Hi Shoenke, Migrating DB volume using ceph-bluestore-tool was a wrong step. It doesn't setup LV tags for underlying volumes which prevents preper OSD devices detection after reboot. One should set these tags manually using lvchange --addtag command. To a major degree DB tags are similar to ones for the block device. But some additional tuning is still required. Unfortunately AFAIK there is no full how-to-do manual available. One of Eugene's link covers the topic just partly. So you should rather use existing valid OSD deployment as a reference. Thanks, Igor On 9/4/2025 3:55 PM, Soenke Schippmann wrote:
Hi,
after upgrading one of our ceph clusters from 18.2.7 to 19.2.3 some OSDs fail to start. For these OSDs, db devices were moved manually months ago from a partition to a lvm volume.
OSD log shows:
2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 set uid:gid to 167:167 (ceph:ceph) 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 7 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 pidfile_write: ignore empty --pid-file 2025-09-04T11:38:22.055+0000 7fec1bbc4740 1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open path /var/lib/ceph/osd/ceph-256/block 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open stat got: (1) Operation not permitted 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-256: (2) No such file or directory
Links to block and block.db within the osd path get deleted after each startup attempt. Recreating the links manually does not help.
ceph-bluestore-tool fsck --path... shows no errors if links to block and block.db are recreated.
Running "ceph-volume activate --osd-id 256" manually within cephadm shell fails with the follwing error:
--> Failed to activate via LVM: could not find db with uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
ceph-volume lvm list shows outdated db uuid and db device:
===== osd.256 ======
[block] /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0
block device /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block uuid GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy cephx lockbox secret cluster fsid 1f3b3198-08b1-418c-a279-7050a2eb1ce3 cluster name ceph crush device class None db device /dev/sdai1 db uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f encrypted 0 osd fsid 93811afc-17a3-4458-8e00-506eb9c92cb0 osd id 256 type block vdo 0 devices /dev/sdc
[db] /dev/sdai1
PARTUUID 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
db was migrated from partition /dev/sdai1 to lvm volume ceph-blockdb-01/osd-db-01 on /dev/sdaa months ago and running fine with Ceph 18.2. Migration was done manually by using "ceph-bluestore-tool bluefs-bdev-migrate" (ceph-volume lvm migrate failed though).
Is there any way to fix this?
Best, Sönke