Hi Sönke, have you verified the OSD's metadata? osd metadata 0 -f json | jq -r '.bluefs_dedicated_db,.devices' And I would also check if the labels are correct: cephadm shell -- ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block | grep osd_key We wrote two blog posts ([0],[1]) about rocksDB migration, maybe they can help you figure out what's wrong with the OSD(s). My suspicion is that the LV tags were not set properly after the migration, but it might be something else, of course. Regards, Eugen [0] https://heiterbiswolkig.blogs.nde.ag/2018/04/08/migrating-bluestores-block-d... [1] https://heiterbiswolkig.blogs.nde.ag/2025/02/05/cephadm-migrate-db-wal-to-ne... Zitat von Soenke Schippmann <schippmann@uni-bremen.de>:
Hi,
after upgrading one of our ceph clusters from 18.2.7 to 19.2.3 some OSDs fail to start. For these OSDs, db devices were moved manually months ago from a partition to a lvm volume.
OSD log shows:
2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 set uid:gid to 167:167 (ceph:ceph) 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 7 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 pidfile_write: ignore empty --pid-file 2025-09-04T11:38:22.055+0000 7fec1bbc4740 1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open path /var/lib/ceph/osd/ceph-256/block 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open stat got: (1) Operation not permitted 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-256: (2) No such file or directory
Links to block and block.db within the osd path get deleted after each startup attempt. Recreating the links manually does not help.
ceph-bluestore-tool fsck --path... shows no errors if links to block and block.db are recreated.
Running "ceph-volume activate --osd-id 256" manually within cephadm shell fails with the follwing error:
--> Failed to activate via LVM: could not find db with uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
ceph-volume lvm list shows outdated db uuid and db device:
===== osd.256 ======
[block] /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0
block device /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block uuid GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy cephx lockbox secret cluster fsid 1f3b3198-08b1-418c-a279-7050a2eb1ce3 cluster name ceph crush device class None db device /dev/sdai1 db uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f encrypted 0 osd fsid 93811afc-17a3-4458-8e00-506eb9c92cb0 osd id 256 type block vdo 0 devices /dev/sdc
[db] /dev/sdai1
PARTUUID 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
db was migrated from partition /dev/sdai1 to lvm volume ceph-blockdb-01/osd-db-01 on /dev/sdaa months ago and running fine with Ceph 18.2. Migration was done manually by using "ceph-bluestore-tool bluefs-bdev-migrate" (ceph-volume lvm migrate failed though).
Is there any way to fix this?
Best, Sönke
-- Sönke Schippmann
Universität Bremen Dezernat 8 - IT Service Center Referat 82 Serverbetrieb
Büroanschrift: Universität Bremen Dez. 8-Bi, SFG 1390 Enrique-Schmidt-Str. 7 28359 Bremen
E-Mail: schippmann@uni-bremen.de Tel: +49 421 218-61327 Fax: +49 421 218-98-61327
http://www.uni-bremen.de/zfn/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io