Hi, thank you, guys. Correcting LV tags on the block LV and setting those on the db volume accordingly did the job: lvchange --deltag "ceph.db_device=/dev/sdai1" ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 lvchange --deltag "ceph.db_uuid=6d676bcd-1f3c-e740-8fdf-6a5156605a3f" ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 lvchange --addtag "ceph.db_device=/dev/ceph-blockdb-01/osd-db-01" ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 lvchange --addtag "ceph.db_uuid=LVKxCf-teCI-1k6G-NamB-7A8o-l0h7-ORHHgx" ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 lvchange --addtag "ceph.block_device=/dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.block_uuid=GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.cephx_lockbox_secret=" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.cluster_fsid=1f3b3198-08b1-418c-a279-7050a2eb1ce3" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.cluster_name=ceph" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.crush_device_class=" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.db_device=/dev/ceph-blockdb-01/osd-db-01" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.db_uuid=LVKxCf-teCI-1k6G-NamB-7A8o-l0h7-ORHHgx" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.encrypted=0" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.osd_fsid=93811afc-17a3-4458-8e00-506eb9c92cb0" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.osd_id=256" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.type=db" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.vdo=0" ceph-blockdb-01/osd-db-01 Best, Sönke Am 04.09.25 um 19:17 schrieb Igor Fedotov:
Hi Shoenke,
Migrating DB volume using ceph-bluestore-tool was a wrong step. It doesn't setup LV tags for underlying volumes which prevents preper OSD devices detection after reboot.
One should set these tags manually using lvchange --addtag command. To a major degree DB tags are similar to ones for the block device. But some additional tuning is still required.
Unfortunately AFAIK there is no full how-to-do manual available. One of Eugene's link covers the topic just partly. So you should rather use existing valid OSD deployment as a reference.
Thanks,
Igor
On 9/4/2025 3:55 PM, Soenke Schippmann wrote:
Hi,
after upgrading one of our ceph clusters from 18.2.7 to 19.2.3 some OSDs fail to start. For these OSDs, db devices were moved manually months ago from a partition to a lvm volume.
OSD log shows:
2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 set uid:gid to 167:167 (ceph:ceph) 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 7 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 pidfile_write: ignore empty --pid-file 2025-09-04T11:38:22.055+0000 7fec1bbc4740 1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open path /var/lib/ceph/osd/ceph-256/block 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open stat got: (1) Operation not permitted 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-256: (2) No such file or directory
Links to block and block.db within the osd path get deleted after each startup attempt. Recreating the links manually does not help.
ceph-bluestore-tool fsck --path... shows no errors if links to block and block.db are recreated.
Running "ceph-volume activate --osd-id 256" manually within cephadm shell fails with the follwing error:
--> Failed to activate via LVM: could not find db with uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
ceph-volume lvm list shows outdated db uuid and db device:
===== osd.256 ======
[block] /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0
block device /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block uuid GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy cephx lockbox secret cluster fsid 1f3b3198-08b1-418c-a279-7050a2eb1ce3 cluster name ceph crush device class None db device /dev/sdai1 db uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f encrypted 0 osd fsid 93811afc-17a3-4458-8e00-506eb9c92cb0 osd id 256 type block vdo 0 devices /dev/sdc
[db] /dev/sdai1
PARTUUID 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
db was migrated from partition /dev/sdai1 to lvm volume ceph-blockdb-01/osd-db-01 on /dev/sdaa months ago and running fine with Ceph 18.2. Migration was done manually by using "ceph-bluestore-tool bluefs-bdev-migrate" (ceph-volume lvm migrate failed though).
Is there any way to fix this?
Best, Sönke
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sönke Schippmann Universität Bremen Dezernat 8 - IT Service Center Referat 82 Serverbetrieb Büroanschrift: Universität Bremen Dez. 8-Bi, SFG 1390 Enrique-Schmidt-Str. 7 28359 Bremen E-Mail: schippmann@uni-bremen.de Tel: +49 421 218-61327 Fax: +49 421 218-98-61327 http://www.uni-bremen.de/zfn/