Ceph OSD with migrated db device fails after upgrade to 19.2.3 from 18.2.7
Hi, after upgrading one of our ceph clusters from 18.2.7 to 19.2.3 some OSDs fail to start. For these OSDs, db devices were moved manually months ago from a partition to a lvm volume. OSD log shows: 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 set uid:gid to 167:167 (ceph:ceph) 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 7 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 pidfile_write: ignore empty --pid-file 2025-09-04T11:38:22.055+0000 7fec1bbc4740 1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open path /var/lib/ceph/osd/ceph-256/block 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open stat got: (1) Operation not permitted 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-256: (2) No such file or directory Links to block and block.db within the osd path get deleted after each startup attempt. Recreating the links manually does not help. ceph-bluestore-tool fsck --path... shows no errors if links to block and block.db are recreated. Running "ceph-volume activate --osd-id 256" manually within cephadm shell fails with the follwing error: --> Failed to activate via LVM: could not find db with uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f ceph-volume lvm list shows outdated db uuid and db device: ===== osd.256 ====== [block] /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block device /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block uuid GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy cephx lockbox secret cluster fsid 1f3b3198-08b1-418c-a279-7050a2eb1ce3 cluster name ceph crush device class None db device /dev/sdai1 db uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f encrypted 0 osd fsid 93811afc-17a3-4458-8e00-506eb9c92cb0 osd id 256 type block vdo 0 devices /dev/sdc [db] /dev/sdai1 PARTUUID 6d676bcd-1f3c-e740-8fdf-6a5156605a3f db was migrated from partition /dev/sdai1 to lvm volume ceph-blockdb-01/osd-db-01 on /dev/sdaa months ago and running fine with Ceph 18.2. Migration was done manually by using "ceph-bluestore-tool bluefs-bdev-migrate" (ceph-volume lvm migrate failed though). Is there any way to fix this? Best, Sönke -- Sönke Schippmann Universität Bremen Dezernat 8 - IT Service Center Referat 82 Serverbetrieb Büroanschrift: Universität Bremen Dez. 8-Bi, SFG 1390 Enrique-Schmidt-Str. 7 28359 Bremen E-Mail: schippmann@uni-bremen.de Tel: +49 421 218-61327 Fax: +49 421 218-98-61327 http://www.uni-bremen.de/zfn/
Hi Sönke, have you verified the OSD's metadata? osd metadata 0 -f json | jq -r '.bluefs_dedicated_db,.devices' And I would also check if the labels are correct: cephadm shell -- ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block | grep osd_key We wrote two blog posts ([0],[1]) about rocksDB migration, maybe they can help you figure out what's wrong with the OSD(s). My suspicion is that the LV tags were not set properly after the migration, but it might be something else, of course. Regards, Eugen [0] https://heiterbiswolkig.blogs.nde.ag/2018/04/08/migrating-bluestores-block-d... [1] https://heiterbiswolkig.blogs.nde.ag/2025/02/05/cephadm-migrate-db-wal-to-ne... Zitat von Soenke Schippmann <schippmann@uni-bremen.de>:
Hi,
after upgrading one of our ceph clusters from 18.2.7 to 19.2.3 some OSDs fail to start. For these OSDs, db devices were moved manually months ago from a partition to a lvm volume.
OSD log shows:
2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 set uid:gid to 167:167 (ceph:ceph) 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 7 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 pidfile_write: ignore empty --pid-file 2025-09-04T11:38:22.055+0000 7fec1bbc4740 1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open path /var/lib/ceph/osd/ceph-256/block 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open stat got: (1) Operation not permitted 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-256: (2) No such file or directory
Links to block and block.db within the osd path get deleted after each startup attempt. Recreating the links manually does not help.
ceph-bluestore-tool fsck --path... shows no errors if links to block and block.db are recreated.
Running "ceph-volume activate --osd-id 256" manually within cephadm shell fails with the follwing error:
--> Failed to activate via LVM: could not find db with uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
ceph-volume lvm list shows outdated db uuid and db device:
===== osd.256 ======
[block] /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0
block device /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block uuid GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy cephx lockbox secret cluster fsid 1f3b3198-08b1-418c-a279-7050a2eb1ce3 cluster name ceph crush device class None db device /dev/sdai1 db uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f encrypted 0 osd fsid 93811afc-17a3-4458-8e00-506eb9c92cb0 osd id 256 type block vdo 0 devices /dev/sdc
[db] /dev/sdai1
PARTUUID 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
db was migrated from partition /dev/sdai1 to lvm volume ceph-blockdb-01/osd-db-01 on /dev/sdaa months ago and running fine with Ceph 18.2. Migration was done manually by using "ceph-bluestore-tool bluefs-bdev-migrate" (ceph-volume lvm migrate failed though).
Is there any way to fix this?
Best, Sönke
-- Sönke Schippmann
Universität Bremen Dezernat 8 - IT Service Center Referat 82 Serverbetrieb
Büroanschrift: Universität Bremen Dez. 8-Bi, SFG 1390 Enrique-Schmidt-Str. 7 28359 Bremen
E-Mail: schippmann@uni-bremen.de Tel: +49 421 218-61327 Fax: +49 421 218-98-61327
http://www.uni-bremen.de/zfn/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Shoenke, Migrating DB volume using ceph-bluestore-tool was a wrong step. It doesn't setup LV tags for underlying volumes which prevents preper OSD devices detection after reboot. One should set these tags manually using lvchange --addtag command. To a major degree DB tags are similar to ones for the block device. But some additional tuning is still required. Unfortunately AFAIK there is no full how-to-do manual available. One of Eugene's link covers the topic just partly. So you should rather use existing valid OSD deployment as a reference. Thanks, Igor On 9/4/2025 3:55 PM, Soenke Schippmann wrote:
Hi,
after upgrading one of our ceph clusters from 18.2.7 to 19.2.3 some OSDs fail to start. For these OSDs, db devices were moved manually months ago from a partition to a lvm volume.
OSD log shows:
2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 set uid:gid to 167:167 (ceph:ceph) 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 7 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 pidfile_write: ignore empty --pid-file 2025-09-04T11:38:22.055+0000 7fec1bbc4740 1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open path /var/lib/ceph/osd/ceph-256/block 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open stat got: (1) Operation not permitted 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-256: (2) No such file or directory
Links to block and block.db within the osd path get deleted after each startup attempt. Recreating the links manually does not help.
ceph-bluestore-tool fsck --path... shows no errors if links to block and block.db are recreated.
Running "ceph-volume activate --osd-id 256" manually within cephadm shell fails with the follwing error:
--> Failed to activate via LVM: could not find db with uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
ceph-volume lvm list shows outdated db uuid and db device:
===== osd.256 ======
[block] /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0
block device /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block uuid GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy cephx lockbox secret cluster fsid 1f3b3198-08b1-418c-a279-7050a2eb1ce3 cluster name ceph crush device class None db device /dev/sdai1 db uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f encrypted 0 osd fsid 93811afc-17a3-4458-8e00-506eb9c92cb0 osd id 256 type block vdo 0 devices /dev/sdc
[db] /dev/sdai1
PARTUUID 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
db was migrated from partition /dev/sdai1 to lvm volume ceph-blockdb-01/osd-db-01 on /dev/sdaa months ago and running fine with Ceph 18.2. Migration was done manually by using "ceph-bluestore-tool bluefs-bdev-migrate" (ceph-volume lvm migrate failed though).
Is there any way to fix this?
Best, Sönke
Hi, thank you, guys. Correcting LV tags on the block LV and setting those on the db volume accordingly did the job: lvchange --deltag "ceph.db_device=/dev/sdai1" ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 lvchange --deltag "ceph.db_uuid=6d676bcd-1f3c-e740-8fdf-6a5156605a3f" ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 lvchange --addtag "ceph.db_device=/dev/ceph-blockdb-01/osd-db-01" ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 lvchange --addtag "ceph.db_uuid=LVKxCf-teCI-1k6G-NamB-7A8o-l0h7-ORHHgx" ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 lvchange --addtag "ceph.block_device=/dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.block_uuid=GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.cephx_lockbox_secret=" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.cluster_fsid=1f3b3198-08b1-418c-a279-7050a2eb1ce3" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.cluster_name=ceph" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.crush_device_class=" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.db_device=/dev/ceph-blockdb-01/osd-db-01" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.db_uuid=LVKxCf-teCI-1k6G-NamB-7A8o-l0h7-ORHHgx" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.encrypted=0" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.osd_fsid=93811afc-17a3-4458-8e00-506eb9c92cb0" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.osd_id=256" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.type=db" ceph-blockdb-01/osd-db-01 lvchange --addtag "ceph.vdo=0" ceph-blockdb-01/osd-db-01 Best, Sönke Am 04.09.25 um 19:17 schrieb Igor Fedotov:
Hi Shoenke,
Migrating DB volume using ceph-bluestore-tool was a wrong step. It doesn't setup LV tags for underlying volumes which prevents preper OSD devices detection after reboot.
One should set these tags manually using lvchange --addtag command. To a major degree DB tags are similar to ones for the block device. But some additional tuning is still required.
Unfortunately AFAIK there is no full how-to-do manual available. One of Eugene's link covers the topic just partly. So you should rather use existing valid OSD deployment as a reference.
Thanks,
Igor
On 9/4/2025 3:55 PM, Soenke Schippmann wrote:
Hi,
after upgrading one of our ceph clusters from 18.2.7 to 19.2.3 some OSDs fail to start. For these OSDs, db devices were moved manually months ago from a partition to a lvm volume.
OSD log shows:
2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 set uid:gid to 167:167 (ceph:ceph) 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 7 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 pidfile_write: ignore empty --pid-file 2025-09-04T11:38:22.055+0000 7fec1bbc4740 1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open path /var/lib/ceph/osd/ceph-256/block 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open stat got: (1) Operation not permitted 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-256: (2) No such file or directory
Links to block and block.db within the osd path get deleted after each startup attempt. Recreating the links manually does not help.
ceph-bluestore-tool fsck --path... shows no errors if links to block and block.db are recreated.
Running "ceph-volume activate --osd-id 256" manually within cephadm shell fails with the follwing error:
--> Failed to activate via LVM: could not find db with uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
ceph-volume lvm list shows outdated db uuid and db device:
===== osd.256 ======
[block] /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0
block device /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block uuid GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy cephx lockbox secret cluster fsid 1f3b3198-08b1-418c-a279-7050a2eb1ce3 cluster name ceph crush device class None db device /dev/sdai1 db uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f encrypted 0 osd fsid 93811afc-17a3-4458-8e00-506eb9c92cb0 osd id 256 type block vdo 0 devices /dev/sdc
[db] /dev/sdai1
PARTUUID 6d676bcd-1f3c-e740-8fdf-6a5156605a3f
db was migrated from partition /dev/sdai1 to lvm volume ceph-blockdb-01/osd-db-01 on /dev/sdaa months ago and running fine with Ceph 18.2. Migration was done manually by using "ceph-bluestore-tool bluefs-bdev-migrate" (ceph-volume lvm migrate failed though).
Is there any way to fix this?
Best, Sönke
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sönke Schippmann Universität Bremen Dezernat 8 - IT Service Center Referat 82 Serverbetrieb Büroanschrift: Universität Bremen Dez. 8-Bi, SFG 1390 Enrique-Schmidt-Str. 7 28359 Bremen E-Mail: schippmann@uni-bremen.de Tel: +49 421 218-61327 Fax: +49 421 218-98-61327 http://www.uni-bremen.de/zfn/
Migrating DB volume using ceph-bluestore-tool was a wrong step.
What's the alternative? Redeploying an entire cluster of spinners doesn't seem very feasible.
It doesn't setup LV tags for underlying volumes which prevents preper OSD devices detection after reboot.
One should set these tags manually using lvchange --addtag command. To a major degree DB tags are similar to ones for the block device. But some additional tuning is still required.
Unfortunately AFAIK there is no full how-to-do manual available. One of Eugene's link covers the topic just partly. So you should rather use existing valid OSD deployment as a reference.
Seems like justification for a tracker bug.
Migrating DB volume using ceph-bluestore-tool was a wrong step.
What's the alternative? Redeploying an entire cluster of spinners doesn't seem very feasible.
As far as I understand, "ceph-volume lvm migrate" is the (better) supported way for db migrations. In our case that tool didn't work. The db was on a partition, not a LV, and that case seems not to be supported by ceph-volume. Best, Sönke -- Sönke Schippmann Universität Bremen Dezernat 8 - IT Service Center Referat 82 Serverbetrieb Büroanschrift: Universität Bremen Dez. 8-Bi, SFG 1390 Enrique-Schmidt-Str. 7 28359 Bremen E-Mail: schippmann@uni-bremen.de Tel: +49 421 218-61327 Fax: +49 421 218-98-61327 http://www.uni-bremen.de/zfn/
Indeed the only time I've done this myself was from colocated to offloaded WAL+DB.
Migrating DB volume using ceph-bluestore-tool was a wrong step.
What's the alternative? Redeploying an entire cluster of spinners doesn't seem very feasible.
As far as I understand, "ceph-volume lvm migrate" is the (better) supported way for db migrations.
In our case that tool didn't work. The db was on a partition, not a LV, and that case seems not to be supported by ceph-volume.
Best, Sönke
-- Sönke Schippmann
Universität Bremen Dezernat 8 - IT Service Center Referat 82 Serverbetrieb
Büroanschrift: Universität Bremen Dez. 8-Bi, SFG 1390 Enrique-Schmidt-Str. 7 28359 Bremen
E-Mail: schippmann@uni-bremen.de Tel: +49 421 218-61327 Fax: +49 421 218-98-61327
http://www.uni-bremen.de/zfn/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Anthony D'Atri
-
Eugen Block
-
Igor Fedotov
-
Soenke Schippmann