- ceph-users - lists.ceph.io

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Reto Gysi 29 Jan '26

29 Jan '26

Hi Boris, I had a similiar problem in the past (I don't remember which ceph release it was) that the block.db links in /var/lib/ceph/<UUID>/osd.N weren't set correctly by the ceph-volume activate script call in the containers unit.run script. The problem was that I had the block.db on a lvm raid1 mirror on SSDs (so a failure of a single SSD doesn't take down multiple OSDs), and the "ceph-volume activate" script call couldn't properly handle this case. It took one of the entries that had the lv_role private instead of public For example: LV Devices Role ceph-db-osd1 ceph-db-osd1_rimage_0(0),ceph-db-osd1_rimage_1(0) public [ceph-db-osd1_rimage_0] ceph-db-osd1_rimage_0_iorig(0) private,raid,image [ceph-db-osd1_rimage_0_imeta] /dev/sdg(8314) private,integrity,metadata [ceph-db-osd1_rimage_0_iorig] /dev/sdg(9216) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_0_iorig] /dev/sdg(82518) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_0_iorig] /dev/sdg(55297) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_0_iorig] /dev/sdg(59888) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_0_iorig] /dev/sdg(62448) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_1] ceph-db-osd1_rimage_1_iorig(0) private,raid,image [ceph-db-osd1_rimage_1_imeta] /dev/sdh(91281) private,integrity,metadata [ceph-db-osd1_rimage_1_iorig] /dev/sdh(1) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_1_iorig] /dev/sdh(84486) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_1_iorig] /dev/sdh(89094) private,integrity,origin,integrityorigin [ceph-db-osd1_rmeta_0] /dev/sdg(46080) private,raid,metadata [ceph-db-osd1_rmeta_1] /dev/sdh(0) private,raid,metadata I think I fixed it by manually changing the ceph-volume activate call in the unit.run to ceph-volume lvm activate ceph-volume lvm activate <ID> <FSID>. Luckily my cluster contains not that many OSDs. With ceph tentacle release I no longer had to manually change the unit.run script call. Cheers, Reto Am Di., 27. Jan. 2026 um 11:55 Uhr schrieb Boris via ceph-users < ceph-users(a)ceph.io>: > Hi, > we are currently facing an issue, that suddenly none of the OSDs will start > after the container started with the new versions. > > This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, > but took really long (like 5 hours) and then every single OSD after that > needed a host reboot to bring the disk back up and continue the update. > > We've stopped after 6 tries. > > And one disk never came back up. We removed and zapped the OSD. The > orchestrator picked the available disk and recreated it. It came up within > seconds. > > We have around 90 clusters and this happened only on a single one. All > others updates within two hours without any issues. > > The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per SSD). > The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of > output from udevadm, lsblk and nsenter > The activation container (ceph-UUID-osd-N-activate) gets killed after a > couple of minutes. > It also looks like the block and block.db links > in /var/lib/ceph/UUID/osd.N/ are not correctly set. > When we restart the daemons that needed a host restart, the OSD doesn't > come up and needs a host restart. > > All OSDs are encrypted. > > Does anyone got some ideas how to debug further? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 29 Jan '26

29 Jan '26

Thanks for chiming in, Anthony. I'm already in contact with Adam who helped unblocking my address from receiving emails. I do receive all mails to the list in my inbox(es), but my responses from the first subscribed address don't land in my inbox. I was wondering why my responses seemed to be ignored until I realized this. But I hope that Adam can help me out again. And yes, this happens to newly created OSDs. I'm not sure if commenting on the open (but resolved) tracker is the right place, I wonder if I should create a separate tracker for this. Since it's production relevant, I would like to get as much attention as possible on that matter. Am Do., 29. Jan. 2026 um 17:12 Uhr schrieb Anthony D'Atri via ceph-users < ceph-users(a)ceph.io>: > > > > Hi, it appears that my mails to the lists are only visible on > lists.ceph.io > > but are not sent as emails, so I'm resending from a different account. > > The ongoing relocation of the Sepia lab seems to have disrupted some > member settings. There’s one for whether a given member address receives > list email, you might auth to the list interface and check yours. I am > subscribed with two addresses because sometimes I goof and send from an > address that I’ve been slowly deprecating, but I don’t want duplicate > delivery, so I have delivery turned off for the latter. If an audit of > settings doesn’t improve your experience, let me know and I’ll forward to > the community manager. > > > Each newly created OSD process grows in RAM usage to around 140 GB > within a > > few minutes, > > Yikes! > > > Anyway, the only way to safely create OSDs is one by one, maybe two at > once > > depending on the host's RAM capacity. An automated (unattended) OSD > > deployment is currently not possible. > > This happens to *new* OSDs? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Anthony D'Atri 29 Jan '26

29 Jan '26

> Hi, it appears that my mails to the lists are only visible on lists.ceph.io > but are not sent as emails, so I'm resending from a different account. The ongoing relocation of the Sepia lab seems to have disrupted some member settings. There’s one for whether a given member address receives list email, you might auth to the list interface and check yours. I am subscribed with two addresses because sometimes I goof and send from an address that I’ve been slowly deprecating, but I don’t want duplicate delivery, so I have delivery turned off for the latter. If an audit of settings doesn’t improve your experience, let me know and I’ll forward to the community manager. > Each newly created OSD process grows in RAM usage to around 140 GB within a > few minutes, Yikes! > Anyway, the only way to safely create OSDs is one by one, maybe two at once > depending on the host's RAM capacity. An automated (unattended) OSD > deployment is currently not possible. This happens to *new* OSDs?

1 0

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Eugen Block 29 Jan '26

29 Jan '26

I was thinking about the same bug you commented, Boris: https://tracker.ceph.com/issues/73107 <https://tracker.ceph.com/issues/73107#change-331393> I am also subscribed to that bug because we upgraded to 19.2.3 a couple of months ago. But since we don’t have that many OSDs per host, we weren’t facing any impact as described in the tracker. Boris via ceph-users <ceph-users(a)ceph.io> schrieb am Mi. 28. Jan. 2026 um 22:19: > Hi Malte, > we just upped the timeout in the service file to 720 (cronjob is going to > set it every minute). > Starting the OSDs takes around 4 minutes. We still think this is an issue > with the ceph-volume activate, because this is taking ages. As soon as the > cryptsetup open the luks devices the rest is going as normal. > > ceph-volume log seems to reflect that: > root@s3db15:~# ls -alh > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log* > -rw-r--r-- 1 root root 2.4G Jan 27 15:59 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log > -rw-r--r-- 1 root root 680M Jan 27 00:00 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.1.gz > -rw-r--r-- 1 root root 5.0M Jan 25 23:46 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.2.gz > -rw-r--r-- 1 root root 4.8M Jan 24 23:35 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.3.gz > -rw-r--r-- 1 root root 4.8M Jan 23 23:45 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.4.gz > -rw-r--r-- 1 root root 4.8M Jan 22 23:38 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.5.gz > -rw-r--r-- 1 root root 4.8M Jan 21 23:39 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.6.gz > -rw-r--r-- 1 root root 4.8M Jan 20 23:38 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.7.gz > --- > # time systemctl restart ceph-dca79fff-ffd0-58f4-1cff-82a2feea05f4(a)osd.135 > > real 3m46.251s > user 0m0.060s > sys 0m0.063s > --- > I will prepare an output of the ceph-collect from 42on. But I need to > remove the sensitive stuff like the rgw config and so. But I don't think > this will give the right clues, as the output won't show what is happening > during the start. > > > Am Di., 27. Jan. 2026 um 13:51 Uhr schrieb Malte Stroem < > malte.stroem(a)gmail.com>: > > > We do not know a lot about you cluster, so it's hard to help. > > > > Give us alle the information one needs. > > > > ceph -s, ceph orch host ls, ceph health detail, all the good things to > > get an overview. > > > > On 1/27/26 12:43, Boris via ceph-users wrote: > > > All disks are 8TB HDD. We have some 16TB HDDs but those are all in the > > > newest host, that updated just fine. > > > > > > Am Di., 27. Jan. 2026 um 12:39 Uhr schrieb Malte Stroem < > > > malte.stroem(a)gmail.com>: > > > > > >> From what I can see, the OSD is running. It's starting up and still > > >> needs time. > > >> > > >> How big is the disk? > > >> > > >> On 1/27/26 12:31, Boris via ceph-users wrote: > > >>> Sure: https://pastebin.com/9RLzyUQs > > >>> > > >>> I've trimmed the log a little bit (removed peering, epoch, trim and > so > > >> on). > > >>> This is the last OSD that we tried that did not work. > > >>> > > >>> We tried another host, where the upgrade just went through. But this > > Host > > >>> also got the newest hardware. > > >>> But we don't think it is a hardware issue, because the first 30 OSDs > > were > > >>> on the two oldest hosts and the first one that failed was on the same > > >> host > > >>> as the last OSD that did not fail. > > >>> > > >>> > > >>> > > >>> Am Di., 27. Jan. 2026 um 12:05 Uhr schrieb Malte Stroem < > > >>> malte.stroem(a)gmail.com>: > > >>> > > >>>> Could be the kind of hardware you are using. Is it different from > the > > >>>> other clusters' hardware? > > >>>> > > >>>> Send us logs, so we can help you out. > > >>>> > > >>>> Example: > > >>>> > > >>>> journalctl -eu ceph-fsid(a)osd.0 > > >>>> > > >>>> Best, > > >>>> Malte > > >>>> > > >>>> On 1/27/26 11:55, Boris via ceph-users wrote: > > >>>>> Hi, > > >>>>> we are currently facing an issue, that suddenly none of the OSDs > will > > >>>> start > > >>>>> after the container started with the new versions. > > >>>>> > > >>>>> This seems to be an issue with some hosts/OSDs. The first 30 OSDs > > >> worked, > > >>>>> but took really long (like 5 hours) and then every single OSD after > > >> that > > >>>>> needed a host reboot to bring the disk back up and continue the > > update. > > >>>>> > > >>>>> We've stopped after 6 tries. > > >>>>> > > >>>>> And one disk never came back up. We removed and zapped the OSD. The > > >>>>> orchestrator picked the available disk and recreated it. It came up > > >>>> within > > >>>>> seconds. > > >>>>> > > >>>>> We have around 90 clusters and this happened only on a single one. > > All > > >>>>> others updates within two hours without any issues. > > >>>>> > > >>>>> The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db > per > > >>>> SSD). > > >>>>> The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot > > of > > >>>>> output from udevadm, lsblk and nsenter > > >>>>> The activation container (ceph-UUID-osd-N-activate) gets killed > > after a > > >>>>> couple of minutes. > > >>>>> It also looks like the block and block.db links > > >>>>> in /var/lib/ceph/UUID/osd.N/ are not correctly set. > > >>>>> When we restart the daemons that needed a host restart, the OSD > > doesn't > > >>>>> come up and needs a host restart. > > >>>>> > > >>>>> All OSDs are encrypted. > > >>>>> > > >>>>> Does anyone got some ideas how to debug further? > > >>>>> _______________________________________________ > > >>>>> ceph-users mailing list -- ceph-users(a)ceph.io > > >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >>>> > > >>>> > > >>> > > >> > > >> > > > > > > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 29 Jan '26

29 Jan '26

Hi, it appears that my mails to the lists are only visible on lists.ceph.io but are not sent as emails, so I'm resending from a different account. This is kind of a follow-up on a two years old thread [0] and I wanted to raise some awareness for the corresponding tracker [1]. Back then we managed to limit the impact on the mon store performance with some paxos configs, but now the OSDs are impacted as well when there are new ones created: Each newly created OSD process grows in RAM usage to around 140 GB within a few minutes, easily causing oom killers on hosts if multiple OSDs are created at once. The residual RAM usage drops to the memory target after it successfully booted. The reason are the purged_snaps that are loaded during OSD boot (snap_mapper.record_purged_snaps purged_snaps), two years ago the customer had more than 42 million purged_snap entries. I don't know how many there are today, I don't have access myself, but I'll try to get a current number. Anyway, the only way to safely create OSDs is one by one, maybe two at once depending on the host's RAM capacity. An automated (unattended) OSD deployment is currently not possible. Unfortunately, the new command [2] doesn't seem to work as expected, at least in my test cluster it didn't have any impact on the number of purged_snaps in the mon store. That's why we haven't tried it on the customer cluster(s) yet. How do other operators/admins/users deal with this kind of scenario? Having many snapshots can't be a corner case, but I can't remember having read anything like this on the list(s). I'd appreciate any comments Thanks! Eugen

1 0

[MDS crash / ceph_assert failure] CephFS MDS crash on Reef 18.2.2 (Filer::_probed assertion)
by Андрей Муханов 29 Jan '26

29 Jan '26

1 0

Re: Adding new osds on new host
by Paul Mezzanini 28 Jan '26

28 Jan '26

I disagree with your blanket statement of "don't do that". Do what fits your needs, skills, and environment. Use the right tools for the job, and if that is a containerized environment then cephadm all the way. If you have the tooling in place to run a package based install and want to be "closer" to the hardware, then do that instead. -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology ________________________________________ From: Malte Stroem via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, January 28, 2026 9:46 AM To: Anthony D'Atri; Malte Stroem Cc: Rok Jaklič; ceph-users Subject: [ceph-users] Re: Adding new osds on new host Yes, don't do that. Use cephadm. On 1/28/26 14:26, Anthony D'Atri via ceph-users wrote: > The OP may be using a package-based install. > >> On Jan 28, 2026, at 3:14 AM, Malte Stroem via ceph-users <ceph-users(a)ceph.io> wrote: >> >> Use cephadm, "ceph orch" commands to do this. >> >>> On 1/27/26 20:51, Rok Jaklič via ceph-users wrote: >>> Hi, >>> i am adding new OSDs from a new host with something like: >>> <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha >>> Running command: /usr/bin/ceph-authtool --gen-print-key >>> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >>> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >>> dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> Running command: vgcreate --force --yes >>> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha >>> stdout: Physical volume "/dev/mapper/mpatha" successfully created. >>> stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" >>> successfully created >>> Running command: lvcreate --yes -l 3814911 -n >>> osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 >>> stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" >>> created. >>> Running command: /usr/bin/ceph-authtool --gen-print-key >>> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/chown -h ceph:ceph >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >>> Running command: /usr/bin/ln -s >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >>> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o >>> /var/lib/ceph/osd/ceph-233/activate.monmap >>> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >>> a keyring on >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >>> (2) No such file or directory >>> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >>> no keyring found at >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >>> disabling cephx >>> stderr: got monmap epoch 6 >>> --> Creating keyring file for osd.233 >>> Running command: /usr/bin/chown -R ceph:ceph >>> /var/lib/ceph/osd/ceph-233/keyring >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ >>> Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore >>> bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap >>> --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid >>> dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >>> --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir >>> --dev >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> --path /var/lib/ceph/osd/ceph-233 --no-mon-config >>> Running command: /usr/bin/ln -snf >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/chown -h ceph:ceph >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/systemctl enable >>> ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> stderr: Created symlink >>> /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service >>> → /usr/lib/systemd/system/ceph-volume@.service. >>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 >>> stderr: Created symlink >>> /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → >>> /usr/lib/systemd/system/ceph-osd@.service. >>> Running command: /usr/bin/systemctl start ceph-osd@233 >>> --> ceph-volume lvm activate successful for osd ID: 233 >>> --> ceph-volume lvm create successful for: /dev/mapper/mpatha >>> --- >>> I get few errors in output like: >>> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >>> a keyring on >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >>> (2) No such file or directory >>> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >>> no keyring found at >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >>> disabling cephx >>> ... and then ... >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >>> ... but cluster seems to add and work on new OSDs anyway. >>> Anyone had a similar issue? Or can I ignore this error? >>> Kind regards, >>> Rok >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Adding new osds on new host
by Malte Stroem 28 Jan '26

28 Jan '26

Yes, don't do that. Use cephadm. On 1/28/26 14:26, Anthony D'Atri via ceph-users wrote: > The OP may be using a package-based install. > >> On Jan 28, 2026, at 3:14 AM, Malte Stroem via ceph-users <ceph-users(a)ceph.io> wrote: >> >> Use cephadm, "ceph orch" commands to do this. >> >>> On 1/27/26 20:51, Rok Jaklič via ceph-users wrote: >>> Hi, >>> i am adding new OSDs from a new host with something like: >>> <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha >>> Running command: /usr/bin/ceph-authtool --gen-print-key >>> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >>> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >>> dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> Running command: vgcreate --force --yes >>> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha >>> stdout: Physical volume "/dev/mapper/mpatha" successfully created. >>> stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" >>> successfully created >>> Running command: lvcreate --yes -l 3814911 -n >>> osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 >>> stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" >>> created. >>> Running command: /usr/bin/ceph-authtool --gen-print-key >>> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/chown -h ceph:ceph >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >>> Running command: /usr/bin/ln -s >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >>> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o >>> /var/lib/ceph/osd/ceph-233/activate.monmap >>> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >>> a keyring on >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >>> (2) No such file or directory >>> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >>> no keyring found at >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >>> disabling cephx >>> stderr: got monmap epoch 6 >>> --> Creating keyring file for osd.233 >>> Running command: /usr/bin/chown -R ceph:ceph >>> /var/lib/ceph/osd/ceph-233/keyring >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ >>> Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore >>> bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap >>> --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid >>> dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >>> --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir >>> --dev >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> --path /var/lib/ceph/osd/ceph-233 --no-mon-config >>> Running command: /usr/bin/ln -snf >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/chown -h ceph:ceph >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/systemctl enable >>> ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> stderr: Created symlink >>> /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service >>> → /usr/lib/systemd/system/ceph-volume@.service. >>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 >>> stderr: Created symlink >>> /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → >>> /usr/lib/systemd/system/ceph-osd@.service. >>> Running command: /usr/bin/systemctl start ceph-osd@233 >>> --> ceph-volume lvm activate successful for osd ID: 233 >>> --> ceph-volume lvm create successful for: /dev/mapper/mpatha >>> --- >>> I get few errors in output like: >>> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >>> a keyring on >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >>> (2) No such file or directory >>> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >>> no keyring found at >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >>> disabling cephx >>> ... and then ... >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >>> ... but cluster seems to add and work on new OSDs anyway. >>> Anyone had a similar issue? Or can I ignore this error? >>> Kind regards, >>> Rok >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Adding new osds on new host
by Anthony D'Atri 28 Jan '26

28 Jan '26

The OP may be using a package-based install. > On Jan 28, 2026, at 3:14 AM, Malte Stroem via ceph-users <ceph-users(a)ceph.io> wrote: > > Use cephadm, "ceph orch" commands to do this. > >> On 1/27/26 20:51, Rok Jaklič via ceph-users wrote: >> Hi, >> i am adding new OSDs from a new host with something like: >> <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha >> Running command: /usr/bin/ceph-authtool --gen-print-key >> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >> dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> Running command: vgcreate --force --yes >> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha >> stdout: Physical volume "/dev/mapper/mpatha" successfully created. >> stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" >> successfully created >> Running command: lvcreate --yes -l 3814911 -n >> osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 >> stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" >> created. >> Running command: /usr/bin/ceph-authtool --gen-print-key >> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 >> Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 >> Running command: /usr/bin/chown -h ceph:ceph >> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >> Running command: /usr/bin/ln -s >> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> /var/lib/ceph/osd/ceph-233/block >> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o >> /var/lib/ceph/osd/ceph-233/activate.monmap >> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >> a keyring on >> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >> (2) No such file or directory >> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >> no keyring found at >> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >> disabling cephx >> stderr: got monmap epoch 6 >> --> Creating keyring file for osd.233 >> Running command: /usr/bin/chown -R ceph:ceph >> /var/lib/ceph/osd/ceph-233/keyring >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ >> Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore >> bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap >> --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid >> dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >> --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir >> --dev >> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> --path /var/lib/ceph/osd/ceph-233 --no-mon-config >> Running command: /usr/bin/ln -snf >> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> /var/lib/ceph/osd/ceph-233/block >> Running command: /usr/bin/chown -h ceph:ceph >> /var/lib/ceph/osd/ceph-233/block >> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >> Running command: /usr/bin/systemctl enable >> ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> stderr: Created symlink >> /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service >> → /usr/lib/systemd/system/ceph-volume@.service. >> Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 >> stderr: Created symlink >> /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → >> /usr/lib/systemd/system/ceph-osd@.service. >> Running command: /usr/bin/systemctl start ceph-osd@233 >> --> ceph-volume lvm activate successful for osd ID: 233 >> --> ceph-volume lvm create successful for: /dev/mapper/mpatha >> --- >> I get few errors in output like: >> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >> a keyring on >> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >> (2) No such file or directory >> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >> no keyring found at >> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >> disabling cephx >> ... and then ... >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >> ... but cluster seems to add and work on new OSDs anyway. >> Anyone had a similar issue? Or can I ignore this error? >> Kind regards, >> Rok >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: upmap-read balancer with CephFS kernel clients
by Janek Bevendorff 28 Jan '26

28 Jan '26

All right, I found a way by using rbd-nbd. However, for that I need to recreate all my Kubernetes StorageClasses, because the parameter fields are immutable. Not ideal. On 28/01/2026 09:17, Janek Bevendorff via ceph-users wrote: > I just noticed that this applies to RBD clients as well. How is this > supposed to work? Can you use only radosgw and ceph-fuse if you want > to use upmap-read? It seems strange to me that the most central Ceph > systems don't work with these new features. > > > On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: >> Hi, >> >> Is there a way to use the new upmap-read balancer mode when I have >> CephFS kernel clients connected? Our cluster runs Squid and all >> connected clients are on kernel 6.11-6.14, so they should be new >> enough. However, they identify themselves as luminous (I could not >> find clear documentation about this, but this seems to be hard-coded >> and by design), so I cannot run ceph osd >> set-require-min-compat-client reef, which is required for the >> balancer module to work. >> >> I have 500 clients connected and migrating them all to FUSE would be >> a headache. Is there a way around that? Some of my pools have >> extremely high read balance scores and we're also struggling with >> performance issues as a result. >> >> Janek >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de

1 0