- ceph-users - lists.ceph.io

Re: upmap-read balancer with CephFS kernel clients
by Janek Bevendorff 28 Jan '26

28 Jan '26

I just noticed that this applies to RBD clients as well. How is this supposed to work? Can you use only radosgw and ceph-fuse if you want to use upmap-read? It seems strange to me that the most central Ceph systems don't work with these new features. On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: > Hi, > > Is there a way to use the new upmap-read balancer mode when I have > CephFS kernel clients connected? Our cluster runs Squid and all > connected clients are on kernel 6.11-6.14, so they should be new > enough. However, they identify themselves as luminous (I could not > find clear documentation about this, but this seems to be hard-coded > and by design), so I cannot run ceph osd set-require-min-compat-client > reef, which is required for the balancer module to work. > > I have 500 clients connected and migrating them all to FUSE would be a > headache. Is there a way around that? Some of my pools have extremely > high read balance scores and we're also struggling with performance > issues as a result. > > Janek > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de

1 0

Re: Adding new osds on new host
by Malte Stroem 28 Jan '26

28 Jan '26

Use cephadm, "ceph orch" commands to do this. On 1/27/26 20:51, Rok Jaklič via ceph-users wrote: > Hi, > > i am adding new OSDs from a new host with something like: > <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > dbb58525-c4f6-4a1e-b03a-ac5966f09809 > Running command: vgcreate --force --yes > ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha > stdout: Physical volume "/dev/mapper/mpatha" successfully created. > stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" > successfully created > Running command: lvcreate --yes -l 3814911 -n > osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 > stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" > created. > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 > Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 > Running command: /usr/bin/chown -h ceph:ceph > /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 > Running command: /usr/bin/ln -s > /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > /var/lib/ceph/osd/ceph-233/block > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o > /var/lib/ceph/osd/ceph-233/activate.monmap > stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find > a keyring on > /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: > (2) No such file or directory > 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) > no keyring found at > /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, > disabling cephx > stderr: got monmap epoch 6 > --> Creating keyring file for osd.233 > Running command: /usr/bin/chown -R ceph:ceph > /var/lib/ceph/osd/ceph-233/keyring > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ > Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore > bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap > --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid > dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid > --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir > --dev > /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > --path /var/lib/ceph/osd/ceph-233 --no-mon-config > Running command: /usr/bin/ln -snf > /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > /var/lib/ceph/osd/ceph-233/block > Running command: /usr/bin/chown -h ceph:ceph > /var/lib/ceph/osd/ceph-233/block > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 > Running command: /usr/bin/systemctl enable > ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > stderr: Created symlink > /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service > → /usr/lib/systemd/system/ceph-volume@.service. > Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 > stderr: Created symlink > /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → > /usr/lib/systemd/system/ceph-osd@.service. > Running command: /usr/bin/systemctl start ceph-osd@233 > --> ceph-volume lvm activate successful for osd ID: 233 > --> ceph-volume lvm create successful for: /dev/mapper/mpatha > > --- > > I get few errors in output like: > stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find > a keyring on > /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: > (2) No such file or directory > 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) > no keyring found at > /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, > disabling cephx > > ... and then ... > > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid > > ... but cluster seems to add and work on new OSDs anyway. > > Anyone had a similar issue? Or can I ignore this error? > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Eugen Block 27 Jan '26

27 Jan '26

Hi, did you by any chance restart the mgr (or simply fail)? The OSD status is not always correctly reported. From your log output you can see that OSD.200 actually booted successfully after around 3 minutes (as Malte also already pointed out): -- Boot 032bb082802d4003a96b9b2248f0e2eb -- Jan 27 10:08:07 host14 systemd[1]: Starting Ceph osd.200 for dca79fff-ffd0-58f4-1cff-82a2feea05f4... ... Jan 27 10:11:31 host14 ceph-osd[100997]: osd.200 1389134 load_pgs opened 135 pgs Jan 27 10:11:31 host14 ceph-osd[100997]: osd.200 1389134 log_to_monitors true Jan 27 10:11:31 host14 conmon[100991]: 2026-01-27T10:11:31.277+0000 7f541f5dd740 -1 osd.200 1389134 log_to_monitors true Jan 27 10:11:32 host14 ceph-osd[100997]: osd.200 1389134 done with init, starting boot process Jan 27 10:11:32 host14 ceph-osd[100997]: osd.200 1389134 start_boot ... Jan 27 10:11:33 host14 ceph-osd[100997]: osd.200 1389172 state: booting -> active Is there anything missing from the logs? Or is this not the right OSD log? Zitat von Boris via ceph-users <ceph-users(a)ceph.io>: > Sure: https://pastebin.com/9RLzyUQs > > I've trimmed the log a little bit (removed peering, epoch, trim and so on). > This is the last OSD that we tried that did not work. > > We tried another host, where the upgrade just went through. But this Host > also got the newest hardware. > But we don't think it is a hardware issue, because the first 30 OSDs were > on the two oldest hosts and the first one that failed was on the same host > as the last OSD that did not fail. > > > > Am Di., 27. Jan. 2026 um 12:05 Uhr schrieb Malte Stroem < > malte.stroem(a)gmail.com>: > >> Could be the kind of hardware you are using. Is it different from the >> other clusters' hardware? >> >> Send us logs, so we can help you out. >> >> Example: >> >> journalctl -eu ceph-fsid(a)osd.0 >> >> Best, >> Malte >> >> On 1/27/26 11:55, Boris via ceph-users wrote: >> > Hi, >> > we are currently facing an issue, that suddenly none of the OSDs will >> start >> > after the container started with the new versions. >> > >> > This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, >> > but took really long (like 5 hours) and then every single OSD after that >> > needed a host reboot to bring the disk back up and continue the update. >> > >> > We've stopped after 6 tries. >> > >> > And one disk never came back up. We removed and zapped the OSD. The >> > orchestrator picked the available disk and recreated it. It came up >> within >> > seconds. >> > >> > We have around 90 clusters and this happened only on a single one. All >> > others updates within two hours without any issues. >> > >> > The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per >> SSD). >> > The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of >> > output from udevadm, lsblk and nsenter >> > The activation container (ceph-UUID-osd-N-activate) gets killed after a >> > couple of minutes. >> > It also looks like the block and block.db links >> > in /var/lib/ceph/UUID/osd.N/ are not correctly set. >> > When we restart the daemons that needed a host restart, the OSD doesn't >> > come up and needs a host restart. >> > >> > All OSDs are encrypted. >> > >> > Does anyone got some ideas how to debug further? >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Adding new osds on new host
by Rok Jaklič 27 Jan '26

27 Jan '26

Hi, i am adding new OSDs from a new host with something like: <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new dbb58525-c4f6-4a1e-b03a-ac5966f09809 Running command: vgcreate --force --yes ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha stdout: Physical volume "/dev/mapper/mpatha" successfully created. stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" successfully created Running command: lvcreate --yes -l 3814911 -n osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" created. Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 Running command: /usr/bin/ln -s /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 /var/lib/ceph/osd/ceph-233/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-233/activate.monmap stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx stderr: got monmap epoch 6 --> Creating keyring file for osd.233 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 --path /var/lib/ceph/osd/ceph-233 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 /var/lib/ceph/osd/ceph-233/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-233/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 Running command: /usr/bin/systemctl enable ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service → /usr/lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → /usr/lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@233 --> ceph-volume lvm activate successful for osd ID: 233 --> ceph-volume lvm create successful for: /dev/mapper/mpatha --- I get few errors in output like: stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx ... and then ... bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid ... but cluster seems to add and work on new OSDs anyway. Anyone had a similar issue? Or can I ignore this error? Kind regards, Rok

1 0

Device prediction life unknown
by Info 27 Jan '26

27 Jan '26

Hello users When checking disk life every time getting unknown, it’s 16.2 with add’s. Ceph device predict-life-expectancy osd.84 unknown Any idea? Regards Dev

1 0

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Malte Stroem 27 Jan '26

27 Jan '26

We do not know a lot about you cluster, so it's hard to help. Give us alle the information one needs. ceph -s, ceph orch host ls, ceph health detail, all the good things to get an overview. On 1/27/26 12:43, Boris via ceph-users wrote: > All disks are 8TB HDD. We have some 16TB HDDs but those are all in the > newest host, that updated just fine. > > Am Di., 27. Jan. 2026 um 12:39 Uhr schrieb Malte Stroem < > malte.stroem(a)gmail.com>: > >> From what I can see, the OSD is running. It's starting up and still >> needs time. >> >> How big is the disk? >> >> On 1/27/26 12:31, Boris via ceph-users wrote: >>> Sure: https://pastebin.com/9RLzyUQs >>> >>> I've trimmed the log a little bit (removed peering, epoch, trim and so >> on). >>> This is the last OSD that we tried that did not work. >>> >>> We tried another host, where the upgrade just went through. But this Host >>> also got the newest hardware. >>> But we don't think it is a hardware issue, because the first 30 OSDs were >>> on the two oldest hosts and the first one that failed was on the same >> host >>> as the last OSD that did not fail. >>> >>> >>> >>> Am Di., 27. Jan. 2026 um 12:05 Uhr schrieb Malte Stroem < >>> malte.stroem(a)gmail.com>: >>> >>>> Could be the kind of hardware you are using. Is it different from the >>>> other clusters' hardware? >>>> >>>> Send us logs, so we can help you out. >>>> >>>> Example: >>>> >>>> journalctl -eu ceph-fsid(a)osd.0 >>>> >>>> Best, >>>> Malte >>>> >>>> On 1/27/26 11:55, Boris via ceph-users wrote: >>>>> Hi, >>>>> we are currently facing an issue, that suddenly none of the OSDs will >>>> start >>>>> after the container started with the new versions. >>>>> >>>>> This seems to be an issue with some hosts/OSDs. The first 30 OSDs >> worked, >>>>> but took really long (like 5 hours) and then every single OSD after >> that >>>>> needed a host reboot to bring the disk back up and continue the update. >>>>> >>>>> We've stopped after 6 tries. >>>>> >>>>> And one disk never came back up. We removed and zapped the OSD. The >>>>> orchestrator picked the available disk and recreated it. It came up >>>> within >>>>> seconds. >>>>> >>>>> We have around 90 clusters and this happened only on a single one. All >>>>> others updates within two hours without any issues. >>>>> >>>>> The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per >>>> SSD). >>>>> The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of >>>>> output from udevadm, lsblk and nsenter >>>>> The activation container (ceph-UUID-osd-N-activate) gets killed after a >>>>> couple of minutes. >>>>> It also looks like the block and block.db links >>>>> in /var/lib/ceph/UUID/osd.N/ are not correctly set. >>>>> When we restart the daemons that needed a host restart, the OSD doesn't >>>>> come up and needs a host restart. >>>>> >>>>> All OSDs are encrypted. >>>>> >>>>> Does anyone got some ideas how to debug further? >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> >>> >> >> >

2 1

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Malte Stroem 27 Jan '26

27 Jan '26

From what I can see, the OSD is running. It's starting up and still needs time. How big is the disk? On 1/27/26 12:31, Boris via ceph-users wrote: > Sure: https://pastebin.com/9RLzyUQs > > I've trimmed the log a little bit (removed peering, epoch, trim and so on). > This is the last OSD that we tried that did not work. > > We tried another host, where the upgrade just went through. But this Host > also got the newest hardware. > But we don't think it is a hardware issue, because the first 30 OSDs were > on the two oldest hosts and the first one that failed was on the same host > as the last OSD that did not fail. > > > > Am Di., 27. Jan. 2026 um 12:05 Uhr schrieb Malte Stroem < > malte.stroem(a)gmail.com>: > >> Could be the kind of hardware you are using. Is it different from the >> other clusters' hardware? >> >> Send us logs, so we can help you out. >> >> Example: >> >> journalctl -eu ceph-fsid(a)osd.0 >> >> Best, >> Malte >> >> On 1/27/26 11:55, Boris via ceph-users wrote: >>> Hi, >>> we are currently facing an issue, that suddenly none of the OSDs will >> start >>> after the container started with the new versions. >>> >>> This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, >>> but took really long (like 5 hours) and then every single OSD after that >>> needed a host reboot to bring the disk back up and continue the update. >>> >>> We've stopped after 6 tries. >>> >>> And one disk never came back up. We removed and zapped the OSD. The >>> orchestrator picked the available disk and recreated it. It came up >> within >>> seconds. >>> >>> We have around 90 clusters and this happened only on a single one. All >>> others updates within two hours without any issues. >>> >>> The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per >> SSD). >>> The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of >>> output from udevadm, lsblk and nsenter >>> The activation container (ceph-UUID-osd-N-activate) gets killed after a >>> couple of minutes. >>> It also looks like the block and block.db links >>> in /var/lib/ceph/UUID/osd.N/ are not correctly set. >>> When we restart the daemons that needed a host restart, the OSD doesn't >>> come up and needs a host restart. >>> >>> All OSDs are encrypted. >>> >>> Does anyone got some ideas how to debug further? >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

2 1

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Malte Stroem 27 Jan '26

27 Jan '26

Could be the kind of hardware you are using. Is it different from the other clusters' hardware? Send us logs, so we can help you out. Example: journalctl -eu ceph-fsid(a)osd.0 Best, Malte On 1/27/26 11:55, Boris via ceph-users wrote: > Hi, > we are currently facing an issue, that suddenly none of the OSDs will start > after the container started with the new versions. > > This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, > but took really long (like 5 hours) and then every single OSD after that > needed a host reboot to bring the disk back up and continue the update. > > We've stopped after 6 tries. > > And one disk never came back up. We removed and zapped the OSD. The > orchestrator picked the available disk and recreated it. It came up within > seconds. > > We have around 90 clusters and this happened only on a single one. All > others updates within two hours without any issues. > > The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per SSD). > The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of > output from udevadm, lsblk and nsenter > The activation container (ceph-UUID-osd-N-activate) gets killed after a > couple of minutes. > It also looks like the block and block.db links > in /var/lib/ceph/UUID/osd.N/ are not correctly set. > When we restart the daemons that needed a host restart, the OSD doesn't > come up and needs a host restart. > > All OSDs are encrypted. > > Does anyone got some ideas how to debug further? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Boris 27 Jan '26

27 Jan '26

Hi, we are currently facing an issue, that suddenly none of the OSDs will start after the container started with the new versions. This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, but took really long (like 5 hours) and then every single OSD after that needed a host reboot to bring the disk back up and continue the update. We've stopped after 6 tries. And one disk never came back up. We removed and zapped the OSD. The orchestrator picked the available disk and recreated it. It came up within seconds. We have around 90 clusters and this happened only on a single one. All others updates within two hours without any issues. The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per SSD). The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of output from udevadm, lsblk and nsenter The activation container (ceph-UUID-osd-N-activate) gets killed after a couple of minutes. It also looks like the block and block.db links in /var/lib/ceph/UUID/osd.N/ are not correctly set. When we restart the daemons that needed a host restart, the OSD doesn't come up and needs a host restart. All OSDs are encrypted. Does anyone got some ideas how to debug further?

1 0

upmap-read balancer with CephFS kernel clients
by Janek Bevendorff 26 Jan '26

26 Jan '26

Hi, Is there a way to use the new upmap-read balancer mode when I have CephFS kernel clients connected? Our cluster runs Squid and all connected clients are on kernel 6.11-6.14, so they should be new enough. However, they identify themselves as luminous (I could not find clear documentation about this, but this seems to be hard-coded and by design), so I cannot run ceph osd set-require-min-compat-client reef, which is required for the balancer module to work. I have 500 clients connected and migrating them all to FUSE would be a headache. Is there a way around that? Some of my pools have extremely high read balance scores and we're also struggling with performance issues as a result. Janek

1 0