Hi, did you by any chance restart the mgr (or simply fail)? The OSD status is not always correctly reported. From your log output you can see that OSD.200 actually booted successfully after around 3 minutes (as Malte also already pointed out): -- Boot 032bb082802d4003a96b9b2248f0e2eb -- Jan 27 10:08:07 host14 systemd[1]: Starting Ceph osd.200 for dca79fff-ffd0-58f4-1cff-82a2feea05f4... ... Jan 27 10:11:31 host14 ceph-osd[100997]: osd.200 1389134 load_pgs opened 135 pgs Jan 27 10:11:31 host14 ceph-osd[100997]: osd.200 1389134 log_to_monitors true Jan 27 10:11:31 host14 conmon[100991]: 2026-01-27T10:11:31.277+0000 7f541f5dd740 -1 osd.200 1389134 log_to_monitors true Jan 27 10:11:32 host14 ceph-osd[100997]: osd.200 1389134 done with init, starting boot process Jan 27 10:11:32 host14 ceph-osd[100997]: osd.200 1389134 start_boot ... Jan 27 10:11:33 host14 ceph-osd[100997]: osd.200 1389172 state: booting -> active Is there anything missing from the logs? Or is this not the right OSD log? Zitat von Boris via ceph-users <ceph-users@ceph.io>:

Sure: https://pastebin.com/9RLzyUQs

I've trimmed the log a little bit (removed peering, epoch, trim and so on). This is the last OSD that we tried that did not work.

We tried another host, where the upgrade just went through. But this Host also got the newest hardware. But we don't think it is a hardware issue, because the first 30 OSDs were on the two oldest hosts and the first one that failed was on the same host as the last OSD that did not fail.

Am Di., 27. Jan. 2026 um 12:05 Uhr schrieb Malte Stroem < malte.stroem@gmail.com>:

...

Could be the kind of hardware you are using. Is it different from the other clusters' hardware?

Send us logs, so we can help you out.

Example:

journalctl -eu ceph-fsid@osd.0

Best, Malte

On 1/27/26 11:55, Boris via ceph-users wrote:

...

Hi, we are currently facing an issue, that suddenly none of the OSDs will start after the container started with the new versions.

This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, but took really long (like 5 hours) and then every single OSD after that needed a host reboot to bring the disk back up and continue the update.

We've stopped after 6 tries.

And one disk never came back up. We removed and zapped the OSD. The orchestrator picked the available disk and recreated it. It came up within seconds.

We have around 90 clusters and this happened only on a single one. All others updates within two hours without any issues.

The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per SSD). The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of output from udevadm, lsblk and nsenter The activation container (ceph-UUID-osd-N-activate) gets killed after a couple of minutes. It also looks like the block and block.db links in /var/lib/ceph/UUID/osd.N/ are not correctly set. When we restart the daemons that needed a host restart, the OSD doesn't come up and needs a host restart.

All OSDs are encrypted.

Does anyone got some ideas how to debug further? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io