Hi, the MGR doesn't always report the correct PG status, so don't rely on that too much. Sometimes it's necessary to restart primary OSDs for stuck PGs, although a repeer could have been sufficient. Your Ceph clients had to refresh their osdmap, that's when they notice that there had been down OSDs. It's not a real-time log in this case, no need to worry. It's a common question though, I think we also asked it 8 to 10 years ago. ;-) Regards, Eugen Zitat von Wannes Smet via ceph-users <ceph-users@ceph.io>:
Hi,
I'm running a Ceph cluster 19.2.2, 23 nodes, 152 OSDs, cephadm deployed. Most SAS SSDs, 12 NVMe SSDs.
Yesterday we experienced a total power failure and everything went down hard. Also our Ceph cluster. There were a couple of things, but this stood out after it got back up:
[ERR] OSD_UNREACHABLE: 2 osds(s) are not reachable osd.53's public address is not in '192.168.11.0/24' subnet osd.86's public address is not in '192.168.11.0/24' subnet
ceph -s did not say reduced data {availability,redundancy} which is a bit "off", given that both OSDs are in separate hosts, failure domain=host. There must have been PGs with less than 3 replicas and also PGs with just one replica left?
So I manually restarted those OSDs with systemctl , a recovery process started and all our VMs, "magically" started booting now. I'm also surprised that the recovery process only started when those OSDs got back up.
I didn't make too much of the above, but now this morning, I'm looking at the kernel ring buffer of our PVE nodes and I notice the logs below. Just a single "blip". All at the same time on all of our PVE nodes (ceph clients):
[Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179035): osd53 down [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179050): osd53 up [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179057): osd86 down [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179074): osd86 up
I don't see anything weird in the Ceph cluster itself, neither in the log files of the ODS.
I'm not sure what to make from this. Why would this happen and what would you do?
Thanks for your insights,
Wannes Smet
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io