Hi, I have a Luminous (12.2.25) cluster with several OSDs down. The daemons start but they're reporting as down. I did see in some osd logs that heartbeats were failing but when I checked the ports for the heartbeats were incorrect for that osd, although another osd was listening on that. How does the osd know what ports to ping other osds on? Is there any way to force an update. The reason this happened is because someone took a VM snapshot of this cluster and restored the snapshot so the osds aren't up. I know this isn't a good implementation or a good idea and this will change going forward. Anyway, I was just wondering about the heartbeat issue and whether attempting to ping on the right ports might bring them up. Thanks, Neil.
Hello Neil, You should never never never do a snapshot on a ceph cluster (in a vm perspective as you say). I have my ceph cluster in virtualbox but i only shutdown my cluster with commands like ceph osd noout, norebalance, pause etc. Regarding the osd heartbeat, here is some articles that might help you. https://ceph.io/planet/dealing-with-some-osd-timeouts/ https://access.redhat.com/solutions/2127471 You might want to adjust any timers if you want but do make sure there is connectivity between the osd nodes and also you may see these heartbeats during scrubbing as well.
participants (2)
-
neil.ashby-senior@bt.com
-
tdados@hotmail.com