Heartbeating is a continuous process — it issues a new ping every 5 or 15 seconds or something. So apparently whatever queue is blocking it (switch, kernel queue, something in Ceph code) is getting a bit shorter over that duration. If you have durable and persistent issues with ping latency, the slow change is not surprising. (Note too that this is aggregating the worst reports from OSDs. If you look at the health detail you may see the source switching around. Or maybe it’s one particular NIC causing trouble and it’s always the same one.) On Thu, Oct 15, 2026 at 11:16 AM Janne Johansson <icepic.dz@gmail.com> wrote:
Den tors 15 okt. 2026 kl 19:25 skrev Gregory Farnum <gfarnum@redhat.com>:
Yes, OSDs heartbeat their peers to make sure each connection is still working and each peer is responsive. The heartbeat can get blocked by other work if it takes long enough, but it skips most queues IIRC.
So this log message is saying an osd pinged a peer and didn’t get the response back for 3 seconds. That’s quite a long time!
That is not the mystery, the mystery is how 2 minutes later, the situation has only improved so that it now takes 1.9 seconds to make that same ping. How can this "delay" either not grow by one second per second (because it is low prio compared to something else and not being done at all) but also not improve more than 0.1s every 15 seconds of wall time?
These lines are 10-15-20 seconds apart, and in that time span, the
"longest" is lowering by .. a few hundred milliseconds. So over the course of two minutes something has gone from 3000 ms to under 1000. Now if this was a queue then it seems weird that it did not sink slower, assuming that the involved parts are working off items on the queue for minutes. If they were NOT working off the queue, one would assume it would grow by one second per second, like when you get
1 ops are blocked > 3276 sec on osd.21 and nothing is improving the situation, then that delay does grow by the second, totally reasonable. But the slow ops feels like a mystery to me, how it can be so specific in milliseconds while often working itself downwards so slowly that it takes literal minutes to get down below the warning limit.
So what does this "Slow OSD heartbeats" actually measure at the time of these log lines? I often get them during expansion of the cluster or rolling OSD restarts, so its not necessarily even a strict error condition either.
-- May the most significant bit of your life be positive.