Yes, OSDs heartbeat their peers to make sure each connection is still working and each peer is responsive. The heartbeat can get blocked by other work if it takes long enough, but it skips most queues IIRC. So this log message is saying an osd pinged a peer and didn’t get the response back for 3 seconds. That’s quite a long time! -Greg On Thu, Oct 15, 2026 at 2:30 AM Janne Johansson <ceph-users@ceph.io> wrote:
I see this at times,
2026-10-15T06:33:57.184296+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 3038.405ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:33:57.184319+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 3038.335ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:07.332951+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 2981.937ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:07.332975+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 2982.992ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:35.591255+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 2038.587ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:35.591280+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 2039.430ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:50.182523+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1284.844ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:50.182545+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1284.781ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:59.363750+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1228.156ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:59.363774+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1228.295ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:35:07.363735+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1220.582ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:35:07.363771+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1220.705ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:35:21.546156+0000 mon.mon-1 [INF] Health check cleared: OSD_SLOW_PING_TIME_BACK (was: Slow OSD heartbeats on back (longest 1220.582ms)) 2026-10-15T06:35:21.546210+0000 mon.mon-1 [INF] Health check cleared: OSD_SLOW_PING_TIME_FRONT (was: Slow OSD heartbeats on front (longest 1220.705ms)
These lines are 10-15-20 seconds apart, and in that time span, the "longest" is lowering by .. a few hundred milliseconds. So over the course of two minutes something has gone from 3000 ms to under 1000. Now if this was a queue then it seems weird that it did not sink slower, assuming that the involved parts are working off items on the queue for minutes. If they were NOT working off the queue, one would assume it would grow by one second per second, like when you get
1 ops are blocked > 3276 sec on osd.21 and nothing is improving the situation, then that delay does grow by the second, totally reasonable. But the slow ops feels like a mystery to me, how it can be so specific in milliseconds while often working itself downwards so slowly that it takes literal minutes to get down below the warning limit.
So what does this "Slow OSD heartbeats" actually measure at the time of these log lines? I often get them during expansion of the cluster or rolling OSD restarts, so its not necessarily even a strict error condition either.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io