Ceph standby-replay metadata server: MDS internal heartbeat is not healthy
Hi all, today we observe that out of the sudden our standby-replay metadata server continuously writes the following logs: 2020-02-13 11:56:50.216102 7fd2ad229700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-02-13 11:56:50.287699 7fd2ad229700 0 mds.beacon.dcucmds401 Skipping beacon heartbeat to monitors (last acked 100.836s ago); MDS internal heartbeat is not healthy! and it's memory is growing until no memory is available any more and the service gets restarted and then stops. The funny thing is that on the active MDS we are not seeing these log messages and any increase of memory. We are running ceph version 12.2.10 on all nodes of our Ceph cluster. Any suggestions? Best, Martin
Hi Martin, On Thu, Feb 13, 2020 at 4:10 AM Martin Palma <martin@palma.bz> wrote:
Hi all,
today we observe that out of the sudden our standby-replay metadata server continuously writes the following logs:
2020-02-13 11:56:50.216102 7fd2ad229700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-02-13 11:56:50.287699 7fd2ad229700 0 mds.beacon.dcucmds401 Skipping beacon heartbeat to monitors (last acked 100.836s ago); MDS internal heartbeat is not healthy!
and it's memory is growing until no memory is available any more and the service gets restarted and then stops. The funny thing is that on the active MDS we are not seeing these log messages and any increase of memory.
We are running ceph version 12.2.10 on all nodes of our Ceph cluster. Any suggestions?
Please increase debugging on the standby-replay daemon and share the logs. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi Patrick, we have performed a minor upgrade to 12.2.13 which resolved the issue. We think it was the following bug: https://tracker.ceph.com/issues/37723 Best, Martin On Thu, Feb 20, 2020 at 5:16 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
Hi Martin,
On Thu, Feb 13, 2020 at 4:10 AM Martin Palma <martin@palma.bz> wrote:
Hi all,
today we observe that out of the sudden our standby-replay metadata server continuously writes the following logs:
2020-02-13 11:56:50.216102 7fd2ad229700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-02-13 11:56:50.287699 7fd2ad229700 0 mds.beacon.dcucmds401 Skipping beacon heartbeat to monitors (last acked 100.836s ago); MDS internal heartbeat is not healthy!
and it's memory is growing until no memory is available any more and the service gets restarted and then stops. The funny thing is that on the active MDS we are not seeing these log messages and any increase of memory.
We are running ceph version 12.2.10 on all nodes of our Ceph cluster. Any suggestions?
Please increase debugging on the standby-replay daemon and share the logs.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
participants (2)
-
Martin Palma
-
Patrick Donnelly