Help needed, ceph fs down due to large stray dir
Hi all, we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted. I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM. Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank? Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Hi Frank, On Fri, Jan 10, 2025 at 12:31 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries.
... why would you restart the daemon? I can't stress this question enough. Usually when CephFS has a "meltdown", the trigger was "I restarted the MDS" hoping that "X relatively minor problem" would go away.
It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Some strays may not be eligible for removal due to hard links or snapshots.
Q1: Is the MDS at least making progress in every restart iteration?
Probably not.
Q2: If not, how do we get this rank up again?
I don't see an easy way to circumvent this problem with any type of hacks/configs. One option you have is to allocate a suitably large swap file for the MDS node to see if it can chew through the stray directories. (More RAM would be better...)
Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Afraid not. You cannot migrate strays and it wouldn't take effect in time anyway.
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
That's probably out-of-date. Checking: this MDS runs out of memory shortly after becoming active right? -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi Patrick and others, thanks for your fast reply. The problem we are in comes from forward scrub ballooning and the memory overuse did not go away even after aborting the scrub. The "official" way to evaluate strays I got from Neha was to restart the rank. I did not expect that the MDS needs to load the entire stray buckets into RAM just for processing it. I expected that this situation was considered during development. The answer to when the problem occurs: yes, the MDS goes active and then starts loading incredible amounts of entries. I can go and give it swap, but might try with RAM only first (we have 512G machines, just need to stop the OSDs on the server). I will report back what happens. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Patrick Donnelly <pdonnell@redhat.com> Sent: Friday, January 10, 2025 6:57 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir Hi Frank, On Fri, Jan 10, 2025 at 12:31 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries.
... why would you restart the daemon? I can't stress this question enough. Usually when CephFS has a "meltdown", the trigger was "I restarted the MDS" hoping that "X relatively minor problem" would go away.
It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Some strays may not be eligible for removal due to hard links or snapshots.
Q1: Is the MDS at least making progress in every restart iteration?
Probably not.
Q2: If not, how do we get this rank up again?
I don't see an easy way to circumvent this problem with any type of hacks/configs. One option you have is to allocate a suitably large swap file for the MDS node to see if it can chew through the stray directories. (More RAM would be better...)
Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Afraid not. You cannot migrate strays and it wouldn't take effect in time anyway.
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
That's probably out-of-date. Checking: this MDS runs out of memory shortly after becoming active right? -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi Frank, Are you able to share any logs from the mds that's crashing? And just to confirm the rank goes into up:active before eventually OOM ? This sounds familiar-ish but i'm also recovering after a nearly 24 hour bender of another ceph related recovery.....trying to rack my brain of similar issues we've seen. Is there much swap space available to the node as well? In the event the daemon is actually making progress but just has lack of resources you may need to extend the time it can remain up with swap. Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 1/10/25 13:30, Frank Schilder wrote:
Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays. You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart. On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything: 2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy! I hope it doesn't get failed by some king of timeout now. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays. You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart. On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all, we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted. I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM. Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank? Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
HI Frank, What is the state of the mds currently? We are probably at a point where we do a bit of hope and waiting for it to come back up. Regards, Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 1/10/25 15:51, Frank Schilder wrote:
Hi all,
I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything:
2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy!
I hope it doesn't get failed by some king of timeout now.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays.
You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, I got the MDS up. however, after quite some time its sitting with almost no CPU load: top - 21:40:02 up 2:49, 1 user, load average: 0.00, 0.02, 0.34 Tasks: 606 total, 1 running, 247 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 503.7 total, 12.3 free, 490.3 used, 1.1 buff/cache GiB Swap: 3577.0 total, 3367.0 free, 210.0 used. 2.9 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 59495 ceph 20 0 685.8g 477.9g 0.0g S 1.0 94.9 53:47.57 ceph-mds I'm not sure if its doing anything at all. Only messages like these keep showing up in the log: 2025-01-10T21:38:08.459+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T21:38:08.459+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3019.23s ago); MDS internal heartbeat is not healthy! The MDS cluster looks healthy from this output: # ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-15 Reqs: 0 /s 255k 248k 5434 1678 1 active ceph-14 Reqs: 2 /s 402k 396k 26.7k 144k 2 active ceph-12 Reqs: 0 /s 86.9M 86.9M 46.2k 3909 3 active ceph-08 Reqs: 0 /s 637k 630k 2663 7457 4 active ceph-11 Reqs: 0 /s 1496k 1492k 113k 103k 5 active ceph-16 Reqs: 2 /s 775k 769k 65.3k 12.9k 6 active ceph-24 Reqs: 0 /s 130k 113k 7294 8670 7 active ceph-13 Reqs: 65 /s 3619k 3609k 469k 47.2k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 7269G con-fs2-meta2 data 0 7258G con-fs2-data data 1225T 2476T con-fs2-data-ec-ssd data 794G 22.6T con-fs2-data2 data 5747T 2253T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-17 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) Did it mark itself out of the cluster and is waiting for the MON to fail it?? Please help. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, January 10, 2025 8:51 PM To: Spencer Macphee Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir Hi all, I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything: 2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy! I hope it doesn't get failed by some king of timeout now. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays. You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart. On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all, we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted. I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM. Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank? Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Frank, You mentioned previously a large number of strays on the mds rank. Are you able to check the rank again to see how many strays there are again? We've previously had a similar issue, and once the MDS came back up we had to stat the filesystem to decrease the number of strays, and which doing so everything returned to normal. ceph tell mds.X perf dump | jq .mds_cache Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 1/10/25 16:42, Frank Schilder wrote:
Hi all,
I got the MDS up. however, after quite some time its sitting with almost no CPU load:
top - 21:40:02 up 2:49, 1 user, load average: 0.00, 0.02, 0.34 Tasks: 606 total, 1 running, 247 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 503.7 total, 12.3 free, 490.3 used, 1.1 buff/cache GiB Swap: 3577.0 total, 3367.0 free, 210.0 used. 2.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 59495 ceph 20 0 685.8g 477.9g 0.0g S 1.0 94.9 53:47.57 ceph-mds
I'm not sure if its doing anything at all. Only messages like these keep showing up in the log:
2025-01-10T21:38:08.459+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T21:38:08.459+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3019.23s ago); MDS internal heartbeat is not healthy!
The MDS cluster looks healthy from this output:
# ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-15 Reqs: 0 /s 255k 248k 5434 1678 1 active ceph-14 Reqs: 2 /s 402k 396k 26.7k 144k 2 active ceph-12 Reqs: 0 /s 86.9M 86.9M 46.2k 3909 3 active ceph-08 Reqs: 0 /s 637k 630k 2663 7457 4 active ceph-11 Reqs: 0 /s 1496k 1492k 113k 103k 5 active ceph-16 Reqs: 2 /s 775k 769k 65.3k 12.9k 6 active ceph-24 Reqs: 0 /s 130k 113k 7294 8670 7 active ceph-13 Reqs: 65 /s 3619k 3609k 469k 47.2k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 7269G con-fs2-meta2 data 0 7258G con-fs2-data data 1225T 2476T con-fs2-data-ec-ssd data 794G 22.6T con-fs2-data2 data 5747T 2253T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-17 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)
Did it mark itself out of the cluster and is waiting for the MON to fail it?? Please help.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, January 10, 2025 8:51 PM To: Spencer Macphee Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
Hi all,
I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything:
2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy!
I hope it doesn't get failed by some king of timeout now.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays.
You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Bailey, thanks for your response. The MDS was actually unresponsive and I had to restart it (ceph tell and ceph daemon commands were hanging, except for "help"). Its currently in clientreplay and loading all the stuff again. I'm really worried that this here is the rescue killer: heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s Do you have any idea how to deal with this timeout? Somewhere in he process the MDS seems to become unresponsive for too long and seems to become unresponsive after that. I have 4T swap now and the MDS comes up to the point where it actually reports back a number for the stray items. However, some time after it becomes unresponsive and the heartbeat messages start showing up. I don't know how to get past this point. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:05 PM To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Frank, You mentioned previously a large number of strays on the mds rank. Are you able to check the rank again to see how many strays there are again? We've previously had a similar issue, and once the MDS came back up we had to stat the filesystem to decrease the number of strays, and which doing so everything returned to normal. ceph tell mds.X perf dump | jq .mds_cache Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 1/10/25 16:42, Frank Schilder wrote:
Hi all,
I got the MDS up. however, after quite some time its sitting with almost no CPU load:
top - 21:40:02 up 2:49, 1 user, load average: 0.00, 0.02, 0.34 Tasks: 606 total, 1 running, 247 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 503.7 total, 12.3 free, 490.3 used, 1.1 buff/cache GiB Swap: 3577.0 total, 3367.0 free, 210.0 used. 2.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 59495 ceph 20 0 685.8g 477.9g 0.0g S 1.0 94.9 53:47.57 ceph-mds
I'm not sure if its doing anything at all. Only messages like these keep showing up in the log:
2025-01-10T21:38:08.459+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T21:38:08.459+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3019.23s ago); MDS internal heartbeat is not healthy!
The MDS cluster looks healthy from this output:
# ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-15 Reqs: 0 /s 255k 248k 5434 1678 1 active ceph-14 Reqs: 2 /s 402k 396k 26.7k 144k 2 active ceph-12 Reqs: 0 /s 86.9M 86.9M 46.2k 3909 3 active ceph-08 Reqs: 0 /s 637k 630k 2663 7457 4 active ceph-11 Reqs: 0 /s 1496k 1492k 113k 103k 5 active ceph-16 Reqs: 2 /s 775k 769k 65.3k 12.9k 6 active ceph-24 Reqs: 0 /s 130k 113k 7294 8670 7 active ceph-13 Reqs: 65 /s 3619k 3609k 469k 47.2k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 7269G con-fs2-meta2 data 0 7258G con-fs2-data data 1225T 2476T con-fs2-data-ec-ssd data 794G 22.6T con-fs2-data2 data 5747T 2253T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-17 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)
Did it mark itself out of the cluster and is waiting for the MON to fail it?? Please help.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, January 10, 2025 8:51 PM To: Spencer Macphee Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
Hi all,
I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything:
2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy!
I hope it doesn't get failed by some king of timeout now.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays.
You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, The value for that is mds_beacon_grace. Default is 15 but you can jack it up. Apply it to the monitor or global to take effect. Just to clarify too, does the MDS daemon come into up:active ? If it does, are you able to also access that portion of the filesystem in that time? If you can access the filesystem, try running a stat on that portion with something like 'find . -ls' in a directory and see if the strays decrease. Regards, Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 1/10/25 17:18, Frank Schilder wrote:
Hi Bailey,
thanks for your response. The MDS was actually unresponsive and I had to restart it (ceph tell and ceph daemon commands were hanging, except for "help"). Its currently in clientreplay and loading all the stuff again. I'm really worried that this here is the rescue killer:
heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s
Do you have any idea how to deal with this timeout? Somewhere in he process the MDS seems to become unresponsive for too long and seems to become unresponsive after that.
I have 4T swap now and the MDS comes up to the point where it actually reports back a number for the stray items. However, some time after it becomes unresponsive and the heartbeat messages start showing up. I don't know how to get past this point.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:05 PM To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Frank,
You mentioned previously a large number of strays on the mds rank. Are you able to check the rank again to see how many strays there are again? We've previously had a similar issue, and once the MDS came back up we had to stat the filesystem to decrease the number of strays, and which doing so everything returned to normal.
ceph tell mds.X perf dump | jq .mds_cache
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 16:42, Frank Schilder wrote:
Hi all,
I got the MDS up. however, after quite some time its sitting with almost no CPU load:
top - 21:40:02 up 2:49, 1 user, load average: 0.00, 0.02, 0.34 Tasks: 606 total, 1 running, 247 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 503.7 total, 12.3 free, 490.3 used, 1.1 buff/cache GiB Swap: 3577.0 total, 3367.0 free, 210.0 used. 2.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 59495 ceph 20 0 685.8g 477.9g 0.0g S 1.0 94.9 53:47.57 ceph-mds
I'm not sure if its doing anything at all. Only messages like these keep showing up in the log:
2025-01-10T21:38:08.459+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T21:38:08.459+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3019.23s ago); MDS internal heartbeat is not healthy!
The MDS cluster looks healthy from this output:
# ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-15 Reqs: 0 /s 255k 248k 5434 1678 1 active ceph-14 Reqs: 2 /s 402k 396k 26.7k 144k 2 active ceph-12 Reqs: 0 /s 86.9M 86.9M 46.2k 3909 3 active ceph-08 Reqs: 0 /s 637k 630k 2663 7457 4 active ceph-11 Reqs: 0 /s 1496k 1492k 113k 103k 5 active ceph-16 Reqs: 2 /s 775k 769k 65.3k 12.9k 6 active ceph-24 Reqs: 0 /s 130k 113k 7294 8670 7 active ceph-13 Reqs: 65 /s 3619k 3609k 469k 47.2k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 7269G con-fs2-meta2 data 0 7258G con-fs2-data data 1225T 2476T con-fs2-data-ec-ssd data 794G 22.6T con-fs2-data2 data 5747T 2253T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-17 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)
Did it mark itself out of the cluster and is waiting for the MON to fail it?? Please help.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, January 10, 2025 8:51 PM To: Spencer Macphee Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
Hi all,
I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything:
2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy!
I hope it doesn't get failed by some king of timeout now.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays.
You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Bailey, I already set that value very high: # ceph config get mds.ceph-12 mds_beacon_grace 600000.000000 To no avail. The 15s heartbeat timeout comes from somewhere else. What I observe is that the MDS loads the stray buckets (up to 87Mio DNS/INOS) and as soon as that happened it seems to start doing something (RAM usage grows without the DNS/INOS changing any more). However, shortly after the timeout happens, everything comes to a standstill. I think the MONs keep the MDS assigned but its no longer part of the file system or the actual MDS worker thread terminates with this timeout. Its reported as up and active, but this report seems just outdated as all status queries to the MDS just hang. My suspicion is that the MONs don't kick it out yet (no fail-over triggered), but the rank is actually not really active. The report just doesn't update. I'm stuck here and am out of ideas what to do about it. Increasing the thread timeout would probably help, but I can't find a config option for that. I'm afraid I need to take a break. I will be looking at my e-mail in about 4h again. Would be great if there are some further ideas for how to proceed. Thanks so far and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:23 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Frank, The value for that is mds_beacon_grace. Default is 15 but you can jack it up. Apply it to the monitor or global to take effect. Just to clarify too, does the MDS daemon come into up:active ? If it does, are you able to also access that portion of the filesystem in that time? If you can access the filesystem, try running a stat on that portion with something like 'find . -ls' in a directory and see if the strays decrease. Regards, Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 1/10/25 17:18, Frank Schilder wrote:
Hi Bailey,
thanks for your response. The MDS was actually unresponsive and I had to restart it (ceph tell and ceph daemon commands were hanging, except for "help"). Its currently in clientreplay and loading all the stuff again. I'm really worried that this here is the rescue killer:
heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s
Do you have any idea how to deal with this timeout? Somewhere in he process the MDS seems to become unresponsive for too long and seems to become unresponsive after that.
I have 4T swap now and the MDS comes up to the point where it actually reports back a number for the stray items. However, some time after it becomes unresponsive and the heartbeat messages start showing up. I don't know how to get past this point.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:05 PM To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Frank,
You mentioned previously a large number of strays on the mds rank. Are you able to check the rank again to see how many strays there are again? We've previously had a similar issue, and once the MDS came back up we had to stat the filesystem to decrease the number of strays, and which doing so everything returned to normal.
ceph tell mds.X perf dump | jq .mds_cache
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 16:42, Frank Schilder wrote:
Hi all,
I got the MDS up. however, after quite some time its sitting with almost no CPU load:
top - 21:40:02 up 2:49, 1 user, load average: 0.00, 0.02, 0.34 Tasks: 606 total, 1 running, 247 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 503.7 total, 12.3 free, 490.3 used, 1.1 buff/cache GiB Swap: 3577.0 total, 3367.0 free, 210.0 used. 2.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 59495 ceph 20 0 685.8g 477.9g 0.0g S 1.0 94.9 53:47.57 ceph-mds
I'm not sure if its doing anything at all. Only messages like these keep showing up in the log:
2025-01-10T21:38:08.459+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T21:38:08.459+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3019.23s ago); MDS internal heartbeat is not healthy!
The MDS cluster looks healthy from this output:
# ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-15 Reqs: 0 /s 255k 248k 5434 1678 1 active ceph-14 Reqs: 2 /s 402k 396k 26.7k 144k 2 active ceph-12 Reqs: 0 /s 86.9M 86.9M 46.2k 3909 3 active ceph-08 Reqs: 0 /s 637k 630k 2663 7457 4 active ceph-11 Reqs: 0 /s 1496k 1492k 113k 103k 5 active ceph-16 Reqs: 2 /s 775k 769k 65.3k 12.9k 6 active ceph-24 Reqs: 0 /s 130k 113k 7294 8670 7 active ceph-13 Reqs: 65 /s 3619k 3609k 469k 47.2k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 7269G con-fs2-meta2 data 0 7258G con-fs2-data data 1225T 2476T con-fs2-data-ec-ssd data 794G 22.6T con-fs2-data2 data 5747T 2253T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-17 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)
Did it mark itself out of the cluster and is waiting for the MON to fail it?? Please help.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, January 10, 2025 8:51 PM To: Spencer Macphee Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
Hi all,
I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything:
2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy!
I hope it doesn't get failed by some king of timeout now.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays.
You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, Can you try `perf top` to find out what the ceph-mds process is doing with that CPU time? Also Mark's profiler is super useful to find those busy loops: https://github.com/markhpc/uwpmp Cheers, Dan -- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com On Fri, Jan 10, 2025 at 2:06 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Bailey,
I already set that value very high:
# ceph config get mds.ceph-12 mds_beacon_grace 600000.000000
To no avail. The 15s heartbeat timeout comes from somewhere else. What I observe is that the MDS loads the stray buckets (up to 87Mio DNS/INOS) and as soon as that happened it seems to start doing something (RAM usage grows without the DNS/INOS changing any more). However, shortly after the timeout happens, everything comes to a standstill. I think the MONs keep the MDS assigned but its no longer part of the file system or the actual MDS worker thread terminates with this timeout.
Its reported as up and active, but this report seems just outdated as all status queries to the MDS just hang. My suspicion is that the MONs don't kick it out yet (no fail-over triggered), but the rank is actually not really active. The report just doesn't update.
I'm stuck here and am out of ideas what to do about it. Increasing the thread timeout would probably help, but I can't find a config option for that.
I'm afraid I need to take a break. I will be looking at my e-mail in about 4h again. Would be great if there are some further ideas for how to proceed.
Thanks so far and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:23 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi Frank,
The value for that is mds_beacon_grace. Default is 15 but you can jack it up. Apply it to the monitor or global to take effect.
Just to clarify too, does the MDS daemon come into up:active ? If it does, are you able to also access that portion of the filesystem in that time?
If you can access the filesystem, try running a stat on that portion with something like 'find . -ls' in a directory and see if the strays decrease.
Regards,
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 17:18, Frank Schilder wrote:
Hi Bailey,
thanks for your response. The MDS was actually unresponsive and I had to restart it (ceph tell and ceph daemon commands were hanging, except for "help"). Its currently in clientreplay and loading all the stuff again. I'm really worried that this here is the rescue killer:
heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s
Do you have any idea how to deal with this timeout? Somewhere in he process the MDS seems to become unresponsive for too long and seems to become unresponsive after that.
I have 4T swap now and the MDS comes up to the point where it actually reports back a number for the stray items. However, some time after it becomes unresponsive and the heartbeat messages start showing up. I don't know how to get past this point.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:05 PM To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Frank,
You mentioned previously a large number of strays on the mds rank. Are you able to check the rank again to see how many strays there are again? We've previously had a similar issue, and once the MDS came back up we had to stat the filesystem to decrease the number of strays, and which doing so everything returned to normal.
ceph tell mds.X perf dump | jq .mds_cache
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 16:42, Frank Schilder wrote:
Hi all,
I got the MDS up. however, after quite some time its sitting with almost no CPU load:
top - 21:40:02 up 2:49, 1 user, load average: 0.00, 0.02, 0.34 Tasks: 606 total, 1 running, 247 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 503.7 total, 12.3 free, 490.3 used, 1.1 buff/cache GiB Swap: 3577.0 total, 3367.0 free, 210.0 used. 2.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 59495 ceph 20 0 685.8g 477.9g 0.0g S 1.0 94.9 53:47.57 ceph-mds
I'm not sure if its doing anything at all. Only messages like these keep showing up in the log:
2025-01-10T21:38:08.459+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T21:38:08.459+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3019.23s ago); MDS internal heartbeat is not healthy!
The MDS cluster looks healthy from this output:
# ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-15 Reqs: 0 /s 255k 248k 5434 1678 1 active ceph-14 Reqs: 2 /s 402k 396k 26.7k 144k 2 active ceph-12 Reqs: 0 /s 86.9M 86.9M 46.2k 3909 3 active ceph-08 Reqs: 0 /s 637k 630k 2663 7457 4 active ceph-11 Reqs: 0 /s 1496k 1492k 113k 103k 5 active ceph-16 Reqs: 2 /s 775k 769k 65.3k 12.9k 6 active ceph-24 Reqs: 0 /s 130k 113k 7294 8670 7 active ceph-13 Reqs: 65 /s 3619k 3609k 469k 47.2k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 7269G con-fs2-meta2 data 0 7258G con-fs2-data data 1225T 2476T con-fs2-data-ec-ssd data 794G 22.6T con-fs2-data2 data 5747T 2253T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-17 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)
Did it mark itself out of the cluster and is waiting for the MON to fail it?? Please help.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, January 10, 2025 8:51 PM To: Spencer Macphee Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
Hi all,
I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything:
2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy!
I hope it doesn't get failed by some king of timeout now.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays.
You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, here an update. The MDS got stuck again doing nothing. Could it be blocklisting? The MDS IP address is in the blocklist together with a bunch of others (see blocklist below). Could this have anything to do with my observation of the MDS coming up but not doing anything? Anyways, following on the suggestions I got I did another restart with these recovery options: global advanced mds_beacon_grace 600000.000000 mon advanced mds_beacon_grace 600000.000000 mon advanced mds_heartbeat_reset_grace 14400 mds advanced mds_heartbeat_reset_grace 14400 All of these were set already in my previous attempt except for mds_heartbeat_reset_grace on the MONs. The options preventing client (re-)connections are not present in pacific. How can I prevent clients from connecting? Well, to the result: the MDS gets stuck again. Here is the log snippet at the moment when it all goes bad: 2025-01-11T02:12:18.930+0100 7f35d0b8c700 1 mds.ceph-12 Updating MDS map to version 1103287 from mon.1 2025-01-11T02:12:31.061+0100 7f35d0b8c700 1 mds.ceph-12 Updating MDS map to version 1103288 from mon.1 2025-01-11T02:12:50.282+0100 7f35ceb88700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-11T02:12:50.282+0100 7f35ceb88700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3.99997s ago); MDS internal heartbeat is not healthy! 2025-01-11T02:12:50.782+0100 7f35ceb88700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-11T02:12:50.782+0100 7f35ceb88700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 4.49996s ago); MDS internal heartbeat is not healthy! This may or may not coincide with the start of using swap. I have it on a really fast enterprise SSD though, so this should be very unlikely. How can I change this internal heartbeat timeout? It seems to be the source of all evil here. After changing the grace-parameters (before restart), why is it still not sending beacons to the MONs?? What timeout is relevant here or is it indeed blocklisting? There has to be something. Can anyone explain the meaning of these log messages please? I installed perf in the container, but I get the error "No permission to enable cycles:u event.". Can't install the ceph symbols on the host either, its CentOS7. Here the blocklist, all addresses are in the public network, the *.76 address is the host with the stubborn MDS. All of these addresses are ceph-osd/mds hosts. # ceph osd blocklist ls 192.168.32.76:6801/1430498156 2025-01-12T01:31:20.754963+0100 192.168.32.76:6800/132383718 2025-01-11T23:27:19.299845+0100 192.168.32.76:6801/132383718 2025-01-11T23:27:19.299845+0100 192.168.32.76:6801/2655583055 2025-01-11T18:47:45.777618+0100 192.168.32.76:6800/2655583055 2025-01-11T18:47:45.777618+0100 192.168.32.76:6801/26503869 2025-01-11T18:36:15.545825+0100 192.168.32.81:6801/1860839812 2025-01-11T19:01:30.106032+0100 192.168.32.81:6800/1860839812 2025-01-11T19:01:30.106032+0100 192.168.32.76:6800/26503869 2025-01-11T18:36:15.545825+0100 192.168.32.81:6801/3624695074 2025-01-11T18:20:10.720157+0100 192.168.32.76:6800/169617095 2025-01-11T18:04:24.331676+0100 192.168.32.76:6801/64253026 2025-01-11T19:57:05.362481+0100 192.168.32.77:6800/4046036080 2025-01-11T14:21:01.619782+0100 192.168.32.72:6800/1597941661 2025-01-11T17:34:26.953025+0100 192.168.32.77:6801/4046036080 2025-01-11T14:21:01.619782+0100 192.168.32.76:6800/237383530 2025-01-11T17:44:22.000558+0100 192.168.32.88:6800/3234284305 2025-01-11T17:30:21.984212+0100 192.168.32.88:6801/3234284305 2025-01-11T17:30:21.984212+0100 192.168.32.79:6800/2508491163 2025-01-11T17:42:51.567239+0100 192.168.32.80:6800/1795253695 2025-01-11T17:32:01.876755+0100 192.168.32.72:6801/2852823438 2025-01-11T17:36:06.126374+0100 192.168.32.81:6800/2603013093 2025-01-11T18:40:25.487782+0100 192.168.32.80:6801/1795253695 2025-01-11T17:32:01.876755+0100 192.168.32.78:6800/436883098 2025-01-11T17:37:19.465988+0100 192.168.32.81:6800/1214560246 2025-01-11T20:08:00.161280+0100 192.168.32.75:6800/2981193183 2025-01-11T17:33:21.493087+0100 192.168.32.75:6801/2981193183 2025-01-11T17:33:21.493087+0100 192.168.32.81:6801/2790245426 2025-01-11T17:35:33.591919+0100 192.168.32.81:6801/803708946 2025-01-11T17:52:45.777544+0100 192.168.32.76:6800/64253026 2025-01-11T19:57:05.362481+0100 192.168.32.72:6801/1597941661 2025-01-11T17:34:26.953025+0100 192.168.32.76:6801/169617095 2025-01-11T18:04:24.331676+0100 192.168.32.81:6800/2790245426 2025-01-11T17:35:33.591919+0100 192.168.32.76:6800/1430498156 2025-01-12T01:31:20.754963+0100 192.168.32.81:6801/2603013093 2025-01-11T18:40:25.487782+0100 192.168.32.72:6800/2852823438 2025-01-11T17:36:06.126374+0100 192.168.32.76:6801/871505229 2025-01-11T21:58:22.567016+0100 192.168.32.79:6801/2508491163 2025-01-11T17:42:51.567239+0100 192.168.32.81:6800/3624695074 2025-01-11T18:20:10.720157+0100 192.168.32.78:6801/436883098 2025-01-11T17:37:19.465988+0100 192.168.32.76:6800/871505229 2025-01-11T21:58:22.567016+0100 192.168.32.81:6801/1214560246 2025-01-11T20:08:00.161280+0100 192.168.32.76:6801/237383530 2025-01-11T17:44:22.000558+0100 192.168.32.81:6800/803708946 2025-01-11T17:52:45.777544+0100 listed 44 entries The current instance of ceph-12 seems not to be listed though (if I interpret he nonces right), it is: # ceph tell mds.ceph-12 status { "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "whoami": 2, "id": 436483146, "want_state": "up:active", "state": "up:active", "fs_name": "con-fs2", "rank_uptime": 2408.8316334639999, "mdsmap_epoch": 1103281, "osdmap_epoch": 3235353, "osdmap_epoch_barrier": 3235353, "uptime": 2409.181400937 } This status output was taken before the MDS stopped responding. The only change from restart to restart is that the MDS seems to load about 100.000 fewer DNS/INOS into cache. So maybe there is some progress with trimming the stray items? However, I can't do 850 restarts in this fashion, there has to be another way. I would be really grateful for any help regarding getting he system in a stable state for further trouble shooting. I would really block all client access to the fs. In addition, any hints as to how to get the MDS stay in the system and trim the stray items is dearly needed. Alternatively, is there a way to do off-line trimming? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dan.vanderster@clyso.com> Sent: Friday, January 10, 2025 11:32 PM To: Frank Schilder Cc: Bailey Allison; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Frank, Can you try `perf top` to find out what the ceph-mds process is doing with that CPU time? Also Mark's profiler is super useful to find those busy loops: https://github.com/markhpc/uwpmp Cheers, Dan -- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com On Fri, Jan 10, 2025 at 2:06 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Bailey,
I already set that value very high:
# ceph config get mds.ceph-12 mds_beacon_grace 600000.000000
To no avail. The 15s heartbeat timeout comes from somewhere else. What I observe is that the MDS loads the stray buckets (up to 87Mio DNS/INOS) and as soon as that happened it seems to start doing something (RAM usage grows without the DNS/INOS changing any more). However, shortly after the timeout happens, everything comes to a standstill. I think the MONs keep the MDS assigned but its no longer part of the file system or the actual MDS worker thread terminates with this timeout.
Its reported as up and active, but this report seems just outdated as all status queries to the MDS just hang. My suspicion is that the MONs don't kick it out yet (no fail-over triggered), but the rank is actually not really active. The report just doesn't update.
I'm stuck here and am out of ideas what to do about it. Increasing the thread timeout would probably help, but I can't find a config option for that.
I'm afraid I need to take a break. I will be looking at my e-mail in about 4h again. Would be great if there are some further ideas for how to proceed.
Thanks so far and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:23 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi Frank,
The value for that is mds_beacon_grace. Default is 15 but you can jack it up. Apply it to the monitor or global to take effect.
Just to clarify too, does the MDS daemon come into up:active ? If it does, are you able to also access that portion of the filesystem in that time?
If you can access the filesystem, try running a stat on that portion with something like 'find . -ls' in a directory and see if the strays decrease.
Regards,
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 17:18, Frank Schilder wrote:
Hi Bailey,
thanks for your response. The MDS was actually unresponsive and I had to restart it (ceph tell and ceph daemon commands were hanging, except for "help"). Its currently in clientreplay and loading all the stuff again. I'm really worried that this here is the rescue killer:
heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s
Do you have any idea how to deal with this timeout? Somewhere in he process the MDS seems to become unresponsive for too long and seems to become unresponsive after that.
I have 4T swap now and the MDS comes up to the point where it actually reports back a number for the stray items. However, some time after it becomes unresponsive and the heartbeat messages start showing up. I don't know how to get past this point.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:05 PM To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Frank,
You mentioned previously a large number of strays on the mds rank. Are you able to check the rank again to see how many strays there are again? We've previously had a similar issue, and once the MDS came back up we had to stat the filesystem to decrease the number of strays, and which doing so everything returned to normal.
ceph tell mds.X perf dump | jq .mds_cache
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 16:42, Frank Schilder wrote:
Hi all,
I got the MDS up. however, after quite some time its sitting with almost no CPU load:
top - 21:40:02 up 2:49, 1 user, load average: 0.00, 0.02, 0.34 Tasks: 606 total, 1 running, 247 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 503.7 total, 12.3 free, 490.3 used, 1.1 buff/cache GiB Swap: 3577.0 total, 3367.0 free, 210.0 used. 2.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 59495 ceph 20 0 685.8g 477.9g 0.0g S 1.0 94.9 53:47.57 ceph-mds
I'm not sure if its doing anything at all. Only messages like these keep showing up in the log:
2025-01-10T21:38:08.459+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T21:38:08.459+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3019.23s ago); MDS internal heartbeat is not healthy!
The MDS cluster looks healthy from this output:
# ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-15 Reqs: 0 /s 255k 248k 5434 1678 1 active ceph-14 Reqs: 2 /s 402k 396k 26.7k 144k 2 active ceph-12 Reqs: 0 /s 86.9M 86.9M 46.2k 3909 3 active ceph-08 Reqs: 0 /s 637k 630k 2663 7457 4 active ceph-11 Reqs: 0 /s 1496k 1492k 113k 103k 5 active ceph-16 Reqs: 2 /s 775k 769k 65.3k 12.9k 6 active ceph-24 Reqs: 0 /s 130k 113k 7294 8670 7 active ceph-13 Reqs: 65 /s 3619k 3609k 469k 47.2k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 7269G con-fs2-meta2 data 0 7258G con-fs2-data data 1225T 2476T con-fs2-data-ec-ssd data 794G 22.6T con-fs2-data2 data 5747T 2253T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-17 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)
Did it mark itself out of the cluster and is waiting for the MON to fail it?? Please help.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, January 10, 2025 8:51 PM To: Spencer Macphee Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
Hi all,
I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything:
2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy!
I hope it doesn't get failed by some king of timeout now.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays.
You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, I don't think the blocklists are related. (Those are blocking the previous running instances of the mds on that host, not the current). Your MDS is burning CPU (you see that with top) but it's unresponsive. Any of these will be closer to finding a clue what it's doing: perf top -p <pidof ceph-mds> top -H uwpmp Cheers, Dan On Fri, Jan 10, 2025 at 5:36 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
here an update. The MDS got stuck again doing nothing. Could it be blocklisting? The MDS IP address is in the blocklist together with a bunch of others (see blocklist below). Could this have anything to do with my observation of the MDS coming up but not doing anything?
Anyways, following on the suggestions I got I did another restart with these recovery options:
global advanced mds_beacon_grace 600000.000000 mon advanced mds_beacon_grace 600000.000000 mon advanced mds_heartbeat_reset_grace 14400 mds advanced mds_heartbeat_reset_grace 14400
All of these were set already in my previous attempt except for mds_heartbeat_reset_grace on the MONs. The options preventing client (re-)connections are not present in pacific. How can I prevent clients from connecting? Well, to the result: the MDS gets stuck again. Here is the log snippet at the moment when it all goes bad:
2025-01-11T02:12:18.930+0100 7f35d0b8c700 1 mds.ceph-12 Updating MDS map to version 1103287 from mon.1 2025-01-11T02:12:31.061+0100 7f35d0b8c700 1 mds.ceph-12 Updating MDS map to version 1103288 from mon.1 2025-01-11T02:12:50.282+0100 7f35ceb88700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-11T02:12:50.282+0100 7f35ceb88700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3.99997s ago); MDS internal heartbeat is not healthy! 2025-01-11T02:12:50.782+0100 7f35ceb88700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-11T02:12:50.782+0100 7f35ceb88700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 4.49996s ago); MDS internal heartbeat is not healthy!
This may or may not coincide with the start of using swap. I have it on a really fast enterprise SSD though, so this should be very unlikely. How can I change this internal heartbeat timeout? It seems to be the source of all evil here.
After changing the grace-parameters (before restart), why is it still not sending beacons to the MONs?? What timeout is relevant here or is it indeed blocklisting? There has to be something. Can anyone explain the meaning of these log messages please?
I installed perf in the container, but I get the error "No permission to enable cycles:u event.". Can't install the ceph symbols on the host either, its CentOS7.
Here the blocklist, all addresses are in the public network, the *.76 address is the host with the stubborn MDS. All of these addresses are ceph-osd/mds hosts.
# ceph osd blocklist ls 192.168.32.76:6801/1430498156 2025-01-12T01:31:20.754963+0100 192.168.32.76:6800/132383718 2025-01-11T23:27:19.299845+0100 192.168.32.76:6801/132383718 2025-01-11T23:27:19.299845+0100 192.168.32.76:6801/2655583055 2025-01-11T18:47:45.777618+0100 192.168.32.76:6800/2655583055 2025-01-11T18:47:45.777618+0100 192.168.32.76:6801/26503869 2025-01-11T18:36:15.545825+0100 192.168.32.81:6801/1860839812 2025-01-11T19:01:30.106032+0100 192.168.32.81:6800/1860839812 2025-01-11T19:01:30.106032+0100 192.168.32.76:6800/26503869 2025-01-11T18:36:15.545825+0100 192.168.32.81:6801/3624695074 2025-01-11T18:20:10.720157+0100 192.168.32.76:6800/169617095 2025-01-11T18:04:24.331676+0100 192.168.32.76:6801/64253026 2025-01-11T19:57:05.362481+0100 192.168.32.77:6800/4046036080 2025-01-11T14:21:01.619782+0100 192.168.32.72:6800/1597941661 2025-01-11T17:34:26.953025+0100 192.168.32.77:6801/4046036080 2025-01-11T14:21:01.619782+0100 192.168.32.76:6800/237383530 2025-01-11T17:44:22.000558+0100 192.168.32.88:6800/3234284305 2025-01-11T17:30:21.984212+0100 192.168.32.88:6801/3234284305 2025-01-11T17:30:21.984212+0100 192.168.32.79:6800/2508491163 2025-01-11T17:42:51.567239+0100 192.168.32.80:6800/1795253695 2025-01-11T17:32:01.876755+0100 192.168.32.72:6801/2852823438 2025-01-11T17:36:06.126374+0100 192.168.32.81:6800/2603013093 2025-01-11T18:40:25.487782+0100 192.168.32.80:6801/1795253695 2025-01-11T17:32:01.876755+0100 192.168.32.78:6800/436883098 2025-01-11T17:37:19.465988+0100 192.168.32.81:6800/1214560246 2025-01-11T20:08:00.161280+0100 192.168.32.75:6800/2981193183 2025-01-11T17:33:21.493087+0100 192.168.32.75:6801/2981193183 2025-01-11T17:33:21.493087+0100 192.168.32.81:6801/2790245426 2025-01-11T17:35:33.591919+0100 192.168.32.81:6801/803708946 2025-01-11T17:52:45.777544+0100 192.168.32.76:6800/64253026 2025-01-11T19:57:05.362481+0100 192.168.32.72:6801/1597941661 2025-01-11T17:34:26.953025+0100 192.168.32.76:6801/169617095 2025-01-11T18:04:24.331676+0100 192.168.32.81:6800/2790245426 2025-01-11T17:35:33.591919+0100 192.168.32.76:6800/1430498156 2025-01-12T01:31:20.754963+0100 192.168.32.81:6801/2603013093 2025-01-11T18:40:25.487782+0100 192.168.32.72:6800/2852823438 2025-01-11T17:36:06.126374+0100 192.168.32.76:6801/871505229 2025-01-11T21:58:22.567016+0100 192.168.32.79:6801/2508491163 2025-01-11T17:42:51.567239+0100 192.168.32.81:6800/3624695074 2025-01-11T18:20:10.720157+0100 192.168.32.78:6801/436883098 2025-01-11T17:37:19.465988+0100 192.168.32.76:6800/871505229 2025-01-11T21:58:22.567016+0100 192.168.32.81:6801/1214560246 2025-01-11T20:08:00.161280+0100 192.168.32.76:6801/237383530 2025-01-11T17:44:22.000558+0100 192.168.32.81:6800/803708946 2025-01-11T17:52:45.777544+0100 listed 44 entries
The current instance of ceph-12 seems not to be listed though (if I interpret he nonces right), it is:
# ceph tell mds.ceph-12 status { "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "whoami": 2, "id": 436483146, "want_state": "up:active", "state": "up:active", "fs_name": "con-fs2", "rank_uptime": 2408.8316334639999, "mdsmap_epoch": 1103281, "osdmap_epoch": 3235353, "osdmap_epoch_barrier": 3235353, "uptime": 2409.181400937 }
This status output was taken before the MDS stopped responding.
The only change from restart to restart is that the MDS seems to load about 100.000 fewer DNS/INOS into cache. So maybe there is some progress with trimming the stray items? However, I can't do 850 restarts in this fashion, there has to be another way.
I would be really grateful for any help regarding getting he system in a stable state for further trouble shooting. I would really block all client access to the fs. In addition, any hints as to how to get the MDS stay in the system and trim the stray items is dearly needed. Alternatively, is there a way to do off-line trimming?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dan van der Ster <dan.vanderster@clyso.com> Sent: Friday, January 10, 2025 11:32 PM To: Frank Schilder Cc: Bailey Allison; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi Frank,
Can you try `perf top` to find out what the ceph-mds process is doing with that CPU time? Also Mark's profiler is super useful to find those busy loops: https://github.com/markhpc/uwpmp
Cheers, Dan
-- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
On Fri, Jan 10, 2025 at 2:06 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Bailey,
I already set that value very high:
# ceph config get mds.ceph-12 mds_beacon_grace 600000.000000
To no avail. The 15s heartbeat timeout comes from somewhere else. What I observe is that the MDS loads the stray buckets (up to 87Mio DNS/INOS) and as soon as that happened it seems to start doing something (RAM usage grows without the DNS/INOS changing any more). However, shortly after the timeout happens, everything comes to a standstill. I think the MONs keep the MDS assigned but its no longer part of the file system or the actual MDS worker thread terminates with this timeout.
Its reported as up and active, but this report seems just outdated as all status queries to the MDS just hang. My suspicion is that the MONs don't kick it out yet (no fail-over triggered), but the rank is actually not really active. The report just doesn't update.
I'm stuck here and am out of ideas what to do about it. Increasing the thread timeout would probably help, but I can't find a config option for that.
I'm afraid I need to take a break. I will be looking at my e-mail in about 4h again. Would be great if there are some further ideas for how to proceed.
Thanks so far and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:23 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi Frank,
The value for that is mds_beacon_grace. Default is 15 but you can jack it up. Apply it to the monitor or global to take effect.
Just to clarify too, does the MDS daemon come into up:active ? If it does, are you able to also access that portion of the filesystem in that time?
If you can access the filesystem, try running a stat on that portion with something like 'find . -ls' in a directory and see if the strays decrease.
Regards,
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 17:18, Frank Schilder wrote:
Hi Bailey,
thanks for your response. The MDS was actually unresponsive and I had to restart it (ceph tell and ceph daemon commands were hanging, except for "help"). Its currently in clientreplay and loading all the stuff again. I'm really worried that this here is the rescue killer:
heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s
Do you have any idea how to deal with this timeout? Somewhere in he process the MDS seems to become unresponsive for too long and seems to become unresponsive after that.
I have 4T swap now and the MDS comes up to the point where it actually reports back a number for the stray items. However, some time after it becomes unresponsive and the heartbeat messages start showing up. I don't know how to get past this point.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:05 PM To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Frank,
You mentioned previously a large number of strays on the mds rank. Are you able to check the rank again to see how many strays there are again? We've previously had a similar issue, and once the MDS came back up we had to stat the filesystem to decrease the number of strays, and which doing so everything returned to normal.
ceph tell mds.X perf dump | jq .mds_cache
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 16:42, Frank Schilder wrote:
Hi all,
I got the MDS up. however, after quite some time its sitting with almost no CPU load:
top - 21:40:02 up 2:49, 1 user, load average: 0.00, 0.02, 0.34 Tasks: 606 total, 1 running, 247 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 503.7 total, 12.3 free, 490.3 used, 1.1 buff/cache GiB Swap: 3577.0 total, 3367.0 free, 210.0 used. 2.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 59495 ceph 20 0 685.8g 477.9g 0.0g S 1.0 94.9 53:47.57 ceph-mds
I'm not sure if its doing anything at all. Only messages like these keep showing up in the log:
2025-01-10T21:38:08.459+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T21:38:08.459+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3019.23s ago); MDS internal heartbeat is not healthy!
The MDS cluster looks healthy from this output:
# ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-15 Reqs: 0 /s 255k 248k 5434 1678 1 active ceph-14 Reqs: 2 /s 402k 396k 26.7k 144k 2 active ceph-12 Reqs: 0 /s 86.9M 86.9M 46.2k 3909 3 active ceph-08 Reqs: 0 /s 637k 630k 2663 7457 4 active ceph-11 Reqs: 0 /s 1496k 1492k 113k 103k 5 active ceph-16 Reqs: 2 /s 775k 769k 65.3k 12.9k 6 active ceph-24 Reqs: 0 /s 130k 113k 7294 8670 7 active ceph-13 Reqs: 65 /s 3619k 3609k 469k 47.2k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 7269G con-fs2-meta2 data 0 7258G con-fs2-data data 1225T 2476T con-fs2-data-ec-ssd data 794G 22.6T con-fs2-data2 data 5747T 2253T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-17 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)
Did it mark itself out of the cluster and is waiting for the MON to fail it?? Please help.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, January 10, 2025 8:51 PM To: Spencer Macphee Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
Hi all,
I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything:
2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy!
I hope it doesn't get failed by some king of timeout now.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays.
You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
Hi Dan, thanks for our continued help, I really appreciate it. Just to clarify:
Your MDS is burning CPU (you see that with top) but it's unresponsive.
Did you mean "is *not* burning CPU"? The MDS is idle - *no* CPU load, yet unresponsive. See below for a more detailed description of observations. For doing the investigation you ask for I need to run perf inside the container, but get the error "No permission to enable cycles:u event." Do you know how I can get perf to work inside the docker container? I use the official one from quay.io and run it with privileged=true. New observations: I had to bring up the OSDs in the host and can now confirm that the heartbeat failure is not related to swapping. This time the MDS needed to start swapping much earlier and it just continues to fill the cache. Slower this time, but it makes good progress. The MDS loads cache items until its done (it always stops at about the same number, which is slowly decreasing per restart; similarly, the reported stray count goes a bit down every restart) and then serves a few requests. Very shortly after that the request/s goes to 0 (dashboard), the heartbeat messages show up in the log and the MDS stops responding to daemon queries. From then on, for a longer period of time it continues to do something until it settles at a certain memory consumption. During this period memory usage is growing. After that period, CPU usage goes down to almost nothing and memory usage stays constant. From then on nothing seems to happen any more. So there is a short time window where stray items are evaluated, but the MDS gets thrown off rather quickly somehow. Dan, the host I'm working on is quite busy, it has 80 OSDs. I can stop these again for a while to be able to see what the MDS is doing with the low CPU load. Output like top -H is dominated by the OSDs. Unless you really meant "high CPU load", in which case you might want to ask me to pull out some other information regarding the opposite question "why is the MDS idle yet unresponsive". Thanks for your help so far! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dan.vanderster@clyso.com> Sent: Saturday, January 11, 2025 3:04 AM To: Frank Schilder Cc: Bailey Allison; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Frank, I don't think the blocklists are related. (Those are blocking the previous running instances of the mds on that host, not the current). Your MDS is burning CPU (you see that with top) but it's unresponsive. Any of these will be closer to finding a clue what it's doing: perf top -p <pidof ceph-mds> top -H uwpmp Cheers, Dan
Hi all, new update: after sleeping after the final MDS restart the MDS is doing something! It is still unresponsive, but it does show CPU load of between 150-200% and I really really hope that this is the trimming of stray items. I will try to find out if I get perf to work inside the container. For now, to facilitate trouble shooting, I will add a swap disk to every MDS host just to be on the safe side if stuff fails over. Just to get my hopes back: can someone (from the dev team) let me know if it is expected that an MDS is unresponsive during stray evaluation? Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 2:36 AM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi all, here an update. The MDS got stuck again doing nothing. Could it be blocklisting? The MDS IP address is in the blocklist together with a bunch of others (see blocklist below). Could this have anything to do with my observation of the MDS coming up but not doing anything? Anyways, following on the suggestions I got I did another restart with these recovery options: global advanced mds_beacon_grace 600000.000000 mon advanced mds_beacon_grace 600000.000000 mon advanced mds_heartbeat_reset_grace 14400 mds advanced mds_heartbeat_reset_grace 14400 All of these were set already in my previous attempt except for mds_heartbeat_reset_grace on the MONs. The options preventing client (re-)connections are not present in pacific. How can I prevent clients from connecting? Well, to the result: the MDS gets stuck again. Here is the log snippet at the moment when it all goes bad: 2025-01-11T02:12:18.930+0100 7f35d0b8c700 1 mds.ceph-12 Updating MDS map to version 1103287 from mon.1 2025-01-11T02:12:31.061+0100 7f35d0b8c700 1 mds.ceph-12 Updating MDS map to version 1103288 from mon.1 2025-01-11T02:12:50.282+0100 7f35ceb88700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-11T02:12:50.282+0100 7f35ceb88700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3.99997s ago); MDS internal heartbeat is not healthy! 2025-01-11T02:12:50.782+0100 7f35ceb88700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-11T02:12:50.782+0100 7f35ceb88700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 4.49996s ago); MDS internal heartbeat is not healthy! This may or may not coincide with the start of using swap. I have it on a really fast enterprise SSD though, so this should be very unlikely. How can I change this internal heartbeat timeout? It seems to be the source of all evil here. After changing the grace-parameters (before restart), why is it still not sending beacons to the MONs?? What timeout is relevant here or is it indeed blocklisting? There has to be something. Can anyone explain the meaning of these log messages please? I installed perf in the container, but I get the error "No permission to enable cycles:u event.". Can't install the ceph symbols on the host either, its CentOS7. Here the blocklist, all addresses are in the public network, the *.76 address is the host with the stubborn MDS. All of these addresses are ceph-osd/mds hosts. # ceph osd blocklist ls 192.168.32.76:6801/1430498156 2025-01-12T01:31:20.754963+0100 192.168.32.76:6800/132383718 2025-01-11T23:27:19.299845+0100 192.168.32.76:6801/132383718 2025-01-11T23:27:19.299845+0100 192.168.32.76:6801/2655583055 2025-01-11T18:47:45.777618+0100 192.168.32.76:6800/2655583055 2025-01-11T18:47:45.777618+0100 192.168.32.76:6801/26503869 2025-01-11T18:36:15.545825+0100 192.168.32.81:6801/1860839812 2025-01-11T19:01:30.106032+0100 192.168.32.81:6800/1860839812 2025-01-11T19:01:30.106032+0100 192.168.32.76:6800/26503869 2025-01-11T18:36:15.545825+0100 192.168.32.81:6801/3624695074 2025-01-11T18:20:10.720157+0100 192.168.32.76:6800/169617095 2025-01-11T18:04:24.331676+0100 192.168.32.76:6801/64253026 2025-01-11T19:57:05.362481+0100 192.168.32.77:6800/4046036080 2025-01-11T14:21:01.619782+0100 192.168.32.72:6800/1597941661 2025-01-11T17:34:26.953025+0100 192.168.32.77:6801/4046036080 2025-01-11T14:21:01.619782+0100 192.168.32.76:6800/237383530 2025-01-11T17:44:22.000558+0100 192.168.32.88:6800/3234284305 2025-01-11T17:30:21.984212+0100 192.168.32.88:6801/3234284305 2025-01-11T17:30:21.984212+0100 192.168.32.79:6800/2508491163 2025-01-11T17:42:51.567239+0100 192.168.32.80:6800/1795253695 2025-01-11T17:32:01.876755+0100 192.168.32.72:6801/2852823438 2025-01-11T17:36:06.126374+0100 192.168.32.81:6800/2603013093 2025-01-11T18:40:25.487782+0100 192.168.32.80:6801/1795253695 2025-01-11T17:32:01.876755+0100 192.168.32.78:6800/436883098 2025-01-11T17:37:19.465988+0100 192.168.32.81:6800/1214560246 2025-01-11T20:08:00.161280+0100 192.168.32.75:6800/2981193183 2025-01-11T17:33:21.493087+0100 192.168.32.75:6801/2981193183 2025-01-11T17:33:21.493087+0100 192.168.32.81:6801/2790245426 2025-01-11T17:35:33.591919+0100 192.168.32.81:6801/803708946 2025-01-11T17:52:45.777544+0100 192.168.32.76:6800/64253026 2025-01-11T19:57:05.362481+0100 192.168.32.72:6801/1597941661 2025-01-11T17:34:26.953025+0100 192.168.32.76:6801/169617095 2025-01-11T18:04:24.331676+0100 192.168.32.81:6800/2790245426 2025-01-11T17:35:33.591919+0100 192.168.32.76:6800/1430498156 2025-01-12T01:31:20.754963+0100 192.168.32.81:6801/2603013093 2025-01-11T18:40:25.487782+0100 192.168.32.72:6800/2852823438 2025-01-11T17:36:06.126374+0100 192.168.32.76:6801/871505229 2025-01-11T21:58:22.567016+0100 192.168.32.79:6801/2508491163 2025-01-11T17:42:51.567239+0100 192.168.32.81:6800/3624695074 2025-01-11T18:20:10.720157+0100 192.168.32.78:6801/436883098 2025-01-11T17:37:19.465988+0100 192.168.32.76:6800/871505229 2025-01-11T21:58:22.567016+0100 192.168.32.81:6801/1214560246 2025-01-11T20:08:00.161280+0100 192.168.32.76:6801/237383530 2025-01-11T17:44:22.000558+0100 192.168.32.81:6800/803708946 2025-01-11T17:52:45.777544+0100 listed 44 entries The current instance of ceph-12 seems not to be listed though (if I interpret he nonces right), it is: # ceph tell mds.ceph-12 status { "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "whoami": 2, "id": 436483146, "want_state": "up:active", "state": "up:active", "fs_name": "con-fs2", "rank_uptime": 2408.8316334639999, "mdsmap_epoch": 1103281, "osdmap_epoch": 3235353, "osdmap_epoch_barrier": 3235353, "uptime": 2409.181400937 } This status output was taken before the MDS stopped responding. The only change from restart to restart is that the MDS seems to load about 100.000 fewer DNS/INOS into cache. So maybe there is some progress with trimming the stray items? However, I can't do 850 restarts in this fashion, there has to be another way. I would be really grateful for any help regarding getting he system in a stable state for further trouble shooting. I would really block all client access to the fs. In addition, any hints as to how to get the MDS stay in the system and trim the stray items is dearly needed. Alternatively, is there a way to do off-line trimming? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dan.vanderster@clyso.com> Sent: Friday, January 10, 2025 11:32 PM To: Frank Schilder Cc: Bailey Allison; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Frank, Can you try `perf top` to find out what the ceph-mds process is doing with that CPU time? Also Mark's profiler is super useful to find those busy loops: https://github.com/markhpc/uwpmp Cheers, Dan -- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com On Fri, Jan 10, 2025 at 2:06 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Bailey,
I already set that value very high:
# ceph config get mds.ceph-12 mds_beacon_grace 600000.000000
To no avail. The 15s heartbeat timeout comes from somewhere else. What I observe is that the MDS loads the stray buckets (up to 87Mio DNS/INOS) and as soon as that happened it seems to start doing something (RAM usage grows without the DNS/INOS changing any more). However, shortly after the timeout happens, everything comes to a standstill. I think the MONs keep the MDS assigned but its no longer part of the file system or the actual MDS worker thread terminates with this timeout.
Its reported as up and active, but this report seems just outdated as all status queries to the MDS just hang. My suspicion is that the MONs don't kick it out yet (no fail-over triggered), but the rank is actually not really active. The report just doesn't update.
I'm stuck here and am out of ideas what to do about it. Increasing the thread timeout would probably help, but I can't find a config option for that.
I'm afraid I need to take a break. I will be looking at my e-mail in about 4h again. Would be great if there are some further ideas for how to proceed.
Thanks so far and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:23 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi Frank,
The value for that is mds_beacon_grace. Default is 15 but you can jack it up. Apply it to the monitor or global to take effect.
Just to clarify too, does the MDS daemon come into up:active ? If it does, are you able to also access that portion of the filesystem in that time?
If you can access the filesystem, try running a stat on that portion with something like 'find . -ls' in a directory and see if the strays decrease.
Regards,
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 17:18, Frank Schilder wrote:
Hi Bailey,
thanks for your response. The MDS was actually unresponsive and I had to restart it (ceph tell and ceph daemon commands were hanging, except for "help"). Its currently in clientreplay and loading all the stuff again. I'm really worried that this here is the rescue killer:
heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s
Do you have any idea how to deal with this timeout? Somewhere in he process the MDS seems to become unresponsive for too long and seems to become unresponsive after that.
I have 4T swap now and the MDS comes up to the point where it actually reports back a number for the stray items. However, some time after it becomes unresponsive and the heartbeat messages start showing up. I don't know how to get past this point.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Friday, January 10, 2025 10:05 PM To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Frank,
You mentioned previously a large number of strays on the mds rank. Are you able to check the rank again to see how many strays there are again? We've previously had a similar issue, and once the MDS came back up we had to stat the filesystem to decrease the number of strays, and which doing so everything returned to normal.
ceph tell mds.X perf dump | jq .mds_cache
Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868
On 1/10/25 16:42, Frank Schilder wrote:
Hi all,
I got the MDS up. however, after quite some time its sitting with almost no CPU load:
top - 21:40:02 up 2:49, 1 user, load average: 0.00, 0.02, 0.34 Tasks: 606 total, 1 running, 247 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 503.7 total, 12.3 free, 490.3 used, 1.1 buff/cache GiB Swap: 3577.0 total, 3367.0 free, 210.0 used. 2.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 59495 ceph 20 0 685.8g 477.9g 0.0g S 1.0 94.9 53:47.57 ceph-mds
I'm not sure if its doing anything at all. Only messages like these keep showing up in the log:
2025-01-10T21:38:08.459+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T21:38:08.459+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 3019.23s ago); MDS internal heartbeat is not healthy!
The MDS cluster looks healthy from this output:
# ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-15 Reqs: 0 /s 255k 248k 5434 1678 1 active ceph-14 Reqs: 2 /s 402k 396k 26.7k 144k 2 active ceph-12 Reqs: 0 /s 86.9M 86.9M 46.2k 3909 3 active ceph-08 Reqs: 0 /s 637k 630k 2663 7457 4 active ceph-11 Reqs: 0 /s 1496k 1492k 113k 103k 5 active ceph-16 Reqs: 2 /s 775k 769k 65.3k 12.9k 6 active ceph-24 Reqs: 0 /s 130k 113k 7294 8670 7 active ceph-13 Reqs: 65 /s 3619k 3609k 469k 47.2k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 7269G con-fs2-meta2 data 0 7258G con-fs2-data data 1225T 2476T con-fs2-data-ec-ssd data 794G 22.6T con-fs2-data2 data 5747T 2253T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-17 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)
Did it mark itself out of the cluster and is waiting for the MON to fail it?? Please help.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, January 10, 2025 8:51 PM To: Spencer Macphee Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
Hi all,
I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything:
2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy!
I hope it doesn't get failed by some king of timeout now.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Spencer Macphee <spencerofsydney@gmail.com> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Help needed, ceph fs down due to large stray dir
I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays.
You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, my hopes are down again. The MDS might look busy but I'm not sure its doing anything interesting. I now see a lot of these in the log (stripped the heartbeat messages): 2025-01-11T12:35:50.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:50.713867+0100) 2025-01-11T12:35:51.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:51.714027+0100) 2025-01-11T12:35:52.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:52.714335+0100) 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 Looks like the auth key for the MDS expired and cannot be renewed. Is there a grace period for that as well? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 11:41 AM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi all, new update: after sleeping after the final MDS restart the MDS is doing something! It is still unresponsive, but it does show CPU load of between 150-200% and I really really hope that this is the trimming of stray items. I will try to find out if I get perf to work inside the container. For now, to facilitate trouble shooting, I will add a swap disk to every MDS host just to be on the safe side if stuff fails over. Just to get my hopes back: can someone (from the dev team) let me know if it is expected that an MDS is unresponsive during stray evaluation? Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
And another small piece of information: Needed to do another restart. This time I managed to capture the approximate length of the period for which the MDS is up and responsive after loading the cache (it reports stats). Its pretty much exactly 60 seconds. This smells like a timeout. Is there any MDS/ceph-fs related timeout with a 60s default somewhere? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 12:46 PM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi all, my hopes are down again. The MDS might look busy but I'm not sure its doing anything interesting. I now see a lot of these in the log (stripped the heartbeat messages): 2025-01-11T12:35:50.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:50.713867+0100) 2025-01-11T12:35:51.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:51.714027+0100) 2025-01-11T12:35:52.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:52.714335+0100) 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 Looks like the auth key for the MDS expired and cannot be renewed. Is there a grace period for that as well? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 11:41 AM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi all, new update: after sleeping after the final MDS restart the MDS is doing something! It is still unresponsive, but it does show CPU load of between 150-200% and I really really hope that this is the trimming of stray items. I will try to find out if I get perf to work inside the container. For now, to facilitate trouble shooting, I will add a swap disk to every MDS host just to be on the safe side if stuff fails over. Just to get my hopes back: can someone (from the dev team) let me know if it is expected that an MDS is unresponsive during stray evaluation? Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, not sure if this already has been mentioned, but this one has 60 seconds timeout: mds_beacon_mon_down_grace ceph config help mds_beacon_mon_down_grace mds_beacon_mon_down_grace - tolerance in seconds for missed MDS beacons to monitors (secs, advanced) Default: 60 Can update at runtime: true Services: [mon] Maybe bumping that up could help here? Zitat von Frank Schilder <frans@dtu.dk>:
And another small piece of information:
Needed to do another restart. This time I managed to capture the approximate length of the period for which the MDS is up and responsive after loading the cache (it reports stats). Its pretty much exactly 60 seconds. This smells like a timeout. Is there any MDS/ceph-fs related timeout with a 60s default somewhere?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 12:46 PM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
my hopes are down again. The MDS might look busy but I'm not sure its doing anything interesting. I now see a lot of these in the log (stripped the heartbeat messages):
2025-01-11T12:35:50.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:50.713867+0100) 2025-01-11T12:35:51.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:51.714027+0100) 2025-01-11T12:35:52.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:52.714335+0100) 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092
Looks like the auth key for the MDS expired and cannot be renewed. Is there a grace period for that as well?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 11:41 AM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
new update: after sleeping after the final MDS restart the MDS is doing something! It is still unresponsive, but it does show CPU load of between 150-200% and I really really hope that this is the trimming of stray items.
I will try to find out if I get perf to work inside the container. For now, to facilitate trouble shooting, I will add a swap disk to every MDS host just to be on the safe side if stuff fails over.
Just to get my hopes back: can someone (from the dev team) let me know if it is expected that an MDS is unresponsive during stray evaluation?
Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, thanks for your reply! Its a long shot, but worth trying. I will give it a go. Since you are following: I also observed cephx timeouts. I'm considering to increase the ttl for auth tickets. Do you think auth_service_ticket_ttl (default 3600) is the right parameter? If so, can I just change it up and down or can this have side effects like all daemons loosing their tickets when the ttl is decreased from a very large value? I mainly need the auth ticket for the MDS to remain valid for a long time. I will wait a bit in case you manage to reply. I would like to adjust both values in the next attempt. Otherwise, I will just increase beacon down grace. Thanks a lot again and have a nice Sunday! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Saturday, January 11, 2025 7:59 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Frank, not sure if this already has been mentioned, but this one has 60 seconds timeout: mds_beacon_mon_down_grace ceph config help mds_beacon_mon_down_grace mds_beacon_mon_down_grace - tolerance in seconds for missed MDS beacons to monitors (secs, advanced) Default: 60 Can update at runtime: true Services: [mon] Maybe bumping that up could help here? Zitat von Frank Schilder <frans@dtu.dk>:
And another small piece of information:
Needed to do another restart. This time I managed to capture the approximate length of the period for which the MDS is up and responsive after loading the cache (it reports stats). Its pretty much exactly 60 seconds. This smells like a timeout. Is there any MDS/ceph-fs related timeout with a 60s default somewhere?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 12:46 PM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
my hopes are down again. The MDS might look busy but I'm not sure its doing anything interesting. I now see a lot of these in the log (stripped the heartbeat messages):
2025-01-11T12:35:50.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:50.713867+0100) 2025-01-11T12:35:51.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:51.714027+0100) 2025-01-11T12:35:52.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:52.714335+0100) 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092
Looks like the auth key for the MDS expired and cannot be renewed. Is there a grace period for that as well?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 11:41 AM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
new update: after sleeping after the final MDS restart the MDS is doing something! It is still unresponsive, but it does show CPU load of between 150-200% and I really really hope that this is the trimming of stray items.
I will try to find out if I get perf to work inside the container. For now, to facilitate trouble shooting, I will add a swap disk to every MDS host just to be on the safe side if stuff fails over.
Just to get my hopes back: can someone (from the dev team) let me know if it is expected that an MDS is unresponsive during stray evaluation?
Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Personally, I would only try one change at a time and wait for a result. Otherwise it can get difficult to tell what exactly helped and what not. I have never played with auth_service_ticket_ttl yet, so I can only refer to the docs here:
When the Ceph Storage Cluster sends a ticket for authentication to a Ceph client, the Ceph Storage Cluster assigns that ticket a Time To Live (TTL).
I wouldn't consider a MDS daemons as clients here, so I doubt this will help. But then again, never tried to change it... I hope you'll get it back up soon so you can enjoy at least some portion of the weekend as well. :-) Zitat von Frank Schilder <frans@dtu.dk>:
Hi Eugen,
thanks for your reply! Its a long shot, but worth trying. I will give it a go.
Since you are following: I also observed cephx timeouts. I'm considering to increase the ttl for auth tickets. Do you think auth_service_ticket_ttl (default 3600) is the right parameter? If so, can I just change it up and down or can this have side effects like all daemons loosing their tickets when the ttl is decreased from a very large value?
I mainly need the auth ticket for the MDS to remain valid for a long time.
I will wait a bit in case you manage to reply. I would like to adjust both values in the next attempt. Otherwise, I will just increase beacon down grace.
Thanks a lot again and have a nice Sunday! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Saturday, January 11, 2025 7:59 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi Frank,
not sure if this already has been mentioned, but this one has 60 seconds timeout:
mds_beacon_mon_down_grace
ceph config help mds_beacon_mon_down_grace mds_beacon_mon_down_grace - tolerance in seconds for missed MDS beacons to monitors (secs, advanced) Default: 60 Can update at runtime: true Services: [mon]
Maybe bumping that up could help here?
Zitat von Frank Schilder <frans@dtu.dk>:
And another small piece of information:
Needed to do another restart. This time I managed to capture the approximate length of the period for which the MDS is up and responsive after loading the cache (it reports stats). Its pretty much exactly 60 seconds. This smells like a timeout. Is there any MDS/ceph-fs related timeout with a 60s default somewhere?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 12:46 PM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
my hopes are down again. The MDS might look busy but I'm not sure its doing anything interesting. I now see a lot of these in the log (stripped the heartbeat messages):
2025-01-11T12:35:50.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:50.713867+0100) 2025-01-11T12:35:51.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:51.714027+0100) 2025-01-11T12:35:52.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:52.714335+0100) 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092
Looks like the auth key for the MDS expired and cannot be renewed. Is there a grace period for that as well?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 11:41 AM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
new update: after sleeping after the final MDS restart the MDS is doing something! It is still unresponsive, but it does show CPU load of between 150-200% and I really really hope that this is the trimming of stray items.
I will try to find out if I get perf to work inside the container. For now, to facilitate trouble shooting, I will add a swap disk to every MDS host just to be on the safe side if stuff fails over.
Just to get my hopes back: can someone (from the dev team) let me know if it is expected that an MDS is unresponsive during stray evaluation?
Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, thanks and yes, let's try one thing at a time. I will report back. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Saturday, January 11, 2025 10:39 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Personally, I would only try one change at a time and wait for a result. Otherwise it can get difficult to tell what exactly helped and what not. I have never played with auth_service_ticket_ttl yet, so I can only refer to the docs here:
When the Ceph Storage Cluster sends a ticket for authentication to a Ceph client, the Ceph Storage Cluster assigns that ticket a Time To Live (TTL).
I wouldn't consider a MDS daemons as clients here, so I doubt this will help. But then again, never tried to change it... I hope you'll get it back up soon so you can enjoy at least some portion of the weekend as well. :-) Zitat von Frank Schilder <frans@dtu.dk>:
Hi Eugen,
thanks for your reply! Its a long shot, but worth trying. I will give it a go.
Since you are following: I also observed cephx timeouts. I'm considering to increase the ttl for auth tickets. Do you think auth_service_ticket_ttl (default 3600) is the right parameter? If so, can I just change it up and down or can this have side effects like all daemons loosing their tickets when the ttl is decreased from a very large value?
I mainly need the auth ticket for the MDS to remain valid for a long time.
I will wait a bit in case you manage to reply. I would like to adjust both values in the next attempt. Otherwise, I will just increase beacon down grace.
Thanks a lot again and have a nice Sunday! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Saturday, January 11, 2025 7:59 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi Frank,
not sure if this already has been mentioned, but this one has 60 seconds timeout:
mds_beacon_mon_down_grace
ceph config help mds_beacon_mon_down_grace mds_beacon_mon_down_grace - tolerance in seconds for missed MDS beacons to monitors (secs, advanced) Default: 60 Can update at runtime: true Services: [mon]
Maybe bumping that up could help here?
Zitat von Frank Schilder <frans@dtu.dk>:
And another small piece of information:
Needed to do another restart. This time I managed to capture the approximate length of the period for which the MDS is up and responsive after loading the cache (it reports stats). Its pretty much exactly 60 seconds. This smells like a timeout. Is there any MDS/ceph-fs related timeout with a 60s default somewhere?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 12:46 PM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
my hopes are down again. The MDS might look busy but I'm not sure its doing anything interesting. I now see a lot of these in the log (stripped the heartbeat messages):
2025-01-11T12:35:50.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:50.713867+0100) 2025-01-11T12:35:51.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:51.714027+0100) 2025-01-11T12:35:52.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:52.714335+0100) 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092
Looks like the auth key for the MDS expired and cannot be renewed. Is there a grace period for that as well?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 11:41 AM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
new update: after sleeping after the final MDS restart the MDS is doing something! It is still unresponsive, but it does show CPU load of between 150-200% and I really really hope that this is the trimming of stray items.
I will try to find out if I get perf to work inside the container. For now, to facilitate trouble shooting, I will add a swap disk to every MDS host just to be on the safe side if stuff fails over.
Just to get my hopes back: can someone (from the dev team) let me know if it is expected that an MDS is unresponsive during stray evaluation?
Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, as promised the result. Unfortunately, increasing this parameter seems not to help. Was worth a try though. I will keep the MDS running and check again tomorrow. Its really annoying that it doesn't come back. Following the reports of other people who were in a similar situation it should just work after adding swap. I wonder what is so special about our case. Thanks for your input and have a good night! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 10:43 PM To: Eugen Block Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Eugen, thanks and yes, let's try one thing at a time. I will report back. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Saturday, January 11, 2025 10:39 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Personally, I would only try one change at a time and wait for a result. Otherwise it can get difficult to tell what exactly helped and what not. I have never played with auth_service_ticket_ttl yet, so I can only refer to the docs here:
When the Ceph Storage Cluster sends a ticket for authentication to a Ceph client, the Ceph Storage Cluster assigns that ticket a Time To Live (TTL).
I wouldn't consider a MDS daemons as clients here, so I doubt this will help. But then again, never tried to change it... I hope you'll get it back up soon so you can enjoy at least some portion of the weekend as well. :-) Zitat von Frank Schilder <frans@dtu.dk>:
Hi Eugen,
thanks for your reply! Its a long shot, but worth trying. I will give it a go.
Since you are following: I also observed cephx timeouts. I'm considering to increase the ttl for auth tickets. Do you think auth_service_ticket_ttl (default 3600) is the right parameter? If so, can I just change it up and down or can this have side effects like all daemons loosing their tickets when the ttl is decreased from a very large value?
I mainly need the auth ticket for the MDS to remain valid for a long time.
I will wait a bit in case you manage to reply. I would like to adjust both values in the next attempt. Otherwise, I will just increase beacon down grace.
Thanks a lot again and have a nice Sunday! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Saturday, January 11, 2025 7:59 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi Frank,
not sure if this already has been mentioned, but this one has 60 seconds timeout:
mds_beacon_mon_down_grace
ceph config help mds_beacon_mon_down_grace mds_beacon_mon_down_grace - tolerance in seconds for missed MDS beacons to monitors (secs, advanced) Default: 60 Can update at runtime: true Services: [mon]
Maybe bumping that up could help here?
Zitat von Frank Schilder <frans@dtu.dk>:
And another small piece of information:
Needed to do another restart. This time I managed to capture the approximate length of the period for which the MDS is up and responsive after loading the cache (it reports stats). Its pretty much exactly 60 seconds. This smells like a timeout. Is there any MDS/ceph-fs related timeout with a 60s default somewhere?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 12:46 PM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
my hopes are down again. The MDS might look busy but I'm not sure its doing anything interesting. I now see a lot of these in the log (stripped the heartbeat messages):
2025-01-11T12:35:50.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:50.713867+0100) 2025-01-11T12:35:51.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:51.714027+0100) 2025-01-11T12:35:52.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:52.714335+0100) 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092
Looks like the auth key for the MDS expired and cannot be renewed. Is there a grace period for that as well?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 11:41 AM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
new update: after sleeping after the final MDS restart the MDS is doing something! It is still unresponsive, but it does show CPU load of between 150-200% and I really really hope that this is the trimming of stray items.
I will try to find out if I get perf to work inside the container. For now, to facilitate trouble shooting, I will add a swap disk to every MDS host just to be on the safe side if stuff fails over.
Just to get my hopes back: can someone (from the dev team) let me know if it is expected that an MDS is unresponsive during stray evaluation?
Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, It's possible that certain parameters you modified at some point, which may have helped the MDS to start up, are now slowing down its operation or preventing it from going further. In that case, resetting these parameters to their default values could help. Just a thought. Another thing... did you have a chance to mount and 'stat' this tree like Bailey suggested on Friday and monitor stray entries with a perf dump or by Spencer's advice (looking at omapkeys of special rados objects in the metadata pool)? Not sure if the 'stat' would help in the current state the MDS is in, but you may want to consider this move if you can. Regards, Frédéric. ________________________________ De : Frank Schilder <frans@dtu.dk> Envoyé : dimanche 12 janvier 2025 00:07 À : Eugen Block Cc: ceph-users@ceph.io Objet : [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Eugen, as promised the result. Unfortunately, increasing this parameter seems not to help. Was worth a try though. I will keep the MDS running and check again tomorrow. Its really annoying that it doesn't come back. Following the reports of other people who were in a similar situation it should just work after adding swap. I wonder what is so special about our case. Thanks for your input and have a good night! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 10:43 PM To: Eugen Block Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Eugen, thanks and yes, let's try one thing at a time. I will report back. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Saturday, January 11, 2025 10:39 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Personally, I would only try one change at a time and wait for a result. Otherwise it can get difficult to tell what exactly helped and what not. I have never played with auth_service_ticket_ttl yet, so I can only refer to the docs here:
When the Ceph Storage Cluster sends a ticket for authentication to a Ceph client, the Ceph Storage Cluster assigns that ticket a Time To Live (TTL).
I wouldn't consider a MDS daemons as clients here, so I doubt this will help. But then again, never tried to change it... I hope you'll get it back up soon so you can enjoy at least some portion of the weekend as well. :-) Zitat von Frank Schilder <frans@dtu.dk>:
Hi Eugen,
thanks for your reply! Its a long shot, but worth trying. I will give it a go.
Since you are following: I also observed cephx timeouts. I'm considering to increase the ttl for auth tickets. Do you think auth_service_ticket_ttl (default 3600) is the right parameter? If so, can I just change it up and down or can this have side effects like all daemons loosing their tickets when the ttl is decreased from a very large value?
I mainly need the auth ticket for the MDS to remain valid for a long time.
I will wait a bit in case you manage to reply. I would like to adjust both values in the next attempt. Otherwise, I will just increase beacon down grace.
Thanks a lot again and have a nice Sunday! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Saturday, January 11, 2025 7:59 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi Frank,
not sure if this already has been mentioned, but this one has 60 seconds timeout:
mds_beacon_mon_down_grace
ceph config help mds_beacon_mon_down_grace mds_beacon_mon_down_grace - tolerance in seconds for missed MDS beacons to monitors (secs, advanced) Default: 60 Can update at runtime: true Services: [mon]
Maybe bumping that up could help here?
Zitat von Frank Schilder <frans@dtu.dk>:
And another small piece of information:
Needed to do another restart. This time I managed to capture the approximate length of the period for which the MDS is up and responsive after loading the cache (it reports stats). Its pretty much exactly 60 seconds. This smells like a timeout. Is there any MDS/ceph-fs related timeout with a 60s default somewhere?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 12:46 PM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
my hopes are down again. The MDS might look busy but I'm not sure its doing anything interesting. I now see a lot of these in the log (stripped the heartbeat messages):
2025-01-11T12:35:50.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:50.713867+0100) 2025-01-11T12:35:51.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:51.714027+0100) 2025-01-11T12:35:52.712+0100 7ff888375700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2025-01-11T11:35:52.714335+0100) 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.084+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.353+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.536+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 auth: could not find secret_id=51092 2025-01-11T12:35:53.573+0100 7ff88cb7e700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=51092
Looks like the auth key for the MDS expired and cannot be renewed. Is there a grace period for that as well?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 11, 2025 11:41 AM To: Dan van der Ster Cc: Bailey Allison; ceph-users@ceph.io Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
new update: after sleeping after the final MDS restart the MDS is doing something! It is still unresponsive, but it does show CPU load of between 150-200% and I really really hope that this is the trimming of stray items.
I will try to find out if I get perf to work inside the container. For now, to facilitate trouble shooting, I will add a swap disk to every MDS host just to be on the safe side if stuff fails over.
Just to get my hopes back: can someone (from the dev team) let me know if it is expected that an MDS is unresponsive during stray evaluation?
Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear all, a quick update and some answers. We set up a dedicated host for running an MDS and debugging the problem. On this host we have 750G RAM, 4T swap and 4T log, both on fast SSDs. Plan is to monitor with "perf top" the MDS becoming the designated MDS for the problematic rank and also pull out a detailed log about the startup until the MDS hangs. I have some questions about that, a new observation that might be relevant and some answers to some suggestions, in that order. I need to install debug info for perf to give useful output. I can't find a meta package that has all ceph-debuginfo rpms as a dependency. Which ones should I install? Also, should I install some kernel debug info packages? Please note that every restart takes about 1h to hit the issue. I would like to have as much as possible installed the first time. A new observation: After every restart cycle the rank loads a little bit less into cache. However, the num_stray count does not decrease. Could that mean the problem is not the high num_stray count but something else? Answer to a suggestion: It is not possible to access anything on the MDS or the entire file system. Hence, trying to stat some files/dirs is not possible. Furthermore, not all stray items can be reintegrated with this method and I'm afraid our stray items are mostly of this nature. This means that in octopus the only way to trim (evaluate) stray items was an MDS restart. For details, the relevant discussions are https://www.spinics.net/lists/ceph-users/msg70459.html and https://www.spinics.net/lists/ceph-users/msg73150.html with the most important info in this message: https://www.spinics.net/lists/ceph-users/msg70849.html . Summary: As part of the debugging back then I executed a recursive stat-ing of files and directories on the *entire* file system only to observe that the stray count didn't change. This was when Gregory finally explained that hard links can block stray removal on snaptrim also for paths that are no longer accessible through the file system or any snapshots, that is, the usual stray evaluation doesn't have any effect. That's the situation we are in, we need the MDS do it itself. A correction: It was actually Venky Shankar participating in this communication and not Neha. Is Venkhy still working on the FS? Thanks for package hints and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Hi Frank, More than ever. You should open a tracker and post debug logs there so anyone can have a look. Regards, Frédéric. ________________________________ De : Frank Schilder <frans@dtu.dk> Envoyé : lundi 13 janvier 2025 17:39 À : ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Objet : [ceph-users] Re: Help needed, ceph fs down due to large stray dir Dear all, a quick update and some answers. We set up a dedicated host for running an MDS and debugging the problem. On this host we have 750G RAM, 4T swap and 4T log, both on fast SSDs. Plan is to monitor with "perf top" the MDS becoming the designated MDS for the problematic rank and also pull out a detailed log about the startup until the MDS hangs. I have some questions about that, a new observation that might be relevant and some answers to some suggestions, in that order. I need to install debug info for perf to give useful output. I can't find a meta package that has all ceph-debuginfo rpms as a dependency. Which ones should I install? Also, should I install some kernel debug info packages? Please note that every restart takes about 1h to hit the issue. I would like to have as much as possible installed the first time. A new observation: After every restart cycle the rank loads a little bit less into cache. However, the num_stray count does not decrease. Could that mean the problem is not the high num_stray count but something else? Answer to a suggestion: It is not possible to access anything on the MDS or the entire file system. Hence, trying to stat some files/dirs is not possible. Furthermore, not all stray items can be reintegrated with this method and I'm afraid our stray items are mostly of this nature. This means that in octopus the only way to trim (evaluate) stray items was an MDS restart. For details, the relevant discussions are https://www.spinics.net/lists/ceph-users/msg70459.html and https://www.spinics.net/lists/ceph-users/msg73150.html with the most important info in this message: https://www.spinics.net/lists/ceph-users/msg70849.html . Summary: As part of the debugging back then I executed a recursive stat-ing of files and directories on the *entire* file system only to observe that the stray count didn't change. This was when Gregory finally explained that hard links can block stray removal on snaptrim also for paths that are no longer accessible through the file system or any snapshots, that is, the usual stray evaluation doesn't have any effect. That's the situation we are in, we need the MDS do it itself. A correction: It was actually Venky Shankar participating in this communication and not Neha. Is Venkhy still working on the FS? Thanks for package hints and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear all, we finally managed to collect perf data and it seems to show a smoking gun. Since this thread is already heavily cluttered on lists.ceph.io I started a new one: "MDS hung in purge_stale_snap_data after populating cache" (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XLYHTZBD4AXN...). Please take a look at this post. Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Tuesday, January 14, 2025 8:25 AM To: Frank Schilder; ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Frank, More than ever. You should open a tracker and post debug logs there so anyone can have a look. Regards, Frédéric. ________________________________ De : Frank Schilder <frans@dtu.dk> Envoyé : lundi 13 janvier 2025 17:39 À : ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Objet : [ceph-users] Re: Help needed, ceph fs down due to large stray dir Dear all, a quick update and some answers. We set up a dedicated host for running an MDS and debugging the problem. On this host we have 750G RAM, 4T swap and 4T log, both on fast SSDs. Plan is to monitor with "perf top" the MDS becoming the designated MDS for the problematic rank and also pull out a detailed log about the startup until the MDS hangs. I have some questions about that, a new observation that might be relevant and some answers to some suggestions, in that order. I need to install debug info for perf to give useful output. I can't find a meta package that has all ceph-debuginfo rpms as a dependency. Which ones should I install? Also, should I install some kernel debug info packages? Please note that every restart takes about 1h to hit the issue. I would like to have as much as possible installed the first time. A new observation: After every restart cycle the rank loads a little bit less into cache. However, the num_stray count does not decrease. Could that mean the problem is not the high num_stray count but something else? Answer to a suggestion: It is not possible to access anything on the MDS or the entire file system. Hence, trying to stat some files/dirs is not possible. Furthermore, not all stray items can be reintegrated with this method and I'm afraid our stray items are mostly of this nature. This means that in octopus the only way to trim (evaluate) stray items was an MDS restart. For details, the relevant discussions are https://www.spinics.net/lists/ceph-users/msg70459.html and https://www.spinics.net/lists/ceph-users/msg73150.html with the most important info in this message: https://www.spinics.net/lists/ceph-users/msg70849.html . Summary: As part of the debugging back then I executed a recursive stat-ing of files and directories on the *entire* file system only to observe that the stray count didn't change. This was when Gregory finally explained that hard links can block stray removal on snaptrim also for paths that are no longer accessible through the file system or any snapshots, that is, the usual stray evaluation doesn't have any effect. That's the situation we are in, we need the MDS do it itself. A correction: It was actually Venky Shankar participating in this communication and not Neha. Is Venkhy still working on the FS? Thanks for package hints and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, looking at the log data (see snippet at end) we suspect a classic "producer–consumer" deadlock since it seems that the same thread that is filling the purge queue at PurgeQueue.cc:L335:journaler.append_entry(bl) in function PurgeQueue::push is also responsible for processing it but the call to scan_stray_dir() at the end of MDCache::populate_mydir() is atomic and blocks processing until it completes. I'm looking through the code right now to find what parameter is used in constructing this queue that allows adjusting its size. I need to triple it in size at least to get scan_stray_dir() to complete and see what happens then. This would, of course, go much faster if someone familiar with the code would tell me what that parameter is. Here the log snippet when the purge queue runs full: 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.ino(0x20028c29e69) purge_stale_snap_data 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.ino(0x20028c29e69) purging old_inode [c7d,dae] 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache clear_dirty_bits_for_stray [inode 0x20028c29e69 [...daf,head] ~mds2/stray7/20028c29e69/ auth v2199183125 snaprealm=0x562c9fb6e600 f(v0 m2024-12-28T20:17:59.487395+0100) n(v0 rc2024-12-28T20:17:59.487395+0100 1=0+1) (iversion lock) 0x562c9fb70b00] 2025-01-16T14:06:00.490+0100 7f25cfa31700 20 mds.2.cache.strays enqueue: purging dn: [dentry #0x102/stray7/20028c29e69 [daf,head] auth (dversion lock) pv=0 v=2199200546 ino=0x20028c29e69 state=1073741844 0x562c9fb6db80] 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.dir(0x61b.000100011011*) auth_pin by 0x55e0f256ff80 on [dir 0x61b.000100011011* ~mds2/stray7/ [2,head] auth v=2199200547 cv=2199200546/2199200546 ap=1928+0 state=1610620929|complete|sticky f(v23 m2025-01-10T13:30:41.594215+0100 2568=0+2568)/f(v23 m2025-01-10T13:30:41.594215+0100 2569=1+2568) n(v2110 rc2025-01-10T13:30:41.594215+0100 2568=0+2568)/n(v2110 rc2025-01-10T13:30:41.594215+0100 b7068 2569=1+2568) hs=2568+0,ss=0+0 | child=1 sticky=1 dirty=1 waiter=0 authpin=1 0x562c98bf0880] count now 1928 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.strays purge [dentry #0x102/stray7/20028c29e69 [daf,head] auth (dversion lock) pv=0 v=2199200546 ino=0x20028c29e69 state=1073741876 | purging=1 0x562c9fb6db80] [inode 0x20028c29e69 [...daf,head] ~mds2/stray7/20028c29e69/ auth v2199183125 snaprealm=0x562c9fb6e600 f(v0 m2024-12-28T20:17:59.487395+0100) n(v0 rc2024-12-28T20:17:59.487395+0100 1=0+1) (iversion lock) 0x562c9fb70b00] 2025-01-16T14:06:00.490+0100 7f25cfa31700 4 mds.2.purge_queue push: pushing inode 0x20028c29e69 2025-01-16T14:06:00.537+0100 7f25d7240700 20 -- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 legacy=0x55e0f3d34000 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=0).process 2025-01-16T14:06:00.537+0100 7f25d7240700 20 -- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 legacy=0x55e0f3d34000 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=1 2025-01-16T14:06:00.537+0100 7f25d7240700 20 --1- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 0x55e0f3d34000 :6801 s=OPENED pgs=828965 cs=1 l=0).handle_message r=0 2025-01-16T14:06:00.537+0100 7f25d7240700 20 --1- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 0x55e0f3d34000 :6801 s=OPENED pgs=828965 cs=1 l=0).handle_message process tag 14 Best regards and have a good weekend. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Hi all, we took a log with setting debug_journaler=20 and managed to track the deadlock down to line https://github.com/ceph/ceph/blob/pacific/src/osdc/Journaler.cc#L583 in Journaler::append_entry(bufferlist& bl): // append size_t delta = bl.length() + journal_stream.get_envelope_size(); // write_buf space is nearly full if (!write_buf_throttle.get_or_fail(delta)) { l.unlock(); ldout(cct, 10) << "write_buf_throttle wait, delta " << delta << dendl; write_buf_throttle.get(delta); //<<<<<<<<< The MDS is stuck here <<<<<<<<< l.lock(); } ldout(cct, 20) << "write_buf_throttle get, delta " << delta << dendl; This is indicated by the last message in the log before the lock up, which reads mds.2.journaler.pq(rw) write_buf_throttle wait, delta 101 and is generated by the line above the call write_buf_throttle.get(delta). All log messages messages before start with "write_buf_throttle get, delta", which means these did not go into the if-statement. Obvious question is, which parameter influences the maximum capacity of Journaler::write_buffer (https://github.com/ceph/ceph/blob/pacific/src/osdc/Journaler.h#L306) in the class definition of class Journaler? Increasing this limit should get us past the deadlock. Note that all the relevant code is identical to branch main, which means that all versions since pacific are affected. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 18, 2025 2:21 PM To: Frédéric Nass; ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi all, looking at the log data (see snippet at end) we suspect a classic "producer–consumer" deadlock since it seems that the same thread that is filling the purge queue at PurgeQueue.cc:L335:journaler.append_entry(bl) in function PurgeQueue::push is also responsible for processing it but the call to scan_stray_dir() at the end of MDCache::populate_mydir() is atomic and blocks processing until it completes. I'm looking through the code right now to find what parameter is used in constructing this queue that allows adjusting its size. I need to triple it in size at least to get scan_stray_dir() to complete and see what happens then. This would, of course, go much faster if someone familiar with the code would tell me what that parameter is. Here the log snippet when the purge queue runs full: 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.ino(0x20028c29e69) purge_stale_snap_data 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.ino(0x20028c29e69) purging old_inode [c7d,dae] 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache clear_dirty_bits_for_stray [inode 0x20028c29e69 [...daf,head] ~mds2/stray7/20028c29e69/ auth v2199183125 snaprealm=0x562c9fb6e600 f(v0 m2024-12-28T20:17:59.487395+0100) n(v0 rc2024-12-28T20:17:59.487395+0100 1=0+1) (iversion lock) 0x562c9fb70b00] 2025-01-16T14:06:00.490+0100 7f25cfa31700 20 mds.2.cache.strays enqueue: purging dn: [dentry #0x102/stray7/20028c29e69 [daf,head] auth (dversion lock) pv=0 v=2199200546 ino=0x20028c29e69 state=1073741844 0x562c9fb6db80] 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.dir(0x61b.000100011011*) auth_pin by 0x55e0f256ff80 on [dir 0x61b.000100011011* ~mds2/stray7/ [2,head] auth v=2199200547 cv=2199200546/2199200546 ap=1928+0 state=1610620929|complete|sticky f(v23 m2025-01-10T13:30:41.594215+0100 2568=0+2568)/f(v23 m2025-01-10T13:30:41.594215+0100 2569=1+2568) n(v2110 rc2025-01-10T13:30:41.594215+0100 2568=0+2568)/n(v2110 rc2025-01-10T13:30:41.594215+0100 b7068 2569=1+2568) hs=2568+0,ss=0+0 | child=1 sticky=1 dirty=1 waiter=0 authpin=1 0x562c98bf0880] count now 1928 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.strays purge [dentry #0x102/stray7/20028c29e69 [daf,head] auth (dversion lock) pv=0 v=2199200546 ino=0x20028c29e69 state=1073741876 | purging=1 0x562c9fb6db80] [inode 0x20028c29e69 [...daf,head] ~mds2/stray7/20028c29e69/ auth v2199183125 snaprealm=0x562c9fb6e600 f(v0 m2024-12-28T20:17:59.487395+0100) n(v0 rc2024-12-28T20:17:59.487395+0100 1=0+1) (iversion lock) 0x562c9fb70b00] 2025-01-16T14:06:00.490+0100 7f25cfa31700 4 mds.2.purge_queue push: pushing inode 0x20028c29e69 2025-01-16T14:06:00.537+0100 7f25d7240700 20 -- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 legacy=0x55e0f3d34000 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=0).process 2025-01-16T14:06:00.537+0100 7f25d7240700 20 -- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 legacy=0x55e0f3d34000 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=1 2025-01-16T14:06:00.537+0100 7f25d7240700 20 --1- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 0x55e0f3d34000 :6801 s=OPENED pgs=828965 cs=1 l=0).handle_message r=0 2025-01-16T14:06:00.537+0100 7f25d7240700 20 --1- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 0x55e0f3d34000 :6801 s=OPENED pgs=828965 cs=1 l=0).handle_message process tag 14 Best regards and have a good weekend. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, with the help of Croit we got back on our feet. I will post a detailed post-mortem later this month including information about how to check if a cluster is in the same situation. Long story short, we hit a deadlock due to competition between MDS cache trimming and purging stale strays. "Disabling" cache trimming by setting a ridiculously high mds_memory_limit on the bad rank did the trick. Purging 100Mio strays is actually no problem and doesn't require much if any RAM by itself (I mean here the purge that happens on MDS restart, I don't know if the forward-scrub purge behaves the same). Our cluster managed to purge about 10K items/s and after a few hours everything was cleaned out. While purging it was serving client IO, so the FS is up right away. A big thank you to everyone who helped with this case. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Sunday, January 19, 2025 5:35 PM To: ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi all, we took a log with setting debug_journaler=20 and managed to track the deadlock down to line https://github.com/ceph/ceph/blob/pacific/src/osdc/Journaler.cc#L583 in Journaler::append_entry(bufferlist& bl): // append size_t delta = bl.length() + journal_stream.get_envelope_size(); // write_buf space is nearly full if (!write_buf_throttle.get_or_fail(delta)) { l.unlock(); ldout(cct, 10) << "write_buf_throttle wait, delta " << delta << dendl; write_buf_throttle.get(delta); //<<<<<<<<< The MDS is stuck here <<<<<<<<< l.lock(); } ldout(cct, 20) << "write_buf_throttle get, delta " << delta << dendl; This is indicated by the last message in the log before the lock up, which reads mds.2.journaler.pq(rw) write_buf_throttle wait, delta 101 and is generated by the line above the call write_buf_throttle.get(delta). All log messages messages before start with "write_buf_throttle get, delta", which means these did not go into the if-statement. Obvious question is, which parameter influences the maximum capacity of Journaler::write_buffer (https://github.com/ceph/ceph/blob/pacific/src/osdc/Journaler.h#L306) in the class definition of class Journaler? Increasing this limit should get us past the deadlock. Note that all the relevant code is identical to branch main, which means that all versions since pacific are affected. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 18, 2025 2:21 PM To: Frédéric Nass; ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi all, looking at the log data (see snippet at end) we suspect a classic "producer–consumer" deadlock since it seems that the same thread that is filling the purge queue at PurgeQueue.cc:L335:journaler.append_entry(bl) in function PurgeQueue::push is also responsible for processing it but the call to scan_stray_dir() at the end of MDCache::populate_mydir() is atomic and blocks processing until it completes. I'm looking through the code right now to find what parameter is used in constructing this queue that allows adjusting its size. I need to triple it in size at least to get scan_stray_dir() to complete and see what happens then. This would, of course, go much faster if someone familiar with the code would tell me what that parameter is. Here the log snippet when the purge queue runs full: 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.ino(0x20028c29e69) purge_stale_snap_data 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.ino(0x20028c29e69) purging old_inode [c7d,dae] 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache clear_dirty_bits_for_stray [inode 0x20028c29e69 [...daf,head] ~mds2/stray7/20028c29e69/ auth v2199183125 snaprealm=0x562c9fb6e600 f(v0 m2024-12-28T20:17:59.487395+0100) n(v0 rc2024-12-28T20:17:59.487395+0100 1=0+1) (iversion lock) 0x562c9fb70b00] 2025-01-16T14:06:00.490+0100 7f25cfa31700 20 mds.2.cache.strays enqueue: purging dn: [dentry #0x102/stray7/20028c29e69 [daf,head] auth (dversion lock) pv=0 v=2199200546 ino=0x20028c29e69 state=1073741844 0x562c9fb6db80] 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.dir(0x61b.000100011011*) auth_pin by 0x55e0f256ff80 on [dir 0x61b.000100011011* ~mds2/stray7/ [2,head] auth v=2199200547 cv=2199200546/2199200546 ap=1928+0 state=1610620929|complete|sticky f(v23 m2025-01-10T13:30:41.594215+0100 2568=0+2568)/f(v23 m2025-01-10T13:30:41.594215+0100 2569=1+2568) n(v2110 rc2025-01-10T13:30:41.594215+0100 2568=0+2568)/n(v2110 rc2025-01-10T13:30:41.594215+0100 b7068 2569=1+2568) hs=2568+0,ss=0+0 | child=1 sticky=1 dirty=1 waiter=0 authpin=1 0x562c98bf0880] count now 1928 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.strays purge [dentry #0x102/stray7/20028c29e69 [daf,head] auth (dversion lock) pv=0 v=2199200546 ino=0x20028c29e69 state=1073741876 | purging=1 0x562c9fb6db80] [inode 0x20028c29e69 [...daf,head] ~mds2/stray7/20028c29e69/ auth v2199183125 snaprealm=0x562c9fb6e600 f(v0 m2024-12-28T20:17:59.487395+0100) n(v0 rc2024-12-28T20:17:59.487395+0100 1=0+1) (iversion lock) 0x562c9fb70b00] 2025-01-16T14:06:00.490+0100 7f25cfa31700 4 mds.2.purge_queue push: pushing inode 0x20028c29e69 2025-01-16T14:06:00.537+0100 7f25d7240700 20 -- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 legacy=0x55e0f3d34000 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=0).process 2025-01-16T14:06:00.537+0100 7f25d7240700 20 -- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 legacy=0x55e0f3d34000 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=1 2025-01-16T14:06:00.537+0100 7f25d7240700 20 --1- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 0x55e0f3d34000 :6801 s=OPENED pgs=828965 cs=1 l=0).handle_message r=0 2025-01-16T14:06:00.537+0100 7f25d7240700 20 --1- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 0x55e0f3d34000 :6801 s=OPENED pgs=828965 cs=1 l=0).handle_message process tag 14 Best regards and have a good weekend. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That’s great to hear, Frank! I’m glad your CephFS is back up! Thanks for the update! Zitat von Frank Schilder <frans@dtu.dk>:
Hi all,
with the help of Croit we got back on our feet. I will post a detailed post-mortem later this month including information about how to check if a cluster is in the same situation.
Long story short, we hit a deadlock due to competition between MDS cache trimming and purging stale strays. "Disabling" cache trimming by setting a ridiculously high mds_memory_limit on the bad rank did the trick. Purging 100Mio strays is actually no problem and doesn't require much if any RAM by itself (I mean here the purge that happens on MDS restart, I don't know if the forward-scrub purge behaves the same). Our cluster managed to purge about 10K items/s and after a few hours everything was cleaned out. While purging it was serving client IO, so the FS is up right away.
A big thank you to everyone who helped with this case.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Sunday, January 19, 2025 5:35 PM To: ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
we took a log with setting debug_journaler=20 and managed to track the deadlock down to line https://github.com/ceph/ceph/blob/pacific/src/osdc/Journaler.cc#L583 in Journaler::append_entry(bufferlist& bl):
// append size_t delta = bl.length() + journal_stream.get_envelope_size(); // write_buf space is nearly full if (!write_buf_throttle.get_or_fail(delta)) { l.unlock(); ldout(cct, 10) << "write_buf_throttle wait, delta " << delta << dendl; write_buf_throttle.get(delta); //<<<<<<<<< The MDS is stuck here <<<<<<<<< l.lock(); } ldout(cct, 20) << "write_buf_throttle get, delta " << delta << dendl;
This is indicated by the last message in the log before the lock up, which reads
mds.2.journaler.pq(rw) write_buf_throttle wait, delta 101
and is generated by the line above the call write_buf_throttle.get(delta). All log messages messages before start with "write_buf_throttle get, delta", which means these did not go into the if-statement.
Obvious question is, which parameter influences the maximum capacity of Journaler::write_buffer (https://github.com/ceph/ceph/blob/pacific/src/osdc/Journaler.h#L306) in the class definition of class Journaler? Increasing this limit should get us past the deadlock.
Note that all the relevant code is identical to branch main, which means that all versions since pacific are affected.
Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Saturday, January 18, 2025 2:21 PM To: Frédéric Nass; ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Subject: [ceph-users] Re: Help needed, ceph fs down due to large stray dir
Hi all,
looking at the log data (see snippet at end) we suspect a classic "producer–consumer" deadlock since it seems that the same thread that is filling the purge queue at PurgeQueue.cc:L335:journaler.append_entry(bl) in function PurgeQueue::push is also responsible for processing it but the call to scan_stray_dir() at the end of MDCache::populate_mydir() is atomic and blocks processing until it completes.
I'm looking through the code right now to find what parameter is used in constructing this queue that allows adjusting its size. I need to triple it in size at least to get scan_stray_dir() to complete and see what happens then.
This would, of course, go much faster if someone familiar with the code would tell me what that parameter is.
Here the log snippet when the purge queue runs full:
2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.ino(0x20028c29e69) purge_stale_snap_data 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.ino(0x20028c29e69) purging old_inode [c7d,dae] 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache clear_dirty_bits_for_stray [inode 0x20028c29e69 [...daf,head] ~mds2/stray7/20028c29e69/ auth v2199183125 snaprealm=0x562c9fb6e600 f(v0 m2024-12-28T20:17:59.487395+0100) n(v0 rc2024-12-28T20:17:59.487395+0100 1=0+1) (iversion lock) 0x562c9fb70b00] 2025-01-16T14:06:00.490+0100 7f25cfa31700 20 mds.2.cache.strays enqueue: purging dn: [dentry #0x102/stray7/20028c29e69 [daf,head] auth (dversion lock) pv=0 v=2199200546 ino=0x20028c29e69 state=1073741844 0x562c9fb6db80] 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.dir(0x61b.000100011011*) auth_pin by 0x55e0f256ff80 on [dir 0x61b.000100011011* ~mds2/stray7/ [2,head] auth v=2199200547 cv=2199200546/2199200546 ap=1928+0 state=1610620929|complete|sticky f(v23 m2025-01-10T13:30:41.594215+0100 2568=0+2568)/f(v23 m2025-01-10T13:30:41.594215+0100 2569=1+2568) n(v2110 rc2025-01-10T13:30:41.594215+0100 2568=0+2568)/n(v2110 rc2025-01-10T13:30:41.594215+0100 b7068 2569=1+2568) hs=2568+0,ss=0+0 | child=1 sticky=1 dirty=1 waiter=0 authpin=1 0x562c98bf0880] count now 1928 2025-01-16T14:06:00.490+0100 7f25cfa31700 10 mds.2.cache.strays purge [dentry #0x102/stray7/20028c29e69 [daf,head] auth (dversion lock) pv=0 v=2199200546 ino=0x20028c29e69 state=1073741876 | purging=1 0x562c9fb6db80] [inode 0x20028c29e69 [...daf,head] ~mds2/stray7/20028c29e69/ auth v2199183125 snaprealm=0x562c9fb6e600 f(v0 m2024-12-28T20:17:59.487395+0100) n(v0 rc2024-12-28T20:17:59.487395+0100 1=0+1) (iversion lock) 0x562c9fb70b00] 2025-01-16T14:06:00.490+0100 7f25cfa31700 4 mds.2.purge_queue push: pushing inode 0x20028c29e69 2025-01-16T14:06:00.537+0100 7f25d7240700 20 -- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 legacy=0x55e0f3d34000 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=0).process 2025-01-16T14:06:00.537+0100 7f25d7240700 20 -- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 legacy=0x55e0f3d34000 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=1 2025-01-16T14:06:00.537+0100 7f25d7240700 20 --1- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 0x55e0f3d34000 :6801 s=OPENED pgs=828965 cs=1 l=0).handle_message r=0 2025-01-16T14:06:00.537+0100 7f25d7240700 20 --1- [v2:192.168.32.85:6800/845135215,v1:192.168.32.85:6801/845135215] >> v1:192.168.48.132:0/2119497078 conn(0x55e0f3d3e400 0x55e0f3d34000 :6801 s=OPENED pgs=828965 cs=1 l=0).handle_message process tag 14
Best regards and have a good weekend. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You could try some of the steps here Frank: https://docs.ceph.com/en/quincy/cephfs/troubleshooting/#avoiding-recovery-ro... mds_heartbeat_reset_grace is probably the only one really relevant to your scenario. On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
+1 to this, and the doc mentioned. Just be aware depending on version the heartbeat grace parameter is different, I believe for 16 and below it's the one I mentioned, and it's to be set on the mon level, and for 17 and newer it is what Spencer mentioned. The doc he has provided also mentions such, and some other helpful configs. Again, if you can access the directories of the mds rank in question when it's active, see if you can stat some of them. Best of luck friend, Regards, Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 1/10/25 18:07, Spencer Macphee wrote:
You could try some of the steps here Frank: https://docs.ceph.com/en/quincy/cephfs/troubleshooting/#avoiding-recovery-ro...
mds_heartbeat_reset_grace is probably the only one really relevant to your scenario.
On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
mds_beacon_grace is, perhaps confusingly, not an MDS configuration. It's applied to MONs. As you've injected it into the MDS that is likely why the heartbeat is still failing: This has the effect of having the MDS continue to send beacons to the monitors even when its internal "heartbeat" mechanism has not been reset (beat) in one hour. Note the previous mechanism for achieving this was via the ` mds_beacon_grace` monitor setting. On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted.
I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM.
Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank?
Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump.
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Bailey Allison
-
Dan van der Ster
-
Eugen Block
-
Frank Schilder
-
Frédéric Nass
-
Patrick Donnelly
-
Spencer Macphee