stuck MDS warning: Client HOST failing to respond to cache pressure
Hi all, I'm affected by a stuck MDS warning for 2 clients: "failing to respond to cache pressure". This is a false alarm as no MDS is under any cache pressure. The warning is stuck already for a couple of days. I found some old threads about cases where the MDS does not update flags/triggers for this warning in certain situations. Dating back to luminous and I'm probably hitting one of these. In these threads I could find a lot except for instructions for how to clear this out in a nice way. Is there something I can do on the clients to clear this warning? I don't want to evict/reboot just because of that. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
On 17-10-2023 09:22, Frank Schilder wrote:
Hi all,
I'm affected by a stuck MDS warning for 2 clients: "failing to respond to cache pressure". This is a false alarm as no MDS is under any cache pressure. The warning is stuck already for a couple of days. I found some old threads about cases where the MDS does not update flags/triggers for this warning in certain situations. Dating back to luminous and I'm probably hitting one of these.
In these threads I could find a lot except for instructions for how to clear this out in a nice way. Is there something I can do on the clients to clear this warning? I don't want to evict/reboot just because of that.
echo 2 > /proc/sys/vm/drop_caches on the clients .... does that help? Gr. Stefan
Hi Stefan, probably. Its 2 compute nodes and there are jobs running. Our epilogue script will drop the caches, at which point I indeed expect the warning to disappear. We have no time limit on these nodes though, so this can be a while. I was hoping there was an alternative to that, say, a user-level command that I could execute on the client without possibly affecting other users jobs. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Tuesday, October 17, 2023 11:13 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] stuck MDS warning: Client HOST failing to respond to cache pressure On 17-10-2023 09:22, Frank Schilder wrote:
Hi all,
I'm affected by a stuck MDS warning for 2 clients: "failing to respond to cache pressure". This is a false alarm as no MDS is under any cache pressure. The warning is stuck already for a couple of days. I found some old threads about cases where the MDS does not update flags/triggers for this warning in certain situations. Dating back to luminous and I'm probably hitting one of these.
In these threads I could find a lot except for instructions for how to clear this out in a nice way. Is there something I can do on the clients to clear this warning? I don't want to evict/reboot just because of that.
echo 2 > /proc/sys/vm/drop_caches on the clients .... does that help? Gr. Stefan
On 17/10/2023 11:27, Frank Schilder wrote:
Hi Stefan,
probably. Its 2 compute nodes and there are jobs running. Our epilogue script will drop the caches, at which point I indeed expect the warning to disappear. We have no time limit on these nodes though, so this can be a while. I was hoping there was an alternative to that, say, a user-level command that I could execute on the client without possibly affecting other users jobs.
Hello, If you know the names of the files to flush from the cache (from /proc/$PID/fd, lsof, batch job script, ...), you can use something like https://github.com/tortay/cache-toys/blob/master/drop-from-pagecache.c on the client. See comments line 16 to 22 of the source code for caveats/limitations. Loïc. -- | Loīc Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
Hi Loïc, thanks for the pointer. Its kind of the opposite extreme to dropping just everything. I need to know the file name that is in cache. I'm looking for a middle way, say, "drop_caches -u USER" that drops all caches of files owned by user USER. This way I could try dropping caches for a bunch of users who are *not* running a job. I guess I have to wait for the jobs to end. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Loïc Tortay <tortay@cc.in2p3.fr> Sent: Tuesday, October 17, 2023 3:40 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: stuck MDS warning: Client HOST failing to respond to cache pressure On 17/10/2023 11:27, Frank Schilder wrote:
Hi Stefan,
probably. Its 2 compute nodes and there are jobs running. Our epilogue script will drop the caches, at which point I indeed expect the warning to disappear. We have no time limit on these nodes though, so this can be a while. I was hoping there was an alternative to that, say, a user-level command that I could execute on the client without possibly affecting other users jobs.
Hello, If you know the names of the files to flush from the cache (from /proc/$PID/fd, lsof, batch job script, ...), you can use something like https://github.com/tortay/cache-toys/blob/master/drop-from-pagecache.c on the client. See comments line 16 to 22 of the source code for caveats/limitations. Loïc. -- | Loīc Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
On 18/10/2023 10:02, Frank Schilder wrote:
Hi Loïc,
thanks for the pointer. Its kind of the opposite extreme to dropping just everything. I need to know the file name that is in cache. I'm looking for a middle way, say, "drop_caches -u USER" that drops all caches of files owned by user USER. This way I could try dropping caches for a bunch of users who are *not* running a job.
Hello, You can use something like the following to get the list of filenames opened by $USER in $USERDIR (CephFS mountpoint): lsof -au $USER $USERDIR | awk '/REG/{print $NF}' | sort -u Some level of "drop_caches -u $USER" can then be achieved with piping the above commands to "xargs -r $SOMEWHERE/drop-from-pagecache". No need to be "root" if run as $USER. Loïc -- | Loīc Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
Hi Stefan, the jobs ended and the warning disappeared as expected. However, a new job started and the warning showed up again. There is something very strange going on and, maybe, you can help out here: We have a low client CAPS limit configured for performance reasons: # ceph config dump | grep client [...] mds advanced mds_max_caps_per_client 65536 The job in question holds more than that: # ceph tell mds.0 session ls | jq -c '[.[] | {id: .id, h: .client_metadata.hostname, addr: .inst, fs: .client_metadata.root, caps: .num_caps, req: .request_load_avg}]|sort_by(.caps)|.[]' | tail [...] {"id":172249397,"h":"sn272...","addr":"client.172249397 v1:192.168.57.143:0/195146548","fs":"/hpc/home","caps":105417,"req":1442} This CAPS allocation is stable over time, the number doesn't change (I queried multiple times with several minutes interval). My guess is that the MDS message is not about cache pressure but rather about caps trimming. We do have clients that regularly exceed the limit though without MDS warnings. My guess is that these return at least some CAPS on request and are, therefore, not flagged. The client above seems to sit on a fixed set of CAPS that doesn't change and this causes the warning to show up. The strange thing now is that very few files (on ceph fs) are actually open on the client: [USER@sn272 ~]$ lsof -u USER | grep -e /home -e /groups -e /apps | wc -l 170 The kworker thread is at about 3% CPU and should be able to release CAPS. I'm wondering why it doesn't happen though. I also don't believe that 170 open files can allocate 105417 client caps. Questions: - Why does the client have so many caps allocated? Is there another way than open files that requires allocations? - Is there a way to find out what these caps are for? - We will look at the code (its python+miniconda), any pointers what to look for? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Tuesday, October 17, 2023 11:27 AM To: Stefan Kooman; ceph-users@ceph.io Subject: [ceph-users] Re: stuck MDS warning: Client HOST failing to respond to cache pressure Hi Stefan, probably. Its 2 compute nodes and there are jobs running. Our epilogue script will drop the caches, at which point I indeed expect the warning to disappear. We have no time limit on these nodes though, so this can be a while. I was hoping there was an alternative to that, say, a user-level command that I could execute on the client without possibly affecting other users jobs. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Tuesday, October 17, 2023 11:13 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] stuck MDS warning: Client HOST failing to respond to cache pressure On 17-10-2023 09:22, Frank Schilder wrote:
Hi all,
I'm affected by a stuck MDS warning for 2 clients: "failing to respond to cache pressure". This is a false alarm as no MDS is under any cache pressure. The warning is stuck already for a couple of days. I found some old threads about cases where the MDS does not update flags/triggers for this warning in certain situations. Dating back to luminous and I'm probably hitting one of these.
In these threads I could find a lot except for instructions for how to clear this out in a nice way. Is there something I can do on the clients to clear this warning? I don't want to evict/reboot just because of that.
echo 2 > /proc/sys/vm/drop_caches on the clients .... does that help? Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I just tried what sending SIGSTOP and SIGCONT do. After stopping the process 3 caps were returned. After resuming the process these 3 caps were allocated again. There seems to be a large number of stale caps that are not released. While the process was stopped the kworker thread continued to show 2% CPU usage even though there was no file IO going on. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Thursday, October 19, 2023 10:02 AM To: Stefan Kooman; ceph-users@ceph.io Subject: [ceph-users] Re: stuck MDS warning: Client HOST failing to respond to cache pressure Hi Stefan, the jobs ended and the warning disappeared as expected. However, a new job started and the warning showed up again. There is something very strange going on and, maybe, you can help out here: We have a low client CAPS limit configured for performance reasons: # ceph config dump | grep client [...] mds advanced mds_max_caps_per_client 65536 The job in question holds more than that: # ceph tell mds.0 session ls | jq -c '[.[] | {id: .id, h: .client_metadata.hostname, addr: .inst, fs: .client_metadata.root, caps: .num_caps, req: .request_load_avg}]|sort_by(.caps)|.[]' | tail [...] {"id":172249397,"h":"sn272...","addr":"client.172249397 v1:192.168.57.143:0/195146548","fs":"/hpc/home","caps":105417,"req":1442} This CAPS allocation is stable over time, the number doesn't change (I queried multiple times with several minutes interval). My guess is that the MDS message is not about cache pressure but rather about caps trimming. We do have clients that regularly exceed the limit though without MDS warnings. My guess is that these return at least some CAPS on request and are, therefore, not flagged. The client above seems to sit on a fixed set of CAPS that doesn't change and this causes the warning to show up. The strange thing now is that very few files (on ceph fs) are actually open on the client: [USER@sn272 ~]$ lsof -u USER | grep -e /home -e /groups -e /apps | wc -l 170 The kworker thread is at about 3% CPU and should be able to release CAPS. I'm wondering why it doesn't happen though. I also don't believe that 170 open files can allocate 105417 client caps. Questions: - Why does the client have so many caps allocated? Is there another way than open files that requires allocations? - Is there a way to find out what these caps are for? - We will look at the code (its python+miniconda), any pointers what to look for? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Tuesday, October 17, 2023 11:27 AM To: Stefan Kooman; ceph-users@ceph.io Subject: [ceph-users] Re: stuck MDS warning: Client HOST failing to respond to cache pressure Hi Stefan, probably. Its 2 compute nodes and there are jobs running. Our epilogue script will drop the caches, at which point I indeed expect the warning to disappear. We have no time limit on these nodes though, so this can be a while. I was hoping there was an alternative to that, say, a user-level command that I could execute on the client without possibly affecting other users jobs. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Tuesday, October 17, 2023 11:13 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] stuck MDS warning: Client HOST failing to respond to cache pressure On 17-10-2023 09:22, Frank Schilder wrote:
Hi all,
I'm affected by a stuck MDS warning for 2 clients: "failing to respond to cache pressure". This is a false alarm as no MDS is under any cache pressure. The warning is stuck already for a couple of days. I found some old threads about cases where the MDS does not update flags/triggers for this warning in certain situations. Dating back to luminous and I'm probably hitting one of these.
In these threads I could find a lot except for instructions for how to clear this out in a nice way. Is there something I can do on the clients to clear this warning? I don't want to evict/reboot just because of that.
echo 2 > /proc/sys/vm/drop_caches on the clients .... does that help? Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Frank Schilder
-
Loïc Tortay
-
Stefan Kooman