So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure". Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time? I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...? We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping... -erich
Hi Erich, in our case the "client failing to respond to cache pressure" situation is/was often caused by users how have vscode connecting via ssh to our HPC head node. vscode makes heavy use of file watchers and we have seen users with > 400k watchers. All these watched files must be held in the MDS cache and if you have multiple users at the same time running vscode it gets problematic. Unfortunately there is no global setting - at least none that we are aware of - for vscode to exclude certain files or directories from being watched. We asked the users to configure their vscode (Remote Settings -> Watcher Exclude) as follows: { "files.watcherExclude": { "**/.git/objects/**": true, "**/.git/subtree-cache/**": true, "**/node_modules/*/**": true, "**/.cache/**": true, "**/.conda/**": true, "**/.local/**": true, "**/.nextflow/**": true, "**/work/**": true } } ~/.vscode-server/data/Machine/settings.json To monitor and find processes with watcher you may use inotify-info <https://github.com/mikesart/inotify-info> HTH Dietmar On 4/23/24 15:47, Erich Weiler wrote:
So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure".
Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time?
I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...?
We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping...
-erich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dietmar, We do in fact have a bunch of users running vscode on our HPC head node as well (in addition to a few of our general purpose interactive compute servers). I'll suggest they make the mods you referenced! Thanks for the tip. cheers, erich On 4/24/24 12:58 PM, Dietmar Rieder wrote:
Hi Erich,
in our case the "client failing to respond to cache pressure" situation is/was often caused by users how have vscode connecting via ssh to our HPC head node. vscode makes heavy use of file watchers and we have seen users with > 400k watchers. All these watched files must be held in the MDS cache and if you have multiple users at the same time running vscode it gets problematic.
Unfortunately there is no global setting - at least none that we are aware of - for vscode to exclude certain files or directories from being watched. We asked the users to configure their vscode (Remote Settings -> Watcher Exclude) as follows:
{ "files.watcherExclude": { "**/.git/objects/**": true, "**/.git/subtree-cache/**": true, "**/node_modules/*/**": true, "**/.cache/**": true, "**/.conda/**": true, "**/.local/**": true, "**/.nextflow/**": true, "**/work/**": true } }
~/.vscode-server/data/Machine/settings.json
To monitor and find processes with watcher you may use inotify-info <https://github.com/mikesart/inotify-info>
HTH Dietmar
On 4/23/24 15:47, Erich Weiler wrote:
So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure".
Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time?
I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...?
We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping...
-erich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Erich, hope it helps. Let us know. Dietmar Am 26. April 2024 15:52:06 MESZ schrieb Erich Weiler <weiler@soe.ucsc.edu>:
Hi Dietmar,
We do in fact have a bunch of users running vscode on our HPC head node as well (in addition to a few of our general purpose interactive compute servers). I'll suggest they make the mods you referenced! Thanks for the tip.
cheers, erich
On 4/24/24 12:58 PM, Dietmar Rieder wrote:
Hi Erich,
in our case the "client failing to respond to cache pressure" situation is/was often caused by users how have vscode connecting via ssh to our HPC head node. vscode makes heavy use of file watchers and we have seen users with > 400k watchers. All these watched files must be held in the MDS cache and if you have multiple users at the same time running vscode it gets problematic.
Unfortunately there is no global setting - at least none that we are aware of - for vscode to exclude certain files or directories from being watched. We asked the users to configure their vscode (Remote Settings -> Watcher Exclude) as follows:
{ "files.watcherExclude": { "**/.git/objects/**": true, "**/.git/subtree-cache/**": true, "**/node_modules/*/**": true, "**/.cache/**": true, "**/.conda/**": true, "**/.local/**": true, "**/.nextflow/**": true, "**/work/**": true } }
~/.vscode-server/data/Machine/settings.json
To monitor and find processes with watcher you may use inotify-info <https://github.com/mikesart/inotify-info>
HTH Dietmar
On 4/23/24 15:47, Erich Weiler wrote:
So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure".
Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time?
I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...?
We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping...
-erich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Actually should I be excluding my whole cephfs filesystem? Like, if I mount it as /cephfs, should my stanza looks something like: { "files.watcherExclude": { "**/.git/objects/**": true, "**/.git/subtree-cache/**": true, "**/node_modules/*/**": true, "**/.cache/**": true, "**/.conda/**": true, "**/.local/**": true, "**/.nextflow/**": true, "**/work/**": true, "**/cephfs/**": true } } On 4/27/24 12:24 AM, Dietmar Rieder wrote:
Hi Erich,
hope it helps. Let us know.
Dietmar
Am 26. April 2024 15:52:06 MESZ schrieb Erich Weiler <weiler@soe.ucsc.edu>:
Hi Dietmar,
We do in fact have a bunch of users running vscode on our HPC head node as well (in addition to a few of our general purpose interactive compute servers). I'll suggest they make the mods you referenced! Thanks for the tip.
cheers, erich
On 4/24/24 12:58 PM, Dietmar Rieder wrote:
Hi Erich,
in our case the "client failing to respond to cache pressure" situation is/was often caused by users how have vscode connecting via ssh to our HPC head node. vscode makes heavy use of file watchers and we have seen users with > 400k watchers. All these watched files must be held in the MDS cache and if you have multiple users at the same time running vscode it gets problematic.
Unfortunately there is no global setting - at least none that we are aware of - for vscode to exclude certain files or directories from being watched. We asked the users to configure their vscode (Remote Settings -> Watcher Exclude) as follows:
{ "files.watcherExclude": { "**/.git/objects/**": true, "**/.git/subtree-cache/**": true, "**/node_modules/*/**": true, "**/.cache/**": true, "**/.conda/**": true, "**/.local/**": true, "**/.nextflow/**": true, "**/work/**": true } }
~/.vscode-server/data/Machine/settings.json
To monitor and find processes with watcher you may use inotify-info <https://github.com/mikesart/inotify-info <https://github.com/mikesart/inotify-info>>
HTH Dietmar
On 4/23/24 15:47, Erich Weiler wrote:
So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure".
Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time?
I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...?
We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping...
-erich ------------------------------------------------------------------------ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
------------------------------------------------------------------------ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
there might be situations/dicrectories for which it makes a lot of sense that they are watched for changes and vscode is informed about these. So excluding the entire cephfs might not always be a good idea. But I guess you need to find out. Dietmar On 4/27/24 16:38, Erich Weiler wrote:
Actually should I be excluding my whole cephfs filesystem? Like, if I mount it as /cephfs, should my stanza looks something like:
{ "files.watcherExclude": { "**/.git/objects/**": true, "**/.git/subtree-cache/**": true, "**/node_modules/*/**": true, "**/.cache/**": true, "**/.conda/**": true, "**/.local/**": true, "**/.nextflow/**": true, "**/work/**": true, "**/cephfs/**": true } }
On 4/27/24 12:24 AM, Dietmar Rieder wrote:
Hi Erich,
hope it helps. Let us know.
Dietmar
Am 26. April 2024 15:52:06 MESZ schrieb Erich Weiler <weiler@soe.ucsc.edu>:
Hi Dietmar,
We do in fact have a bunch of users running vscode on our HPC head node as well (in addition to a few of our general purpose interactive compute servers). I'll suggest they make the mods you referenced! Thanks for the tip.
cheers, erich
On 4/24/24 12:58 PM, Dietmar Rieder wrote:
Hi Erich,
in our case the "client failing to respond to cache pressure" situation is/was often caused by users how have vscode connecting via ssh to our HPC head node. vscode makes heavy use of file watchers and we have seen users with > 400k watchers. All these watched files must be held in the MDS cache and if you have multiple users at the same time running vscode it gets problematic.
Unfortunately there is no global setting - at least none that we are aware of - for vscode to exclude certain files or directories from being watched. We asked the users to configure their vscode (Remote Settings -> Watcher Exclude) as follows:
{ "files.watcherExclude": { "**/.git/objects/**": true, "**/.git/subtree-cache/**": true, "**/node_modules/*/**": true, "**/.cache/**": true, "**/.conda/**": true, "**/.local/**": true, "**/.nextflow/**": true, "**/work/**": true } }
~/.vscode-server/data/Machine/settings.json
To monitor and find processes with watcher you may use inotify-info <https://github.com/mikesart/inotify-info <https://github.com/mikesart/inotify-info>>
HTH Dietmar
On 4/23/24 15:47, Erich Weiler wrote:
So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure".
Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time?
I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...?
We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping...
-erich
------------------------------------------------------------------------ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
------------------------------------------------------------------------ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Erich, Erich Weiler schreef op 2024-04-23 15:47:
So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure".
Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time?
I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...?
We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping...
We are seeing "client failing to respond to cache pressure" on a daily basis. Remounting on the client usually 'fixes' the issue. Sometimes, remounting on all clients that have the same directory mounted is needed. Also, a larger MDS cache seems to help. As Dietmar said, VS Code may cause this. Quite funny to read, actually, because we've been dealing with this issue for over a year, and yesterday was the very first time Ceph complained about a client and we saw VS Code's remote stuff running. Coincidence.
-erich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Met vriendelijke groeten, William Edwards
As Dietmar said, VS Code may cause this. Quite funny to read, actually, because we've been dealing with this issue for over a year, and yesterday was the very first time Ceph complained about a client and we saw VS Code's remote stuff running. Coincidence.
I'm holding my breath that the vscode issue is the one affecting us - I got my users to tweak their vscode configs and the problem seemed to go away, but I guess I won't consider it 'solved' until a few days pass without it coming back... :)
On 4/26/24 23:51, Erich Weiler wrote:
As Dietmar said, VS Code may cause this. Quite funny to read, actually, because we've been dealing with this issue for over a year, and yesterday was the very first time Ceph complained about a client and we saw VS Code's remote stuff running. Coincidence.
I'm holding my breath that the vscode issue is the one affecting us - I got my users to tweak their vscode configs and the problem seemed to go away, but I guess I won't consider it 'solved' until a few days pass without it coming back... :)
I wonder if the vscode configs solved your issues, or if you still see the cache pressure messages? Dietmar
I still saw client cache pressure messages, although I think it did in general help a bit. What I additionally just did (like 5 minutes ago), was reduce "mds_recall_max_caps" from 30,000 to 10,000 after looking at this post: https://www.spinics.net/lists/ceph-users/msg73188.html And will try further reducing mds_recall_max_caps if the pressure messages keep coming up. After reducing it to 10,000 a few client cache pressure warnings cleared but I don't know yet if that was the reason it cleared or if it was just luck. If I see it stay clear then I'll call it solved. -erich On 5/7/24 6:55 AM, Dietmar Rieder wrote:
On 4/26/24 23:51, Erich Weiler wrote:
As Dietmar said, VS Code may cause this. Quite funny to read, actually, because we've been dealing with this issue for over a year, and yesterday was the very first time Ceph complained about a client and we saw VS Code's remote stuff running. Coincidence.
I'm holding my breath that the vscode issue is the one affecting us - I got my users to tweak their vscode configs and the problem seemed to go away, but I guess I won't consider it 'solved' until a few days pass without it coming back... :)
I wonder if the vscode configs solved your issues, or if you still see the cache pressure messages?
Dietmar
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Dietmar Rieder
-
Erich Weiler
-
William Edwards