23 Apr
2024
23 Apr
'24
4:47 p.m.
So I'm trying to figure out ways to reduce the number of warnings I'm
getting and I'm thinking about the one "client failing to respond to
cache pressure".
Is there maybe a way to tell a client (or all clients) to reduce the
amount of cache it uses or to release caches quickly? Like, all the time?
I know the linux kernel (and maybe ceph) likes to cache everything for a
while, and rightfully so, but I suspect in my use case it may be more
efficient to more quickly purge the cache or to in general just cache
way less overall...?
We have many thousands of threads all doing different things that are
hitting our filesystem, so I suspect the caching isn't really doing me
much good anyway due to the churn, and probably is causing more problems
than it helping...
-erich
24 Apr
24 Apr
10:58 p.m.
New subject: [EXTERN] cache pressure?
Hi Erich,
in our case the "client failing to respond to cache pressure" situation
is/was often caused by users how have vscode connecting via ssh to our
HPC head node. vscode makes heavy use of file watchers and we have seen
users with > 400k watchers. All these watched files must be held in the
MDS cache and if you have multiple users at the same time running vscode
it gets problematic.
Unfortunately there is no global setting - at least none that we are
aware of - for vscode to exclude certain files or directories from being
watched. We asked the users to configure their vscode (Remote Settings
-> Watcher Exclude) as follows:
{
"files.watcherExclude": {
"**/.git/objects/**": true,
"**/.git/subtree-cache/**": true,
"**/node_modules/*/**": true,
"**/.cache/**": true,
"**/.conda/**": true,
"**/.local/**": true,
"**/.nextflow/**": true,
"**/work/**": true
}
}
~/.vscode-server/data/Machine/settings.json
To monitor and find processes with watcher you may use inotify-info
<https://github.com/mikesart/inotify-info>
HTH
Dietmar
On 4/23/24 15:47, Erich Weiler wrote:
So I'm trying to figure out ways to reduce the
number of warnings I'm
getting and I'm thinking about the one "client failing to respond to
cache pressure".
Is there maybe a way to tell a client (or all clients) to reduce the
amount of cache it uses or to release caches quickly? Like, all the time?
I know the linux kernel (and maybe ceph) likes to cache everything for a
while, and rightfully so, but I suspect in my use case it may be more
efficient to more quickly purge the cache or to in general just cache
way less overall...?
We have many thousands of threads all doing different things that are
hitting our filesystem, so I suspect the caching isn't really doing me
much good anyway due to the churn, and probably is causing more problems
than it helping...
-erich
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
26 Apr
26 Apr
4:52 p.m.
New subject: [EXTERN] cache pressure?
Hi Dietmar,
We do in fact have a bunch of users running vscode on our HPC head node
as well (in addition to a few of our general purpose interactive compute
servers). I'll suggest they make the mods you referenced! Thanks for
the tip.
cheers,
erich
On 4/24/24 12:58 PM, Dietmar Rieder wrote:
> Hi Erich,
>
> in our case the "client failing to respond to cache pressure" situation
> is/was often caused by users how have vscode connecting via ssh to our
> HPC head node. vscode makes heavy use of file watchers and we have seen
> users with > 400k watchers. All these watched files must be held in the
> MDS cache and if you have multiple users at the same time running vscode
> it gets problematic.
>
> Unfortunately there is no global setting - at least none that we are
> aware of - for vscode to exclude certain files or directories from being
> watched. We asked the users to configure their vscode (Remote Settings
> -> Watcher Exclude) as follows:
>
> {
> "files.watcherExclude": {
> "**/.git/objects/**": true,
> "**/.git/subtree-cache/**": true,
> "**/node_modules/*/**": true,
> "**/.cache/**": true,
> "**/.conda/**": true,
> "**/.local/**": true,
> "**/.nextflow/**": true,
> "**/work/**": true
> }
> }
>
> ~/.vscode-server/data/Machine/settings.json
>
> To monitor and find processes with watcher you may use inotify-info
> <https://github.com/mikesart/inotify-info>
>
> HTH
> Dietmar
>
> On 4/23/24 15:47, Erich Weiler wrote:
>> So I'm trying to figure out ways to reduce the number of warnings I'm
>> getting and I'm thinking about the one "client failing to respond to
>> cache pressure".
>>
>> Is there maybe a way to tell a client (or all clients) to reduce the
>> amount of cache it uses or to release caches quickly? Like, all the
>> time?
>>
>> I know the linux kernel (and maybe ceph) likes to cache everything for
>> a while, and rightfully so, but I suspect in my use case it may be
>> more efficient to more quickly purge the cache or to in general just
>> cache way less overall...?
>>
>> We have many thousands of threads all doing different things that are
>> hitting our filesystem, so I suspect the caching isn't really doing me
>> much good anyway due to the churn, and probably is causing more
>> problems than it helping...
>>
>> -erich
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:27 p.m.
Hi Erich,
Erich Weiler schreef op 2024-04-23 15:47:
So I'm trying to figure out ways to reduce the
number of warnings I'm
getting and I'm thinking about the one "client failing to respond to
cache pressure".
Is there maybe a way to tell a client (or all clients) to reduce the
amount of cache it uses or to release caches quickly? Like, all the
time?
I know the linux kernel (and maybe ceph) likes to cache everything for
a while, and rightfully so, but I suspect in my use case it may be more
efficient to more quickly purge the cache or to in general just cache
way less overall...?
We have many thousands of threads all doing different things that are
hitting our filesystem, so I suspect the caching isn't really doing me
much good anyway due to the churn, and probably is causing more
problems than it helping...
We are seeing "client failing to respond to cache pressure" on a daily
basis.
Remounting on the client usually 'fixes' the issue. Sometimes,
remounting on all clients that have the same directory mounted is
needed. Also, a larger MDS cache seems to help.
As Dietmar said, VS Code may cause this. Quite funny to read, actually,
because we've been dealing with this issue for over a year, and
yesterday was the very first time Ceph complained about a client and we
saw VS Code's remote stuff running. Coincidence.
-erich
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Met vriendelijke groeten,
William Edwards
27 Apr
27 Apr
12:51 a.m.
As Dietmar said, VS Code may cause this. Quite funny
to read, actually,
because we've been dealing with this issue for over a year, and
yesterday was the very first time Ceph complained about a client and we
saw VS Code's remote stuff running. Coincidence.
I'm holding my breath that the vscode issue is the one affecting us - I
got my users to tweak their vscode configs and the problem seemed to go
away, but I guess I won't consider it 'solved' until a few days pass
without it coming back... :)
10:24 a.m.
New subject: [EXTERN] cache pressure?
Hi Erich,
hope it helps. Let us know.
Dietmar
Am 26. April 2024 15:52:06 MESZ schrieb Erich Weiler <weiler(a)soe.ucsc.edu>du>:
Hi Dietmar,
We do in fact have a bunch of users running vscode on our HPC head node as well (in
addition to a few of our general purpose interactive compute servers). I'll suggest
they make the mods you referenced! Thanks for the tip.
cheers,
erich
On 4/24/24 12:58 PM, Dietmar Rieder wrote:
> Hi Erich,
>
> in our case the "client failing to respond to cache pressure" situation
is/was often caused by users how have vscode connecting via ssh to our HPC head node.
vscode makes heavy use of file watchers and we have seen users with > 400k watchers.
All these watched files must be held in the MDS cache and if you have multiple users at
the same time running vscode it gets problematic.
>
> Unfortunately there is no global setting - at least none that we are aware of - for
vscode to exclude certain files or directories from being watched. We asked the users to
configure their vscode (Remote Settings -> Watcher Exclude) as follows:
>
> {
> "files.watcherExclude": {
> "**/.git/objects/**": true,
> "**/.git/subtree-cache/**": true,
> "**/node_modules/*/**": true,
> "**/.cache/**": true,
> "**/.conda/**": true,
> "**/.local/**": true,
> "**/.nextflow/**": true,
> "**/work/**": true
> }
> }
>
> ~/.vscode-server/data/Machine/settings.json
>
> To monitor and find processes with watcher you may use inotify-info
> <https://github.com/mikesart/inotify-info>
>
> HTH
> Dietmar
>
> On 4/23/24 15:47, Erich Weiler wrote:
>> So I'm trying to figure out ways to reduce the number of warnings I'm
getting and I'm thinking about the one "client failing to respond to cache
pressure".
>>
>> Is there maybe a way to tell a client (or all clients) to reduce the amount of
cache it uses or to release caches quickly? Like, all the time?
>>
>> I know the linux kernel (and maybe ceph) likes to cache everything for a while,
and rightfully so, but I suspect in my use case it may be more efficient to more quickly
purge the cache or to in general just cache way less overall...?
>>
>> We have many thousands of threads all doing different things that are hitting our
filesystem, so I suspect the caching isn't really doing me much good anyway due to the
churn, and probably is causing more problems than it helping...
>>
>> -erich
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:38 p.m.
New subject: [EXTERN] cache pressure?
Actually should I be excluding my whole cephfs filesystem? Like, if I
mount it as /cephfs, should my stanza looks something like:
{
"files.watcherExclude": {
"**/.git/objects/**": true,
"**/.git/subtree-cache/**": true,
"**/node_modules/*/**": true,
"**/.cache/**": true,
"**/.conda/**": true,
"**/.local/**": true,
"**/.nextflow/**": true,
"**/work/**": true,
"**/cephfs/**": true
}
}
On 4/27/24 12:24 AM, Dietmar Rieder wrote:
> Hi Erich,
>
> hope it helps. Let us know.
>
> Dietmar
>
>
> Am 26. April 2024 15:52:06 MESZ schrieb Erich Weiler <weiler(a)soe.ucsc.edu>du>:
>
> Hi Dietmar,
>
> We do in fact have a bunch of users running vscode on our HPC head
> node as well (in addition to a few of our general purpose
> interactive compute servers). I'll suggest they make the mods you
> referenced! Thanks for the tip.
>
> cheers,
> erich
>
> On 4/24/24 12:58 PM, Dietmar Rieder wrote:
>
> Hi Erich,
>
> in our case the "client failing to respond to cache pressure"
> situation is/was often caused by users how have vscode
> connecting via ssh to our HPC head node. vscode makes heavy use
> of file watchers and we have seen users with > 400k watchers.
> All these watched files must be held in the MDS cache and if you
> have multiple users at the same time running vscode it gets
> problematic.
>
> Unfortunately there is no global setting - at least none that we
> are aware of - for vscode to exclude certain files or
> directories from being watched. We asked the users to configure
> their vscode (Remote Settings -> Watcher Exclude) as follows:
>
> {
> "files.watcherExclude": {
> "**/.git/objects/**": true,
> "**/.git/subtree-cache/**": true,
> "**/node_modules/*/**": true,
> "**/.cache/**": true,
> "**/.conda/**": true,
> "**/.local/**": true,
> "**/.nextflow/**": true,
> "**/work/**": true
> }
> }
>
> ~/.vscode-server/data/Machine/settings.json
>
> To monitor and find processes with watcher you may use inotify-info
> <https://github.com/mikesart/inotify-info
> <https://github.com/mikesart/inotify-info>>
>
> HTH
> Dietmar
>
> On 4/23/24 15:47, Erich Weiler wrote:
>
> So I'm trying to figure out ways to reduce the number of
> warnings I'm getting and I'm thinking about the one "client
> failing to respond to cache pressure".
>
> Is there maybe a way to tell a client (or all clients) to
> reduce the amount of cache it uses or to release caches
> quickly? Like, all the time?
>
> I know the linux kernel (and maybe ceph) likes to cache
> everything for a while, and rightfully so, but I suspect in
> my use case it may be more efficient to more quickly purge
> the cache or to in general just cache way less overall...?
>
> We have many thousands of threads all doing different things
> that are hitting our filesystem, so I suspect the caching
> isn't really doing me much good anyway due to the churn, and
> probably is causing more problems than it helping...
>
> -erich
> ------------------------------------------------------------------------
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> ------------------------------------------------------------------------
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
29 Apr
29 Apr
11:08 a.m.
New subject: [EXTERN] cache pressure?
there might be situations/dicrectories for which it makes a lot of sense
that they are watched for changes and vscode is informed about these. So
excluding the entire cephfs might not always be a good idea. But I guess
you need to find out.
Dietmar
On 4/27/24 16:38, Erich Weiler wrote:
Actually should I be excluding my whole cephfs
filesystem? Like, if I
mount it as /cephfs, should my stanza looks something like:
{
"files.watcherExclude": {
"**/.git/objects/**": true,
"**/.git/subtree-cache/**": true,
"**/node_modules/*/**": true,
"**/.cache/**": true,
"**/.conda/**": true,
"**/.local/**": true,
"**/.nextflow/**": true,
"**/work/**": true,
"**/cephfs/**": true
}
}
On 4/27/24 12:24 AM, Dietmar Rieder wrote:
> Hi Erich,
>
> hope it helps. Let us know.
>
> Dietmar
>
>
> Am 26. April 2024 15:52:06 MESZ schrieb Erich Weiler
> <weiler(a)soe.ucsc.edu>du>:
>
> Hi Dietmar,
>
> We do in fact have a bunch of users running vscode on our HPC head
> node as well (in addition to a few of our general purpose
> interactive compute servers). I'll suggest they make the mods you
> referenced! Thanks for the tip.
>
> cheers,
> erich
>
> On 4/24/24 12:58 PM, Dietmar Rieder wrote:
>
> Hi Erich,
>
> in our case the "client failing to respond to cache pressure"
> situation is/was often caused by users how have vscode
> connecting via ssh to our HPC head node. vscode makes heavy use
> of file watchers and we have seen users with > 400k watchers.
> All these watched files must be held in the MDS cache and if you
> have multiple users at the same time running vscode it gets
> problematic.
>
> Unfortunately there is no global setting - at least none that we
> are aware of - for vscode to exclude certain files or
> directories from being watched. We asked the users to configure
> their vscode (Remote Settings -> Watcher Exclude) as follows:
>
> {
> "files.watcherExclude": {
> "**/.git/objects/**": true,
> "**/.git/subtree-cache/**": true,
> "**/node_modules/*/**": true,
> "**/.cache/**": true,
> "**/.conda/**": true,
> "**/.local/**": true,
> "**/.nextflow/**": true,
> "**/work/**": true
> }
> }
>
> ~/.vscode-server/data/Machine/settings.json
>
> To monitor and find processes with watcher you may use
> inotify-info
> <https://github.com/mikesart/inotify-info
> <https://github.com/mikesart/inotify-info>>
>
> HTH
> Dietmar
>
> On 4/23/24 15:47, Erich Weiler wrote:
>
> So I'm trying to figure out ways to reduce the number of
> warnings I'm getting and I'm thinking about the one
"client
> failing to respond to cache pressure".
>
> Is there maybe a way to tell a client (or all clients) to
> reduce the amount of cache it uses or to release caches
> quickly? Like, all the time?
>
> I know the linux kernel (and maybe ceph) likes to cache
> everything for a while, and rightfully so, but I suspect in
> my use case it may be more efficient to more quickly purge
> the cache or to in general just cache way less overall...?
>
> We have many thousands of threads all doing different things
> that are hitting our filesystem, so I suspect the caching
> isn't really doing me much good anyway due to the churn, and
> probably is causing more problems than it helping...
>
> -erich
>
> ------------------------------------------------------------------------
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> ------------------------------------------------------------------------
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
7 May
7 May
4:55 p.m.
New subject: [EXTERN] Re: cache pressure?
On 4/26/24 23:51, Erich Weiler wrote:
I wonder if the vscode configs solved your issues, or if you still see
the cache pressure messages?
Dietmar
As Dietmar
said, VS Code may cause this. Quite funny to read,
actually, because we've been dealing with this issue for over a year,
and yesterday was the very first time Ceph complained about a client
and we saw VS Code's remote stuff running. Coincidence.
I'm holding my breath that the vscode issue is the one affecting us - I
got my users to tweak their vscode configs and the problem seemed to go
away, but I guess I won't consider it 'solved' until a few days pass
without it coming back... :)
5:03 p.m.
New subject: [EXTERN] Re: cache pressure?
I still saw client cache pressure messages, although I think it did in
general help a bit. What I additionally just did (like 5 minutes ago),
was reduce "mds_recall_max_caps" from 30,000 to 10,000 after looking at
this post:
https://www.spinics.net/lists/ceph-users/msg73188.html
And will try further reducing mds_recall_max_caps if the pressure
messages keep coming up. After reducing it to 10,000 a few client cache
pressure warnings cleared but I don't know yet if that was the reason it
cleared or if it was just luck. If I see it stay clear then I'll call
it solved.
-erich
On 5/7/24 6:55 AM, Dietmar Rieder wrote:
> On 4/26/24 23:51, Erich Weiler wrote:
>>> As Dietmar said, VS Code may cause this. Quite funny to read,
>>> actually, because we've been dealing with this issue for over a year,
>>> and yesterday was the very first time Ceph complained about a client
>>> and we saw VS Code's remote stuff running. Coincidence.
>>
>> I'm holding my breath that the vscode issue is the one affecting us -
>> I got my users to tweak their vscode configs and the problem seemed to
>> go away, but I guess I won't consider it 'solved' until a few days
>> pass without it coming back... :)
>
> I wonder if the vscode configs solved your issues, or if you still see
> the cache pressure messages?
>
> Dietmar
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
13
days inactive
27
days old
9 comments
3 participants
participants (3)
-
Dietmar Rieder -
Erich Weiler -
William Edwards