MDS: cache pressure warnings with Ganesha exports
Hey folks, I keep getting ceph health warnings about clients failing to respond to cache pressure. They always refer to sessions from ganesha exports. I've read all threads regarding this issue, but none of my changes resolved it. What I’ve done so far: Ganesha.conf: MDCACHE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; } Attr_Expiration_Time = 0 in every export mds_cache_memory_limit = 17179869184 on MDS Servers I even set “client_oc = false” on the ganesha server, but this doesn’t seem to be applied My setup is ceph version 14.2.8 (All Server and clients. One active MDS. Ganesha 2.8.3 runs on a dedicated Server. ceph daemon mds.<active_mds> dump_mempools (filtered out the empty pools): "mempool": { "by_pool": { "bloom_filter": { "items": 1414723, "bytes": 1414723 }, "buffer_anon": { "items": 180992, "bytes": 2404306271 }, "buffer_meta": { "items": 178660, "bytes": 15722080 }, "osdmap": { "items": 4121, "bytes": 75912 }, "mds_co": { "items": 221728924, "bytes": 16320868177 }, }, "total": { "items": 223507420, "bytes": 18742387163 Every hint how to resolve the issue are welcome. If more information is needed, I am glad to provide it. Regards Felix ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------
Hi Felix, we've had cache pressure messages for a long time in our small production cluster without seeing any negative impact to clients (or the cluster). We don't use Ganesha but we export CephFS directories via NFS. I guess NFS is the common denominator here. In our case we started with a very low mds_cache_memory_limit value (around 2 or 4 GB, we also started with only a few clients) and increased it step by step, currently it is around 8 GB. Cache status currently shows ~ 7.5 GB while top output shows 10.5 GB usage. That seems to be enough for around 70 connected clients, although not all of them are constantly active. So I would suggest to increase memory in small steps to see if those warnings go away. Regards, Eugen Zitat von "Stolte, Felix" <f.stolte@fz-juelich.de>:
Hey folks,
I keep getting ceph health warnings about clients failing to respond to cache pressure. They always refer to sessions from ganesha exports. I've read all threads regarding this issue, but none of my changes resolved it. What I’ve done so far:
Ganesha.conf: MDCACHE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; }
Attr_Expiration_Time = 0 in every export
mds_cache_memory_limit = 17179869184 on MDS Servers
I even set “client_oc = false” on the ganesha server, but this doesn’t seem to be applied
My setup is ceph version 14.2.8 (All Server and clients. One active MDS. Ganesha 2.8.3 runs on a dedicated Server.
ceph daemon mds.<active_mds> dump_mempools (filtered out the empty pools):
"mempool": { "by_pool": { "bloom_filter": { "items": 1414723, "bytes": 1414723 }, "buffer_anon": { "items": 180992, "bytes": 2404306271 }, "buffer_meta": { "items": 178660, "bytes": 15722080 }, "osdmap": { "items": 4121, "bytes": 75912 }, "mds_co": { "items": 221728924, "bytes": 16320868177 }, }, "total": { "items": 223507420, "bytes": 18742387163
Every hint how to resolve the issue are welcome. If more information is needed, I am glad to provide it.
Regards Felix
------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, 2020-04-07 at 07:34 +0000, Stolte, Felix wrote:
Hey folks,
I keep getting ceph health warnings about clients failing to respond to cache pressure. They always refer to sessions from ganesha exports. I've read all threads regarding this issue, but none of my changes resolved it. What I’ve done so far:
Ganesha.conf: MDCACHE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; }
Attr_Expiration_Time = 0 in every export
mds_cache_memory_limit = 17179869184 on MDS Servers
I even set “client_oc = false” on the ganesha server, but this doesn’t seem to be applied
My setup is ceph version 14.2.8 (All Server and clients. One active MDS. Ganesha 2.8.3 runs on a dedicated Server.
ceph daemon mds.<active_mds> dump_mempools (filtered out the empty pools):
"mempool": { "by_pool": { "bloom_filter": { "items": 1414723, "bytes": 1414723 }, "buffer_anon": { "items": 180992, "bytes": 2404306271 }, "buffer_meta": { "items": 178660, "bytes": 15722080 }, "osdmap": { "items": 4121, "bytes": 75912 }, "mds_co": { "items": 221728924, "bytes": 16320868177 }, }, "total": { "items": 223507420, "bytes": 18742387163
Every hint how to resolve the issue are welcome. If more information is needed, I am glad to provide it.
You may want to follow the suggestion here and see if it helps: https://tracker.ceph.com/issues/44976#note-23 -- Jeff Layton <jlayton@redhat.com>
Hi Jeff, thank you for the hint. I set Entries_HWMark = 100 in MDCACHE Section of ganesha.conf and upgraded ganesha to 3.2 this weekend. Cache Pressure warnings still keep accuring, but not as frequent as before. Is there another suggestion I did miss? Regards Felix ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Am 09.04.20, 14:10 schrieb "Jeff Layton" <jlayton@redhat.com>: On Tue, 2020-04-07 at 07:34 +0000, Stolte, Felix wrote: > Hey folks, > > I keep getting ceph health warnings about clients failing to respond to cache pressure. They always refer to sessions from ganesha exports. I've read all threads regarding this issue, but none of my changes resolved it. What I’ve done so far: > > Ganesha.conf: > MDCACHE { > Dir_Chunk = 0; > NParts = 1; > Cache_Size = 1; > } > > Attr_Expiration_Time = 0 in every export > > mds_cache_memory_limit = 17179869184 on MDS Servers > > I even set “client_oc = false” on the ganesha server, but this doesn’t seem to be applied > > My setup is ceph version 14.2.8 (All Server and clients. One active MDS. Ganesha 2.8.3 runs on a dedicated Server. > > ceph daemon mds.<active_mds> dump_mempools (filtered out the empty pools): > > "mempool": { > "by_pool": { > "bloom_filter": { > "items": 1414723, > "bytes": 1414723 > }, > "buffer_anon": { > "items": 180992, > "bytes": 2404306271 > }, > "buffer_meta": { > "items": 178660, > "bytes": 15722080 > }, > "osdmap": { > "items": 4121, > "bytes": 75912 > }, > "mds_co": { > "items": 221728924, > "bytes": 16320868177 > }, > }, > "total": { > "items": 223507420, > "bytes": 18742387163 > > > Every hint how to resolve the issue are welcome. If more information is needed, I am glad to provide it. > You may want to follow the suggestion here and see if it helps: https://tracker.ceph.com/issues/44976#note-23 -- Jeff Layton <jlayton@redhat.com>
On Tue, 2020-04-14 at 06:27 +0000, Stolte, Felix wrote:
Hi Jeff,
thank you for the hint. I set Entries_HWMark = 100 in MDCACHE Section of ganesha.conf and upgraded ganesha to 3.2 this weekend. Cache Pressure warnings still keep accuring, but not as frequent as before. Is there another suggestion I did miss?
Great. If it doesn't help, then it would be good to see the "ganesha_stats inode" output. That script is part of the nfs-ganesha-utils package. The output from my (idle) server looks like this: ----------------8<------------------- $ ./ganesha_stats.py inode Timestamp: Tue Apr 14 15:24:18 2020795806304 nsecs Inode Cache statistics Cache Requests: 0 Cache Hits: 2412 Cache Misses: 0 Cache Conflicts: 0 Cache Adds: 193 Cache Mapping: 386 LRU Utilization Data FSAL opened FD count : 0 System limit on FDs : 524288 FD usage : Below Low Water Mark LRU entries in use : 12 Chunks in use : 0 ----------------8<------------------- That should tell us something about the cached inodes that ganesha is holding onto. -- Jeff Layton <jlayton@redhat.com>
Hi Jeff, Output of ganesha_stats inode: Timestamp: Wed Apr 15 09:37:32 2020522216985 nsecs Inode Cache Requests: 0 Inode Cache Hits: 361976 Inode Cache Misses: 0 Inode Cache Conflicts:: 2 Inode Cache Adds: 307597 Inode Cache Mapping: 615198 Best regards Felix ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Am 14.04.20, 21:26 schrieb "Jeff Layton" <jlayton@redhat.com>: On Tue, 2020-04-14 at 06:27 +0000, Stolte, Felix wrote: > Hi Jeff, > > thank you for the hint. I set Entries_HWMark = 100 in MDCACHE Section > of ganesha.conf and upgraded ganesha to 3.2 this weekend. Cache > Pressure warnings still keep accuring, but not as frequent as before. > Is there another suggestion I did miss? > Great. If it doesn't help, then it would be good to see the "ganesha_stats inode" output. That script is part of the nfs-ganesha-utils package. The output from my (idle) server looks like this: ----------------8<------------------- $ ./ganesha_stats.py inode Timestamp: Tue Apr 14 15:24:18 2020795806304 nsecs Inode Cache statistics Cache Requests: 0 Cache Hits: 2412 Cache Misses: 0 Cache Conflicts: 0 Cache Adds: 193 Cache Mapping: 386 LRU Utilization Data FSAL opened FD count : 0 System limit on FDs : 524288 FD usage : Below Low Water Mark LRU entries in use : 12 Chunks in use : 0 ----------------8<------------------- That should tell us something about the cached inodes that ganesha is holding onto. -- Jeff Layton <jlayton@redhat.com>
On Wed, 2020-04-15 at 12:06 +0000, Stolte, Felix wrote:
Hi Jeff,
Output of ganesha_stats inode:
Timestamp: Wed Apr 15 09:37:32 2020522216985 nsecs Inode Cache Requests: 0 Inode Cache Hits: 361976 Inode Cache Misses: 0 Inode Cache Conflicts:: 2 Inode Cache Adds: 307597 Inode Cache Mapping: 615198
LRU Utilization part is missing. Ahh ok...the interesting bit (the LRU Utilization part) was only added very recently (Feb 2020), so that won't be as helpful as I had hoped on older ganesha versions. One thing you could do is hook up the debugger to get the interesting part. Install the nfs-ganesha package debuginfo and: $ gdb /path/to/ganesha.nfsd gdb> p lru_state ...and collect what it prints. That'll stall the server for a bit while you're poking around in there though The entries_used field is what we're most interested in. Each entry holds a reference to a libcephfs Inode object, and that consumes resources on the MDS. We want to keep that value pretty low as a matter of course. With Entries_HWMark = 100, it should be below 100 most of the time. It can and does go above that though, so it'd be interesting to see what that looks like, particularly when the MDS is complaining about client resource utilization. -- Jeff Layton <jlayton@redhat.com>
Hi Jeff, my ganesha instances are running on ubuntu 18.04 with packages from http://ppa.launchpad.net/nfs-ganesha/nfs-ganesha-3.0/ubuntu Unfortunately they do not provide the debug package, so I cannot poke around due to the missing debug symbols. Do you have another approach to get the information needed? I cloned https://github.com/nfs-ganesha/nfs-ganesha and tried to use the ganesha_stats.py script from there, but with the inode parameter I get: ' Error: Can't talk to ganesha service on d-bus. Looks like Ganesha is down'. All other parameters are working (list_clients, etc.) The cache pressure issue seems to be related to our tape backup we do over a kernel mount. During backup mds cache increases by ~50 Million items per hour. While the kernel mount seems to handle cache pressure in time, ganesha does not. Cache pressure warnings are always for sessions from ganesha. To work around the issue I was wondering if its possible to mount the ceph filesystem without object caching on mds side. Backup does not really benefit from it, because every file (or it's metadata) is only read once. Regards Felix PS: Sorry for the previous mail, forgot to disable signing ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Am 15.04.20, 14:57 schrieb "Jeff Layton" <jlayton@redhat.com>: On Wed, 2020-04-15 at 12:06 +0000, Stolte, Felix wrote: > Hi Jeff, > > Output of ganesha_stats inode: > > Timestamp: Wed Apr 15 09:37:32 2020522216985 nsecs > Inode Cache Requests: 0 > Inode Cache Hits: 361976 > Inode Cache Misses: 0 > Inode Cache Conflicts:: 2 > Inode Cache Adds: 307597 > Inode Cache Mapping: 615198 > LRU Utilization part is missing. Ahh ok...the interesting bit (the LRU Utilization part) was only added very recently (Feb 2020), so that won't be as helpful as I had hoped on older ganesha versions. One thing you could do is hook up the debugger to get the interesting part. Install the nfs-ganesha package debuginfo and: $ gdb /path/to/ganesha.nfsd gdb> p lru_state ...and collect what it prints. That'll stall the server for a bit while you're poking around in there though The entries_used field is what we're most interested in. Each entry holds a reference to a libcephfs Inode object, and that consumes resources on the MDS. We want to keep that value pretty low as a matter of course. With Entries_HWMark = 100, it should be below 100 most of the time. It can and does go above that though, so it'd be interesting to see what that looks like, particularly when the MDS is complaining about client resource utilization. -- Jeff Layton <jlayton@redhat.com>
participants (3)
-
Eugen Block
-
Jeff Layton
-
Stolte, Felix