There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not the usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services? Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E... quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep the free cache size in the desired range. I need to make it aggressive but this should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?