Hi Kasper, that's exactly what we usually do if we have identified some misbehavior, trying to find the right setting to mitigate the issue. If you see cache pressure messages, it might be more helpful to rather decrease mds_recall_max_caps (default: 30000) than to increase it (your setting is 33000). Mykola once helped explaining that [0], maybe this could help here as well. I can't recall having tweaked mds_max_caps_per_client myself yet, but yeah, I would try to make sense of the settings and the observed behavior. Regards, Eugen [0] https://tracker.ceph.com/issues/57115 Zitat von Kasper Rasmussen <kasper_steengaard@hotmail.com>:
I have a CephFS where workloads use many small files. I see cache pressure / MDS_CLIENT_RECALL warnings ones in awhile (due to clients exceeding mds_max_caps_per_client) and it seems if they linger to long, it ends up with more warnings e.g. MDS_SLOW_REQUESTS, and some directories locks up.
Anyway - Currently I have the mds_max_caps_per_client set to 2M which by looking into the output of
sudo ceph tell mds.<name> counter dump .. .. "counters": { "cap_hits": 8122912454, "cap_miss": 497593, "avg_read_latency": 0.000000028, "avg_write_latency": 0.000000000, "avg_metadata_latency": 0.000000000, "dentry_lease_hits": 5630994071, "dentry_lease_miss": 174816044, "opened_files": 65, "opened_inodes": 2106823, "pinned_icaps": 2106823, "total_inodes": 2106823, "total_read_ops": 309938, "total_read_size": 191662499168, "total_write_ops": 371242, "total_write_size": 414398493835 .. ..
It Is not enough. However there is not a lot of open files
Checking the "ceph_mds_client_metrics_<fs_name>_pinned_icaps" gauge in prometheus tells the same story.. The client is constantly hidden the max caps roof over days (clients have long running jobs)
Can anyone share experience in regards to changing mds_max_caps_per_client to accomodate for such workloads. When changing this, should other config variables be taken into account like -
mds_cache_memory_limit - currently: 36GB mds_cache_trim_decay_rate - currently: 0.9 mds_cache_trim_threshold - currently: 288358 mds_recall_max_caps - currently: 33000 mds_recall_max_decay_rate - currently: 1.35
Or should the be tuned on a observe-and-change-as-needed basis.
Thanks in advance. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io