Hi Kotresh, Thanks for the reply. 1)There are no customer configs defined 2) not enabled subtree pinning 3) there were no warning related to rados So wanted to know In order to fix this should we increase default mds_cache_memory_limit from 4Gb to 6Gb or more? Or is there any other solution for this issue? On Fri, 17 May, 2024, 12:11 pm Kotresh Hiremath Ravishankar, < khiremat@redhat.com> wrote:
Hi,
~6K log segments to be trimmed, that's huge.
1. Are there any custom configs configured on this setup ? 2. Is subtree pinning enabled ? 3. Are there any warnings w.r.t rados slowness ? 4. Please share the mds perf dump to check for latencies and other stuff. $ceph tell mds.<id> perf dump
Thanks and Regards, Kotresh H R
On Fri, May 17, 2024 at 11:01 AM Akash Warkhade <a.warkhade98@gmail.com> wrote:
Hi,
We are using rook-ceph with operator 1.10.8 and ceph 17.2.5. we are using ceph filesystem with 4 mds i.e 2 active & 2 standby MDS every 3-4 weeks filesystem is having issue i.e in ceph status we can see below warnings warnings :
2 MDS reports slow requests 2 MDS Behind on Trimming mds.myfs-a(mds.1) : behind on trimming (6378/128) max_segments:128, num_segments: 6378 mds.myfs-c(mds.1): behind on trimming (6560/128) max_segments:128, num_segments: 6560
to fix it, we have to restart all MDS pods one by one. this is happening every 4-5 weeks.
We have seen many ceph issues related to it on ceph tracker and many people are suggesting to increase mds_cache_memory_limit currently for our cluster *mds_cache_memory_limit* is set to default 4GB *mds_log_max_segments* is set to default 128 Should we increase *mds_cache_memory_limit* to 8GB from default 4GB or is there any solution to fix this issue permanently? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io