Hi All, We have a slurm cluster with 25 clients, each with 256 cores, each mounting a cephfs filesystem as their main storage target. The workload can be heavy at times. We have two active MDS daemons and one standby. A lot of the time everything is healthy but we sometimes get warnings about MDS daemons being slow on requests, behind on trimming, etc. I realize their may be a bug in play, but also, I was wondering if we simply didn't have enough MDS daemons to handle the load. Is there a way to know if adding a MDS daemon would help? We could add a third active MDS if needed. But I don't want to start adding a bunch of MDS's if that won't help. The OSD servers seem fine. It's mainly the MDS instances that are complaining. We are running reef 18.2.1. For reference, when things look healthy: # ceph fs status slugfs slugfs - 34 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active slugfs.pr-md-03.mclckv Reqs: 273 /s 2759k 2636k 362k 1079k 1 active slugfs.pr-md-01.xdtppo Reqs: 194 /s 868k 674k 67.3k 351k POOL TYPE USED AVAIL cephfs_metadata metadata 127G 3281G cephfs_md_and_data data 0 98.3T cephfs_data data 740T 196T STANDBY MDS slugfs.pr-md-02.sbblqq MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_OK services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-01.jemmdf(active, since 5w), standbys: pr-md-02.emffhz mds: 2/2 daemons up, 1 standby osd: 46 osds: 46 up (since 8d), 46 in (since 4w) data: volumes: 1/1 healthy pools: 4 pools, 1313 pgs objects: 271.17M objects, 493 TiB usage: 744 TiB used, 384 TiB / 1.1 PiB avail pgs: 1307 active+clean 4 active+clean+scrubbing 2 active+clean+scrubbing+deep io: client: 39 MiB/s rd, 108 MiB/s wr, 1.96k op/s rd, 54 op/s wr But when things are in "warning" mode, it looks like this: # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded 1 clients failing to advance oldest client/flush tid 1 MDSs report slow requests 1 MDSs behind on trimming services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-01.jemmdf(active, since 5w), standbys: pr-md-02.emffhz mds: 2/2 daemons up, 1 standby osd: 46 osds: 46 up (since 8d), 46 in (since 4w) data: volumes: 1/1 healthy pools: 4 pools, 1313 pgs objects: 271.28M objects, 494 TiB usage: 746 TiB used, 382 TiB / 1.1 PiB avail pgs: 1307 active+clean 5 active+clean+scrubbing 1 active+clean+scrubbing+deep io: client: 55 MiB/s rd, 2.6 MiB/s wr, 15 op/s rd, 46 op/s wr And this: # ceph health detail HEALTH_WARN 2 clients failing to advance oldest client/flush tid; 2 MDSs report slow requests; 1 MDSs behind on trimming [WRN] MDS_CLIENT_OLDEST_TID: 2 clients failing to advance oldest client/flush tid mds.slugfs.pr-md-01.xdtppo(mds.0): Client phoenix-06.prism failing to advance its oldest client/flush tid. client_id: 125780 mds.slugfs.pr-md-02.sbblqq(mds.1): Client phoenix-00.prism failing to advance its oldest client/flush tid. client_id: 99385 [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests mds.slugfs.pr-md-01.xdtppo(mds.0): 4 slow requests are blocked > 30 secs mds.slugfs.pr-md-02.sbblqq(mds.1): 67 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.slugfs.pr-md-02.sbblqq(mds.1): Behind on trimming (109410/250) max_segments: 250, num_segments: 109410 The "cure" is the restart the active MDS daemons, one at a time. Then everything becomes healthy again, for a time. We also have the following MDS config items in play: mds_cache_memory_limit = 8589934592 mds_cache_trim_decay_rate = .6 mds_log_max_segments = 250 Thanks for any pointers! cheers, erich