Hi Tim, We've been seeing something that may be similar to you with concurrent MDS_CLIENT_LATE_RELEASE and MDS_SLOW_REQUEST warning messages as well as frequently MDS_CLIENT_RECALL and MDS_SLOW_METADATA_IO warnings from the same MDS referring to the same client. We are using 1 MDS for our non-containerised filesystem on a 5.7 PiB sized Alma8.8 cluster with 29 nodes and 348 spinning OSDs for bulk data and 4 OSDs for the metadata on NVMe SSDs with 174 million files (we also have additional NVMe drive partitions as DB and WAL devices for each OSD). All of our clients are using the kernel mount where we also have a SMB gateway which kernel mounts the filesystem and shares it to Windows and Mac machines. This problem seems to have various symptoms including but not limited to: (i) a particular file or directory hanging on openfs, read, and or statx system calls for all clients mounting the filesystem, (ii) all our connected clients hanging on the aforementioned system calls when performing any metadata or bulk data I/O on the filesystem, (iii) 1 client being unable to stat or cd into the filesystem whilst all other clients are completely unaffected, and (iv) possibly a near complete loss of metadata due to the MDS journal swelling beyond the size of the MDS pool where we eventually had some success following the recovery docs (https://docs.ceph.com/en/quincy/cephfs/disaster-recovery-experts/) but opted for a backup restore instead (we were able to restore the bulk data to a read-only state but not all the metadata where we would be more than happy to write a "worked example" of this if the community would find it useful). We use to solve this by forcefully re-mounting the filesystem on the client but as we run multi-user systems and have 100s of clients, this was not a sustainable option for us. We notice generally 3 classes of error: * The oldest slow or blocked request in the ceph-mds log is waiting to acquire locks for an inode (visible via `ceph tell mds.N dump_blocked_ops`). By tracing the oldest FS client which has the capability for this file and seeing which OSD the client is waiting for with `watch 'cat /sys/kernel/debug/ceph/*/osdc'` or by looking at the output of `ceph tell mds.N objecter requests`, we can determine which exact OSD is being slow (always a spinning disk). These issues are due to hardware issues or the OSD crashing (where a restart seems to remedy the problem) and thus counterintuitively (at least for me) suggests that our error messages for the MDS can refer to OSD hard-drive problems. These HDDs are in a completely separate pool to where I would have thought the metadata I/O would be being performed since our metadata pool is exclusively SSDs. * The oldest, or near oldest, blocked MDS op being an "internal op fragmentdir" operation and many client_requests at the flag_point "failed to authpin, dir is being fragmented". This seems to be an MDS bug as the documentation suggests that this should only happen when in a multi-MDS system but we have been using only 1 MDS. A simple restart of the MDS solves this problem. * Client has a crash and either has been blocklisted or should be. Blocklisted clients may not immediately know they've been evicted and thus may be hanging indefinitely when stating the filesystem. This can be detected by searching through the client list via `ceph tell mds.N client ls` and seeing whether there is an ID associated with the client. However whilst a manual re-mount will allow the former client to establish new session, we prefer to use the "recover_session=clean" mount option to do this automatically and now we have no incidents of this. Yesterday for example, we had an incident of multiple MDS warning messages of: MDS_SLOW_REQUEST,  MDS_TRIM, MDS_CLIENT_RECALL, and MDS_CLIENT_LATE_RELEASE. This was caused by a non-responsive hard-drive leading to a build up of the MDS cache and trims being unable to be completed where we managed to narrow down the hard-drive for the inode which the blocked client was waiting for a rdlock on and restarted the OSD for that the drive. Notably this hard-drive had no errors with smartctl or elsewhere and only had the following slow ops message on OSD systemctl status/ log:     osd.281 123892 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.49576654.0:9101 3.d9bs0 3.ee6f2d9b (undecoded) ondisk+read+known_if_redirected e123889) Whilst restarting the hard-drive that the client with the oldest blocked op was waiting for did "clear" this /sys/kernel/debug/ceph/*/osdc queue, the oldest blocked MDS op then became an "internal op fragmentdir:mds.0:1" one where restarting the active MDS cleared this. Alas, this resulted in another blocked getattr op at the flag point "failed to authpin, dir is being fragmented" which was similarly tackled by restarting the MDS that just took over. This finally resulted in only two clients failing to respond to caps releases on inodes they were holding (despite rebooting at the time) where performing a "ceph tell mds.N session kill CLIENT_ID" removed them from the session map and allow the MDS' cache to become manageable again, thereby clearing all of these warning messages. We've had this problem since the beginning of this year and upgrading from octopus to quincy has unfortunately not solved our problem. We've only really been able to solve this problem by undergoing an aggressive campaign of replacing hard-drives which were reaching the end of their lives. This has substantially reduced the amount of problems we've had in relation to this. We would be very interested to hear about the rest of the community's experience in relation to this and I would recommend looking at your underlying OSDs Tim to see whether there are any timeout or uncorrectable errors. We would also be very eager to hear if these approaches are sub-optimal and whether anyone else has any insight into our problems. Sorry as well for resurrecting an old thread but we thought our experiences may be helpfully for others! Kindest regards, Ivan Clayson On 19/09/2023 12:35, Tim Bishop wrote: -- Ivan Clayson ----------------- Scientific Computing Officer MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH