Hi Ivan, I don't think we're necessarily seeing the same issue. Mine didn't seem to be related to OSDs, and in fact I could unblock it by killing the backup job on our backup server and unmounting the filesystem. This would then release all other stuck ops on the MDS. I've been waiting to follow-up, so as not to tempt fate, but I upgraded to Quincy at the end of last week and so far I haven't had a single issue. Whether this is due to a bugfix in the MDS code, or running MDS and client versions that are nearer to each other (the client was much newer), I don't know. But for now, I'm relieved. I did find that enabling snapshots on CephFS exacerbated the problem, so for now they're disabled. Once I'm happy things are working I'll give them another go. Hopefully your detailed response will be useful to others facing similar issues. Tim. On Thu, Oct 12, 2023 at 09:22:03AM +0100, Ivan Clayson wrote:

Hi Tim,

We've been seeing something that may be similar to you with concurrent MDS_CLIENT_LATE_RELEASE and MDS_SLOW_REQUEST warning messages as well as frequently MDS_CLIENT_RECALL and MDS_SLOW_METADATA_IO warnings from the same MDS referring to the same client. We are using 1 MDS for our non-containerised filesystem on a 5.7 PiB sized Alma8.8 cluster with 29 nodes and 348 spinning OSDs for bulk data and 4 OSDs for the metadata on NVMe SSDs with 174 million files (we also have additional NVMe drive partitions as DB and WAL devices for each OSD). All of our clients are using the kernel mount where we also have a SMB gateway which kernel mounts the filesystem and shares it to Windows and Mac machines.

This problem seems to have various symptoms including but not limited to: (i) a particular file or directory hanging on openfs, read, and or statx system calls for all clients mounting the filesystem, (ii) all our connected clients hanging on the aforementioned system calls when performing any metadata or bulk data I/O on the filesystem, (iii) 1 client being unable to stat or cd into the filesystem whilst all other clients are completely unaffected, and (iv) possibly a near complete loss of metadata due to the MDS journal swelling beyond the size of the MDS pool where we eventually had some success following the recovery docs (https://docs.ceph.com/en/quincy/cephfs/disaster-recovery-experts/) but opted for a backup restore instead (we were able to restore the bulk data to a read-only state but not all the metadata where we would be more than happy to write a "worked example" of this if the community would find it useful). We use to solve this by forcefully re-mounting the filesystem on the client but as we run multi-user systems and have 100s of clients, this was not a sustainable option for us.

We notice generally 3 classes of error:

* The oldest slow or blocked request in the ceph-mds log is waiting to acquire locks for an inode (visible via `ceph tell mds.N dump_blocked_ops`). By tracing the oldest FS client which has the capability for this file and seeing which OSD the client is waiting for with `watch 'cat /sys/kernel/debug/ceph/*/osdc'` or by looking at the output of `ceph tell mds.N objecter requests`, we can determine which exact OSD is being slow (always a spinning disk). These issues are due to hardware issues or the OSD crashing (where a restart seems to remedy the problem) and thus counterintuitively (at least for me) suggests that our error messages for the MDS can refer to OSD hard-drive problems. These HDDs are in a completely separate pool to where I would have thought the metadata I/O would be being performed since our metadata pool is exclusively SSDs. * The oldest, or near oldest, blocked MDS op being an "internal op fragmentdir" operation and many client_requests at the flag_point "failed to authpin, dir is being fragmented". This seems to be an MDS bug as the documentation suggests that this should only happen when in a multi-MDS system but we have been using only 1 MDS. A simple restart of the MDS solves this problem. * Client has a crash and either has been blocklisted or should be. Blocklisted clients may not immediately know they've been evicted and thus may be hanging indefinitely when stating the filesystem. This can be detected by searching through the client list via `ceph tell mds.N client ls` and seeing whether there is an ID associated with the client. However whilst a manual re-mount will allow the former client to establish new session, we prefer to use the "recover_session=clean" mount option to do this automatically and now we have no incidents of this.

Yesterday for example, we had an incident of multiple MDS warning messages of: MDS_SLOW_REQUEST,  MDS_TRIM, MDS_CLIENT_RECALL, and MDS_CLIENT_LATE_RELEASE. This was caused by a non-responsive hard-drive leading to a build up of the MDS cache and trims being unable to be completed where we managed to narrow down the hard-drive for the inode which the blocked client was waiting for a rdlock on and restarted the OSD for that the drive. Notably this hard-drive had no errors with smartctl or elsewhere and only had the following slow ops message on OSD systemctl status/ log:

    osd.281 123892 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.49576654.0:9101 3.d9bs0 3.ee6f2d9b (undecoded) ondisk+read+known_if_redirected e123889)

Whilst restarting the hard-drive that the client with the oldest blocked op was waiting for did "clear" this /sys/kernel/debug/ceph/*/osdc queue, the oldest blocked MDS op then became an "internal op fragmentdir:mds.0:1" one where restarting the active MDS cleared this. Alas, this resulted in another blocked getattr op at the flag point "failed to authpin, dir is being fragmented" which was similarly tackled by restarting the MDS that just took over. This finally resulted in only two clients failing to respond to caps releases on inodes they were holding (despite rebooting at the time) where performing a "ceph tell mds.N session kill CLIENT_ID" removed them from the session map and allow the MDS' cache to become manageable again, thereby clearing all of these warning messages.

We've had this problem since the beginning of this year and upgrading from octopus to quincy has unfortunately not solved our problem. We've only really been able to solve this problem by undergoing an aggressive campaign of replacing hard-drives which were reaching the end of their lives. This has substantially reduced the amount of problems we've had in relation to this.

We would be very interested to hear about the rest of the community's experience in relation to this and I would recommend looking at your underlying OSDs Tim to see whether there are any timeout or uncorrectable errors. We would also be very eager to hear if these approaches are sub-optimal and whether anyone else has any insight into our problems. Sorry as well for resurrecting an old thread but we thought our experiences may be helpfully for others!

Kindest regards,

Ivan Clayson

On 19/09/2023 12:35, Tim Bishop wrote:

...

Hi,

I've seen this issue mentioned in the past, but with older releases. So I'm wondering if anybody has any pointers.

The Ceph cluster is running Pacific 16.2.13 on Ubuntu 20.04. Almost all clients are working fine, with the exception of our backup server. This is using the kernel CephFS client on Ubuntu 22.04 with kernel 6.2.0 [1] (so I suspect a newer Ceph version?).

The backup server has multiple (12) CephFS mount points. One of them, the busiest, regularly causes this error on the cluster:

HEALTH_WARN 1 clients failing to respond to capability release [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release mds.mds-server(mds.0): Client backupserver:cephfs-backupserver failing to respond to capability release client_id: 521306112

And occasionally, which may be unrelated, but occurs at the same time:

[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.mds-server(mds.0): 1 slow requests are blocked > 30 secs

The second one clears itself, but the first sticks until I can unmount the filesystem on the client after the backup completes.

It appears that whilst it's in this stuck state there may be one or more directory trees that are inaccessible to all clients. The backup server is walking the whole tree but never gets stuck itself, so either the inaccessible directory entry is caused after it has gone past, or it's not affected. Maybe the backup server is holding a directory when it shouldn't?

It may be that an upgrade to Quincy resolves this, since it's more likely to be inline with the kernel client version wise, but I don't want to knee-jerk upgrade just to try and fix this problem.

Thanks for any advice.

Tim.

[1] The reason for the newer kernel is that the backup performance from CephFS was terrible with older kernels. This newer kernel does at least resolve that issue. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

-- CAUTION: This email originated from outside of the LMB. Do not click links or open attachments unless you recognize the sender and know the content is safe. .-ceph-users-bounces@ceph.io-.

-- Ivan Clayson ----------------- Scientific Computing Officer MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

Tim. -- Tim Bishop http://www.bishnet.net/tim/ PGP Key: 0x6C226B37FDF38D55