Same problem here with Ceph 17.2.6 on Ubuntu 22.04 and Clients Debian 11, Kernel 6.0.12-1~bpo11+1. We are still looking for a solution. At the time being we let restart the Orchestrator MDS daemons by removig/adding labels to the servers. We use multiple MDS and have many CPU cores and memory. The problem should not be due to a lack of resources. Am Di., 19. Sept. 2023 um 13:36 Uhr schrieb Tim Bishop < tim-lists@bishnet.net>:
Hi,
I've seen this issue mentioned in the past, but with older releases. So I'm wondering if anybody has any pointers.
The Ceph cluster is running Pacific 16.2.13 on Ubuntu 20.04. Almost all clients are working fine, with the exception of our backup server. This is using the kernel CephFS client on Ubuntu 22.04 with kernel 6.2.0 [1] (so I suspect a newer Ceph version?).
The backup server has multiple (12) CephFS mount points. One of them, the busiest, regularly causes this error on the cluster:
HEALTH_WARN 1 clients failing to respond to capability release [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release mds.mds-server(mds.0): Client backupserver:cephfs-backupserver failing to respond to capability release client_id: 521306112
And occasionally, which may be unrelated, but occurs at the same time:
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.mds-server(mds.0): 1 slow requests are blocked > 30 secs
The second one clears itself, but the first sticks until I can unmount the filesystem on the client after the backup completes.
It appears that whilst it's in this stuck state there may be one or more directory trees that are inaccessible to all clients. The backup server is walking the whole tree but never gets stuck itself, so either the inaccessible directory entry is caused after it has gone past, or it's not affected. Maybe the backup server is holding a directory when it shouldn't?
It may be that an upgrade to Quincy resolves this, since it's more likely to be inline with the kernel client version wise, but I don't want to knee-jerk upgrade just to try and fix this problem.
Thanks for any advice.
Tim.
[1] The reason for the newer kernel is that the backup performance from CephFS was terrible with older kernels. This newer kernel does at least resolve that issue. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io