CephFS error: currently failed to rdlock, waiting. clients crashing and evicted - ceph-users

17 Nov 2020

Hi all!

Hopefully some of you can shed some light on this. We have big problems with samba
crashing when macOS smb clients access certain/random folders/files over vfs_ceph.

When browsing cephfs folder in question directly on a cephnode where cephfs is mouted we
experience some issues like slow dir listing. We suspect that maybe macOS fetching of
xattr metadata creates a lot of traffic, but it should not lockup the cluster like this.
In logs we see both rdlock and wrlock, but mostly rdlocks.

End clients experience spurious disconnects when issue occurs, roughly up to a handfull
times a day. Is this a config issue? Have we hit a bug? It's certainly not a feature
:/

Any pointers on how to troubleshoot or rectify this problem is most welcome.

ceph version 14.2.11
samba version 4.12.10-SerNet-Ubuntu-10.focal
Supermicro X11, Intel Silver 4110, 9 ceph nodes, 2x40gbe network, 150OSD spinners, NVMe
db/journal

--

2020-11-17 22:09:07.525706 [WRN] evicting unresponsive client bo-samba-03 (3887652779),
after 301.746 seconds
2020-11-17 22:09:07.525580 [INF] Evicting (and blacklisting) client session 3877970532
(10.40.30.133:0/3971626932)
2020-11-17 22:09:07.525536 [WRN] evicting unresponsive client bo-samba-03 (3877970532),
after 302.034 seconds
2020-11-17 22:07:23.915412 [INF] Cluster is now healthy
2020-11-17 22:07:23.915381 [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs
report slow requests)
2020-11-17 22:07:23.915330 [INF] Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1
clients failing to respond to capability release)
2020-11-17 22:07:23.064492 [INF] MDS health message cleared (mds.?): 1 slow requests are
blocked > 30 secs
2020-11-17 22:07:23.064457 [INF] MDS health message cleared (mds.?): Client bo-samba-03
failing to respond to capability release
2020-11-17 22:07:17.524023 [WRN] client.3887663354 isn't responding to
mclientcaps(revoke), ino 0x10001202b55 pending pAsLsXsFs issued pAsLsXsFsx, sent 63.325997
seconds ago
2020-11-17 22:07:17.523987 [INF] Evicting (and blacklisting) client session 3887663354
(10.40.30.133:0/3230547239)
2020-11-17 22:07:17.523967 [WRN] evicting unresponsive client bo-samba-03 (3887663354),
after 64.5412 seconds
2020-11-17 22:07:17.523610 [WRN] slow request 63.325528 seconds old, received at
2020-11-17 22:06:14.197986: client_request(client.3878823430:4 lookup #0x100011f9a68/mappe
uten navn 2020-11-17 22:06:14.197908 caller_uid=111139, caller_gid=110513{}) currently
failed to rdlock, waiting
2020-11-17 22:07:17.523596 [WRN] 1 slow requests, 1 included below; oldest blocked for
> 63.325529 secs
2020-11-17 22:07:19.255177 [WRN] Health check failed: 1 clients failing to respond to
capability release (MDS_CLIENT_LATE_RELEASE)
2020-11-17 22:07:12.523453 [WRN] 1 slow requests, 0 included below; oldest blocked for
> 58.325433 secs
2020-11-17 22:07:07.523382 [WRN] 1 slow requests, 0 included below; oldest blocked for
> 53.325362 secs
2020-11-17 22:07:02.523360 [WRN] 1 slow requests, 0 included below; oldest blocked for
> 48.325307 secs
2020-11-17 22:06:57.523218 [WRN] 1 slow requests, 0 included below; oldest blocked for
> 43.325199 secs
2020-11-17 22:06:52.523203 [WRN] 1 slow requests, 0 included below; oldest blocked for
> 38.325158 secs
2020-11-17 22:06:47.523105 [WRN] slow request 33.325065 seconds old, received at
2020-11-17 22:06:14.197986: client_request(client.3878823430:4 lookup #0x100011f9a68/mappe
uten navn 2020-11-17 22:06:14.197908 caller_uid=111139, caller_gid=110513{}) currently
failed to rdlock, waiting
2020-11-17 22:06:47.523100 [WRN] 1 slow requests, 1 included below; oldest blocked for
> 33.325065 secs
2020-11-17 22:06:51.431745 [WRN] Health check failed: 1 MDSs report slow requests
(MDS_SLOW_REQUEST)
2020-11-17 22:06:20.045030 [INF] Cluster is now healthy
2020-11-17 22:06:20.045008 [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs
report slow requests)
2020-11-17 22:06:20.044960 [INF] Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1
clients failing to respond to capability release)
2020-11-17 22:06:19.062307 [INF] MDS health message cleared (mds.?): 1 slow requests are
blocked > 30 secs
2020-11-17 22:06:19.062253 [INF] MDS health message cleared (mds.?): Client bo-samba-03
failing to respond to capability release
2020-11-17 22:06:15.936150 [WRN] Health check failed: 1 clients failing to respond to
capability release (MDS_CLIENT_LATE_RELEASE)
2020-11-17 22:06:12.522624 [WRN] client.3869410498 isn't responding to
mclientcaps(revoke), ino 0x10001202b55 pending pAsLsXsFs issued pAsLsXsFsx, sent 64.045677
seconds ago

--thomas

--
Thomas Hukkelberg
thomas(a)hovedkvarteret.no
+47 971 81 192
--
support(a)hovedkvarteret.no
+47 966 44 999