Hi Giuseppe, There are likely one or two clients whose op is blocking the reconnect/replay. If you increase debug_mds perhaps you can find the guilty client and disconnect it / block it from mounting. Or for a more disruptive recovery you can try this "Deny all reconnect to clients " option: https://docs.ceph.com/en/reef/cephfs/troubleshooting/#avoiding-recovery-road... Hope that helps, Dan On Mon, Nov 27, 2023 at 1:08 AM Lo Re Giuseppe <giuseppe.lore@cscs.ch> wrote:
Hi, We have upgraded one ceph cluster from 17.2.7 to 18.2.0. Since then we are having CephFS issues. For example this morning: “”” [root@naret-monitor01 ~]# ceph -s cluster: id: 63334166-d991-11eb-99de-40a6b72108d0 health: HEALTH_WARN 1 filesystem is degraded 3 clients failing to advance oldest client/flush tid 3 MDSs report slow requests 6 pgs not scrubbed in time 29 daemons have recently crashed … “””
The ceph orch, ceph crash and ceph fs status commands were hanging.
After a “ceph mgr fail” those commands started to respond. Then I have noticed that there was one mds with most of the slow operations,
“”” [WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests mds.cephfs.naret-monitor01.nuakzo(mds.0): 18 slow requests are blocked > 30 secs mds.cephfs.naret-monitor01.uvevbf(mds.1): 1683 slow requests are blocked > 30 secs mds.cephfs.naret-monitor02.exceuo(mds.2): 1 slow requests are blocked > 30 secs “””
Then I tried to restart it with
“”” [root@naret-monitor01 ~]# ceph orch daemon restart mds.cephfs.naret-monitor01.uvevbf Scheduled to restart mds.cephfs.naret-monitor01.uvevbf on host 'naret-monitor01' “””
After the cephfs entered into this situation: “”” [root@naret-monitor01 ~]# ceph fs status cephfs - 198 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.naret-monitor01.nuakzo Reqs: 0 /s 17.2k 16.2k 1892 14.3k 1 active cephfs.naret-monitor02.ztdghf Reqs: 0 /s 28.1k 10.3k 752 6881 2 clientreplay cephfs.naret-monitor02.exceuo 63.0k 6491 541 66 3 active cephfs.naret-monitor03.lqppte Reqs: 0 /s 16.7k 13.4k 8233 990 POOL TYPE USED AVAIL cephfs.cephfs.meta metadata 5888M 18.5T cephfs.cephfs.data data 119G 215T cephfs.cephfs.data.e_4_2 data 2289G 3241T cephfs.cephfs.data.e_8_3 data 9997G 470T STANDBY MDS cephfs.naret-monitor03.eflouf cephfs.naret-monitor01.uvevbf MDS version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) “””
The file system is totally unresponsive (we can mount it on client nodes but any operations like a simple ls hangs).
During the night we had a lot of mds crashes, I can share the content.
Does anybody have an idea on how to tackle this problem?
Best,
Giuseppe _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io