I hope this message finds you well. I have a cephfs cluster with 3 active mds, and use 3-node samba to export through the kernel. Currently, there are 2 node mds experiencing slow requests. We have tried restarting the mds. After a few hours, the replay log status became active. But the slow request reappears. The slow request does not seem to come from the client, but from the request of the mds node. Looking forward to your prompt response. HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests mds.osd44(mds.0): 2 slow requests are blocked > 30 secs mds.osd43(mds.1): 2 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 2 MDSs behind on trimming mds.osd44(mds.0): Behind on trimming (18642/1024) max_segments: 1024, num_segments: 18642 mds.osd43(mds.1): Behind on trimming (976612/1024) max_segments: 1024, num_segments: 976612 mds.0 { "ops": [ { "description": "peer_request:mds.1:1", "initiated_at": "2023-12-31T11:19:38.679925+0800", "age": 4358.8009461359998, "duration": 4358.8009636369998, "type_data": { "flag_point": "dispatched", "reqid": "mds.1:1", "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2023-12-31T11:19:38.679925+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:38.679925+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:38.679925+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:38.679936+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:38.679940+0800", "event": "dispatched" } ] } }, { "description": "peer_request:mds.1:2", "initiated_at": "2023-12-31T11:19:38.679938+0800", "age": 4358.8009326969996, "duration": 4358.8009763549999, "type_data": { "flag_point": "dispatched", "reqid": "mds.1:2", "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2023-12-31T11:19:38.679938+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:38.679938+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:38.679938+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:38.679941+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:38.679991+0800", "event": "dispatched" } ] } } ], "complaint_time": 30, "num_blocked_ops": 2 } mds.1 { "ops": [ { "description": "internal op exportdir:mds.1:1", "initiated_at": "2023-12-31T11:19:34.416451+0800", "age": 4384.38814198, "duration": 4384.3881617610004, "type_data": { "flag_point": "failed to wrlock, waiting", "reqid": "mds.1:1", "op_type": "internal_op", "internal_op": 5377, "op_name": "exportdir", "events": [ { "time": "2023-12-31T11:19:34.416451+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "dispatched" }, { "time": "2023-12-31T11:19:38.679923+0800", "event": "requesting remote authpins" }, { "time": "2023-12-31T11:19:38.693981+0800", "event": "failed to wrlock, waiting" } ] } }, { "description": "internal op exportdir:mds.1:2", "initiated_at": "2023-12-31T11:19:34.416482+0800", "age": 4384.3881117999999, "duration": 4384.3881714600002, "type_data": { "flag_point": "failed to wrlock, waiting", "reqid": "mds.1:2", "op_type": "internal_op", "internal_op": 5377, "op_name": "exportdir", "events": [ { "time": "2023-12-31T11:19:34.416482+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "dispatched" }, { "time": "2023-12-31T11:19:38.679929+0800", "event": "requesting remote authpins" }, { "time": "2023-12-31T11:19:38.693995+0800", "event": "failed to wrlock, waiting" } ] } } ], "complaint_time": 30, "num_blocked_ops": 2 } I can't find any other solution other than restarting the mds service with slow requests. Currently, the backlog of mds logs in the metadata pool exceeds 4TB. Best regards,