hello. I am using ROK CEPH and have 20 MDSs in use. 10 are in rank 0-9 and 10 are in standby. I have one ceph filesystem, and 2 mds are trimming. Under one FILESYSTEM, there are 6 MDSs in RESOLVE, 1 MDS in REPLAY, and 3 in ACTIVE. For some reason, since 36 hours ago, RESOLVE is stuck in TRIMMING, and so are the MDSs in REPLAY. I've also tried FAILing each MDS, but to no avail. I think something should change when the MDS in REPLAY goes to RESOLVE, but I don't know what. Even looking at the logs of the REPLAY MDS, it's hard to see any messages other than it is TERMINATED every 11 minutes. I'm desperate for someone's help.
if possible, could you share the mds logs at debug level 20 you'll need to set debug_mds = 20 in the conf file until the crash and revert the level to the default after mds crash On Tue, Jul 18, 2023 at 9:12 PM <dxodnd@naver.com> wrote:
hello. I am using ROK CEPH and have 20 MDSs in use. 10 are in rank 0-9 and 10 are in standby. I have one ceph filesystem, and 2 mds are trimming. Under one FILESYSTEM, there are 6 MDSs in RESOLVE, 1 MDS in REPLAY, and 3 in ACTIVE. For some reason, since 36 hours ago, RESOLVE is stuck in TRIMMING, and so are the MDSs in REPLAY. I've also tried FAILing each MDS, but to no avail. I think something should change when the MDS in REPLAY goes to RESOLVE, but I don't know what. Even looking at the logs of the REPLAY MDS, it's hard to see any messages other than it is TERMINATED every 11 minutes. I'm desperate for someone's help. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Milind
I think the rook-ceph is not responding to the liveness probe (confirmed by k8s describe mds pod) I don't think it's the memory as I don't limit it, and I have the cpu set to 500m per mds, but what direction should I go from here?
This issue has been closed. If any rook-ceph users see this, when mds replay takes a long time, look at the logs in mds pod. If it's going well and then abruptly terminates, try describing the mds pod, and if liveness probe terminated, try increasing the threadhold of liveness probe.
On 7/20/23 11:36, dxodnd@naver.com wrote:
This issue has been closed. If any rook-ceph users see this, when mds replay takes a long time, look at the logs in mds pod. If it's going well and then abruptly terminates, try describing the mds pod, and if liveness probe terminated, try increasing the threadhold of liveness probe.
Do you mean increasing the pod grace period ? Thanks - Xiubo
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If any rook-ceph users see the situation that mds is stuck in replay, then look at the logs of the mds pod. When it runs and then terminates repeatedly, check if there is "liveness probe termninated" error message by typing "kubectl describe pod -n (namspace) (mds' pod name)" If there is the error message, it's helpful to increase the threshold about "liveness probe" In my case, it resolved the issue. Thanks
On Thu, Jul 20, 2023 at 11:19 PM <dxodnd@naver.com> wrote:
If any rook-ceph users see the situation that mds is stuck in replay, then look at the logs of the mds pod.
When it runs and then terminates repeatedly, check if there is "liveness probe termninated" error message by typing "kubectl describe pod -n (namspace) (mds' pod name)"
If there is the error message, it's helpful to increase the threshold about "liveness probe"
In my case, it resolved the issue.
Would you mind sharing what version of ceph (mds) was used? In a particular (pacific) release, the mds would abort when it received an metric update message (from a client) that it did not understand.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
participants (4)
-
dxodnd＠naver.com
-
Milind Changire
-
Venky Shankar
-
Xiubo Li