Ceph MDS failing because of corrupted dentries in lost+found after update from 17.2.7 to 18.2.0
After we updated our ceph cluster from 17.2.7 to 18.2.0 the MDS kept being marked as damaged and stuck in up:standby with these errors in the log. debug -12> 2024-07-14T21:22:19.962+0000 7f020cf3a700 1 mds.0.cache.den(0x4 1000b3bcfea) loaded already corrupt dentry: [dentry #0x1/lost+found/1000b3bcfea [head,head] rep@0.0 NULL (dversion lock) pv=0 v=2 ino=(nil) state=0 0x558ca63b6500] debug -11> 2024-07-14T21:22:19.962+0000 7f020cf3a700 10 mds.0.cache.dir(0x4) go_bad_dentry 1000b3bcfea these log lines are repeated a bunch of times in our MDS logs, all on dentries that are within the lost+found directory. After reading this mailing list post <https://www.spinics.net/lists/ceph-users/msg77325.html>, we tried setting ceph config set mds mds_go_bad_corrupt_dentry false. This seemed to successfully circumvent the issue, however, after a few seconds our MDS crashes. Our 3 MDS are now stuck in a cycle of active -> crash -> standby -> back to active. Because of this our actual ceph fs is extremely laggy. We read here <https://docs.ceph.com/en/latest/releases/reef/#cephfs> that reef now makes it possible to delete the lost+found directory, which might solve our problem, but it is inaccessible, to cd, ls, rm, etc. Has anyone seen this type of issue or know how to solve it? Thanks!
Hi Justin, You should able to delete inodes from the lost+found dirs just by simply `sudo rm -rf lost+found/<ino>` What do you get when you try to delete? Do you get `EROFS`? On Fri, Aug 2, 2024 at 8:42 AM Justin Lee <justin.adam.lee@gmail.com> wrote:
After we updated our ceph cluster from 17.2.7 to 18.2.0 the MDS kept being marked as damaged and stuck in up:standby with these errors in the log.
debug -12> 2024-07-14T21:22:19.962+0000 7f020cf3a700 1 mds.0.cache.den(0x4 1000b3bcfea) loaded already corrupt dentry: [dentry #0x1/lost+found/1000b3bcfea [head,head] rep@0.0 NULL (dversion lock) pv=0 v=2 ino=(nil) state=0 0x558ca63b6500] debug -11> 2024-07-14T21:22:19.962+0000 7f020cf3a700 10 mds.0.cache.dir(0x4) go_bad_dentry 1000b3bcfea
these log lines are repeated a bunch of times in our MDS logs, all on dentries that are within the lost+found directory. After reading this mailing list post <https://www.spinics.net/lists/ceph-users/msg77325.html>, we tried setting ceph config set mds mds_go_bad_corrupt_dentry false. This seemed to successfully circumvent the issue, however, after a few seconds our MDS crashes. Our 3 MDS are now stuck in a cycle of active -> crash -> standby -> back to active. Because of this our actual ceph fs is extremely laggy.
We read here <https://docs.ceph.com/en/latest/releases/reef/#cephfs> that reef now makes it possible to delete the lost+found directory, which might solve our problem, but it is inaccessible, to cd, ls, rm, etc.
Has anyone seen this type of issue or know how to solve it? Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dhairya, Thanks for the response! We tried removing it as you suggested with `rm -rf` but the command just hangs indefinitely with no output. We are also unable to `ls lost_found`, or otherwise interact with the directory's contents. Best, Justin lee On Fri, Aug 2, 2024 at 8:24 AM Dhairya Parmar <dparmar@redhat.com> wrote:
Hi Justin,
You should able to delete inodes from the lost+found dirs just by simply `sudo rm -rf lost+found/<ino>`
What do you get when you try to delete? Do you get `EROFS`?
On Fri, Aug 2, 2024 at 8:42 AM Justin Lee <justin.adam.lee@gmail.com> wrote:
After we updated our ceph cluster from 17.2.7 to 18.2.0 the MDS kept being marked as damaged and stuck in up:standby with these errors in the log.
debug -12> 2024-07-14T21:22:19.962+0000 7f020cf3a700 1 mds.0.cache.den(0x4 1000b3bcfea) loaded already corrupt dentry: [dentry #0x1/lost+found/1000b3bcfea [head,head] rep@0.0 NULL (dversion lock) pv=0 v=2 ino=(nil) state=0 0x558ca63b6500] debug -11> 2024-07-14T21:22:19.962+0000 7f020cf3a700 10 mds.0.cache.dir(0x4) go_bad_dentry 1000b3bcfea
these log lines are repeated a bunch of times in our MDS logs, all on dentries that are within the lost+found directory. After reading this mailing list post <https://www.spinics.net/lists/ceph-users/msg77325.html>, we tried setting ceph config set mds mds_go_bad_corrupt_dentry false. This seemed to successfully circumvent the issue, however, after a few seconds our MDS crashes. Our 3 MDS are now stuck in a cycle of active -> crash -> standby -> back to active. Because of this our actual ceph fs is extremely laggy.
We read here <https://docs.ceph.com/en/latest/releases/reef/#cephfs> that reef now makes it possible to delete the lost+found directory, which might solve our problem, but it is inaccessible, to cd, ls, rm, etc.
Has anyone seen this type of issue or know how to solve it? Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So the mount hung? Can you see anything suspicious in the logs? On Fri, Aug 2, 2024 at 7:17 PM Justin Lee <justin.adam.lee@gmail.com> wrote:
Hi Dhairya,
Thanks for the response! We tried removing it as you suggested with `rm -rf` but the command just hangs indefinitely with no output. We are also unable to `ls lost_found`, or otherwise interact with the directory's contents.
Best, Justin lee
On Fri, Aug 2, 2024 at 8:24 AM Dhairya Parmar <dparmar@redhat.com> wrote:
Hi Justin,
You should able to delete inodes from the lost+found dirs just by simply `sudo rm -rf lost+found/<ino>`
What do you get when you try to delete? Do you get `EROFS`?
On Fri, Aug 2, 2024 at 8:42 AM Justin Lee <justin.adam.lee@gmail.com> wrote:
After we updated our ceph cluster from 17.2.7 to 18.2.0 the MDS kept being marked as damaged and stuck in up:standby with these errors in the log.
debug -12> 2024-07-14T21:22:19.962+0000 7f020cf3a700 1 mds.0.cache.den(0x4 1000b3bcfea) loaded already corrupt dentry: [dentry #0x1/lost+found/1000b3bcfea [head,head] rep@0.0 NULL (dversion lock) pv=0 v=2 ino=(nil) state=0 0x558ca63b6500] debug -11> 2024-07-14T21:22:19.962+0000 7f020cf3a700 10 mds.0.cache.dir(0x4) go_bad_dentry 1000b3bcfea
these log lines are repeated a bunch of times in our MDS logs, all on dentries that are within the lost+found directory. After reading this mailing list post <https://www.spinics.net/lists/ceph-users/msg77325.html>, we tried setting ceph config set mds mds_go_bad_corrupt_dentry false. This seemed to successfully circumvent the issue, however, after a few seconds our MDS crashes. Our 3 MDS are now stuck in a cycle of active -> crash -> standby -> back to active. Because of this our actual ceph fs is extremely laggy.
We read here <https://docs.ceph.com/en/latest/releases/reef/#cephfs> that reef now makes it possible to delete the lost+found directory, which might solve our problem, but it is inaccessible, to cd, ls, rm, etc.
Has anyone seen this type of issue or know how to solve it? Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The actual mount command doesn't hang, we just can't interact with any of the directory's contents once mounted. I couldn't find anything unusual in the logs. Best, Justin Lee On Fri, Aug 2, 2024 at 10:38 AM Dhairya Parmar <dparmar@redhat.com> wrote:
So the mount hung? Can you see anything suspicious in the logs?
On Fri, Aug 2, 2024 at 7:17 PM Justin Lee <justin.adam.lee@gmail.com> wrote:
Hi Dhairya,
Thanks for the response! We tried removing it as you suggested with `rm -rf` but the command just hangs indefinitely with no output. We are also unable to `ls lost_found`, or otherwise interact with the directory's contents.
Best, Justin lee
On Fri, Aug 2, 2024 at 8:24 AM Dhairya Parmar <dparmar@redhat.com> wrote:
Hi Justin,
You should able to delete inodes from the lost+found dirs just by simply `sudo rm -rf lost+found/<ino>`
What do you get when you try to delete? Do you get `EROFS`?
On Fri, Aug 2, 2024 at 8:42 AM Justin Lee <justin.adam.lee@gmail.com> wrote:
After we updated our ceph cluster from 17.2.7 to 18.2.0 the MDS kept being marked as damaged and stuck in up:standby with these errors in the log.
debug -12> 2024-07-14T21:22:19.962+0000 7f020cf3a700 1 mds.0.cache.den(0x4 1000b3bcfea) loaded already corrupt dentry: [dentry #0x1/lost+found/1000b3bcfea [head,head] rep@0.0 NULL (dversion lock) pv=0 v=2 ino=(nil) state=0 0x558ca63b6500] debug -11> 2024-07-14T21:22:19.962+0000 7f020cf3a700 10 mds.0.cache.dir(0x4) go_bad_dentry 1000b3bcfea
these log lines are repeated a bunch of times in our MDS logs, all on dentries that are within the lost+found directory. After reading this mailing list post <https://www.spinics.net/lists/ceph-users/msg77325.html>, we tried setting ceph config set mds mds_go_bad_corrupt_dentry false. This seemed to successfully circumvent the issue, however, after a few seconds our MDS crashes. Our 3 MDS are now stuck in a cycle of active -> crash -> standby -> back to active. Because of this our actual ceph fs is extremely laggy.
We read here <https://docs.ceph.com/en/latest/releases/reef/#cephfs> that reef now makes it possible to delete the lost+found directory, which might solve our problem, but it is inaccessible, to cd, ls, rm, etc.
Has anyone seen this type of issue or know how to solve it? Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Dhairya Parmar
-
Justin Lee