MDS_CLIENTS_BROKEN_ROOTSQUASH error after upgrade to 18.2.4
Dear Ceph developers, I originally posted this question on the users mailing list and got the suggestion to post it also here. Yesterday I updated my Cephadm-managed cluster from 18.2.2 to 18.2.4, and since then I get a MDS_CLIENTS_BROKEN_ROOTSQUASH error which I didn't get with 18.2.2. As far as I could go with web search, I understood that this error is due to a missing feature in the CephFS client, but also that the feature is going to be introduced with version 19. So I'd need to understand why this error comes out and if it's possible how to fix it. One thing that might be relevant is that some clients trigger the error while some other doesn't: - Ceph 18.2.2 kernel client on Rocky Linux 8: error - Ceph 18.2.4 kernel client on Rocky Linux 9: error - Ceph 18.2.4 kernel client on Archlinux (on kernel 6.10.6): no error From the above facts I'd say that the error might be triggered by some compilation flags, since there's no correlation with client version, is it plausible? Thanks, Nicola
Hi Nicola, Is it possible please to provide the exact kernel version you are using in Rocky Linux and also in Archlinux ? It's curious to see that Ceph 18.2.4 on Rocky Linux is not working but on Archlinux you have no issues. From what I know Arch Linux tends to ship much more recent kernels than Rocky Linux. Thank you, Bogdan Velica croit.io On Thu, Aug 22, 2024 at 11:34 AM Nicola Mori <mori@fi.infn.it> wrote:
Dear Ceph developers,
I originally posted this question on the users mailing list and got the suggestion to post it also here. Yesterday I updated my Cephadm-managed cluster from 18.2.2 to 18.2.4, and since then I get a MDS_CLIENTS_BROKEN_ROOTSQUASH error which I didn't get with 18.2.2. As far as I could go with web search, I understood that this error is due to a missing feature in the CephFS client, but also that the feature is going to be introduced with version 19. So I'd need to understand why this error comes out and if it's possible how to fix it. One thing that might be relevant is that some clients trigger the error while some other doesn't:
- Ceph 18.2.2 kernel client on Rocky Linux 8: error - Ceph 18.2.4 kernel client on Rocky Linux 9: error - Ceph 18.2.4 kernel client on Archlinux (on kernel 6.10.6): no error
From the above facts I'd say that the error might be triggered by some compilation flags, since there's no correlation with client version, is it plausible? Thanks,
Nicola _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Bogdan, here's the info you requested: Rocky Linux 8: $ uname -a Linux bofur 4.18.0-553.8.1.el8_10.x86_64 #1 SMP Tue Jul 2 17:10:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Rocky Linux 9: $ uname -a Linux fili 5.14.0-427.31.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 14 16:15:25 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Archlinux: $ uname -a Linux stryke 6.10.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 19 Aug 2024 17:02:39 +0000 x86_64 GNU/Linux Cheers, Nicola
Thank you, The Rocky Linux 8 (4.18.x) kernel is quite old (2018), and Rocky Linux 9 (5.14.x) is newer (2021), but still behind I think.... Both may lack recent features or patches related to CephFS rootsquash. In contrast, Arch Linux (6.10.x) uses a cutting-edge, up-to-date kernel that includes the latest support and improvements, explaining the issue differences you're seeing. I would suggest upgrading the kernel clients if possible and see if this issue is still reproducible . Just my 2 cents. Thank you, Bogdan V. croit.io On Thu, Aug 22, 2024 at 1:30 PM Nicola Mori <mori@fi.infn.it> wrote:
Hi Bogdan,
here's the info you requested:
Rocky Linux 8: $ uname -a Linux bofur 4.18.0-553.8.1.el8_10.x86_64 #1 SMP Tue Jul 2 17:10:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Rocky Linux 9: $ uname -a Linux fili 5.14.0-427.31.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 14 16:15:25 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Archlinux: $ uname -a Linux stryke 6.10.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 19 Aug 2024 17:02:39 +0000 x86_64 GNU/Linux
Cheers,
Nicola _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Thank you Bogdan, I'll see if I can install a more recent kernel on a Rocky Linux machine and will let you know. Cheers, Nicola
Hi Bogdan, I upgraded the kernel on Rocky Linux 8: $ uname -a Linux bofur 6.10.6-1.el8.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 19 13:21:58 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux and 9: $ uname -a Linux fili 6.10.6-1.el9.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 19 13:18:57 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux Now the client on Rocky Linux 9 does not trigger the error anymore. So I'd say it all boils down to the kernel version in use. By the way, I still have one client triggering the error, but the machine is not under my control so I cannot make extensive tests; anyway the Ceph file systems dashboard reports kernel "6.8.0-36-generic" for this client (probably it's a Ubuntu 24.04 machine). So I'd say that kernel >= 6.9 is needed, does it sound reasonable to you? Thanks for your help, Nicola
Sorry, I forgot to mention that also on Rocky 8 with the new kernel the client does not trigger the error anymore. Cheers, Nicola
Glad that worked for you, All the best, Bogdan Velica croit.io On Thu, Aug 22, 2024 at 4:08 PM Nicola Mori <mori@fi.infn.it> wrote:
Sorry, I forgot to mention that also on Rocky 8 with the new kernel the client does not trigger the error anymore. Cheers,
Nicola _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (2)
-
Bogdan Adrian Velica
-
Nicola Mori