Ceph 17.2.7 to 18.2.0 issues
Hi All, Recently I upgraded my cluster from Quincy to Reef. Everything appeared to go smoothly and without any issues arising. I was forced to poweroff the cluster, performing the ususal procedures beforehand and everything appears to have come back fine. Every service reports green across the board except.... If i try to copy any files from a cephfs mountpoint whether kernel or fuse the actual copy will hang. ls/stat etc all work which indicates metadata appears fine but copying always hangs. I can copy objects direct using the rados toolset which indicates the underlying data exists. The system itself reports no errors and thinks its healthy. The entire cluster and cephfs clients are all Rocky9. Any advice would be much appreciatd. I'd find this easier to deal with if the cluster actually gave me an error....
Hi, did you unmount your clients after the cluster poweroff? You could also enable debug logs in mds to see more information. Are there any blocked requests? You can query the mds daemon via cephadm shell or with ad admin keyring like this: # ceph tell mds.cephfs.storage.lgmyqv dump_blocked_ops Regards, Eugen Zitat von pclark6063@outlook.com:
Hi All,
Recently I upgraded my cluster from Quincy to Reef. Everything appeared to go smoothly and without any issues arising. I was forced to poweroff the cluster, performing the ususal procedures beforehand and everything appears to have come back fine. Every service reports green across the board except....
If i try to copy any files from a cephfs mountpoint whether kernel or fuse the actual copy will hang. ls/stat etc all work which indicates metadata appears fine but copying always hangs.
I can copy objects direct using the rados toolset which indicates the underlying data exists.
The system itself reports no errors and thinks its healthy.
The entire cluster and cephfs clients are all Rocky9.
Any advice would be much appreciatd. I'd find this easier to deal with if the cluster actually gave me an error.... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, Dec 7, 2023 at 12:49 PM Eugen Block <eblock@nde.ag> wrote:
Hi, did you unmount your clients after the cluster poweroff?
If this is the case, then a remount would kick things back working.
You could also enable debug logs in mds to see more information. Are there any blocked requests? You can query the mds daemon via cephadm shell or with ad admin keyring like this:
# ceph tell mds.cephfs.storage.lgmyqv dump_blocked_ops
Regards, Eugen
Zitat von pclark6063@outlook.com:
Hi All,
Recently I upgraded my cluster from Quincy to Reef. Everything appeared to go smoothly and without any issues arising. I was forced to poweroff the cluster, performing the ususal procedures beforehand and everything appears to have come back fine. Every service reports green across the board except....
If i try to copy any files from a cephfs mountpoint whether kernel or fuse the actual copy will hang. ls/stat etc all work which indicates metadata appears fine but copying always hangs.
I can copy objects direct using the rados toolset which indicates the underlying data exists.
The system itself reports no errors and thinks its healthy.
The entire cluster and cephfs clients are all Rocky9.
Any advice would be much appreciatd. I'd find this easier to deal with if the cluster actually gave me an error.... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
Thanks for this, I've replied above but sadly a client eviction and remount didn't help.
Hi, Thank you very much for the reply. So I evicted all my clients and still no luck. Check for blocked ops returns 0 from each mds service. Each mds service is serving a different pool suffering the same issue. If I write any recent files I can both stat and pull those so I have zero issues writing into the pool or pulling those newer files. Everything pre-reboot however just hangs when you try an actual copy. I've tried both fuse and kernel mounts and may try a nfs server and see if that makes a difference. The whole cluster still reports healthy system and healthy volumes with no pgs stuck in deep scrub. Tempted to reboot the system....
participants (3)
-
Eugen Block
-
pclark6063＠outlook.com
-
Venky Shankar