Hi, here is the sequence of commands I executed to change the directory to a different pools, and then exact sequence in reverse. system("ceph osd crush rule create-replicated replicated_hdd_rule default host hdd"); system("ceph osd pool create tier2mit_rep_hdd replicated --bulk"); system("ceph osd pool set tier2mit_rep_hdd crush_rule replicated_hdd_rule"); system("ceph osd pool set tier2mit_rep_hdd size 2"); system("ceph fs add_data_pool tier2mit tier2mit_rep_hdd"); system("cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_rep_hdd"); cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_ec_hdd then removed pool from fs, and deleted it. One of the things we tried was to put pool in failed state and then rejoin it (ceph fs fail tier2mit; ceph fs set tier2mit joinable true), here is a log from MDS, you can see where it puts itself in readonly mode. https://www.cmsaf.mit.edu/various/messages Thankyou, --Max ________________________________ From: Eugen Block <eblock@nde.ag> Sent: Monday, May 5, 2025 5:47 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state Hi, do you have logs from the MDS? And how exactly did you move the /temp directory to a different pool? That might help understand what happened. There was a thread here a couple of weeks ago how about moving contents to a different CephFS pool. Zitat von Maxim Goncharov <maxi@mit.edu>:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io