Hi, it seems like you only changed the xattr, but that doesn't really move the objects into the new pool, only new objects would be created in the additional pool. There was a thread [0] on this list a few months ago, pointing to [1] how to actually migrate data to a different pool. I'm just not sure how to get out of this situation. Based on the description in the docs [2], writes to the metadata pool are blocked. So one option could be to reset the journal (take a backup first) as described in the troubleshooting guide [3]. Maybe someone else has other ideas... Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/M72ZAQPMX3S... [1] https://ewal.dev/cephfs-migrating-files-between-pools [2] https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-health-read-only [3] https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#journal-tr... Zitat von Maxim Goncharov <maxi@mit.edu>:
Hi, here is the sequence of commands I executed to change the directory to a different pools, and then exact sequence in reverse.
system("ceph osd crush rule create-replicated replicated_hdd_rule default host hdd"); system("ceph osd pool create tier2mit_rep_hdd replicated --bulk"); system("ceph osd pool set tier2mit_rep_hdd crush_rule replicated_hdd_rule"); system("ceph osd pool set tier2mit_rep_hdd size 2"); system("ceph fs add_data_pool tier2mit tier2mit_rep_hdd"); system("cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_rep_hdd");
cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_ec_hdd then removed pool from fs, and deleted it.
One of the things we tried was to put pool in failed state and then rejoin it (ceph fs fail tier2mit; ceph fs set tier2mit joinable true), here is a log from MDS, you can see where it puts itself in readonly mode.
https://www.cmsaf.mit.edu/various/messages
Thankyou, --Max
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Monday, May 5, 2025 5:47 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state
Hi,
do you have logs from the MDS? And how exactly did you move the /temp directory to a different pool? That might help understand what happened. There was a thread here a couple of weeks ago how about moving contents to a different CephFS pool.
Zitat von Maxim Goncharov <maxi@mit.edu>:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io