Need help with MDS stuck in readonly state
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool. I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again? Thank you, --Max
Hi, do you have logs from the MDS? And how exactly did you move the /temp directory to a different pool? That might help understand what happened. There was a thread here a couple of weeks ago how about moving contents to a different CephFS pool. Zitat von Maxim Goncharov <maxi@mit.edu>:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, here is the sequence of commands I executed to change the directory to a different pools, and then exact sequence in reverse. system("ceph osd crush rule create-replicated replicated_hdd_rule default host hdd"); system("ceph osd pool create tier2mit_rep_hdd replicated --bulk"); system("ceph osd pool set tier2mit_rep_hdd crush_rule replicated_hdd_rule"); system("ceph osd pool set tier2mit_rep_hdd size 2"); system("ceph fs add_data_pool tier2mit tier2mit_rep_hdd"); system("cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_rep_hdd"); cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_ec_hdd then removed pool from fs, and deleted it. One of the things we tried was to put pool in failed state and then rejoin it (ceph fs fail tier2mit; ceph fs set tier2mit joinable true), here is a log from MDS, you can see where it puts itself in readonly mode. https://www.cmsaf.mit.edu/various/messages Thankyou, --Max ________________________________ From: Eugen Block <eblock@nde.ag> Sent: Monday, May 5, 2025 5:47 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state Hi, do you have logs from the MDS? And how exactly did you move the /temp directory to a different pool? That might help understand what happened. There was a thread here a couple of weeks ago how about moving contents to a different CephFS pool. Zitat von Maxim Goncharov <maxi@mit.edu>:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, it seems like you only changed the xattr, but that doesn't really move the objects into the new pool, only new objects would be created in the additional pool. There was a thread [0] on this list a few months ago, pointing to [1] how to actually migrate data to a different pool. I'm just not sure how to get out of this situation. Based on the description in the docs [2], writes to the metadata pool are blocked. So one option could be to reset the journal (take a backup first) as described in the troubleshooting guide [3]. Maybe someone else has other ideas... Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/M72ZAQPMX3S... [1] https://ewal.dev/cephfs-migrating-files-between-pools [2] https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-health-read-only [3] https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#journal-tr... Zitat von Maxim Goncharov <maxi@mit.edu>:
Hi, here is the sequence of commands I executed to change the directory to a different pools, and then exact sequence in reverse.
system("ceph osd crush rule create-replicated replicated_hdd_rule default host hdd"); system("ceph osd pool create tier2mit_rep_hdd replicated --bulk"); system("ceph osd pool set tier2mit_rep_hdd crush_rule replicated_hdd_rule"); system("ceph osd pool set tier2mit_rep_hdd size 2"); system("ceph fs add_data_pool tier2mit tier2mit_rep_hdd"); system("cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_rep_hdd");
cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_ec_hdd then removed pool from fs, and deleted it.
One of the things we tried was to put pool in failed state and then rejoin it (ceph fs fail tier2mit; ceph fs set tier2mit joinable true), here is a log from MDS, you can see where it puts itself in readonly mode.
https://www.cmsaf.mit.edu/various/messages
Thankyou, --Max
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Monday, May 5, 2025 5:47 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state
Hi,
do you have logs from the MDS? And how exactly did you move the /temp directory to a different pool? That might help understand what happened. There was a thread here a couple of weeks ago how about moving contents to a different CephFS pool.
Zitat von Maxim Goncharov <maxi@mit.edu>:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, we knew that only newly created data would belong to the new pool. Why is it a problem though? Is it possible to declare all data in the /temp directory corrupt, allow me to delete it, and move on with life? Thank you, --Max ________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, May 6, 2025 2:53 AM To: Maxim Goncharov <maxi@mit.edu> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Need help with MDS stuck in readonly state Hi, it seems like you only changed the xattr, but that doesn't really move the objects into the new pool, only new objects would be created in the additional pool. There was a thread [0] on this list a few months ago, pointing to [1] how to actually migrate data to a different pool. I'm just not sure how to get out of this situation. Based on the description in the docs [2], writes to the metadata pool are blocked. So one option could be to reset the journal (take a backup first) as described in the troubleshooting guide [3]. Maybe someone else has other ideas... Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/M72ZAQPMX3S... [1] https://ewal.dev/cephfs-migrating-files-between-pools [2] https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-health-read-only [3] https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#journal-tr... Zitat von Maxim Goncharov <maxi@mit.edu>:
Hi, here is the sequence of commands I executed to change the directory to a different pools, and then exact sequence in reverse.
system("ceph osd crush rule create-replicated replicated_hdd_rule default host hdd"); system("ceph osd pool create tier2mit_rep_hdd replicated --bulk"); system("ceph osd pool set tier2mit_rep_hdd crush_rule replicated_hdd_rule"); system("ceph osd pool set tier2mit_rep_hdd size 2"); system("ceph fs add_data_pool tier2mit tier2mit_rep_hdd"); system("cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_rep_hdd");
cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool tier2mit_ec_hdd then removed pool from fs, and deleted it.
One of the things we tried was to put pool in failed state and then rejoin it (ceph fs fail tier2mit; ceph fs set tier2mit joinable true), here is a log from MDS, you can see where it puts itself in readonly mode.
https://www.cmsaf.mit.edu/various/messages
Thankyou, --Max
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Monday, May 5, 2025 5:47 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state
Hi,
do you have logs from the MDS? And how exactly did you move the /temp directory to a different pool? That might help understand what happened. There was a thread here a couple of weeks ago how about moving contents to a different CephFS pool.
Zitat von Maxim Goncharov <maxi@mit.edu>:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Max, This could be related to the reported issue #63259. See: https://tracker.ceph.com/issues/63259#note-11 Milan -- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing On 5/1/25 08:22, Maxim Goncharov wrote:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Hi Milan, Great catch! The fix is in Reef v18.2.5 and has been merged to Squid branch, so it should be in v19.2.3, I think. No fix for EOL Quincy. @Maxim, there might be a way to recover from this situation (without going through the disaster recovery procedure) by fixing the backtraces or removing the /temp directory as a whole - if the MDS allows to do so before switching to readonly mode - and flushing the journal, but this is not something I tested. Your best move from now would be to upgrade to v18.2.7 when it's out. Note that upgrading to v18.2.5 or v18.2.6 has been highly discouraged recently due to bluestore issues. Regards, Frédéric. ----- Le 6 Mai 25, à 19:51, Milan Kupcevic milan_kupcevic@harvard.edu a écrit :
Hi Max,
This could be related to the reported issue #63259. See:
https://tracker.ceph.com/issues/63259#note-11
Milan
-- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing
On 5/1/25 08:22, Maxim Goncharov wrote:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frederic, We are running 19.2.2 squid (stable), all fixes should have been already in, I hope. I do not really understand what commands you are advising us to issue, is there web page that lists a sequence of commands we should run, or please put them in e-mail? Thank you, --Max ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, May 7, 2025 6:04 AM To: Milan Kupcevic <milan_kupcevic@harvard.edu> Cc: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state Hi Milan, Great catch! The fix is in Reef v18.2.5 and has been merged to Squid branch, so it should be in v19.2.3, I think. No fix for EOL Quincy. @Maxim, there might be a way to recover from this situation (without going through the disaster recovery procedure) by fixing the backtraces or removing the /temp directory as a whole - if the MDS allows to do so before switching to readonly mode - and flushing the journal, but this is not something I tested. Your best move from now would be to upgrade to v18.2.7 when it's out. Note that upgrading to v18.2.5 or v18.2.6 has been highly discouraged recently due to bluestore issues. Regards, Frédéric. ----- Le 6 Mai 25, à 19:51, Milan Kupcevic milan_kupcevic@harvard.edu a écrit :
Hi Max,
This could be related to the reported issue #63259. See:
https://tracker.ceph.com/issues/63259#note-11
Milan
-- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing
On 5/1/25 08:22, Maxim Goncharov wrote:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Maxim, I reproduced this issue on a v19.2.1 lab and I think I find an easy way out. Here below is what worked on my side (tested multiple times): - unmount de filesystem from any clients accessing /temp directory - restart active MDS(s) - filesystem should now be readwrite (until next flush of the journal) - umount and remount the fs on an admin node - remove any files/subdirs in the "temp" tree that's not accessible (or all of them if you can afford it) using the linux 'rm' command. If the filesystem already switched to readonly mode due to client activity, do it anyway. - flush the journal (ceph tell mds.0 flush journal). - filesystem should stay readwrite. If it switches to readonly again, then try restarting MDSs and flush again. If it still goes to readonly mode then check MDS logs for any files left over and repeat this procedure. Let me know how it goes. Best, Frédéric. ----- Le 7 Mai 25, à 12:29, Maxim Goncharov <maxi@mit.edu> a écrit :
Hi Frederic,
We are running 19.2.2 squid (stable), all fixes should have been already in, I hope. I do not really understand what commands you are advising us to issue, is there web page that lists a sequence of commands we should run, or please put them in e-mail?
Thank you, --Max
From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, May 7, 2025 6:04 AM To: Milan Kupcevic <milan_kupcevic@harvard.edu> Cc: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state Hi Milan,
Great catch! The fix is in Reef v18.2.5 and has been merged to Squid branch, so it should be in v19.2.3, I think. No fix for EOL Quincy.
@Maxim, there might be a way to recover from this situation (without going through the disaster recovery procedure) by fixing the backtraces or removing the /temp directory as a whole - if the MDS allows to do so before switching to readonly mode - and flushing the journal, but this is not something I tested.
Your best move from now would be to upgrade to v18.2.7 when it's out. Note that upgrading to v18.2.5 or v18.2.6 has been highly discouraged recently due to bluestore issues.
Regards, Frédéric.
----- Le 6 Mai 25, à 19:51, Milan Kupcevic milan_kupcevic@harvard.edu a écrit :
Hi Max,
This could be related to the reported issue #63259. See:
[ https://tracker.ceph.com/issues/63259#note-11 | https://tracker.ceph.com/issues/63259#note-11 ]
Milan
-- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing
On 5/1/25 08:22, Maxim Goncharov wrote:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frederic, Thank you for looking into it more. We indeed tried the steps that you describe. The problem was that the MDS would turn readonly before deletion of that /temp directory could run. Now we can't try anything else because yesterday we wiped everything out and bootstrapped a new cluster. Please please put somewhere in documentation in big red letters not to do what we have done so that other people would not run into the same issue. Thank you, --Max ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, May 7, 2025 9:36 AM To: Maxim Goncharov <maxi@mit.edu> Cc: Milan Kupcevic <milan_kupcevic@harvard.edu>; ceph-users <ceph-users@ceph.io> Subject: Re: Need help with MDS stuck in readonly state Maxim, I reproduced this issue on a v19.2.1 lab and I think I find an easy way out. Here below is what worked on my side (tested multiple times): - unmount de filesystem from any clients accessing /temp directory - restart active MDS(s) - filesystem should now be readwrite (until next flush of the journal) - umount and remount the fs on an admin node - remove any files/subdirs in the "temp" tree that's not accessible (or all of them if you can afford it) using the linux 'rm' command. If the filesystem already switched to readonly mode due to client activity, do it anyway. - flush the journal (ceph tell mds.0 flush journal). - filesystem should stay readwrite. If it switches to readonly again, then try restarting MDSs and flush again. If it still goes to readonly mode then check MDS logs for any files left over and repeat this procedure. Let me know how it goes. Best, Frédéric. ----- Le 7 Mai 25, à 12:29, Maxim Goncharov <maxi@mit.edu> a écrit : Hi Frederic, We are running 19.2.2 squid (stable), all fixes should have been already in, I hope. I do not really understand what commands you are advising us to issue, is there web page that lists a sequence of commands we should run, or please put them in e-mail? Thank you, --Max ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, May 7, 2025 6:04 AM To: Milan Kupcevic <milan_kupcevic@harvard.edu> Cc: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state Hi Milan, Great catch! The fix is in Reef v18.2.5 and has been merged to Squid branch, so it should be in v19.2.3, I think. No fix for EOL Quincy. @Maxim, there might be a way to recover from this situation (without going through the disaster recovery procedure) by fixing the backtraces or removing the /temp directory as a whole - if the MDS allows to do so before switching to readonly mode - and flushing the journal, but this is not something I tested. Your best move from now would be to upgrade to v18.2.7 when it's out. Note that upgrading to v18.2.5 or v18.2.6 has been highly discouraged recently due to bluestore issues. Regards, Frédéric. ----- Le 6 Mai 25, à 19:51, Milan Kupcevic milan_kupcevic@harvard.edu a écrit :
Hi Max,
This could be related to the reported issue #63259. See:
https://tracker.ceph.com/issues/63259#note-11
Milan
-- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing
On 5/1/25 08:22, Maxim Goncharov wrote:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
----- Le 7 Mai 25, à 15:45, Maxim Goncharov maxi@mit.edu a écrit :
Hi Frederic,
Thank you for looking into it more. We indeed tried the steps that you describe. The problem was that the MDS would turn readonly before deletion of that /temp directory could run.
Preventing access to the filesystem (ceph fs set cephfs refuse_client_session true) may have helped with that, I believe.
Now we can't try anything else because yesterday we wiped everything out and bootstrapped a new cluster. Please please put somewhere in documentation in big red letters not to do what we have done so that other people would not run into the same issue.
Well, this issue was discussed [1], tracked [2] and fixed [3] 8 years ago in Jewel. I don't know why this could happen again as the fix made back then is still present in the code of latest Ceph releases. Something the devs might want to investigate. Given this is a bug that's been fixed in Reef and further releases, it would seem odd to add it to the documentation, or maybe to Quincy documentation? Regards, Frédéric. [1] https://ceph-users.ceph.narkive.com/h9GQKIe4/mds-read-only-state-in-producti... [2] https://tracker.ceph.com/issues/19401 [3] https://github.com/ceph/ceph/pull/14682
Thank you, --Max
From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, May 7, 2025 9:36 AM To: Maxim Goncharov <maxi@mit.edu> Cc: Milan Kupcevic <milan_kupcevic@harvard.edu>; ceph-users <ceph-users@ceph.io> Subject: Re: Need help with MDS stuck in readonly state Maxim,
I reproduced this issue on a v19.2.1 lab and I think I find an easy way out. Here below is what worked on my side (tested multiple times):
- unmount de filesystem from any clients accessing /temp directory - restart active MDS(s) - filesystem should now be readwrite (until next flush of the journal) - umount and remount the fs on an admin node - remove any files/subdirs in the "temp" tree that's not accessible (or all of them if you can afford it) using the linux 'rm' command. If the filesystem already switched to readonly mode due to client activity, do it anyway. - flush the journal (ceph tell mds.0 flush journal). - filesystem should stay readwrite. If it switches to readonly again, then try restarting MDSs and flush again. If it still goes to readonly mode then check MDS logs for any files left over and repeat this procedure.
Let me know how it goes.
Best, Frédéric.
----- Le 7 Mai 25, à 12:29, Maxim Goncharov <maxi@mit.edu> a écrit :
Hi Frederic,
We are running 19.2.2 squid (stable), all fixes should have been already in, I hope. I do not really understand what commands you are advising us to issue, is there web page that lists a sequence of commands we should run, or please put them in e-mail?
Thank you, --Max
From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, May 7, 2025 6:04 AM To: Milan Kupcevic <milan_kupcevic@harvard.edu> Cc: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state Hi Milan,
Great catch! The fix is in Reef v18.2.5 and has been merged to Squid branch, so it should be in v19.2.3, I think. No fix for EOL Quincy.
@Maxim, there might be a way to recover from this situation (without going through the disaster recovery procedure) by fixing the backtraces or removing the /temp directory as a whole - if the MDS allows to do so before switching to readonly mode - and flushing the journal, but this is not something I tested.
Your best move from now would be to upgrade to v18.2.7 when it's out. Note that upgrading to v18.2.5 or v18.2.6 has been highly discouraged recently due to bluestore issues.
Regards, Frédéric.
----- Le 6 Mai 25, à 19:51, Milan Kupcevic milan_kupcevic@harvard.edu a écrit :
Hi Max,
This could be related to the reported issue #63259. See:
[ https://tracker.ceph.com/issues/63259#note-11 | https://tracker.ceph.com/issues/63259#note-11 ]
Milan
-- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing
On 5/1/25 08:22, Maxim Goncharov wrote:
Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
Frédéric Nass
-
Maxim Goncharov
-
Milan Kupcevic