How to remove one of two filesystems
Hello, I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds. mds1 is managing fs1 mds2 manages fs2 mds3 is standby I want to completely remove fs1. It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it and then delete the data and metadata pools with ceph osd pool delete but in many threads I noticed that you must shutdown the mds before running ceph fs rm. Is it still the case ? What happens in my configuration (I have 2 fs) ? If I stop mds1, the mds3 will take the management. If I stop mds3 what will mds2 do (try to manage the 2 fs or continue only with fs2) ? Thanks for your advices. F.
Hi,
It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it and then delete the data and metadata pools with ceph osd pool delete but in many threads I noticed that you must shutdown the mds before running ceph fs rm. Is it still the case ?
Yes.
What happens in my configuration (I have 2 fs) ? If I stop mds1, the mds3 will take the management. If I stop mds3 what will mds2 do (try to manage the 2 fs or continue only with fs2) ?
First stop the standby mds3 (temporarily no standby mds available), this shouldn't have any impact on the two active mds, both will only serve their assigned fs. Then stop mds1 and remove fs1. Ceph will get into WARN state because of missing standby daemons. Start mds3 to get back a standby mds, then you can clean up the pools. Zitat von Francois Legrand <fleg@lpnhe.in2p3.fr>:
Hello, I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds. mds1 is managing fs1 mds2 manages fs2 mds3 is standby
I want to completely remove fs1. It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it and then delete the data and metadata pools with ceph osd pool delete but in many threads I noticed that you must shutdown the mds before running ceph fs rm. Is it still the case ? What happens in my configuration (I have 2 fs) ? If I stop mds1, the mds3 will take the management. If I stop mds3 what will mds2 do (try to manage the 2 fs or continue only with fs2) ? Thanks for your advices. F. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Use ceph fs set <fs_name> down true after this all mdses of fs fs_name will become standbys. Now you can cleanly remove everything. Wait for the fs to be shown as down in ceph status, the command above is non-blocking but the shutdown takes a long time. Try to disconnect all clients first. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 22 June 2020 15:56:56 To: ceph-users Subject: [ceph-users] How to remove one of two filesystems Hello, I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds. mds1 is managing fs1 mds2 manages fs2 mds3 is standby I want to completely remove fs1. It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it and then delete the data and metadata pools with ceph osd pool delete but in many threads I noticed that you must shutdown the mds before running ceph fs rm. Is it still the case ? What happens in my configuration (I have 2 fs) ? If I stop mds1, the mds3 will take the management. If I stop mds3 what will mds2 do (try to manage the 2 fs or continue only with fs2) ? Thanks for your advices. F. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph fs set <fs_name> down true
That's much better! Zitat von Frank Schilder <frans@dtu.dk>:
Use
ceph fs set <fs_name> down true
after this all mdses of fs fs_name will become standbys. Now you can cleanly remove everything.
Wait for the fs to be shown as down in ceph status, the command above is non-blocking but the shutdown takes a long time. Try to disconnect all clients first.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 22 June 2020 15:56:56 To: ceph-users Subject: [ceph-users] How to remove one of two filesystems
Hello, I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds. mds1 is managing fs1 mds2 manages fs2 mds3 is standby
I want to completely remove fs1. It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it and then delete the data and metadata pools with ceph osd pool delete but in many threads I noticed that you must shutdown the mds before running ceph fs rm. Is it still the case ? What happens in my configuration (I have 2 fs) ? If I stop mds1, the mds3 will take the management. If I stop mds3 what will mds2 do (try to manage the 2 fs or continue only with fs2) ? Thanks for your advices. F. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, Jun 22, 2020 at 7:29 AM Frank Schilder <frans@dtu.dk> wrote:
Use
ceph fs set <fs_name> down true
after this all mdses of fs fs_name will become standbys. Now you can cleanly remove everything.
Wait for the fs to be shown as down in ceph status, the command above is non-blocking but the shutdown takes a long time. Try to disconnect all clients first.
If you're planning to delete the file system, it is faster to just do: ceph fs fail <fs_name> which will remove all the MDS and mark the cluster as not joinable. See also: https://docs.ceph.com/docs/master/cephfs/administration/#taking-the-cluster-... -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Thanks a lot. It works. I could delete the filesystem and remove the pools (data and metadata). But now I am facing another problem which is that the removal of the pools seems to take a incredible time to free the space (the pool I deleted was about 100TB and in 36h I got back only 10TB). In the meantime, the cluster is extremely slow (a rbd extract takes ~30 mn for a 9 GB image and writing 10MB in cephfs takes half a minute !!) which makes the cluster almost unusable. It seems that the removal of deleted pg is done by deep-scrubs according to https://medium.com/opsops/a-very-slow-pool-removal-7089e4ac8301 But I couldn't find a way to speedup the process or to get back the cluster to a decent reactivity ? Do you have a suggestion ? F. Le 22/06/2020 à 16:40, Patrick Donnelly a écrit :
On Mon, Jun 22, 2020 at 7:29 AM Frank Schilder <frans@dtu.dk> wrote:
Use
ceph fs set <fs_name> down true
after this all mdses of fs fs_name will become standbys. Now you can cleanly remove everything.
Wait for the fs to be shown as down in ceph status, the command above is non-blocking but the shutdown takes a long time. Try to disconnect all clients first. If you're planning to delete the file system, it is faster to just do:
ceph fs fail <fs_name>
which will remove all the MDS and mark the cluster as not joinable. See also: https://docs.ceph.com/docs/master/cephfs/administration/#taking-the-cluster-...
Hi Francois, I have seen reports of poor performance from Nautilus onwards and you might be hit by this. This might require a ticket. There is a hypothesis that a regression occurred that affects the cluster's ability to run background operations properly. What you observe should not happen and I didn't see any of this on mimic when removing a 120TB file system. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 24 June 2020 00:25:03 To: Patrick Donnelly; Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] Re: How to remove one of two filesystems Thanks a lot. It works. I could delete the filesystem and remove the pools (data and metadata). But now I am facing another problem which is that the removal of the pools seems to take a incredible time to free the space (the pool I deleted was about 100TB and in 36h I got back only 10TB). In the meantime, the cluster is extremely slow (a rbd extract takes ~30 mn for a 9 GB image and writing 10MB in cephfs takes half a minute !!) which makes the cluster almost unusable. It seems that the removal of deleted pg is done by deep-scrubs according to https://medium.com/opsops/a-very-slow-pool-removal-7089e4ac8301 But I couldn't find a way to speedup the process or to get back the cluster to a decent reactivity ? Do you have a suggestion ? F. Le 22/06/2020 à 16:40, Patrick Donnelly a écrit : On Mon, Jun 22, 2020 at 7:29 AM Frank Schilder <frans@dtu.dk><mailto:frans@dtu.dk> wrote: Use ceph fs set <fs_name> down true after this all mdses of fs fs_name will become standbys. Now you can cleanly remove everything. Wait for the fs to be shown as down in ceph status, the command above is non-blocking but the shutdown takes a long time. Try to disconnect all clients first. If you're planning to delete the file system, it is faster to just do: ceph fs fail <fs_name> which will remove all the MDS and mark the cluster as not joinable. See also: https://docs.ceph.com/docs/master/cephfs/administration/#taking-the-cluster-...
Here is a thread that seems most relevant: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/W4M5XQRDBLXF... I do not see this issue on mimic, but it seems to be a problem from nautilus onwards. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 24 June 2020 09:34:24 To: Patrick Donnelly; fleg@lpnhe.in2p3.fr Cc: ceph-users Subject: [ceph-users] Re: How to remove one of two filesystems Hi Francois, I have seen reports of poor performance from Nautilus onwards and you might be hit by this. This might require a ticket. There is a hypothesis that a regression occurred that affects the cluster's ability to run background operations properly. What you observe should not happen and I didn't see any of this on mimic when removing a 120TB file system. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 24 June 2020 00:25:03 To: Patrick Donnelly; Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] Re: How to remove one of two filesystems Thanks a lot. It works. I could delete the filesystem and remove the pools (data and metadata). But now I am facing another problem which is that the removal of the pools seems to take a incredible time to free the space (the pool I deleted was about 100TB and in 36h I got back only 10TB). In the meantime, the cluster is extremely slow (a rbd extract takes ~30 mn for a 9 GB image and writing 10MB in cephfs takes half a minute !!) which makes the cluster almost unusable. It seems that the removal of deleted pg is done by deep-scrubs according to https://medium.com/opsops/a-very-slow-pool-removal-7089e4ac8301 But I couldn't find a way to speedup the process or to get back the cluster to a decent reactivity ? Do you have a suggestion ? F. Le 22/06/2020 à 16:40, Patrick Donnelly a écrit : On Mon, Jun 22, 2020 at 7:29 AM Frank Schilder <frans@dtu.dk><mailto:frans@dtu.dk> wrote: Use ceph fs set <fs_name> down true after this all mdses of fs fs_name will become standbys. Now you can cleanly remove everything. Wait for the fs to be shown as down in ceph status, the command above is non-blocking but the shutdown takes a long time. Try to disconnect all clients first. If you're planning to delete the file system, it is faster to just do: ceph fs fail <fs_name> which will remove all the MDS and mark the cluster as not joinable. See also: https://docs.ceph.com/docs/master/cephfs/administration/#taking-the-cluster-... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
Francois Legrand
-
Frank Schilder
-
Patrick Donnelly