22 Jun
2020
22 Jun
'20
6:26 p.m.
Hello,
I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds.
mds1 is managing fs1
mds2 manages fs2
mds3 is standby
I want to completely remove fs1.
It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it
and then delete the data and metadata pools with ceph osd pool delete
but in many threads I noticed that you must shutdown the mds before
running ceph fs rm.
Is it still the case ?
What happens in my configuration (I have 2 fs) ? If I stop mds1, the
mds3 will take the management. If I stop mds3 what will mds2 do (try to
manage the 2 fs or continue only with fs2) ?
Thanks for your advices.
F.
22 Jun
22 Jun
6:57 p.m.
Hi,
It seems that the command to use is ceph fs rm fs1
--yes-i-really-mean-it
and then delete the data and metadata pools with ceph osd pool delete
but in many threads I noticed that you must shutdown the mds before
running ceph fs rm.
Is it still the case ?
Yes.
What happens in my configuration (I have 2 fs) ? If I
stop mds1, the
mds3 will take the management. If I stop mds3 what will mds2 do (try
to manage the 2 fs or continue only with fs2) ?
First stop the standby mds3 (temporarily no standby mds available),
this shouldn't have any impact on the two active mds, both will only
serve their assigned fs. Then stop mds1 and remove fs1. Ceph will get
into WARN state because of missing standby daemons. Start mds3 to get
back a standby mds, then you can clean up the pools.
Zitat von Francois Legrand <fleg(a)lpnhe.in2p3.fr>fr>:
> Hello,
> I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds.
> mds1 is managing fs1
> mds2 manages fs2
> mds3 is standby
>
> I want to completely remove fs1.
> It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it
> and then delete the data and metadata pools with ceph osd pool delete
> but in many threads I noticed that you must shutdown the mds before
> running ceph fs rm.
> Is it still the case ?
What happens in my configuration (I have 2 fs) ? If I
stop mds1, the
mds3 will take the management. If I stop mds3 what will mds2 do (try
to manage the 2 fs or continue only with fs2) ?
> Thanks for your advices.
> F.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:57 p.m.
Use
ceph fs set <fs_name> down true
after this all mdses of fs fs_name will become standbys. Now you can cleanly remove
everything.
Wait for the fs to be shown as down in ceph status, the command above is non-blocking but
the shutdown takes a long time. Try to disconnect all clients first.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Francois Legrand <fleg(a)lpnhe.in2p3.fr>
Sent: 22 June 2020 15:56:56
To: ceph-users
Subject: [ceph-users] How to remove one of two filesystems
Hello,
I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds.
mds1 is managing fs1
mds2 manages fs2
mds3 is standby
I want to completely remove fs1.
It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it
and then delete the data and metadata pools with ceph osd pool delete
but in many threads I noticed that you must shutdown the mds before
running ceph fs rm.
Is it still the case ?
What happens in my configuration (I have 2 fs) ? If I stop mds1, the
mds3 will take the management. If I stop mds3 what will mds2 do (try to
manage the 2 fs or continue only with fs2) ?
Thanks for your advices.
F.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:05 p.m.
ceph fs set <fs_name> down true
That's much better!
Zitat von Frank Schilder <frans(a)dtu.dk>dk>:
> Use
>
ceph fs set <fs_name> down true
>
> after this all mdses of fs fs_name will become standbys. Now you can
> cleanly remove everything.
>
> Wait for the fs to be shown as down in ceph status, the command
> above is non-blocking but the shutdown takes a long time. Try to
> disconnect all clients first.
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Francois Legrand <fleg(a)lpnhe.in2p3.fr>
> Sent: 22 June 2020 15:56:56
> To: ceph-users
> Subject: [ceph-users] How to remove one of two filesystems
>
> Hello,
> I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds.
> mds1 is managing fs1
> mds2 manages fs2
> mds3 is standby
>
> I want to completely remove fs1.
> It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it
> and then delete the data and metadata pools with ceph osd pool delete
> but in many threads I noticed that you must shutdown the mds before
> running ceph fs rm.
> Is it still the case ?
> What happens in my configuration (I have 2 fs) ? If I stop mds1, the
> mds3 will take the management. If I stop mds3 what will mds2 do (try to
> manage the 2 fs or continue only with fs2) ?
> Thanks for your advices.
> F.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:10 p.m.
On Mon, Jun 22, 2020 at 7:29 AM Frank Schilder <frans(a)dtu.dk> wrote:
Use
ceph fs set <fs_name> down true
after this all mdses of fs fs_name will become standbys. Now you can cleanly remove
everything.
Wait for the fs to be shown as down in ceph status, the command above is non-blocking but
the shutdown takes a long time. Try to disconnect all clients first.
If you're planning to delete the file system, it is faster to just do:
ceph fs fail <fs_name>
which will remove all the MDS and mark the cluster as not joinable.
See also:
https://docs.ceph.com/docs/master/cephfs/administration/#taking-the-cluster…
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
24 Jun
24 Jun
2:55 a.m.
Thanks a lot. It works.
I could delete the filesystem and remove the pools (data and metadata).
But now I am facing another problem which is that the removal of the
pools seems to take a incredible time to free the space (the pool I
deleted was about 100TB and in 36h I got back only 10TB). In the
meantime, the cluster is extremely slow (a rbd extract takes ~30 mn for
a 9 GB image and writing 10MB in cephfs takes half a minute !!) which
makes the cluster almost unusable.
It seems that the removal of deleted pg is done by deep-scrubs according
to https://medium.com/opsops/a-very-slow-pool-removal-7089e4ac8301
But I couldn't find a way to speedup the process or to get back the
cluster to a decent reactivity ?
Do you have a suggestion ?
F.
Le 22/06/2020 à 16:40, Patrick Donnelly a écrit :
On Mon, Jun 22, 2020 at 7:29 AM Frank Schilder
<frans(a)dtu.dk> wrote:
Use
ceph fs set <fs_name> down true
after this all mdses of fs fs_name will become standbys. Now you can cleanly remove
everything.
Wait for the fs to be shown as down in ceph status, the command above is non-blocking but
the shutdown takes a long time. Try to disconnect all clients first.
If you're
planning to delete the file system, it is faster to just do:
ceph fs fail <fs_name>
which will remove all the MDS and mark the cluster as not joinable.
See also:
https://docs.ceph.com/docs/master/cephfs/administration/#taking-the-cluster…
12:04 p.m.
Hi Francois,
I have seen reports of poor performance from Nautilus onwards and you might be hit by
this. This might require a ticket. There is a hypothesis that a regression occurred that
affects the cluster's ability to run background operations properly.
What you observe should not happen and I didn't see any of this on mimic when removing
a 120TB file system.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Francois Legrand <fleg(a)lpnhe.in2p3.fr>
Sent: 24 June 2020 00:25:03
To: Patrick Donnelly; Frank Schilder
Cc: ceph-users
Subject: Re: [ceph-users] Re: How to remove one of two filesystems
Thanks a lot. It works.
I could delete the filesystem and remove the pools (data and metadata).
But now I am facing another problem which is that the removal of the pools seems to take a
incredible time to free the space (the pool I deleted was about 100TB and in 36h I got
back only 10TB). In the meantime, the cluster is extremely slow (a rbd extract takes ~30
mn for a 9 GB image and writing 10MB in cephfs takes half a minute !!) which makes the
cluster almost unusable.
It seems that the removal of deleted pg is done by deep-scrubs according to
https://medium.com/opsops/a-very-slow-pool-removal-7089e4ac8301
But I couldn't find a way to speedup the process or to get back the cluster to a
decent reactivity ?
Do you have a suggestion ?
F.
Le 22/06/2020 à 16:40, Patrick Donnelly a écrit :
On Mon, Jun 22, 2020 at 7:29 AM Frank Schilder
<frans@dtu.dk><mailto:frans@dtu.dk> wrote:
Use
ceph fs set <fs_name> down true
after this all mdses of fs fs_name will become standbys. Now you can cleanly remove
everything.
Wait for the fs to be shown as down in ceph status, the command above is non-blocking but
the shutdown takes a long time. Try to disconnect all clients first.
If you're planning to delete the file system, it is faster to just do:
ceph fs fail <fs_name>
which will remove all the MDS and mark the cluster as not joinable.
See also:
https://docs.ceph.com/docs/master/cephfs/administration/#taking-the-cluster…
1:12 p.m.
Here is a thread that seems most relevant:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/W4M5XQRDBLX…
I do not see this issue on mimic, but it seems to be a problem from nautilus onwards.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: 24 June 2020 09:34:24
To: Patrick Donnelly; fleg(a)lpnhe.in2p3.fr
Cc: ceph-users
Subject: [ceph-users] Re: How to remove one of two filesystems
Hi Francois,
I have seen reports of poor performance from Nautilus onwards and you might be hit by
this. This might require a ticket. There is a hypothesis that a regression occurred that
affects the cluster's ability to run background operations properly.
What you observe should not happen and I didn't see any of this on mimic when removing
a 120TB file system.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Francois Legrand <fleg(a)lpnhe.in2p3.fr>
Sent: 24 June 2020 00:25:03
To: Patrick Donnelly; Frank Schilder
Cc: ceph-users
Subject: Re: [ceph-users] Re: How to remove one of two filesystems
Thanks a lot. It works.
I could delete the filesystem and remove the pools (data and metadata).
But now I am facing another problem which is that the removal of the pools seems to take a
incredible time to free the space (the pool I deleted was about 100TB and in 36h I got
back only 10TB). In the meantime, the cluster is extremely slow (a rbd extract takes ~30
mn for a 9 GB image and writing 10MB in cephfs takes half a minute !!) which makes the
cluster almost unusable.
It seems that the removal of deleted pg is done by deep-scrubs according to
https://medium.com/opsops/a-very-slow-pool-removal-7089e4ac8301
But I couldn't find a way to speedup the process or to get back the cluster to a
decent reactivity ?
Do you have a suggestion ?
F.
Le 22/06/2020 à 16:40, Patrick Donnelly a écrit :
On Mon, Jun 22, 2020 at 7:29 AM Frank Schilder
<frans@dtu.dk><mailto:frans@dtu.dk> wrote:
Use
ceph fs set <fs_name> down true
after this all mdses of fs fs_name will become standbys. Now you can cleanly remove
everything.
Wait for the fs to be shown as down in ceph status, the command above is non-blocking but
the shutdown takes a long time. Try to disconnect all clients first.
If you're planning to delete the file system, it is faster to just do:
ceph fs fail <fs_name>
which will remove all the MDS and mark the cluster as not joinable.
See also:
https://docs.ceph.com/docs/master/cephfs/administration/#taking-the-cluster…
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1427
days inactive
1429
days old
7 comments
4 participants
participants (4)
-
Eugen Block -
Francois Legrand -
Frank Schilder -
Patrick Donnelly