6 Jan
2021
6 Jan
'21
6:32 p.m.
Hi everybody
Does somebody had experience with important performance degradations during
a pool deletion?
We are asking because we are going to delete a 370 TiB with 120 M objects and have never
done this in the past.
The pool is using erasure coding 8+2 on nvme ssd's with rocksdb/wal on nvme optane
disks.
Openstack VM's are running on the other rbd pools.
Thank you in advance for your feedback!
Cheers
Francois
PS:
this is no option, as it take about 100 year to complete ;-) :
rados -p ch-zh1-az1.rgw.buckets.data ls | while read i; do rados -p
ch-zh1-az1.rgw.buckets.data rm "$i"; done
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer(a)everyware.ch
web: http://www.everyware.ch
6 Jan
6 Jan
10:17 p.m.
Hi,
one of our customers just recently tried that in a non-production
cluster (Nautilus) and it was horrible. They wanted to test a couple
of failure resistency scenarios and needed to purge old data (I
believe around 200 TB or so), we ended up recreating the OSDs instead
of waiting for the deletion to finish. They have HDDs with separate
rocksDB on SSDs (ratio 1:5) and the SSDs were completely saturated
according to iostat.
It's a known issue (see [1], [2]), I'll just quote Igor:
there are two issues with massive data removals
1) It might take pretty long time to complete since it performs 15
object removals per second per PG/OSD. Hence it might takes days
2)RocksDB starts to perform poorly from performance point of view
after (or even during) such removals
As far as I can see the backports for the fix are pending for Mimic,
Nautilus and Octopus.
Regards,
Eugen
[1] https://tracker.ceph.com/issues/45765
[2] https://tracker.ceph.com/issues/47044
Zitat von Scheurer François <francois.scheurer(a)everyware.ch>ch>:
Hi everybody
Does somebody had experience with important performance degradations during
a pool deletion?
We are asking because we are going to delete a 370 TiB with 120 M
objects and have never done this in the past.
The pool is using erasure coding 8+2 on nvme ssd's with rocksdb/wal
on nvme optane disks.
Openstack VM's are running on the other rbd pools.
Thank you in advance for your feedback!
Cheers
Francois
PS:
this is no option, as it take about 100 year to complete ;-) :
rados -p ch-zh1-az1.rgw.buckets.data ls | while read i; do rados -p
ch-zh1-az1.rgw.buckets.data rm "$i"; done
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer(a)everyware.ch
web: http://www.everyware.ch
11 Jan
11 Jan
12:22 p.m.
Many Thanks Eugen for you experience sharing!! very useful information
(I thought I was maybe to paranoid.. thanks god I asked the maillist first!)
Cheers
Francois
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer(a)everyware.ch
web: http://www.everyware.ch
________________________________
From: Eugen Block <eblock(a)nde.ag>
Sent: Wednesday, January 6, 2021 8:17 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: performance impact by pool deletion?
Hi,
one of our customers just recently tried that in a non-production
cluster (Nautilus) and it was horrible. They wanted to test a couple
of failure resistency scenarios and needed to purge old data (I
believe around 200 TB or so), we ended up recreating the OSDs instead
of waiting for the deletion to finish. They have HDDs with separate
rocksDB on SSDs (ratio 1:5) and the SSDs were completely saturated
according to iostat.
It's a known issue (see [1], [2]), I'll just quote Igor:
there are two issues with massive data removals
1) It might take pretty long time to complete since it performs 15
object removals per second per PG/OSD. Hence it might takes days
2)RocksDB starts to perform poorly from performance point of view
after (or even during) such removals
As far as I can see the backports for the fix are pending for Mimic,
Nautilus and Octopus.
Regards,
Eugen
[1] https://tracker.ceph.com/issues/45765
[2] https://tracker.ceph.com/issues/47044
Zitat von Scheurer François <francois.scheurer(a)everyware.ch>ch>:
Hi everybody
Does somebody had experience with important performance degradations during
a pool deletion?
We are asking because we are going to delete a 370 TiB with 120 M
objects and have never done this in the past.
The pool is using erasure coding 8+2 on nvme ssd's with rocksdb/wal
on nvme optane disks.
Openstack VM's are running on the other rbd pools.
Thank you in advance for your feedback!
Cheers
Francois
PS:
this is no option, as it take about 100 year to complete ;-) :
rados -p ch-zh1-az1.rgw.buckets.data ls | while read i; do rados -p
ch-zh1-az1.rgw.buckets.data rm "$i"; done
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer(a)everyware.ch
web: http://www.everyware.ch
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9 Jan
9 Jan
10:15 a.m.
I deleted a 240TB rgw pool a few weeks ago, caused a huge slowdown. Lucky it wasn't a
important cluster otherwise it would've taken it down for a week.
From: Scheurer François <francois.scheurer(a)everyware.ch>
Sent: Wednesday, 6 January 2021 11:32 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] performance impact by pool deletion?
Hi everybody
Does somebody had experience with important performance degradations during
a pool deletion?
We are asking because we are going to delete a 370 TiB with 120 M objects and have never
done this in the past.
The pool is using erasure coding 8+2 on nvme ssd's with rocksdb/wal on nvme optane
disks.
Openstack VM's are running on the other rbd pools.
Thank you in advance for your feedback!
Cheers
Francois
PS:
this is no option, as it take about 100 year to complete ;-) :
rados -p ch-zh1-az1.rgw.buckets.data ls | while read i; do rados -p
ch-zh1-az1.rgw.buckets.data rm "$i"; done
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@everyware.ch<mailto:francois.scheurer@everyware.ch>
web: http://www.everyware.ch
This e-mail is intended solely for the benefit of the addressee(s) and any other named
recipient. It is confidential and may contain legally privileged or confidential
information. If you are not the recipient, any use, distribution, disclosure or copying of
this e-mail is prohibited. The confidentiality and legal privilege attached to this
communication is not waived or lost by reason of the mistaken transmission or delivery to
you. If you have received this e-mail in error, please notify us immediately.
2:10 p.m.
Hi all,
I deleted a ceph fs data pool (EC 8+2) of size 240TB with about 150M objects and it had no
observable impact at all. Client IO and admin operations worked just like before. In fact,
I was surprised how fast it went and how fast the capacity became available again. It was
probably just a few days, but don't remember the exact times any more.
My version back then was mimic 13.2.8. All OSDs had collocated WAL/DB, everything on
spindle. My impression from reports in this list is, that this started becoming a problem
with changes made in nautilus.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Glen Baars <glen(a)onsitecomputers.com.au>
Sent: 09 January 2021 08:15:51
To: Scheurer François; ceph-users(a)ceph.io
Subject: [ceph-users] Re: performance impact by pool deletion?
I deleted a 240TB rgw pool a few weeks ago, caused a huge slowdown. Lucky it wasn't a
important cluster otherwise it would've taken it down for a week.
From: Scheurer François <francois.scheurer(a)everyware.ch>
Sent: Wednesday, 6 January 2021 11:32 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] performance impact by pool deletion?
Hi everybody
Does somebody had experience with important performance degradations during
a pool deletion?
We are asking because we are going to delete a 370 TiB with 120 M objects and have never
done this in the past.
The pool is using erasure coding 8+2 on nvme ssd's with rocksdb/wal on nvme optane
disks.
Openstack VM's are running on the other rbd pools.
Thank you in advance for your feedback!
Cheers
Francois
PS:
this is no option, as it take about 100 year to complete ;-) :
rados -p ch-zh1-az1.rgw.buckets.data ls | while read i; do rados -p
ch-zh1-az1.rgw.buckets.data rm "$i"; done
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@everyware.ch<mailto:francois.scheurer@everyware.ch>
web: http://www.everyware.ch
This e-mail is intended solely for the benefit of the addressee(s) and any other named
recipient. It is confidential and may contain legally privileged or confidential
information. If you are not the recipient, any use, distribution, disclosure or copying of
this e-mail is prohibited. The confidentiality and legal privilege attached to this
communication is not waived or lost by reason of the mistaken transmission or delivery to
you. If you have received this e-mail in error, please notify us immediately.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11 Jan
11 Jan
12:17 p.m.
Thank you Glen and Frank for your experience sharing!
Cheers
Francois
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer(a)everyware.ch
web: http://www.everyware.ch
________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: Saturday, January 9, 2021 12:10 PM
To: Glen Baars; Scheurer François; ceph-users(a)ceph.io
Subject: Re: performance impact by pool deletion?
Hi all,
I deleted a ceph fs data pool (EC 8+2) of size 240TB with about 150M objects and it had no
observable impact at all. Client IO and admin operations worked just like before. In fact,
I was surprised how fast it went and how fast the capacity became available again. It was
probably just a few days, but don't remember the exact times any more.
My version back then was mimic 13.2.8. All OSDs had collocated WAL/DB, everything on
spindle. My impression from reports in this list is, that this started becoming a problem
with changes made in nautilus.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Glen Baars <glen(a)onsitecomputers.com.au>
Sent: 09 January 2021 08:15:51
To: Scheurer François; ceph-users(a)ceph.io
Subject: [ceph-users] Re: performance impact by pool deletion?
I deleted a 240TB rgw pool a few weeks ago, caused a huge slowdown. Lucky it wasn't a
important cluster otherwise it would've taken it down for a week.
From: Scheurer François <francois.scheurer(a)everyware.ch>
Sent: Wednesday, 6 January 2021 11:32 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] performance impact by pool deletion?
Hi everybody
Does somebody had experience with important performance degradations during
a pool deletion?
We are asking because we are going to delete a 370 TiB with 120 M objects and have never
done this in the past.
The pool is using erasure coding 8+2 on nvme ssd's with rocksdb/wal on nvme optane
disks.
Openstack VM's are running on the other rbd pools.
Thank you in advance for your feedback!
Cheers
Francois
PS:
this is no option, as it take about 100 year to complete ;-) :
rados -p ch-zh1-az1.rgw.buckets.data ls | while read i; do rados -p
ch-zh1-az1.rgw.buckets.data rm "$i"; done
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@everyware.ch<mailto:francois.scheurer@everyware.ch>
web: http://www.everyware.ch
This e-mail is intended solely for the benefit of the addressee(s) and any other named
recipient. It is confidential and may contain legally privileged or confidential
information. If you are not the recipient, any use, distribution, disclosure or copying of
this e-mail is prohibited. The confidentiality and legal privilege attached to this
communication is not waived or lost by reason of the mistaken transmission or delivery to
you. If you have received this e-mail in error, please notify us immediately.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1209
days inactive
1214
days old
5 comments
4 participants
participants (4)
-
Eugen Block -
Frank Schilder -
Glen Baars -
Scheurer François