Snaptriming speed degrade with pg increase
Hi, When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04) -osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2 We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours. Is there any good solution that we are missing to fix this? On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still. Thank you
Dear Istvan, The first thing that stands out: Ubuntu 20.04 (EOL in April 2025) and Ceph v15 Octopus (EOL since 2022) Is there a possibility to upgrade these things? Best regards Gunnar --- Original Nachricht --- Betreff: [ceph-users] Snaptriming speed degrade with pg increase Von: "Szabo, Istvan (Agoda)" An: "Ceph Users" Datum: 29-11-2024 3:30 Hi, When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04) -osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2 We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours. Is there any good solution that we are missing to fix this? On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still. Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Let's say yes if that is the issue. Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ From: Bandelow, Gunnar Sent: Friday, November 29, 2024 1:47 PM To: Szabo, Istvan (Agoda); Ceph Users Subject: Re: [ceph-users] Snaptriming speed degrade with pg increase Dear Istvan, The first thing that stands out: Ubuntu 20.04 (EOL in April 2025) and Ceph v15 Octopus (EOL since 2022) Is there a possibility to upgrade these things? Best regards Gunnar --- Original Nachricht --- Betreff: [ceph-users] Snaptriming speed degrade with pg increase Von: "Szabo, Istvan (Agoda)" <istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>> An: "Ceph Users" <ceph-users@ceph.io<mailto:ceph-users@ceph.io>> Datum: 29-11-2024 3:30 Hi, When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04) -osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2 We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours. Is there any good solution that we are missing to fix this? On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still. Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hi Istvan, Did the PG split involved using more OSDs than before? If so then increasing these values (apart from the sleep) should not have a negative impact on clients I/O compared to before the split and should accelerate the whole process. Did you reshard the buckets as discussed in the other thread? Regards, Frédéric. ----- Le 29 Nov 24, à 3:30, Istvan Szabo, Agoda Istvan.Szabo@agoda.com a écrit :
Hi,
When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04)
-osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2
We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours.
Is there any good solution that we are missing to fix this?
On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still.
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Increased from 9 servers to 11 so let's say 20% capacity and performance added. This is a different cluster purely rbd. (For the other topic can't be resharded because in multisite it will disappear all the data disappear on remote site, need to create new bucket and migrate data first to a higher sharded bucket). Istvan ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Friday, November 29, 2024 4:58:52 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Snaptriming speed degrade with pg increase Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Hi Istvan, Did the PG split involved using more OSDs than before? If so then increasing these values (apart from the sleep) should not have a negative impact on clients I/O compared to before the split and should accelerate the whole process. Did you reshard the buckets as discussed in the other thread? Regards, Frédéric. ----- Le 29 Nov 24, à 3:30, Istvan Szabo, Agoda Istvan.Szabo@agoda.com a écrit :
Hi,
When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04)
-osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2
We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours.
Is there any good solution that we are missing to fix this?
On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still.
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
----- Le 29 Nov 24, à 11:11, Istvan Szabo, Agoda <Istvan.Szabo@agoda.com> a écrit :
Increased from 9 servers to 11 so let's say 20% capacity and performance added.
This is a different cluster purely rbd.
I see, so big objects. You might want to increase osd_max_trimming_pgs and eventually osd_pg_max_concurrent_snap_trims and see how it goes.
(For the other topic can't be resharded because in multisite it will disappear all the data disappear on remote site, need to create new bucket and migrate data first to a higher sharded bucket).
Hum... You have fallen significantly behind on Ceph versions, which must be hindering you in many operational tasks today. Another option would be to catch up and reshard into a recent version in multi-site mode. Frédéric.
Istvan
From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Friday, November 29, 2024 4:58:52 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Snaptriming speed degrade with pg increase Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi Istvan,
Did the PG split involved using more OSDs than before? If so then increasing these values (apart from the sleep) should not have a negative impact on clients I/O compared to before the split and should accelerate the whole process.
Did you reshard the buckets as discussed in the other thread?
Regards, Frédéric.
----- Le 29 Nov 24, à 3:30, Istvan Szabo, Agoda Istvan.Szabo@agoda.com a écrit :
Hi,
When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04)
-osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2
We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours.
Is there any good solution that we are missing to fix this?
On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still.
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The reshard topic is running on quincy 17.2.7, but tested today the reshard, objects gone. Istvan ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Friday, November 29, 2024 5:17:27 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Snaptriming speed degrade with pg increase Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ ----- Le 29 Nov 24, à 11:11, Istvan Szabo, Agoda <Istvan.Szabo@agoda.com> a écrit : Increased from 9 servers to 11 so let's say 20% capacity and performance added. This is a different cluster purely rbd. I see, so big objects. You might want to increase osd_max_trimming_pgs and eventually osd_pg_max_concurrent_snap_trims and see how it goes. (For the other topic can't be resharded because in multisite it will disappear all the data disappear on remote site, need to create new bucket and migrate data first to a higher sharded bucket). Hum... You have fallen significantly behind on Ceph versions, which must be hindering you in many operational tasks today. Another option would be to catch up and reshard into a recent version in multi-site mode. Frédéric. Istvan ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Friday, November 29, 2024 4:58:52 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Snaptriming speed degrade with pg increase Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Hi Istvan, Did the PG split involved using more OSDs than before? If so then increasing these values (apart from the sleep) should not have a negative impact on clients I/O compared to before the split and should accelerate the whole process. Did you reshard the buckets as discussed in the other thread? Regards, Frédéric. ----- Le 29 Nov 24, à 3:30, Istvan Szabo, Agoda Istvan.Szabo@agoda.com a écrit :
Hi,
When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04)
-osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2
We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours.
Is there any good solution that we are missing to fix this?
On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still.
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Istvan, Yeah, it's a been a known bug. Thus my previous recommendation to first upgrade your cluster a more recent version of Ceph and then only reshard the mult-site synced bucket. Cheers, Frédéric. ----- Le 29 Nov 24, à 13:11, Istvan Szabo, Agoda <Istvan.Szabo@agoda.com> a écrit :
The reshard topic is running on quincy 17.2.7, but tested today the reshard, objects gone.
Istvan
From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Friday, November 29, 2024 5:17:27 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Snaptriming speed degrade with pg increase Email received from the internet. If in doubt, don't click any link nor open any attachment !
----- Le 29 Nov 24, à 11:11, Istvan Szabo, Agoda <Istvan.Szabo@agoda.com> a écrit :
Increased from 9 servers to 11 so let's say 20% capacity and performance added.
This is a different cluster purely rbd.
I see, so big objects. You might want to increase osd_max_trimming_pgs and eventually osd_pg_max_concurrent_snap_trims and see how it goes.
(For the other topic can't be resharded because in multisite it will disappear all the data disappear on remote site, need to create new bucket and migrate data first to a higher sharded bucket).
Hum... You have fallen significantly behind on Ceph versions, which must be hindering you in many operational tasks today. Another option would be to catch up and reshard into a recent version in multi-site mode.
Frédéric.
Istvan
From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Friday, November 29, 2024 4:58:52 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Snaptriming speed degrade with pg increase Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi Istvan,
Did the PG split involved using more OSDs than before? If so then increasing these values (apart from the sleep) should not have a negative impact on clients I/O compared to before the split and should accelerate the whole process.
Did you reshard the buckets as discussed in the other thread?
Regards, Frédéric.
----- Le 29 Nov 24, à 3:30, Istvan Szabo, Agoda Istvan.Szabo@agoda.com a écrit :
Hi,
When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04)
-osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2
We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours.
Is there any good solution that we are missing to fix this?
On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still.
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Quick update on this topic, seems to be the solution for us to offline compact all osds. After that all snaptrimming can finish in an hour rather than a day. ________________________________ From: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Sent: Friday, November 29, 2024 2:31:33 PM To: Bandelow, Gunnar <gunnar.bandelow@uni-greifswald.de>; Ceph Users <ceph-users@ceph.io> Subject: [ceph-users] Re: Snaptriming speed degrade with pg increase Let's say yes if that is the issue. Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ From: Bandelow, Gunnar Sent: Friday, November 29, 2024 1:47 PM To: Szabo, Istvan (Agoda); Ceph Users Subject: Re: [ceph-users] Snaptriming speed degrade with pg increase Dear Istvan, The first thing that stands out: Ubuntu 20.04 (EOL in April 2025) and Ceph v15 Octopus (EOL since 2022) Is there a possibility to upgrade these things? Best regards Gunnar --- Original Nachricht --- Betreff: [ceph-users] Snaptriming speed degrade with pg increase Von: "Szabo, Istvan (Agoda)" <istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>> An: "Ceph Users" <ceph-users@ceph.io<mailto:ceph-users@ceph.io>> Datum: 29-11-2024 3:30 Hi, When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04) -osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2 We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours. Is there any good solution that we are missing to fix this? On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still. Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Istvan,
Quick update on this topic, seems to be the solution for us to offline compact all osds. After that all snaptrimming can finish in an hour rather than a day.
Ah, this might be tombstone accumulation, then. You'd probably benefit from going to at least latest Pacific, enabling rocksdb_cf_compact_on_deletion (https://github.com/ceph/ceph/pull/50894), and resharding rocksdb on your OSDs (otherwise it won't take effect). Josh
participants (4)
-
Bandelow, Gunnar
-
Frédéric Nass
-
Joshua Baergen
-
Szabo, Istvan (Agoda)