Increase number of objects in flight during recovery
Hi all, I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect. Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge: NAME ID USED %USED MAX AVAIL OBJECTS con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115 con-fs2-meta2 13 0 B 0 933 GiB 118389897 con-fs2-data 14 698 TiB 72.15 270 TiB 286826739 Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a bad idea, the meta2 pool should have a much higher PG count or a much more aggressive recovery policy. I now need to rebalance PGs on meta2 and it is going way too slow compared with the performance of the SSDs it is located on. In a way, I would like to keep the PG count where it is, but increase the recovery rate for this pool by a factor of 10. Please let me know what options I have. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder <frans@dtu.dk>:
I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect. Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge:
I don't run cephfs so it might not map 100%, but I think that pools for which ceph stores file/object metadata (radosgw pools in my case) will show a completely "false" numbers while recovering, which I think is because there are tons of object metadata applied as metadata on 0-sized objects. This means recovery will look like it does one object per second or something, while in fact it does 100s of metadatas on that one object but the recovery doesn't list this. Also, it made old ceph df and rados df say "this pool is almost empty" but when you try to dump or move the pool it takes far longer than it should take to move an almost-empty pool. And the pool dump gets huge. I would take a look at iostat output for those OSD drives and see if there are 8 iops or lots more actually. -- May the most significant bit of your life be positive.
Hi Janne, looked at it already. The recovery rate is unbearably slow and I would like to increase it. The % misplaced objects is decreasing unnecessarily slow. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Janne Johansson <icepic.dz@gmail.com> Sent: 03 December 2020 10:41:29 To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Increase number of objects in flight during recovery Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>: I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect. Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge: I don't run cephfs so it might not map 100%, but I think that pools for which ceph stores file/object metadata (radosgw pools in my case) will show a completely "false" numbers while recovering, which I think is because there are tons of object metadata applied as metadata on 0-sized objects. This means recovery will look like it does one object per second or something, while in fact it does 100s of metadatas on that one object but the recovery doesn't list this. Also, it made old ceph df and rados df say "this pool is almost empty" but when you try to dump or move the pool it takes far longer than it should take to move an almost-empty pool. And the pool dump gets huge. I would take a look at iostat output for those OSD drives and see if there are 8 iops or lots more actually. -- May the most significant bit of your life be positive.
Hi Frank, out of curiosity, can you share the recovery rates you are seeing? I would appreciate it, thanks! On 12/03 09:44, Frank Schilder wrote:
Hi Janne,
looked at it already. The recovery rate is unbearably slow and I would like to increase it. The % misplaced objects is decreasing unnecessarily slow.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Janne Johansson <icepic.dz@gmail.com> Sent: 03 December 2020 10:41:29 To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Increase number of objects in flight during recovery
Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>: I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect. Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge:
I don't run cephfs so it might not map 100%, but I think that pools for which ceph stores file/object metadata (radosgw pools in my case) will show a completely "false" numbers while recovering, which I think is because there are tons of object metadata applied as metadata on 0-sized objects. This means recovery will look like it does one object per second or something, while in fact it does 100s of metadatas on that one object but the recovery doesn't list this. Also, it made old ceph df and rados df say "this pool is almost empty" but when you try to dump or move the pool it takes far longer than it should take to move an almost-empty pool. And the pool dump gets huge.
I would take a look at iostat output for those OSD drives and see if there are 8 iops or lots more actually.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
[root@gnosis ~]# ceph status cluster: id: health: HEALTH_WARN 8283238/3566503213 objects misplaced (0.232%) 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-02(active), standbys: ceph-03, ceph-01 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 315 osds: 295 up, 295 in; 8 remapped pgs data: pools: 11 pools, 3215 pgs objects: 455.8 M objects, 817 TiB usage: 1.0 PiB used, 1.0 PiB / 2.1 PiB avail pgs: 8283238/3566503213 objects misplaced (0.232%) 3200 active+clean 8 active+remapped+backfilling 7 active+clean+scrubbing+deep io: client: 85 MiB/s rd, 93 MiB/s wr, 1.35 kop/s rd, 856 op/s wr recovery: 0 B/s, 57 objects/s With this rate I will wait for ages. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: David Caro <dcaro@wikimedia.org> Sent: 03 December 2020 11:03:28 To: Frank Schilder Cc: Janne Johansson; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Increase number of objects in flight during recovery Hi Frank, out of curiosity, can you share the recovery rates you are seeing? I would appreciate it, thanks! On 12/03 09:44, Frank Schilder wrote:
Hi Janne,
looked at it already. The recovery rate is unbearably slow and I would like to increase it. The % misplaced objects is decreasing unnecessarily slow.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Janne Johansson <icepic.dz@gmail.com> Sent: 03 December 2020 10:41:29 To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Increase number of objects in flight during recovery
Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>: I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect. Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge:
I don't run cephfs so it might not map 100%, but I think that pools for which ceph stores file/object metadata (radosgw pools in my case) will show a completely "false" numbers while recovering, which I think is because there are tons of object metadata applied as metadata on 0-sized objects. This means recovery will look like it does one object per second or something, while in fact it does 100s of metadatas on that one object but the recovery doesn't list this. Also, it made old ceph df and rados df say "this pool is almost empty" but when you try to dump or move the pool it takes far longer than it should take to move an almost-empty pool. And the pool dump gets huge.
I would take a look at iostat output for those OSD drives and see if there are 8 iops or lots more actually.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
Hi, There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will change includes: osd_max_backfill osd_recovery_max_active osd_recovery_max_single_start osd_recovery_sleep Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively. Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder <frans@dtu.dk> 写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115 con-fs2-meta2 13 0 B 0 933 GiB 118389897 con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a bad idea, the meta2 pool should have a much higher PG count or a much more aggressive recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the performance of the SSDs it is located on. In a way, I would like to keep the PG count where it is, but increase the recovery rate for this pool by a factor of 10. Please let me know what options I have.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Did this already. It doesn't change the number of objects in flight. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: 胡 玮文 <huww98@outlook.com> Sent: 03 December 2020 12:35:03 To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Increase number of objects in flight during recovery Hi, There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will change includes: osd_max_backfill osd_recovery_max_active osd_recovery_max_single_start osd_recovery_sleep Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively. Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder <frans@dtu.dk> 写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115 con-fs2-meta2 13 0 B 0 933 GiB 118389897 con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a bad idea, the meta2 pool should have a much higher PG count or a much more aggressive recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the performance of the SSDs it is located on. In a way, I would like to keep the PG count where it is, but increase the recovery rate for this pool by a factor of 10. Please let me know what options I have.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry, just tried "osd_recovery_sleep=0" (was 0.05) and the number of objects in flight did increase dramatically: recovery: 0 B/s, 8.64 kobjects/s Would be nice if there was a way to set this per pool. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 03 December 2020 12:53:28 To: 胡 玮文 Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Increase number of objects in flight during recovery Did this already. It doesn't change the number of objects in flight. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: 胡 玮文 <huww98@outlook.com> Sent: 03 December 2020 12:35:03 To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Increase number of objects in flight during recovery Hi, There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will change includes: osd_max_backfill osd_recovery_max_active osd_recovery_max_single_start osd_recovery_sleep Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively. Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder <frans@dtu.dk> 写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115 con-fs2-meta2 13 0 B 0 933 GiB 118389897 con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a bad idea, the meta2 pool should have a much higher PG count or a much more aggressive recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the performance of the SSDs it is located on. In a way, I would like to keep the PG count where it is, but increase the recovery rate for this pool by a factor of 10. Please let me know what options I have.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, this values we used to reduce the recovery impact before luminous. #reduce recovery impact osd max backfills osd recovery max active osd recovery max single start osd recovery op priority osd recovery threads osd backfill scan max osd backfill scan min I do not know how many osds and pgs you have in your cluster. But the backfill performance depends on osds, pgs and objects/pg. Regards, Joachim ___________________________________ Clyso GmbH Am 03.12.2020 um 12:35 schrieb 胡 玮文:
Hi,
There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will change includes:
osd_max_backfill osd_recovery_max_active osd_recovery_max_single_start osd_recovery_sleep
Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively. Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder <frans@dtu.dk> 写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115 con-fs2-meta2 13 0 B 0 933 GiB 118389897 con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a bad idea, the meta2 pool should have a much higher PG count or a much more aggressive recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the performance of the SSDs it is located on. In a way, I would like to keep the PG count where it is, but increase the recovery rate for this pool by a factor of 10. Please let me know what options I have.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just to update the case for others: Setting ceph config set osd/class:ssd osd_recovery_sleep 0.001 ceph config set osd/class:hdd osd_recovery_sleep 0.05 had the desired effect. I'm running another massive rebalancing operation right now and these settings seem to help. It would be nice if one could use a pool name in a filter though (osd/pool:NAME). I have 2 different pools on the same SSDs and only objects from one of these pools require the lower sleep setting. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Joachim Kraftmayer <joachim.kraftmayer@clyso.com> Sent: 03 December 2020 16:49:51 To: 胡 玮文; Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Increase number of objects in flight during recovery Hi Frank, this values we used to reduce the recovery impact before luminous. #reduce recovery impact osd max backfills osd recovery max active osd recovery max single start osd recovery op priority osd recovery threads osd backfill scan max osd backfill scan min I do not know how many osds and pgs you have in your cluster. But the backfill performance depends on osds, pgs and objects/pg. Regards, Joachim ___________________________________ Clyso GmbH Am 03.12.2020 um 12:35 schrieb 胡 玮文:
Hi,
There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will change includes:
osd_max_backfill osd_recovery_max_active osd_recovery_max_single_start osd_recovery_sleep
Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively. Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder <frans@dtu.dk> 写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115 con-fs2-meta2 13 0 B 0 933 GiB 118389897 con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a bad idea, the meta2 pool should have a much higher PG count or a much more aggressive recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the performance of the SSDs it is located on. In a way, I would like to keep the PG count where it is, but increase the recovery rate for this pool by a factor of 10. Please let me know what options I have.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
David Caro
-
Frank Schilder
-
Janne Johansson
-
Joachim Kraftmayer
-
胡 玮文