3 Dec
2020
3 Dec
'20
12:40 p.m.
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I
need to increase the number of objects in flight during rebalance. It is already all
remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are
transferred per PG at a time. The pools sits on high-performance SSDs and could easily
handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase
the number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs,
which stores exclusively the kind of meta data that goes into this pool. Storage
consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS
con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115
con-fs2-meta2 13 0 B 0 933 GiB 118389897
con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so
I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a
bad idea, the meta2 pool should have a much higher PG count or a much more aggressive
recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the
performance of the SSDs it is located on. In a way, I would like to keep the PG count
where it is, but increase the recovery rate for this pool by a factor of 10. Please let me
know what options I have.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
3 Dec
3 Dec
1:11 p.m.
Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder <frans(a)dtu.dk>dk>:
I have the opposite problem as discussed in "slow
down keys/s in
recovery". I need to increase the number of objects in flight during
rebalance. It is already all remapped PGs in state backfilling, but it
looks like no more than 8 objects/sec are transferred per PG at a time. The
pools sits on high-performance SSDs and could easily handle a transfer of
100 or more objects/sec simultaneously. Is there any way to increase the
number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool
of a ceph fs, which stores exclusively the kind of meta data that goes into
this pool. Storage consumption is reported as 0, but the number of objects
is huge:
I don't run cephfs so it might not map 100%, but I think that pools for
which ceph stores file/object metadata (radosgw pools in my case) will show
a completely "false" numbers while recovering, which I think is because
there are tons of object metadata applied as metadata on 0-sized objects.
This means recovery will look like it does one object per second or
something, while in fact it does 100s of metadatas on that one object but
the recovery doesn't list this. Also, it made old ceph df and rados df say
"this pool is almost empty" but when you try to dump or move the pool it
takes far longer than it should take to move an almost-empty pool. And the
pool dump gets huge.
I would take a look at iostat output for those OSD drives and see if there
are 8 iops or lots more actually.
--
May the most significant bit of your life be positive.
1:14 p.m.
Hi Janne,
looked at it already. The recovery rate is unbearably slow and I would like to increase
it. The % misplaced objects is decreasing unnecessarily slow.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Janne Johansson <icepic.dz(a)gmail.com>
Sent: 03 December 2020 10:41:29
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Increase number of objects in flight during recovery
Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>>:
I have the opposite problem as discussed in "slow down keys/s in recovery". I
need to increase the number of objects in flight during rebalance. It is already all
remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are
transferred per PG at a time. The pools sits on high-performance SSDs and could easily
handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase
the number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs,
which stores exclusively the kind of meta data that goes into this pool. Storage
consumption is reported as 0, but the number of objects is huge:
I don't run cephfs so it might not map 100%, but I think that pools for which ceph
stores file/object metadata (radosgw pools in my case) will show a completely
"false" numbers while recovering, which I think is because there are tons of
object metadata applied as metadata on 0-sized objects. This means recovery will look like
it does one object per second or something, while in fact it does 100s of metadatas on
that one object but the recovery doesn't list this. Also, it made old ceph df and
rados df say "this pool is almost empty" but when you try to dump or move the
pool it takes far longer than it should take to move an almost-empty pool. And the pool
dump gets huge.
I would take a look at iostat output for those OSD drives and see if there are 8 iops or
lots more actually.
--
May the most significant bit of your life be positive.
1:33 p.m.
Hi Frank,
out of curiosity, can you share the recovery rates you are seeing?
I would appreciate it, thanks!
On 12/03 09:44, Frank Schilder wrote:
Hi Janne,
looked at it already. The recovery rate is unbearably slow and I would like to increase
it. The % misplaced objects is decreasing unnecessarily slow.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Janne Johansson <icepic.dz(a)gmail.com>
Sent: 03 December 2020 10:41:29
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Increase number of objects in flight during recovery
Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>>:
I have the opposite problem as discussed in "slow down keys/s in recovery". I
need to increase the number of objects in flight during rebalance. It is already all
remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are
transferred per PG at a time. The pools sits on high-performance SSDs and could easily
handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase
the number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs,
which stores exclusively the kind of meta data that goes into this pool. Storage
consumption is reported as 0, but the number of objects is huge:
I don't run cephfs so it might not map 100%, but I think that pools for which ceph
stores file/object metadata (radosgw pools in my case) will show a completely
"false" numbers while recovering, which I think is because there are tons of
object metadata applied as metadata on 0-sized objects. This means recovery will look like
it does one object per second or something, while in fact it does 100s of metadatas on
that one object but the recovery doesn't list this. Also, it made old ceph df and
rados df say "this pool is almost empty" but when you try to dump or move the
pool it takes far longer than it should take to move an almost-empty pool. And the pool
dump gets huge.
I would take a look at iostat output for those OSD drives and see if there are 8 iops or
lots more actually.
--
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
1:44 p.m.
[root@gnosis ~]# ceph status
cluster:
id:
health: HEALTH_WARN
8283238/3566503213 objects misplaced (0.232%)
1 pools nearfull
services:
mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03
mgr: ceph-02(active), standbys: ceph-03, ceph-01
mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay
osd: 315 osds: 295 up, 295 in; 8 remapped pgs
data:
pools: 11 pools, 3215 pgs
objects: 455.8 M objects, 817 TiB
usage: 1.0 PiB used, 1.0 PiB / 2.1 PiB avail
pgs: 8283238/3566503213 objects misplaced (0.232%)
3200 active+clean
8 active+remapped+backfilling
7 active+clean+scrubbing+deep
io:
client: 85 MiB/s rd, 93 MiB/s wr, 1.35 kop/s rd, 856 op/s wr
recovery: 0 B/s, 57 objects/s
With this rate I will wait for ages.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: David Caro <dcaro(a)wikimedia.org>
Sent: 03 December 2020 11:03:28
To: Frank Schilder
Cc: Janne Johansson; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: Increase number of objects in flight during recovery
Hi Frank,
out of curiosity, can you share the recovery rates you are seeing?
I would appreciate it, thanks!
On 12/03 09:44, Frank Schilder wrote:
Hi Janne,
looked at it already. The recovery rate is unbearably slow and I would like to increase
it. The % misplaced objects is decreasing unnecessarily slow.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Janne Johansson <icepic.dz(a)gmail.com>
Sent: 03 December 2020 10:41:29
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Increase number of objects in flight during recovery
Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>>:
I have the opposite problem as discussed in "slow down keys/s in recovery". I
need to increase the number of objects in flight during rebalance. It is already all
remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are
transferred per PG at a time. The pools sits on high-performance SSDs and could easily
handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase
the number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs,
which stores exclusively the kind of meta data that goes into this pool. Storage
consumption is reported as 0, but the number of objects is huge:
I don't run cephfs so it might not map 100%, but I think that pools for which ceph
stores file/object metadata (radosgw pools in my case) will show a completely
"false" numbers while recovering, which I think is because there are tons of
object metadata applied as metadata on 0-sized objects. This means recovery will look like
it does one object per second or something, while in fact it does 100s of metadatas on
that one object but the recovery doesn't list this. Also, it made old ceph df and
rados df say "this pool is almost empty" but when you try to dump or move the
pool it takes far longer than it should take to move an almost-empty pool. And the pool
dump gets huge.
I would take a look at iostat output for those OSD drives and see if there are 8 iops or
lots more actually.
--
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
3:05 p.m.
Hi,
There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will
change includes:
osd_max_backfill
osd_recovery_max_active
osd_recovery_max_single_start
osd_recovery_sleep
Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively.
Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder <frans(a)dtu.dk>
写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I
need to increase the number of objects in flight during rebalance. It is already all
remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are
transferred per PG at a time. The pools sits on high-performance SSDs and could easily
handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase
the number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs,
which stores exclusively the kind of meta data that goes into this pool. Storage
consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS
con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115
con-fs2-meta2 13 0 B 0 933 GiB 118389897
con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so
I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a
bad idea, the meta2 pool should have a much higher PG count or a much more aggressive
recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the
performance of the SSDs it is located on. In a way, I would like to keep the PG count
where it is, but increase the recovery rate for this pool by a factor of 10. Please let me
know what options I have.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:23 p.m.
Did this already. It doesn't change the number of objects in flight.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: 胡 玮文 <huww98(a)outlook.com>
Sent: 03 December 2020 12:35:03
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Increase number of objects in flight during recovery
Hi,
There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will
change includes:
osd_max_backfill
osd_recovery_max_active
osd_recovery_max_single_start
osd_recovery_sleep
Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively.
Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder <frans(a)dtu.dk>
写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I
need to increase the number of objects in flight during rebalance. It is already all
remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are
transferred per PG at a time. The pools sits on high-performance SSDs and could easily
handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase
the number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs,
which stores exclusively the kind of meta data that goes into this pool. Storage
consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS
con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115
con-fs2-meta2 13 0 B 0 933 GiB 118389897
con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so
I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a
bad idea, the meta2 pool should have a much higher PG count or a much more aggressive
recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the
performance of the SSDs it is located on. In a way, I would like to keep the PG count
where it is, but increase the recovery rate for this pool by a factor of 10. Please let me
know what options I have.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:44 p.m.
Sorry, just tried "osd_recovery_sleep=0" (was 0.05) and the number of objects in
flight did increase dramatically:
recovery: 0 B/s, 8.64 kobjects/s
Would be nice if there was a way to set this per pool.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: 03 December 2020 12:53:28
To: 胡 玮文
Cc: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Increase number of objects in flight during recovery
Did this already. It doesn't change the number of objects in flight.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: 胡 玮文 <huww98(a)outlook.com>
Sent: 03 December 2020 12:35:03
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Increase number of objects in flight during recovery
Hi,
There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will
change includes:
osd_max_backfill
osd_recovery_max_active
osd_recovery_max_single_start
osd_recovery_sleep
Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively.
Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder <frans(a)dtu.dk>
写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I
need to increase the number of objects in flight during rebalance. It is already all
remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are
transferred per PG at a time. The pools sits on high-performance SSDs and could easily
handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase
the number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs,
which stores exclusively the kind of meta data that goes into this pool. Storage
consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS
con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115
con-fs2-meta2 13 0 B 0 933 GiB 118389897
con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so
I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a
bad idea, the meta2 pool should have a much higher PG count or a much more aggressive
recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the
performance of the SSDs it is located on. In a way, I would like to keep the PG count
where it is, but increase the recovery rate for this pool by a factor of 10. Please let me
know what options I have.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:19 p.m.
Hi Frank,
this values we used to reduce the recovery impact before luminous.
#reduce recovery impact
osd max backfills
osd recovery max active
osd recovery max single start
osd recovery op priority
osd recovery threads
osd backfill scan max
osd backfill scan min
I do not know how many osds and pgs you have in your cluster. But the
backfill performance depends on osds, pgs and objects/pg.
Regards, Joachim
___________________________________
Clyso GmbH
Am 03.12.2020 um 12:35 schrieb 胡 玮文:
Hi,
There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will
change includes:
osd_max_backfill
osd_recovery_max_active
osd_recovery_max_single_start
osd_recovery_sleep
Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively.
Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder
<frans(a)dtu.dk> 写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I
need to increase the number of objects in flight during rebalance. It is already all
remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are
transferred per PG at a time. The pools sits on high-performance SSDs and could easily
handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase
the number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs,
which stores exclusively the kind of meta data that goes into this pool. Storage
consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS
con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115
con-fs2-meta2 13 0 B 0 933 GiB 118389897
con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so
I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a
bad idea, the meta2 pool should have a much higher PG count or a much more aggressive
recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the
performance of the SSDs it is located on. In a way, I would like to keep the PG count
where it is, but increase the recovery rate for this pool by a factor of 10. Please let me
know what options I have.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6 Dec
6 Dec
1:01 p.m.
Just to update the case for others: Setting
ceph config set osd/class:ssd osd_recovery_sleep 0.001
ceph config set osd/class:hdd osd_recovery_sleep 0.05
had the desired effect. I'm running another massive rebalancing operation right now
and these settings seem to help. It would be nice if one could use a pool name in a filter
though (osd/pool:NAME). I have 2 different pools on the same SSDs and only objects from
one of these pools require the lower sleep setting.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Joachim Kraftmayer <joachim.kraftmayer(a)clyso.com>
Sent: 03 December 2020 16:49:51
To: 胡 玮文; Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: Increase number of objects in flight during recovery
Hi Frank,
this values we used to reduce the recovery impact before luminous.
#reduce recovery impact
osd max backfills
osd recovery max active
osd recovery max single start
osd recovery op priority
osd recovery threads
osd backfill scan max
osd backfill scan min
I do not know how many osds and pgs you have in your cluster. But the
backfill performance depends on osds, pgs and objects/pg.
Regards, Joachim
___________________________________
Clyso GmbH
Am 03.12.2020 um 12:35 schrieb 胡 玮文:
Hi,
There is a “OSD recovery priority” dialog box in web dashboard. Configurations it will
change includes:
osd_max_backfill
osd_recovery_max_active
osd_recovery_max_single_start
osd_recovery_sleep
Tune these config may helps. “High” priority corresponding to 4, 4, 4, 0, respectively.
Some of these also have a _ssd/_hdd variant.
在 2020年12月3日，17:11，Frank Schilder
<frans(a)dtu.dk> 写道：
Hi all,
I have the opposite problem as discussed in "slow down keys/s in recovery". I
need to increase the number of objects in flight during rebalance. It is already all
remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are
transferred per PG at a time. The pools sits on high-performance SSDs and could easily
handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase
the number of transfers/sec or simultaneous transfers? Increasing the options
osd_max_backfills and osd_recovery_max_active has no effect.
Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs,
which stores exclusively the kind of meta data that goes into this pool. Storage
consumption is reported as 0, but the number of objects is huge:
NAME ID USED %USED MAX AVAIL OBJECTS
con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115
con-fs2-meta2 13 0 B 0 933 GiB 118389897
con-fs2-data 14 698 TiB 72.15 270 TiB 286826739
Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so
I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a
bad idea, the meta2 pool should have a much higher PG count or a much more aggressive
recovery policy.
I now need to rebalance PGs on meta2 and it is going way too slow compared with the
performance of the SSDs it is located on. In a way, I would like to keep the PG count
where it is, but increase the recovery rate for this pool by a factor of 10. Please let me
know what options I have.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1247
days inactive
1250
days old
9 comments
5 participants
participants (5)
-
David Caro -
Frank Schilder -
Janne Johansson -
Joachim Kraftmayer -
胡 玮文