Ceph Cluster Taking An Awful Long Time To Rebalance
Hi Guys, Is the below "ceph -s" normal? This is a brand new cluster with (at the moment) a single Monitor and 7 OSDs (each 6 GiB) that has no data in it (yet), and yet its taking almost a day to "heal itself" from adding in the 2nd OSD. ~~~ cluster: id: [REDACTED] health: HEALTH_WARN Reduced data availability: 256 pgs inactive, 256 pgs incomplete Degraded data redundancy: 12 pgs undersized services: mon: 1 daemons, quorum [REDACTED] (age 22h) mgr: [REDACTED](active, since 22h) osd: 7 osds: 7 up (since 21h), 7 in (since 21h); 32 remapped pgs data: pools: 5 pools, 288 pgs objects: 7 objects, 0 B usage: 7.1 GiB used, 38 TiB / 38 TiB avail pgs: 88.889% pgs not active 6/21 objects misplaced (28.571%) 256 creating+incomplete 18 active+clean 12 active+undersized+remapped 2 active+clean+remapped progress: Rebalancing after osd.1 marked in (22h) [............................] PG autoscaler decreasing pool 1 PGs from 32 to 1 (19h) [............................] ~~~ Thanks in advance Matthew J -- Peregrine IT Signature *Matthew J BLACK* M.Inf.Tech.(Data Comms) MBA B.Sc. MACS (Snr), CP, IP3P When you want it done /right/ ‒ the first time! Phone: +61 4 0411 0089 Email: matthew@peregrineit.net <mailto:matthew@peregrineit.net> Web: www.peregrineit.net <http://www.peregrineit.net> View Matthew J BLACK's profile on LinkedIn <http://au.linkedin.com/in/mjblack> This Email is intended only for the addressee. Its use is limited to that intended by the author at the time and it is not to be distributed without the author’s consent. You must not use or disclose the contents of this Email, or add the sender’s Email address to any database, list or mailing list unless you are expressly authorised to do so. Unless otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the contents of this Email except where subsequently confirmed in writing. The opinions expressed in this Email are those of the author and do not necessarily represent the views of Peregrine I.T. Pty Ltd. This Email is confidential and may be subject to a claim of legal privilege. If you have received this Email in error, please notify the author and delete this message immediately.
I think your issue is you also have the PG scaler trying to change to 1PG. Due to the size of the OSD/no data it thinks you only need 1PG, I would suggest disabling the PG Auto Scaler on small test clusters. Thanks On 16 Mar 2021, 10:50 +0800, duluxoz <duluxoz@gmail.com>, wrote:
Hi Guys,
Is the below "ceph -s" normal?
This is a brand new cluster with (at the moment) a single Monitor and 7 OSDs (each 6 GiB) that has no data in it (yet), and yet its taking almost a day to "heal itself" from adding in the 2nd OSD.
~~~
cluster: id: [REDACTED] health: HEALTH_WARN Reduced data availability: 256 pgs inactive, 256 pgs incomplete Degraded data redundancy: 12 pgs undersized
services: mon: 1 daemons, quorum [REDACTED] (age 22h) mgr: [REDACTED](active, since 22h) osd: 7 osds: 7 up (since 21h), 7 in (since 21h); 32 remapped pgs
data: pools: 5 pools, 288 pgs objects: 7 objects, 0 B usage: 7.1 GiB used, 38 TiB / 38 TiB avail pgs: 88.889% pgs not active 6/21 objects misplaced (28.571%) 256 creating+incomplete 18 active+clean 12 active+undersized+remapped 2 active+clean+remapped
progress: Rebalancing after osd.1 marked in (22h) [............................] PG autoscaler decreasing pool 1 PGs from 32 to 1 (19h) [............................]
~~~
Thanks in advance
Matthew J
-- Peregrine IT Signature
*Matthew J BLACK* M.Inf.Tech.(Data Comms) MBA B.Sc. MACS (Snr), CP, IP3P
When you want it done /right/ ‒ the first time!
Phone: +61 4 0411 0089 Email: matthew@peregrineit.net <mailto:matthew@peregrineit.net> Web: www.peregrineit.net <http://www.peregrineit.net>
View Matthew J BLACK's profile on LinkedIn <http://au.linkedin.com/in/mjblack>
This Email is intended only for the addressee. Its use is limited to that intended by the author at the time and it is not to be distributed without the author’s consent. You must not use or disclose the contents of this Email, or add the sender’s Email address to any database, list or mailing list unless you are expressly authorised to do so. Unless otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the contents of this Email except where subsequently confirmed in writing. The opinions expressed in this Email are those of the author and do not necessarily represent the views of Peregrine I.T. Pty Ltd. This Email is confidential and may be subject to a claim of legal privilege.
If you have received this Email in error, please notify the author and delete this message immediately.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for that - I'll disable the autoscaler on all 5 pools and see what happens Cheers Matthew J On 16/03/2021 13:53, ashley@amerrick.co.uk wrote:
I think your issue is you also have the PG scaler trying to change to 1PG.
Due to the size of the OSD/no data it thinks you only need 1PG, I would suggest disabling the PG Auto Scaler on small test clusters.
Thanks On 16 Mar 2021, 10:50 +0800, duluxoz <duluxoz@gmail.com>, wrote:
Hi Guys,
Is the below "ceph -s" normal?
This is a brand new cluster with (at the moment) a single Monitor and 7
OSDs (each 6 GiB) that has no data in it (yet), and yet its taking
almost a day to "heal itself" from adding in the 2nd OSD.
~~~
cluster:
id: [REDACTED]
health: HEALTH_WARN
Reduced data availability: 256 pgs inactive, 256 pgs incomplete
Degraded data redundancy: 12 pgs undersized
services:
mon: 1 daemons, quorum [REDACTED] (age 22h)
mgr: [REDACTED](active, since 22h)
osd: 7 osds: 7 up (since 21h), 7 in (since 21h); 32 remapped pgs
data:
pools: 5 pools, 288 pgs
objects: 7 objects, 0 B
usage: 7.1 GiB used, 38 TiB / 38 TiB avail
pgs: 88.889% pgs not active
6/21 objects misplaced (28.571%)
256 creating+incomplete
18 active+clean
12 active+undersized+remapped
2 active+clean+remapped
progress:
Rebalancing after osd.1 marked in (22h)
[............................]
PG autoscaler decreasing pool 1 PGs from 32 to 1 (19h)
[............................]
~~~
Thanks in advance
Matthew J
--
Peregrine IT Signature
*Matthew J BLACK*
M.Inf.Tech.(Data Comms)
MBA
B.Sc.
MACS (Snr), CP, IP3P
When you want it done /right/ ‒ the first time!
Phone: +61 4 0411 0089
Email: matthew@peregrineit.net <mailto:matthew@peregrineit.net>
Web: www.peregrineit.net <http://www.peregrineit.net>
View Matthew J BLACK's profile on LinkedIn
<http://au.linkedin.com/in/mjblack>
This Email is intended only for the addressee. Its use is limited to
that intended by the author at the time and it is not to be distributed
without the author’s consent. You must not use or disclose the contents
of this Email, or add the sender’s Email address to any database, list
or mailing list unless you are expressly authorised to do so. Unless
otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the
contents of this Email except where subsequently confirmed in
writing. The opinions expressed in this Email are those of the author
and do not necessarily represent the views of Peregrine I.T. Pty
Ltd. This Email is confidential and may be subject to a claim of legal
privilege.
If you have received this Email in error, please notify the author and
delete this message immediately.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
-- Peregrine IT Signature *Matthew J BLACK* M.Inf.Tech.(Data Comms) MBA B.Sc. MACS (Snr), CP, IP3P When you want it done /right/ ‒ the first time! Phone: +61 4 0411 0089 Email: matthew@peregrineit.net <mailto:matthew@peregrineit.net> Web: www.peregrineit.net <http://www.peregrineit.net> View Matthew J BLACK's profile on LinkedIn <http://au.linkedin.com/in/mjblack> This Email is intended only for the addressee. Its use is limited to that intended by the author at the time and it is not to be distributed without the author’s consent. You must not use or disclose the contents of this Email, or add the sender’s Email address to any database, list or mailing list unless you are expressly authorised to do so. Unless otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the contents of this Email except where subsequently confirmed in writing. The opinions expressed in this Email are those of the author and do not necessarily represent the views of Peregrine I.T. Pty Ltd. This Email is confidential and may be subject to a claim of legal privilege. If you have received this Email in error, please notify the author and delete this message immediately.
OK, so I set autoscaling to off for all five pools, and the "ceph -s" has not changed: ~~~ cluster: id: [REDACTED] health: HEALTH_WARN Reduced data availability: 256 pgs inactive, 256 pgs incomplete Degraded data redundancy: 12 pgs undersized services: mon: 1 daemons, quorum [REDACTED] (age 23h) mgr: [REDACTED](active, since 23h) osd: 7 osds: 7 up (since 22h), 7 in (since 22h); 32 remapped pgs data: pools: 5 pools, 288 pgs objects: 7 objects, 0 B usage: 7.1 GiB used, 38 TiB / 38 TiB avail pgs: 88.889% pgs not active 6/21 objects misplaced (28.571%) 256 creating+incomplete 18 active+clean 12 active+undersized+remapped 2 active+clean+remapped progress: Rebalancing after osd.1 marked in (23h) [............................] PG autoscaler decreasing pool 1 PGs from 32 to 1 (21h) [............................] ~~~ Any ideas - or is this normal ie does this normally take this long? (I'm wondering if I shouldn't tear down the cluster and start again?) Cheers Matthew J
pgs: 88.889% pgs not active 6/21 objects misplaced (28.571%) 256 creating+incomplete
For new clusters, "creating+incomplete" sounds like you created a pool (with 256 PGs) with some crush rule that doesn't allow it to find suitable placements, like "replication = 3" and "failure domain = host" but only having 2 hosts, or something to that effect. Unless you add hosts (in my example), this will not "fix itself" until you either add hosts, or change the crush rules to something less reliable. -- May the most significant bit of your life be positive.
Ah, right, that makes sense - I'll have a go at that Thank you On 16/03/2021 19:12, Janne Johansson wrote:
pgs: 88.889% pgs not active 6/21 objects misplaced (28.571%) 256 creating+incomplete
For new clusters, "creating+incomplete" sounds like you created a pool (with 256 PGs) with some crush rule that doesn't allow it to find suitable placements, like "replication = 3" and "failure domain = host" but only having 2 hosts, or something to that effect. Unless you add hosts (in my example), this will not "fix itself" until you either add hosts, or change the crush rules to something less reliable.
-- Peregrine IT Signature *Matthew J BLACK* M.Inf.Tech.(Data Comms) MBA B.Sc. MACS (Snr), CP, IP3P When you want it done /right/ ‒ the first time! Phone: +61 4 0411 0089 Email: matthew@peregrineit.net <mailto:matthew@peregrineit.net> Web: www.peregrineit.net <http://www.peregrineit.net> View Matthew J BLACK's profile on LinkedIn <http://au.linkedin.com/in/mjblack> This Email is intended only for the addressee. Its use is limited to that intended by the author at the time and it is not to be distributed without the author’s consent. You must not use or disclose the contents of this Email, or add the sender’s Email address to any database, list or mailing list unless you are expressly authorised to do so. Unless otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the contents of this Email except where subsequently confirmed in writing. The opinions expressed in this Email are those of the author and do not necessarily represent the views of Peregrine I.T. Pty Ltd. This Email is confidential and may be subject to a claim of legal privilege. If you have received this Email in error, please notify the author and delete this message immediately.
Yeap - that was the issue: an incorrect CRUSH rule Thanks for the help Dulux-Oz
participants (3)
-
ashley＠amerrick.co.uk
-
duluxoz
-
Janne Johansson