16 Mar
2021
16 Mar
'21
5:48 a.m.
Hi Guys,
Is the below "ceph -s" normal?
This is a brand new cluster with (at the moment) a single Monitor and 7
OSDs (each 6 GiB) that has no data in it (yet), and yet its taking
almost a day to "heal itself" from adding in the 2nd OSD.
~~~
cluster:
id: [REDACTED]
health: HEALTH_WARN
Reduced data availability: 256 pgs inactive, 256 pgs incomplete
Degraded data redundancy: 12 pgs undersized
services:
mon: 1 daemons, quorum [REDACTED] (age 22h)
mgr: [REDACTED](active, since 22h)
osd: 7 osds: 7 up (since 21h), 7 in (since 21h); 32 remapped pgs
data:
pools: 5 pools, 288 pgs
objects: 7 objects, 0 B
usage: 7.1 GiB used, 38 TiB / 38 TiB avail
pgs: 88.889% pgs not active
6/21 objects misplaced (28.571%)
256 creating+incomplete
18 active+clean
12 active+undersized+remapped
2 active+clean+remapped
progress:
Rebalancing after osd.1 marked in (22h)
[............................]
PG autoscaler decreasing pool 1 PGs from 32 to 1 (19h)
[............................]
~~~
Thanks in advance
Matthew J
--
Peregrine IT Signature
*Matthew J BLACK*
M.Inf.Tech.(Data Comms)
MBA
B.Sc.
MACS (Snr), CP, IP3P
When you want it done /right/ ‒ the first time!
Phone: +61 4 0411 0089
Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net>
Web: www.peregrineit.net <http://www.peregrineit.net>
View Matthew J BLACK's profile on LinkedIn
<http://au.linkedin.com/in/mjblack>
This Email is intended only for the addressee. Its use is limited to
that intended by the author at the time and it is not to be distributed
without the author’s consent. You must not use or disclose the contents
of this Email, or add the sender’s Email address to any database, list
or mailing list unless you are expressly authorised to do so. Unless
otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the
contents of this Email except where subsequently confirmed in
writing. The opinions expressed in this Email are those of the author
and do not necessarily represent the views of Peregrine I.T. Pty
Ltd. This Email is confidential and may be subject to a claim of legal
privilege.
If you have received this Email in error, please notify the author and
delete this message immediately.
16 Mar
16 Mar
5:53 a.m.
I think your issue is you also have the PG scaler trying to change to 1PG.
Due to the size of the OSD/no data it thinks you only need 1PG, I would suggest disabling
the PG Auto Scaler on small test clusters.
Thanks
On 16 Mar 2021, 10:50 +0800, duluxoz <duluxoz(a)gmail.com>om>, wrote:
Hi Guys,
Is the below "ceph -s" normal?
This is a brand new cluster with (at the moment) a single Monitor and 7
OSDs (each 6 GiB) that has no data in it (yet), and yet its taking
almost a day to "heal itself" from adding in the 2nd OSD.
~~~
cluster:
id: [REDACTED]
health: HEALTH_WARN
Reduced data availability: 256 pgs inactive, 256 pgs incomplete
Degraded data redundancy: 12 pgs undersized
services:
mon: 1 daemons, quorum [REDACTED] (age 22h)
mgr: [REDACTED](active, since 22h)
osd: 7 osds: 7 up (since 21h), 7 in (since 21h); 32 remapped pgs
data:
pools: 5 pools, 288 pgs
objects: 7 objects, 0 B
usage: 7.1 GiB used, 38 TiB / 38 TiB avail
pgs: 88.889% pgs not active
6/21 objects misplaced (28.571%)
256 creating+incomplete
18 active+clean
12 active+undersized+remapped
2 active+clean+remapped
progress:
Rebalancing after osd.1 marked in (22h)
[............................]
PG autoscaler decreasing pool 1 PGs from 32 to 1 (19h)
[............................]
~~~
Thanks in advance
Matthew J
--
Peregrine IT Signature
*Matthew J BLACK*
M.Inf.Tech.(Data Comms)
MBA
B.Sc.
MACS (Snr), CP, IP3P
When you want it done /right/ ‒ the first time!
Phone: +61 4 0411 0089
Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net>
Web: www.peregrineit.net <http://www.peregrineit.net>
View Matthew J BLACK's profile on LinkedIn
<http://au.linkedin.com/in/mjblack>
This Email is intended only for the addressee. Its use is limited to
that intended by the author at the time and it is not to be distributed
without the author’s consent. You must not use or disclose the contents
of this Email, or add the sender’s Email address to any database, list
or mailing list unless you are expressly authorised to do so. Unless
otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the
contents of this Email except where subsequently confirmed in
writing. The opinions expressed in this Email are those of the author
and do not necessarily represent the views of Peregrine I.T. Pty
Ltd. This Email is confidential and may be subject to a claim of legal
privilege.
If you have received this Email in error, please notify the author and
delete this message immediately.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:55 a.m.
Thanks for that - I'll disable the autoscaler on all 5 pools and see
what happens
Cheers
Matthew J
On 16/03/2021 13:53, ashley(a)amerrick.co.uk wrote:
I think your issue is you also have the PG scaler
trying to change to 1PG.
Due to the size of the OSD/no data it thinks you only need 1PG, I
would suggest disabling the PG Auto Scaler on small test clusters.
Thanks
On 16 Mar 2021, 10:50 +0800, duluxoz <duluxoz(a)gmail.com>om>, wrote:
Hi Guys,
Is the below "ceph -s" normal?
This is a brand new cluster with (at the moment) a single Monitor
and 7
OSDs (each 6 GiB) that has no data in it (yet), and yet its taking
almost a day to "heal itself" from adding in the 2nd OSD.
~~~
cluster:
id: [REDACTED]
health: HEALTH_WARN
Reduced data availability: 256 pgs inactive, 256 pgs
incomplete
Degraded data redundancy: 12 pgs undersized
services:
mon: 1 daemons, quorum [REDACTED] (age 22h)
mgr: [REDACTED](active, since 22h)
osd: 7 osds: 7 up (since 21h), 7 in (since 21h); 32 remapped pgs
data:
pools: 5 pools, 288 pgs
objects: 7 objects, 0 B
usage: 7.1 GiB used, 38 TiB / 38 TiB avail
pgs: 88.889% pgs not active
6/21 objects misplaced (28.571%)
256 creating+incomplete
18 active+clean
12 active+undersized+remapped
2 active+clean+remapped
progress:
Rebalancing after osd.1 marked in (22h)
[............................]
PG autoscaler decreasing pool 1 PGs from 32 to 1 (19h)
[............................]
~~~
Thanks in advance
Matthew J
--
Peregrine IT Signature
*Matthew J BLACK*
M.Inf.Tech.(Data Comms)
MBA
B.Sc.
MACS (Snr), CP, IP3P
When you want it done /right/ ‒ the first time!
Phone: +61 4 0411 0089
Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net>
Web: www.peregrineit.net <http://www.peregrineit.net>
View Matthew J BLACK's profile on LinkedIn
<http://au.linkedin.com/in/mjblack>
This Email is intended only for the addressee. Its use is limited to
that intended by the author at the time and it is not to be
distributed
without the author’s consent. You must not use or disclose the
contents
of this Email, or add the sender’s Email address to any database, list
or mailing list unless you are expressly authorised to do so. Unless
otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the
contents of this Email except where subsequently confirmed in
writing. The opinions expressed in this Email are those of the author
and do not necessarily represent the views of Peregrine I.T. Pty
Ltd. This Email is confidential and may be subject to a claim of
legal
privilege.
If you have received this Email in error, please notify the author and
delete this message immediately.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Peregrine IT Signature
*Matthew J BLACK*
M.Inf.Tech.(Data Comms)
MBA
B.Sc.
MACS (Snr), CP, IP3P
When you want it done /right/ ‒ the first time!
Phone: +61 4 0411 0089
Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net>
Web: www.peregrineit.net <http://www.peregrineit.net>
View Matthew J BLACK's profile on LinkedIn
<http://au.linkedin.com/in/mjblack>
This Email is intended only for the addressee. Its use is limited to
that intended by the author at the time and it is not to be distributed
without the author’s consent. You must not use or disclose the contents
of this Email, or add the sender’s Email address to any database, list
or mailing list unless you are expressly authorised to do so. Unless
otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the
contents of this Email except where subsequently confirmed in
writing. The opinions expressed in this Email are those of the author
and do not necessarily represent the views of Peregrine I.T. Pty
Ltd. This Email is confidential and may be subject to a claim of legal
privilege.
If you have received this Email in error, please notify the author and
delete this message immediately.
7:10 a.m.
OK, so I set autoscaling to off for all five pools, and the "ceph -s"
has not changed:
~~~
cluster:
id: [REDACTED]
health: HEALTH_WARN
Reduced data availability: 256 pgs inactive, 256 pgs incomplete
Degraded data redundancy: 12 pgs undersized
services:
mon: 1 daemons, quorum [REDACTED] (age 23h)
mgr: [REDACTED](active, since 23h)
osd: 7 osds: 7 up (since 22h), 7 in (since 22h); 32 remapped pgs
data:
pools: 5 pools, 288 pgs
objects: 7 objects, 0 B
usage: 7.1 GiB used, 38 TiB / 38 TiB avail
pgs: 88.889% pgs not active
6/21 objects misplaced (28.571%)
256 creating+incomplete
18 active+clean
12 active+undersized+remapped
2 active+clean+remapped
progress:
Rebalancing after osd.1 marked in (23h)
[............................]
PG autoscaler decreasing pool 1 PGs from 32 to 1 (21h)
[............................]
~~~
Any ideas - or is this normal ie does this normally take this long?
(I'm wondering if I shouldn't tear down the cluster and start again?)
Cheers
Matthew J
11:12 a.m.
pgs: 88.889% pgs not active
6/21 objects misplaced (28.571%)
256 creating+incomplete
For new clusters, "creating+incomplete" sounds like you created a pool
(with 256 PGs) with some crush rule that doesn't allow it to find
suitable placements, like "replication = 3" and "failure domain =
host" but only having 2 hosts, or something to that effect. Unless you
add hosts (in my example), this will not "fix itself" until you either
add hosts, or change the crush rules to something less reliable.
--
May the most significant bit of your life be positive.
11:38 a.m.
Ah, right, that makes sense - I'll have a go at that
Thank you
On 16/03/2021 19:12, Janne Johansson wrote:
--
Peregrine IT Signature
*Matthew J BLACK*
M.Inf.Tech.(Data Comms)
MBA
B.Sc.
MACS (Snr), CP, IP3P
When you want it done /right/ ‒ the first time!
Phone: +61 4 0411 0089
Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net>
Web: www.peregrineit.net <http://www.peregrineit.net>
View Matthew J BLACK's profile on LinkedIn
<http://au.linkedin.com/in/mjblack>
This Email is intended only for the addressee. Its use is limited to
that intended by the author at the time and it is not to be distributed
without the author’s consent. You must not use or disclose the contents
of this Email, or add the sender’s Email address to any database, list
or mailing list unless you are expressly authorised to do so. Unless
otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the
contents of this Email except where subsequently confirmed in
writing. The opinions expressed in this Email are those of the author
and do not necessarily represent the views of Peregrine I.T. Pty
Ltd. This Email is confidential and may be subject to a claim of legal
privilege.
If you have received this Email in error, please notify the author and
delete this message immediately.
pgs:
88.889% pgs not active
6/21 objects misplaced (28.571%)
256 creating+incomplete
For new clusters,
"creating+incomplete" sounds like you created a pool
(with 256 PGs) with some crush rule that doesn't allow it to find
suitable placements, like "replication = 3" and "failure domain =
host" but only having 2 hosts, or something to that effect. Unless you
add hosts (in my example), this will not "fix itself" until you either
add hosts, or change the crush rules to something less reliable.
17 Mar
17 Mar
8:30 a.m.
Yeap - that was the issue: an incorrect CRUSH rule
Thanks for the help
Dulux-Oz
1160
days inactive
1161
days old
6 comments
3 participants
participants (3)
-
ashley＠amerrick.co.uk -
duluxoz -
Janne Johansson