30 Mar
2021
30 Mar
'21
4 p.m.
Hello,
i run a ceph Nautilus cluster with 9 hosts and 144 OSDs. Last night we
lost two disks, so two OSDs (67,90) are down. The two disks are on two
different hosts. A third ODS on a third host repotrts slow ops. ceph is
repairing at the moment.
Pools affected are eg these ones:
pool 35 'pxa-rbd' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 256 pgp_num 256 last_change 192082 lfor
0/27841/27845 flags hashpspool,selfmanaged_snaps stripe_width 0
pg_num_min 128 target_size_ratio 0.0001 application rbd
pool 36 'pxa-ec' erasure size 6 min_size 5 crush_rule 7 object_hash
rjenkins pg_num 512 pgp_num 512 last_change 192177 lfor 0/172580/172578
flags hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 16384
pg_num_min 512 target_size_ratio 0.15 application rbd
At the mmoment the proxmox-cluster using storage from the seperate ceph
cluster hangs. The ppols with date are erasure coded with the following
profile:
crush-device-class=
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=2
plugin=jerasure
technique=reed_sol_van
w=8
What I do not understand is why access on the virtualization seem to
block. Could that be related to min_size of the pools cause this
behaviour? How can I find out if this is true or what else is causing
the blocking behaviour seen?
This is the current status:
health: HEALTH_WARN
Reduced data availability: 1 pg inactive, 1 pg incomplete
Degraded data redundancy: 42384/130014984 objects degraded
(0.033%), 4 pgs degraded, 5 pgs undersized
15 daemons have recently crashed
150 slow ops, oldest one blocked for 15901 sec, daemons
[osd.60,osd.67] have slow ops.
services:
mon: 3 daemons, quorum ceph2,ceph5,ceph8 (age 4h)
mgr: ceph2(active, since 7w), standbys: ceph5, ceph8, ceph-admin
mds: cephfsrz:1 {0=ceph6=up:active} 2 up:standby
osd: 144 osds: 142 up (since 4h), 142 in (since 5h); 6 remapped pgs
task status:
scrub status:
mds.ceph6: idle
data:
pools: 15 pools, 2632 pgs
objects: 21.70M objects, 80 TiB
usage: 139 TiB used, 378 TiB / 517 TiB avail
pgs: 0.038% pgs not active
42384/130014984 objects degraded (0.033%)
2623 active+clean
3 active+undersized+degraded+remapped+backfilling
3 active+clean+scrubbing+deep
1 active+undersized+degraded+remapped+backfill_wait
1 active+undersized+remapped+backfill_wait
1 remapped+incomplete
io:
client: 2.2 MiB/s rd, 3.6 MiB/s wr, 8 op/s rd, 179 op/s wr
recovery: 51 MiB/s, 12 objects/s
Thanks a lot
Rainer
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
30 Mar
30 Mar
6:41 p.m.
Hi,
from what you've sent my conclusion about the stalled I/O would be
indeed the min_size of the EC pool.
There's only one PG reported as incomplete, I assume that is the EC
pool, not the replicated pxa-rbd, right? Both pools are for rbd so I'm
guessing the rbd headers are in pxa-rbd while the data is stored in
pxa-ec, could you confirm that?
You could add 'ceph health detail' output to your question to see
which PG is incomplete.
I assume that both down OSDs are in the acting set of the inactive PG,
and since the pool's min_size is 5 the I/O pauses. If you can't wait
for recovery to finish and can't bring up at least one of those OSDs
you could set the min_size of pxa-ec to 4, but if you do, be aware
that one more disk failure could mean data loss! So think carefully
about it (maybe you could instead speed up recovery?) and don't forget
to increase min_size back to 5 when the recovery has finished, that's
very important!
Regards,
Eugen
Zitat von Rainer Krienke <krienke(a)uni-koblenz.de>de>:
Hello,
i run a ceph Nautilus cluster with 9 hosts and 144 OSDs. Last night
we lost two disks, so two OSDs (67,90) are down. The two disks are
on two different hosts. A third ODS on a third host repotrts slow
ops. ceph is repairing at the moment.
Pools affected are eg these ones:
pool 35 'pxa-rbd' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 256 pgp_num 256 last_change 192082 lfor
0/27841/27845 flags hashpspool,selfmanaged_snaps stripe_width 0
pg_num_min 128 target_size_ratio 0.0001 application rbd
pool 36 'pxa-ec' erasure size 6 min_size 5 crush_rule 7 object_hash
rjenkins pg_num 512 pgp_num 512 last_change 192177 lfor
0/172580/172578 flags hashpspool,ec_overwrites,selfmanaged_snaps
stripe_width 16384 pg_num_min 512 target_size_ratio 0.15 application
rbd
At the mmoment the proxmox-cluster using storage from the seperate
ceph cluster hangs. The ppols with date are erasure coded with the
following profile:
crush-device-class=
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=2
plugin=jerasure
technique=reed_sol_van
w=8
What I do not understand is why access on the virtualization seem to
block. Could that be related to min_size of the pools cause this
behaviour? How can I find out if this is true or what else is
causing the blocking behaviour seen?
This is the current status:
health: HEALTH_WARN
Reduced data availability: 1 pg inactive, 1 pg incomplete
Degraded data redundancy: 42384/130014984 objects
degraded (0.033%), 4 pgs degraded, 5 pgs undersized
15 daemons have recently crashed
150 slow ops, oldest one blocked for 15901 sec, daemons
[osd.60,osd.67] have slow ops.
services:
mon: 3 daemons, quorum ceph2,ceph5,ceph8 (age 4h)
mgr: ceph2(active, since 7w), standbys: ceph5, ceph8, ceph-admin
mds: cephfsrz:1 {0=ceph6=up:active} 2 up:standby
osd: 144 osds: 142 up (since 4h), 142 in (since 5h); 6 remapped pgs
task status:
scrub status:
mds.ceph6: idle
data:
pools: 15 pools, 2632 pgs
objects: 21.70M objects, 80 TiB
usage: 139 TiB used, 378 TiB / 517 TiB avail
pgs: 0.038% pgs not active
42384/130014984 objects degraded (0.033%)
2623 active+clean
3 active+undersized+degraded+remapped+backfilling
3 active+clean+scrubbing+deep
1 active+undersized+degraded+remapped+backfill_wait
1 active+undersized+remapped+backfill_wait
1 remapped+incomplete
io:
client: 2.2 MiB/s rd, 3.6 MiB/s wr, 8 op/s rd, 179 op/s wr
recovery: 51 MiB/s, 12 objects/s
Thanks a lot
Rainer
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax:
+49261287 1001312
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:05 p.m.
Hello,
yes your assumptions are correct pxa-rbd ist the metadata pool for
pxa-ec which uses a erasure coding 4+2 profile.
In the last hours ceph repaired most of the damage. One inactive PG
remained and in ceph health detail then told me:
---------
HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg incomplete;
15 daemons have recently crashed; 150 slow ops, oldest one blocked for
26716 sec, daemons [osd.60,osd.67] have slow ops.
PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg incomplete
pg 36.15b is remapped+incomplete, acting
[60,2147483647,23,96,2147483647,36] (reducing pool pxa-ec min_size from
5 may help; search ceph.com/docs for 'incomplete')
RECENT_CRASH 15 daemons have recently crashed
osd.90 crashed on host ceph6 at 2021-03-29 21:14:10.442314Z
osd.67 crashed on host ceph5 at 2021-03-30 02:21:23.944205Z
osd.67 crashed on host ceph5 at 2021-03-30 01:39:14.452610Z
osd.90 crashed on host ceph6 at 2021-03-29 21:14:24.222223Z
osd.67 crashed on host ceph5 at 2021-03-30 02:35:43.373845Z
osd.67 crashed on host ceph5 at 2021-03-30 01:19:58.762393Z
osd.67 crashed on host ceph5 at 2021-03-30 02:09:42.297941Z
osd.67 crashed on host ceph5 at 2021-03-30 02:28:29.981528Z
osd.67 crashed on host ceph5 at 2021-03-30 01:50:05.374278Z
osd.90 crashed on host ceph6 at 2021-03-29 21:13:51.896849Z
osd.67 crashed on host ceph5 at 2021-03-30 02:00:22.593745Z
osd.67 crashed on host ceph5 at 2021-03-30 01:29:39.170134Z
osd.90 crashed on host ceph6 at 2021-03-29 21:14:38.114768Z
osd.67 crashed on host ceph5 at 2021-03-30 00:54:06.629808Z
osd.67 crashed on host ceph5 at 2021-03-30 01:10:21.824447Z
---------
All osds except for 67 and 90 are up and I followed the hint in health
detail and lowered min_size from 5 to 4 for pxa-ec. Since then ceph is
again repairing and in between some VMs in the attached proxmox cluster
are working again.
So I hope that after repairing all PGs are up, so that I can restart all
VMs again.
Thanks
Rainer
Am 30.03.21 um 11:41 schrieb Eugen Block:
Hi,
from what you've sent my conclusion about the stalled I/O would be
indeed the min_size of the EC pool.
There's only one PG reported as incomplete, I assume that is the EC
pool, not the replicated pxa-rbd, right? Both pools are for rbd so I'm
guessing the rbd headers are in pxa-rbd while the data is stored in
pxa-ec, could you confirm that?
You could add 'ceph health detail' output to your question to see which
PG is incomplete.
I assume that both down OSDs are in the acting set of the inactive PG,
and since the pool's min_size is 5 the I/O pauses. If you can't wait for
recovery to finish and can't bring up at least one of those OSDs you
could set the min_size of pxa-ec to 4, but if you do, be aware that one
more disk failure could mean data loss! So think carefully about it
(maybe you could instead speed up recovery?) and don't forget to
increase min_size back to 5 when the recovery has finished, that's very
important!
Regards,
Eugen
Zitat von Rainer Krienke <krienke(a)uni-koblenz.de>de>:
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
Hello,
i run a ceph Nautilus cluster with 9 hosts and 144 OSDs. Last night we
lost two disks, so two OSDs (67,90) are down. The two disks are on two
different hosts. A third ODS on a third host repotrts slow ops. ceph
is repairing at the moment.
Pools affected are eg these ones:
pool 35 'pxa-rbd' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 256 pgp_num 256 last_change 192082 lfor
0/27841/27845 flags hashpspool,selfmanaged_snaps stripe_width 0
pg_num_min 128 target_size_ratio 0.0001 application rbd
pool 36 'pxa-ec' erasure size 6 min_size 5 crush_rule 7 object_hash
rjenkins pg_num 512 pgp_num 512 last_change 192177 lfor
0/172580/172578 flags hashpspool,ec_overwrites,selfmanaged_snaps
stripe_width 16384 pg_num_min 512 target_size_ratio 0.15 application rbd
At the mmoment the proxmox-cluster using storage from the seperate
ceph cluster hangs. The ppols with date are erasure coded with the
following profile:
crush-device-class=
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=2
plugin=jerasure
technique=reed_sol_van
w=8
What I do not understand is why access on the virtualization seem to
block. Could that be related to min_size of the pools cause this
behaviour? How can I find out if this is true or what else is causing
the blocking behaviour seen?
This is the current status:
health: HEALTH_WARN
Reduced data availability: 1 pg inactive, 1 pg incomplete
Degraded data redundancy: 42384/130014984 objects degraded
(0.033%), 4 pgs degraded, 5 pgs undersized
15 daemons have recently crashed
150 slow ops, oldest one blocked for 15901 sec, daemons
[osd.60,osd.67] have slow ops.
services:
mon: 3 daemons, quorum ceph2,ceph5,ceph8 (age 4h)
mgr: ceph2(active, since 7w), standbys: ceph5, ceph8, ceph-admin
mds: cephfsrz:1 {0=ceph6=up:active} 2 up:standby
osd: 144 osds: 142 up (since 4h), 142 in (since 5h); 6 remapped pgs
task status:
scrub status:
mds.ceph6: idle
data:
pools: 15 pools, 2632 pgs
objects: 21.70M objects, 80 TiB
usage: 139 TiB used, 378 TiB / 517 TiB avail
pgs: 0.038% pgs not active
42384/130014984 objects degraded (0.033%)
2623 active+clean
3 active+undersized+degraded+remapped+backfilling
3 active+clean+scrubbing+deep
1 active+undersized+degraded+remapped+backfill_wait
1 active+undersized+remapped+backfill_wait
1 remapped+incomplete
io:
client: 2.2 MiB/s rd, 3.6 MiB/s wr, 8 op/s rd, 179 op/s wr
recovery: 51 MiB/s, 12 objects/s
Thanks a lot
Rainer
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287
1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
8:20 p.m.
Hi, this is odd. The problem with recovery when sufficiently many but less than min_size
shards are present should have been resolved with osd_allow_recovery_below_min_size=true.
It is really dangerous to reduce min_size below k+1 and, in fact, should never be
necessary for recovery. Can you check if this option is present and set to true? If it is
not working as intended, a tracker ticker might be in order.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Rainer Krienke <krienke(a)uni-koblenz.de>
Sent: 30 March 2021 13:05:56
To: Eugen Block; ceph-users(a)ceph.io
Subject: [ceph-users] Re: ceph Nautilus lost two disk over night everything hangs
Hello,
yes your assumptions are correct pxa-rbd ist the metadata pool for
pxa-ec which uses a erasure coding 4+2 profile.
In the last hours ceph repaired most of the damage. One inactive PG
remained and in ceph health detail then told me:
---------
HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg incomplete;
15 daemons have recently crashed; 150 slow ops, oldest one blocked for
26716 sec, daemons [osd.60,osd.67] have slow ops.
PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg incomplete
pg 36.15b is remapped+incomplete, acting
[60,2147483647,23,96,2147483647,36] (reducing pool pxa-ec min_size from
5 may help; search ceph.com/docs for 'incomplete')
RECENT_CRASH 15 daemons have recently crashed
osd.90 crashed on host ceph6 at 2021-03-29 21:14:10.442314Z
osd.67 crashed on host ceph5 at 2021-03-30 02:21:23.944205Z
osd.67 crashed on host ceph5 at 2021-03-30 01:39:14.452610Z
osd.90 crashed on host ceph6 at 2021-03-29 21:14:24.222223Z
osd.67 crashed on host ceph5 at 2021-03-30 02:35:43.373845Z
osd.67 crashed on host ceph5 at 2021-03-30 01:19:58.762393Z
osd.67 crashed on host ceph5 at 2021-03-30 02:09:42.297941Z
osd.67 crashed on host ceph5 at 2021-03-30 02:28:29.981528Z
osd.67 crashed on host ceph5 at 2021-03-30 01:50:05.374278Z
osd.90 crashed on host ceph6 at 2021-03-29 21:13:51.896849Z
osd.67 crashed on host ceph5 at 2021-03-30 02:00:22.593745Z
osd.67 crashed on host ceph5 at 2021-03-30 01:29:39.170134Z
osd.90 crashed on host ceph6 at 2021-03-29 21:14:38.114768Z
osd.67 crashed on host ceph5 at 2021-03-30 00:54:06.629808Z
osd.67 crashed on host ceph5 at 2021-03-30 01:10:21.824447Z
---------
All osds except for 67 and 90 are up and I followed the hint in health
detail and lowered min_size from 5 to 4 for pxa-ec. Since then ceph is
again repairing and in between some VMs in the attached proxmox cluster
are working again.
So I hope that after repairing all PGs are up, so that I can restart all
VMs again.
Thanks
Rainer
Am 30.03.21 um 11:41 schrieb Eugen Block:
Hi,
from what you've sent my conclusion about the stalled I/O would be
indeed the min_size of the EC pool.
There's only one PG reported as incomplete, I assume that is the EC
pool, not the replicated pxa-rbd, right? Both pools are for rbd so I'm
guessing the rbd headers are in pxa-rbd while the data is stored in
pxa-ec, could you confirm that?
You could add 'ceph health detail' output to your question to see which
PG is incomplete.
I assume that both down OSDs are in the acting set of the inactive PG,
and since the pool's min_size is 5 the I/O pauses. If you can't wait for
recovery to finish and can't bring up at least one of those OSDs you
could set the min_size of pxa-ec to 4, but if you do, be aware that one
more disk failure could mean data loss! So think carefully about it
(maybe you could instead speed up recovery?) and don't forget to
increase min_size back to 5 when the recovery has finished, that's very
important!
Regards,
Eugen
Zitat von Rainer Krienke <krienke(a)uni-koblenz.de>de>:
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hello,
i run a ceph Nautilus cluster with 9 hosts and 144 OSDs. Last night we
lost two disks, so two OSDs (67,90) are down. The two disks are on two
different hosts. A third ODS on a third host repotrts slow ops. ceph
is repairing at the moment.
Pools affected are eg these ones:
pool 35 'pxa-rbd' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 256 pgp_num 256 last_change 192082 lfor
0/27841/27845 flags hashpspool,selfmanaged_snaps stripe_width 0
pg_num_min 128 target_size_ratio 0.0001 application rbd
pool 36 'pxa-ec' erasure size 6 min_size 5 crush_rule 7 object_hash
rjenkins pg_num 512 pgp_num 512 last_change 192177 lfor
0/172580/172578 flags hashpspool,ec_overwrites,selfmanaged_snaps
stripe_width 16384 pg_num_min 512 target_size_ratio 0.15 application rbd
At the mmoment the proxmox-cluster using storage from the seperate
ceph cluster hangs. The ppols with date are erasure coded with the
following profile:
crush-device-class=
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=2
plugin=jerasure
technique=reed_sol_van
w=8
What I do not understand is why access on the virtualization seem to
block. Could that be related to min_size of the pools cause this
behaviour? How can I find out if this is true or what else is causing
the blocking behaviour seen?
This is the current status:
health: HEALTH_WARN
Reduced data availability: 1 pg inactive, 1 pg incomplete
Degraded data redundancy: 42384/130014984 objects degraded
(0.033%), 4 pgs degraded, 5 pgs undersized
15 daemons have recently crashed
150 slow ops, oldest one blocked for 15901 sec, daemons
[osd.60,osd.67] have slow ops.
services:
mon: 3 daemons, quorum ceph2,ceph5,ceph8 (age 4h)
mgr: ceph2(active, since 7w), standbys: ceph5, ceph8, ceph-admin
mds: cephfsrz:1 {0=ceph6=up:active} 2 up:standby
osd: 144 osds: 142 up (since 4h), 142 in (since 5h); 6 remapped pgs
task status:
scrub status:
mds.ceph6: idle
data:
pools: 15 pools, 2632 pgs
objects: 21.70M objects, 80 TiB
usage: 139 TiB used, 378 TiB / 517 TiB avail
pgs: 0.038% pgs not active
42384/130014984 objects degraded (0.033%)
2623 active+clean
3 active+undersized+degraded+remapped+backfilling
3 active+clean+scrubbing+deep
1 active+undersized+degraded+remapped+backfill_wait
1 active+undersized+remapped+backfill_wait
1 remapped+incomplete
io:
client: 2.2 MiB/s rd, 3.6 MiB/s wr, 8 op/s rd, 179 op/s wr
recovery: 51 MiB/s, 12 objects/s
Thanks a lot
Rainer
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287
1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:30 p.m.
Hello Frank,
the option is actually set. On one of my monitors:
# ceph daemon /var/run/ceph/ceph-mon.*.asok config show|grep
osd_allow_recovery_below_min_size
"osd_allow_recovery_below_min_size": "true",
Thank you very much
Rainer
Am 30.03.21 um 13:20 schrieb Frank Schilder:
Hi, this is odd. The problem with recovery when
sufficiently many but less than min_size shards are present should have been resolved with
osd_allow_recovery_below_min_size=true. It is really dangerous to reduce min_size below
k+1 and, in fact, should never be necessary for recovery. Can you check if this option is
present and set to true? If it is not working as intended, a tracker ticker might be in
order.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
9:53 p.m.
Dear Rainer,
hmm, maybe the option is ignored or not implemented properly. This option set to true
should have the same effect as reducing min_size *except* that new writes will not go to
non-redundant storage. When reducing min-size, a critically degraded PG will accept new
writes, which is the danger of data-loss mentioned before and avoided if only recovery ops
are allowed on such PGs.
Can you open a tracker about your observation that reducing min-size was necessary and
helped despite osd_allow_recovery_below_min_size=true?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Rainer Krienke <krienke(a)uni-koblenz.de>
Sent: 30 March 2021 13:30:00
To: Frank Schilder; Eugen Block; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ceph Nautilus lost two disk over night everything hangs
Hello Frank,
the option is actually set. On one of my monitors:
# ceph daemon /var/run/ceph/ceph-mon.*.asok config show|grep
osd_allow_recovery_below_min_size
"osd_allow_recovery_below_min_size": "true",
Thank you very much
Rainer
Am 30.03.21 um 13:20 schrieb Frank Schilder:
Hi, this is odd. The problem with recovery when
sufficiently many but less than min_size shards are present should have been resolved with
osd_allow_recovery_below_min_size=true. It is really dangerous to reduce min_size below
k+1 and, in fact, should never be necessary for recovery. Can you check if this option is
present and set to true? If it is not working as intended, a tracker ticker might be in
order.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
9:58 p.m.
Sorry about the flow of messages.
I forgot to mention this. Looking at other replies, in particular, that the PG in question
remained at 4 out of 6 OSDs until you reduced min_size might indicate that peering was
blocked for some reason, but completed after the reduction. If this was the order of
events, it seems like an important detail.
It is true that recovery will need to wait until the PG has the missing OSDs assigned. If
this assignment is somehow blocked by min-size>k, the flag
osd_allow_recovery_below_min_size itself will have no effect.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder
Sent: 30 March 2021 14:53:18
To: Rainer Krienke; Eugen Block; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ceph Nautilus lost two disk over night everything hangs
Dear Rainer,
hmm, maybe the option is ignored or not implemented properly. This option set to true
should have the same effect as reducing min_size *except* that new writes will not go to
non-redundant storage. When reducing min-size, a critically degraded PG will accept new
writes, which is the danger of data-loss mentioned before and avoided if only recovery ops
are allowed on such PGs.
Can you open a tracker about your observation that reducing min-size was necessary and
helped despite osd_allow_recovery_below_min_size=true?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Rainer Krienke <krienke(a)uni-koblenz.de>
Sent: 30 March 2021 13:30:00
To: Frank Schilder; Eugen Block; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ceph Nautilus lost two disk over night everything hangs
Hello Frank,
the option is actually set. On one of my monitors:
# ceph daemon /var/run/ceph/ceph-mon.*.asok config show|grep
osd_allow_recovery_below_min_size
"osd_allow_recovery_below_min_size": "true",
Thank you very much
Rainer
Am 30.03.21 um 13:20 schrieb Frank Schilder:
Hi, this is odd. The problem with recovery when
sufficiently many but less than min_size shards are present should have been resolved with
osd_allow_recovery_below_min_size=true. It is really dangerous to reduce min_size below
k+1 and, in fact, should never be necessary for recovery. Can you check if this option is
present and set to true? If it is not working as intended, a tracker ticker might be in
order.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
10:23 p.m.
I thought that recovery below min_size for EC pools wasn't expected to work
until Octopus. From the Octopus release notes: "Ceph will allow recovery
below min_size for Erasure coded pools, wherever possible."
Josh
On Tue, Mar 30, 2021 at 6:53 AM Frank Schilder <frans(a)dtu.dk> wrote:
Dear Rainer,
hmm, maybe the option is ignored or not implemented properly. This option
set to true should have the same effect as reducing min_size *except* that
new writes will not go to non-redundant storage. When reducing min-size, a
critically degraded PG will accept new writes, which is the danger of
data-loss mentioned before and avoided if only recovery ops are allowed on
such PGs.
Can you open a tracker about your observation that reducing min-size was
necessary and helped despite osd_allow_recovery_below_min_size=true?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Rainer Krienke <krienke(a)uni-koblenz.de>
Sent: 30 March 2021 13:30:00
To: Frank Schilder; Eugen Block; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ceph Nautilus lost two disk over night
everything hangs
Hello Frank,
the option is actually set. On one of my monitors:
# ceph daemon /var/run/ceph/ceph-mon.*.asok config show|grep
osd_allow_recovery_below_min_size
"osd_allow_recovery_below_min_size": "true",
Thank you very much
Rainer
Am 30.03.21 um 13:20 schrieb Frank Schilder:
Hi, this is odd. The problem with recovery when
sufficiently many but
less than min_size shards are present should have been
resolved with
osd_allow_recovery_below_min_size=true. It is really dangerous to reduce
min_size below k+1 and, in fact, should never be necessary for recovery.
Can you check if this option is present and set to true? If it is not
working as intended, a tracker ticker might be in order.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287
1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:34 p.m.
Ahh, right. I saw it fixed here https://tracker.ceph.com/issues/18749 a long time ago, but
it seems the back-port never happened.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Josh Baergen <jbaergen(a)digitalocean.com>
Sent: 30 March 2021 15:23:10
To: Frank Schilder
Cc: Rainer Krienke; Eugen Block; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ceph Nautilus lost two disk over night everything hangs
I thought that recovery below min_size for EC pools wasn't expected to work until
Octopus. From the Octopus release notes: "Ceph will allow recovery below min_size for
Erasure coded pools, wherever possible."
Josh
On Tue, Mar 30, 2021 at 6:53 AM Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
Dear Rainer,
hmm, maybe the option is ignored or not implemented properly. This option set to true
should have the same effect as reducing min_size *except* that new writes will not go to
non-redundant storage. When reducing min-size, a critically degraded PG will accept new
writes, which is the danger of data-loss mentioned before and avoided if only recovery ops
are allowed on such PGs.
Can you open a tracker about your observation that reducing min-size was necessary and
helped despite osd_allow_recovery_below_min_size=true?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Rainer Krienke <krienke@uni-koblenz.de<mailto:krienke@uni-koblenz.de>>
Sent: 30 March 2021 13:30:00
To: Frank Schilder; Eugen Block; ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: [ceph-users] Re: ceph Nautilus lost two disk over night everything hangs
Hello Frank,
the option is actually set. On one of my monitors:
# ceph daemon /var/run/ceph/ceph-mon.*.asok config show|grep
osd_allow_recovery_below_min_size
"osd_allow_recovery_below_min_size": "true",
Thank you very much
Rainer
Am 30.03.21 um 13:20 schrieb Frank Schilder:
Hi, this is odd. The problem with recovery when
sufficiently many but less than min_size shards are present should have been resolved with
osd_allow_recovery_below_min_size=true. It is really dangerous to reduce min_size below
k+1 and, in fact, should never be necessary for recovery. Can you check if this option is
present and set to true? If it is not working as intended, a tracker ticker might be in
order.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
8:32 p.m.
Hi,
On 30.03.21 13:05, Rainer Krienke wrote:
Hello,
yes your assumptions are correct pxa-rbd ist the metadata pool for
pxa-ec which uses a erasure coding 4+2 profile.
In the last hours ceph repaired most of the damage. One inactive PG
remained and in ceph health detail then told me:
---------
HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg incomplete;
15 daemons have recently crashed; 150 slow ops, oldest one blocked for
26716 sec, daemons [osd.60,osd.67] have slow ops.
PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg incomplete
pg 36.15b is remapped+incomplete, acting
[60,2147483647,23,96,2147483647,36] (reducing pool pxa-ec min_size
from 5 may help; search ceph.com/docs for 'incomplete')
*snipsnap*
2147483647 is (uint32)(-1), which mean no associated OSD. So this PG
does not have six independent OSDs, and no backfilling is happening
since there are no targets to backfill.
You mentioned 9 hosts, so if you use a simple host based crush rule ceph
should be able to find new OSDs for that PG. If you do not use standard
crush rules please check that ceph is able to derive enough OSDs to
satisfy the PG requirements (six different OSDs).
The 'incomplete' part might be a problem. If just a chunk would be
missing, the state should be undersized, not incomplete...
Regards,
Burkhard
9:27 p.m.
Hello,
in between ceph is runing again normally, except for the two osds that
are down because of the failed disks.
What really helped in my situation was to lower min_size from 5 (k+1)
to 4 in my 4+2 erasure code setup. So I am also greatful for the
programmer who put the helping hint in ceph health detail for this
situation.
Thanks very much to everyone who answered my request to help out.
What is left now is to replace the disks and then bring the two osds up
again.
Have a nice day
Rainer
Am 30.03.21 um 13:32 schrieb Burkhard Linke:
Hi,
On 30.03.21 13:05, Rainer Krienke wrote:
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
Hello,
yes your assumptions are correct pxa-rbd ist the metadata pool for
pxa-ec which uses a erasure coding 4+2 profile.
In the last hours ceph repaired most of the damage. One inactive PG
remained and in ceph health detail then told me:
---------
HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg incomplete;
15 daemons have recently crashed; 150 slow ops, oldest one blocked for
26716 sec, daemons [osd.60,osd.67] have slow ops.
PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg incomplete
pg 36.15b is remapped+incomplete, acting
[60,2147483647,23,96,2147483647,36] (reducing pool pxa-ec min_size
from 5 may help; search ceph.com/docs for 'incomplete')
*snipsnap*
2147483647 is (uint32)(-1), which mean no associated OSD. So this PG
does not have six independent OSDs, and no backfilling is happening
since there are no targets to backfill.
You mentioned 9 hosts, so if you use a simple host based crush rule ceph
should be able to find new OSDs for that PG. If you do not use standard
crush rules please check that ceph is able to derive enough OSDs to
satisfy the PG requirements (six different OSDs).
The 'incomplete' part might be a problem. If just a chunk would be
missing, the state should be undersized, not incomplete...
Regards,
Burkhard
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1125
days inactive
1125
days old
10 comments
5 participants
participants (5)
-
Burkhard Linke -
Eugen Block -
Frank Schilder -
Josh Baergen -
Rainer Krienke