18 May
2020
18 May
'20
10:25 p.m.
In a recent cluster reorganization, we ended up with a lot of
undersized/degraded PGs and a day of recovery from them, when all we
expected was moving some data around. After retracing my steps, I found
something odd. If I crush reweight an OSD to 0 while it is down - it
results in the PGs of that OSD ending up degraded even after the OSD is
restarted. If I do the same reweighting while the OSD is up - data gets
moved without any degraded/undersized states. I would not expect this -
so I wonder if this is a bug or is somehow intended. This is on ceph
Nautilus 14.2.8. Below are the details.
Andras
First the case that works as I would expect:
# Healthy cluster ...
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 2m), 270 in (since 4h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
# Reweight an OSD to 0
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
reweighted item id 0 name 'osd.0' to 0 in crush map
# Crush map changes - data movement is set up, no degraded PGs:
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 10m), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 2562045/232996662 objects misplaced (1.100%)
5137 active+clean
172 active+remapped+backfilling
3 active+remapped+backfill_wait
# Reweight it back to the original weight
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
# Cluster goes back to clean
reweighted item id 0 name 'osd.0' to 8 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 11m), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
#
# Now the problematic case
#
# Stop an OSD
[root@xorphosd00 ~]# systemctl stop ceph-osd@0
# We get degraded PGs - as expected
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
1 osds down
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 269 up (since 16s), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
5230 active+clean
82 active+undersized+degraded
# Reweight the OSD to 0:
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
# Still degraded - as expected
reweighted item id 0 name 'osd.0' to 0 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
1 osds down
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 269 up (since 59s), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
1688081/232996662 objects misplaced (0.725%)
5137 active+clean
93 active+remapped+backfilling
82 active+undersized+degraded+remapped+backfilling
# Restarting the OSD
[root@xorphosd00 ~]# systemctl start ceph-osd@0
# And the PGs still stay degraded - THIS IS UNEXPECTED!!!
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 14s), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
1688081/232996662 objects misplaced (0.725%)
5137 active+clean
93 active+remapped+backfilling
82 active+undersized+degraded+remapped+backfilling
# Now for something even more odd - reweight the OSD back to its
original weigh
# and all the data gets magically FOUND again on that OSD!!!
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
reweighted item id 0 name 'osd.0' to 8 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 51s), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
19 May
19 May
9:49 a.m.
Hi Andreas,
I made exactly the same observation in another scenario. I added some OSDs while other
OSDs were down.
This is expected.
The crush map is an a-priory algorithm to compute the location of objects without
contacting a central server. Hence, *any*change of a crush map while an OSD is down will
lead to a change of locations of objects/PGs of the down OSD. Consequently, these
objects/PGs will become degraded, because no up OSD reports these. Once the peering is
over after setting the weight to 0, the cluster must assume they are lost.
Changing a weight is a change of the crush map.
The way to get the cluster to re-scan after starting the down OSDs is to restore the crush
map to exactly the state as it was before the OSD went down. In your case,
- starting the down OSD,
- setting weight back to original value
will find all missing objects. After the cluster is clean, set the weight back to 0 and
now the OSD will be vacated as expected.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Andras Pataki <apataki(a)flatironinstitute.org>
Sent: 18 May 2020 22:25:37
To: ceph-users
Subject: [ceph-users] Reweighting OSD while down results in undersized+degraded PGs
In a recent cluster reorganization, we ended up with a lot of
undersized/degraded PGs and a day of recovery from them, when all we
expected was moving some data around. After retracing my steps, I found
something odd. If I crush reweight an OSD to 0 while it is down - it
results in the PGs of that OSD ending up degraded even after the OSD is
restarted. If I do the same reweighting while the OSD is up - data gets
moved without any degraded/undersized states. I would not expect this -
so I wonder if this is a bug or is somehow intended. This is on ceph
Nautilus 14.2.8. Below are the details.
Andras
First the case that works as I would expect:
# Healthy cluster ...
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 2m), 270 in (since 4h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
# Reweight an OSD to 0
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
reweighted item id 0 name 'osd.0' to 0 in crush map
# Crush map changes - data movement is set up, no degraded PGs:
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 10m), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 2562045/232996662 objects misplaced (1.100%)
5137 active+clean
172 active+remapped+backfilling
3 active+remapped+backfill_wait
# Reweight it back to the original weight
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
# Cluster goes back to clean
reweighted item id 0 name 'osd.0' to 8 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 11m), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
#
# Now the problematic case
#
# Stop an OSD
[root@xorphosd00 ~]# systemctl stop ceph-osd@0
# We get degraded PGs - as expected
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
1 osds down
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 269 up (since 16s), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
5230 active+clean
82 active+undersized+degraded
# Reweight the OSD to 0:
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
# Still degraded - as expected
reweighted item id 0 name 'osd.0' to 0 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
1 osds down
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 269 up (since 59s), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
1688081/232996662 objects misplaced (0.725%)
5137 active+clean
93 active+remapped+backfilling
82 active+undersized+degraded+remapped+backfilling
# Restarting the OSD
[root@xorphosd00 ~]# systemctl start ceph-osd@0
# And the PGs still stay degraded - THIS IS UNEXPECTED!!!
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 14s), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
1688081/232996662 objects misplaced (0.725%)
5137 active+clean
93 active+remapped+backfilling
82 active+undersized+degraded+remapped+backfilling
# Now for something even more odd - reweight the OSD back to its
original weigh
# and all the data gets magically FOUND again on that OSD!!!
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
reweighted item id 0 name 'osd.0' to 8 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 51s), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:57 p.m.
Hi Frank,
My understanding was that once a cluster is in a degraded state (an OSD
is down), ceph stores all changed cluster maps until the cluster is
healthy again exactly for the reason of finding missing objects. If
there is a real disaster of some kind, and many OSDs go up and down at
various times, we have to have a way of retracing where parts of a PG
were in the past. And ... most of the time this does work - I just
don't understand why this current scenario is different.
Andras
On 5/19/20 3:49 AM, Frank Schilder wrote:
Hi Andreas,
I made exactly the same observation in another scenario. I added some OSDs while other
OSDs were down.
This is expected.
The crush map is an a-priory algorithm to compute the location of objects without
contacting a central server. Hence, *any*change of a crush map while an OSD is down will
lead to a change of locations of objects/PGs of the down OSD. Consequently, these
objects/PGs will become degraded, because no up OSD reports these. Once the peering is
over after setting the weight to 0, the cluster must assume they are lost.
Changing a weight is a change of the crush map.
The way to get the cluster to re-scan after starting the down OSDs is to restore the
crush map to exactly the state as it was before the OSD went down. In your case,
- starting the down OSD,
- setting weight back to original value
will find all missing objects. After the cluster is clean, set the weight back to 0 and
now the OSD will be vacated as expected.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Andras Pataki <apataki(a)flatironinstitute.org>
Sent: 18 May 2020 22:25:37
To: ceph-users
Subject: [ceph-users] Reweighting OSD while down results in undersized+degraded PGs
In a recent cluster reorganization, we ended up with a lot of
undersized/degraded PGs and a day of recovery from them, when all we
expected was moving some data around. After retracing my steps, I found
something odd. If I crush reweight an OSD to 0 while it is down - it
results in the PGs of that OSD ending up degraded even after the OSD is
restarted. If I do the same reweighting while the OSD is up - data gets
moved without any degraded/undersized states. I would not expect this -
so I wonder if this is a bug or is somehow intended. This is on ceph
Nautilus 14.2.8. Below are the details.
Andras
First the case that works as I would expect:
# Healthy cluster ...
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 2m), 270 in (since 4h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
# Reweight an OSD to 0
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
reweighted item id 0 name 'osd.0' to 0 in crush map
# Crush map changes - data movement is set up, no degraded PGs:
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 10m), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 2562045/232996662 objects misplaced (1.100%)
5137 active+clean
172 active+remapped+backfilling
3 active+remapped+backfill_wait
# Reweight it back to the original weight
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
# Cluster goes back to clean
reweighted item id 0 name 'osd.0' to 8 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 11m), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
#
# Now the problematic case
#
# Stop an OSD
[root@xorphosd00 ~]# systemctl stop ceph-osd@0
# We get degraded PGs - as expected
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
1 osds down
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 269 up (since 16s), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
5230 active+clean
82 active+undersized+degraded
# Reweight the OSD to 0:
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
# Still degraded - as expected
reweighted item id 0 name 'osd.0' to 0 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
1 osds down
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 269 up (since 59s), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
1688081/232996662 objects misplaced (0.725%)
5137 active+clean
93 active+remapped+backfilling
82 active+undersized+degraded+remapped+backfilling
# Restarting the OSD
[root@xorphosd00 ~]# systemctl start ceph-osd@0
# And the PGs still stay degraded - THIS IS UNEXPECTED!!!
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 14s), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
1688081/232996662 objects misplaced (0.725%)
5137 active+clean
93 active+remapped+backfilling
82 active+undersized+degraded+remapped+backfilling
# Now for something even more odd - reweight the OSD back to its
original weigh
# and all the data gets magically FOUND again on that OSD!!!
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
reweighted item id 0 name 'osd.0' to 8 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 51s), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:23 p.m.
Hi Andreas,
the cluster map and crush map are not the same thing. If you change the crush map while
the cluster is in degraded state, you basically modify this history of cluster maps
explicitly and have to live with the consequences (keeping history under crush map changes
is limited to up+in OSDs). Only OSDs that can peer are able to respond to changes of the
crush map.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Andras Pataki <apataki(a)flatironinstitute.org>
Sent: 19 May 2020 15:57:49
To: Frank Schilder; ceph-users
Subject: Re: [ceph-users] Reweighting OSD while down results in undersized+degraded PGs
Hi Frank,
My understanding was that once a cluster is in a degraded state (an OSD
is down), ceph stores all changed cluster maps until the cluster is
healthy again exactly for the reason of finding missing objects. If
there is a real disaster of some kind, and many OSDs go up and down at
various times, we have to have a way of retracing where parts of a PG
were in the past. And ... most of the time this does work - I just
don't understand why this current scenario is different.
Andras
On 5/19/20 3:49 AM, Frank Schilder wrote:
Hi Andreas,
I made exactly the same observation in another scenario. I added some OSDs while other
OSDs were down.
This is expected.
The crush map is an a-priory algorithm to compute the location of objects without
contacting a central server. Hence, *any*change of a crush map while an OSD is down will
lead to a change of locations of objects/PGs of the down OSD. Consequently, these
objects/PGs will become degraded, because no up OSD reports these. Once the peering is
over after setting the weight to 0, the cluster must assume they are lost.
Changing a weight is a change of the crush map.
The way to get the cluster to re-scan after starting the down OSDs is to restore the
crush map to exactly the state as it was before the OSD went down. In your case,
- starting the down OSD,
- setting weight back to original value
will find all missing objects. After the cluster is clean, set the weight back to 0 and
now the OSD will be vacated as expected.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Andras Pataki <apataki(a)flatironinstitute.org>
Sent: 18 May 2020 22:25:37
To: ceph-users
Subject: [ceph-users] Reweighting OSD while down results in undersized+degraded PGs
In a recent cluster reorganization, we ended up with a lot of
undersized/degraded PGs and a day of recovery from them, when all we
expected was moving some data around. After retracing my steps, I found
something odd. If I crush reweight an OSD to 0 while it is down - it
results in the PGs of that OSD ending up degraded even after the OSD is
restarted. If I do the same reweighting while the OSD is up - data gets
moved without any degraded/undersized states. I would not expect this -
so I wonder if this is a bug or is somehow intended. This is on ceph
Nautilus 14.2.8. Below are the details.
Andras
First the case that works as I would expect:
# Healthy cluster ...
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 2m), 270 in (since 4h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
# Reweight an OSD to 0
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
reweighted item id 0 name 'osd.0' to 0 in crush map
# Crush map changes - data movement is set up, no degraded PGs:
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 10m), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 2562045/232996662 objects misplaced (1.100%)
5137 active+clean
172 active+remapped+backfilling
3 active+remapped+backfill_wait
# Reweight it back to the original weight
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
# Cluster goes back to clean
reweighted item id 0 name 'osd.0' to 8 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 11m), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
#
# Now the problematic case
#
# Stop an OSD
[root@xorphosd00 ~]# systemctl stop ceph-osd@0
# We get degraded PGs - as expected
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
1 osds down
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 269 up (since 16s), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
5230 active+clean
82 active+undersized+degraded
# Reweight the OSD to 0:
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
# Still degraded - as expected
reweighted item id 0 name 'osd.0' to 0 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
1 osds down
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 269 up (since 59s), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
1688081/232996662 objects misplaced (0.725%)
5137 active+clean
93 active+remapped+backfilling
82 active+undersized+degraded+remapped+backfilling
# Restarting the OSD
[root@xorphosd00 ~]# systemctl start ceph-osd@0
# And the PGs still stay degraded - THIS IS UNEXPECTED!!!
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
Degraded data redundancy: 873964/232996662 objects degraded
(0.375%), 82 pgs degraded
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 14s), 270 in (since 5h); 175 remapped pgs
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 873964/232996662 objects degraded (0.375%)
1688081/232996662 objects misplaced (0.725%)
5137 active+clean
93 active+remapped+backfilling
82 active+undersized+degraded+remapped+backfilling
# Now for something even more odd - reweight the OSD back to its
original weigh
# and all the data gets magically FOUND again on that OSD!!!
[root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
reweighted item id 0 name 'osd.0' to 8 in crush map
[root@xorphosd00 ~]# ceph -s
cluster:
id: 86d8a1b9-761b-4099-a960-6a303b951236
health: HEALTH_WARN
noout,nobackfill,noscrub,nodeep-scrub flag(s) set
services:
mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
osd: 270 osds: 270 up (since 51s), 270 in (since 5h)
flags noout,nobackfill,noscrub,nodeep-scrub
data:
pools: 4 pools, 5312 pgs
objects: 75.87M objects, 287 TiB
usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
pgs: 5312 active+clean
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
20 May
20 May
4:18 p.m.
Hi Frank,
Thanks for the explanation - I wasn't aware of this subtle point. So
when some OSDs are down, one has to be very careful with changing the
cluster then. I guess one could even end up with incomplete PGs this
way that ceph can't recover from in an automated fashion?
Andras
On 5/19/20 11:23 AM, Frank Schilder wrote:
Hi Andreas,
the cluster map and crush map are not the same thing. If you change the crush map while
the cluster is in degraded state, you basically modify this history of cluster maps
explicitly and have to live with the consequences (keeping history under crush map changes
is limited to up+in OSDs). Only OSDs that can peer are able to respond to changes of the
crush map.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Andras Pataki <apataki(a)flatironinstitute.org>
Sent: 19 May 2020 15:57:49
To: Frank Schilder; ceph-users
Subject: Re: [ceph-users] Reweighting OSD while down results in undersized+degraded PGs
Hi Frank,
My understanding was that once a cluster is in a degraded state (an OSD
is down), ceph stores all changed cluster maps until the cluster is
healthy again exactly for the reason of finding missing objects. If
there is a real disaster of some kind, and many OSDs go up and down at
various times, we have to have a way of retracing where parts of a PG
were in the past. And ... most of the time this does work - I just
don't understand why this current scenario is different.
Andras
On 5/19/20 3:49 AM, Frank Schilder wrote:
> Hi Andreas,
>
> I made exactly the same observation in another scenario. I added some OSDs while
other OSDs were down.
>
> This is expected.
>
> The crush map is an a-priory algorithm to compute the location of objects without
contacting a central server. Hence, *any*change of a crush map while an OSD is down will
lead to a change of locations of objects/PGs of the down OSD. Consequently, these
objects/PGs will become degraded, because no up OSD reports these. Once the peering is
over after setting the weight to 0, the cluster must assume they are lost.
>
> Changing a weight is a change of the crush map.
>
> The way to get the cluster to re-scan after starting the down OSDs is to restore the
crush map to exactly the state as it was before the OSD went down. In your case,
>
> - starting the down OSD,
> - setting weight back to original value
>
> will find all missing objects. After the cluster is clean, set the weight back to 0
and now the OSD will be vacated as expected.
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Andras Pataki <apataki(a)flatironinstitute.org>
> Sent: 18 May 2020 22:25:37
> To: ceph-users
> Subject: [ceph-users] Reweighting OSD while down results in undersized+degraded PGs
>
> In a recent cluster reorganization, we ended up with a lot of
> undersized/degraded PGs and a day of recovery from them, when all we
> expected was moving some data around. After retracing my steps, I found
> something odd. If I crush reweight an OSD to 0 while it is down - it
> results in the PGs of that OSD ending up degraded even after the OSD is
> restarted. If I do the same reweighting while the OSD is up - data gets
> moved without any degraded/undersized states. I would not expect this -
> so I wonder if this is a bug or is somehow intended. This is on ceph
> Nautilus 14.2.8. Below are the details.
>
> Andras
>
>
> First the case that works as I would expect:
>
> # Healthy cluster ...
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 2m), 270 in (since 4h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 5312 active+clean
>
> # Reweight an OSD to 0
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
> reweighted item id 0 name 'osd.0' to 0 in crush map
>
> # Crush map changes - data movement is set up, no degraded PGs:
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 10m), 270 in (since 5h); 175 remapped pgs
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 2562045/232996662 objects misplaced (1.100%)
> 5137 active+clean
> 172 active+remapped+backfilling
> 3 active+remapped+backfill_wait
>
> # Reweight it back to the original weight
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
>
> # Cluster goes back to clean
> reweighted item id 0 name 'osd.0' to 8 in crush map
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 11m), 270 in (since 5h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 5312 active+clean
>
>
>
>
> #
> # Now the problematic case
> #
>
> # Stop an OSD
> [root@xorphosd00 ~]# systemctl stop ceph-osd@0
>
> # We get degraded PGs - as expected
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
> 1 osds down
> Degraded data redundancy: 873964/232996662 objects degraded
> (0.375%), 82 pgs degraded
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 269 up (since 16s), 270 in (since 5h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 873964/232996662 objects degraded (0.375%)
> 5230 active+clean
> 82 active+undersized+degraded
>
> # Reweight the OSD to 0:
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
>
> # Still degraded - as expected
> reweighted item id 0 name 'osd.0' to 0 in crush map
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
> 1 osds down
> Degraded data redundancy: 873964/232996662 objects degraded
> (0.375%), 82 pgs degraded
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 269 up (since 59s), 270 in (since 5h); 175 remapped pgs
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 873964/232996662 objects degraded (0.375%)
> 1688081/232996662 objects misplaced (0.725%)
> 5137 active+clean
> 93 active+remapped+backfilling
> 82 active+undersized+degraded+remapped+backfilling
>
> # Restarting the OSD
> [root@xorphosd00 ~]# systemctl start ceph-osd@0
>
> # And the PGs still stay degraded - THIS IS UNEXPECTED!!!
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
> Degraded data redundancy: 873964/232996662 objects degraded
> (0.375%), 82 pgs degraded
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 14s), 270 in (since 5h); 175 remapped pgs
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 873964/232996662 objects degraded (0.375%)
> 1688081/232996662 objects misplaced (0.725%)
> 5137 active+clean
> 93 active+remapped+backfilling
> 82 active+undersized+degraded+remapped+backfilling
>
> # Now for something even more odd - reweight the OSD back to its
> original weigh
> # and all the data gets magically FOUND again on that OSD!!!
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
> reweighted item id 0 name 'osd.0' to 8 in crush map
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 51s), 270 in (since 5h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 5312 active+clean
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:41 a.m.
Hi Andras,
To me it looks like the osd.0 is not peering when it starts with crush weight 0.
I would try forcing the re-peering with `ceph osd down osd.0` when the
PGs are unexpectedly degraded. (e.g start the osd when crush weight is
0, then obverve the PGs are still degraded, then force the re-peering
-- does it help?)
Otherwise I agree, to me this is an unexpected behaviour -- maybe open a ticket?
Cheers, Dan
P.S. For some reason all of your mails are repeatedly landing in my
spam folder. I think this is the reason:
ARC-Authentication-Results: i=1; mx.google.com;
dkim=neutral (body hash did not verify)
header.i=(a)flatironinstitute.org header.s=google header.b=NvX+wag9;
spf=fail (google.com: domain of ceph-users-bounces(a)ceph.io does
not designate 217.70.178.232 as permitted sender)
smtp.mailfrom=ceph-users-bounces(a)ceph.io;
dmarc=fail (p=REJECT sp=REJECT dis=QUARANTINE)
header.from=flatironinstitute.org
On Mon, May 18, 2020 at 10:26 PM Andras Pataki
<apataki(a)flatironinstitute.org> wrote:
>
> In a recent cluster reorganization, we ended up with a lot of
> undersized/degraded PGs and a day of recovery from them, when all we
> expected was moving some data around. After retracing my steps, I found
> something odd. If I crush reweight an OSD to 0 while it is down - it
> results in the PGs of that OSD ending up degraded even after the OSD is
> restarted. If I do the same reweighting while the OSD is up - data gets
> moved without any degraded/undersized states. I would not expect this -
> so I wonder if this is a bug or is somehow intended. This is on ceph
> Nautilus 14.2.8. Below are the details.
>
> Andras
>
>
> First the case that works as I would expect:
>
> # Healthy cluster ...
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 2m), 270 in (since 4h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 5312 active+clean
>
> # Reweight an OSD to 0
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
> reweighted item id 0 name 'osd.0' to 0 in crush map
>
> # Crush map changes - data movement is set up, no degraded PGs:
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 10m), 270 in (since 5h); 175 remapped pgs
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 2562045/232996662 objects misplaced (1.100%)
> 5137 active+clean
> 172 active+remapped+backfilling
> 3 active+remapped+backfill_wait
>
> # Reweight it back to the original weight
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
>
> # Cluster goes back to clean
> reweighted item id 0 name 'osd.0' to 8 in crush map
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 11m), 270 in (since 5h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 5312 active+clean
>
>
>
>
> #
> # Now the problematic case
> #
>
> # Stop an OSD
> [root@xorphosd00 ~]# systemctl stop ceph-osd@0
>
> # We get degraded PGs - as expected
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
> 1 osds down
> Degraded data redundancy: 873964/232996662 objects degraded
> (0.375%), 82 pgs degraded
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 269 up (since 16s), 270 in (since 5h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 873964/232996662 objects degraded (0.375%)
> 5230 active+clean
> 82 active+undersized+degraded
>
> # Reweight the OSD to 0:
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
>
> # Still degraded - as expected
> reweighted item id 0 name 'osd.0' to 0 in crush map
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
> 1 osds down
> Degraded data redundancy: 873964/232996662 objects degraded
> (0.375%), 82 pgs degraded
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 269 up (since 59s), 270 in (since 5h); 175 remapped pgs
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 873964/232996662 objects degraded (0.375%)
> 1688081/232996662 objects misplaced (0.725%)
> 5137 active+clean
> 93 active+remapped+backfilling
> 82 active+undersized+degraded+remapped+backfilling
>
> # Restarting the OSD
> [root@xorphosd00 ~]# systemctl start ceph-osd@0
>
> # And the PGs still stay degraded - THIS IS UNEXPECTED!!!
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
> Degraded data redundancy: 873964/232996662 objects degraded
> (0.375%), 82 pgs degraded
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 14s), 270 in (since 5h); 175 remapped pgs
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 873964/232996662 objects degraded (0.375%)
> 1688081/232996662 objects misplaced (0.725%)
> 5137 active+clean
> 93 active+remapped+backfilling
> 82 active+undersized+degraded+remapped+backfilling
>
> # Now for something even more odd - reweight the OSD back to its
> original weigh
> # and all the data gets magically FOUND again on that OSD!!!
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
> reweighted item id 0 name 'osd.0' to 8 in crush map
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 51s), 270 in (since 5h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 5312 active+clean
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:16 p.m.
Hi Dan,
Unfortunately 'ceph osd down osd.0' doesn't help - it is marked down and
soon after back up, but it doesn't peer still. I tried reweighting the
OSD to half its weight, 4.0 instead of 0.0, and that results in about
half the PGs staying degraded. So this is not specific to zero weight.
It looks like when the OSD starts, it gets a crush map and tries to peer
the PGs that are in that map. It doesn't look like it tries to peer for
PGs it has data for but it isn't responsible for in the crush map
currently. I guess I don't fully understand the design on what the
intended behavior is.
I also get a lot of the mailing list going to spam. The reason has to
do with how the mailing list is set up. It resends all messages keeping
the sender (From:) the same using servers that are unrelated to the
sender. Thus messages looks like spam a lot - sending mails in my name
using a server not registered as allowed to send for my domain. Not
sure what can be done about it.
Andras
On 5/20/20 3:41 AM, Dan van der Ster wrote:
Hi Andras,
To me it looks like the osd.0 is not peering when it starts with crush weight 0.
I would try forcing the re-peering with `ceph osd down osd.0` when the
PGs are unexpectedly degraded. (e.g start the osd when crush weight is
0, then obverve the PGs are still degraded, then force the re-peering
-- does it help?)
Otherwise I agree, to me this is an unexpected behaviour -- maybe open a ticket?
Cheers, Dan
P.S. For some reason all of your mails are repeatedly landing in my
spam folder. I think this is the reason:
ARC-Authentication-Results: i=1; mx.google.com;
dkim=neutral (body hash did not verify)
header.i=(a)flatironinstitute.org header.s=google header.b=NvX+wag9;
spf=fail (google.com: domain of ceph-users-bounces(a)ceph.io does
not designate 217.70.178.232 as permitted sender)
smtp.mailfrom=ceph-users-bounces(a)ceph.io;
dmarc=fail (p=REJECT sp=REJECT dis=QUARANTINE)
header.from=flatironinstitute.org
On Mon, May 18, 2020 at 10:26 PM Andras Pataki
<apataki(a)flatironinstitute.org> wrote:
> In a recent cluster reorganization, we ended up with a lot of
> undersized/degraded PGs and a day of recovery from them, when all we
> expected was moving some data around. After retracing my steps, I found
> something odd. If I crush reweight an OSD to 0 while it is down - it
> results in the PGs of that OSD ending up degraded even after the OSD is
> restarted. If I do the same reweighting while the OSD is up - data gets
> moved without any degraded/undersized states. I would not expect this -
> so I wonder if this is a bug or is somehow intended. This is on ceph
> Nautilus 14.2.8. Below are the details.
>
> Andras
>
>
> First the case that works as I would expect:
>
> # Healthy cluster ...
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 2m), 270 in (since 4h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 5312 active+clean
>
> # Reweight an OSD to 0
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
> reweighted item id 0 name 'osd.0' to 0 in crush map
>
> # Crush map changes - data movement is set up, no degraded PGs:
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 10m), 270 in (since 5h); 175 remapped pgs
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 2562045/232996662 objects misplaced (1.100%)
> 5137 active+clean
> 172 active+remapped+backfilling
> 3 active+remapped+backfill_wait
>
> # Reweight it back to the original weight
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
>
> # Cluster goes back to clean
> reweighted item id 0 name 'osd.0' to 8 in crush map
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 11m), 270 in (since 5h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 5312 active+clean
>
>
>
>
> #
> # Now the problematic case
> #
>
> # Stop an OSD
> [root@xorphosd00 ~]# systemctl stop ceph-osd@0
>
> # We get degraded PGs - as expected
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
> 1 osds down
> Degraded data redundancy: 873964/232996662 objects degraded
> (0.375%), 82 pgs degraded
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 269 up (since 16s), 270 in (since 5h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 873964/232996662 objects degraded (0.375%)
> 5230 active+clean
> 82 active+undersized+degraded
>
> # Reweight the OSD to 0:
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0
>
> # Still degraded - as expected
> reweighted item id 0 name 'osd.0' to 0 in crush map
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
> 1 osds down
> Degraded data redundancy: 873964/232996662 objects degraded
> (0.375%), 82 pgs degraded
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 269 up (since 59s), 270 in (since 5h); 175 remapped pgs
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 873964/232996662 objects degraded (0.375%)
> 1688081/232996662 objects misplaced (0.725%)
> 5137 active+clean
> 93 active+remapped+backfilling
> 82 active+undersized+degraded+remapped+backfilling
>
> # Restarting the OSD
> [root@xorphosd00 ~]# systemctl start ceph-osd@0
>
> # And the PGs still stay degraded - THIS IS UNEXPECTED!!!
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
> Degraded data redundancy: 873964/232996662 objects degraded
> (0.375%), 82 pgs degraded
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 14s), 270 in (since 5h); 175 remapped pgs
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 873964/232996662 objects degraded (0.375%)
> 1688081/232996662 objects misplaced (0.725%)
> 5137 active+clean
> 93 active+remapped+backfilling
> 82 active+undersized+degraded+remapped+backfilling
>
> # Now for something even more odd - reweight the OSD back to its
> original weigh
> # and all the data gets magically FOUND again on that OSD!!!
> [root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0
> reweighted item id 0 name 'osd.0' to 8 in crush map
> [root@xorphosd00 ~]# ceph -s
> cluster:
> id: 86d8a1b9-761b-4099-a960-6a303b951236
> health: HEALTH_WARN
> noout,nobackfill,noscrub,nodeep-scrub flag(s) set
>
> services:
> mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d)
> mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00
> mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby
> osd: 270 osds: 270 up (since 51s), 270 in (since 5h)
> flags noout,nobackfill,noscrub,nodeep-scrub
>
> data:
> pools: 4 pools, 5312 pgs
> objects: 75.87M objects, 287 TiB
> usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail
> pgs: 5312 active+clean
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1458
days inactive
1460
days old
6 comments
3 participants
participants (3)
-
Andras Pataki -
Dan van der Ster -
Frank Schilder