31 Mar
2023
31 Mar
'23
8:38 p.m.
Dear all;
Up until a few hours ago, I had a seemingly normally-behaving cluster
(Quincy, 17.2.5) with 36 OSDs, evenly distributed across 3 of its 6
nodes. The cluster is only used for CephFS and the only non-standard
configuration I can think of is that I had 2 active MDSs, but only 1
standby. I had also doubled mds_cache_memory limit to 8 GB (all OSD
hosts have 256 G of RAM) at some point in the past.
Then I rebooted one of the OSD nodes. The rebooted node held one of the
active MDSs. Now the node is back up: ceph -s says the cluster is
healthy, but all PGs are in a active+clean+remapped state and 166.67% of
the objects are misplaced (dashboard: -66.66% healthy).
The data pool is a threefold replica with 5.4M object, the number of
misplaced objects is reported as 27087410/16252446. The denominator in
the ratio makes sense to me (16.2M / 3 = 5.4M), but the numerator does
not. I also note that the ratio is *exactly* 5 / 3. The filesystem is
still mounted and appears to be usable, but df reports it as 100% full;
I suspect it would say 167% but that is capped somewhere.
Any ideas about what is going on? Any suggestions for recovery?
// Best wishes; Johan
1 Apr
1 Apr
3:07 a.m.
Need to know some more about your cluster...
Ceph -s
Ceph osd df tree
Replica or ec?
...
Perhaps this can give us some insight
Mehmet
Am 31. März 2023 18:08:38 MESZ schrieb Johan Hattne <johan(a)hattne.se>se>:
Dear all;
Up until a few hours ago, I had a seemingly normally-behaving cluster (Quincy, 17.2.5)
with 36 OSDs, evenly distributed across 3 of its 6 nodes. The cluster is only used for
CephFS and the only non-standard configuration I can think of is that I had 2 active MDSs,
but only 1 standby. I had also doubled mds_cache_memory limit to 8 GB (all OSD hosts have
256 G of RAM) at some point in the past.
Then I rebooted one of the OSD nodes. The rebooted node held one of the active MDSs. Now
the node is back up: ceph -s says the cluster is healthy, but all PGs are in a
active+clean+remapped state and 166.67% of the objects are misplaced (dashboard: -66.66%
healthy).
The data pool is a threefold replica with 5.4M object, the number of misplaced objects is
reported as 27087410/16252446. The denominator in the ratio makes sense to me (16.2M / 3
= 5.4M), but the numerator does not. I also note that the ratio is *exactly* 5 / 3. The
filesystem is still mounted and appears to be usable, but df reports it as 100% full; I
suspect it would say 167% but that is capped somewhere.
Any ideas about what is going on? Any suggestions for recovery?
// Best wishes; Johan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:31 a.m.
Here goes:
# ceph -s
cluster:
id: e1327a10-8b8c-11ed-88b9-3cecef0e3946
health: HEALTH_OK
services:
mon: 5 daemons, quorum
bcgonen-a,bcgonen-b,bcgonen-c,bcgonen-r0h0,bcgonen-r0h1 (age 16h)
mgr: bcgonen-b.furndm(active, since 8d), standbys: bcgonen-a.qmmqxj
mds: 1/1 daemons up, 2 standby
osd: 36 osds: 36 up (since 16h), 36 in (since 3d); 1041 remapped pgs
data:
volumes: 1/1 healthy
pools: 3 pools, 1041 pgs
objects: 5.42M objects, 6.5 TiB
usage: 19 TiB used, 428 TiB / 447 TiB avail
pgs: 27087125/16252275 objects misplaced (166.667%)
1039 active+clean+remapped
2 active+clean+remapped+scrubbing+deep
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-14 149.02008 rack rack-1
-7 149.02008 host bcgonen-r1h0
20 hdd 14.55269 osd.20 up 1.00000 1.00000
21 hdd 14.55269 osd.21 up 1.00000 1.00000
22 hdd 14.55269 osd.22 up 1.00000 1.00000
23 hdd 14.55269 osd.23 up 1.00000 1.00000
24 hdd 14.55269 osd.24 up 1.00000 1.00000
25 hdd 14.55269 osd.25 up 1.00000 1.00000
26 hdd 14.55269 osd.26 up 1.00000 1.00000
27 hdd 14.55269 osd.27 up 1.00000 1.00000
28 hdd 14.55269 osd.28 up 1.00000 1.00000
29 hdd 14.55269 osd.29 up 1.00000 1.00000
34 ssd 1.74660 osd.34 up 1.00000 1.00000
35 ssd 1.74660 osd.35 up 1.00000 1.00000
-13 298.04016 rack rack-0
-3 149.02008 host bcgonen-r0h0
0 hdd 14.55269 osd.0 up 1.00000 1.00000
1 hdd 14.55269 osd.1 up 1.00000 1.00000
2 hdd 14.55269 osd.2 up 1.00000 1.00000
3 hdd 14.55269 osd.3 up 1.00000 1.00000
4 hdd 14.55269 osd.4 up 1.00000 1.00000
5 hdd 14.55269 osd.5 up 1.00000 1.00000
6 hdd 14.55269 osd.6 up 1.00000 1.00000
7 hdd 14.55269 osd.7 up 1.00000 1.00000
8 hdd 14.55269 osd.8 up 1.00000 1.00000
9 hdd 14.55269 osd.9 up 1.00000 1.00000
30 ssd 1.74660 osd.30 up 1.00000 1.00000
31 ssd 1.74660 osd.31 up 1.00000 1.00000
-5 149.02008 host bcgonen-r0h1
10 hdd 14.55269 osd.10 up 1.00000 1.00000
11 hdd 14.55269 osd.11 up 1.00000 1.00000
12 hdd 14.55269 osd.12 up 1.00000 1.00000
13 hdd 14.55269 osd.13 up 1.00000 1.00000
14 hdd 14.55269 osd.14 up 1.00000 1.00000
15 hdd 14.55269 osd.15 up 1.00000 1.00000
16 hdd 14.55269 osd.16 up 1.00000 1.00000
17 hdd 14.55269 osd.17 up 1.00000 1.00000
18 hdd 14.55269 osd.18 up 1.00000 1.00000
19 hdd 14.55269 osd.19 up 1.00000 1.00000
32 ssd 1.74660 osd.32 up 1.00000 1.00000
33 ssd 1.74660 osd.33 up 1.00000 1.00000
-1 0 root default
# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 31 flags
hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr
pool 2 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 2
object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change
9833 lfor 0/0/584 flags hashpspool stripe_width 0 pg_autoscale_bias 4
pg_num_min 16 recovery_priority 5 application cephfs
pool 3 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 1
object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on
last_change 7630 lfor 0/1831/6544 flags hashpspool,bulk stripe_width 0
application cephfs
crush_rules 1 and 2 are just used to assign the data and meta pool to
HDD and SSD, respectively (failure domain: host).
// J
On 2023-03-31 15:37, ceph(a)elchaka.de wrote:
Need to know some more about your cluster...
Ceph -s
Ceph osd df tree
Replica or ec?
...
Perhaps this can give us some insight
Mehmet
Am 31. März 2023 18:08:38 MESZ schrieb Johan Hattne <johan(a)hattne.se>se>:
Dear all;
Up until a few hours ago, I had a seemingly normally-behaving cluster (Quincy,
17.2.5) with 36 OSDs, evenly distributed across 3 of its 6 nodes. The cluster is only
used for CephFS and the only non-standard configuration I can think of is that I had 2
active MDSs, but only 1 standby. I had also doubled mds_cache_memory limit to 8 GB (all
OSD hosts have 256 G of RAM) at some point in the past.
Then I rebooted one of the OSD nodes. The rebooted node held one of the active MDSs.
Now the node is back up: ceph -s says the cluster is healthy, but all PGs are in a
active+clean+remapped state and 166.67% of the objects are misplaced (dashboard: -66.66%
healthy).
The data pool is a threefold replica with 5.4M object, the number of misplaced
objects is reported as 27087410/16252446. The denominator in the ratio makes sense to me
(16.2M / 3 = 5.4M), but the numerator does not. I also note that the ratio is *exactly* 5
/ 3. The filesystem is still mounted and appears to be usable, but df reports it as 100%
full; I suspect it would say 167% but that is capped somewhere.
Any ideas about what is going on? Any suggestions for recovery?
// Best wishes; Johan
------------------------------------------------------------------------
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4 Apr
4 Apr
12:01 a.m.
Thanks Mehmet; I took a closer look at what I sent you and the problem
appears to be in the CRUSH map. At some point since anything was last
rebooted, I created rack buckets and moved the OSD nodes in under them:
# ceph osd crush add-bucket rack-0 rack
# ceph osd crush add-bucket rack-1 rack
# ceph osd crush move bcgonen-r0h0 rack=rack-0
# ceph osd crush move bcgonen-r0h1 rack=rack-0
# ceph osd crush move bcgonen-r1h0 rack=rack-1
All seemed fine at the time; it was not until bcgonen-r1h0 was rebooted
that stuff got weird. But as per "ceph osd tree" output, those rack
buckets were sitting next to the default root as opposed to under it.
Now that's fixed, and the cluster is backfilling remapped PGs.
// J
On 2023-03-31 16:01, Johan Hattne wrote:
Here goes:
# ceph -s
cluster:
id: e1327a10-8b8c-11ed-88b9-3cecef0e3946
health: HEALTH_OK
services:
mon: 5 daemons, quorum
bcgonen-a,bcgonen-b,bcgonen-c,bcgonen-r0h0,bcgonen-r0h1 (age 16h)
mgr: bcgonen-b.furndm(active, since 8d), standbys: bcgonen-a.qmmqxj
mds: 1/1 daemons up, 2 standby
osd: 36 osds: 36 up (since 16h), 36 in (since 3d); 1041 remapped pgs
data:
volumes: 1/1 healthy
pools: 3 pools, 1041 pgs
objects: 5.42M objects, 6.5 TiB
usage: 19 TiB used, 428 TiB / 447 TiB avail
pgs: 27087125/16252275 objects misplaced (166.667%)
1039 active+clean+remapped
2 active+clean+remapped+scrubbing+deep
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-14 149.02008 rack rack-1
-7 149.02008 host bcgonen-r1h0
20 hdd 14.55269 osd.20 up 1.00000 1.00000
21 hdd 14.55269 osd.21 up 1.00000 1.00000
22 hdd 14.55269 osd.22 up 1.00000 1.00000
23 hdd 14.55269 osd.23 up 1.00000 1.00000
24 hdd 14.55269 osd.24 up 1.00000 1.00000
25 hdd 14.55269 osd.25 up 1.00000 1.00000
26 hdd 14.55269 osd.26 up 1.00000 1.00000
27 hdd 14.55269 osd.27 up 1.00000 1.00000
28 hdd 14.55269 osd.28 up 1.00000 1.00000
29 hdd 14.55269 osd.29 up 1.00000 1.00000
34 ssd 1.74660 osd.34 up 1.00000 1.00000
35 ssd 1.74660 osd.35 up 1.00000 1.00000
-13 298.04016 rack rack-0
-3 149.02008 host bcgonen-r0h0
0 hdd 14.55269 osd.0 up 1.00000 1.00000
1 hdd 14.55269 osd.1 up 1.00000 1.00000
2 hdd 14.55269 osd.2 up 1.00000 1.00000
3 hdd 14.55269 osd.3 up 1.00000 1.00000
4 hdd 14.55269 osd.4 up 1.00000 1.00000
5 hdd 14.55269 osd.5 up 1.00000 1.00000
6 hdd 14.55269 osd.6 up 1.00000 1.00000
7 hdd 14.55269 osd.7 up 1.00000 1.00000
8 hdd 14.55269 osd.8 up 1.00000 1.00000
9 hdd 14.55269 osd.9 up 1.00000 1.00000
30 ssd 1.74660 osd.30 up 1.00000 1.00000
31 ssd 1.74660 osd.31 up 1.00000 1.00000
-5 149.02008 host bcgonen-r0h1
10 hdd 14.55269 osd.10 up 1.00000 1.00000
11 hdd 14.55269 osd.11 up 1.00000 1.00000
12 hdd 14.55269 osd.12 up 1.00000 1.00000
13 hdd 14.55269 osd.13 up 1.00000 1.00000
14 hdd 14.55269 osd.14 up 1.00000 1.00000
15 hdd 14.55269 osd.15 up 1.00000 1.00000
16 hdd 14.55269 osd.16 up 1.00000 1.00000
17 hdd 14.55269 osd.17 up 1.00000 1.00000
18 hdd 14.55269 osd.18 up 1.00000 1.00000
19 hdd 14.55269 osd.19 up 1.00000 1.00000
32 ssd 1.74660 osd.32 up 1.00000 1.00000
33 ssd 1.74660 osd.33 up 1.00000 1.00000
-1 0 root default
# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 31 flags
hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr
pool 2 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 2
object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change
9833 lfor 0/0/584 flags hashpspool stripe_width 0 pg_autoscale_bias 4
pg_num_min 16 recovery_priority 5 application cephfs
pool 3 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 1
object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on
last_change 7630 lfor 0/1831/6544 flags hashpspool,bulk stripe_width 0
application cephfs
crush_rules 1 and 2 are just used to assign the data and meta pool to
HDD and SSD, respectively (failure domain: host).
// J
On 2023-03-31 15:37, ceph(a)elchaka.de wrote:
Need to know some more about your cluster...
Ceph -s
Ceph osd df tree
Replica or ec?
...
Perhaps this can give us some insight
Mehmet
Am 31. März 2023 18:08:38 MESZ schrieb Johan Hattne <johan(a)hattne.se>se>:
Dear all;
Up until a few hours ago, I had a seemingly normally-behaving
cluster (Quincy, 17.2.5) with 36 OSDs, evenly distributed across 3 of
its 6 nodes. The cluster is only used for CephFS and the only
non-standard configuration I can think of is that I had 2 active MDSs,
but only 1 standby. I had also doubled mds_cache_memory limit to 8 GB
(all OSD hosts have 256 G of RAM) at some point in the past.
Then I rebooted one of the OSD nodes. The rebooted node held one
of the active MDSs. Now the node is back up: ceph -s says the cluster
is healthy, but all PGs are in a active+clean+remapped state and
166.67% of the objects are misplaced (dashboard: -66.66% healthy).
The data pool is a threefold replica with 5.4M object, the number
of misplaced objects is reported as 27087410/16252446. The
denominator in the ratio makes sense to me (16.2M / 3 = 5.4M), but the
numerator does not. I also note that the ratio is *exactly* 5 / 3.
The filesystem is still mounted and appears to be usable, but df
reports it as 100% full; I suspect it would say 167% but that is
capped somewhere.
Any ideas about what is going on? Any suggestions for recovery?
// Best wishes; Johan
------------------------------------------------------------------------
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5 Apr
5 Apr
11:31 p.m.
I guess this is related to your crush rules..
Unfortunaly i dont know much about creating the rules...
But someone cloud give more insights when you also provide
crush rule dump
.... your "-1 0 root default" is a bit strange
Am 1. April 2023 01:01:39 MESZ schrieb Johan Hattne <johan(a)hattne.se>se>:
Here goes:
# ceph -s
cluster:
id: e1327a10-8b8c-11ed-88b9-3cecef0e3946
health: HEALTH_OK
services:
mon: 5 daemons, quorum bcgonen-a,bcgonen-b,bcgonen-c,bcgonen-r0h0,bcgonen-r0h1 (age
16h)
mgr: bcgonen-b.furndm(active, since 8d), standbys: bcgonen-a.qmmqxj
mds: 1/1 daemons up, 2 standby
osd: 36 osds: 36 up (since 16h), 36 in (since 3d); 1041 remapped pgs
data:
volumes: 1/1 healthy
pools: 3 pools, 1041 pgs
objects: 5.42M objects, 6.5 TiB
usage: 19 TiB used, 428 TiB / 447 TiB avail
pgs: 27087125/16252275 objects misplaced (166.667%)
1039 active+clean+remapped
2 active+clean+remapped+scrubbing+deep
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-14 149.02008 rack rack-1
-7 149.02008 host bcgonen-r1h0
20 hdd 14.55269 osd.20 up 1.00000 1.00000
21 hdd 14.55269 osd.21 up 1.00000 1.00000
22 hdd 14.55269 osd.22 up 1.00000 1.00000
23 hdd 14.55269 osd.23 up 1.00000 1.00000
24 hdd 14.55269 osd.24 up 1.00000 1.00000
25 hdd 14.55269 osd.25 up 1.00000 1.00000
26 hdd 14.55269 osd.26 up 1.00000 1.00000
27 hdd 14.55269 osd.27 up 1.00000 1.00000
28 hdd 14.55269 osd.28 up 1.00000 1.00000
29 hdd 14.55269 osd.29 up 1.00000 1.00000
34 ssd 1.74660 osd.34 up 1.00000 1.00000
35 ssd 1.74660 osd.35 up 1.00000 1.00000
-13 298.04016 rack rack-0
-3 149.02008 host bcgonen-r0h0
0 hdd 14.55269 osd.0 up 1.00000 1.00000
1 hdd 14.55269 osd.1 up 1.00000 1.00000
2 hdd 14.55269 osd.2 up 1.00000 1.00000
3 hdd 14.55269 osd.3 up 1.00000 1.00000
4 hdd 14.55269 osd.4 up 1.00000 1.00000
5 hdd 14.55269 osd.5 up 1.00000 1.00000
6 hdd 14.55269 osd.6 up 1.00000 1.00000
7 hdd 14.55269 osd.7 up 1.00000 1.00000
8 hdd 14.55269 osd.8 up 1.00000 1.00000
9 hdd 14.55269 osd.9 up 1.00000 1.00000
30 ssd 1.74660 osd.30 up 1.00000 1.00000
31 ssd 1.74660 osd.31 up 1.00000 1.00000
-5 149.02008 host bcgonen-r0h1
10 hdd 14.55269 osd.10 up 1.00000 1.00000
11 hdd 14.55269 osd.11 up 1.00000 1.00000
12 hdd 14.55269 osd.12 up 1.00000 1.00000
13 hdd 14.55269 osd.13 up 1.00000 1.00000
14 hdd 14.55269 osd.14 up 1.00000 1.00000
15 hdd 14.55269 osd.15 up 1.00000 1.00000
16 hdd 14.55269 osd.16 up 1.00000 1.00000
17 hdd 14.55269 osd.17 up 1.00000 1.00000
18 hdd 14.55269 osd.18 up 1.00000 1.00000
19 hdd 14.55269 osd.19 up 1.00000 1.00000
32 ssd 1.74660 osd.32 up 1.00000 1.00000
33 ssd 1.74660 osd.33 up 1.00000 1.00000
-1 0 root default
# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins
pg_num 1 pgp_num 1 autoscale_mode on last_change 31 flags hashpspool stripe_width 0
pg_num_max 32 pg_num_min 1 application mgr
pool 2 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 2 object_hash
rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 9833 lfor 0/0/584 flags
hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5
application cephfs
pool 3 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 1 object_hash
rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 7630 lfor 0/1831/6544
flags hashpspool,bulk stripe_width 0 application cephfs
crush_rules 1 and 2 are just used to assign the data and meta pool to HDD and SSD,
respectively (failure domain: host).
// J
On 2023-03-31 15:37, ceph(a)elchaka.de wrote:
Need to know some more about your cluster...
Ceph -s
Ceph osd df tree
Replica or ec?
...
Perhaps this can give us some insight
Mehmet
Am 31. März 2023 18:08:38 MESZ schrieb Johan Hattne <johan(a)hattne.se>se>:
Dear all;
Up until a few hours ago, I had a seemingly normally-behaving cluster (Quincy,
17.2.5) with 36 OSDs, evenly distributed across 3 of its 6 nodes. The cluster is only
used for CephFS and the only non-standard configuration I can think of is that I had 2
active MDSs, but only 1 standby. I had also doubled mds_cache_memory limit to 8 GB (all
OSD hosts have 256 G of RAM) at some point in the past.
Then I rebooted one of the OSD nodes. The rebooted node held one of the active MDSs.
Now the node is back up: ceph -s says the cluster is healthy, but all PGs are in a
active+clean+remapped state and 166.67% of the objects are misplaced (dashboard: -66.66%
healthy).
The data pool is a threefold replica with 5.4M object, the number of misplaced
objects is reported as 27087410/16252446. The denominator in the ratio makes sense to me
(16.2M / 3 = 5.4M), but the numerator does not. I also note that the ratio is *exactly* 5
/ 3. The filesystem is still mounted and appears to be usable, but df reports it as 100%
full; I suspect it would say 167% but that is capped somewhere.
Any ideas about what is going on? Any suggestions for recovery?
// Best wishes; Johan
------------------------------------------------------------------------
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6 Apr
6 Apr
12:51 a.m.
I think this is resolved—and you're right about the 0-weight of the root
bucket being strange. I had created the rack buckets with
# ceph osd crush add-bucket rack-0 rack
whereas I should have used something like
# ceph osd crush add-bucket rack-0 rack root=default
There's a bit in the documentation
(https://docs.ceph.com/en/quincy/rados/operations/crush-map) that says
"Not all keys need to be specified" (in a different context, I admit).
I might have saved a second or two by omitting "root=default" and maybe
half a minute by not checking the CRUSH map carefully afterwards. It
was not worth it.
// J
On 2023-04-05 12:01, ceph(a)elchaka.de wrote:
I guess this is related to your crush rules..
Unfortunaly i dont know much about creating the rules...
But someone cloud give more insights when you also provide
crush rule dump
.... your "-1 0 root default" is a bit strange
Am 1. April 2023 01:01:39 MESZ schrieb Johan Hattne <johan(a)hattne.se>se>:
Here goes:
# ceph -s
cluster:
id: e1327a10-8b8c-11ed-88b9-3cecef0e3946
health: HEALTH_OK
services:
mon: 5 daemons, quorum bcgonen-a,bcgonen-b,bcgonen-c,bcgonen-r0h0,bcgonen-r0h1
(age 16h)
mgr: bcgonen-b.furndm(active, since 8d), standbys: bcgonen-a.qmmqxj
mds: 1/1 daemons up, 2 standby
osd: 36 osds: 36 up (since 16h), 36 in (since 3d); 1041 remapped pgs
data:
volumes: 1/1 healthy
pools: 3 pools, 1041 pgs
objects: 5.42M objects, 6.5 TiB
usage: 19 TiB used, 428 TiB / 447 TiB avail
pgs: 27087125/16252275 objects misplaced (166.667%)
1039 active+clean+remapped
2 active+clean+remapped+scrubbing+deep
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-14 149.02008 rack rack-1
-7 149.02008 host bcgonen-r1h0
20 hdd 14.55269 osd.20 up 1.00000 1.00000
21 hdd 14.55269 osd.21 up 1.00000 1.00000
22 hdd 14.55269 osd.22 up 1.00000 1.00000
23 hdd 14.55269 osd.23 up 1.00000 1.00000
24 hdd 14.55269 osd.24 up 1.00000 1.00000
25 hdd 14.55269 osd.25 up 1.00000 1.00000
26 hdd 14.55269 osd.26 up 1.00000 1.00000
27 hdd 14.55269 osd.27 up 1.00000 1.00000
28 hdd 14.55269 osd.28 up 1.00000 1.00000
29 hdd 14.55269 osd.29 up 1.00000 1.00000
34 ssd 1.74660 osd.34 up 1.00000 1.00000
35 ssd 1.74660 osd.35 up 1.00000 1.00000
-13 298.04016 rack rack-0
-3 149.02008 host bcgonen-r0h0
0 hdd 14.55269 osd.0 up 1.00000 1.00000
1 hdd 14.55269 osd.1 up 1.00000 1.00000
2 hdd 14.55269 osd.2 up 1.00000 1.00000
3 hdd 14.55269 osd.3 up 1.00000 1.00000
4 hdd 14.55269 osd.4 up 1.00000 1.00000
5 hdd 14.55269 osd.5 up 1.00000 1.00000
6 hdd 14.55269 osd.6 up 1.00000 1.00000
7 hdd 14.55269 osd.7 up 1.00000 1.00000
8 hdd 14.55269 osd.8 up 1.00000 1.00000
9 hdd 14.55269 osd.9 up 1.00000 1.00000
30 ssd 1.74660 osd.30 up 1.00000 1.00000
31 ssd 1.74660 osd.31 up 1.00000 1.00000
-5 149.02008 host bcgonen-r0h1
10 hdd 14.55269 osd.10 up 1.00000 1.00000
11 hdd 14.55269 osd.11 up 1.00000 1.00000
12 hdd 14.55269 osd.12 up 1.00000 1.00000
13 hdd 14.55269 osd.13 up 1.00000 1.00000
14 hdd 14.55269 osd.14 up 1.00000 1.00000
15 hdd 14.55269 osd.15 up 1.00000 1.00000
16 hdd 14.55269 osd.16 up 1.00000 1.00000
17 hdd 14.55269 osd.17 up 1.00000 1.00000
18 hdd 14.55269 osd.18 up 1.00000 1.00000
19 hdd 14.55269 osd.19 up 1.00000 1.00000
32 ssd 1.74660 osd.32 up 1.00000 1.00000
33 ssd 1.74660 osd.33 up 1.00000 1.00000
-1 0 root default
# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins
pg_num 1 pgp_num 1 autoscale_mode on last_change 31 flags hashpspool stripe_width 0
pg_num_max 32 pg_num_min 1 application mgr
pool 2 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 2
object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 9833 lfor 0/0/584
flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5
application cephfs
pool 3 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 1
object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 7630 lfor
0/1831/6544 flags hashpspool,bulk stripe_width 0 application cephfs
crush_rules 1 and 2 are just used to assign the data and meta pool to HDD and SSD,
respectively (failure domain: host).
// J
On 2023-03-31 15:37, ceph(a)elchaka.de wrote:
Need to know some more about your cluster...
Ceph -s
Ceph osd df tree
Replica or ec?
...
Perhaps this can give us some insight
Mehmet
Am 31. März 2023 18:08:38 MESZ schrieb Johan Hattne
<johan(a)hattne.se>se>:
Dear all;
Up until a few hours ago, I had a seemingly normally-behaving
cluster (Quincy, 17.2.5) with 36 OSDs, evenly distributed across
3 of its 6 nodes. The cluster is only used for CephFS and the
only non-standard configuration I can think of is that I had 2
active MDSs, but only 1 standby. I had also doubled
mds_cache_memory limit to 8 GB (all OSD hosts have 256 G of RAM)
at some point in the past.
Then I rebooted one of the OSD nodes. The rebooted node held one
of the active MDSs. Now the node is back up: ceph -s says the
cluster is healthy, but all PGs are in a active+clean+remapped
state and 166.67% of the objects are misplaced (dashboard:
-66.66% healthy).
The data pool is a threefold replica with 5.4M object, the
number of misplaced objects is reported as 27087410/16252446.
The denominator in the ratio makes sense to me (16.2M / 3 =
5.4M), but the numerator does not. I also note that the ratio is
*exactly* 5 / 3. The filesystem is still mounted and appears to
be usable, but df reports it as 100% full; I suspect it would
say 167% but that is capped somewhere.
Any ideas about what is going on? Any suggestions for recovery?
// Best wishes; Johan
------------------------------------------------------------------------
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:38 a.m.
Perhaps this option triggered the crush map change:
osd crush update on start
Each time the OSD starts, it verifies it is in the correct location in
the CRUSH map and, if it is not, it moves itself.
https://docs.ceph.com/en/quincy/rados/operations/crush-map/
Joachim
Johan Hattne <johan(a)hattne.se> schrieb am Mi., 5. Apr. 2023, 22:21:
I think this is resolved—and you're right about
the 0-weight of the root
bucket being strange. I had created the rack buckets with
# ceph osd crush add-bucket rack-0 rack
whereas I should have used something like
# ceph osd crush add-bucket rack-0 rack root=default
There's a bit in the documentation
(https://docs.ceph.com/en/quincy/rados/operations/crush-map) that says
"Not all keys need to be specified" (in a different context, I admit).
I might have saved a second or two by omitting "root=default" and maybe
half a minute by not checking the CRUSH map carefully afterwards. It
was not worth it.
// J
On 2023-04-05 12:01, ceph(a)elchaka.de wrote:
I guess this is related to your crush rules..
Unfortunaly i dont know much about creating the rules...
But someone cloud give more insights when you also provide
crush rule dump
.... your "-1 0 root default" is a bit strange
Am 1. April 2023 01:01:39 MESZ schrieb Johan Hattne <johan(a)hattne.se>se>:
Here goes:
# ceph -s
cluster:
id: e1327a10-8b8c-11ed-88b9-3cecef0e3946
health: HEALTH_OK
services:
mon: 5 daemons, quorum
bcgonen-a,bcgonen-b,bcgonen-c,bcgonen-r0h0,bcgonen-r0h1 (age 16h)
mgr: bcgonen-b.furndm(active, since 8d),
standbys:
bcgonen-a.qmmqxj
mds: 1/1 daemons up, 2 standby
osd: 36 osds: 36 up (since 16h), 36 in (since 3d); 1041
remapped pgs
data:
volumes: 1/1 healthy
pools: 3 pools, 1041 pgs
objects: 5.42M objects, 6.5 TiB
usage: 19 TiB used, 428 TiB / 447 TiB avail
pgs: 27087125/16252275 objects misplaced (166.667%)
1039 active+clean+remapped
2 active+clean+remapped+scrubbing+deep
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT
PRI-AFF
-14 149.02008 rack rack-1
-7 149.02008 host bcgonen-r1h0
20 hdd 14.55269 osd.20 up 1.00000
1.00000
21 hdd 14.55269 osd.21
up 1.00000
1.00000
22 hdd 14.55269 osd.22
up 1.00000
1.00000
23 hdd 14.55269 osd.23
up 1.00000
1.00000
24 hdd 14.55269 osd.24
up 1.00000
1.00000
25 hdd 14.55269 osd.25
up 1.00000
1.00000
26 hdd 14.55269 osd.26
up 1.00000
1.00000
27 hdd 14.55269 osd.27
up 1.00000
1.00000
28 hdd 14.55269 osd.28
up 1.00000
1.00000
29 hdd 14.55269 osd.29
up 1.00000
1.00000
34 ssd 1.74660 osd.34
up 1.00000
1.00000
35 ssd 1.74660 osd.35
up 1.00000
1.00000
-13 298.04016 rack rack-0
-3 149.02008 host bcgonen-r0h0
0 hdd 14.55269 osd.0 up 1.00000
1.00000
1 hdd 14.55269 osd.1
up 1.00000
1.00000
2 hdd 14.55269 osd.2
up 1.00000
1.00000
3 hdd 14.55269 osd.3
up 1.00000
1.00000
4 hdd 14.55269 osd.4
up 1.00000
1.00000
5 hdd 14.55269 osd.5
up 1.00000
1.00000
6 hdd 14.55269 osd.6
up 1.00000
1.00000
7 hdd 14.55269 osd.7
up 1.00000
1.00000
8 hdd 14.55269 osd.8
up 1.00000
1.00000
9 hdd 14.55269 osd.9
up 1.00000
1.00000
30 ssd 1.74660 osd.30
up 1.00000
1.00000
31 ssd 1.74660 osd.31
up 1.00000
1.00000
-5 149.02008 host
bcgonen-r0h1
10 hdd 14.55269 osd.10 up 1.00000
1.00000
11 hdd 14.55269 osd.11
up 1.00000
1.00000
12 hdd 14.55269 osd.12
up 1.00000
1.00000
13 hdd 14.55269 osd.13
up 1.00000
1.00000
14 hdd 14.55269 osd.14
up 1.00000
1.00000
15 hdd 14.55269 osd.15
up 1.00000
1.00000
16 hdd 14.55269 osd.16
up 1.00000
1.00000
17 hdd 14.55269 osd.17
up 1.00000
1.00000
18 hdd 14.55269 osd.18
up 1.00000
1.00000
19 hdd 14.55269 osd.19
up 1.00000
1.00000
32 ssd 1.74660 osd.32
up 1.00000
1.00000
33 ssd 1.74660 osd.33
up 1.00000
1.00000
-1 0 root default
# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 31 flags
hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr
pool 2 'cephfs.cephfs.meta'
replicated size 3 min_size 2 crush_rule
2 object_hash rjenkins pg_num 16 pgp_num 16
autoscale_mode on last_change
9833 lfor 0/0/584 flags hashpspool stripe_width 0 pg_autoscale_bias 4
pg_num_min 16 recovery_priority 5 application cephfs
pool 3 'cephfs.cephfs.data'
replicated size 3 min_size 2 crush_rule
1 object_hash rjenkins pg_num 1024 pgp_num
1024 autoscale_mode on
last_change 7630 lfor 0/1831/6544 flags hashpspool,bulk stripe_width 0
application cephfs
crush_rules 1 and 2 are just used to assign the data and meta pool
to HDD and
SSD, respectively (failure domain: host).
// J
On 2023-03-31 15:37, ceph(a)elchaka.de wrote:
Need to know some more about your cluster...
Ceph -s
Ceph osd df tree
Replica or ec?
...
Perhaps this can give us some insight
Mehmet
Am 31. März 2023 18:08:38 MESZ schrieb Johan Hattne
<johan(a)hattne.se>se>:
Dear all;
Up until a few hours ago, I had a seemingly normally-behaving
cluster (Quincy, 17.2.5) with 36 OSDs, evenly distributed across
3 of its 6 nodes. The cluster is only used for CephFS and the
only non-standard configuration I can think of is that I had 2
active MDSs, but only 1 standby. I had also doubled
mds_cache_memory limit to 8 GB (all OSD hosts have 256 G of RAM)
at some point in the past.
Then I rebooted one of the OSD nodes. The rebooted node held one
of the active MDSs. Now the node is back up: ceph -s says the
cluster is healthy, but all PGs are in a active+clean+remapped
state and 166.67% of the objects are misplaced (dashboard:
-66.66% healthy).
The data pool is a threefold replica with 5.4M object, the
number of misplaced objects is reported as 27087410/16252446.
The denominator in the ratio makes sense to me (16.2M / 3 =
5.4M), but the numerator does not. I also note that the ratio is
*exactly* 5 / 3. The filesystem is still mounted and appears to
be usable, but df reports it as 100% full; I suspect it would
say 167% but that is capped somewhere.
Any ideas about what is going on? Any suggestions for recovery?
// Best wishes; Johan
------------------------------------------------------------------------
ceph-users mailing list --
ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
387
days inactive
393
days old
6 comments
3 participants
participants (3)
-
ceph＠elchaka.de -
Joachim Kraftmayer -
Johan Hattne