25 Oct
2020
25 Oct
'20
4:39 a.m.
Hi, my cluster was crashed by going down one of my DC and 'ceph -s'
status dont show me the current working status and nothing change in
large time, how can i see what is ceph doing really:
cluster:
health: HEALTH_ERR
mons fond-beagle,guided-tuna are using a lot of disk space
1/3 mons down, quorum fond-beagle,guided-tuna
18/404368 objects unfound (0.004%)
Reduced data availability: 235 pgs inactive, 72 pgs down, 9
pgs incomplete
Possible data damage: 3 pgs recovery_unfound
Degraded data redundancy: 306574/2607020 objects degraded
(11.760%), 10 pgs degraded, 10 pgs undersized
2 pgs not deep-scrubbed in time
32408 slow ops, oldest one blocked for 62348 sec, daemons
[osd.0,osd.10,osd.11,osd.13,osd.14,osd.15,osd.16,osd.17,osd.18,osd.19]...
have slow ops.
services:
mon: 3 daemons, quorum fond-beagle,guided-tuna (age 31m), out of
quorum: alive-lynx
mgr: fond-beagle(active, since 31m)
osd: 52 osds: 28 up (since 30m), 28 in (since 11h); 3 remapped pgs
data:
pools: 7 pools, 2305 pgs
objects: 404.37k objects, 1.7 TiB
usage: 2.7 TiB used, 22 TiB / 24 TiB avail
pgs: 6.681% pgs unknown
3.514% pgs not active
306574/2607020 objects degraded (11.760%)
18/404368 objects unfound (0.004%)
2060 active+clean
154 unknown
72 down
9 incomplete
7 active+undersized+degraded
3 active+recovery_unfound+undersized+degraded+remapped