27 Jan
2023
27 Jan
'23
12:59 p.m.
Hi,
my ceph cluster started to show HEALTH_WARN, there are no healthy pgs left,
all are unknown, but it seems my cephfs is still readable, how to
investigate this any further?
$ sudo ceph -s
cluster:
id: ddb7ebd8-65b5-11ed-84d7-22aca0408523
health: HEALTH_WARN
failed to probe daemons or devices
noout flag(s) set
Reduced data availability: 339 pgs inactive
services:
mon: 1 daemons, quorum flucky-server (age 3m)
mgr: flucky-server.cupbak(active, since 3m)
mds: 1/1 daemons up
osd: 18 osds: 18 up (since 26h), 18 in (since 7w)
flags noout
rgw: 1 daemon active (1 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 11 pools, 339 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs: 100.000% pgs unknown
339 unknown
$ sudo ceph fs status
cephfs - 2 clients
======
RANK STATE MDS ACTIVITY DNS INOS
DIRS CAPS
0 active cephfs.flucky-server.ldzavv Reqs: 0 /s 61.9k 61.9k
17.1k 54.5k
POOL TYPE USED AVAIL
cephfs_metadata metadata 0 0
cephfs_data data 0 0
MDS version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757)
quincy (stable)
$ docker logs ceph-ddb7ebd8-65b5-11ed-84d7-22aca0408523-mon-flucky-server
cluster 2023-01-27T12:15:30.437140+0000 mgr.flucky-server.cupbak
(mgr.144098) 200 : cluster [DBG] pgmap v189: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
debug 2023-01-27T12:15:31.995+0000 7fa90b3f7700 1
mon.flucky-server(a)0(leader).osd
e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232
full_alloc: 348127232 kv_alloc: 322961408
cluster 2023-01-27T12:15:32.437854+0000 mgr.flucky-server.cupbak
(mgr.144098) 201 : cluster [DBG] pgmap v190: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:32.373735+0000 osd.9 (osd.9) 123948 : cluster
[DBG] 9.a deep-scrub starts
cluster 2023-01-27T12:15:33.013990+0000 osd.2 (osd.2) 41797 : cluster [DBG]
5.6 scrub starts
cluster 2023-01-27T12:15:33.402881+0000 osd.9 (osd.9) 123949 : cluster
[DBG] 9.13 scrub starts
cluster 2023-01-27T12:15:34.438591+0000 mgr.flucky-server.cupbak
(mgr.144098) 202 : cluster [DBG] pgmap v191: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:35.461575+0000 osd.9 (osd.9) 123950 : cluster
[DBG] 7.16 deep-scrub starts
debug 2023-01-27T12:15:37.005+0000 7fa90b3f7700 1
mon.flucky-server(a)0(leader).osd
e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232
full_alloc: 348127232 kv_alloc: 322961408
cluster 2023-01-27T12:15:36.439416+0000 mgr.flucky-server.cupbak
(mgr.144098) 203 : cluster [DBG] pgmap v192: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:36.925368+0000 osd.2 (osd.2) 41798 : cluster [DBG]
7.15 deep-scrub starts
cluster 2023-01-27T12:15:37.960907+0000 osd.2 (osd.2) 41799 : cluster [DBG]
6.6 scrub starts
cluster 2023-01-27T12:15:38.440099+0000 mgr.flucky-server.cupbak
(mgr.144098) 204 : cluster [DBG] pgmap v193: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:38.482333+0000 osd.9 (osd.9) 123951 : cluster
[DBG] 2.2 scrub starts
cluster 2023-01-27T12:15:38.959557+0000 osd.2 (osd.2) 41800 : cluster [DBG]
9.47 scrub starts
cluster 2023-01-27T12:15:39.519980+0000 osd.9 (osd.9) 123952 : cluster
[DBG] 4.b scrub starts
cluster 2023-01-27T12:15:40.440711+0000 mgr.flucky-server.cupbak
(mgr.144098) 205 : cluster [DBG] pgmap v194: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
debug 2023-01-27T12:15:42.012+0000 7fa90b3f7700 1
mon.flucky-server(a)0(leader).osd
e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232
full_alloc: 348127232 kv_alloc: 322961408
cluster 2023-01-27T12:15:41.536421+0000 osd.9 (osd.9) 123953 : cluster
[DBG] 2.7 scrub starts
cluster 2023-01-27T12:15:42.441314+0000 mgr.flucky-server.cupbak
(mgr.144098) 206 : cluster [DBG] pgmap v195: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:43.954128+0000 osd.2 (osd.2) 41801 : cluster [DBG]
9.4f scrub starts
cluster 2023-01-27T12:15:44.441897+0000 mgr.flucky-server.cupbak
(mgr.144098) 207 : cluster [DBG] pgmap v196: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:45.944038+0000 osd.2 (osd.2) 41802 : cluster [DBG]
1.1f deep-scrub starts
debug 2023-01-27T12:15:47.019+0000 7fa90b3f7700 1
mon.flucky-server(a)0(leader).osd
e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232
full_alloc: 348127232 kv_alloc: 322961408
cluster 2023-01-27T12:15:46.442532+0000 mgr.flucky-server.cupbak
(mgr.144098) 208 : cluster [DBG] pgmap v197: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:47.543275+0000 osd.9 (osd.9) 123954 : cluster
[DBG] 2.3 scrub starts
cluster 2023-01-27T12:15:48.443081+0000 mgr.flucky-server.cupbak
(mgr.144098) 209 : cluster [DBG] pgmap v198: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:48.515994+0000 osd.9 (osd.9) 123955 : cluster
[DBG] 1.19 scrub starts
cluster 2023-01-27T12:15:49.957501+0000 osd.2 (osd.2) 41803 : cluster [DBG]
7.11 scrub starts
cluster 2023-01-27T12:15:50.443740+0000 mgr.flucky-server.cupbak
(mgr.144098) 210 : cluster [DBG] pgmap v199: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:50.473278+0000 osd.9 (osd.9) 123956 : cluster
[DBG] 5.10 scrub starts
debug 2023-01-27T12:15:52.026+0000 7fa90b3f7700 1
mon.flucky-server(a)0(leader).osd
e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232
full_alloc: 348127232 kv_alloc: 322961408
cluster 2023-01-27T12:15:51.506790+0000 osd.9 (osd.9) 123957 : cluster
[DBG] 5.1b deep-scrub starts
cluster 2023-01-27T12:15:51.957026+0000 osd.2 (osd.2) 41804 : cluster [DBG]
4.16 scrub starts
cluster 2023-01-27T12:15:52.444197+0000 mgr.flucky-server.cupbak
(mgr.144098) 211 : cluster [DBG] pgmap v200: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:52.939466+0000 osd.2 (osd.2) 41805 : cluster [DBG]
5.1c scrub starts
cluster 2023-01-27T12:15:53.470511+0000 osd.9 (osd.9) 123958 : cluster
[DBG] 8.8 scrub starts
cluster 2023-01-27T12:15:53.916653+0000 osd.2 (osd.2) 41806 : cluster [DBG]
5.6 deep-scrub starts
cluster 2023-01-27T12:15:54.422547+0000 osd.9 (osd.9) 123959 : cluster
[DBG] 9.3b deep-scrub starts
cluster 2023-01-27T12:15:54.444675+0000 mgr.flucky-server.cupbak
(mgr.144098) 212 : cluster [DBG] pgmap v201: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:55.409322+0000 osd.9 (osd.9) 123960 : cluster
[DBG] 9.34 deep-scrub starts
cluster 2023-01-27T12:15:55.921989+0000 osd.2 (osd.2) 41807 : cluster [DBG]
7.15 deep-scrub starts
debug 2023-01-27T12:15:57.029+0000 7fa90b3f7700 1
mon.flucky-server(a)0(leader).osd
e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232
full_alloc: 348127232 kv_alloc: 322961408
audit 2023-01-27T12:15:56.339185+0000 mgr.flucky-server.cupbak (mgr.144098)
213 : audit [DBG] from='client.144120 -' entity='client.admin'
cmd=[{"prefix": "fs status", "target": ["mon-mgr",
""]}]: dispatch
cluster 2023-01-27T12:15:56.445186+0000 mgr.flucky-server.cupbak
(mgr.144098) 214 : cluster [DBG] pgmap v202: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:57.883819+0000 osd.2 (osd.2) 41808 : cluster [DBG]
6.6 deep-scrub starts
cluster 2023-01-27T12:15:58.445697+0000 mgr.flucky-server.cupbak
(mgr.144098) 215 : cluster [DBG] pgmap v203: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:15:59.415908+0000 osd.9 (osd.9) 123961 : cluster
[DBG] 9.25 scrub starts
cluster 2023-01-27T12:16:00.446210+0000 mgr.flucky-server.cupbak
(mgr.144098) 216 : cluster [DBG] pgmap v204: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
debug 2023-01-27T12:16:02.033+0000 7fa90b3f7700 1
mon.flucky-server(a)0(leader).osd
e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232
full_alloc: 348127232 kv_alloc: 322961408
cluster 2023-01-27T12:16:02.446670+0000 mgr.flucky-server.cupbak
(mgr.144098) 217 : cluster [DBG] pgmap v205: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
debug 2023-01-27T12:16:04.953+0000 7fa908bf2700 0 mon.flucky-server@0(leader)
e1 handle_command mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/mirror_snapshot_schedule"}
v 0) v1
debug 2023-01-27T12:16:04.953+0000 7fa908bf2700 0 log_channel(audit) log
[INF] : from='mgr.144098 172.18.0.1:0/3192812764'
entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/mirror_snapshot_schedule"}]:
dispatch
debug 2023-01-27T12:16:04.969+0000 7fa908bf2700 0 mon.flucky-server@0(leader)
e1 handle_command mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/trash_purge_schedule"}
v 0) v1
debug 2023-01-27T12:16:04.969+0000 7fa908bf2700 0 log_channel(audit) log
[INF] : from='mgr.144098 172.18.0.1:0/3192812764'
entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/trash_purge_schedule"}]:
dispatch
cluster 2023-01-27T12:16:04.447207+0000 mgr.flucky-server.cupbak
(mgr.144098) 218 : cluster [DBG] pgmap v206: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:16:04.537785+0000 osd.9 (osd.9) 123962 : cluster
[DBG] 9.27 scrub starts
cluster 2023-01-27T12:16:04.795757+0000 osd.2 (osd.2) 41809 : cluster [DBG]
9.47 scrub starts
audit 2023-01-27T12:16:04.956941+0000 mon.flucky-server (mon.0) 304 : audit
[INF] from='mgr.144098 172.18.0.1:0/3192812764'
entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/mirror_snapshot_schedule"}]:
dispatch
audit 2023-01-27T12:16:04.973875+0000 mon.flucky-server (mon.0) 305 : audit
[INF] from='mgr.144098 172.18.0.1:0/3192812764'
entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/trash_purge_schedule"}]:
dispatch
debug 2023-01-27T12:16:07.039+0000 7fa90b3f7700 1
mon.flucky-server(a)0(leader).osd
e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232
full_alloc: 348127232 kv_alloc: 322961408
cluster 2023-01-27T12:16:06.447964+0000 mgr.flucky-server.cupbak
(mgr.144098) 219 : cluster [DBG] pgmap v207: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:16:07.606921+0000 osd.9 (osd.9) 123963 : cluster
[DBG] 9.1c scrub starts
cluster 2023-01-27T12:16:08.448450+0000 mgr.flucky-server.cupbak
(mgr.144098) 220 : cluster [DBG] pgmap v208: 339 pgs: 339 unknown; 0 B
data, 0 B used, 0 B / 0 B avail
cluster 2023-01-27T12:16:08.629529+0000 osd.9 (osd.9) 123964 : cluster
[DBG] 9.2c scrub starts