18 Apr
2024
18 Apr
'24
2:42 p.m.
Dear colleagues, hope that anybody can help us.
The initial point: Ceph cluster v15.2 (installed and controlled by the Proxmox) with 3
nodes based on physical servers rented from a cloud provider. CephFS is installed also.
Yesterday we discovered that some of the applications stopped working. During the
investigation we recognized that we have the problem with Ceph, more precisely with СephFS
- MDS daemons suddenly crashed. We tried to restart them and found that they crashed again
immediately after the start. The crash information:
2024-04-17T17:47:42.841+0000 7f959ced9700 1 mds.0.29134 recovery_done -- successful
recovery!
2024-04-17T17:47:42.853+0000 7f959ced9700 1 mds.0.29134 active_start
2024-04-17T17:47:42.881+0000 7f959ced9700 1 mds.0.29134 cluster recovered.
2024-04-17T17:47:43.825+0000 7f959aed5700 -1 ./src/mds/OpenFileTable.cc: In function
'void OpenFileTable::commit(MDSContext*, uint64_t, int)' thread 7f959aed5700 time
2024-04-17T17:47:43.831243+0000
./src/mds/OpenFileTable.cc: 549: FAILED ceph_assert(count > 0)
Next hours we read the tons of articles, studied the documentation, and checked the common
state of Ceph cluster by the various diagnostic commands – but didn’t find anything wrong.
At evening we decided to upgrade it up to v16, and finally to v17.2.7. Unfortunately, it
didn’t solve the problem, MDS continue to crash with the same error. The only difference
that we found is “1 MDSs report damaged metadata” in the output of ceph -s – see it
below.
I supposed that it may be the well-known bug, but couldn’t find the same one on
https://tracker.ceph.com - there are several bugs associated with file OpenFileTable.cc
but not related to ceph_assert(count > 0)
We tried to check the source code of OpenFileTable.cc also, here is a fragment of it, in
function OpenFileTable::_journal_finish
int omap_idx = anchor.omap_idx;
unsigned& count = omap_num_items.at(omap_idx);
ceph_assert(count > 0);
So, we guess that the object map is empty for some object in Ceph, and it is unexpected
behavior. But again, we found nothing wrong in our cluster…
Next, we started with https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/
article – tried to reset the journal (despite that it was Ok all the time) and wipe the
sessions using cephfs-table-tool all reset session command. No result…
Now I decided to continue following this article and run cephfs-data-scan scan_extents
command, it is working just now. But I have a doubt that it will solve the issue because
of no problem with our objects in Ceph.
Is it the new bug? or something else? Any idea is welcome!
The important outputs:
----- ceph -s
cluster:
id: 4cd1c477-c8d0-4855-a1f1-cb71d89427ed
health: HEALTH_ERR
1 MDSs report damaged metadata
insufficient standby MDS daemons available
83 daemons have recently crashed
3 mgr modules have recently crashed
services:
mon: 3 daemons, quorum asrv-dev-stor-2,asrv-dev-stor-3,asrv-dev-stor-1 (age 22h)
mgr: asrv-dev-stor-2(active, since 22h), standbys: asrv-dev-stor-1
mds: 1/1 daemons up
osd: 18 osds: 18 up (since 22h), 18 in (since 29h)
data:
volumes: 1/1 healthy
pools: 5 pools, 289 pgs
objects: 29.72M objects, 5.6 TiB
usage: 21 TiB used, 47 TiB / 68 TiB avail
pgs: 287 active+clean
2 active+clean+scrubbing+deep
io:
client: 2.5 KiB/s rd, 172 KiB/s wr, 261 op/s rd, 195 op/s wr
-----ceph fs dump
e29480
enable_multiple, ever_enabled_multiple: 0,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned
encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file
layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'cephfs' (1)
fs_name cephfs
epoch 29480
flags 12 joinable allow_snaps allow_multimds_snaps
created 2022-11-25T15:56:08.507407+0000
modified 2024-04-18T16:52:29.970504+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 14728
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default
file layouts on dirs,4=dir inode in separate object,5=mds uses versioned
encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file
layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=156636152}
failed
damaged
stopped
data_pools [5]
metadata_pool 6
inline_data disabled
balancer
standby_count_wanted 1
[mds.asrv-dev-stor-1{0:156636152} state up:active seq 6 laggy since
2024-04-18T16:52:29.970479+0000 addr
[v2:172.22.2.91:6800/2487054023,v1:172.22.2.91:6801/2487054023] compat
{c=[1],r=[1],i=[7ff]}]
-----cephfs-journal-tool --rank=cephfs:0 journal inspect
Overall journal integrity: OK
-----ceph pg dump summary
version 41137
stamp 2024-04-18T21:17:59.133536+0000
last_osdmap_epoch 0
last_pg_scan 0
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES
OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
sum 29717605 0 0 0 0 6112544251872
13374192956 28493480 1806575 1806575
OSD_STAT USED AVAIL USED_RAW TOTAL
sum 21 TiB 47 TiB 21 TiB 68 TiB
-----ceph pg dump pools
POOLID OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES
OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
8 31771 0 0 0 0 131337887503
2482 140 401246 401246
7 839707 0 0 0 0 3519034650971
736 61 399328 399328
6 1319576 0 0 0 0 421044421
13374189738 28493279 206749 206749
5 27526539 0 0 0 0 2461702171417
0 0 792165 792165
2 12 0 0 0 0 48497560
0 0 6991 6991
26 Apr
26 Apr
2:25 a.m.
Hello,
'almost all diagnostic ceph subcommands hang!' -> this triggered my bell.
We've had a similar issue with many ceph commands hanging due to a missing L3 ACL
between MGRs and a new MDS machine that we added to the cluster.
I second Eugen analysis: network issue, whatever the OSI layer.
Regards,
Frédéric.
----- Le 26 Avr 24, à 9:31, Eugen Block eblock(a)nde.ag a écrit :
> Hi, it's unlikely that all OSDs fail at the same time, it seems like a
> network issue. Do you have an active MGR? Just a couple of days ago
> someone reported incorrect OSD stats because no MGR was up. Although
> your 'ceph health detail' output doesn't mention that, there are still
> issues when MGR processes are active according to ceph but don't
> respond anymore.
> I would probably start with basic network debugging, e. g. iperf,
> pings on public and cluster networks (if present) and so on.
>
> Regards,
> Eugen
>
> Zitat von Alexey GERASIMOV <alexey.gerasimov(a)opencascade.com>om>:
>
>> Colleagues, I have the update.
>>
>> Starting from yestrerday the situation with ceph health is much
>> worse than it was previously.
>> We found that
>> - ceph -s informs us that some PGs are in stale state
>> - almost all diagnostic ceph subcommands hang! For example, "ceph
>> osd ls" , "ceph osd dump", "ceph osd tree", "ceph
health detail"
>> provide the output - but "ceph osd status", all the commands
"ceph
>> pg ..." and other ones hang.
>>
>> So, it looks that the crashes of MDS daemons were the first signs of
>> problems only.
>> I read that "stale" state for PGs means that all nodes storing this
>> placement group may be down - but it's wrong, all osd daemons are up
>> on all three nodes:
>>
>> ------- ceph osd tree
>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
>> -1 68.05609 root default
>> -3 22.68536 host asrv-dev-stor-1
>> 0 hdd 5.45799 osd.0 up 1.00000 1.00000
>> 1 hdd 5.45799 osd.1 up 1.00000 1.00000
>> 2 hdd 5.45799 osd.2 up 1.00000 1.00000
>> 3 hdd 5.45799 osd.3 up 1.00000 1.00000
>> 12 ssd 0.42670 osd.12 up 1.00000 1.00000
>> 13 ssd 0.42670 osd.13 up 1.00000 1.00000
>> -5 22.68536 host asrv-dev-stor-2
>> 4 hdd 5.45799 osd.4 up 1.00000 1.00000
>> 5 hdd 5.45799 osd.5 up 1.00000 1.00000
>> 6 hdd 5.45799 osd.6 up 1.00000 1.00000
>> 7 hdd 5.45799 osd.7 up 1.00000 1.00000
>> 14 ssd 0.42670 osd.14 up 1.00000 1.00000
>> 15 ssd 0.42670 osd.15 up 1.00000 1.00000
>> -7 22.68536 host asrv-dev-stor-3
>> 8 hdd 5.45799 osd.8 up 1.00000 1.00000
>> 10 hdd 5.45799 osd.10 up 1.00000 1.00000
>> 11 hdd 5.45799 osd.11 up 1.00000 1.00000
>> 18 hdd 5.45799 osd.18 up 1.00000 1.00000
>> 16 ssd 0.42670 osd.16 up 1.00000 1.00000
>> 17 ssd 0.42670 osd.17 up 1.00000 1.00000
>>
>> May it be the physical problem with our drives? "smartctl -a"
>> informs nothing wrong. We started the surface check using dd
>> command also but it will be 7 hours per drive at least...
>>
>> What should we do also?
>>
>> The output of "ceph health detail":
>>
>> ceph health detail
>> HEALTH_ERR 1 MDSs report damaged metadata; insufficient standby MDS
>> daemons available; Reduced data availability: 50 pgs stale; 90
>> daemons have recently crashed; 3 mgr modules have recently crashed
>> [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata
>> mds.asrv-dev-stor-2(mds.0): Metadata damage detected
>> [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
>> have 0; want 1 more
>> [WRN] PG_AVAILABILITY: Reduced data availability: 50 pgs stale
>> pg 5.0 is stuck stale for 67m, current state stale+active+clean,
>> last acting [4,1,11]
>> pg 5.13 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,0,10]
>> pg 5.18 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,11,2]
>> pg 5.19 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,3,10]
>> pg 5.1e is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,7,11]
>> pg 5.22 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,6,18]
>> pg 5.26 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,1,18]
>> pg 5.29 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,11,6]
>> pg 5.2b is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,18,6]
>> pg 5.30 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,8,7]
>> pg 5.37 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,10,0]
>> pg 5.3c is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,10,3]
>> pg 5.43 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,6,18]
>> pg 5.44 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,2,11]
>> pg 5.45 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,11,3]
>> pg 5.47 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,10,1]
>> pg 5.48 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,5,11]
>> pg 5.60 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,10,7]
>> pg 7.2 is stuck stale for 67m, current state stale+active+clean,
>> last acting [4,2,10]
>> pg 7.4 is stuck stale for 67m, current state stale+active+clean,
>> last acting [4,18,3]
>> pg 7.f is stuck stale for 10h, current state stale+active+clean,
>> last acting [0,4,8]
>> pg 7.13 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,7,11]
>> pg 7.18 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,0,10]
>> pg 7.1b is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,8,0]
>> pg 7.1f is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,5,11]
>> pg 7.2a is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,6,8]
>> pg 7.35 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,6,10]
>> pg 7.36 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,2,8]
>> pg 7.37 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,8,7]
>> pg 7.38 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,6,11]
>> pg 9.10 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,0,8]
>> pg 9.16 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,4,11]
>> pg 9.20 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,3,8]
>> pg 9.2a is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,8,0]
>> pg 9.33 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,18,5]
>> pg 9.3a is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,8,5]
>> pg 9.48 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,2,11]
>> pg 9.4b is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,7,11]
>> pg 9.4f is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,6,10]
>> pg 9.52 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,8,0]
>> pg 9.53 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,11,7]
>> pg 9.56 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,5,18]
>> pg 9.5a is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,7,8]
>> pg 9.5d is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,6,10]
>> pg 9.6b is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,11,0]
>> pg 9.6f is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,2,18]
>> pg 9.73 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,2,10]
>> pg 9.76 is stuck stale for 67m, current state
>> stale+active+clean, last acting [4,10,2]
>> pg 9.79 is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,6,8]
>> pg 9.7f is stuck stale for 10h, current state
>> stale+active+clean, last acting [0,10,5]
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:31 a.m.
Hi, it's unlikely that all OSDs fail at the same time, it seems like a
network issue. Do you have an active MGR? Just a couple of days ago
someone reported incorrect OSD stats because no MGR was up. Although
your 'ceph health detail' output doesn't mention that, there are still
issues when MGR processes are active according to ceph but don't
respond anymore.
I would probably start with basic network debugging, e. g. iperf,
pings on public and cluster networks (if present) and so on.
Regards,
Eugen
Zitat von Alexey GERASIMOV <alexey.gerasimov(a)opencascade.com>om>:
Colleagues, I have the update.
Starting from yestrerday the situation with ceph health is much
worse than it was previously.
We found that
- ceph -s informs us that some PGs are in stale state
- almost all diagnostic ceph subcommands hang! For example, "ceph
osd ls" , "ceph osd dump", "ceph osd tree", "ceph health
detail"
provide the output - but "ceph osd status", all the commands "ceph
pg ..." and other ones hang.
So, it looks that the crashes of MDS daemons were the first signs of
problems only.
I read that "stale" state for PGs means that all nodes storing this
placement group may be down - but it's wrong, all osd daemons are up
on all three nodes:
------- ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 68.05609 root default
-3 22.68536 host asrv-dev-stor-1
0 hdd 5.45799 osd.0 up 1.00000 1.00000
1 hdd 5.45799 osd.1 up 1.00000 1.00000
2 hdd 5.45799 osd.2 up 1.00000 1.00000
3 hdd 5.45799 osd.3 up 1.00000 1.00000
12 ssd 0.42670 osd.12 up 1.00000 1.00000
13 ssd 0.42670 osd.13 up 1.00000 1.00000
-5 22.68536 host asrv-dev-stor-2
4 hdd 5.45799 osd.4 up 1.00000 1.00000
5 hdd 5.45799 osd.5 up 1.00000 1.00000
6 hdd 5.45799 osd.6 up 1.00000 1.00000
7 hdd 5.45799 osd.7 up 1.00000 1.00000
14 ssd 0.42670 osd.14 up 1.00000 1.00000
15 ssd 0.42670 osd.15 up 1.00000 1.00000
-7 22.68536 host asrv-dev-stor-3
8 hdd 5.45799 osd.8 up 1.00000 1.00000
10 hdd 5.45799 osd.10 up 1.00000 1.00000
11 hdd 5.45799 osd.11 up 1.00000 1.00000
18 hdd 5.45799 osd.18 up 1.00000 1.00000
16 ssd 0.42670 osd.16 up 1.00000 1.00000
17 ssd 0.42670 osd.17 up 1.00000 1.00000
May it be the physical problem with our drives? "smartctl -a"
informs nothing wrong. We started the surface check using dd
command also but it will be 7 hours per drive at least...
What should we do also?
The output of "ceph health detail":
ceph health detail
HEALTH_ERR 1 MDSs report damaged metadata; insufficient standby MDS
daemons available; Reduced data availability: 50 pgs stale; 90
daemons have recently crashed; 3 mgr modules have recently crashed
[ERR] MDS_DAMAGE: 1 MDSs report damaged metadata
mds.asrv-dev-stor-2(mds.0): Metadata damage detected
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
have 0; want 1 more
[WRN] PG_AVAILABILITY: Reduced data availability: 50 pgs stale
pg 5.0 is stuck stale for 67m, current state stale+active+clean,
last acting [4,1,11]
pg 5.13 is stuck stale for 67m, current state
stale+active+clean, last acting [4,0,10]
pg 5.18 is stuck stale for 67m, current state
stale+active+clean, last acting [4,11,2]
pg 5.19 is stuck stale for 67m, current state
stale+active+clean, last acting [4,3,10]
pg 5.1e is stuck stale for 10h, current state
stale+active+clean, last acting [0,7,11]
pg 5.22 is stuck stale for 10h, current state
stale+active+clean, last acting [0,6,18]
pg 5.26 is stuck stale for 67m, current state
stale+active+clean, last acting [4,1,18]
pg 5.29 is stuck stale for 10h, current state
stale+active+clean, last acting [0,11,6]
pg 5.2b is stuck stale for 10h, current state
stale+active+clean, last acting [0,18,6]
pg 5.30 is stuck stale for 10h, current state
stale+active+clean, last acting [0,8,7]
pg 5.37 is stuck stale for 67m, current state
stale+active+clean, last acting [4,10,0]
pg 5.3c is stuck stale for 67m, current state
stale+active+clean, last acting [4,10,3]
pg 5.43 is stuck stale for 10h, current state
stale+active+clean, last acting [0,6,18]
pg 5.44 is stuck stale for 67m, current state
stale+active+clean, last acting [4,2,11]
pg 5.45 is stuck stale for 67m, current state
stale+active+clean, last acting [4,11,3]
pg 5.47 is stuck stale for 67m, current state
stale+active+clean, last acting [4,10,1]
pg 5.48 is stuck stale for 10h, current state
stale+active+clean, last acting [0,5,11]
pg 5.60 is stuck stale for 10h, current state
stale+active+clean, last acting [0,10,7]
pg 7.2 is stuck stale for 67m, current state stale+active+clean,
last acting [4,2,10]
pg 7.4 is stuck stale for 67m, current state stale+active+clean,
last acting [4,18,3]
pg 7.f is stuck stale for 10h, current state stale+active+clean,
last acting [0,4,8]
pg 7.13 is stuck stale for 10h, current state
stale+active+clean, last acting [0,7,11]
pg 7.18 is stuck stale for 67m, current state
stale+active+clean, last acting [4,0,10]
pg 7.1b is stuck stale for 67m, current state
stale+active+clean, last acting [4,8,0]
pg 7.1f is stuck stale for 10h, current state
stale+active+clean, last acting [0,5,11]
pg 7.2a is stuck stale for 10h, current state
stale+active+clean, last acting [0,6,8]
pg 7.35 is stuck stale for 10h, current state
stale+active+clean, last acting [0,6,10]
pg 7.36 is stuck stale for 67m, current state
stale+active+clean, last acting [4,2,8]
pg 7.37 is stuck stale for 10h, current state
stale+active+clean, last acting [0,8,7]
pg 7.38 is stuck stale for 10h, current state
stale+active+clean, last acting [0,6,11]
pg 9.10 is stuck stale for 67m, current state
stale+active+clean, last acting [4,0,8]
pg 9.16 is stuck stale for 10h, current state
stale+active+clean, last acting [0,4,11]
pg 9.20 is stuck stale for 67m, current state
stale+active+clean, last acting [4,3,8]
pg 9.2a is stuck stale for 67m, current state
stale+active+clean, last acting [4,8,0]
pg 9.33 is stuck stale for 10h, current state
stale+active+clean, last acting [0,18,5]
pg 9.3a is stuck stale for 10h, current state
stale+active+clean, last acting [0,8,5]
pg 9.48 is stuck stale for 67m, current state
stale+active+clean, last acting [4,2,11]
pg 9.4b is stuck stale for 10h, current state
stale+active+clean, last acting [0,7,11]
pg 9.4f is stuck stale for 10h, current state
stale+active+clean, last acting [0,6,10]
pg 9.52 is stuck stale for 67m, current state
stale+active+clean, last acting [4,8,0]
pg 9.53 is stuck stale for 10h, current state
stale+active+clean, last acting [0,11,7]
pg 9.56 is stuck stale for 10h, current state
stale+active+clean, last acting [0,5,18]
pg 9.5a is stuck stale for 10h, current state
stale+active+clean, last acting [0,7,8]
pg 9.5d is stuck stale for 10h, current state
stale+active+clean, last acting [0,6,10]
pg 9.6b is stuck stale for 67m, current state
stale+active+clean, last acting [4,11,0]
pg 9.6f is stuck stale for 67m, current state
stale+active+clean, last acting [4,2,18]
pg 9.73 is stuck stale for 67m, current state
stale+active+clean, last acting [4,2,10]
pg 9.76 is stuck stale for 67m, current state
stale+active+clean, last acting [4,10,2]
pg 9.79 is stuck stale for 10h, current state
stale+active+clean, last acting [0,6,8]
pg 9.7f is stuck stale for 10h, current state
stale+active+clean, last acting [0,10,5]
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 Apr
25 Apr
8:44 a.m.
Colleagues, I have the update.
Starting from yestrerday the situation with ceph health is much worse than it was
previously.
We found that
- ceph -s informs us that some PGs are in stale state
- almost all diagnostic ceph subcommands hang! For example, "ceph osd ls" ,
"ceph osd dump", "ceph osd tree", "ceph health detail"
provide the output - but "ceph osd status", all the commands "ceph pg
..." and other ones hang.
So, it looks that the crashes of MDS daemons were the first signs of problems only.
I read that "stale" state for PGs means that all nodes storing this placement
group may be down - but it's wrong, all osd daemons are up on all three nodes:
------- ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 68.05609 root default
-3 22.68536 host asrv-dev-stor-1
0 hdd 5.45799 osd.0 up 1.00000 1.00000
1 hdd 5.45799 osd.1 up 1.00000 1.00000
2 hdd 5.45799 osd.2 up 1.00000 1.00000
3 hdd 5.45799 osd.3 up 1.00000 1.00000
12 ssd 0.42670 osd.12 up 1.00000 1.00000
13 ssd 0.42670 osd.13 up 1.00000 1.00000
-5 22.68536 host asrv-dev-stor-2
4 hdd 5.45799 osd.4 up 1.00000 1.00000
5 hdd 5.45799 osd.5 up 1.00000 1.00000
6 hdd 5.45799 osd.6 up 1.00000 1.00000
7 hdd 5.45799 osd.7 up 1.00000 1.00000
14 ssd 0.42670 osd.14 up 1.00000 1.00000
15 ssd 0.42670 osd.15 up 1.00000 1.00000
-7 22.68536 host asrv-dev-stor-3
8 hdd 5.45799 osd.8 up 1.00000 1.00000
10 hdd 5.45799 osd.10 up 1.00000 1.00000
11 hdd 5.45799 osd.11 up 1.00000 1.00000
18 hdd 5.45799 osd.18 up 1.00000 1.00000
16 ssd 0.42670 osd.16 up 1.00000 1.00000
17 ssd 0.42670 osd.17 up 1.00000 1.00000
May it be the physical problem with our drives? "smartctl -a" informs nothing
wrong. We started the surface check using dd command also but it will be 7 hours per
drive at least...
What should we do also?
The output of "ceph health detail":
ceph health detail
HEALTH_ERR 1 MDSs report damaged metadata; insufficient standby MDS daemons available;
Reduced data availability: 50 pgs stale; 90 daemons have recently crashed; 3 mgr modules
have recently crashed
[ERR] MDS_DAMAGE: 1 MDSs report damaged metadata
mds.asrv-dev-stor-2(mds.0): Metadata damage detected
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
have 0; want 1 more
[WRN] PG_AVAILABILITY: Reduced data availability: 50 pgs stale
pg 5.0 is stuck stale for 67m, current state stale+active+clean, last acting
[4,1,11]
pg 5.13 is stuck stale for 67m, current state stale+active+clean, last acting
[4,0,10]
pg 5.18 is stuck stale for 67m, current state stale+active+clean, last acting
[4,11,2]
pg 5.19 is stuck stale for 67m, current state stale+active+clean, last acting
[4,3,10]
pg 5.1e is stuck stale for 10h, current state stale+active+clean, last acting
[0,7,11]
pg 5.22 is stuck stale for 10h, current state stale+active+clean, last acting
[0,6,18]
pg 5.26 is stuck stale for 67m, current state stale+active+clean, last acting
[4,1,18]
pg 5.29 is stuck stale for 10h, current state stale+active+clean, last acting
[0,11,6]
pg 5.2b is stuck stale for 10h, current state stale+active+clean, last acting
[0,18,6]
pg 5.30 is stuck stale for 10h, current state stale+active+clean, last acting
[0,8,7]
pg 5.37 is stuck stale for 67m, current state stale+active+clean, last acting
[4,10,0]
pg 5.3c is stuck stale for 67m, current state stale+active+clean, last acting
[4,10,3]
pg 5.43 is stuck stale for 10h, current state stale+active+clean, last acting
[0,6,18]
pg 5.44 is stuck stale for 67m, current state stale+active+clean, last acting
[4,2,11]
pg 5.45 is stuck stale for 67m, current state stale+active+clean, last acting
[4,11,3]
pg 5.47 is stuck stale for 67m, current state stale+active+clean, last acting
[4,10,1]
pg 5.48 is stuck stale for 10h, current state stale+active+clean, last acting
[0,5,11]
pg 5.60 is stuck stale for 10h, current state stale+active+clean, last acting
[0,10,7]
pg 7.2 is stuck stale for 67m, current state stale+active+clean, last acting
[4,2,10]
pg 7.4 is stuck stale for 67m, current state stale+active+clean, last acting
[4,18,3]
pg 7.f is stuck stale for 10h, current state stale+active+clean, last acting [0,4,8]
pg 7.13 is stuck stale for 10h, current state stale+active+clean, last acting
[0,7,11]
pg 7.18 is stuck stale for 67m, current state stale+active+clean, last acting
[4,0,10]
pg 7.1b is stuck stale for 67m, current state stale+active+clean, last acting
[4,8,0]
pg 7.1f is stuck stale for 10h, current state stale+active+clean, last acting
[0,5,11]
pg 7.2a is stuck stale for 10h, current state stale+active+clean, last acting
[0,6,8]
pg 7.35 is stuck stale for 10h, current state stale+active+clean, last acting
[0,6,10]
pg 7.36 is stuck stale for 67m, current state stale+active+clean, last acting
[4,2,8]
pg 7.37 is stuck stale for 10h, current state stale+active+clean, last acting
[0,8,7]
pg 7.38 is stuck stale for 10h, current state stale+active+clean, last acting
[0,6,11]
pg 9.10 is stuck stale for 67m, current state stale+active+clean, last acting
[4,0,8]
pg 9.16 is stuck stale for 10h, current state stale+active+clean, last acting
[0,4,11]
pg 9.20 is stuck stale for 67m, current state stale+active+clean, last acting
[4,3,8]
pg 9.2a is stuck stale for 67m, current state stale+active+clean, last acting
[4,8,0]
pg 9.33 is stuck stale for 10h, current state stale+active+clean, last acting
[0,18,5]
pg 9.3a is stuck stale for 10h, current state stale+active+clean, last acting
[0,8,5]
pg 9.48 is stuck stale for 67m, current state stale+active+clean, last acting
[4,2,11]
pg 9.4b is stuck stale for 10h, current state stale+active+clean, last acting
[0,7,11]
pg 9.4f is stuck stale for 10h, current state stale+active+clean, last acting
[0,6,10]
pg 9.52 is stuck stale for 67m, current state stale+active+clean, last acting
[4,8,0]
pg 9.53 is stuck stale for 10h, current state stale+active+clean, last acting
[0,11,7]
pg 9.56 is stuck stale for 10h, current state stale+active+clean, last acting
[0,5,18]
pg 9.5a is stuck stale for 10h, current state stale+active+clean, last acting
[0,7,8]
pg 9.5d is stuck stale for 10h, current state stale+active+clean, last acting
[0,6,10]
pg 9.6b is stuck stale for 67m, current state stale+active+clean, last acting
[4,11,0]
pg 9.6f is stuck stale for 67m, current state stale+active+clean, last acting
[4,2,18]
pg 9.73 is stuck stale for 67m, current state stale+active+clean, last acting
[4,2,10]
pg 9.76 is stuck stale for 67m, current state stale+active+clean, last acting
[4,10,2]
pg 9.79 is stuck stale for 10h, current state stale+active+clean, last acting
[0,6,8]
pg 9.7f is stuck stale for 10h, current state stale+active+clean, last acting
[0,10,5]
22 Apr
22 Apr
1:48 a.m.
Right, I just figured from the health output you would have a couple
of seconds or so to query the daemon:
> mds: 1/1 daemons up
Zitat von Alexey GERASIMOV <alexey.gerasimov(a)opencascade.com>om>:
> Ok, we will create the ticket.
>
> Eugen Block - ceph tell command needs to communicate with the MDS
> daemon running, but it is crashed. So, I just have the information
> about the impossibility to receive the information from daemon:
>
> ceph tell mds.0 damage ls
> Error ENOENT: problem getting command descriptions from mds.0
>
> ---
> Best regards,
>
> Alexey Gerasimov
> System Manager
>
>
> www.opencascade.com
> www.capgemini.com
>
>
>
> -----Original Message-----
> From: Xiubo Li <xiubli(a)redhat.com>
> Sent: Monday, April 22, 2024 2:21 AM
> To: Alexey GERASIMOV <alexey.gerasimov(a)opencascade.com>om>; ceph-users(a)ceph.io
> Subject: Re: [ceph-users] MDS crash
>
> Hi Alexey,
>
> This looks a new issue for me. Please create a tracker for it and
> provide the detail call trace there.
>
> Thanks
>
> - Xiubo
>
> On 4/19/24 05:42, alexey.gerasimov(a)opencascade.com wrote:
>> Dear colleagues, hope that anybody can help us.
>>
>> The initial point: Ceph cluster v15.2 (installed and controlled by
>> the Proxmox) with 3 nodes based on physical servers rented from a
>> cloud provider. CephFS is installed also.
>>
>> Yesterday we discovered that some of the applications stopped
>> working. During the investigation we recognized that we have the
>> problem with Ceph, more precisely with СephFS - MDS daemons
>> suddenly crashed. We tried to restart them and found that they
>> crashed again immediately after the start. The crash information:
>> 2024-04-17T17:47:42.841+0000 7f959ced9700 1 mds.0.29134
>> recovery_done -- successful recovery!
>> 2024-04-17T17:47:42.853+0000 7f959ced9700 1 mds.0.29134 active_start
>> 2024-04-17T17:47:42.881+0000 7f959ced9700 1 mds.0.29134 cluster recovered.
>> 2024-04-17T17:47:43.825+0000 7f959aed5700 -1
>> ./src/mds/OpenFileTable.cc: In function 'void
>> OpenFileTable::commit(MDSContext*, uint64_t, int)' thread 7f959aed5700
>> time 2024-04-17T17:47:43.831243+0000
>> ./src/mds/OpenFileTable.cc: 549: FAILED ceph_assert(count > 0)
>>
>> Next hours we read the tons of articles, studied the documentation,
>> and checked the common state of Ceph cluster by the various
>> diagnostic commands – but didn’t find anything wrong. At evening we
>> decided to upgrade it up to v16, and finally to v17.2.7.
>> Unfortunately, it didn’t solve the problem, MDS continue to crash
>> with the same error. The only difference that we found is “1 MDSs
>> report damaged metadata” in the output of ceph -s – see it below.
>>
>> I supposed that it may be the well-known bug, but couldn’t find the
>> same one on https://tracker.ceph.com - there are several bugs
>> associated with file OpenFileTable.cc but not related to
>> ceph_assert(count > 0)
>>
>> We tried to check the source code of OpenFileTable.cc also, here is
>> a fragment of it, in function OpenFileTable::_journal_finish
>> int omap_idx = anchor.omap_idx;
>> unsigned& count = omap_num_items.at(omap_idx);
>> ceph_assert(count > 0);
>> So, we guess that the object map is empty for some object in Ceph, and
>> it is unexpected behavior. But again, we found nothing wrong in our
>> cluster…
>>
>> Next, we started with
>> https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/
>> article – tried to reset the journal (despite that it was Ok all
>> the time) and wipe the sessions using cephfs-table-tool all reset
>> session command. No result… Now I decided to continue following
>> this article and run cephfs-data-scan scan_extents command, it is
>> working just now. But I have a doubt that it will solve the issue
>> because of no problem with our objects in Ceph.
>>
>> Is it the new bug? or something else? Any idea is welcome!
>>
>> The important outputs:
>>
>> ----- ceph -s
>> cluster:
>> id: 4cd1c477-c8d0-4855-a1f1-cb71d89427ed
>> health: HEALTH_ERR
>> 1 MDSs report damaged metadata
>> insufficient standby MDS daemons available
>> 83 daemons have recently crashed
>> 3 mgr modules have recently crashed
>>
>> services:
>> mon: 3 daemons, quorum
>> asrv-dev-stor-2,asrv-dev-stor-3,asrv-dev-stor-1 (age 22h)
>> mgr: asrv-dev-stor-2(active, since 22h), standbys: asrv-dev-stor-1
> mds: 1/1 daemons up
>> osd:
18 osds: 18 up (since 22h), 18 in (since 29h)
>>
>> data:
>> volumes: 1/1 healthy
>> pools: 5 pools, 289 pgs
>> objects: 29.72M objects, 5.6 TiB
>> usage: 21 TiB used, 47 TiB / 68 TiB avail
>> pgs: 287 active+clean
>> 2 active+clean+scrubbing+deep
>>
>> io:
>> client: 2.5 KiB/s rd, 172 KiB/s wr, 261 op/s rd, 195 op/s wr
>>
>> -----ceph fs dump
>> e29480
>> enable_multiple, ever_enabled_multiple: 0,1 default compat:
>> compat={},rocompat={},incompat={1=base v0.20,2=client writeable
>> ranges,3=default file layouts on dirs,4=dir inode in separate
>> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds
>> uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
>> legacy client fscid: 1
>>
>> Filesystem 'cephfs' (1)
>> fs_name cephfs
>> epoch 29480
>> flags 12 joinable allow_snaps allow_multimds_snaps
>> created 2022-11-25T15:56:08.507407+0000
>> modified 2024-04-18T16:52:29.970504+0000
>> tableserver 0
>> root 0
>> session_timeout 60
>> session_autoclose 300
>> max_file_size 1099511627776
>> required_client_features {}
>> last_failure 0
>> last_failure_osd_epoch 14728
>> compat compat={},rocompat={},incompat={1=base v0.20,2=client
>> writeable ranges,3=default file layouts on dirs,4=dir inode in
>> separate object,5=mds uses versioned encoding,6=dirfrag is stored
>> in omap,7=mds uses inline data,8=no anchor table,9=file layout
>> v2,10=snaprealm v2} max_mds 1
>> in 0
>> up {0=156636152}
>> failed
>> damaged
>> stopped
>> data_pools [5]
>> metadata_pool 6
>> inline_data disabled
>> balancer
>> standby_count_wanted 1
>> [mds.asrv-dev-stor-1{0:156636152} state up:active seq 6 laggy since
>> 2024-04-18T16:52:29.970479+0000 addr
>> [v2:172.22.2.91:6800/2487054023,v1:172.22.2.91:6801/2487054023] compat
>> {c=[1],r=[1],i=[7ff]}]
>>
>> -----cephfs-journal-tool --rank=cephfs:0 journal inspect Overall
>> journal integrity: OK
>>
>> -----ceph pg dump summary
>> version 41137
>> stamp 2024-04-18T21:17:59.133536+0000
>> last_osdmap_epoch 0
>> last_pg_scan 0
>> PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND
>> BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
>> sum 29717605 0 0 0 0
>> 6112544251872 13374192956 28493480 1806575 1806575
>> OSD_STAT USED AVAIL USED_RAW TOTAL
>> sum 21 TiB 47 TiB 21 TiB 68 TiB
>>
>> -----ceph pg dump pools
>> POOLID OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND
>> BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
>> 8 31771 0 0 0 0
>> 131337887503 2482 140 401246 401246
>> 7 839707 0 0 0 0
>> 3519034650971 736 61 399328 399328
>> 6 1319576 0 0 0 0
>> 421044421 13374189738 28493279 206749 206749
>> 5 27526539 0 0 0 0
>> 2461702171417 0 0 792165 792165
>> 2 12 0 0 0 0
>> 48497560 0 0 6991 6991
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
>> email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:43 a.m.
Ok, we will create the ticket.
Eugen Block - ceph tell command needs to communicate with the MDS daemon running, but it
is crashed. So, I just have the information about the impossibility to receive the
information from daemon:
ceph tell mds.0 damage ls
Error ENOENT: problem getting command descriptions from mds.0
---
Best regards,
Alexey Gerasimov
System Manager
www.opencascade.com
www.capgemini.com
-----Original Message-----
From: Xiubo Li <xiubli(a)redhat.com>
Sent: Monday, April 22, 2024 2:21 AM
To: Alexey GERASIMOV <alexey.gerasimov(a)opencascade.com>om>; ceph-users(a)ceph.io
Subject: Re: [ceph-users] MDS crash
Hi Alexey,
This looks a new issue for me. Please create a tracker for it and provide the detail call
trace there.
Thanks
- Xiubo
On 4/19/24 05:42, alexey.gerasimov(a)opencascade.com wrote:
Dear colleagues, hope that anybody can help us.
The initial point: Ceph cluster v15.2 (installed and controlled by the Proxmox) with 3
nodes based on physical servers rented from a cloud provider. CephFS is installed also.
Yesterday we discovered that some of the applications stopped working. During the
investigation we recognized that we have the problem with Ceph, more precisely with СephFS
- MDS daemons suddenly crashed. We tried to restart them and found that they crashed again
immediately after the start. The crash information:
2024-04-17T17:47:42.841+0000 7f959ced9700 1 mds.0.29134 recovery_done -- successful
recovery!
2024-04-17T17:47:42.853+0000 7f959ced9700 1 mds.0.29134 active_start
2024-04-17T17:47:42.881+0000 7f959ced9700 1 mds.0.29134 cluster recovered.
2024-04-17T17:47:43.825+0000 7f959aed5700 -1
./src/mds/OpenFileTable.cc: In function 'void
OpenFileTable::commit(MDSContext*, uint64_t, int)' thread 7f959aed5700
time 2024-04-17T17:47:43.831243+0000
./src/mds/OpenFileTable.cc: 549: FAILED ceph_assert(count > 0)
Next hours we read the tons of articles, studied the documentation, and checked the
common state of Ceph cluster by the various diagnostic commands – but didn’t find anything
wrong. At evening we decided to upgrade it up to v16, and finally to v17.2.7.
Unfortunately, it didn’t solve the problem, MDS continue to crash with the same error. The
only difference that we found is “1 MDSs report damaged metadata” in the output of ceph -s
– see it below.
I supposed that it may be the well-known bug, but couldn’t find the
same one on https://tracker.ceph.com - there are several bugs
associated with file OpenFileTable.cc but not related to
ceph_assert(count > 0)
We tried to check the source code of OpenFileTable.cc also, here is a fragment of it, in
function OpenFileTable::_journal_finish
int omap_idx = anchor.omap_idx;
unsigned& count = omap_num_items.at(omap_idx);
ceph_assert(count > 0);
So, we guess that the object map is empty for some object in Ceph, and
it is unexpected behavior. But again, we found nothing wrong in our
cluster…
Next, we started with
https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ article – tried to
reset the journal (despite that it was Ok all the time) and wipe the sessions using
cephfs-table-tool all reset session command. No result… Now I decided to continue
following this article and run cephfs-data-scan scan_extents command, it is working just
now. But I have a doubt that it will solve the issue because of no problem with our
objects in Ceph.
Is it the new bug? or something else? Any idea is welcome!
The important outputs:
----- ceph -s
cluster:
id: 4cd1c477-c8d0-4855-a1f1-cb71d89427ed
health: HEALTH_ERR
1 MDSs report damaged metadata
insufficient standby MDS daemons available
83 daemons have recently crashed
3 mgr modules have recently crashed
services:
mon: 3 daemons, quorum asrv-dev-stor-2,asrv-dev-stor-3,asrv-dev-stor-1 (age 22h)
mgr: asrv-dev-stor-2(active, since 22h), standbys: asrv-dev-stor-1
mds: 1/1 daemons up
osd: 18 osds: 18 up (since 22h), 18 in (since 29h)
data:
volumes: 1/1 healthy
pools: 5 pools, 289 pgs
objects: 29.72M objects, 5.6 TiB
usage: 21 TiB used, 47 TiB / 68 TiB avail
pgs: 287 active+clean
2 active+clean+scrubbing+deep
io:
client: 2.5 KiB/s rd, 172 KiB/s wr, 261 op/s rd, 195 op/s wr
-----ceph fs dump
e29480
enable_multiple, ever_enabled_multiple: 0,1 default compat:
compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds
uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'cephfs' (1)
fs_name cephfs
epoch 29480
flags 12 joinable allow_snaps allow_multimds_snaps
created 2022-11-25T15:56:08.507407+0000
modified 2024-04-18T16:52:29.970504+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 14728
compat compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses
versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor
table,9=file layout v2,10=snaprealm v2} max_mds 1
in 0
up {0=156636152}
failed
damaged
stopped
data_pools [5]
metadata_pool 6
inline_data disabled
balancer
standby_count_wanted 1
[mds.asrv-dev-stor-1{0:156636152} state up:active seq 6 laggy since
2024-04-18T16:52:29.970479+0000 addr
[v2:172.22.2.91:6800/2487054023,v1:172.22.2.91:6801/2487054023] compat
{c=[1],r=[1],i=[7ff]}]
-----cephfs-journal-tool --rank=cephfs:0 journal inspect Overall
journal integrity: OK
-----ceph pg dump summary
version 41137
stamp 2024-04-18T21:17:59.133536+0000
last_osdmap_epoch 0
last_pg_scan 0
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES
OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
sum 29717605 0 0 0 0 6112544251872
13374192956 28493480 1806575 1806575
OSD_STAT USED AVAIL USED_RAW TOTAL
sum 21 TiB 47 TiB 21 TiB 68 TiB
-----ceph pg dump pools
POOLID OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES
OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
8 31771 0 0 0 0 131337887503
2482 140 401246 401246
7 839707 0 0 0 0 3519034650971
736 61 399328 399328
6 1319576 0 0 0 0 421044421
13374189738 28493279 206749 206749
5 27526539 0 0 0 0 2461702171417
0 0 792165 792165
2 12 0 0 0 0 48497560
0 0 6991 6991
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
21 Apr
21 Apr
6:21 p.m.
Hi Alexey,
This looks a new issue for me. Please create a tracker for it and
provide the detail call trace there.
Thanks
- Xiubo
On 4/19/24 05:42, alexey.gerasimov(a)opencascade.com wrote:
Dear colleagues, hope that anybody can help us.
The initial point: Ceph cluster v15.2 (installed and controlled by the Proxmox) with 3
nodes based on physical servers rented from a cloud provider. CephFS is installed also.
Yesterday we discovered that some of the applications stopped working. During the
investigation we recognized that we have the problem with Ceph, more precisely with СephFS
- MDS daemons suddenly crashed. We tried to restart them and found that they crashed again
immediately after the start. The crash information:
2024-04-17T17:47:42.841+0000 7f959ced9700 1 mds.0.29134 recovery_done -- successful
recovery!
2024-04-17T17:47:42.853+0000 7f959ced9700 1 mds.0.29134 active_start
2024-04-17T17:47:42.881+0000 7f959ced9700 1 mds.0.29134 cluster recovered.
2024-04-17T17:47:43.825+0000 7f959aed5700 -1 ./src/mds/OpenFileTable.cc: In function
'void OpenFileTable::commit(MDSContext*, uint64_t, int)' thread 7f959aed5700 time
2024-04-17T17:47:43.831243+0000
./src/mds/OpenFileTable.cc: 549: FAILED ceph_assert(count > 0)
Next hours we read the tons of articles, studied the documentation, and checked the
common state of Ceph cluster by the various diagnostic commands – but didn’t find anything
wrong. At evening we decided to upgrade it up to v16, and finally to v17.2.7.
Unfortunately, it didn’t solve the problem, MDS continue to crash with the same error. The
only difference that we found is “1 MDSs report damaged metadata” in the output of ceph -s
– see it below.
I supposed that it may be the well-known bug, but couldn’t find the same one on
https://tracker.ceph.com - there are several bugs associated with file OpenFileTable.cc
but not related to ceph_assert(count > 0)
We tried to check the source code of OpenFileTable.cc also, here is a fragment of it, in
function OpenFileTable::_journal_finish
int omap_idx = anchor.omap_idx;
unsigned& count = omap_num_items.at(omap_idx);
ceph_assert(count > 0);
So, we guess that the object map is empty for some object in Ceph, and it is unexpected
behavior. But again, we found nothing wrong in our cluster…
Next, we started with https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/
article – tried to reset the journal (despite that it was Ok all the time) and wipe the
sessions using cephfs-table-tool all reset session command. No result…
Now I decided to continue following this article and run cephfs-data-scan scan_extents
command, it is working just now. But I have a doubt that it will solve the issue because
of no problem with our objects in Ceph.
Is it the new bug? or something else? Any idea is welcome!
The important outputs:
----- ceph -s
cluster:
id: 4cd1c477-c8d0-4855-a1f1-cb71d89427ed
health: HEALTH_ERR
1 MDSs report damaged metadata
insufficient standby MDS daemons available
83 daemons have recently crashed
3 mgr modules have recently crashed
services:
mon: 3 daemons, quorum asrv-dev-stor-2,asrv-dev-stor-3,asrv-dev-stor-1 (age 22h)
mgr: asrv-dev-stor-2(active, since 22h), standbys: asrv-dev-stor-1
mds: 1/1 daemons up
osd: 18 osds: 18 up (since 22h), 18 in (since 29h)
data:
volumes: 1/1 healthy
pools: 5 pools, 289 pgs
objects: 29.72M objects, 5.6 TiB
usage: 21 TiB used, 47 TiB / 68 TiB avail
pgs: 287 active+clean
2 active+clean+scrubbing+deep
io:
client: 2.5 KiB/s rd, 172 KiB/s wr, 261 op/s rd, 195 op/s wr
-----ceph fs dump
e29480
enable_multiple, ever_enabled_multiple: 0,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned
encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file
layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'cephfs' (1)
fs_name cephfs
epoch 29480
flags 12 joinable allow_snaps allow_multimds_snaps
created 2022-11-25T15:56:08.507407+0000
modified 2024-04-18T16:52:29.970504+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 14728
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default
file layouts on dirs,4=dir inode in separate object,5=mds uses versioned
encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file
layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=156636152}
failed
damaged
stopped
data_pools [5]
metadata_pool 6
inline_data disabled
balancer
standby_count_wanted 1
[mds.asrv-dev-stor-1{0:156636152} state up:active seq 6 laggy since
2024-04-18T16:52:29.970479+0000 addr
[v2:172.22.2.91:6800/2487054023,v1:172.22.2.91:6801/2487054023] compat
{c=[1],r=[1],i=[7ff]}]
-----cephfs-journal-tool --rank=cephfs:0 journal inspect
Overall journal integrity: OK
-----ceph pg dump summary
version 41137
stamp 2024-04-18T21:17:59.133536+0000
last_osdmap_epoch 0
last_pg_scan 0
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES
OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
sum 29717605 0 0 0 0 6112544251872
13374192956 28493480 1806575 1806575
OSD_STAT USED AVAIL USED_RAW TOTAL
sum 21 TiB 47 TiB 21 TiB 68 TiB
-----ceph pg dump pools
POOLID OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES
OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
8 31771 0 0 0 0 131337887503
2482 140 401246 401246
7 839707 0 0 0 0 3519034650971
736 61 399328 399328
6 1319576 0 0 0 0 421044421
13374189738 28493279 206749 206749
5 27526539 0 0 0 0 2461702171417
0 0 792165 792165
2 12 0 0 0 0 48497560
0 0 6991 6991
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:18 a.m.
What’s the output of:
ceph tell mds.0 damage ls
Zitat von alexey.gerasimov(a)opencascade.com:
Dear colleagues, hope that anybody can help us.
The initial point: Ceph cluster v15.2 (installed and controlled by
the Proxmox) with 3 nodes based on physical servers rented from a
cloud provider. CephFS is installed also.
Yesterday we discovered that some of the applications stopped
working. During the investigation we recognized that we have the
problem with Ceph, more precisely with СephFS - MDS daemons suddenly
crashed. We tried to restart them and found that they crashed again
immediately after the start. The crash information:
2024-04-17T17:47:42.841+0000 7f959ced9700 1 mds.0.29134
recovery_done -- successful recovery!
2024-04-17T17:47:42.853+0000 7f959ced9700 1 mds.0.29134 active_start
2024-04-17T17:47:42.881+0000 7f959ced9700 1 mds.0.29134 cluster recovered.
2024-04-17T17:47:43.825+0000 7f959aed5700 -1
./src/mds/OpenFileTable.cc: In function 'void
OpenFileTable::commit(MDSContext*, uint64_t, int)' thread
7f959aed5700 time 2024-04-17T17:47:43.831243+0000
./src/mds/OpenFileTable.cc: 549: FAILED ceph_assert(count > 0)
Next hours we read the tons of articles, studied the documentation,
and checked the common state of Ceph cluster by the various
diagnostic commands – but didn’t find anything wrong. At evening we
decided to upgrade it up to v16, and finally to v17.2.7.
Unfortunately, it didn’t solve the problem, MDS continue to crash
with the same error. The only difference that we found is “1 MDSs
report damaged metadata” in the output of ceph -s – see it below.
I supposed that it may be the well-known bug, but couldn’t find the
same one on https://tracker.ceph.com - there are several bugs
associated with file OpenFileTable.cc but not related to
ceph_assert(count > 0)
We tried to check the source code of OpenFileTable.cc also, here is
a fragment of it, in function OpenFileTable::_journal_finish
int omap_idx = anchor.omap_idx;
unsigned& count = omap_num_items.at(omap_idx);
ceph_assert(count > 0);
So, we guess that the object map is empty for some object in Ceph,
and it is unexpected behavior. But again, we found nothing wrong in
our cluster…
Next, we started with
https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/
article – tried to reset the journal (despite that it was Ok all the
time) and wipe the sessions using cephfs-table-tool all reset
session command. No result…
Now I decided to continue following this article and run
cephfs-data-scan scan_extents command, it is working just now. But I
have a doubt that it will solve the issue because of no problem with
our objects in Ceph.
Is it the new bug? or something else? Any idea is welcome!
The important outputs:
----- ceph -s
cluster:
id: 4cd1c477-c8d0-4855-a1f1-cb71d89427ed
health: HEALTH_ERR
1 MDSs report damaged metadata
insufficient standby MDS daemons available
83 daemons have recently crashed
3 mgr modules have recently crashed
services:
mon: 3 daemons, quorum
asrv-dev-stor-2,asrv-dev-stor-3,asrv-dev-stor-1 (age 22h)
mgr: asrv-dev-stor-2(active, since 22h), standbys: asrv-dev-stor-1
mds: 1/1 daemons up
osd: 18 osds: 18 up (since 22h), 18 in (since 29h)
data:
volumes: 1/1 healthy
pools: 5 pools, 289 pgs
objects: 29.72M objects, 5.6 TiB
usage: 21 TiB used, 47 TiB / 68 TiB avail
pgs: 287 active+clean
2 active+clean+scrubbing+deep
io:
client: 2.5 KiB/s rd, 172 KiB/s wr, 261 op/s rd, 195 op/s wr
-----ceph fs dump
e29480
enable_multiple, ever_enabled_multiple: 0,1
default compat: compat={},rocompat={},incompat={1=base
v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir
inode in separate object,5=mds uses versioned encoding,6=dirfrag is
stored in omap,7=mds uses inline data,8=no anchor table,9=file
layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'cephfs' (1)
fs_name cephfs
epoch 29480
flags 12 joinable allow_snaps allow_multimds_snaps
created 2022-11-25T15:56:08.507407+0000
modified 2024-04-18T16:52:29.970504+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 14728
compat compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in
omap,7=mds uses inline data,8=no anchor table,9=file layout
v2,10=snaprealm v2}
max_mds 1
in 0
up {0=156636152}
failed
damaged
stopped
data_pools [5]
metadata_pool 6
inline_data disabled
balancer
standby_count_wanted 1
[mds.asrv-dev-stor-1{0:156636152} state up:active seq 6 laggy since
2024-04-18T16:52:29.970479+0000 addr
[v2:172.22.2.91:6800/2487054023,v1:172.22.2.91:6801/2487054023]
compat {c=[1],r=[1],i=[7ff]}]
-----cephfs-journal-tool --rank=cephfs:0 journal inspect
Overall journal integrity: OK
-----ceph pg dump summary
version 41137
stamp 2024-04-18T21:17:59.133536+0000
last_osdmap_epoch 0
last_pg_scan 0
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND
BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
sum 29717605 0 0 0 0
6112544251872 13374192956 28493480 1806575 1806575
OSD_STAT USED AVAIL USED_RAW TOTAL
sum 21 TiB 47 TiB 21 TiB 68 TiB
-----ceph pg dump pools
POOLID OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND
BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG
8 31771 0 0 0 0
131337887503 2482 140 401246 401246
7 839707 0 0 0 0
3519034650971 736 61 399328 399328
6 1319576 0 0 0 0
421044421 13374189738 28493279 206749 206749
5 27526539 0 0 0 0
2461702171417 0 0 792165 792165
2 12 0 0 0 0
48497560 0 0 6991 6991
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
13
days inactive
21
days old
7 comments
5 participants
participants (5)
-
Alexey GERASIMOV -
alexey.gerasimov＠opencascade.com
-
Eugen Block -
Frédéric Nass -
Xiubo Li