18 Jan
2023
18 Jan
'23
9:12 p.m.
Hey all! I’ve run into an MDS crash on a cluster recently upgraded from Ceph 16.2.7 to
16.2.10. I’m hitting an assert nearly identical to this one gathered by the telemetry
module:
https://tracker.ceph.com/issues/54747
I have a new build compiling to test whether https://github.com/ceph/ceph/pull/43184/
makes a difference or not, when setting mds_inject_skip_replaying_inotable.
Relevant logs are below, but I’m wondering if anyone has hit anything like this? Thanks
in advance!
=== BEGIN LOG SNIPPET ===
-2> 2023-01-18T20:16:29.789+0000 7f6190243700 -1 log_channel(cluster) log [ERR] :
journal replay alloc 0x10000000010 not in free
[0x10000000011~0x3dc,0x100000003fb~0x1e8,0x100000005e5~0x2,0x100000009d4~0x2,0x1000005cc6d~0x4,0x10001c6b44e~0x4,0x10001cb91f4~0x1f4,0x10001cb93f4~0x3dd,0x10007582c15~0x279,0x10007582e90~0x1f4,0x10007583094~0xfff8a7cf6c]
-1> 2023-01-18T20:16:29.789+0000 7f6190243700 -1
/builds/66321/e7c73776/ceph/-build//WORKDIR/ceph-16.2.10/src/mds/journal.cc: In function
'void EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)' thread 7f6190243700
time 2023-01-18T20:16:29.794189+0000
/WORKDIR/ceph-16.2.10/src/mds/journal.cc: 1577: FAILED ceph_assert(inotablev ==
mds->inotable->get_version())
ceph version 16.2.10 (e7c73776b3136f6d18a35febeb38f5fdd41be364) pacific (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14c)
[0x7f619d548645]
2: /usr/lib/ceph/libceph-common.so.2(+0x27182f) [0x7f619d54882f]
3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x5815) [0x560bfd1c6935]
4: (EUpdate::replay(MDSRank*)+0x3c) [0x560bfd1c7ecc]
5: (MDLog::_replay_thread()+0xca9) [0x560bfd153de9]
6: (MDLog::ReplayThread::entry()+0xd) [0x560bfce78fdd]
7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f619cf29fa3]
8: clone()
0> 2023-01-18T20:16:29.793+0000 7f6190243700 -1 *** Caught signal (Aborted) **
in thread 7f6190243700 thread_name:md_log_replay
ceph version 16.2.10 (e7c73776b3136f6d18a35febeb38f5fdd41be364) pacific (stable)
1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f619cf34730]
2: gsignal()
3: abort()
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19d)
[0x7f619d548696]
5: /usr/lib/ceph/libceph-common.so.2(+0x27182f) [0x7f619d54882f]
6: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x5815) [0x560bfd1c6935]
7: (EUpdate::replay(MDSRank*)+0x3c) [0x560bfd1c7ecc]
8: (MDLog::_replay_thread()+0xca9) [0x560bfd153de9]
9: (MDLog::ReplayThread::entry()+0xd) [0x560bfce78fdd]
10: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f619cf29fa3]
11: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
=== END LOG SNIPPET ===