8 Jun
2020
8 Jun
'20
1:48 p.m.
This is again about our bad cluster, with far too many objects. Now
another OSD crashes immediately at startup:
/build/ceph-14.2.8/src/os/bluestore/KernelDevice.cc: 944: FAILED
ceph_assert(is_valid_io(off, len))
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x152) [0x5601938e0e92]
2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char
const*, char const*, ...)+0) [0x5601938e106d]
3: (KernelDevice::read(unsigned long, unsigned long,
ceph::buffer::v14_2_0::list*, IOContext*, bool)+0x8e0) [0x560193f2ae90]
4: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*,
unsigned long, unsigned long, ceph::buffer::v14_2_0::list*,
char*)+0x4f6) [0x560193ee0506]
5: (BlueFS::_replay(bool, bool)+0x489) [0x560193ee14e9]
6: (BlueFS::mount()+0x219) [0x560193ef4319]
7: (BlueStore::_open_bluefs(bool)+0x41) [0x560193de2281]
8: (BlueStore::_open_db(bool, bool, bool)+0x88c) [0x560193de347c]
9: (BlueStore::_open_db_and_around(bool)+0x44) [0x560193dfa134]
10: (BlueStore::_mount(bool, bool)+0x584) [0x560193e4a804]
11: (OSD::init()+0x3b7) [0x56019398f957]
12: (main()+0x3cdb) [0x5601938e85cb]
13: (__libc_start_main()+0xe7) [0x7f54cdf1bb97]
14: (_start()+0x2a) [0x56019391b08a]
2020-06-08 13:44:52.063 7f54d169ec00 -1 *** Caught signal (Aborted) **
in thread 7f54d169ec00 thread_name:ceph-osd
ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb)
nautilus (stable)
1: (()+0x12890) [0x7f54cf286890]
2: (gsignal()+0xc7) [0x7f54cdf38e97]
3: (abort()+0x141) [0x7f54cdf3a801]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1a3) [0x5601938e0ee3]
5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char
const*, char const*, ...)+0) [0x5601938e106d]
6: (KernelDevice::read(unsigned long, unsigned long,
ceph::buffer::v14_2_0::list*, IOContext*, bool)+0x8e0) [0x560193f2ae90]
7: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*,
unsigned long, unsigned long, ceph::buffer::v14_2_0::list*,
char*)+0x4f6) [0x560193ee0506]
8: (BlueFS::_replay(bool, bool)+0x489) [0x560193ee14e9]
9: (BlueFS::mount()+0x219) [0x560193ef4319]
10: (BlueStore::_open_bluefs(bool)+0x41) [0x560193de2281]
11: (BlueStore::_open_db(bool, bool, bool)+0x88c) [0x560193de347c]
12: (BlueStore::_open_db_and_around(bool)+0x44) [0x560193dfa134]
13: (BlueStore::_mount(bool, bool)+0x584) [0x560193e4a804]
14: (OSD::init()+0x3b7) [0x56019398f957]
15: (main()+0x3cdb) [0x5601938e85cb]
16: (__libc_start_main()+0xe7) [0x7f54cdf1bb97]
17: (_start()+0x2a) [0x56019391b08a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
There was a preceding assert (earlier written about):
/build/ceph-14.2.8/src/os/bluestore/BlueFS.cc: 2261: FAILED
ceph_assert(h->file->fnode.ino != 1)
Any ideas that I could try to save this OSDs?
Cheers
Harry