Hi,
My turn.
We suddenly have a big outage which is similar/identical to
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036519.html
Some of the osds are runnable, but most crash when they start -- crc
error in osdmap::decode.
I'm able to extract an osd map from a good osd and it decodes well
with osdmaptool:
# ceph-objectstore-tool --op get-osdmap --data-path
/var/lib/ceph/osd/ceph-680/ --file osd.680.map
But when I try on one of the bad osds I get:
# ceph-objectstore-tool --op get-osdmap --data-path
/var/lib/ceph/osd/ceph-666/ --file osd.666.map
terminate called after throwing an instance of 'ceph::buffer::malformed_input'
what(): buffer::malformed_input: bad crc, actual 822724616 !=
expected 2334082500
*** Caught signal (Aborted) **
in thread 7f600aa42d00 thread_name:ceph-objectstor
ceph version 13.2.7 (71bd687b6e8b9424dd5e5974ed542595d8977416) mimic (stable)
1: (()+0xf5f0) [0x7f5ffefc45f0]
2: (gsignal()+0x37) [0x7f5ffdbae337]
3: (abort()+0x148) [0x7f5ffdbafa28]
4: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f5ffe4be7d5]
5: (()+0x5e746) [0x7f5ffe4bc746]
6: (()+0x5e773) [0x7f5ffe4bc773]
7: (()+0x5e993) [0x7f5ffe4bc993]
8: (OSDMap::decode(ceph::buffer::list::iterator&)+0x160e) [0x7f6000f4168e]
9: (OSDMap::decode(ceph::buffer::list&)+0x31) [0x7f6000f42e31]
10: (get_osdmap(ObjectStore*, unsigned int, OSDMap&,
ceph::buffer::list&)+0x1d0) [0x55d30a489190]
11: (main()+0x5340) [0x55d30a3aae70]
12: (__libc_start_main()+0xf5) [0x7f5ffdb9a505]
13: (()+0x3a0f40) [0x55d30a483f40]
Aborted (core dumped)
I think I want to inject the osdmap, but can't:
# ceph-objectstore-tool --op set-osdmap --data-path
/var/lib/ceph/osd/ceph-666/ --file osd.680.map
osdmap (#-1:b65b78ab:::osdmap.2983572:0#) does not exist.
How do I do this?
Thanks for any help!
dan