Hi All, from the log output (the line with "Malformed input" string) it rather looks like a device label (the very first 4K data block at main OSD device containing some basic OSD meta, e.g. OSD UUID) corruption. There are some chances that wrong device has been attached too. Alison, to investigate further could you please share the 4K superblock content (can be retrieved using dd tool: "dd if=/var/lib/ceph/osd/ceph-665/block count=1 bs=4096 of=./superb.out") and /var/lib/ceph/osd/ceph-665 file listing. Thanks, Igor On 8/25/2023 8:58 PM, Eugen Block wrote:
Hi, one thing coming to mind is maybe the device names have changed from /dev/sdX to /dev/sdY? Something like that has been reported a couple of times in the last months.
Zitat von Alison Peisker <apeisker@fnal.gov>:
Hi all,
We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm.
The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory
The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this?
Thanks, Alison _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx