We running a small Ceph cluster with two nodes. Our failureDomain is set to host to have the data replicated between the two hosts. The other night one host crashed hard and three OSDs won't recovert with either debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 -1 osd.23 0 OSD::init() : unable to read osd superblock debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 1 bluestore(/var/lib/ceph/osd/ceph-23) umount debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) allocation stats probe 0: cnt: 0 frags: 0 size: 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -1: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -2: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -4: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -8: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -16: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) ------------ debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 4 rocksdb: [db/db_impl.cc:563] Shutdown complete debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 1 bluefs umount debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 1 bdev(0x557150e20700 /var/lib/ceph/osd/ceph-23/block) close debug 2021-01-13T08:13:18.167+0000 7f9bfbd6ef40 1 freelist shutdown debug 2021-01-13T08:13:18.167+0000 7f9bfbd6ef40 1 bdev(0x557150e20000 /var/lib/ceph/osd/ceph-23/block) close debug 2021-01-13T08:13:18.411+0000 7f9bfbd6ef40 -1 ** ERROR: osd init failed: (22) Invalid argument or debug -2> 2021-01-13T08:13:29.991+0000 7f402c5f9700 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: expected 2795871023, got 2381104739 in db/000060.sst offset 748408 size 3819 code = 2 Rocksdb transaction: How can I delete the OSDs to get them back fully operational? Any help appreciated! /Fabian
Hello, I suspect there was unwritten data in RAM which didn't make it to the disk. This shoudn't happen, that's why the journal is in place. If you have size=2 in you pool, there is one copy on the other host. Do delete the OSD you could probably do ceph osd crush remove osd.x ceph osd rm osd.x ceph auth del osd.x maybe "wipefs -a /dev/sdxxx" or dd if=/dev/zero of=dev/sdxx count=1 bs=1m ... Then you should be able deploy the disk again with the tool that you used originally. The disk should be "fresh". rgds, derjohn. On 13.01.21 15:45, Pfannes, Fabian wrote:
failed: (22) Invalid argument
-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet
Hi, thanks to derJohn I was send to the right path. Our setup is a bit more difficult as we're using more than one OSD per drive. The setup is done using rook. Therefore I used the docs at https://github.com/rook/rook/blob/master/Documentation/ceph-osd-mgmt.md#remo... But we ended up to remove all OSDs of the drive: - using "lsblk" to get the UUIDs of the OSDs placed on that drive - getting the UUIDs of the OSDs via "ceph osd dump" - matching them and remove the OSDs deployments - waiting for ceph to shift all data zapping the whole drive (https://github.com/rook/rook/blob/master/Documentation/ceph-teardown.md#dele...) and reinstalling it into the rook deployment. Hope it helps other people looking for issues like this. /Fabian Am Mittwoch, dem 13.01.2021 um 22:35 +0100 schrieb Andreas John:
Hello,
I suspect there was unwritten data in RAM which didn't make it to the disk. This shoudn't happen, that's why the journal is in place.
If you have size=2 in you pool, there is one copy on the other host. Do delete the OSD you could probably do
ceph osd crush remove osd.x
ceph osd rm osd.x
ceph auth del osd.x
maybe "wipefs -a /dev/sdxxx" or dd if=/dev/zero of=dev/sdxx count=1 bs=1m ...
Then you should be able deploy the disk again with the tool that you used originally. The disk should be "fresh".
rgds,
derjohn.
On 13.01.21 15:45, Pfannes, Fabian wrote:
failed: (22) Invalid argument
-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net
Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dipl.-Ing. Fabian Pfannes Maon GmbH Bismarckstraße 10-12, 10625 Berlin fabian.pfannes@maon.eu
participants (2)
-
Andreas John
-
Pfannes, Fabian