[ceph-users] bluestore label returned: (2) No such file or directory

Hi List I have a osd (83) that fails to start. It is made up of one 4TB drive and an 80GB DB on nvme. There was a cluster-full situation that is now solved, however I am quite sure the issue with this particular osd is unrelated. When I try to start the osd it failes to read the label of the block.db device failing with the followint lines:     1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-83/block.db size 80 GiB     -1 bluestore(/var/lib/ceph/osd/ceph-83) _minimal_open_bluefs check block device(/var/lib/ceph/osd/ceph-83/block.db) label returned: (2) No such file or directory     1 bdev(0x563d8822a700 /var/lib/ceph/osd/ceph-83/block.db) close     1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) close     -1 osd.83 0 OSD:init: unable to mount object store     -1  ** ERROR: osd init failed: (2) No such file or directory (Full log: https://gist.github.com/NightDog/7b50349da1410bb05bd7f4d54a02f055) The last thing that happened to the OSD before it started to fail booting was it being terminated during what I believe was a reboot: received  signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Last lines++ of the last successful run: https://gist.github.com/NightDog/fd9b4b7b3e0c0c2ba29ce5d325bb97c6 When I try to run ceph-bluestore-tool --log-level 30 show-label on the block.db it returns: "unable to read label for /dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be: (2) No such file or directory" The block returns the label fine (see master gist): https://gist.github.com/NightDog/4518bf11b364170911e5743b5ed0f614 The strange thing is however that lvs -o lv_tags returns just fine for the block.db: root@ceph-node201:~# lvs -o lv_tags /dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be   LV Tags ceph.block_device=/dev/ceph-ff60b68a-26fe-4294-8bec-4a9c329e858d/osd-block-73ab12e6-7758-4ebe-9319-5935309fcacd,ceph.block_uuid=nbRXYl-fRrQ-qyYP-D93c-IGct-yKg4-rujDOX,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=f4495398-a8c4-4ad9-8219-80c48625abdf,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be,ceph.db_uuid=K09p3L-QV06-LOLO-uVeT-2ulz-GD3O-CEyRcs,ceph.encrypted=0,ceph.osd_fsid=73ab12e6-7758-4ebe-9319-5935309fcacd,ceph.osd_id=83,ceph.osdspec_affinity=osd-spec-2xx,ceph.type=db,ceph.vdo=0 So it seems to me that for some reason, ceph-bluestore-tool fails to read the label of the block.db device, even tho it is there, and then fails the startup of the OSD. Trying to write keys with ceph-bluestore-tool set-label-key fails with the same error message. I see no reason why there should be any damage to either the .db or block device, and since the labels are there in LVM, guess ceph-bluestore-tool errors out on something else? Would it be possible to get some help with regards to getting this .db and OSD back up again? Thanks! PS: Running version 15.2.8, also tried with 16.2.3-> show-label, with same result. -- Regards Karl M. Kittilsen