Hi Marco, Have you checked the output of: dd if=/dev/ceph-xxxxxxx/osd-block-xxxxxxxxx of=/tmp/foo bs=4K count=2 hexdump -C /tmp/foo and: /usr/bin/ceph-bluestore-tool show-label --log-level=30 --dev /dev/nvmexxx -l /var/log/ceph/ceph-volume.log to see if it's aligned with OSD's metadata. You may also want to check this discussion [1] and this tracker [2] for useful commands. Regards, Frédéric [1] https://marc.info/?l=ceph-users&m=171395775626007&w=2 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1755956 ----- Le 2 Sep 24, à 11:26, m@marcofaggian.com a écrit :
FYI: Also posted in L1Techs forum: https://forum.level1techs.com/t/recover-bluestore-osd-in-ceph-cluster/215715.
## The epic intro Through self-inflicted pain, I’m writing here to ask for volunteers in the journey of recovering the lost partitions housing CephFS metadata pool.
## The setup 1 proxmox host (I know) 1 replication rule only for NVMes (2x OSD) 1 replication rule only for HDDs (8x OSD) Each with failure domain to osd. Each OSD configured to use bluestore backend in an LVM. No backup (I know, I know).
## The cause (me) Long story short: I needed the PCIe lanes and decided to remove the two NVMEs that were hosting the metadata pool for CephFS and .mgr pool. I proceeded to remove the two OSDs (out and destroy). This is where I’ve done goof: I didn’t change the replication rule to HDDs’ one, so the cluster never moved the PGs stored in the NVMes, to the HDDs.
## What I’ve done untill now 1. Re-seated the NVMes to their original place. 2. Found out that the LVM didn’t have the OSD’s labels applied 3. Forced the backed-up LVM config to the two NVMes (thanks to the holy entity that thought that archiving LVM config was a good thing, it payed back) 4. Trying ceph-volume lvm activate 8 <id> to find out that it’s unable to decode label at offset 102 in the LVM for that ODS.
## Wishes 1. Does anyone know a way to recover what I feel is a lost partition, given that the “file system” is ceph’s bluestore? 2. Is there a way to know, if it is, how the partition has been nuked? And possibly find a way to reverse that process.
## Closing statement Eternal reminder: If you don’t want to lose it, back it up. Thanks for your time, to the kind souls that are willing to die on this hill with me, or come up victorious! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io