The journey to CephFS metadata pool’s recovery
FYI: Also posted in L1Techs forum: https://forum.level1techs.com/t/recover-bluestore-osd-in-ceph-cluster/215715.
## The epic intro Through self-inflicted pain, I’m writing here to ask for volunteers in the journey of recovering the lost partitions housing CephFS metadata pool. ## The setup 1 proxmox host (I know) 1 replication rule only for NVMes (2x OSD) 1 replication rule only for HDDs (8x OSD) Each with failure domain to osd. Each OSD configured to use bluestore backend in an LVM. No backup (I know, I know). ## The cause (me) Long story short: I needed the PCIe lanes and decided to remove the two NVMEs that were hosting the metadata pool for CephFS and .mgr pool. I proceeded to remove the two OSDs (out and destroy). This is where I’ve done goof: I didn’t change the replication rule to HDDs’ one, so the cluster never moved the PGs stored in the NVMes, to the HDDs. ## What I’ve done untill now 1. Re-seated the NVMes to their original place. 2. Found out that the LVM didn’t have the OSD’s labels applied 3. Forced the backed-up LVM config to the two NVMes (thanks to the holy entity that thought that archiving LVM config was a good thing, it payed back) 4. Trying ceph-volume lvm activate 8 <id> to find out that it’s unable to decode label at offset 102 in the LVM for that ODS. ## Wishes 1. Does anyone know a way to recover what I feel is a lost partition, given that the “file system” is ceph’s bluestore? 2. Is there a way to know, if it is, how the partition has been nuked? And possibly find a way to reverse that process. ## Closing statement Eternal reminder: If you don’t want to lose it, back it up. Thanks for your time, to the kind souls that are willing to die on this hill with me, or come up victorious!
Hi Marco, Have you checked the output of: dd if=/dev/ceph-xxxxxxx/osd-block-xxxxxxxxx of=/tmp/foo bs=4K count=2 hexdump -C /tmp/foo and: /usr/bin/ceph-bluestore-tool show-label --log-level=30 --dev /dev/nvmexxx -l /var/log/ceph/ceph-volume.log to see if it's aligned with OSD's metadata. You may also want to check this discussion [1] and this tracker [2] for useful commands. Regards, Frédéric [1] https://marc.info/?l=ceph-users&m=171395775626007&w=2 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1755956 ----- Le 2 Sep 24, à 11:26, m@marcofaggian.com a écrit :
FYI: Also posted in L1Techs forum: https://forum.level1techs.com/t/recover-bluestore-osd-in-ceph-cluster/215715.
## The epic intro Through self-inflicted pain, I’m writing here to ask for volunteers in the journey of recovering the lost partitions housing CephFS metadata pool.
## The setup 1 proxmox host (I know) 1 replication rule only for NVMes (2x OSD) 1 replication rule only for HDDs (8x OSD) Each with failure domain to osd. Each OSD configured to use bluestore backend in an LVM. No backup (I know, I know).
## The cause (me) Long story short: I needed the PCIe lanes and decided to remove the two NVMEs that were hosting the metadata pool for CephFS and .mgr pool. I proceeded to remove the two OSDs (out and destroy). This is where I’ve done goof: I didn’t change the replication rule to HDDs’ one, so the cluster never moved the PGs stored in the NVMes, to the HDDs.
## What I’ve done untill now 1. Re-seated the NVMes to their original place. 2. Found out that the LVM didn’t have the OSD’s labels applied 3. Forced the backed-up LVM config to the two NVMes (thanks to the holy entity that thought that archiving LVM config was a good thing, it payed back) 4. Trying ceph-volume lvm activate 8 <id> to find out that it’s unable to decode label at offset 102 in the LVM for that ODS.
## Wishes 1. Does anyone know a way to recover what I feel is a lost partition, given that the “file system” is ceph’s bluestore? 2. Is there a way to know, if it is, how the partition has been nuked? And possibly find a way to reverse that process.
## Closing statement Eternal reminder: If you don’t want to lose it, back it up. Thanks for your time, to the kind souls that are willing to die on this hill with me, or come up victorious! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frédéric, Thanks a lot for the pointers! So, using testdisk I’ve created images of both the LVs. I’ve looked at the hexdump and it’s filled with 0x00 until 00a00000. Then for curiosity I’ve compared them and they’re identical until byte 12726273. Also unfortunately the issue is that ceph-bluestore-tool show-label, like dumpe2fs -h /dev/ceph-3.. are both erroring out in the same way: unable to read label for /dev/ceph-454751de-44ab-4aa6-b3ae-50abc22250b3/osd-block-b7745d63-0bf8-4ba4-9274-e034f1c15d7b: 2024-09-03T11:27:22.938+0200 7f2d89f19a00 -1 bluestore(/dev/ceph-454751de-44ab-4aa6-b3ae-50abc22250b3/osd-block-b7745d63-0bf8-4ba4-9274-e034f1c15d7b) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] (2) No such file or directory Unfortunately the thread on the tracker doesn’t seem to point to a solution, even though the issue seems to be identical. The hexdump explains why it’s not finding the label, might it be that the LV is not correctly mapped? Basically here the question is: is there a way to recover the data of an OSD in an LV, if it was ceph osd purge before the cluster had a chance to replicate it (after ceph osd out)? Thanks for your time! fm
On 3 Sep 2024, at 10:35, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Marco,
Have you checked the output of:
dd if=/dev/ceph-xxxxxxx/osd-block-xxxxxxxxx of=/tmp/foo bs=4K count=2 hexdump -C /tmp/foo
and:
/usr/bin/ceph-bluestore-tool show-label --log-level=30 --dev /dev/nvmexxx -l /var/log/ceph/ceph-volume.log
to see if it's aligned with OSD's metadata.
You may also want to check this discussion [1] and this tracker [2] for useful commands.
Regards, Frédéric
[1] https://u44306567.ct.sendgrid.net/ls/click?upn=u001.OhTBFyWWodn3bJkXMXGAv-2B... [2] https://u44306567.ct.sendgrid.net/ls/click?upn=u001.OhTBFyWWodn3bJkXMXGAv52B...
----- Le 2 Sep 24, à 11:26, m@marcofaggian.com a écrit :
FYI: Also posted in L1Techs forum: https://u44306567.ct.sendgrid.net/ls/click?upn=u001.OhTBFyWWodn3bJkXMXGAv0db...
## The epic intro Through self-inflicted pain, I’m writing here to ask for volunteers in the journey of recovering the lost partitions housing CephFS metadata pool.
## The setup 1 proxmox host (I know) 1 replication rule only for NVMes (2x OSD) 1 replication rule only for HDDs (8x OSD) Each with failure domain to osd. Each OSD configured to use bluestore backend in an LVM. No backup (I know, I know).
## The cause (me) Long story short: I needed the PCIe lanes and decided to remove the two NVMEs that were hosting the metadata pool for CephFS and .mgr pool. I proceeded to remove the two OSDs (out and destroy). This is where I’ve done goof: I didn’t change the replication rule to HDDs’ one, so the cluster never moved the PGs stored in the NVMes, to the HDDs.
## What I’ve done untill now 1. Re-seated the NVMes to their original place. 2. Found out that the LVM didn’t have the OSD’s labels applied 3. Forced the backed-up LVM config to the two NVMes (thanks to the holy entity that thought that archiving LVM config was a good thing, it payed back) 4. Trying ceph-volume lvm activate 8 <id> to find out that it’s unable to decode label at offset 102 in the LVM for that ODS.
## Wishes 1. Does anyone know a way to recover what I feel is a lost partition, given that the “file system” is ceph’s bluestore? 2. Is there a way to know, if it is, how the partition has been nuked? And possibly find a way to reverse that process.
## Closing statement Eternal reminder: If you don’t want to lose it, back it up. Thanks for your time, to the kind souls that are willing to die on this hill with me, or come up victorious! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Have you tried to hexdump the actual NVMEs devices instead of the testdisk images? `hexdump -C -n 4096 /dev/nvmexxx` should show LVM LABELONE with PV UUID and `hexdump -C -n 4096 /dev/ceph-454751de-44ab-4aa6-b3ae-50abc22250b3/osd-block-b7745d63-0bf8-4ba4-9274-e034f1c15d7b` should show bluestore metadata AKA 'label' that you can get with `ceph-bluestore-tool show-label` If nothing shows up than you may need to rewrite the label as described here [1] by Igor. Thing is, you'd have to make sure you're dealing with the right disk, meaning both NVMEs were not swapped. To help you with that part, you could run `strings /dev/nvmexxx | less` and try to link the VG informations with OSD config files if you're able to. Hope that helps. Regards, Frédéric. [1] https://www.spinics.net/lists/ceph-users/msg81813.html ----- Le 3 Sep 24, à 11:56, Marco Faggian <m@marcofaggian.com> a écrit :
Hi Frédéric, Thanks a lot for the pointers!
So, using testdisk I’ve created images of both the LVs. I’ve looked at the hexdump and it’s filled with 0x00 until 00a00000. Then for curiosity I’ve compared them and they’re identical until byte 12726273.
Also unfortunately the issue is that ceph-bluestore-tool show-label, like dumpe2fs -h /dev/ceph-3.. are both erroring out in the same way:
unable to read label for /dev/ceph-454751de-44ab-4aa6-b3ae-50abc22250b3/osd-block-b7745d63-0bf8-4ba4-9274-e034f1c15d7b: 2024-09-03T11:27:22.938+0200 7f2d89f19a00 -1 bluestore(/dev/ceph-454751de-44ab-4aa6-b3ae-50abc22250b3/osd-block-b7745d63-0bf8-4ba4-9274-e034f1c15d7b) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
(2) No such file or directory
Unfortunately the thread on the tracker doesn’t seem to point to a solution, even though the issue seems to be identical.
The hexdump explains why it’s not finding the label, might it be that the LV is not correctly mapped?
Basically here the question is: is there a way to recover the data of an OSD in an LV, if it was ceph osd purge before the cluster had a chance to replicate it (after ceph osd out )?
Thanks for your time! fm
On 3 Sep 2024, at 10:35, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Marco,
Have you checked the output of:
dd if=/dev/ceph-xxxxxxx/osd-block-xxxxxxxxx of=/tmp/foo bs=4K count=2 hexdump -C /tmp/foo
and:
/usr/bin/ceph-bluestore-tool show-label --log-level=30 --dev /dev/nvmexxx -l /var/log/ceph/ceph-volume.log
to see if it's aligned with OSD's metadata.
You may also want to check this discussion [1] and this tracker [2] for useful commands.
Regards, Frédéric
[1] https://marc.info/?l=ceph-users&m=171395775626007&w=2 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1755956
----- Le 2 Sep 24, à 11:26, m@marcofaggian.com a écrit :
FYI: Also posted in L1Techs forum: https://forum.level1techs.com/t/recover-bluestore-osd-in-ceph-cluster/215715.
## The epic intro Through self-inflicted pain, I’m writing here to ask for volunteers in the journey of recovering the lost partitions housing CephFS metadata pool.
## The setup 1 proxmox host (I know) 1 replication rule only for NVMes (2x OSD) 1 replication rule only for HDDs (8x OSD) Each with failure domain to osd. Each OSD configured to use bluestore backend in an LVM. No backup (I know, I know).
## The cause (me) Long story short: I needed the PCIe lanes and decided to remove the two NVMEs that were hosting the metadata pool for CephFS and .mgr pool. I proceeded to remove the two OSDs (out and destroy). This is where I’ve done goof: I didn’t change the replication rule to HDDs’ one, so the cluster never moved the PGs stored in the NVMes, to the HDDs.
## What I’ve done untill now 1. Re-seated the NVMes to their original place. 2. Found out that the LVM didn’t have the OSD’s labels applied 3. Forced the backed-up LVM config to the two NVMes (thanks to the holy entity that thought that archiving LVM config was a good thing, it payed back) 4. Trying ceph-volume lvm activate 8 <id> to find out that it’s unable to decode label at offset 102 in the LVM for that ODS.
## Wishes 1. Does anyone know a way to recover what I feel is a lost partition, given that the “file system” is ceph’s bluestore? 2. Is there a way to know, if it is, how the partition has been nuked? And possibly find a way to reverse that process.
## Closing statement Eternal reminder: If you don’t want to lose it, back it up. Thanks for your time, to the kind souls that are willing to die on this hill with me, or come up victorious! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Frédéric Nass
-
m＠marcofaggian.com
-
Marco Faggian