Hi, I think that the approach with exporting and importing PGs would be a-priori more successful than the one based on pvmove or ddrescue. The reason is that you don't need to export/import all data that the failed disk holds, but only the PGs that Ceph cannot recover otherwise. The logic here is that these are, likely, not the same PGs due to which tools are crashing. Note that after the export/import operation Ceph might still think "I need a copy from that failed disk and not the one that you gave me", in this case just export a copy of the same PG from the other failed OSD and import elsewhere, up to the total number of copies. If even that desn't help, "ceph osd lost XX" would be the last (very dangerous) words to convince Cepth that osd.XX will not be seen in the future. On Sat, Feb 3, 2024 at 5:35 AM Eugen Block <eblock@nde.ag> wrote:

Hi,

if the OSDs are deployed as LVs (by ceph-volume) you could try to do a pvmove to a healthy disk. There was a thread here a couple of weeks ago explaining the steps. I don’t have it at hand right now, but it should be easy to find. Of course, there’s no guarantee that this will be successful. I also can’t tell if Igor‘s approach is more promising.

Zitat von Igor Fedotov <igor.fedotov@croit.io>:

...

Hi Carl,

you might want to use ceph-objectstore-tool to export PGs from faulty OSDs and import them back to healthy ones.

The process could be quite tricky though.

There is also pending PR (https://github.com/ceph/ceph/pull/54991) to make the tool more tolerant to disk errors.

The patch worth trying in some cases, not a silver bullet though.

And generally whether the recovery doable greatly depends on the actual error(s).

Thanks,

Igor

On 02/02/2024 19:03, Carl J Taylor wrote:

...

Hi, I have a small cluster with some faulty disks within it and I want to clone the data from the faulty disks onto new ones.

The cluster is currently down and I am unable to do things like ceph-bluestore-fsck but ceph-bluestore-tool bluefs-export does appear to be working.

Any help would be appreciated

Many thanks Carl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Alexander E. Patrakov

Hi,

I think that the approach with exporting and importing PGs would be a-priori more successful than the one based on pvmove or ddrescue. The reason is that you don't need to export/import all data that the failed disk holds, but only the PGs that Ceph cannot recover otherwise. The logic here is that these are, likely, not the same PGs due to which tools are crashing.

that makes sense. Thanks! Zitat von "Alexander E. Patrakov" <patrakov@gmail.com>:

Hi,

I think that the approach with exporting and importing PGs would be a-priori more successful than the one based on pvmove or ddrescue. The reason is that you don't need to export/import all data that the failed disk holds, but only the PGs that Ceph cannot recover otherwise. The logic here is that these are, likely, not the same PGs due to which tools are crashing.

Note that after the export/import operation Ceph might still think "I need a copy from that failed disk and not the one that you gave me", in this case just export a copy of the same PG from the other failed OSD and import elsewhere, up to the total number of copies. If even that desn't help, "ceph osd lost XX" would be the last (very dangerous) words to convince Cepth that osd.XX will not be seen in the future.

On Sat, Feb 3, 2024 at 5:35 AM Eugen Block <eblock@nde.ag> wrote:

...

Hi,

if the OSDs are deployed as LVs (by ceph-volume) you could try to do a pvmove to a healthy disk. There was a thread here a couple of weeks ago explaining the steps. I don’t have it at hand right now, but it should be easy to find. Of course, there’s no guarantee that this will be successful. I also can’t tell if Igor‘s approach is more promising.

Zitat von Igor Fedotov <igor.fedotov@croit.io>:

...

Hi Carl,

you might want to use ceph-objectstore-tool to export PGs from faulty OSDs and import them back to healthy ones.

The process could be quite tricky though.

There is also pending PR (https://github.com/ceph/ceph/pull/54991) to make the tool more tolerant to disk errors.

The patch worth trying in some cases, not a silver bullet though.

And generally whether the recovery doable greatly depends on the actual error(s).

Thanks,

Igor

On 02/02/2024 19:03, Carl J Taylor wrote:

...

Hi, I have a small cluster with some faulty disks within it and I want to clone the data from the faulty disks onto new ones.

The cluster is currently down and I am unable to do things like ceph-bluestore-fsck but ceph-bluestore-tool bluefs-export does appear to be working.

Any help would be appreciated

Many thanks Carl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Alexander E. Patrakov