Command for trying the export was: [rook@rook-ceph-tools-recovery-77495958d9-plfch ~]$ rados export -p cephfs-replicated /mnt/recovery/backup-rados-cephfs-replicated We made sure we had enough space for this operation, and mounted the /mnt/recovery path using hostPath in the modified rook "toolbox" deployment. Regards On Mon, Jun 17, 2024 at 11:56 AM cellosofia1@gmail.com < cellosofia1@gmail.com> wrote:
Hi,
I understand,
We had to re-create the OSDs because of backing storage hardware failure, so recovering from old OSDs is not possible.
From your current understanding, is there a possibility to at least recover some of the information, at least the fragments that are not missing.
I ask this because I tried to export the pool contents, but it gets stuck (I/O blocked) because of "incomplete" PGs, maybe marking the PGs as complete with ceph-objectstore-tool would be an option? Or using the *dangerous* osd_find_best_info_ignore_history_les option for the affected OSDs?
Regards Pablo
On Mon, Jun 17, 2024 at 11:46 AM Matthias Grandl <matthias.grandl@croit.io> wrote:
We are missing info here. Ceph status claims all OSDs are up. Did an OSD die and was it already removed from the CRUSH map? If so the only chance I see at preventing data loss is exporting the PGs off of that OSD and importing to another OSD. But yeah as David I am not too optimistic.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 17:26, cellosofia1@gmail.com wrote:
Hi everyone,
Thanks for your kind responses
I know the following is not the best scenario, but sadly I didn't have the opportunity of installing this cluster
More information about the problem:
* We use replicated pools * Replica 2, min replicas 1. * Ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) * Virtual Machines setup, 2 MGR Nodes, 2 OSD Nodes, 4 VMs in total. * 27 OSDs right now * Rook environment: rook: v1.9.5 * Kubernetes Server Version: v1.24.1
I attach a .txt with the result of some diagnostic commands for reference
What do you think?
Regards Pablo
On Mon, Jun 17, 2024 at 11:01 AM Matthias Grandl < matthias.grandl@croit.io> wrote:
Ah scratch that, my first paragraph about replicated pools is actually incorrect. If it’s a replicated pool and it shows incomplete, it means the most recent copy of the PG is missing. So ideal would be to recover the PG from dead OSDs in any case if possible.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:56, Matthias Grandl <matthias.grandl@croit.io> wrote:
Hi Pablo,
It depends. If it’s a replicated setup, it might be as easy as marking dead OSDs as lost to get the PGs to recover. In that case it basically just means that you are below the pools min_size.
If it is an EC setup, it might be quite a bit more painful, depending on what happened to the dead OSDs and whether they are at all recoverable.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:46, David C. <david.casier@aevoo.fr> wrote:
Hi Pablo,
Could you tell us a little more about how that happened?
Do you have a min_size >= 2 (or E/C equivalent) ? ________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le lun. 17 juin 2024 à 16:26, cellosofia1@gmail.com < cellosofia1@gmail.com> a écrit :
Hi community!
Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked.
Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from.
We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details.
Thanks in advance
Best Regards Pablo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
<diagnostic-commands-ceph.txt>