Incomplete PGs. Ceph Consultant Wanted
Hi community! Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked. Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from. We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details. Thanks in advance Best Regards Pablo
Hi Pablo, Could you tell us a little more about how that happened? Do you have a min_size >= 2 (or E/C equivalent) ? ________________________________________________________ Cordialement, *David CASIER* ________________________________________________________ Le lun. 17 juin 2024 à 16:26, cellosofia1@gmail.com <cellosofia1@gmail.com> a écrit :
Hi community!
Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked.
Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from.
We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details.
Thanks in advance
Best Regards Pablo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Pablo, It depends. If it’s a replicated setup, it might be as easy as marking dead OSDs as lost to get the PGs to recover. In that case it basically just means that you are below the pools min_size. If it is an EC setup, it might be quite a bit more painful, depending on what happened to the dead OSDs and whether they are at all recoverable. Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:46, David C. <david.casier@aevoo.fr> wrote:
Hi Pablo,
Could you tell us a little more about how that happened?
Do you have a min_size >= 2 (or E/C equivalent) ? ________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le lun. 17 juin 2024 à 16:26, cellosofia1@gmail.com <cellosofia1@gmail.com> a écrit :
Hi community!
Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked.
Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from.
We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details.
Thanks in advance
Best Regards Pablo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ah scratch that, my first paragraph about replicated pools is actually incorrect. If it’s a replicated pool and it shows incomplete, it means the most recent copy of the PG is missing. So ideal would be to recover the PG from dead OSDs in any case if possible. Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:56, Matthias Grandl <matthias.grandl@croit.io> wrote:
Hi Pablo,
It depends. If it’s a replicated setup, it might be as easy as marking dead OSDs as lost to get the PGs to recover. In that case it basically just means that you are below the pools min_size.
If it is an EC setup, it might be quite a bit more painful, depending on what happened to the dead OSDs and whether they are at all recoverable.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:46, David C. <david.casier@aevoo.fr> wrote:
Hi Pablo,
Could you tell us a little more about how that happened?
Do you have a min_size >= 2 (or E/C equivalent) ? ________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le lun. 17 juin 2024 à 16:26, cellosofia1@gmail.com <cellosofia1@gmail.com> a écrit :
Hi community!
Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked.
Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from.
We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details.
Thanks in advance
Best Regards Pablo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi everyone, Thanks for your kind responses I know the following is not the best scenario, but sadly I didn't have the opportunity of installing this cluster More information about the problem: * We use replicated pools * Replica 2, min replicas 1. * Ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) * Virtual Machines setup, 2 MGR Nodes, 2 OSD Nodes, 4 VMs in total. * 27 OSDs right now * Rook environment: rook: v1.9.5 * Kubernetes Server Version: v1.24.1 I attach a .txt with the result of some diagnostic commands for reference What do you think? Regards Pablo On Mon, Jun 17, 2024 at 11:01 AM Matthias Grandl <matthias.grandl@croit.io> wrote:
Ah scratch that, my first paragraph about replicated pools is actually incorrect. If it’s a replicated pool and it shows incomplete, it means the most recent copy of the PG is missing. So ideal would be to recover the PG from dead OSDs in any case if possible.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:56, Matthias Grandl <matthias.grandl@croit.io> wrote:
Hi Pablo,
It depends. If it’s a replicated setup, it might be as easy as marking dead OSDs as lost to get the PGs to recover. In that case it basically just means that you are below the pools min_size.
If it is an EC setup, it might be quite a bit more painful, depending on what happened to the dead OSDs and whether they are at all recoverable.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:46, David C. <david.casier@aevoo.fr> wrote:
Hi Pablo,
Could you tell us a little more about how that happened?
Do you have a min_size >= 2 (or E/C equivalent) ? ________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le lun. 17 juin 2024 à 16:26, cellosofia1@gmail.com < cellosofia1@gmail.com> a écrit :
Hi community!
Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked.
Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from.
We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details.
Thanks in advance
Best Regards Pablo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Pablo, Since some PGs are empty and all OSDs are enabled, I'm not optimistic about the future at all. Was the command "ceph osd force-create-pg" executed with missing OSDs ? Le lun. 17 juin 2024 à 17:26, cellosofia1@gmail.com <cellosofia1@gmail.com> a écrit :
Hi everyone,
Thanks for your kind responses
I know the following is not the best scenario, but sadly I didn't have the opportunity of installing this cluster
More information about the problem:
* We use replicated pools * Replica 2, min replicas 1. * Ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) * Virtual Machines setup, 2 MGR Nodes, 2 OSD Nodes, 4 VMs in total. * 27 OSDs right now * Rook environment: rook: v1.9.5 * Kubernetes Server Version: v1.24.1
I attach a .txt with the result of some diagnostic commands for reference
What do you think?
Regards Pablo
On Mon, Jun 17, 2024 at 11:01 AM Matthias Grandl <matthias.grandl@croit.io> wrote:
Ah scratch that, my first paragraph about replicated pools is actually incorrect. If it’s a replicated pool and it shows incomplete, it means the most recent copy of the PG is missing. So ideal would be to recover the PG from dead OSDs in any case if possible.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:56, Matthias Grandl <matthias.grandl@croit.io> wrote:
Hi Pablo,
It depends. If it’s a replicated setup, it might be as easy as marking dead OSDs as lost to get the PGs to recover. In that case it basically just means that you are below the pools min_size.
If it is an EC setup, it might be quite a bit more painful, depending on what happened to the dead OSDs and whether they are at all recoverable.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:46, David C. <david.casier@aevoo.fr> wrote:
Hi Pablo,
Could you tell us a little more about how that happened?
Do you have a min_size >= 2 (or E/C equivalent) ? ________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le lun. 17 juin 2024 à 16:26, cellosofia1@gmail.com < cellosofia1@gmail.com> a écrit :
Hi community!
Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked.
Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from.
We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details.
Thanks in advance
Best Regards Pablo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Perhaps Ceph itself should also have a warning pop up (in "ceph -s", "ceph health detail" etc) when replica and min_size=1 or in an EC if min_size < k+1. Of course it could be muted but it would give an operator pause initially when setting that. I think a lot of people assume replica size=2 is safe enough. I imagine this must have been proposed before. Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Mon, Jun 17, 2024 at 1:07 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
* We use replicated pools * Replica 2, min replicas 1.
Note to self: Change the docs and default to discourage this. This is rarely appropriate in production.
You had multiple overlapping drive failures?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
In Pablo's unfortunate incident, it was because of a SAN incident, so it's possible that Replica 3 didn't save him. In this scenario, the architecture is more the origin of the incident than the number of replicas. It seems to me that replica 3 exists, by default, since firefly => make replica 2, this is intentional. So I'm not sure if adding a warning (again) is necessary. For HDD, apart from special cases (buffer volume, etc.), it is difficult to justify Replica 2 (especially on platforms several years old). However, I'd rather see a full flash Replica 2 platform with solid backups than Replica 3 without backups (well obviously, Replica 3, or E/C + backup are much better). Le lun. 17 juin 2024 à 19:14, Wesley Dillingham <wes@wesdillingham.com> a écrit :
Perhaps Ceph itself should also have a warning pop up (in "ceph -s", "ceph health detail" etc) when replica and min_size=1 or in an EC if min_size < k+1. Of course it could be muted but it would give an operator pause initially when setting that. I think a lot of people assume replica size=2 is safe enough. I imagine this must have been proposed before.
Respectfully,
*Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com
On Mon, Jun 17, 2024 at 1:07 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
* We use replicated pools * Replica 2, min replicas 1.
Note to self: Change the docs and default to discourage this. This is rarely appropriate in production.
You had multiple overlapping drive failures?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ohhh, so multiple OSD failure domains on a single SAN node? I suspected as much. I've experienced a Ceph cluster built on SanDisk InfiniFlash, was was somewhere between SAN and DAS arguably. Each of 4 IF chassis drive 4x OSD nodes via SAS, but it was zoned such that the chassis was the failure domain in the CRUSH tree.
On Jun 17, 2024, at 16:52, David C. <david.casier@aevoo.fr> wrote:
In Pablo's unfortunate incident, it was because of a SAN incident, so it's possible that Replica 3 didn't save him. In this scenario, the architecture is more the origin of the incident than the number of replicas.
It seems to me that replica 3 exists, by default, since firefly => make replica 2, this is intentional.
The default EC profile though is 2,1 and that makes it too easy for someone to understandably assume that the default is suitable for production. I have an action item to update docs and code to default to, say, 2,2 so that it still works on smaller configurations like sandboxes but is less dangerous.
However, I'd rather see a full flash Replica 2 platform with solid backups than Replica 3 without backups (well obviously, Replica 3, or E/C + backup are much better).
Tangent, but yeah RAID or replication != backups. SolidFire was RF2 flash, their assertion was that resilvering was fast enough that it was safe. With Ceph we know there's more to it than that, but I'm not sure if they had special provisions to address the sequences of events that can cause problems with Ceph RF2. They did have limited scalability, though.
Hello Wesley, I couldn't find any tracker related to this and since min_size=1 has been involved in many critical situations with data loss, I created this one: https://tracker.ceph.com/issues/66641 Regards, Frédéric. ----- Le 17 Juin 24, à 19:14, Wesley Dillingham wes@wesdillingham.com a écrit :
Perhaps Ceph itself should also have a warning pop up (in "ceph -s", "ceph health detail" etc) when replica and min_size=1 or in an EC if min_size < k+1. Of course it could be muted but it would give an operator pause initially when setting that. I think a lot of people assume replica size=2 is safe enough. I imagine this must have been proposed before.
Respectfully,
*Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com
On Mon, Jun 17, 2024 at 1:07 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
* We use replicated pools * Replica 2, min replicas 1.
Note to self: Change the docs and default to discourage this. This is rarely appropriate in production.
You had multiple overlapping drive failures?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I understand, We had to re-create the OSDs because of backing storage hardware failure, so recovering from old OSDs is not possible. From your current understanding, is there a possibility to at least recover some of the information, at least the fragments that are not missing. I ask this because I tried to export the pool contents, but it gets stuck (I/O blocked) because of "incomplete" PGs, maybe marking the PGs as complete with ceph-objectstore-tool would be an option? Or using the *dangerous* osd_find_best_info_ignore_history_les option for the affected OSDs? Regards Pablo On Mon, Jun 17, 2024 at 11:46 AM Matthias Grandl <matthias.grandl@croit.io> wrote:
We are missing info here. Ceph status claims all OSDs are up. Did an OSD die and was it already removed from the CRUSH map? If so the only chance I see at preventing data loss is exporting the PGs off of that OSD and importing to another OSD. But yeah as David I am not too optimistic.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 17:26, cellosofia1@gmail.com wrote:
Hi everyone,
Thanks for your kind responses
I know the following is not the best scenario, but sadly I didn't have the opportunity of installing this cluster
More information about the problem:
* We use replicated pools * Replica 2, min replicas 1. * Ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) * Virtual Machines setup, 2 MGR Nodes, 2 OSD Nodes, 4 VMs in total. * 27 OSDs right now * Rook environment: rook: v1.9.5 * Kubernetes Server Version: v1.24.1
I attach a .txt with the result of some diagnostic commands for reference
What do you think?
Regards Pablo
On Mon, Jun 17, 2024 at 11:01 AM Matthias Grandl <matthias.grandl@croit.io> wrote:
Ah scratch that, my first paragraph about replicated pools is actually incorrect. If it’s a replicated pool and it shows incomplete, it means the most recent copy of the PG is missing. So ideal would be to recover the PG from dead OSDs in any case if possible.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:56, Matthias Grandl <matthias.grandl@croit.io> wrote:
Hi Pablo,
It depends. If it’s a replicated setup, it might be as easy as marking dead OSDs as lost to get the PGs to recover. In that case it basically just means that you are below the pools min_size.
If it is an EC setup, it might be quite a bit more painful, depending on what happened to the dead OSDs and whether they are at all recoverable.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:46, David C. <david.casier@aevoo.fr> wrote:
Hi Pablo,
Could you tell us a little more about how that happened?
Do you have a min_size >= 2 (or E/C equivalent) ? ________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le lun. 17 juin 2024 à 16:26, cellosofia1@gmail.com < cellosofia1@gmail.com> a écrit :
Hi community!
Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked.
Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from.
We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details.
Thanks in advance
Best Regards Pablo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
<diagnostic-commands-ceph.txt>
Command for trying the export was: [rook@rook-ceph-tools-recovery-77495958d9-plfch ~]$ rados export -p cephfs-replicated /mnt/recovery/backup-rados-cephfs-replicated We made sure we had enough space for this operation, and mounted the /mnt/recovery path using hostPath in the modified rook "toolbox" deployment. Regards On Mon, Jun 17, 2024 at 11:56 AM cellosofia1@gmail.com < cellosofia1@gmail.com> wrote:
Hi,
I understand,
We had to re-create the OSDs because of backing storage hardware failure, so recovering from old OSDs is not possible.
From your current understanding, is there a possibility to at least recover some of the information, at least the fragments that are not missing.
I ask this because I tried to export the pool contents, but it gets stuck (I/O blocked) because of "incomplete" PGs, maybe marking the PGs as complete with ceph-objectstore-tool would be an option? Or using the *dangerous* osd_find_best_info_ignore_history_les option for the affected OSDs?
Regards Pablo
On Mon, Jun 17, 2024 at 11:46 AM Matthias Grandl <matthias.grandl@croit.io> wrote:
We are missing info here. Ceph status claims all OSDs are up. Did an OSD die and was it already removed from the CRUSH map? If so the only chance I see at preventing data loss is exporting the PGs off of that OSD and importing to another OSD. But yeah as David I am not too optimistic.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 17:26, cellosofia1@gmail.com wrote:
Hi everyone,
Thanks for your kind responses
I know the following is not the best scenario, but sadly I didn't have the opportunity of installing this cluster
More information about the problem:
* We use replicated pools * Replica 2, min replicas 1. * Ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) * Virtual Machines setup, 2 MGR Nodes, 2 OSD Nodes, 4 VMs in total. * 27 OSDs right now * Rook environment: rook: v1.9.5 * Kubernetes Server Version: v1.24.1
I attach a .txt with the result of some diagnostic commands for reference
What do you think?
Regards Pablo
On Mon, Jun 17, 2024 at 11:01 AM Matthias Grandl < matthias.grandl@croit.io> wrote:
Ah scratch that, my first paragraph about replicated pools is actually incorrect. If it’s a replicated pool and it shows incomplete, it means the most recent copy of the PG is missing. So ideal would be to recover the PG from dead OSDs in any case if possible.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:56, Matthias Grandl <matthias.grandl@croit.io> wrote:
Hi Pablo,
It depends. If it’s a replicated setup, it might be as easy as marking dead OSDs as lost to get the PGs to recover. In that case it basically just means that you are below the pools min_size.
If it is an EC setup, it might be quite a bit more painful, depending on what happened to the dead OSDs and whether they are at all recoverable.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:46, David C. <david.casier@aevoo.fr> wrote:
Hi Pablo,
Could you tell us a little more about how that happened?
Do you have a min_size >= 2 (or E/C equivalent) ? ________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le lun. 17 juin 2024 à 16:26, cellosofia1@gmail.com < cellosofia1@gmail.com> a écrit :
Hi community!
Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked.
Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from.
We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details.
Thanks in advance
Best Regards Pablo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
<diagnostic-commands-ceph.txt>
1 pg / 16 is missing, in the meta pool, it is already enough to have great difficulty browsing the FS Your difficulty is to locate important objects in the data pool. Try, perhaps, to target the important objects by retrieving the layout/parent attributes on the objects in the cephfs-replicated pool: Example: # rados -p cephfs-replicated ls |while read inode_chunk; do ... rados -p cephfs-replicated getxattr <inode> layout rados -p cephfs-replicated getxattr <inode> parent Le lun. 17 juin 2024 à 17:59, cellosofia1@gmail.com <cellosofia1@gmail.com> a écrit :
Command for trying the export was:
[rook@rook-ceph-tools-recovery-77495958d9-plfch ~]$ rados export -p cephfs-replicated /mnt/recovery/backup-rados-cephfs-replicated
We made sure we had enough space for this operation, and mounted the /mnt/recovery path using hostPath in the modified rook "toolbox" deployment.
Regards
On Mon, Jun 17, 2024 at 11:56 AM cellosofia1@gmail.com < cellosofia1@gmail.com> wrote:
Hi,
I understand,
We had to re-create the OSDs because of backing storage hardware failure, so recovering from old OSDs is not possible.
From your current understanding, is there a possibility to at least recover some of the information, at least the fragments that are not missing.
I ask this because I tried to export the pool contents, but it gets stuck (I/O blocked) because of "incomplete" PGs, maybe marking the PGs as complete with ceph-objectstore-tool would be an option? Or using the *dangerous* osd_find_best_info_ignore_history_les option for the affected OSDs?
Regards Pablo
On Mon, Jun 17, 2024 at 11:46 AM Matthias Grandl < matthias.grandl@croit.io> wrote:
We are missing info here. Ceph status claims all OSDs are up. Did an OSD die and was it already removed from the CRUSH map? If so the only chance I see at preventing data loss is exporting the PGs off of that OSD and importing to another OSD. But yeah as David I am not too optimistic.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 17:26, cellosofia1@gmail.com wrote:
Hi everyone,
Thanks for your kind responses
I know the following is not the best scenario, but sadly I didn't have the opportunity of installing this cluster
More information about the problem:
* We use replicated pools * Replica 2, min replicas 1. * Ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) * Virtual Machines setup, 2 MGR Nodes, 2 OSD Nodes, 4 VMs in total. * 27 OSDs right now * Rook environment: rook: v1.9.5 * Kubernetes Server Version: v1.24.1
I attach a .txt with the result of some diagnostic commands for reference
What do you think?
Regards Pablo
On Mon, Jun 17, 2024 at 11:01 AM Matthias Grandl < matthias.grandl@croit.io> wrote:
Ah scratch that, my first paragraph about replicated pools is actually incorrect. If it’s a replicated pool and it shows incomplete, it means the most recent copy of the PG is missing. So ideal would be to recover the PG from dead OSDs in any case if possible.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:56, Matthias Grandl <matthias.grandl@croit.io> wrote:
Hi Pablo,
It depends. If it’s a replicated setup, it might be as easy as marking dead OSDs as lost to get the PGs to recover. In that case it basically just means that you are below the pools min_size.
If it is an EC setup, it might be quite a bit more painful, depending on what happened to the dead OSDs and whether they are at all recoverable.
Matthias Grandl Head Storage Engineer matthias.grandl@croit.io
On 17. Jun 2024, at 16:46, David C. <david.casier@aevoo.fr> wrote:
Hi Pablo,
Could you tell us a little more about how that happened?
Do you have a min_size >= 2 (or E/C equivalent) ? ________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le lun. 17 juin 2024 à 16:26, cellosofia1@gmail.com < cellosofia1@gmail.com> a écrit :
> Hi community! > > Recently we had a major outage in production and after running the > automated ceph recovery, some PGs remain in "incomplete" state, and IO > operations are blocked. > > Searching in documentation, forums, and this mailing list archive, I > haven't found yet if this means this data is recoverable or not. We don't > have any "unknown" objects or PGs, so I believe this is somehow an > intermediate stage where we have to tell ceph which version of the objects > to recover from. > > We are willing to work with a Ceph Consultant Specialist, because the data > at stage is very critical, so if you're interested please let me know > off-list, to discuss the details. > > Thanks in advance > > Best Regards > Pablo > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
<diagnostic-commands-ceph.txt>
participants (6)
-
Anthony D'Atri
-
cellosofia1＠gmail.com
-
David C.
-
Frédéric Nass
-
Matthias Grandl
-
Wesley Dillingham