OBJECT_UNFOUND and PG_DAMAGED (recovery_unfound) errors
Hi everyone. Help me settle a debate. My coworker is seeing OBJECT_UNFOUND and PG_DAMAGED (recovery_unfound) errors. We both agree they are caused by bad drives. The fix is to mark the drive as out, replace it and add it back in. Whenever we see this error on Ceph we see corresponding read errors on the physical drive. I'm saying that even though the drive is bad since there are two more copies, only 1 of 3 drives has bad sectors preventing the data from being accessed which is what dmesg is showing ie critical medium error, dev sd..., sector 12345... There should not be OBJECT_UNFOUND since Ceph compares the remaining two copies and assuming the data matches, it should be able to recover on it's own and move the data to another PG or maybe OBJECT_UNFOUND and PG_DAMAGED are warnings not errors. My coworker is saying because the primary OSD responsible for coordinating the PG was the one which failed, and is the "source of truth" the cluster goes into error state. His argument doesn't make sense to me since there should be no single point of failure, but I'm also not sure about my argument since I don't know enough about how Ceph works under the hood. Thanks.
Hi, I'm not sure if I can settle a debate, but I can try. Here's a quote from the docs [0]:
One or more objects in the cluster cannot be found. More precisely, the OSDs know that a new or updated copy of an object should exist, but no such copy has been found on OSDs that are currently online.
This is not the usual case when one copy is corrupted, that would more likely display "inconsistent PG" as an error or warning. And that would still allow reading/writing to that PG. Running ceph tell <pgid> query could help understand the issue. If the primary OSD fails, another one takes over, so you're right about that part, it's not the single source of truth. To me it sounds like you have multiple down OSDs so that min_size cannot be fulfilled. If you provided more information, we could help a bit more: ceph -s ceph osd tree ceph tell <pgid> query ceph osd pool ls detail And the crush rule dump in use by the affected pool. Regards, Eugen [0] https://docs.ceph.com/en/latest/rados/operations/health-checks/#object-unfou... Zitat von Alex <mr.alexey@gmail.com>:
Hi everyone. Help me settle a debate.
My coworker is seeing OBJECT_UNFOUND and PG_DAMAGED (recovery_unfound) errors. We both agree they are caused by bad drives. The fix is to mark the drive as out, replace it and add it back in. Whenever we see this error on Ceph we see corresponding read errors on the physical drive.
I'm saying that even though the drive is bad since there are two more copies, only 1 of 3 drives has bad sectors preventing the data from being accessed which is what dmesg is showing ie critical medium error, dev sd..., sector 12345... There should not be OBJECT_UNFOUND since Ceph compares the remaining two copies and assuming the data matches, it should be able to recover on it's own and move the data to another PG or maybe OBJECT_UNFOUND and PG_DAMAGED are warnings not errors.
My coworker is saying because the primary OSD responsible for coordinating the PG was the one which failed, and is the "source of truth" the cluster goes into error state.
His argument doesn't make sense to me since there should be no single point of failure, but I'm also not sure about my argument since I don't know enough about how Ceph works under the hood.
Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Eugen. The cluster is currently HEALTHY, we have 3 replicas of all the data and currently only the one failed HDD is out, the rest are in/up. If I'm understanding you correctly it's likely that there are other bad drives which have not yet been detected causing the failure? That's very likely given the age of our cluster. Other people have also independently suggested that that could be the issue.
So the OBJECT_UNFOUND error cleared without intervention? Maybe you could give some more details what happened. It’s difficult to give a clear answer without knowing the history. As I said, I wouldn’t expect OBJECT_UNFOUND with only one misbehaving drive. Zitat von Alex <mr.alexey@gmail.com>:
Thanks Eugen.
The cluster is currently HEALTHY, we have 3 replicas of all the data and currently only the one failed HDD is out, the rest are in/up. If I'm understanding you correctly it's likely that there are other bad drives which have not yet been detected causing the failure? That's very likely given the age of our cluster. Other people have also independently suggested that that could be the issue.
Sorry I didn't make it clear. We marked the osd as out, the error cleared, then we replaced the drive.
I see. Maybe the drive failed exactly in that moment when the primary OSD got a new write, tried to replicate it to secondary and tertiary, but the ack could not be committed, resulting in the unfound object because the other two OSDs knew there was a newer copy but couldn't be found. I'm not sure if this makes sense, but it could explain it a little. But I feel like this isn't going to settle your debate. ;-) Zitat von Alex <mr.alexey@gmail.com>:
Sorry I didn't make it clear. We marked the osd as out, the error cleared, then we replaced the drive.
Some cases that would lead to unfound objects: 1) If min size = 1. In this case you may have only 1 OSD acting for the pg with the others 2 OSDs being down. The 2 down OSDs could be due to problems with drives themselves or their hosts going down or under load the heartbeats are not received and the 2 OSDs will be marked down and could be flapping (up/down). When recovery kicks in and other OSDs are assigned as replacement or (most likely in your case) downed OSDs are back up, a new epoch is formed and PGs start peering and agree on state of pg, the pg will become active and backfill and/or recovery will occur. If then the primary has a hardware failure before all changes were synced, the remaining OSDs will be aware of the new/updated changes (since they peered) but that they do not have all the data synced. So the new state of the pg will show unfound objects. 2) Power outage with incorrect/bad hardware. If you have power outage affecting all/many OSDs and if you have HDDs on controller with writeback cache and the controller does not have battery back protection. Or you have sub-grade consumer SSD without PLP. It is possible that some new writes transactions do not fully persist. 3) I think in EC, if you have more than m+1 failures, the remaining k-1 copies will peer but will declare all their objects as unfound as they only have shards and cannot construct the objects. In contrast in replicated pools, if you lose all replicas, the pg will be down or unknown. /Maged On 12/05/2025 21:44, Alex wrote:
Hi everyone. Help me settle a debate.
My coworker is seeing OBJECT_UNFOUND and PG_DAMAGED (recovery_unfound) errors. We both agree they are caused by bad drives. The fix is to mark the drive as out, replace it and add it back in. Whenever we see this error on Ceph we see corresponding read errors on the physical drive.
I'm saying that even though the drive is bad since there are two more copies, only 1 of 3 drives has bad sectors preventing the data from being accessed which is what dmesg is showing ie critical medium error, dev sd..., sector 12345... There should not be OBJECT_UNFOUND since Ceph compares the remaining two copies and assuming the data matches, it should be able to recover on it's own and move the data to another PG or maybe OBJECT_UNFOUND and PG_DAMAGED are warnings not errors.
My coworker is saying because the primary OSD responsible for coordinating the PG was the one which failed, and is the "source of truth" the cluster goes into error state.
His argument doesn't make sense to me since there should be no single point of failure, but I'm also not sure about my argument since I don't know enough about how Ceph works under the hood.
Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Maged I don't think our min size = 1 No power outage fingers crossed. #3 Not sure I understand, what's EC?
EC: erasure coded pools. For second point: make sure you do not have controllers with writeback cache without battery back protection and if you use SSD, only use enterprise SSDs. On 15/05/2025 18:46, Alex wrote:
Hi Maged
I don't think our min size = 1 No power outage fingers crossed. #3 Not sure I understand, what's EC?
participants (3)
-
Alex
-
Eugen Block
-
Maged Mokhtar