Hi, I'm not sure if I can settle a debate, but I can try. Here's a quote from the docs [0]:
One or more objects in the cluster cannot be found. More precisely, the OSDs know that a new or updated copy of an object should exist, but no such copy has been found on OSDs that are currently online.
This is not the usual case when one copy is corrupted, that would more likely display "inconsistent PG" as an error or warning. And that would still allow reading/writing to that PG. Running ceph tell <pgid> query could help understand the issue. If the primary OSD fails, another one takes over, so you're right about that part, it's not the single source of truth. To me it sounds like you have multiple down OSDs so that min_size cannot be fulfilled. If you provided more information, we could help a bit more: ceph -s ceph osd tree ceph tell <pgid> query ceph osd pool ls detail And the crush rule dump in use by the affected pool. Regards, Eugen [0] https://docs.ceph.com/en/latest/rados/operations/health-checks/#object-unfou... Zitat von Alex <mr.alexey@gmail.com>:
Hi everyone. Help me settle a debate.
My coworker is seeing OBJECT_UNFOUND and PG_DAMAGED (recovery_unfound) errors. We both agree they are caused by bad drives. The fix is to mark the drive as out, replace it and add it back in. Whenever we see this error on Ceph we see corresponding read errors on the physical drive.
I'm saying that even though the drive is bad since there are two more copies, only 1 of 3 drives has bad sectors preventing the data from being accessed which is what dmesg is showing ie critical medium error, dev sd..., sector 12345... There should not be OBJECT_UNFOUND since Ceph compares the remaining two copies and assuming the data matches, it should be able to recover on it's own and move the data to another PG or maybe OBJECT_UNFOUND and PG_DAMAGED are warnings not errors.
My coworker is saying because the primary OSD responsible for coordinating the PG was the one which failed, and is the "source of truth" the cluster goes into error state.
His argument doesn't make sense to me since there should be no single point of failure, but I'm also not sure about my argument since I don't know enough about how Ceph works under the hood.
Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io