9 out of 11 missing shards of shadow object in ERC 8:3 pool.
After an upgrade from Nautilus to Pacific the scrub has found an inconsistent object and reports that 9 out of 11 shards are missing. (However, we're not sure this has to do with the upgrade). We have been able to trace it to a S3 bucket, but not to a specific S3 object. # radosgw-admin object stat --bucket=$BUCKET --object=$OBJECT ERROR: failed to stat object, returned error: (2) No such file or directory By design, we have a complete mirror of the bucket in another Ceph cluster and the amount of objects in the buckets match between the clusters. We are therefore somewhat confident that we are not missing any objects. Could this be a failed garbage collection where perhaps the primary OSD failed during gc? The garbage collector does not show anything that seems relevant though... radosgw-admin gc list --include-all | grep "eaa6801e-3967-4541-9b8ca98aa5c2.791015596" Any suggestions on how we can trace and/or fix this inconsistent object? # rados list-inconsistent-obj 11.3ff | jq { "epoch": 177981, "inconsistents": [ { "object": { "name": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "nspace": "", "locator": "", "snap": "head", "version": 109853 }, "errors": [], "union_shard_errors": [ "missing" ], "selected_object_info": { "oid": { "oid": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "key": "", "snapid": -2, "hash": 4294967295, "max": 0, "pool": 11, "namespace": "" }, "version": "17636'109853", "prior_version": "0'0", "last_reqid": "client.791015590.0:449317175", "user_version": 109853, "size": 8388608, "mtime": "2022-01-24T03:33:42.457722+0000", "local_mtime": "2022-01-24T03:33:42.471042+0000", "lost": 0, "flags": [ "dirty", "data_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xe588978d", "omap_digest": "0xffffffff", "expected_object_size": 0, "expected_write_size": 0, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 14, "primary": true, "shard": 0, "errors": [], "size": 1048576 }, { "osd": 67, "primary": false, "shard": 1, "errors": [ "missing" ] }, { "osd": 77, "primary": false, "shard": 4, "errors": [], "size": 1048576 }, { "osd": 225, "primary": false, "shard": 9, "errors": [ "missing" ] }, { "osd": 253, "primary": false, "shard": 8, "errors": [ "missing" ] }, { "osd": 327, "primary": false, "shard": 6, "errors": [ "missing" ] }, { "osd": 568, "primary": false, "shard": 2, "errors": [ "missing" ] }, { "osd": 610, "primary": false, "shard": 7, "errors": [ "missing" ] }, { "osd": 700, "primary": false, "shard": 3, "errors": [ "missing" ] }, { "osd": 736, "primary": false, "shard": 10, "errors": [ "missing" ] }, { "osd": 764, "primary": false, "shard": 5, "errors": [ "missing" ] } ] } ] }
We saw this a fair bit in Nautilus, and I also suspected that there was something up with GC'd and/or deleted objects, but we never determined the cause. Notably it seemed to happen on PGs ending in 'ff' or 'fff', which was extra suspicious. We haven't seen it since Pacific. Josh On Fri, Oct 4, 2024 at 3:06 AM Robert Kihlberg <robkih@gmail.com> wrote:
After an upgrade from Nautilus to Pacific the scrub has found an inconsistent object and reports that 9 out of 11 shards are missing. (However, we're not sure this has to do with the upgrade).
We have been able to trace it to a S3 bucket, but not to a specific S3 object.
# radosgw-admin object stat --bucket=$BUCKET --object=$OBJECT ERROR: failed to stat object, returned error: (2) No such file or directory
By design, we have a complete mirror of the bucket in another Ceph cluster and the amount of objects in the buckets match between the clusters. We are therefore somewhat confident that we are not missing any objects.
Could this be a failed garbage collection where perhaps the primary OSD failed during gc?
The garbage collector does not show anything that seems relevant though... radosgw-admin gc list --include-all | grep "eaa6801e-3967-4541-9b8ca98aa5c2.791015596"
Any suggestions on how we can trace and/or fix this inconsistent object?
# rados list-inconsistent-obj 11.3ff | jq { "epoch": 177981, "inconsistents": [ { "object": { "name": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "nspace": "", "locator": "", "snap": "head", "version": 109853 }, "errors": [], "union_shard_errors": [ "missing" ], "selected_object_info": { "oid": { "oid": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "key": "", "snapid": -2, "hash": 4294967295, "max": 0, "pool": 11, "namespace": "" }, "version": "17636'109853", "prior_version": "0'0", "last_reqid": "client.791015590.0:449317175", "user_version": 109853, "size": 8388608, "mtime": "2022-01-24T03:33:42.457722+0000", "local_mtime": "2022-01-24T03:33:42.471042+0000", "lost": 0, "flags": [ "dirty", "data_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xe588978d", "omap_digest": "0xffffffff", "expected_object_size": 0, "expected_write_size": 0, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 14, "primary": true, "shard": 0, "errors": [], "size": 1048576 }, { "osd": 67, "primary": false, "shard": 1, "errors": [ "missing" ] }, { "osd": 77, "primary": false, "shard": 4, "errors": [], "size": 1048576 }, { "osd": 225, "primary": false, "shard": 9, "errors": [ "missing" ] }, { "osd": 253, "primary": false, "shard": 8, "errors": [ "missing" ] }, { "osd": 327, "primary": false, "shard": 6, "errors": [ "missing" ] }, { "osd": 568, "primary": false, "shard": 2, "errors": [ "missing" ] }, { "osd": 610, "primary": false, "shard": 7, "errors": [ "missing" ] }, { "osd": 700, "primary": false, "shard": 3, "errors": [ "missing" ] }, { "osd": 736, "primary": false, "shard": 10, "errors": [ "missing" ] }, { "osd": 764, "primary": false, "shard": 5, "errors": [ "missing" ] } ] } ] } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This reminds me of this tracker: https://tracker.ceph.com/issues/50351 IIRC, the information could be actually lost on the OSDs. I’m surprised that the number of missing shards is that high though. If you have the objects mirrored, maybe importing it with objectstore-tool could be a way forward. But I’m really not sure if that would be the right approach. Zitat von Robert Kihlberg <robkih@gmail.com>:
After an upgrade from Nautilus to Pacific the scrub has found an inconsistent object and reports that 9 out of 11 shards are missing. (However, we're not sure this has to do with the upgrade).
We have been able to trace it to a S3 bucket, but not to a specific S3 object.
# radosgw-admin object stat --bucket=$BUCKET --object=$OBJECT ERROR: failed to stat object, returned error: (2) No such file or directory
By design, we have a complete mirror of the bucket in another Ceph cluster and the amount of objects in the buckets match between the clusters. We are therefore somewhat confident that we are not missing any objects.
Could this be a failed garbage collection where perhaps the primary OSD failed during gc?
The garbage collector does not show anything that seems relevant though... radosgw-admin gc list --include-all | grep "eaa6801e-3967-4541-9b8ca98aa5c2.791015596"
Any suggestions on how we can trace and/or fix this inconsistent object?
# rados list-inconsistent-obj 11.3ff | jq { "epoch": 177981, "inconsistents": [ { "object": { "name": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "nspace": "", "locator": "", "snap": "head", "version": 109853 }, "errors": [], "union_shard_errors": [ "missing" ], "selected_object_info": { "oid": { "oid": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "key": "", "snapid": -2, "hash": 4294967295, "max": 0, "pool": 11, "namespace": "" }, "version": "17636'109853", "prior_version": "0'0", "last_reqid": "client.791015590.0:449317175", "user_version": 109853, "size": 8388608, "mtime": "2022-01-24T03:33:42.457722+0000", "local_mtime": "2022-01-24T03:33:42.471042+0000", "lost": 0, "flags": [ "dirty", "data_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xe588978d", "omap_digest": "0xffffffff", "expected_object_size": 0, "expected_write_size": 0, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 14, "primary": true, "shard": 0, "errors": [], "size": 1048576 }, { "osd": 67, "primary": false, "shard": 1, "errors": [ "missing" ] }, { "osd": 77, "primary": false, "shard": 4, "errors": [], "size": 1048576 }, { "osd": 225, "primary": false, "shard": 9, "errors": [ "missing" ] }, { "osd": 253, "primary": false, "shard": 8, "errors": [ "missing" ] }, { "osd": 327, "primary": false, "shard": 6, "errors": [ "missing" ] }, { "osd": 568, "primary": false, "shard": 2, "errors": [ "missing" ] }, { "osd": 610, "primary": false, "shard": 7, "errors": [ "missing" ] }, { "osd": 700, "primary": false, "shard": 3, "errors": [ "missing" ] }, { "osd": 736, "primary": false, "shard": 10, "errors": [ "missing" ] }, { "osd": 764, "primary": false, "shard": 5, "errors": [ "missing" ] } ] } ] } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Josh and Eugen, I did not manage to trace this object to an S3 object. Instead I read all files in the suspected S3 bucket and actually hit a bad one. Since we had a known good mirror I deleted the broken S3 object (which succeeded fine) and uploaded the good one (also succeeded). Data wise we're fine. After this, a deep-scrub still reported the PG to be inconsistent. [DBG] 11.3ff deep-scrub starts [ERR] 11.3ffs0 deep-scrub : stat mismatch, got 417274/417273 objects, 0/0 clones, 417274/417273 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 2997719087159/2997710698551 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. [ERR] 11.3ff deep-scrub 1 errors However, rados list-inconsistent-obj 11.3ff did not report any more details. { "epoch": 177981, "inconsistents": [] } We were hesitant to run a repair on it since we didn't know what would happen, but I accidently did it anyway... [DBG] 11.3ff repair starts [ERR] 11.3ffs0 repair : stat mismatch, got 420131/420130 objects, 0/0 clones, 420131/420130 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 3020439864945/3020431476337 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. [ERR] 11.3ff repair 1 errors, 1 fixed It appears to have fixed it and scrubbing it afterwards reports no issues. So this issue seems solved for now. Best Regards, Robert Den lör 5 okt. 2024 kl 14:45 skrev Eugen Block <eblock@nde.ag>:
This reminds me of this tracker:
https://tracker.ceph.com/issues/50351
IIRC, the information could be actually lost on the OSDs. I’m surprised that the number of missing shards is that high though. If you have the objects mirrored, maybe importing it with objectstore-tool could be a way forward. But I’m really not sure if that would be the right approach.
Zitat von Robert Kihlberg <robkih@gmail.com>:
After an upgrade from Nautilus to Pacific the scrub has found an inconsistent object and reports that 9 out of 11 shards are missing. (However, we're not sure this has to do with the upgrade).
We have been able to trace it to a S3 bucket, but not to a specific S3 object.
# radosgw-admin object stat --bucket=$BUCKET --object=$OBJECT ERROR: failed to stat object, returned error: (2) No such file or directory
By design, we have a complete mirror of the bucket in another Ceph cluster and the amount of objects in the buckets match between the clusters. We are therefore somewhat confident that we are not missing any objects.
Could this be a failed garbage collection where perhaps the primary OSD failed during gc?
The garbage collector does not show anything that seems relevant though... radosgw-admin gc list --include-all | grep "eaa6801e-3967-4541-9b8ca98aa5c2.791015596"
Any suggestions on how we can trace and/or fix this inconsistent object?
# rados list-inconsistent-obj 11.3ff | jq { "epoch": 177981, "inconsistents": [ { "object": { "name":
"eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5",
"nspace": "", "locator": "", "snap": "head", "version": 109853 }, "errors": [], "union_shard_errors": [ "missing" ], "selected_object_info": { "oid": { "oid":
"eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5",
"key": "", "snapid": -2, "hash": 4294967295, "max": 0, "pool": 11, "namespace": "" }, "version": "17636'109853", "prior_version": "0'0", "last_reqid": "client.791015590.0:449317175", "user_version": 109853, "size": 8388608, "mtime": "2022-01-24T03:33:42.457722+0000", "local_mtime": "2022-01-24T03:33:42.471042+0000", "lost": 0, "flags": [ "dirty", "data_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xe588978d", "omap_digest": "0xffffffff", "expected_object_size": 0, "expected_write_size": 0, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 14, "primary": true, "shard": 0, "errors": [], "size": 1048576 }, { "osd": 67, "primary": false, "shard": 1, "errors": [ "missing" ] }, { "osd": 77, "primary": false, "shard": 4, "errors": [], "size": 1048576 }, { "osd": 225, "primary": false, "shard": 9, "errors": [ "missing" ] }, { "osd": 253, "primary": false, "shard": 8, "errors": [ "missing" ] }, { "osd": 327, "primary": false, "shard": 6, "errors": [ "missing" ] }, { "osd": 568, "primary": false, "shard": 2, "errors": [ "missing" ] }, { "osd": 610, "primary": false, "shard": 7, "errors": [ "missing" ] }, { "osd": 700, "primary": false, "shard": 3, "errors": [ "missing" ] }, { "osd": 736, "primary": false, "shard": 10, "errors": [ "missing" ] }, { "osd": 764, "primary": false, "shard": 5, "errors": [ "missing" ] } ] } ] } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Robert, thanks for the update, it's great that the issue is resolved. Zitat von Robert Kihlberg <robkih@gmail.com>:
Thanks Josh and Eugen,
I did not manage to trace this object to an S3 object. Instead I read all files in the suspected S3 bucket and actually hit a bad one. Since we had a known good mirror I deleted the broken S3 object (which succeeded fine) and uploaded the good one (also succeeded). Data wise we're fine.
After this, a deep-scrub still reported the PG to be inconsistent. [DBG] 11.3ff deep-scrub starts [ERR] 11.3ffs0 deep-scrub : stat mismatch, got 417274/417273 objects, 0/0 clones, 417274/417273 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 2997719087159/2997710698551 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. [ERR] 11.3ff deep-scrub 1 errors
However, rados list-inconsistent-obj 11.3ff did not report any more details. { "epoch": 177981, "inconsistents": [] }
We were hesitant to run a repair on it since we didn't know what would happen, but I accidently did it anyway... [DBG] 11.3ff repair starts [ERR] 11.3ffs0 repair : stat mismatch, got 420131/420130 objects, 0/0 clones, 420131/420130 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 3020439864945/3020431476337 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. [ERR] 11.3ff repair 1 errors, 1 fixed
It appears to have fixed it and scrubbing it afterwards reports no issues.
So this issue seems solved for now.
Best Regards, Robert
Den lör 5 okt. 2024 kl 14:45 skrev Eugen Block <eblock@nde.ag>:
This reminds me of this tracker:
https://tracker.ceph.com/issues/50351
IIRC, the information could be actually lost on the OSDs. I’m surprised that the number of missing shards is that high though. If you have the objects mirrored, maybe importing it with objectstore-tool could be a way forward. But I’m really not sure if that would be the right approach.
Zitat von Robert Kihlberg <robkih@gmail.com>:
After an upgrade from Nautilus to Pacific the scrub has found an inconsistent object and reports that 9 out of 11 shards are missing. (However, we're not sure this has to do with the upgrade).
We have been able to trace it to a S3 bucket, but not to a specific S3 object.
# radosgw-admin object stat --bucket=$BUCKET --object=$OBJECT ERROR: failed to stat object, returned error: (2) No such file or directory
By design, we have a complete mirror of the bucket in another Ceph cluster and the amount of objects in the buckets match between the clusters. We are therefore somewhat confident that we are not missing any objects.
Could this be a failed garbage collection where perhaps the primary OSD failed during gc?
The garbage collector does not show anything that seems relevant though... radosgw-admin gc list --include-all | grep "eaa6801e-3967-4541-9b8ca98aa5c2.791015596"
Any suggestions on how we can trace and/or fix this inconsistent object?
# rados list-inconsistent-obj 11.3ff | jq { "epoch": 177981, "inconsistents": [ { "object": { "name":
"eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5",
"nspace": "", "locator": "", "snap": "head", "version": 109853 }, "errors": [], "union_shard_errors": [ "missing" ], "selected_object_info": { "oid": { "oid":
"eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5",
"key": "", "snapid": -2, "hash": 4294967295, "max": 0, "pool": 11, "namespace": "" }, "version": "17636'109853", "prior_version": "0'0", "last_reqid": "client.791015590.0:449317175", "user_version": 109853, "size": 8388608, "mtime": "2022-01-24T03:33:42.457722+0000", "local_mtime": "2022-01-24T03:33:42.471042+0000", "lost": 0, "flags": [ "dirty", "data_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xe588978d", "omap_digest": "0xffffffff", "expected_object_size": 0, "expected_write_size": 0, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 14, "primary": true, "shard": 0, "errors": [], "size": 1048576 }, { "osd": 67, "primary": false, "shard": 1, "errors": [ "missing" ] }, { "osd": 77, "primary": false, "shard": 4, "errors": [], "size": 1048576 }, { "osd": 225, "primary": false, "shard": 9, "errors": [ "missing" ] }, { "osd": 253, "primary": false, "shard": 8, "errors": [ "missing" ] }, { "osd": 327, "primary": false, "shard": 6, "errors": [ "missing" ] }, { "osd": 568, "primary": false, "shard": 2, "errors": [ "missing" ] }, { "osd": 610, "primary": false, "shard": 7, "errors": [ "missing" ] }, { "osd": 700, "primary": false, "shard": 3, "errors": [ "missing" ] }, { "osd": 736, "primary": false, "shard": 10, "errors": [ "missing" ] }, { "osd": 764, "primary": false, "shard": 5, "errors": [ "missing" ] } ] } ] } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Joshua Baergen
-
Robert Kihlberg