Thanks, we are currently scanning our object storage. It looks like we can detect the missing objects that return “No Such Key” looking at all “__multipart_” objects returned by radosgw-admin bucket radoslist, and checking if they exist using rados stat. We are currently not looking at shadow objects as our approach already yields more instances of this problem.
On 19 Nov 2020, at 09:09, Janek Bevendorff <janek.bevendorff@uni-weimar.de> wrote:
- The head object had a size of 0. - There was an object with a ’shadow’ in its name, belonging to that path. That is normal. What is not normal is if there are NO shadow objects.
On 18/11/2020 10:06, Denis Krienbühl wrote:
It looks like a single-part object. But we did replace that object last night from backup, so I can’t know for sure if the lost one was like that.
Another engineer that looked at the Rados objects last night did notice two things:
- The head object had a size of 0. - There was an object with a ’shadow’ in its name, belonging to that path.
I’m not knowledgable about Rados, so I’m not sure this is helpful.
On 18 Nov 2020, at 10:01, Janek Bevendorff <janek.bevendorff@uni-weimar.de> wrote:
Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot the "object" there)
On 18/11/2020 09:58, Janek Bevendorff wrote:
The object, a Docker layer, that went missing has not been touched in 2 months. It worked for a while, but then suddenly went missing. Was the object a multipart object? You can check by running radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something "ns": "multipart" in the output. If it says "ns": "shadow", it's a single-part object.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io