13 Nov
2020
13 Nov
'20
5:14 p.m.
I have some questions for those who’ve experienced this issue.
1. It seems like those reporting this issue are seeing it strictly after upgrading to
Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus?
Mimic? Luminous?
2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please
describe.
3. Is anyone making copies of the affected objects (to same or to a different bucket)
prior to seeing the issue? And if they are making copies, does the destination bucket have
lifecycle rules? And if they are making copies, are those copies ever being removed?
4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'?
That would allow us to see a request from an RGW to either remove a tail object or
decrement its reference counter (and when its counter reaches 0 it will be deleted).
Thanks,
Eric
On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn
wrote:
Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly
identified and fixed.
best regards,
Samuel
huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>
From: Janek Bevendorff
Date: 2020-11-12 18:17
To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael
Lopez
CC: Robin H. Johnson; ceph-users
Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk
I have never seen this on Luminous. I recently upgraded to Octopus and the issue started
occurring only few weeks later.
On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote:
which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the
latest?
any idea?
samuel
huxiaoyu(a)horebdata.cn
From: EDH - Manuel Rios
Date: 2020-11-12 14:27
To: Janek Bevendorff; Rafael Lopez
CC: Robin H. Johnson; ceph-users
Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk
This same error caused us to wipe a full cluster of 300TB... will be related to some
rados index/database bug not to s3.
As Janek exposed is a mayor issue, because the error silent happend and you can only
detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey.
Error is not related to S3 logic ..
Hope this time dev's can take enought time to find and resolve the issue. Error
happens with low ec profiles, even with replica x3 in some cases.
Regards
-----Mensaje original-----
De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de>>
Enviado el: jueves, 12 de noviembre de 2020 14:06
Para: Rafael Lopez <rafael.lopez(a)monash.edu
<mailto:rafael.lopez@monash.edu>>
CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>;
ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>>
Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk
Here is a bug report concerning (probably) this exact issue:
https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866>
I left a comment describing the situation and my (limited) experiences with it.
On 11/11/2020 10:04, Janek Bevendorff wrote:
Yeah, that seems to be it. There are 239 objects prefixed
.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none
of the multiparts from the other file to be found and the head object
is 0 bytes.
I checked another multipart object with an end pointer of 11.
Surprisingly, it had way more than 11 parts (39 to be precise) named
.1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I
could find them in the dump at least.
I have no idea why the objects disappeared. I ran a Spark job over all
buckets, read 1 byte of every object and recorded errors. Of the 78
buckets, two are missing objects. One bucket is missing one object,
the other 15. So, luckily, the incidence is still quite low, but the
problem seems to be expanding slowly.
On 10/11/2020 23:46, Rafael Lopez wrote:
> Hi Janek,
>
> What you said sounds right - an S3 single part obj won't have an S3
> multipart string as part of the prefix. S3 multipart string looks
> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme".
>
> From memory, single part S3 objects that don't fit in a single rados
> object are assigned a random prefix that has nothing to do with
> the object name, and the rados tail/data objects (not the head
> object) have that prefix.
> As per your working example, the prefix for that would be
> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239)
"shadow"
> objects with names containing that prefix, and if you add up the
> sizes it should be the size of your S3 object.
>
> You should look at working and non working examples of both single
> and multipart S3 objects, as they are probably all a bit different
> when you look in rados.
>
> I agree it is a serious issue, because once objects are no longer in
> rados, they cannot be recovered. If it was a case that there was a
> link broken or rados objects renamed, then we could work to
> recover...but as far as I can tell, it looks like stuff is just
> vanishing from rados. The only explanation I can think of is some
> (rgw or rados) background process is incorrectly doing something with
> these objects (eg. renaming/deleting). I had thought perhaps it was a
> bug with the rgw garbage collector..but that is pure speculation.
>
> Once you can articulate the problem, I'd recommend logging a bug
> tracker upstream.
>
>
> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff
> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>
> <mailto:janek.bevendorff@uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de>>> wrote:
>
> Here's something else I noticed: when I stat objects that work
> via radosgw-admin, the stat info contains a "begin_iter" JSON
> object with RADOS key info like this
>
>
> "key": {
> "name":
>
"29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz",
> "instance": "",
> "ns": ""
> }
>
>
> and then "end_iter" with key info like this:
>
>
> "key": {
> "name":
> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239",
> "instance": "",
> "ns": "shadow"
> }
>
> However, when I check the broken 0-byte object, the "begin_iter"
> and "end_iter" keys look like this:
>
>
> "key": {
> "name":
>
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1",
> "instance": "",
> "ns": "multipart"
> }
>
> [...]
>
>
> "key": {
> "name":
>
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19",
> "instance": "",
> "ns": "multipart"
> }
>
> So, it's the full name plus a suffix and the namespace is
> multipart, not shadow (or empty). This in itself may just be an
> artefact of whether the object was uploaded in one go or as a
> multipart object, but the second difference is that I cannot find
> any of the multipart objects in my pool's object name dump. I
> can, however, find the shadow RADOS object of the intact S3 object.
>
>
>
>
> --
> *Rafael Lopez*
> Devops Systems Engineer
> Monash University eResearch Centre
>
> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118>
> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>
>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
13 Nov
13 Nov
5:22 p.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
1. It seems like those reporting this issue are seeing it strictly
after upgrading to Octopus. From what version did each of these sites
upgrade to Octopus? From Nautilus? Mimic? Luminous?
I upgraded from the latest Luminous release.
2. Does anyone have any lifecycle rules on a bucket experiencing this
issue? If so, please describe.
Nope.
3. Is anyone making copies of the affected objects (to same or to a
different bucket) prior to seeing the issue? And if they are making
copies, does the destination bucket have lifecycle rules? And if they
are making copies, are those copies ever being removed?
We are not making copies, but we have bucket ACLs in place, which allow
different users to access the objects. I doubt this is the problem
though, otherwise we probably would have lost terabytes upon terabytes
and not 16 objects so far.
4. Is anyone experiencing this issue willing to run their RGWs with
'debug_ms=1'? That would allow us to see a request from an RGW to
either remove a tail object or decrement its reference counter (and
when its counter reaches 0 it will be deleted).
I haven't had any new data loss in the last few days (at least I think
so, I read 1byte from all objects, but didn't compare checksums, so I
cannot say if all objects are complete, but at least all are there).
Thanks,
Eric
On Nov 12, 2020, at 4:54 PM,
huxiaoyu(a)horebdata.cn
<mailto:huxiaoyu@horebdata.cn> wrote:
Looks like this is a very dangerous bug for data safety. Hope the bug
would be quickly identified and fixed.
best regards,
Samuel
huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>
From: Janek Bevendorff
Date: 2020-11-12 18:17
To:huxiaoyu@horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel
Rios; Rafael Lopez
CC: Robin H. Johnson; ceph-users
Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3
list/radosgw bk
I have never seen this on Luminous. I recently upgraded to Octopus
and the issue started occurring only few weeks later.
On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn
<mailto:huxiaoyu@horebdata.cn> wrote:
which Ceph versions are affected by this RGW bug/issues? Luminous,
Mimic, Octupos, or the latest?
any idea?
samuel
huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>
From: EDH - Manuel Rios
Date: 2020-11-12 14:27
To: Janek Bevendorff; Rafael Lopez
CC: Robin H. Johnson; ceph-users
Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3
list/radosgw bk
This same error caused us to wipe a full cluster of 300TB... will be
related to some rados index/database bug not to s3.
As Janek exposed is a mayor issue, because the error silent happend
and you can only detect it with S3, when you're going to delete/purge
a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic ..
Hope this time dev's can take enought time to find and resolve the
issue. Error happens with low ec profiles, even with replica x3 in
some cases.
Regards
-----Mensaje original-----
De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de>>
Enviado el: jueves, 12 de noviembre de 2020 14:06
Para: Rafael Lopez <rafael.lopez(a)monash.edu
<mailto:rafael.lopez@monash.edu>>
CC: Robin H. Johnson <robbat2(a)gentoo.org
<mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>>
Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3
list/radosgw bk
Here is a bug report concerning (probably) this exact issue:
https://tracker.ceph.com/issues/47866
I left a comment describing the situation and my (limited)
experiences with it.
On 11/11/2020 10:04, Janek Bevendorff wrote:
Yeah, that seems to be it. There are 239 objects prefixed
.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none
of the multiparts from the other file to be found and the head object
is 0 bytes.
I checked another multipart object with an end pointer of 11.
Surprisingly, it had way more than 11 parts (39 to be precise) named
.1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I
could find them in the dump at least.
I have no idea why the objects disappeared. I ran a Spark job over all
buckets, read 1 byte of every object and recorded errors. Of the 78
buckets, two are missing objects. One bucket is missing one object,
the other 15. So, luckily, the incidence is still quite low, but the
problem seems to be expanding slowly.
On 10/11/2020 23:46, Rafael Lopez wrote:
> Hi Janek,
>
> What you said sounds right - an S3 single part obj won't have an S3
> multipart string as part of the prefix. S3 multipart string looks
> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme".
>
> From memory, single part S3 objects that don't fit in a single rados
> object are assigned a random prefix that has nothing to do with
> the object name, and the rados tail/data objects (not the head
> object) have that prefix.
> As per your working example, the prefix for that would be
> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239)
"shadow"
> objects with names containing that prefix, and if you add up the
> sizes it should be the size of your S3 object.
>
> You should look at working and non working examples of both single
> and multipart S3 objects, as they are probably all a bit different
> when you look in rados.
>
> I agree it is a serious issue, because once objects are no longer in
> rados, they cannot be recovered. If it was a case that there was a
> link broken or rados objects renamed, then we could work to
> recover...but as far as I can tell, it looks like stuff is just
> vanishing from rados. The only explanation I can think of is some
> (rgw or rados) background process is incorrectly doing something with
> these objects (eg. renaming/deleting). I had thought perhaps it was a
> bug with the rgw garbage collector..but that is pure speculation.
>
> Once you can articulate the problem, I'd recommend logging a bug
> tracker upstream.
>
>
> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff
> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>
> <mailto:janek.bevendorff@uni-weimar.de>> wrote:
>
> Here's something else I noticed: when I stat objects that work
> via radosgw-admin, the stat info contains a "begin_iter" JSON
> object with RADOS key info like this
>
>
> "key": {
> "name":
>
"29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz",
> "instance": "",
> "ns": ""
> }
>
>
> and then "end_iter" with key info like this:
>
>
> "key": {
> "name":
> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239",
> "instance": "",
> "ns": "shadow"
> }
>
> However, when I check the broken 0-byte object, the "begin_iter"
> and "end_iter" keys look like this:
>
>
> "key": {
> "name":
>
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1",
> "instance": "",
> "ns": "multipart"
> }
>
> [...]
>
>
> "key": {
> "name":
>
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19",
> "instance": "",
> "ns": "multipart"
> }
>
> So, it's the full name plus a suffix and the namespace is
> multipart, not shadow (or empty). This in itself may just be an
> artefact of whether the object was uploaded in one go or as a
> multipart object, but the second difference is that I cannot find
> any of the multipart objects in my pool's object name dump. I
> can, however, find the shadow RADOS object of the intact S3 object.
>
>
>
>
> --
> *Rafael Lopez*
> Devops Systems Engineer
> Monash University eResearch Centre
>
> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118>
> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>
> <mailto:rafael.lopez@monash.edu>
>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
7:14 p.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
Thank you for the answers to those questions, Janek.
And in case anyone hasn’t seen it, we do have a tracker for this issue:
https://tracker.ceph.com/issues/47866
We may want to move most of the conversation to the comments there, so everything’s
together.
I do want to follow up on your answer to Question 4, Janek:
On Nov 13, 2020, at 12:22 PM, Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de> wrote:
With multipart uploads I believe this is a sufficient test, as the first bit of
data is in the first tail object, and it’s tail objects that seem to be disappearing.
However if the object is not uploaded via multipart and if it does have tail (_shadow_)
objects, then the initial data is stored in the head object. So this test would not be
truly diagnostic. This could be done with a large object, for example, with `s3cmd put
--disable-multipart …`.
Eric
--
J. Eric Ivancich
he / him / his
Red Hat Storage
Ann Arbor, Michigan, USA
4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'?
That would allow us to see a request from an RGW to either remove a tail object or
decrement its reference counter (and when its counter reaches 0 it will be deleted).
I haven't had any new data loss in the last few days (at least I think so, I
read 1byte from all objects, but didn't compare checksums, so I cannot say if all
objects are complete, but at least all are there).
16 Nov
16 Nov
8:48 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
As noted in the bug report, the issue has affected only multipart
objects at this time. I have added some more remarks there.
And yes, multipart objects tend to have 0 byte head objects in general.
The affected objects are simply missing all shadow objects, leaving us
with nothing but the empty head object and a few metadata.
On 13/11/2020 20:14, Eric Ivancich wrote:
Thank you for the answers to those questions, Janek.
And in case anyone hasn’t seen it, we do have a tracker for this issue:
https://tracker.ceph.com/issues/47866
We may want to move most of the conversation to the comments there, so
everything’s together.
I do want to follow up on your answer to Question 4, Janek:
On Nov 13, 2020, at 12:22 PM, Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de>> wrote:
With multipart uploads I believe this is a sufficient test, as the
first bit of data is in the first tail object, and it’s tail objects
that seem to be disappearing.
However if the object is not uploaded via multipart and if it does
have tail (_shadow_) objects, then the initial data is stored in the
head object. So this test would not be truly diagnostic. This could be
done with a large object, for example, with `s3cmd put
--disable-multipart …`.
Eric
--
J. Eric Ivancich
he / him / his
Red Hat Storage
Ann Arbor, Michigan, USA
4. Is anyone experiencing this issue willing to run their RGWs with
'debug_ms=1'? That would allow us to see a request from an RGW to
either remove a tail object or decrement its reference counter (and
when its counter reaches 0 it will be deleted).
I haven't had any new data loss in the last few days (at least I
think so, I read 1byte from all objects, but didn't compare
checksums, so I cannot say if all objects are complete, but at least
all are there).
17 Nov
17 Nov
1:22 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
I’m wondering if anyone experiencing this bug would mind running `radosgw-admin gc list
--include-all` on a schedule and saving the results. I’d like to know whether these tail
objects are getting removed by the gc process. If we find that that’s the case then
there’s the issue of how they got on the gc list.
Eric
On Nov 16, 2020, at 3:48 AM, Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de> wrote:
As noted in the bug report, the issue has affected only multipart objects at this time. I
have added some more remarks there.
And yes, multipart objects tend to have 0 byte head objects in general. The affected
objects are simply missing all shadow objects, leaving us with nothing but the empty head
object and a few metadata.
On 13/11/2020 20:14, Eric Ivancich wrote:
> Thank you for the answers to those questions, Janek.
>
> And in case anyone hasn’t seen it, we do have a tracker for this issue:
>
> https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866>
>
> We may want to move most of the conversation to the comments there, so everything’s
together.
>
> I do want to follow up on your answer to Question 4, Janek:
>
>> On Nov 13, 2020, at 12:22 PM, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de>> wrote:
>>>
>>> 4. Is anyone experiencing this issue willing to run their RGWs with
'debug_ms=1'? That would allow us to see a request from an RGW to either remove a
tail object or decrement its reference counter (and when its counter reaches 0 it will be
deleted).
>> I haven't had any new data loss in the last few days (at least I think so, I
read 1byte from all objects, but didn't compare checksums, so I cannot say if all
objects are complete, but at least all are there).
>>
> With multipart uploads I believe this is a sufficient test, as the first bit of data
is in the first tail object, and it’s tail objects that seem to be disappearing.
>
> However if the object is not uploaded via multipart and if it does have tail
(_shadow_) objects, then the initial data is stored in the head object. So this test would
not be truly diagnostic. This could be done with a large object, for example, with `s3cmd
put --disable-multipart …`.
>
> Eric
>
> --
> J. Eric Ivancich
> he / him / his
> Red Hat Storage
> Ann Arbor, Michigan, USA
8:22 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
I have run radosgw-admin gc list (without --include-all) a few times
already, but the list was always empty. I will create a cron job running
it every few minutes and writing out the results.
On 17/11/2020 02:22, Eric Ivancich wrote:
I’m wondering if anyone experiencing this bug would
mind running
`radosgw-admin gc list --include-all` on a schedule and saving the
results. I’d like to know whether these tail objects are getting
removed by the gc process. If we find that that’s the case then
there’s the issue of how they got on the gc list.
Eric
On Nov 16, 2020, at 3:48 AM, Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de>> wrote:
As noted in the bug report, the issue has affected only multipart
objects at this time. I have added some more remarks there.
And yes, multipart objects tend to have 0 byte head objects in
general. The affected objects are simply missing all shadow objects,
leaving us with nothing but the empty head object and a few metadata.
On 13/11/2020 20:14, Eric Ivancich wrote:
> Thank you for the answers to those questions, Janek.
>
> And in case anyone hasn’t seen it, we do have a tracker for this issue:
>
> https://tracker.ceph.com/issues/47866
>
> We may want to move most of the conversation to the comments there,
> so everything’s together.
>
> I do want to follow up on your answer to Question 4, Janek:
>
>> On Nov 13, 2020, at 12:22 PM, Janek Bevendorff
>> <janek.bevendorff(a)uni-weimar.de
>> <mailto:janek.bevendorff@uni-weimar.de>> wrote:
>>>
>>> 4. Is anyone experiencing this issue willing to run their RGWs
>>> with 'debug_ms=1'? That would allow us to see a request from an
>>> RGW to either remove a tail object or decrement its reference
>>> counter (and when its counter reaches 0 it will be deleted).
>>
>> I haven't had any new data loss in the last few days (at least I
>> think so, I read 1byte from all objects, but didn't compare
>> checksums, so I cannot say if all objects are complete, but at
>> least all are there).
>>
> With multipart uploads I believe this is a sufficient test, as the
> first bit of data is in the first tail object, and it’s tail objects
> that seem to be disappearing.
>
> However if the object is not uploaded via multipart and if it does
> have tail (_shadow_) objects, then the initial data is stored in the
> head object. So this test would not be truly diagnostic. This could
> be done with a large object, for example, with `s3cmd put
> --disable-multipart …`.
>
> Eric
>
> --
> J. Eric Ivancich
> he / him / his
> Red Hat Storage
> Ann Arbor, Michigan, USA
18 Nov
18 Nov
8:20 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
FYI: I have radosgw-admin gc list --include-all running every three
minutes for a day, but the list has stayed empty. Though I haven't seen
any further data loss, either. I will keep it running until the next
time I seen an object vanish.
On 17/11/2020 09:22, Janek Bevendorff wrote:
I have run radosgw-admin gc list (without --include-all) a few times
already, but the list was always empty. I will create a cron job
running it every few minutes and writing out the results.
On 17/11/2020 02:22, Eric Ivancich wrote:
> I’m wondering if anyone experiencing this bug would mind running
> `radosgw-admin gc list --include-all` on a schedule and saving the
> results. I’d like to know whether these tail objects are getting
> removed by the gc process. If we find that that’s the case then
> there’s the issue of how they got on the gc list.
>
> Eric
>
>
>> On Nov 16, 2020, at 3:48 AM, Janek Bevendorff
>> <janek.bevendorff(a)uni-weimar.de
>> <mailto:janek.bevendorff@uni-weimar.de>> wrote:
>>
>> As noted in the bug report, the issue has affected only multipart
>> objects at this time. I have added some more remarks there.
>>
>> And yes, multipart objects tend to have 0 byte head objects in
>> general. The affected objects are simply missing all shadow objects,
>> leaving us with nothing but the empty head object and a few metadata.
>>
>>
>> On 13/11/2020 20:14, Eric Ivancich wrote:
>>> Thank you for the answers to those questions, Janek.
>>>
>>> And in case anyone hasn’t seen it, we do have a tracker for this issue:
>>>
>>> https://tracker.ceph.com/issues/47866
>>>
>>> We may want to move most of the conversation to the comments there,
>>> so everything’s together.
>>>
>>> I do want to follow up on your answer to Question 4, Janek:
>>>
>>>> On Nov 13, 2020, at 12:22 PM, Janek Bevendorff
>>>> <janek.bevendorff(a)uni-weimar.de
>>>> <mailto:janek.bevendorff@uni-weimar.de>> wrote:
>>>>>
>>>>> 4. Is anyone experiencing this issue willing to run their RGWs
>>>>> with 'debug_ms=1'? That would allow us to see a request from
an
>>>>> RGW to either remove a tail object or decrement its reference
>>>>> counter (and when its counter reaches 0 it will be deleted).
>>>>
>>>> I haven't had any new data loss in the last few days (at least I
>>>> think so, I read 1byte from all objects, but didn't compare
>>>> checksums, so I cannot say if all objects are complete, but at
>>>> least all are there).
>>>>
>>> With multipart uploads I believe this is a sufficient test, as the
>>> first bit of data is in the first tail object, and it’s tail
>>> objects that seem to be disappearing.
>>>
>>> However if the object is not uploaded via multipart and if it does
>>> have tail (_shadow_) objects, then the initial data is stored in
>>> the head object. So this test would not be truly diagnostic. This
>>> could be done with a large object, for example, with `s3cmd put
>>> --disable-multipart …`.
>>>
>>> Eric
>>>
>>> --
>>> J. Eric Ivancich
>>> he / him / his
>>> Red Hat Storage
>>> Ann Arbor, Michigan, USA
>
8:55 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
We’ve hit this issue as well last night. If it wasn’t for a backup of the file that
vanished, it would have been lost.
On 13 Nov 2020, at 18:14, Eric Ivancich
<ivancich(a)redhat.com> wrote:
I have some questions for those who’ve experienced this issue.
1. It seems like those reporting this issue are seeing it strictly after upgrading to
Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus?
Mimic? Luminous?
We were running Mimic, then upgraded to Nautilus then Octopus in short succession.
2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so,
please describe.
The bucket where we hit this yesterday doesn’t have lifecycle rules, nor object
versioning.
3. Is anyone making copies of the affected objects (to same or to a different bucket)
prior to seeing the issue? And if they are making copies, does the destination bucket have
lifecycle rules? And if they are making copies, are those copies ever being removed?
The object, a Docker layer, that went missing has not been touched in 2 months. It worked
for a while, but then suddenly went missing.
Thanks,
Eric
On Nov 12, 2020, at 4:54 PM,
huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> wrote:
Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly
identified and fixed.
best regards,
Samuel
huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>
<mailto:huxiaoyu@horebdata.cn <mailto:huxiaoyu@horebdata.cn>>
From: Janek Bevendorff
Date: 2020-11-12 18:17
To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>
<mailto:huxiaoyu@horebdata.cn <mailto:huxiaoyu@horebdata.cn>>; EDH - Manuel
Rios; Rafael Lopez
CC: Robin H. Johnson; ceph-users
Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk
I have never seen this on Luminous. I recently upgraded to Octopus and the issue started
occurring only few weeks later.
On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote:
which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the
latest?
any idea?
samuel
huxiaoyu(a)horebdata.cn
From: EDH - Manuel Rios
Date: 2020-11-12 14:27
To: Janek Bevendorff; Rafael Lopez
CC: Robin H. Johnson; ceph-users
Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk
This same error caused us to wipe a full cluster of 300TB... will be related to some
rados index/database bug not to s3.
As Janek exposed is a mayor issue, because the error silent happend and you can only
detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey.
Error is not related to S3 logic ..
Hope this time dev's can take enought time to find and resolve the issue. Error
happens with low ec profiles, even with replica x3 in some cases.
Regards
-----Mensaje original-----
De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de> <mailto:janek.bevendorff@uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de>>>
Enviado el: jueves, 12 de noviembre de 2020 14:06
Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>
<mailto:rafael.lopez@monash.edu <mailto:rafael.lopez@monash.edu>>>
CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>
<mailto:robbat2@gentoo.org <mailto:robbat2@gentoo.org>>>; ceph-users
<ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>>
Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk
Here is a bug report concerning (probably) this exact issue:
https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866>
<https://tracker.ceph.com/issues/47866
<https://tracker.ceph.com/issues/47866>>
I left a comment describing the situation and my (limited) experiences with it.
On 11/11/2020 10:04, Janek Bevendorff wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
Yeah, that seems to be it. There are 239 objects prefixed
.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none
of the multiparts from the other file to be found and the head object
is 0 bytes.
I checked another multipart object with an end pointer of 11.
Surprisingly, it had way more than 11 parts (39 to be precise) named
.1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I
could find them in the dump at least.
I have no idea why the objects disappeared. I ran a Spark job over all
buckets, read 1 byte of every object and recorded errors. Of the 78
buckets, two are missing objects. One bucket is missing one object,
the other 15. So, luckily, the incidence is still quite low, but the
problem seems to be expanding slowly.
On 10/11/2020 23:46, Rafael Lopez wrote:
> Hi Janek,
>
> What you said sounds right - an S3 single part obj won't have an S3
> multipart string as part of the prefix. S3 multipart string looks
> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme".
>
> From memory, single part S3 objects that don't fit in a single rados
> object are assigned a random prefix that has nothing to do with
> the object name, and the rados tail/data objects (not the head
> object) have that prefix.
> As per your working example, the prefix for that would be
> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239)
"shadow"
> objects with names containing that prefix, and if you add up the
> sizes it should be the size of your S3 object.
>
> You should look at working and non working examples of both single
> and multipart S3 objects, as they are probably all a bit different
> when you look in rados.
>
> I agree it is a serious issue, because once objects are no longer in
> rados, they cannot be recovered. If it was a case that there was a
> link broken or rados objects renamed, then we could work to
> recover...but as far as I can tell, it looks like stuff is just
> vanishing from rados. The only explanation I can think of is some
> (rgw or rados) background process is incorrectly doing something with
> these objects (eg. renaming/deleting). I had thought perhaps it was a
> bug with the rgw garbage collector..but that is pure speculation.
>
> Once you can articulate the problem, I'd recommend logging a bug
> tracker upstream.
>
>
> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff
> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>
<mailto:janek.bevendorff@uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de>>
> <mailto:janek.bevendorff@uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de> <mailto:janek.bevendorff@uni-weimar.de
<mailto:janek.bevendorff@uni-weimar.de>>>> wrote:
>
> Here's something else I noticed: when I stat objects that work
> via radosgw-admin, the stat info contains a "begin_iter" JSON
> object with RADOS key info like this
>
>
> "key": {
> "name":
>
"29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz",
> "instance": "",
> "ns": ""
> }
>
>
> and then "end_iter" with key info like this:
>
>
> "key": {
> "name":
> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239",
> "instance": "",
> "ns": "shadow"
> }
>
> However, when I check the broken 0-byte object, the "begin_iter"
> and "end_iter" keys look like this:
>
>
> "key": {
> "name":
>
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1",
> "instance": "",
> "ns": "multipart"
> }
>
> [...]
>
>
> "key": {
> "name":
>
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19",
> "instance": "",
> "ns": "multipart"
> }
>
> So, it's the full name plus a suffix and the namespace is
> multipart, not shadow (or empty). This in itself may just be an
> artefact of whether the object was uploaded in one go or as a
> multipart object, but the second difference is that I cannot find
> any of the multipart objects in my pool's object name dump. I
> can, however, find the shadow RADOS object of the intact S3 object.
>
>
>
>
> --
> *Rafael Lopez*
> Devops Systems Engineer
> Monash University eResearch Centre
>
> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118>
> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>
>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:58 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
The object, a Docker layer, that went missing has not been touched in
2 months. It worked for a while, but then suddenly went missing.
Was the object a
multipart object? You can check by running
radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should
say something "ns": "multipart" in the output. If it says
"ns":
"shadow", it's a single-part object.
9:01 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME
--object=OBJECTNAME (forgot the "object" there)
On 18/11/2020 09:58, Janek Bevendorff wrote:
>>
>> The object, a Docker layer, that went missing has not been touched in
>> 2 months. It worked for a while, but then suddenly went missing.
> Was the object a multipart object? You can check by running
> radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should
> say something "ns": "multipart" in the output. If it says
"ns":
> "shadow", it's a single-part object.
9:06 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
It looks like a single-part object. But we did replace that object last night from backup,
so I can’t know for sure if the lost one was like that.
Another engineer that looked at the Rados objects last night did notice two things:
- The head object had a size of 0.
- There was an object with a ’shadow’ in its name, belonging to that path.
I’m not knowledgable about Rados, so I’m not sure this is helpful.
On 18 Nov 2020, at 10:01, Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de> wrote:
Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot
the "object" there)
On 18/11/2020 09:58, Janek Bevendorff wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
The object, a Docker layer, that went missing has not been touched in 2 months. It worked
for a while, but then suddenly went missing.
Was the object a multipart object? You
can check by running radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should
say something "ns": "multipart" in the output. If it says
"ns": "shadow", it's a single-part object.
19 Nov
19 Nov
8:09 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
- The head object had a size of 0.
- There was an object with a ’shadow’ in its name, belonging to that path.
That is
normal. What is not normal is if there are NO shadow objects.
On 18/11/2020 10:06, Denis Krienbühl wrote:
> It looks like a single-part object. But we did replace that object last night from
backup, so I can’t know for sure if the lost one was like that.
>
> Another engineer that looked at the Rados objects last night did notice two things:
>
- The head object had a size of 0.
- There was an object with a ’shadow’ in its name, belonging to that path.
>
> I’m not knowledgable about Rados, so I’m not sure this is helpful.
>
>> On 18 Nov 2020, at 10:01, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de>
wrote:
>>
>> Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME
(forgot the "object" there)
>>
>> On 18/11/2020 09:58, Janek Bevendorff wrote:
>>>> The object, a Docker layer, that went missing has not been touched in 2
months. It worked for a while, but then suddenly went missing.
>>> Was the object a multipart object? You can check by running radosgw-admin
stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something "ns":
"multipart" in the output. If it says "ns": "shadow",
it's a single-part object.
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:12 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
Thanks, we are currently scanning our object storage. It looks like we can detect the
missing objects that return “No Such Key” looking at all “__multipart_” objects returned
by radosgw-admin bucket radoslist, and checking if they exist using rados stat. We are
currently not looking at shadow objects as our approach already yields more instances of
this problem.
On 19 Nov 2020, at 09:09, Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de> wrote:
- The head object had a size of 0.
- There was an object with a ’shadow’ in its name, belonging to that path.
That is
normal. What is not normal is if there are NO shadow objects.
On 18/11/2020 10:06, Denis Krienbühl wrote:
> It looks like a single-part object. But we did replace that object last night from
backup, so I can’t know for sure if the lost one was like that.
>
> Another engineer that looked at the Rados objects last night did notice two things:
>
- The head object had a size of 0.
- There was an object with a ’shadow’ in its name, belonging to that path.
>
> I’m not knowledgable about Rados, so I’m not sure this is helpful.
>
>> On 18 Nov 2020, at 10:01, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de>
wrote:
>>
>> Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME
(forgot the "object" there)
>>
>> On 18/11/2020 09:58, Janek Bevendorff wrote:
>>>> The object, a Docker layer, that went missing has not been touched in 2
months. It worked for a while, but then suddenly went missing.
>>> Was the object a multipart object? You can check by running radosgw-admin
stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something "ns":
"multipart" in the output. If it says "ns": "shadow",
it's a single-part object.
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:21 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
I would recommend you get a dump with rados ls -p poolname (can be
several GB, mine is 61GB) and grep (or ack, which is faster) for the
names there to get an overview of what is there and what isn't. Looking
up the names directly can easily give you the wrong picture, because it
is kinda complicated to derive the correct RADOS name from an S3 object
name and a single typo will give you a not-found error, even if the
object is there.
On 19/11/2020 09:12, Denis Krienbühl wrote:
> Thanks, we are currently scanning our object storage. It looks like we can detect the
missing objects that return “No Such Key” looking at all “__multipart_” objects returned
by radosgw-admin bucket radoslist, and checking if they exist using rados stat. We are
currently not looking at shadow objects as our approach already yields more instances of
this problem.
>
>> On 19 Nov 2020, at 09:09, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de>
wrote:
>>
>>> - The head object had a size of 0.
>>> - There was an object with a ’shadow’ in its name, belonging to that path.
>> That is normal. What is not normal is if there are NO shadow objects.
>>
>> On 18/11/2020 10:06, Denis Krienbühl wrote:
>>> It looks like a single-part object. But we did replace that object last night
from backup, so I can’t know for sure if the lost one was like that.
>>>
>>> Another engineer that looked at the Rados objects last night did notice two
things:
>>>
>>> - The head object had a size of 0.
>>> - There was an object with a ’shadow’ in its name, belonging to that path.
>>>
>>> I’m not knowledgable about Rados, so I’m not sure this is helpful.
>>>
>>>> On 18 Nov 2020, at 10:01, Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de> wrote:
>>>>
>>>> Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME
--object=OBJECTNAME (forgot the "object" there)
>>>>
>>>> On 18/11/2020 09:58, Janek Bevendorff wrote:
>>>>>> The object, a Docker layer, that went missing has not been
touched in 2 months. It worked for a while, but then suddenly went missing.
>>>>> Was the object a multipart object? You can check by running
radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something
"ns": "multipart" in the output. If it says "ns":
"shadow", it's a single-part object.
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:39 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
We are doing that as well. But we need to be able to
check specific buckets additionally. For that we use this second approach.
Since we double-check all output from our script anyway (to see if NoSuchKey actually
happens), we can rule out false positives.
So far all the files detected this way actually have the issue (they show up in the list
of S3 Objects, but return a 404 on GET).
For our purposes, I wrote a Python script
that enumerates all objects:
s3 = boto3.resource('s3', endpoint_url=endpoint,
aws_access_key_id=access_key, aws_secret_access_key=secret_key)
file_list = []
for f in tqdm(s3.Bucket(bucket).objects desc='Listing objects...',
unit=' obj'):
file_list.append(f.key)
And then runs a Spark job that reads 1 byte from every object, recording
any botocore.exceptions.ClientError:
def test_object_absent(s3, obj_name):
try:
s3.Object(bucket, obj_name).get()['Body'].read(1)
return False
except ClientError:
return True
With 600 executors on 130 hosts, it takes about 30 seconds for a 300k
object bucket.
On 19/11/2020 09:21, Janek Bevendorff wrote:
> I would recommend you get a dump with rados ls -p poolname (can be
> several GB, mine is 61GB) and grep (or ack, which is faster) for the
> names there to get an overview of what is there and what isn't.
> Looking up the names directly can easily give you the wrong picture,
> because it is kinda complicated to derive the correct RADOS name from
> an S3 object name and a single typo will give you a not-found error,
> even if the object is there.
>
> On 19/11/2020 09:12, Denis Krienbühl wrote:
>> Thanks, we are currently scanning our object storage. It looks like
>> we can detect the missing objects that return “No Such Key” looking
>> at all “__multipart_” objects returned by radosgw-admin bucket
>> radoslist, and checking if they exist using rados stat. We are
>> currently not looking at shadow objects as our approach already
>> yields more instances of this problem.
>>
>>> On 19 Nov 2020, at 09:09, Janek Bevendorff
>>> <janek.bevendorff(a)uni-weimar.de> wrote:
>>>
>>>> - The head object had a size of 0.
>>>> - There was an object with a ’shadow’ in its name, belonging to
>>>> that path.
>>> That is normal. What is not normal is if there are NO shadow objects.
>>>
>>> On 18/11/2020 10:06, Denis Krienbühl wrote:
>>>> It looks like a single-part object. But we did replace that object
>>>> last night from backup, so I can’t know for sure if the lost one
>>>> was like that.
>>>>
>>>> Another engineer that looked at the Rados objects last night did
>>>> notice two things:
>>>>
>>>> - The head object had a size of 0.
>>>> - There was an object with a ’shadow’ in its name, belonging to
>>>> that path.
>>>>
>>>> I’m not knowledgable about Rados, so I’m not sure this is helpful.
>>>>
>>>>> On 18 Nov 2020, at 10:01, Janek Bevendorff
>>>>> <janek.bevendorff(a)uni-weimar.de> wrote:
>>>>>
>>>>> Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME
>>>>> --object=OBJECTNAME (forgot the "object" there)
>>>>>
>>>>> On 18/11/2020 09:58, Janek Bevendorff wrote:
>>>>>>> The object, a Docker layer, that went missing has not been
>>>>>>> touched in 2 months. It worked for a while, but then suddenly
>>>>>>> went missing.
>>>>>> Was the object a multipart object? You can check by running
>>>>>> radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It
>>>>>> should say something "ns": "multipart" in the
output. If it says
>>>>>> "ns": "shadow", it's a single-part
object.
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
18 Nov
18 Nov
9:46 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
I can now confirm that last night’s missing object was a multi-part file.
On 18 Nov 2020, at 10:01, Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de> wrote:
Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot
the "object" there)
On 18/11/2020 09:58, Janek Bevendorff wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
The object, a Docker layer, that went missing has not been touched in 2 months. It worked
for a while, but then suddenly went missing.
Was the object a multipart object? You
can check by running radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should
say something "ns": "multipart" in the output. If it says
"ns": "shadow", it's a single-part object.
10:27 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
By the way, since there’s some probability that this is a GC refcount issue, would it be
possible and sane to somehow slow the GC down or disable it altogether? Is that something
we could implement on our end as a stop-gap measure to prevent dataloss?
On 18 Nov 2020, at 10:46, Denis Krienbühl
<denis(a)href.ch> wrote:
I can now confirm that last night’s missing object was a multi-part file.
On 18 Nov 2020, at 10:01, Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de> wrote:
Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot
the "object" there)
On 18/11/2020 09:58, Janek Bevendorff wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
The object, a Docker layer, that went missing has not been touched in 2 months. It worked
for a while, but then suddenly went missing.
Was the object a multipart object? You
can check by running radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should
say something "ns": "multipart" in the output. If it says
"ns": "shadow", it's a single-part object. 
23 Nov
23 Nov
10:56 a.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
Hi Denis,
You might want to look at rgw_gc_obj_min_wait from [1] and try
increasing the default value of 7200s (2 hours) to whatever suits your
need < 2^64.
Just remind that at some point you'll have to get these objects
processed by the gc. Or manually through the API [2].
One thing that comes to mind regarding the "last night's missing object"
is maybe it was multi-part re-written and the re-write failed somehow
and the object was then enlisted by the gc. But that supposes this
particular object sometimes gets re-written which may not be the case.
Regards,
Frédéric.
[1] https://docs.ceph.com/en/latest/radosgw/config-ref/
[2]
https://docs.ceph.com/en/latest/dev/radosgw/admin/adminops_nonimplemented/#…
Le 18/11/2020 à 11:27, Denis Krienbühl a écrit :
By the way, since there’s some probability that this
is a GC refcount issue, would it be possible and sane to somehow slow the GC down or
disable it altogether? Is that something we could implement on our end as a stop-gap
measure to prevent dataloss?
On 18 Nov 2020, at 10:46, Denis Krienbühl
<denis(a)href.ch> wrote:
I can now confirm that last night’s missing object was a multi-part file.
> On 18 Nov 2020, at 10:01, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de>
wrote:
>
> Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME
(forgot the "object" there)
>
> On 18/11/2020 09:58, Janek Bevendorff wrote:
>>> The object, a Docker layer, that went missing has not been touched in 2
months. It worked for a while, but then suddenly went missing.
>> Was the object a multipart object? You can check by running radosgw-admin stat
--bucket=BUCKETNAME --object=OBJECTNAME. It should say something "ns":
"multipart" in the output. If it says "ns": "shadow",
it's a single-part object.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:24 p.m.
New subject: NoSuchKey on key that is visible in s3 list/radosgw bk
Thanks Frédéric, we’ve done that in the meantime to work around issue #47866.
The error has been reproduced and there’s a PR associated with the issue:
https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866>
Cheers,
Denis
On 23 Nov 2020, at 11:56, Frédéric Nass
<frederic.nass(a)univ-lorraine.fr> wrote:
Hi Denis,
You might want to look at rgw_gc_obj_min_wait from [1] and try increasing the default
value of 7200s (2 hours) to whatever suits your need < 2^64.
Just remind that at some point you'll have to get these objects processed by the gc.
Or manually through the API [2].
One thing that comes to mind regarding the "last night's missing object" is
maybe it was multi-part re-written and the re-write failed somehow and the object was then
enlisted by the gc. But that supposes this particular object sometimes gets re-written
which may not be the case.
Regards,
Frédéric.
[1] https://docs.ceph.com/en/latest/radosgw/config-ref/
[2]
https://docs.ceph.com/en/latest/dev/radosgw/admin/adminops_nonimplemented/#…
Le 18/11/2020 à 11:27, Denis Krienbühl a écrit :
By the way, since there’s some probability that
this is a GC refcount issue, would it be possible and sane to somehow slow the GC down or
disable it altogether? Is that something we could implement on our end as a stop-gap
measure to prevent dataloss?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io On 18 Nov 2020, at 10:46, Denis Krienbühl
<denis(a)href.ch> wrote:
I can now confirm that last night’s missing object was a multi-part file.
> On 18 Nov 2020, at 10:01, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de>
wrote:
>
> Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME
(forgot the "object" there)
>
> On 18/11/2020 09:58, Janek Bevendorff wrote:
>>> The object, a Docker layer, that went missing has not been touched in 2
months. It worked for a while, but then suddenly went missing.
>> Was the object a multipart object? You can check by running radosgw-admin stat
--bucket=BUCKETNAME --object=OBJECTNAME. It should say something "ns":
"multipart" in the output. If it says "ns": "shadow",
it's a single-part object.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
24 Nov
24 Nov
9 p.m.
New subject: RGW Data Loss Bug in Octopus 15.2.0 through 15.2.6
Starting in stable release Octopus 15.2.0 and continuing through Octopus 15.2.6 there is a
bug in RGW that could result in data loss. There is both an immediate configuration
work-around and a fix is intended for Octopus 15.2.7. [Note: the bug was first merged in a
pre-stable release — Octopus 15.1.0]
This bug is triggered when a read of a large RGW object (i.e., one with at least one tail
segment) takes longer than 1/2 (one half) the time specified in the configuration option
rgw_gc_obj_min_wait (2 hours by default, specified in seconds). The bug causes the tail
segments of that read object to be added to the RGW garbage collection queue, which will
in turn cause them to be deleted after a period of time.
The configuration work-around is to set rgw_gc_obj_min_wait to a large enough time
(specified in seconds) that will exceed twice the time of the longest read you expect. The
downside of this configuration change is that it will delay GC of deleted objects and will
tend to cause the GC queue to become longer. Given that there’s a finite amount of space
for that queue, if it ever becomes full then tail segments will be deleted in-line with
object removal operations, and that could degrade performance slightly.
The Octopus backport tracker is: https://tracker.ceph.com/issues/48331
The Octopus backport PR is: https://github.com/ceph/ceph/pull/38249
The master branch tracker, which has the history of the bug, is:
https://tracker.ceph.com/issues/47866
Tracking down this bug was a group effort and many people participated. See the master
branch tracker for that history. Thanks to everyone who helped out.
Eric
--
J. Eric Ivancich
he / him / his
Red Hat Storage
Ann Arbor, Michigan, USA
27 Mar
27 Mar
12:13 p.m.
New subject: RGW Data Loss Bug in Octopus 15.2.0 through 15.2.6
Hi, Eric Ivancich
I have similar problem in ceph version 16.2.5. Has this problem been completely
resolved in Pacific version？
Our bucket has no lifecycle rules and no copy operation. This is a very serious data loss
issue for us and It happens occasionally in our environment.
Detail describe:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/XQRUOEPZ7Y…
thanks.
28 Mar
28 Mar
3:11 p.m.
New subject: RGW Data Loss Bug in Octopus 15.2.0 through 15.2.6
Hello,
Maybe your issue depends to this https://tracker.ceph.com/issues/63642
On Wed, Mar 27, 2024 at 7:31 PM xu chenhui <xuchenhuig(a)gmail.com> wrote:
Hi, Eric Ivancich
I have similar problem in ceph version 16.2.5. Has this problem been
completely resolved in Pacific version？
Our bucket has no lifecycle rules and no copy operation. This is a very
serious data loss issue for us and It happens occasionally in our
environment.
Detail describe:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/XQRUOEPZ7Y…
thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jonas
3 Apr
3 Apr
2:52 a.m.
New subject: RGW Data Loss Bug in Octopus 15.2.0 through 15.2.6
Jonas Nemeiksis wrote:
Hello,
Maybe your issue depends to this https://tracker.ceph.com/issues/63642
On Wed, Mar 27, 2024 at 7:31 PM xu chenhui <xuchenhuig(a)gmail.com> wrote:
> Hi, Eric Ivancich
> I have similar problem in ceph version 16.2.5. Has this problem been
> completely resolved in Pacific version？
> Our bucket has no lifecycle rules and no copy operation. This is a very
> serious data loss issue for us and It happens occasionally in our
> environment.
>
> Detail describe:
>
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/XQRUOEPZ7Y…
>
> thanks.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
Hi,
My problem is different from https://tracker.ceph.com/issues/63642. All of
multiparts and shadow object had lost and only have head object in our environment. Maybe
this is new issue that happened in low probability and I haven't reproduce.
Is there other information that can help locate root cause or reduce data loss ？
thanks.
25
days inactive
1262
days old
22 comments
6 participants
participants (6)
-
Denis Krienbühl -
Eric Ivancich -
Frédéric Nass -
Janek Bevendorff -
Jonas Nemeiksis -
xu chenhui