RGW objects has same marker and bucket id in different buckets.
Hello. I have a rgw s3 user and the user have 2 bucket. I tried to copy objects from old.bucket to new.bucket with rclone. (in the rgw client server) After I checked the object with "radosgw-admin --bucket=new.bucket object stat $i" and I saw old.bucket id and marker id also old bucket name in the object stats. Is rgw doing this for deduplication or is it a bug? If it's not a bug then If I delete the old bucket what will happen to these objects???
Hi Morphin, Yes, this is by design. When an RGW object has tail chunks and is copied so as to duplicate an entire tail chunk, RGW causes the coincident chunk(s) to be shared. Tail chunks are refcounted to avoid leaks. Matt On Wed, Apr 21, 2021 at 4:21 PM by morphin <morphinwithyou@gmail.com> wrote:
Hello.
I have a rgw s3 user and the user have 2 bucket. I tried to copy objects from old.bucket to new.bucket with rclone. (in the rgw client server) After I checked the object with "radosgw-admin --bucket=new.bucket object stat $i" and I saw old.bucket id and marker id also old bucket name in the object stats.
Is rgw doing this for deduplication or is it a bug? If it's not a bug then If I delete the old bucket what will happen to these objects??? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Thanks for the answer. I have 2 question: 1- If I use different user and a bucket what will happen? Is this design only for same user or user independent? 2- If I delete the Source bucket with radosgw-admin or via S3 delete, what will happen to these objects? Also I have more questions if you have time :) Some objects are pending state due to zone-sync error. I've removed the master zone and set secondary to master. I still see the pending objects in the bucket. (below) radosgw-admin --id radosgw.srv1 object stat --bucket=descript --object=2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017 { "name": "2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017", "size": 0, "tag": "", "attrs": { "user.rgw.manifest": "", "user.rgw.olh.idtag": "ivlde1avu2l3lli6i349h62c0d79ao4u", "user.rgw.olh.info": "\u0001\u0001�", "user.rgw.olh.pending.00000000607d4d5be0hh3lpzjd7vzt2j": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607d4d5c9uhlh9sf93j8lf7l": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607d4d5cpip1i8z8rytcnkqf": "\u0001\u0001\u0008", "user.rgw.olh.ver": "3" } } I overwrite these objects with rclone from old zone bucket to new created bucket on the same user at master zone. After a while I noticed that I'm getting a warning for these objects in rgw client log and the overwritten objects switching back to the corrupted objects. 2021-04-22 10:27:55.445 7f2d85fd4700 0 WARNING: couldn't find acl header for object, generating default 2021-04-22 10:27:55.445 7f2d85fd4700 1 ====== req done req=0x55a441452710 op status=0 http_status=200 latency=0.0200002s ====== 2021-04-22 10:27:55.445 7f2d85fd4700 1 beast: 0x55a441452710: 10.151.101.15 - - [2021-04-22 10:27:55.0.44549s] "GET /descript/2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017 HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6 vendor/Oracle_Corporation" - Am I doing something wrong? Also "sync error trim" does not work. How can I clean these errors and these pending objects? ceph version 14.2.16 Have a great day. Regards. Matt Benjamin <mbenjami@redhat.com>, 22 Nis 2021 Per, 06:08 tarihinde şunu yazdı:
Hi Morphin,
Yes, this is by design. When an RGW object has tail chunks and is copied so as to duplicate an entire tail chunk, RGW causes the coincident chunk(s) to be shared. Tail chunks are refcounted to avoid leaks.
Matt
On Wed, Apr 21, 2021 at 4:21 PM by morphin <morphinwithyou@gmail.com> wrote:
Hello.
I have a rgw s3 user and the user have 2 bucket. I tried to copy objects from old.bucket to new.bucket with rclone. (in the rgw client server) After I checked the object with "radosgw-admin --bucket=new.bucket object stat $i" and I saw old.bucket id and marker id also old bucket name in the object stats.
Is rgw doing this for deduplication or is it a bug? If it's not a bug then If I delete the old bucket what will happen to these objects??? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Hi Morphin, On Thu, Apr 22, 2021 at 3:40 AM by morphin <morphinwithyou@gmail.com> wrote:
Thanks for the answer.
I have 2 question: 1- If I use different user and a bucket what will happen? Is this design only for same user or user independent?
It's user independent.
2- If I delete the Source bucket with radosgw-admin or via S3 delete, what will happen to these objects?
The refcount on each shared object will be decreased by 1. If there are no more references, the objects will be garbage collected.
Also I have more questions if you have time :)
I don't have the expertise in multisite replication to debug this. I don't think the issue is related to RGW's copy-sharing. Matt
Some objects are pending state due to zone-sync error. I've removed the master zone and set secondary to master. I still see the pending objects in the bucket. (below)
radosgw-admin --id radosgw.srv1 object stat --bucket=descript --object=2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017 { "name": "2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017", "size": 0, "tag": "", "attrs": { "user.rgw.manifest": "", "user.rgw.olh.idtag": "ivlde1avu2l3lli6i349h62c0d79ao4u", "user.rgw.olh.info": "\u0001\u0001�", "user.rgw.olh.pending.00000000607d4d5be0hh3lpzjd7vzt2j": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607d4d5c9uhlh9sf93j8lf7l": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607d4d5cpip1i8z8rytcnkqf": "\u0001\u0001\u0008", "user.rgw.olh.ver": "3" } }
I overwrite these objects with rclone from old zone bucket to new created bucket on the same user at master zone.
After a while I noticed that I'm getting a warning for these objects in rgw client log and the overwritten objects switching back to the corrupted objects.
2021-04-22 10:27:55.445 7f2d85fd4700 0 WARNING: couldn't find acl header for object, generating default 2021-04-22 10:27:55.445 7f2d85fd4700 1 ====== req done req=0x55a441452710 op status=0 http_status=200 latency=0.0200002s ====== 2021-04-22 10:27:55.445 7f2d85fd4700 1 beast: 0x55a441452710: 10.151.101.15 - - [2021-04-22 10:27:55.0.44549s] "GET /descript/2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017 HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6 vendor/Oracle_Corporation" -
Am I doing something wrong? Also "sync error trim" does not work. How can I clean these errors and these pending objects?
ceph version 14.2.16
Have a great day. Regards.
Matt Benjamin <mbenjami@redhat.com>, 22 Nis 2021 Per, 06:08 tarihinde şunu yazdı:
Hi Morphin,
Yes, this is by design. When an RGW object has tail chunks and is copied so as to duplicate an entire tail chunk, RGW causes the coincident chunk(s) to be shared. Tail chunks are refcounted to avoid leaks.
Matt
On Wed, Apr 21, 2021 at 4:21 PM by morphin <morphinwithyou@gmail.com> wrote:
Hello.
I have a rgw s3 user and the user have 2 bucket. I tried to copy objects from old.bucket to new.bucket with rclone. (in the rgw client server) After I checked the object with "radosgw-admin --bucket=new.bucket object stat $i" and I saw old.bucket id and marker id also old bucket name in the object stats.
Is rgw doing this for deduplication or is it a bug? If it's not a bug then If I delete the old bucket what will happen to these objects??? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Its crystal clear now. Thank you so much for the answers. The problem is I cant delete these pending objects with radosgw-admin object rm $obj. Also sync error trim does nothing. Due to this problem I've created a New bucket and copied these objects from the main source (out of cluster) and also copied other files from old.bucket because they was not on main source. Ive checked all the files after the copy and they was all ok. Somehow the copies started to change it self to these pending objects after s3 get operations. I think somehow rgw reads the file and make current obj to the sync errored obj. Only log is this: WARNING: couldn't find acl header for object, generating default I need to clear sync error logs and the pending objects. But can not do both. Even deleting rados object directly does not change the situation. When I try to copy after rados rm, the object turns into pending object again. Maybe deleting old periods will help. What do you think? Its very Hard to explain. I tried my best. Regards. 22 Nis 2021 Per 18:26 tarihinde Matt Benjamin <mbenjami@redhat.com> şunu yazdı:
Hi Morphin,
On Thu, Apr 22, 2021 at 3:40 AM by morphin <morphinwithyou@gmail.com> wrote:
Thanks for the answer.
I have 2 question: 1- If I use different user and a bucket what will happen? Is this design only for same user or user independent?
It's user independent.
2- If I delete the Source bucket with radosgw-admin or via S3 delete, what will happen to these objects?
The refcount on each shared object will be decreased by 1. If there are no more references, the objects will be garbage collected.
Also I have more questions if you have time :)
I don't have the expertise in multisite replication to debug this. I don't think the issue is related to RGW's copy-sharing.
Matt
Some objects are pending state due to zone-sync error. I've removed the master zone and set secondary to master. I still see the pending objects in the bucket. (below)
radosgw-admin --id radosgw.srv1 object stat --bucket=descript --object=2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017 { "name": "2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017", "size": 0, "tag": "", "attrs": { "user.rgw.manifest": "", "user.rgw.olh.idtag": "ivlde1avu2l3lli6i349h62c0d79ao4u", "user.rgw.olh.info": "\u0001\u0001�", "user.rgw.olh.pending.00000000607d4d5be0hh3lpzjd7vzt2j": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607d4d5c9uhlh9sf93j8lf7l": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607d4d5cpip1i8z8rytcnkqf": "\u0001\u0001\u0008", "user.rgw.olh.ver": "3" } }
I overwrite these objects with rclone from old zone bucket to new created bucket on the same user at master zone.
After a while I noticed that I'm getting a warning for these objects in rgw client log and the overwritten objects switching back to the corrupted objects.
2021-04-22 10:27:55.445 7f2d85fd4700 0 WARNING: couldn't find acl header for object, generating default 2021-04-22 10:27:55.445 7f2d85fd4700 1 ====== req done req=0x55a441452710 op status=0 http_status=200 latency=0.0200002s ====== 2021-04-22 10:27:55.445 7f2d85fd4700 1 beast: 0x55a441452710: 10.151.101.15 - - [2021-04-22 10:27:55.0.44549s] "GET /descript/2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017 HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6 vendor/Oracle_Corporation" -
Am I doing something wrong? Also "sync error trim" does not work. How can I clean these errors and these pending objects?
ceph version 14.2.16
Have a great day. Regards.
Matt Benjamin <mbenjami@redhat.com>, 22 Nis 2021 Per, 06:08 tarihinde şunu yazdı:
Hi Morphin,
Yes, this is by design. When an RGW object has tail chunks and is copied so as to duplicate an entire tail chunk, RGW causes the coincident chunk(s) to be shared. Tail chunks are refcounted to avoid leaks.
Matt
On Wed, Apr 21, 2021 at 4:21 PM by morphin <morphinwithyou@gmail.com>
wrote:
Hello.
I have a rgw s3 user and the user have 2 bucket. I tried to copy objects from old.bucket to new.bucket with rclone.
(in
the rgw client server) After I checked the object with "radosgw-admin --bucket=new.bucket object stat $i" and I saw old.bucket id and marker id also old bucket name in the object stats.
Is rgw doing this for deduplication or is it a bug? If it's not a bug then If I delete the old bucket what will happen to these objects??? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
participants (2)
-
by morphin
-
Matt Benjamin