radosgw - Etags suffixed with #x0e
Hi, We had a strange problem with some buckets. After a s3cmd sync, some objects got ETAGs with the suffix "#x0e". This rendered the XML output of "GET /" e.g. (s3cmd du) invalid. Unfortunately, this behaviour was not reproducable but could be fixed by "GET /{object}" + "PUT /{object}" (s3cmd get + s3cmd put). I am not sure, how this appeared and how to avoid that. Just now, we have nautilus mons and osds with jewel radosgws. At the time of first appearence, also a nautilus gateway had been online, but the requests had been handled by both types. Any ideas? best regards, Ingo -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001
Hi, i was able to track down the issue further: * Upload had been a multipart upload - Metadata are correct * After that, the object had been copied (PUT with x-amz-copy-source) - ETag got broken "s3cmd du" or "s3cmd ls" fails with the copied object: [...] DEBUG: Sending request method_string='GET', uri='/?prefix=XXXXXXXXXXX-ZZZ%2Fdata%2FL%2FKKKKK', headers={'x-amz-date': '20191219T133746Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=XXXXXXXXXXX/20191219/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[ommited]', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (http://appliance-12f587e9ea79f037733f378276712a39.dcs.dunkel.de#1) DEBUG: Response: {'data': b'<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http:' b'//s3.amazonaws.com/doc/2006-03-01/"><Name>[ommitedbucketname........' b'...............]</Name><Prefix>XXXXXXXXXXX-ZZZ/data/L/KKKKK</Prefix>' b'<Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTrunca' b'ted><Contents><Key>XXXXXXXXXXX-ZZZ/data/L/KKKKK</Key><LastModified>2' b'019-12-18T22:46:50.889Z</LastModified><ETag>"ae6992f728ff1763f6' b'b2425acf6811a5-19"</ETag><Size>296557676</Size><StorageCl' b'ass>STANDARD</StorageClass><Owner><ID>[ommited]</ID><DisplayName>[..' b'username.]</DisplayName></Owner><Type>Normal</Type></Contents></List' b'BucketResult>', 'headers': {'content-type': 'application/xml', 'date': 'Thu, 19 Dec 2019 13:37:46 GMT', 'transfer-encoding': 'chunked', 'x-amz-request-id': 'tx0000000000000000463e6-005dfb7d2a-1c798e3c-default'}, 'reason': 'OK', 'status': 200} ERROR: Error parsing xml: reference to invalid character number: line 1, column 377 Surprisingly, when i fetch metadata of both objects, i don`t see any pecularities: root@cephosd01:~# radosgw-admin object stat --bucket=[ommitted] --object="XXXXXXXXXX-YYY/data/L/KKKKK" { "name": "XXXXXXXXXX-YYY/data/L/KKKKK", [..] "etag": "ae6992f728ff1763f6b2425acf6811a5-19", "tag": "default.477728316.187337", "manifest": { "objs": [], "obj_size": 296557676, "explicit_objs": "false", "head_size": 0, "max_head_size": 0, "prefix": "XXXXXXXXXX-YYY/data/L/KKKKK.2~8sxiSL8gadZHI9VHRJiRcxo5SYMWGcV", broken: radosgw-admin object stat --bucket=appliance-12f587e9ea79f037733f378276712a39 --object="XXXXXXXXXXX-ZZZ/data/L/KKKKK" [..] "etag": "ae6992f728ff1763f6b2425acf6811a5-19", "tag": "default.477200610.3445398", "manifest": { "objs": [], "obj_size": 296557676, "explicit_objs": "false", "head_size": 0, "max_head_size": 0, "prefix": "XXXXXXXXXXX-ZZZ/data/L/KKKKK.2~8sxiSL8gadZHI9VHRJiRcxo5SYMWGcV", [..] root@cephosd01:~# rados -p rgw.buckets getxattr "default.375689600.58_XXXXXXXXXXX-YYY/data/L/KKKKK" user.rgw.etag > /tmp/etag.broken root@cephosd01:~# rados -p rgw.buckets getxattr "default.375689600.58_XXXXXXXXXXX-ZZZ/data/L/KKKKK" user.rgw.etag > /tmp/etag.ok root@cephosd01:~# diff /tmp/etag.* root@cephosd01:~# cat /tmp/etag.broken ae6992f728ff1763f6b2425acf6811a5-19root@cephosd01:~# I may read and write or even copy (s3cmd copy) the object again, and all is fine. Still not clear, at what point the issue appears. Any clues? kind regards, Ingo ----- Ursprüngliche Mail ----- Von: "Ingo Reimann" <ireimann@dunkel.de> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Mittwoch, 18. Dezember 2019 11:40:39 Betreff: [ceph-users] radosgw - Etags suffixed with #x0e Hi, We had a strange problem with some buckets. After a s3cmd sync, some objects got ETAGs with the suffix "#x0e". This rendered the XML output of "GET /" e.g. (s3cmd du) invalid. Unfortunately, this behaviour was not reproducable but could be fixed by "GET /{object}" + "PUT /{object}" (s3cmd get + s3cmd put). I am not sure, how this appeared and how to avoid that. Just now, we have nautilus mons and osds with jewel radosgws. At the time of first appearence, also a nautilus gateway had been online, but the requests had been handled by both types. Any ideas? best regards, Ingo -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Anyone? ----- Ursprüngliche Mail ----- Von: "Ingo Reimann" <ireimann@dunkel.de> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Donnerstag, 19. Dezember 2019 15:56:21 Betreff: [ceph-users] Re: radosgw - Etags suffixed with #x0e Hi, i was able to track down the issue further: * Upload had been a multipart upload - Metadata are correct * After that, the object had been copied (PUT with x-amz-copy-source) - ETag got broken "s3cmd du" or "s3cmd ls" fails with the copied object: [...] DEBUG: Sending request method_string='GET', uri='/?prefix=XXXXXXXXXXX-ZZZ%2Fdata%2FL%2FKKKKK', headers={'x-amz-date': '20191219T133746Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=XXXXXXXXXXX/20191219/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[ommited]', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (http://appliance-12f587e9ea79f037733f378276712a39.dcs.dunkel.de#1) DEBUG: Response: {'data': b'<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http:' b'//s3.amazonaws.com/doc/2006-03-01/"><Name>[ommitedbucketname........' b'...............]</Name><Prefix>XXXXXXXXXXX-ZZZ/data/L/KKKKK</Prefix>' b'<Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTrunca' b'ted><Contents><Key>XXXXXXXXXXX-ZZZ/data/L/KKKKK</Key><LastModified>2' b'019-12-18T22:46:50.889Z</LastModified><ETag>"ae6992f728ff1763f6' b'b2425acf6811a5-19"</ETag><Size>296557676</Size><StorageCl' b'ass>STANDARD</StorageClass><Owner><ID>[ommited]</ID><DisplayName>[..' b'username.]</DisplayName></Owner><Type>Normal</Type></Contents></List' b'BucketResult>', 'headers': {'content-type': 'application/xml', 'date': 'Thu, 19 Dec 2019 13:37:46 GMT', 'transfer-encoding': 'chunked', 'x-amz-request-id': 'tx0000000000000000463e6-005dfb7d2a-1c798e3c-default'}, 'reason': 'OK', 'status': 200} ERROR: Error parsing xml: reference to invalid character number: line 1, column 377 Surprisingly, when i fetch metadata of both objects, i don`t see any pecularities: root@cephosd01:~# radosgw-admin object stat --bucket=[ommitted] --object="XXXXXXXXXX-YYY/data/L/KKKKK" { "name": "XXXXXXXXXX-YYY/data/L/KKKKK", [..] "etag": "ae6992f728ff1763f6b2425acf6811a5-19", "tag": "default.477728316.187337", "manifest": { "objs": [], "obj_size": 296557676, "explicit_objs": "false", "head_size": 0, "max_head_size": 0, "prefix": "XXXXXXXXXX-YYY/data/L/KKKKK.2~8sxiSL8gadZHI9VHRJiRcxo5SYMWGcV", broken: radosgw-admin object stat --bucket=appliance-12f587e9ea79f037733f378276712a39 --object="XXXXXXXXXXX-ZZZ/data/L/KKKKK" [..] "etag": "ae6992f728ff1763f6b2425acf6811a5-19", "tag": "default.477200610.3445398", "manifest": { "objs": [], "obj_size": 296557676, "explicit_objs": "false", "head_size": 0, "max_head_size": 0, "prefix": "XXXXXXXXXXX-ZZZ/data/L/KKKKK.2~8sxiSL8gadZHI9VHRJiRcxo5SYMWGcV", [..] root@cephosd01:~# rados -p rgw.buckets getxattr "default.375689600.58_XXXXXXXXXXX-YYY/data/L/KKKKK" user.rgw.etag > /tmp/etag.broken root@cephosd01:~# rados -p rgw.buckets getxattr "default.375689600.58_XXXXXXXXXXX-ZZZ/data/L/KKKKK" user.rgw.etag > /tmp/etag.ok root@cephosd01:~# diff /tmp/etag.* root@cephosd01:~# cat /tmp/etag.broken ae6992f728ff1763f6b2425acf6811a5-19root@cephosd01:~# I may read and write or even copy (s3cmd copy) the object again, and all is fine. Still not clear, at what point the issue appears. Any clues? kind regards, Ingo ----- Ursprüngliche Mail ----- Von: "Ingo Reimann" <ireimann@dunkel.de> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Mittwoch, 18. Dezember 2019 11:40:39 Betreff: [ceph-users] radosgw - Etags suffixed with #x0e Hi, We had a strange problem with some buckets. After a s3cmd sync, some objects got ETAGs with the suffix "#x0e". This rendered the XML output of "GET /" e.g. (s3cmd du) invalid. Unfortunately, this behaviour was not reproducable but could be fixed by "GET /{object}" + "PUT /{object}" (s3cmd get + s3cmd put). I am not sure, how this appeared and how to avoid that. Just now, we have nautilus mons and osds with jewel radosgws. At the time of first appearence, also a nautilus gateway had been online, but the requests had been handled by both types. Any ideas? best regards, Ingo -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Dec 18, 2019 at 11:41 AM Ingo Reimann <ireimann@dunkel.de> wrote:
Hi,
We had a strange problem with some buckets. After a s3cmd sync, some objects got ETAGs with the suffix "#x0e". This rendered the XML output of "GET /" e.g. (s3cmd du) invalid. Unfortunately, this behaviour was not reproducable but could be fixed by "GET /{object}" + "PUT /{object}" (s3cmd get + s3cmd put).
I am not sure, how this appeared and how to avoid that. Just now, we have nautilus mons and osds with jewel radosgws.
Please don't mix versions like that. Running nautilus and jewel at the same time is unsupported. Upgrade everything and check if that solves your problem. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
At the time of first appearence, also a nautilus gateway had been online, but the requests had been handled by both types.
Any ideas?
best regards, Ingo
-- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Paul, thanks for your advise. I didn`t expect this consequences, as there is no hint in the release notes. On Monday, we will perform the overall upgrade to nautilus. Nevertheless, there seems to be a bug in radosgw: If an ETag contains characters below 0x20, they are coded as eg. . This renders the XML invalid. Meanwhile, I modified the code of radosgw to show, that the 0x0e really exists at the end of meta.etag and remove it in the output. That helps identifying the problematic objects, but does not solve the root cause. best regards, ingo Von: "Paul Emmerich" <paul.emmerich@croit.io> An: "Ingo Reimann" <ireimann@dunkel.de> CC: "ceph-users" <ceph-users@ceph.io> Gesendet: Donnerstag, 2. Januar 2020 21:26:03 Betreff: Re: [ceph-users] radosgw - Etags suffixed with #x0e On Wed, Dec 18, 2019 at 11:41 AM Ingo Reimann < [ mailto:ireimann@dunkel.de | ireimann@dunkel.de ] > wrote: Hi, We had a strange problem with some buckets. After a s3cmd sync, some objects got ETAGs with the suffix "#x0e". This rendered the XML output of "GET /" e.g. (s3cmd du) invalid. Unfortunately, this behaviour was not reproducable but could be fixed by "GET /{object}" + "PUT /{object}" (s3cmd get + s3cmd put). I am not sure, how this appeared and how to avoid that. Just now, we have nautilus mons and osds with jewel radosgws. Please don't mix versions like that. Running nautilus and jewel at the same time is unsupported. Upgrade everything and check if that solves your problem. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at [ https://croit.io/ | https://croit.io ] croit GmbH Freseniusstr. 31h 81247 München [ http://www.croit.io/ | www.croit.io ] Tel: +49 89 1896585 90 BQ_BEGIN At the time of first appearence, also a nautilus gateway had been online, but the requests had been handled by both types. Any ideas? best regards, Ingo -- Ingo Reimann Teamleiter Technik [ [ https://www.dunkel.de/ | https://www.dunkel.de/ ] ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: [ mailto:support@dunkel.de | support@dunkel.de ] [ http://www.dunkel.de/ | http://www.Dunkel.de/ ] Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001 _______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ] BQ_END
participants (2)
-
Ingo Reimann
-
Paul Emmerich