30 Sep
2020
30 Sep
'20
12:55 p.m.
On Wed, Sep 30, 2020 at 5:20 AM Eugeniy Khvastunov <khvastunov(a)gmail.com> wrote:
Hi ceph-users,
Looks like we faced a broken S3 multipart upload in Ceph 12.2.11 Luminous
(MCP 2019.2.0) during thanos setup.
The request to initialize the multipart fails.
S3cmd log:
DEBUG: Sending request method_string='POST',
uri='/staging-prometheus-k8s-metrics/test4/dd-50M.rnd?uploads',
headers={'x-amz-meta-s3cmd-attrs': 'uid:1011/gname:
yevhen.kh/uname:yevhen.kh/gid:1011/mode:33188/mtime:1600198080/atime:1600198530/md5:730ee931c6058cad5bd4d87a6c8f3178/ctime:1600198363',
'content-type': 'application/octet-stream', 'Authorization':
'AWS
ZLMVD7JZZ8V09VVK66C3:YX5Oe+Ig0FPDbRpIYO2m1WHITOE=', 'x-amz-date': 'Tue,
15
Sep 2020 19:38:33 +0000', 'x-amz-storage-class': 'STANDARD'}, body=(0
bytes)
DEBUG: Response: {'status': 400, 'headers': {'content-length':
'253',
'accept-ranges': 'bytes', 'server': 'nginx',
'connection': 'keep-alive',
'x-amz-request-id': 'tx0000000000000001e019a-005f611839-deb80f-default',
'date': 'Tue, 15 Sep 2020 19:38:37 GMT', 'content-type':
'application/xml'}, 'reason': 'Bad Request', 'data':
'<?xml version="1.0"
encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>staging-prometheus-k8s-metrics</BucketName><RequestId>tx0000000000000001e019a-005f611839-deb80f-default</RequestId><HostId>deb80f-default-default</HostId></Error>'}
DEBUG: ConnMan.put(): connection put back to pool (
https://s3.cloud.f.in.company.com:8080#1)
DEBUG: S3Error: 400 (Bad Request)
DEBUG: HttpHeader: content-length: 253
DEBUG: HttpHeader: accept-ranges: bytes
DEBUG: HttpHeader: server: nginx
DEBUG: HttpHeader: connection: keep-alive
DEBUG: HttpHeader: x-amz-request-id:
tx0000000000000001e019a-005f611839-deb80f-default
DEBUG: HttpHeader: date: Tue, 15 Sep 2020 19:38:37 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'InvalidArgument'
DEBUG: ErrorXML: BucketName: 'staging-prometheus-k8s-metrics'
DEBUG: ErrorXML: RequestId:
'tx0000000000000001e019a-005f611839-deb80f-default'
DEBUG: ErrorXML: HostId: 'deb80f-default-default'
ERROR: S3 error: 400 (InvalidArgument)
CEPH log:
> 2020-09-10 08:26:26.238814 7f7482fdd700 1 -- 10.14.2.66:0/3902939317 <==
> mon.0 10.14.2.66:6789/0 19 ==== pool_op_reply(tid 1290 (22) Invalid
> argument v22601) v1 ==== 43+0+0 (3815426539 0 0) 0x55faeb4e4a80 con
> 0x55faeabc8800
it looks like the EINVAL error is coming from pool creation. we use
the data_extra_pool (ie default.rgw.buckets.non-ec) to track some
state for multipart uploads, so that's probably the one it's trying to
create here
2020-09-10
08:26:26.241214 7f74607de700 2 req 484:1.595459:s3:POST
/staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:completing
2020-09-10 08:26:26.241389 7f74607de700 2 req 484:1.595632:s3:POST
/staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:op
status=-22
2020-09-10 08:26:26.241414 7f74607de700 2 req 484:1.595658:s3:POST
/staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:http
status=400
2020-09-10 08:26:26.241420 7f74607de700 1 ====== req done
req=0x7f74607d8140 op status=-22 http_status=400 ======
2020-09-10 08:26:26.241443 7f74607de700 20 process_request() returned -22
2020-09-10 08:26:26.241492 7f74607de700 1 civetweb: 0x55faeaedf000:
10.14.5.203 - - [10/Sep/2020:08:26:24 +0000] "POST
/staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001?uploads=
HTTP/1.1" 400 0 - MinIO (linux; amd64) minio-go/v7.0.2
thanos-sidecar/0.15.0 (go1.14.2)
The bucket exists. The issue is repeated on the newly created bucket and on
the existing one.
If s3cmd runs with `--multipart-chunk-size-mb=` option and the value is
larger than the file size(for example, if the file is 50 mb, set
--multipart-chunk-size-mb=200 mb) then the file uploads.
Example:
s3cmd -c /root/f-s3-thanos.cfg put
/root/dd-50M.rnd
s3://staging-prometheus-k8s-metrics/test4/dd-50M.rnd -d
--multipart-chunk-size-mb=200
If s3cmd runs with `--disable-multipart` that works as well.
Have you faced such problems?
I would be grateful for your advice.
--
wbr. Eugeniy Khvastunov,
System administrator.
[FMGH-UANIC]
http://blog.unlimite.net
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io