2 Sep
2020
2 Sep
'20
10:15 p.m.
Good day,
I am having an issue with some multipart uploads to radosgw. I
recently upgraded my cluster from Mimic to Nautilus and began having
problems with multipart uploads from clients using the Java AWS SDK
(specifically 1.11.219). I do NOT have issues with multipart uploads
with other clients such as the aws2 cli. The error that surfaces is
http error 500. Small files that are not multipart uploads work fine
with the Java client.
I enabled debugging for debug_ms = 1 and debug_rgw = 20 and then
performed a multipart upload with the offending client and this is
what I see (full log @
https://object.cancercollaboratory.org:9080/swift/v1/ceph-debug-pub/filtere…
)
- just a snippet -
2020-09-02 18:03:43.339 7f9277474700 1 -- 172.25.13.93:0/458188962
--> [v2:172.25.12.152:6936/12180,v1:172.25.12.152:6937/12180] --
osd_op(unknown.0.0:6331 17.325
17:a4c2f681:::2020-09-02-18-default.262831163.2-score-rdpc-collab-qa:head
[append 0~817] snapc 0=[] ondisk+write+known_if_redirected e1493993)
v8 -- 0x55e9fefb9000 con 0x55e9f3e61f80
2020-09-02 18:03:43.339 7f9277474700 2 req 1714 70.432s s3:put_obj op
status=-35
2020-09-02 18:03:43.339 7f9277474700 2 req 1714 70.432s s3:put_obj
http status=500
haproxy log:
Sep 2 13:33:36 localhost haproxy[14495]: 206.108.127.16:46088
[02/Sep/2020:13:32:06.263] radosgw_public_https_11~
radosgw_public_https_11/storage2-r5 0/0/0/71677/89948 500 477 - - ----
12/12/11/1/0 0/0 {} "PUT
/score-rdpc-collab-qa/data/1c9e4395-677c-55c1-8437-b6467c806173?uploadId=2~zTiBm6p4sB2SyMu2cJ_dwd3ECMpELh3&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200902T173202Z&X-Amz-SignedHeaders=host&X-Amz-Expires=518400&X-Amz-Credential=TO1DRTS4H14HJ388I26R%2F20200902%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=225b6f111cc342c97ad48f8a39f34af044647ac04da532ef949d770509e65f76
HTTP/1.1"
Does anyone know if there may be incompatiblities with Java AWS SDK
v1? I see that v2 is out so one of our devs is going to run some tests
using Java AWS SDK v2 but this may take a bit of time.
Thanks all,
Jared Baker
Ontario Institute for Cancer Research