[ceph-users] possible bug in radosgw-admin bucket radoslist

4 May 2021

Hi there,

I think I found a bug in the radosgw-admin bucket radoslist command. I'm not 100% sure
so would like to check here first before I fill a bug report.
I have a bucket called bucket3. If I do a multipart upload and stop it halfway for example
and start a new upload with the same name and abort it, I'll get duplicate lines in
the radoslist output.

For example:

root@alpha:~# radosgw-admin bucket radoslist --bucket bucket3
root@alpha:~#
root@alpha:~# s3cmd put -P 100MB.bin s3://bucket3/multipart-obj-fail
upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail'  [part 1 of 7,
15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    35.14 MB/s  done
upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail'  [part 2 of 7,
15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    33.59 MB/s  done
upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail'  [part 3 of 7,
15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    33.88 MB/s  done
upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail'  [part 4 of 7,
15MB] [1 of 1]
    65536 of 15728640     0% in    0s   806.10 KB/s^CERROR:
Upload of '100MB.bin' part 4 failed. Use
  /usr/bin/s3cmd abortmp s3://bucket3/multipart-obj-fail
2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX
to abort the upload, or
  /usr/bin/s3cmd --upload-id 2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX put ...
to continue the

If I now run a radoslist it looks fine:

646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_3

However, if I do a second upload with the same name:

root@alpha:~# s3cmd put -P 100MB.bin s3://bucket3/multipart-obj-fail
upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail'  [part 1 of 7,
15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    23.18 MB/s  done
upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail'  [part 2 of 7,
15MB] [1 of 1]
 15728640 of 15728640   100% in    1s    13.10 MB/s  done
upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail'  [part 3 of 7,
15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    29.93 MB/s  done
upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail'  [part 4 of 7,
15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    39.53 MB/s  done
upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail'  [part 5 of 7,
15MB] [1 of 1]
^CERROR:
Upload of '100MB.bin' part 5 failed. Use
  /usr/bin/s3cmd abortmp s3://bucket3/multipart-obj-fail
2~yS7Tzru_FSP6rkg4yeO28os207nDtTw
to abort the upload, or
  /usr/bin/s3cmd --upload-id 2~yS7Tzru_FSP6rkg4yeO28os207nDtTw put ...
to continue the upload.
See ya!

Note that the IDs are unique, run1: 2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX  run2:
2~yS7Tzru_FSP6rkg4yeO28os207nDtTw
But if we look at the radoslist output again:

646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3_3
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4_1
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4_2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4_3

the duplicates are(output from radoslist | sort | uniq -c):

      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4

If I again do a new upload and stop it I get a new upload ID
2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX but again with duplicate entries(output from radoslist |
sort | uniq -c):

      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~iNQha2hAznnKdSLukUAbJT1-4nXoWUy.1
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~iNQha2hAznnKdSLukUAbJT1-4nXoWUy.2
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~iNQha2hAznnKdSLukUAbJT1-4nXoWUy.3
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~iNQha2hAznnKdSLukUAbJT1-4nXoWUy.4
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3
      2
646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4

I'm running Ceph Octopus deployed by cephadm:

root@alpha:~# ceph versions
{
    "overall": {
        "ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus
(stable)": 12
    }
}

Is this expected behaviour or should I make a bug report for this?

Kind regards,

Rob

2024

2023

2022

2021

2020

2019

[ceph-users] possible bug in radosgw-admin bucket radoslist