The latest version of quincy seems to be having problems cleaning up multipart fragments from canceled uploads.
The bucket is empty:
% s3cmd -c .s3cfg ls s3://warp-benchmark
%
However, it's got 11TB of data and 700k objects.
# radosgw-admin bucket stats --bucket=warp-benchmark
{
"bucket": "warp-benchmark",
"num_shards": 10,
"tenant": "",
"zonegroup": "6be863e8-a9f2-42c9-b114-c8651b1f1afa",
"placement_rule": "ssd.ec63",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "aa099b5e-01d5-4394-b287-df99a4d63298.18924.1",
"marker": "aa099b5e-01d5-4394-b287-df99a4d63298.37403.1",
"index_type": "Normal",
"owner": "warp_benchmark",
"ver": "0#5580404,1#5593184,2#5586262,3#5591427,4#5591937,5#5588120,6#5589760,7#5582923,8#5579062,9#5578699",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0",
"mtime": "0.000000",
"creation_time": "2023-02-10T21:45:12.721604Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#",
"usage": {
"rgw.main": {
"size": 12047620866048,
"size_actual": 12047620866048,
"size_utilized": 12047620866048,
"size_kb": 11765254752,
"size_kb_actual": 11765254752,
"size_kb_utilized": 11765254752,
"num_objects": 736113
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
A bucket list shows that they are all multipart fragments
# radosgw-admin bucket list --bucket=warp-benchmark
[
... (LOTS OF THESE)
{
"name": "_multipart_(2F3(gCS/1.GagoUCrCRqawswb6.rnd.tg1efLm7-es41Xg3i-Nm6bYjS-c-No79.12",
"instance": "",
"ver": {
"pool": 20,
"epoch": 30984
},
"locator": "",
"exists": "true",
"meta": {
"category": 1,
"size": 16777216,
"mtime": "2023-02-16T00:03:01.586472Z",
"etag": "e7475bca6a58de35648ca5f25d6653bf",
"storage_class": "",
"owner": "warp_benchmark",
"owner_display_name": "Warp Benchmark",
"content_type": "",
"accounted_size": 16777216,
"user_data": "",
"appendable": "false"
},
"tag": "_YdopX7yxnVrvg2h35MIQGN3vsPyZx5W",
"flags": 0,
"pending_map": [],
"versioned_epoch": 0
}
]
Note that the timestamp is from 2 weeks ago so a lifecycle policy of "cleanup after 1 day" should delete them.
cat cleanup-multipart.xml
<LifecycleConfiguration>
<Rule>
<ID>abort-multipart-rule</ID>
<Filter>
<Prefix></Prefix>
</Filter>
<Status>Enabled</Status>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>1</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
</Rule>
</LifecycleConfiguration>
% s3cmd dellifecycle s3://warp-benchmark
s3://warp-benchmark/: Lifecycle Policy deleted
% s3cmd setlifecycle cleanup-multipart.xml s3://warp-benchmark
s3://warp-benchmark/: Lifecycle Policy updated
A secondary problem is that the lifecycle policy never runs automatically and is stuck in the UNINITIAL state. This problem is for another day of debugging.
# radosgw-admin lc list
[
{
"bucket": ":warp-benchmark:aa099b5e-01d5-4394-b287-df99a4d63298.37403.1",
"started": "Thu, 01 Jan 1970 00:00:00 GMT",
"status": "UNINITIAL"
}
]
However, it can be started manually
# radosgw-admin lc process
# radosgw-admin lc list
[
{
"bucket": ":warp-benchmark:aa099b5e-01d5-4394-b287-df99a4d63298.37403.1",
"started": "Wed, 01 Mar 2023 17:35:27 GMT",
"status": "COMPLETE"
}
]
This has no effect on the bucket and the bucket stats show the exact same size and object count (output omitted for brevity).
Running a gc pass also has no effect
# radosgw-admin gc list
[]
# radosgw-admin gc process
# radosgw-admin gc list
[]
Any ideas?
Hello
We are planning to start QE validation release next week.
If you have PRs that are to be part of it, please let us know by
adding "needs-qa" for 'quincy' milestone ASAP.
Thx
YuriW
