The latest version of quincy seems to be having problems cleaning up multipart fragments from canceled uploads. The bucket is empty: % s3cmd -c .s3cfg ls s3://warp-benchmark % However, it's got 11TB of data and 700k objects. # radosgw-admin bucket stats --bucket=warp-benchmark { "bucket": "warp-benchmark", "num_shards": 10, "tenant": "", "zonegroup": "6be863e8-a9f2-42c9-b114-c8651b1f1afa", "placement_rule": "ssd.ec63", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "aa099b5e-01d5-4394-b287-df99a4d63298.18924.1", "marker": "aa099b5e-01d5-4394-b287-df99a4d63298.37403.1", "index_type": "Normal", "owner": "warp_benchmark", "ver": "0#5580404,1#5593184,2#5586262,3#5591427,4#5591937,5#5588120,6#5589760,7#5582923,8#5579062,9#5578699", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0", "mtime": "0.000000", "creation_time": "2023-02-10T21:45:12.721604Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#", "usage": { "rgw.main": { "size": 12047620866048, "size_actual": 12047620866048, "size_utilized": 12047620866048, "size_kb": 11765254752, "size_kb_actual": 11765254752, "size_kb_utilized": 11765254752, "num_objects": 736113 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } A bucket list shows that they are all multipart fragments # radosgw-admin bucket list --bucket=warp-benchmark [ ... (LOTS OF THESE) { "name": "_multipart_(2F3(gCS/1.GagoUCrCRqawswb6.rnd.tg1efLm7-es41Xg3i-Nm6bYjS-c-No79.12", "instance": "", "ver": { "pool": 20, "epoch": 30984 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 16777216, "mtime": "2023-02-16T00:03:01.586472Z", "etag": "e7475bca6a58de35648ca5f25d6653bf", "storage_class": "", "owner": "warp_benchmark", "owner_display_name": "Warp Benchmark", "content_type": "", "accounted_size": 16777216, "user_data": "", "appendable": "false" }, "tag": "_YdopX7yxnVrvg2h35MIQGN3vsPyZx5W", "flags": 0, "pending_map": [], "versioned_epoch": 0 } ] Note that the timestamp is from 2 weeks ago so a lifecycle policy of "cleanup after 1 day" should delete them. cat cleanup-multipart.xml <LifecycleConfiguration> <Rule> <ID>abort-multipart-rule</ID> <Filter> <Prefix></Prefix> </Filter> <Status>Enabled</Status> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> </Rule> </LifecycleConfiguration> % s3cmd dellifecycle s3://warp-benchmark s3://warp-benchmark/: Lifecycle Policy deleted % s3cmd setlifecycle cleanup-multipart.xml s3://warp-benchmark s3://warp-benchmark/: Lifecycle Policy updated A secondary problem is that the lifecycle policy never runs automatically and is stuck in the UNINITIAL state. This problem is for another day of debugging. # radosgw-admin lc list [ { "bucket": ":warp-benchmark:aa099b5e-01d5-4394-b287-df99a4d63298.37403.1", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] However, it can be started manually # radosgw-admin lc process # radosgw-admin lc list [ { "bucket": ":warp-benchmark:aa099b5e-01d5-4394-b287-df99a4d63298.37403.1", "started": "Wed, 01 Mar 2023 17:35:27 GMT", "status": "COMPLETE" } ] This has no effect on the bucket and the bucket stats show the exact same size and object count (output omitted for brevity). Running a gc pass also has no effect # radosgw-admin gc list [] # radosgw-admin gc process # radosgw-admin gc list [] Any ideas?