Hi all, We are running into a problem cleaning up abandoned multipart uploads on RGW and would appreciate a sanity check of our analysis, plus any advice on a better approach. Environment ----------- - Ceph Squid 19.2.6, cephadm-deployed, single zone. - Data pool: EC on HDD (12 x 24 TB per host, DB/WAL on NVMe). Index pool on NVMe. - 12 RGW daemons behind the cephadm ingress (HAProxy). - rgw_gc_max_objs = 128, rgw_gc_max_queue_size at its default (~128 MiB). What happened ------------- A customer's backup appliance (a NAS uploading backup files over S3) left 9 incomplete multipart uploads in one bucket: 3 objects x 3 attempts each. Each upload has ~26,000 parts of 128 MiB (~3.3 TiB per upload, ~31.6 TB in total), and the bucket has 0 visible objects. We suspect the uploads exceeded the 10,000-part limit, so CompleteMultipartUpload failed and the appliance retried from scratch, leaving the previous upload behind. We have not confirmed this yet (see question 5). Each 128 MiB part is 1 __multipart_ head plus 31 x 4 MiB __shadow_ objects, so each upload is ~830k RADOS objects and the bucket ~7.5M. Symptoms -------- 1. The customer's DeleteBucket timed out. As we understand it, RGW aborts pending MPUs synchronously inside DeleteBucket, which cannot finish within an HTTP request at this size. AbortMultipartUpload from the client returned 503. 2. "radosgw-admin bucket rm --purge-objects --bypass-gc" immediately logs: garbage collection: RGWGC::send_split_chain - send chain returned error: -28 abort: gc->send_chain() returned -28 delete_objs_inline: removing <data-pool>:<marker>__shadow_<key>.2~<upload-id>.1_1 delete_objs_inline: refcount put returned error -2 ... It then deletes the rest of the chain inline, one object at a time, from a single thread. Each refcount put takes between ~10 ms and ~2 s, so it averages roughly 1 object/s. --max-concurrent-ios has no effect on this phase, and --bypass-gc does not seem to apply to the MPU abort. 3. The GC as a whole was nearly empty at the time. The -28 is not global pressure. Our analysis (from reading the Squid source; please correct us) --------------------------------------------------------------- - RadosMultipartUpload::abort() collects the manifests of all parts into ONE chain and sends it to the GC with tag = upload id. - The GC shard is chosen from the tag (hash(tag) % rgw_gc_max_objs). RGWGC::send_split_chain() splits the chain into rgw_max_chunk_size (4 MiB) entries, but every entry keeps the same tag and therefore goes to the SAME gc.N shard. - Each shard is a single cls_queue object capped at rgw_gc_max_queue_size (~128 MiB, bounded by osd_max_object_size). A ~830k-object chain encodes to roughly 130-160 MB, so it cannot fit even in an empty shard. We observed 31 x 4 MiB entries with that upload's tag filling one shard. The enqueue then fails with ENOSPC, and the remainder of the chain is deleted inline via delete_objs_inline(). Consequences ------------ a) The multipart meta object is only removed at the very end of the abort. Any interruption (client timeout, Ctrl-C, RGW restart) means the next attempt restarts from part 1. It re-walks objects that are already gone (refcount put -> -2, which is still ~1 obj/s), and the shard is still full from the previous attempt, so this time the whole chain goes inline. b) While a shard is full, any unrelated deletion whose tag hashes to that shard also falls back to inline deletion. c) Raising rgw_gc_max_queue_size does not help: it is capped by osd_max_object_size, and existing queues are not resized (queue_init returns EEXIST). Workaround we are using ----------------------- 1. Drain the GC first (radosgw-admin gc process --include-all). 2. Put an AbortIncompleteMultipartUpload lifecycle rule on the bucket and run: radosgw-admin lc process --bucket=<bucket> --rgw-lc-max-wp-worker=9 This aborts all 9 uploads in parallel. Each upload fills its own GC shard (~75% of its chain), and the remaining ~25% is deleted inline, but with 9 threads instead of 1. Note that in this code path the inline deletion is logged under debug_rgw_lifecycle, not debug_rgw. 3. Run several "radosgw-admin gc process --include-all" in parallel, at most one per full shard, since each shard is processed under its own lock. 4. Finally, bucket rm --purge-objects --bypass-gc, which should be fast once no MPUs remain. Questions --------- 1. Is this a known issue? Is there a tracker for MPU aborts overflowing a single GC queue shard? 2. Would it make sense for the abort path to honour --bypass-gc in radosgw-admin, or to use AIO with concurrency for delete_objs_inline() instead of deleting one object at a time? 3. Could the abort make incremental progress (e.g. drop part index entries as they are released) so that a retry does not restart from part 1? 4. Is there any safe, supported way to resize existing GC queue objects, or a recommended setting for workloads with very large MPUs? 5. Does RGW enforce the 10,000-part limit at UploadPart time, or only at CompleteMultipartUpload (rgw_multipart_part_upload_limit)? If UploadPart accepts part numbers above 10,000, a client can build an upload that can never complete. Rejecting it early would avoid this whole situation. 6. Is there a better way to purge tens of TB of abandoned multipart uploads? Thanks in advance, Manuel Rios