Hi, We've been trying to set up multi-site sync on two test VMs before rolling things out on actual production hardware. Both are running Ceph 18.2.4 deployed via cephadm. Host OS is Debian 12, container runtime is podman (switched from Debian 11 and docker.io, same error there). There is only one RGW daemon on each site. Ceph config is pretty much defaults. One thing I did change was setting rgw_relaxed_region_enforcement to true because the zonegroup got renamed from "default" during the switch to multi-site using the dashboard's assistant. There's nothing special like server-side encryption either. Our end goal is to replicate all RGW data from our current cluster to a new one. The Multi-Site configuration itself went pretty smoothly through the dashboard and pre-existing data started syncing right away. Unfortunately, not all objects made it. To be precise, none of the larger objects over the multipart threshold got synced. This is consistent for newly uploaded multipart objects as well. Curiously, it's working fine in the other direction, i.e. multipart uploads from the secondary zone do get synced to the master. Here are some relevant logs: From `radosgw-admin sync error list`: { "shard_id": 26, "entries": [ { "id": "1_1722598249.479766_23730.1", "section": "data", "name": "foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7/logstash_1%3a8.12.2-1_amd64.deb", "timestamp": "2024-08-02T11:30:49.479766Z", "info": { "source_zone": "5160b406-4428-4fdc-9c5d-5ec9fe9404c0", "error_code": 35, "message": "failed to sync object(35) Resource deadlock avoided" } } ] }, From RGW on the receiving end: Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.474+0000 7f3a6243e640 0 rgw async rados processor: store->fetch_remote_obj() returned r=-35 Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.474+0000 7f3a36b7b640 2 req 7168648379339657593 0.000000000s :list_data_changes_log normalizing buckets and tenants Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.474+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log init permissions Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log recalculating target Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log reading permissions Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log init op Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log verifying op mask Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log verifying op permissions Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 overriding permissions due to system operation Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log verifying op params Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a5241e640 0 RGW-SYNC:data:sync:shard[28]:entry[foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7[0]]:bucket_sync_sources[source=foobar:new[5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3]):7:source_zone=5160b406-4428-4fdc-9c5d-5ec9fe9404c0]:bucket[foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3<-foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7]:inc_sync[foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7]:entry[logstash_1%3a8.12.2-1_amd64.deb]: ERROR: failed to sync object: foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7/logstash_1%3a8.12.2-1_amd64.deb And from the sender: Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0acfdb2640 1 ====== req done req=0x7f0ab50e4710 op status=-104 http_status=200 latency=0.419986606s ====== Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0ba9f66640 2 req 5943847843579143466 0.000000000s initializing for trans_id = tx00000527cca1f3381a52a-0066acc369-c052e6-eu2 Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0acfdb2640 1 beast: 0x7f0ab50e4710: 10.139.0.151 - synchronization-user [02/Aug/2024:11:30:49.056 +0000] "GET /foobar%3Anew/logstash_1%253a8.12.2-1_amd64.deb?rgwx-zonegroup=9c1ee979-4362-45a1-ae70-2a83a30ea9fc&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a0fab4b8-ec26-4a11-85dd-abab2e3205fa%3Afoobar%2Fnew%3A5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3 HTTP/1.1" 200 138413732 - - - latency=0.419986606s Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0ba9f66640 2 req 5943847843579143466 0.000000000s getting op 0 Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0ba9f66640 2 req 5943847843579143466 0.000000000s :list_metadata_log verifying requester They all keep running into the same error: "failed to sync object(35) Resource deadlock avoided" Any ideas? Thanks! -- Mit freundlichem Gruß // Best regards, Tino Lehnig Cloud Architect Contabo GmbH Aschauer Straße 32a 81549 München https://contabo.com E-Mail: tino.lehnig@contabo.de Amtsgericht München HRB 180722 Vertretungsberechtigte Geschäftsführer: Dr. Christian Böing & Thomas Schimmel