This is a 18.2.4 reef cluster Looking for help with cloud sync module. Here is my setup 1. Two Ceph clusters with RGW setup in a MultiSite configuration. There is no data synchronization between the clusters. No data sync groups, flows, or pipes. 2. We use a multi tenant setup with a small number of tenants (10). We have one tenant that is writing to the same bucket in different rgw zones. The names of files are unique in each zone. 3. We are using the cloud-sync module to copy data from our ceph clusters to aws. I have pasted the commands used to create the sync zone The cloud-sync module works but for one issue that I can not figure out. For some reason another bucket in another tenant is showing up as target to sync but was never specified in the original config. I can never get a clean radosgw-admin sync status because there are recovering shards from a bucket in another tenant that I never intended to sync to the cloud. The data sync status of the recovering shard is pointing to a bucket that never had any cloud sync specified. tenant1/bucket1 was never part of any tier config. radosgw-admin sync error list --rgw-zone cloud-sync shows a never ending list of objects that can not be synced from tenant1/bucket1 there are 4 different types of error messages. "message": "failed to sync bucket instance: (22) Invalid argument" "message": "failed to sync bucket instance: (39) Directory not empty" "message": "failed to sync object(16) Device or resource busy" "message": "failed to sync object(39) Directory not empty" One other point is that radosgw-admin sync info --rgw-zone cloud-sync --bucket tenant1/bucket1 seems to indicate that that is part of the cloud-sync config. What I just want to do is remove tenant1/bucket1 from the sync config and stop the data sync such that the shards recover and the error log stop accumulating messages. I would really like a clean sync status for my cloud tier zone. It would also be nice to help understand what is happening here. All and any help is greatly appreciated. # # sync and data sync status # radosgw-admin sync status --rgw-zone=cloud-sync realm 02b82a81-7b16-4760-ab94-08b79176397e (realm) zonegroup 0e44f33d-4e70-4ba6-8c75-794d3d0b69f8 (zonegroup) zone 34e5298b-4537-4ada-b0d9-75579dc9640a (cloud-sync) current time 2025-02-11T23:56:52Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 792edbc3-8516-4014-aad4-8e7b6ebe2757 (zone-1) syncing full sync: 0/128 shards incremental sync: 128/128 shards 1 shards are recovering recovering shards: [72] source: f3e72377-2fb1-4ab6-aa0a-76b91d157fb3 (zone-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 4 shards behind shards: [72,75,77,78] oldest incremental change not applied: 2025-02-11T15:56:34.654956-0800 [75] 10 shards are recovering recovering shards: [68,69,70,71,72,73,74,76,77,78] radosgw-admin data sync status --rgw-zone cloud-sync --source-zone zone-2 --shard-id 68 { "shard_id": 68, "marker": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000357645", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2025-02-11T23:42:34.266680Z" }, "pending_buckets": [ "tenant1/bucket1:f3e72377-2fb1-4ab6-aa0a-76b91d157fb3.2251620.1:0" ], "recovering_buckets": [ "tenant1/bucket1:f3e72377-2fb1-4ab6-aa0a-76b91d157fb3.2251620.1:0[0]" ], "current_time": "2025-02-11T23:58:08Z" } # # cloud sync zone creation # radosgw-admin zone create \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --endpoints=https://ceph-server:4430 \ --tier-type=cloud radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=s-cloud-sync \ --access-key=<ACCESS> \ --secret=<SECRET> radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=connections[].id=aws-cloud-sync radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=connections[-1].endpoint=https://s3.amazonaws.com,connections[-1].access_key=<ACCESS>,connections[-1].secret=<SECRET> radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=profiles[].connection_id=aws-cloud-sync radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=profiles[-1].connection_id=aws-cloud-sync,profiles[-1].source_bucket='tenant0:bucket0',profiles[-1].target_path=aws-bucket radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=connection_id=aws-cloud-sync,target_path=aws-bucket