Hi ceph gurus, I am experimenting with rgw multisite sync feature using Quincy release (17.2.5). I am using the zone-level sync, not bucket-level sync policy. During my experiment, somehow my setup got into a situation that it doesn't seem to get out of. One zone is perpetually behind the other, although there is no ongoing client request. Here is the output of my "sync status": root@mon1-z1:~# radosgw-admin sync status realm f90e4356-3aa7-46eb-a6b7-117dfa4607c4 (test-realm) zonegroup a5f23c9c-0640-41f2-956f-a8523eccecb3 (zg) zone bbe3e2a1-bdba-4977-affb-80596a6fe2b9 (z1) metadata sync no sync (zone is master) data sync source: 9645a68b-012e-4889-bf24-096e7478f786 (z2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 14 shards behind shards: [56,61,63,107,108,109,110,111,112,113,114,115,116,117] It stays behind forever while rgw is almost completely idle (1% of CPU). Any suggestion on how to drill deeper to see what happened? Thanks, Yixin