So, I ended up checking all datalog shards with:
radosgw-admin data sync status --shard-id=XY --source-zone=us-east-1
and found one with a few hundred references to a bucket that had been deleted.
I ended up shutting down HAProxy on both ends and runningĀ
radosgw-admin data sync init
This seemed to clear out the shards and things are running normally now.
radosgw-admin stale instances list
didn't find these for some reason.
thx
Frank