in multisite, these async notifications are http messages that get
periodically broadcast to peer zones as new entries are added to a
shard of the mdlog or datalog. on the destination zones, they serve
two purposes:

* wake up the coroutine that was processing the given log shards, in
case they were sleeping because there was nothing to do the last time
they polled

* for data sync only, these messages also carry the keys of each new
datalog entry so we can trigger sync on the related bucket shards (in
addition to the buckets we're already syncing from the datalog itself)

these notifications have been in since jewel. as i understand it, the
goal was to make replication feel more responsive to updates, but the
model has two major flaws:

* it doesn't scale to more than one gateway per zone. when
broadcasting these notifications, we choose one radosgw endpoint from
each peer zone - but we have no way to know which one of those is
actually processing the log shards we're trying to notify. on receipt,
data sync will cache all of these keys in a map of 'modified_shards',
and the entries will just pile up in memory for the shards it isn't
processing

* it reduces the apparent latency of sync on some buckets at the
expense of overall sync throughput. not only does it prioritize sync
of 'hot' buckets over buckets in the backlog, but for every bucket we
sync via a notification, we'll re-sync it again when we get to its
entry in the log. i don't think this tradeoff is a good one


what does everyone else think? are there other reasons to keep sending these?