multisite's metadata sync relies on some abstractions in order to
transfer metadata between zones in json format. it also requires that
any time we write changes to a metadata object, we also write to the
metadata log so other zones know when/what they need to sync
the main abstraction is the RGWMetadataHandler, which we implement for
each kind of metadata. so we have a RGWBucketInstanceMetadataHandler
that knows how to json-encode/decode and read/write bucket instances,
a RGWUserMetadataHandler for users, etc etc
in addition to json and reading/writing, these handlers also have some
important logic in them. for example, when
RGWBucketInstanceMetadataHandler writes a new bucket instance we
haven't seen before, it will also create and initialize its bucket
index objects
the 'archive zone' uses special handler wrappers like
RGWArchiveBucketMetadataHandler and
RGWArchiveBucketInstanceMetadataHandler to force-enable object
versioning, preserve deleted buckets, etc
in
https://github.com/ceph/ceph/pull/28679, we made a lot of changes
to move the actual rados reads/writes into 'metadata backends'. Yehuda
did a good job documenting the motivations and design decisions in the
PR description, so i'd encourage everyone to read through that
this predated the zipper work, but shared a similar goal of non-rados
backends. however, this metadata backend stuff is complicated without
providing any tangible benefits, while also making it significantly
more difficult to add new types of metadata (see Abhishek's work to
support role metadata in
https://github.com/ceph/ceph/pull/37679). i'd
like to see these backends reimagined in terms of zipper to allow
metadata sync between different stores, but i think there are some
open questions here:
* zipper just has a Bucket interface - the distinction between the
'bucket entrypoint' and 'bucket instance' metadata is specific to the
rados store
* the split between MetadataHandler logic and the backends doesn't
seem right, at least in the case of the
RGWBucketInstanceMetadataHandler creating index objects, where the
bucket index itself is a detail of the rados store
i feel like a better approach would be for each zipper store to
implement the MetadataHandlers itself instead of trying to maintain
the separation between the metadata handlers and backends. so each
store would have functions like create_bucket_metadata_handler(),
create_user_metadata_handler(), etc
then for the archive zone, its handlers would wrap the handlers it
gets from zipper, and use generic zipper APIs to implement the extra
stuff like enabling versioning