Sync Info: Information about the sync entities (for example: meta, data, bucket). * Rationale To simplify the sync code and to separate the sync logic from the info layout. To provide a framework that could enable alternative sync entity providers. * Overview The rgw sync code handles sync of both metadata and buckets data. It pulls information about the sync entities from 3 separate types of providers: meta, data (which buckets need to sync), bucket. Sync for each of the types is split into two different stages: full sync and incremental sync. Full sync means that all the relevant data is being iterated over: listing of all meta keys, listing of all bucket instances, listing of all keys in every bucket. Incremental sync means that the different changes logs are being read and changes are being applied. Stemming from this is that we have 6 separate implementations for the core sync process that vary in the way sync information is fetched, but repeat the same general logic: fetch information, apply changes, update markers, handle errors. Therefore there is an opportunity to consolidate the sync logic into common generic code. At the minimum we could combine the full sync and incremental stages of each entity type. Other considerations, like multi-stage bucket sync that supports resharding could benefit from that. * Details new module: sync info provider The sync info provider will be responsible for providing meta-information about the sync entity (e.g., how many shards in the different sync stages, current state for each stage), and will provide serial sync info by marker. The marker itself will be opaque to the target and will reflect the sync stage. The sync info provider will return the list of source entities that need to be synced, and the target should be able to apply those changes without specific knowledge of each different stage. The info provided will start with the 'full sync' data (e.g., for metadata sync it will include all the metadata keys), and when that is exhausted, it will include all the metadata log entries. The structure of each info entry will be the same whichever stage it is. Each stage could have different sharding properties. For example: the first stage (full sync) can have a single shard, and the second stage (incremental sync) can have many more shards. When starting the sync, the target will send init request that will return a list of initial markers for each shard in any of the existing stages. The shard id and the stage id information will be embedded within the marker. Trimming: We can leverage this system to simplify the trim logic. The sync info provider could keep a list of all the targets and their corresponding current synced marker position (that the targets will provide). The trimmers could then use that information instead of polling the targets for their current state. We can get rid of the trimmers polling scheme altogether and the sync info providers could maintain a central in-memory list of trimming targets that will periodically be trimmed. (Need to consider backward compatibility) The sync info providers should be running at the source zone and a RESTful api should be created to access their functionality. They should read in the source zone so that sync info about targets can be aggregated (for trimming purposes). However, we should create a target-side interface that would provide a functional interface for the sync code. We can create alternative target-side implementations for managing sources that do not support this API (e.g., backward compatibility and other non-rgw sources). A sync info provider client wrapper can be created to enable cases like full sync of metadata where the source info is not sharded. The generic wrapper would fetch the data and store it in temporary queues (the same as the current full metadata sync) so that the sync process itself could happen concurrently by multiple shards. It should be transparent to the caller, and from its point of view it would fetch the data from a sharded source. sync_info_init: input: { my_id entity_id } output: { sync_info_id stages[] = { stage_id num_shards markers[] = { string marker } } } sync_info_fetch: input: { sync_info_id marker max_entries optional: sync_position (for trimming) } output: { status = { have_more | done | stage_done | who_are_you } entries[] = { marker_id info (depending on entity type) } } * The entry info will depend on the entity type. It will usually include a timestamp and other type-specific fields. For example, the bucket entity type would include an op field that data entities would not have. * When the data of a specific sync stage is exhausted, the status will reflect it. * The source might decide to remove a target from its list of targets if a target hasn't contacted it for a while. This can happen if a target went down, and is needed to allow the source to trim its logs. If the target returns then the source will send an error message that will require the target to re-initialize its sync process. * if there is no more data in the current stage, the stage_done status will be returned. The target will only start working on the next stage after all the current stage shards are complete. When all shards are complete, the sync process should initiate a sync on all the new stage shards. If target does not have any information for the next stage (e.g., after bucket reshard), it will query the source for that information sync_info_update_position: input: { sync_info_id sync_position_marker } output: { status } sync_info_next_stage_info: input: { cur_stage_id } output: { status next_stage_id num_shards markers[] = { string marker } } Note that the returned markers will be the needed markers for transitioning to the next stage. The markers returned when transitioning from full sync to incremental sync reflect the max logs position when the sync started. The markers returned when transitioning from different incremental sync stages (e.g., different reshard generations) are the minimum log positions (or even empty position) of the next generation. * Development Initial work & Metadata Sync * Source-side sync info provider core * define functional interfaces * abstract SyncInfoProvider * abstract SIPEntity * Control * store/read target state * marker tools * First implementation * create provider for metadata * SyncInfoProvider_Meta * SIPEntity_Meta * radosgw-admin to control SyncInfoProvider hooks * REST api * Target-side core * define abstract SIPClient * implement SIPClient_REST * Coroutines implementation * radosgw-admin to control SIPClient hooks * Meta sync * modify sync init * convert incremental sync to use SIP * stage transition * remove full sync * Source side trimming module * generic core * implement mdlog trimmer TODO: - backward compatibility plan - trimming in mixed versions environment? - radosgw-admin sync status - testing Data & Bucket Sync * Data Sync * source side SyncInfoProvider_Data * convert sync code similar to meta sync * Bucket Sync * SyncInfoProvider_BucketInstance * bucket instance sync code * Optional * common sync core * modify all implementations to use common core Yehuda