[multisite] The purpose of zonegroup
Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin
you're correct that the distinction is between metadata and data; metadata like users and buckets will replicate to all zonegroups, while object data only replicates within a single zonegroup. any given bucket is 'owned' by the zonegroup that creates it (or overridden by the LocationConstraint on creation). requests for data in that bucket sent to other zonegroups should redirect to the zonegroup where it resides the ability to create multiple zonegroups can be useful in cases where you want some isolation for the datasets, but a shared namespace of users and buckets. you may have several connected sites sharing storage, but only require a single backup for purposes of disaster recovery. there it could make sense to create several zonegroups with only two zones each to avoid replicating all objects to all zones in other cases, it could make more sense to isolate things in separate realms with a single zonegroup each. zonegroups just provide some flexibility to control the isolation of data and metadata separately On Thu, Jun 29, 2023 at 5:48 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks! This is something that should be copy-pasted at the top of https://docs.ceph.com/en/latest/radosgw/multisite/ Actually, I reported a documentation bug for something very similar. On Fri, Jun 30, 2023 at 11:30 PM Casey Bodley <cbodley@redhat.com> wrote:
you're correct that the distinction is between metadata and data; metadata like users and buckets will replicate to all zonegroups, while object data only replicates within a single zonegroup. any given bucket is 'owned' by the zonegroup that creates it (or overridden by the LocationConstraint on creation). requests for data in that bucket sent to other zonegroups should redirect to the zonegroup where it resides
the ability to create multiple zonegroups can be useful in cases where you want some isolation for the datasets, but a shared namespace of users and buckets. you may have several connected sites sharing storage, but only require a single backup for purposes of disaster recovery. there it could make sense to create several zonegroups with only two zones each to avoid replicating all objects to all zones
in other cases, it could make more sense to isolate things in separate realms with a single zonegroup each. zonegroups just provide some flexibility to control the isolation of data and metadata separately
On Thu, Jun 29, 2023 at 5:48 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov
cc Zac, who has been working on multisite docs in https://tracker.ceph.com/issues/58632 On Fri, Jun 30, 2023 at 11:37 AM Alexander E. Patrakov <patrakov@gmail.com> wrote:
Thanks! This is something that should be copy-pasted at the top of https://docs.ceph.com/en/latest/radosgw/multisite/
Actually, I reported a documentation bug for something very similar.
On Fri, Jun 30, 2023 at 11:30 PM Casey Bodley <cbodley@redhat.com> wrote:
you're correct that the distinction is between metadata and data; metadata like users and buckets will replicate to all zonegroups, while object data only replicates within a single zonegroup. any given bucket is 'owned' by the zonegroup that creates it (or overridden by the LocationConstraint on creation). requests for data in that bucket sent to other zonegroups should redirect to the zonegroup where it resides
the ability to create multiple zonegroups can be useful in cases where you want some isolation for the datasets, but a shared namespace of users and buckets. you may have several connected sites sharing storage, but only require a single backup for purposes of disaster recovery. there it could make sense to create several zonegroups with only two zones each to avoid replicating all objects to all zones
in other cases, it could make more sense to isolate things in separate realms with a single zonegroup each. zonegroups just provide some flexibility to control the isolation of data and metadata separately
On Thu, Jun 29, 2023 at 5:48 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov
The information provided by Casey has been added to doc/radosgw/multisite.rst in this PR: https://github.com/ceph/ceph/pull/52324 Zac Dover Upstream Docs Ceph Foundation ------- Original Message ------- On Saturday, July 1st, 2023 at 1:45 AM, Casey Bodley <cbodley@redhat.com> wrote:
cc Zac, who has been working on multisite docs in https://tracker.ceph.com/issues/58632
On Fri, Jun 30, 2023 at 11:37 AM Alexander E. Patrakov patrakov@gmail.com wrote:
Thanks! This is something that should be copy-pasted at the top of https://docs.ceph.com/en/latest/radosgw/multisite/
Actually, I reported a documentation bug for something very similar.
On Fri, Jun 30, 2023 at 11:30 PM Casey Bodley cbodley@redhat.com wrote:
you're correct that the distinction is between metadata and data; metadata like users and buckets will replicate to all zonegroups, while object data only replicates within a single zonegroup. any given bucket is 'owned' by the zonegroup that creates it (or overridden by the LocationConstraint on creation). requests for data in that bucket sent to other zonegroups should redirect to the zonegroup where it resides
the ability to create multiple zonegroups can be useful in cases where you want some isolation for the datasets, but a shared namespace of users and buckets. you may have several connected sites sharing storage, but only require a single backup for purposes of disaster recovery. there it could make sense to create several zonegroups with only two zones each to avoid replicating all objects to all zones
in other cases, it could make more sense to isolate things in separate realms with a single zonegroup each. zonegroups just provide some flexibility to control the isolation of data and metadata separately
On Thu, Jun 29, 2023 at 5:48 PM Yixin Jin yjin77@yahoo.ca wrote:
Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov
Hi Casey, Thanks a lot for the clarification. I feel that zonegroup made a great sense at the beginning when multisite feature was conceived and (I suspect) zones were always syncing from all other zones within a zonegroup. However, once the "sync_from" was introduced and later the sync policy further enhanced the granularity of the control over data sync, it seems not much advantage is left with zonegroup. Both "sync_from" and sync policy could be moved up to realm level while the isolation of datasets can still be maintained. On the other hand, if some new features are introduced to enable some isolation of metadata within the same realm, probably at zonegroup level, its usefulness may be more justified. Regards,Yixin On Friday, June 30, 2023 at 11:29:16 a.m. EDT, Casey Bodley <cbodley@redhat.com> wrote: you're correct that the distinction is between metadata and data; metadata like users and buckets will replicate to all zonegroups, while object data only replicates within a single zonegroup. any given bucket is 'owned' by the zonegroup that creates it (or overridden by the LocationConstraint on creation). requests for data in that bucket sent to other zonegroups should redirect to the zonegroup where it resides the ability to create multiple zonegroups can be useful in cases where you want some isolation for the datasets, but a shared namespace of users and buckets. you may have several connected sites sharing storage, but only require a single backup for purposes of disaster recovery. there it could make sense to create several zonegroups with only two zones each to avoid replicating all objects to all zones in other cases, it could make more sense to isolate things in separate realms with a single zonegroup each. zonegroups just provide some flexibility to control the isolation of data and metadata separately On Thu, Jun 29, 2023 at 5:48 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
thanks Yixin, On Tue, Jul 4, 2023 at 1:20 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi Casey, Thanks a lot for the clarification. I feel that zonegroup made a great sense at the beginning when multisite feature was conceived and (I suspect) zones were always syncing from all other zones within a zonegroup. However, once the "sync_from" was introduced and later the sync policy further enhanced the granularity of the control over data sync, it seems not much advantage is left with zonegroup.
both sync_from and sync policy do offer finer-grained control over which zones sync from which, but they can't represent a bucket's 'residency' the way that the zonegroup-based LocationConstraint does. by redirecting requests to the bucket's resident zonegroup, the goal is to present a single eventually-consistent set of objects per bucket. while features like sync_from and bucket replication policy do complicate this picture, i think this concept of residency and redirects are important to make sense of s3's LocationConstraint. but perhaps the sync policy model could be extended to take over the zonegroup's role here?
Both "sync_from" and sync policy could be moved up to realm level while the isolation of datasets can still be maintained. On the other hand, if some new features are introduced to enable some isolation of metadata within the same realm, probably at zonegroup level, its usefulness may be more justified.> Regards,Yixin
On Friday, June 30, 2023 at 11:29:16 a.m. EDT, Casey Bodley <cbodley@redhat.com> wrote:
you're correct that the distinction is between metadata and data; metadata like users and buckets will replicate to all zonegroups, while object data only replicates within a single zonegroup. any given bucket is 'owned' by the zonegroup that creates it (or overridden by the LocationConstraint on creation). requests for data in that bucket sent to other zonegroups should redirect to the zonegroup where it resides
the ability to create multiple zonegroups can be useful in cases where you want some isolation for the datasets, but a shared namespace of users and buckets. you may have several connected sites sharing storage, but only require a single backup for purposes of disaster recovery. there it could make sense to create several zonegroups with only two zones each to avoid replicating all objects to all zones
in other cases, it could make more sense to isolate things in separate realms with a single zonegroup each. zonegroups just provide some flexibility to control the isolation of data and metadata separately
On Thu, Jun 29, 2023 at 5:48 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Alexander E. Patrakov
-
Casey Bodley
-
Yixin Jin
-
Zac Dover