Supporting localise Read for RGW
Hi, I am planning to support read localise feature for RGW servers similar to what RBD volumes support. From code reading, it looks to pass "librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this will be per server config option. I don't know about any other technical hurdles in implementing this feature. Please share your thoughts on the same. Thanks and regards, Jiffin
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan@redhat.com> wrote:
Hi,
I am planning to support read localise feature for RGW servers similar to what RBD volumes support. From code reading, it looks to pass "librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this will be per server config option. I don't know about any other technical hurdles in implementing this feature. Please share your thoughts on the same.
Thanks and regards, Jiffin
thanks Jiffin, that sounds simple enough, though rgw issues librados reads in many different places that would need to manage this flag i'm curious why this is a per-op flag, rather than a global librados setting. is there a reason that rgw would only want *some* of its reads localized? are there any implications for read-after-write consistency that we need to worry about here?
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan@redhat.com> wrote:
Hi,
I am planning to support read localise feature for RGW servers similar to what RBD volumes support. From code reading, it looks to pass "librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this will be per server config option. I don't know about any other technical hurdles in implementing this feature. Please share your thoughts on the same.
Thanks and regards, Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados setting. is there a reason that rgw would only want *some* of its reads localized? are there any implications for read-after-write consistency that we need to worry about here?
Hi Casey, I'm not aware of any read-after-write or other consistency implications. Read-from-replica is safe for general use since Octopus (OSD-side issues were fixed in [1]). Ultimately all flags are per-op. The reason that there is no global librados setting for BALANCE_READS and LOCALIZE_READS seems to be a combination of a) the fact that these flags could only be used in special circumstances in the past and b) the fact that librados doesn't have a generic API for applying a given flag globally like it does for applying a given snapshot ID/context to all reads/writes. [1] https://github.com/ceph/ceph/pull/32381 Thanks, Ilya
One more point - from a business perspective, this flag is per pool and not global so the API should be per pool and not global. Regards, Josh On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan@redhat.com>
Hi,
I am planning to support read localise feature for RGW servers similar
to what RBD volumes support. From code reading, it looks to pass "librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this will be per server config
wrote: option. I don't know about any other technical hurdles in implementing this feature. Please share your thoughts on the same.
Thanks and regards, Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados setting. is there a reason that rgw would only want *some* of its reads localized? are there any implications for read-after-write consistency that we need to worry about here?
Hi Casey,
I'm not aware of any read-after-write or other consistency implications. Read-from-replica is safe for general use since Octopus (OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global librados setting for BALANCE_READS and LOCALIZE_READS seems to be a combination of a) the fact that these flags could only be used in special circumstances in the past and b) the fact that librados doesn't have a generic API for applying a given flag globally like it does for applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
It would probably make sense to add support in the objecter for picking up certain flags (such as LOCALIZE_READS) from the pool config. -Sam On Sun, Jun 18, 2023 at 1:34 AM Josh Salomon <jsalomon@redhat.com> wrote:
One more point - from a business perspective, this flag is per pool and not global so the API should be per pool and not global. Regards,
Josh
On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan@redhat.com> wrote:
Hi,
I am planning to support read localise feature for RGW servers similar to what RBD volumes support. From code reading, it looks to pass "librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this will be per server config option. I don't know about any other technical hurdles in implementing this feature. Please share your thoughts on the same.
Thanks and regards, Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados setting. is there a reason that rgw would only want *some* of its reads localized? are there any implications for read-after-write consistency that we need to worry about here?
Hi Casey,
I'm not aware of any read-after-write or other consistency implications. Read-from-replica is safe for general use since Octopus (OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global librados setting for BALANCE_READS and LOCALIZE_READS seems to be a combination of a) the fact that these flags could only be used in special circumstances in the past and b) the fact that librados doesn't have a generic API for applying a given flag globally like it does for applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Tue, Jun 27, 2023 at 7:12 PM Sam Just <sjust@redhat.com> wrote:
It would probably make sense to add support in the objecter for picking up certain flags (such as LOCALIZE_READS) from the pool config. -Sam
thanks Sam. Jiffin, would you be willing to look into this Objecter part? i think it would make your project easier if the feature was transparent to rgw
On Sun, Jun 18, 2023 at 1:34 AM Josh Salomon <jsalomon@redhat.com> wrote:
One more point - from a business perspective, this flag is per pool and not global so the API should be per pool and not global. Regards,
Josh
On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan@redhat.com> wrote:
Hi,
I am planning to support read localise feature for RGW servers similar to what RBD volumes support. From code reading, it looks to pass "librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this will be per server config option. I don't know about any other technical hurdles in implementing this feature. Please share your thoughts on the same.
Thanks and regards, Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados setting. is there a reason that rgw would only want *some* of its reads localized? are there any implications for read-after-write consistency that we need to worry about here?
Hi Casey,
I'm not aware of any read-after-write or other consistency implications. Read-from-replica is safe for general use since Octopus (OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global librados setting for BALANCE_READS and LOCALIZE_READS seems to be a combination of a) the fact that these flags could only be used in special circumstances in the past and b) the fact that librados doesn't have a generic API for applying a given flag globally like it does for applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
is there a global/cluster-wide version of such a flag, or to automate applying it? it seems like the scenarios I hear about are stretched clusters, and that it would be attractive to not have to manage that by exception? Matt On Mon, Jul 3, 2023 at 12:25 PM Casey Bodley <cbodley@redhat.com> wrote:
On Tue, Jun 27, 2023 at 7:12 PM Sam Just <sjust@redhat.com> wrote:
It would probably make sense to add support in the objecter for picking up certain flags (such as LOCALIZE_READS) from the pool config. -Sam
thanks Sam. Jiffin, would you be willing to look into this Objecter part? i think it would make your project easier if the feature was transparent to rgw
On Sun, Jun 18, 2023 at 1:34 AM Josh Salomon <jsalomon@redhat.com>
One more point - from a business perspective, this flag is per pool
and not global so the API should be per pool and not global.
Regards,
Josh
On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley@redhat.com>
wrote:
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan@redhat.com>
wrote:
Hi,
I am planning to support read localise feature for RGW servers
similar to what RBD volumes support. From code reading, it looks to pass "librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this will be per server config
wrote: option. I don't know about any other technical hurdles in implementing this feature. Please share your thoughts on the same.
Thanks and regards, Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados setting. is there a reason that rgw would only want *some* of its reads localized? are there any implications for read-after-write consistency that we need to worry about here?
Hi Casey,
I'm not aware of any read-after-write or other consistency implications. Read-from-replica is safe for general use since Octopus (OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global librados setting for BALANCE_READS and LOCALIZE_READS seems to be a combination of a) the fact that these flags could only be used in special circumstances in the past and b) the fact that librados doesn't have a generic API for applying a given flag globally like it does for applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
For userspace clients I think we can implement a normal config option (rados_localize_reads) and set it to true in the mon config store, and then clients will pick it up. But that would be global to all ops — for it to be pool-based, we’d have to extend the OSDMap with client information and make some more extensive Objecter changes (and kclient ones). That’s not impossible but feels a bit weird. :/ -Greg On Mon, Jul 3, 2023 at 11:36 AM Matt Benjamin <mbenjami@redhat.com> wrote:
is there a global/cluster-wide version of such a flag, or to automate applying it? it seems like the scenarios I hear about are stretched clusters, and that it would be attractive to not have to manage that by exception?
Matt
On Mon, Jul 3, 2023 at 12:25 PM Casey Bodley <cbodley@redhat.com> wrote:
On Tue, Jun 27, 2023 at 7:12 PM Sam Just <sjust@redhat.com> wrote:
It would probably make sense to add support in the objecter for picking up certain flags (such as LOCALIZE_READS) from the pool config. -Sam
thanks Sam. Jiffin, would you be willing to look into this Objecter part? i think it would make your project easier if the feature was transparent to rgw
On Sun, Jun 18, 2023 at 1:34 AM Josh Salomon <jsalomon@redhat.com>
One more point - from a business perspective, this flag is per pool
and not global so the API should be per pool and not global.
Regards,
Josh
On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley@redhat.com>
wrote:
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <
jthottan@redhat.com> wrote:
> > Hi, > > I am planning to support read localise feature for RGW servers similar to what RBD volumes support. From code reading, it looks to pass "librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this will be per server config
> > Thanks and regards, > Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global
wrote: option. I don't know about any other technical hurdles in implementing this feature. Please share your thoughts on the same. librados
setting. is there a reason that rgw would only want *some* of its reads localized? are there any implications for read-after-write consistency that we need to worry about here?
Hi Casey,
I'm not aware of any read-after-write or other consistency implications. Read-from-replica is safe for general use since Octopus (OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global librados setting for BALANCE_READS and LOCALIZE_READS seems to be a combination of a) the fact that these flags could only be used in special circumstances in the past and b) the fact that librados doesn't have a generic API for applying a given flag globally like it does for applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A <https://www.google.com/maps/search/315+West+Huron+Street,+Suite+140A+Ann+Arbor,+Michigan+48103?entry=gmail&source=g> Ann Arbor, Michigan 48103 <https://www.google.com/maps/search/315+West+Huron+Street,+Suite+140A+Ann+Arbor,+Michigan+48103?entry=gmail&source=g>
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Josh, I don't follow. First, are you saying that there actually is already a per pool flag, or that the use case "should be" per pool ("business perspective")? If the latter, can you explain why the use case wants to be per pool? Matt On Sun, Jun 18, 2023 at 4:33 AM Josh Salomon <jsalomon@redhat.com> wrote:
One more point - from a business perspective, this flag is per pool and not global so the API should be per pool and not global. Regards,
Josh
On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan@redhat.com>
Hi,
I am planning to support read localise feature for RGW servers
similar to what RBD volumes support. From code reading, it looks to pass "librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this will be per server config
wrote: option. I don't know about any other technical hurdles in implementing this feature. Please share your thoughts on the same.
Thanks and regards, Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados setting. is there a reason that rgw would only want *some* of its reads localized? are there any implications for read-after-write consistency that we need to worry about here?
Hi Casey,
I'm not aware of any read-after-write or other consistency implications. Read-from-replica is safe for general use since Octopus (OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global librados setting for BALANCE_READS and LOCALIZE_READS seems to be a combination of a) the fact that these flags could only be used in special circumstances in the past and b) the fact that librados doesn't have a generic API for applying a given flag globally like it does for applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
participants (7)
-
Casey Bodley
-
Gregory Farnum
-
Ilya Dryomov
-
Jiffin Thottan
-
Josh Salomon
-
Matt Benjamin
-
Sam Just