16 Jun
2023
16 Jun
'23
12:21 a.m.
Hi,
I am planning to support read localise feature for RGW servers similar to
what RBD volumes support. From code reading, it looks to pass
"librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS.
I have created a tracker issue for this feature
https://tracker.ceph.com/issues/61701, this will be per server config
option. I don't know about any other technical hurdles in implementing this
feature. Please share your thoughts on the same.
Thanks and regards,
Jiffin
16 Jun
16 Jun
8:12 a.m.
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan(a)redhat.com> wrote:
Hi,
I am planning to support read localise feature for RGW servers similar to what RBD
volumes support. From code reading, it looks to pass
"librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I
have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this
will be per server config option. I don't know about any other technical hurdles in
implementing this feature. Please share your thoughts on the same.
Thanks and regards,
Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many
different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados
setting. is there a reason that rgw would only want *some* of its
reads localized? are there any implications for read-after-write
consistency that we need to worry about here?
17 Jun
17 Jun
6:21 a.m.
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley(a)redhat.com> wrote:
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan(a)redhat.com> wrote:
Hi Casey,
I'm not aware of any read-after-write or other consistency
implications. Read-from-replica is safe for general use since Octopus
(OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global
librados setting for BALANCE_READS and LOCALIZE_READS seems to be
a combination of a) the fact that these flags could only be used in
special circumstances in the past and b) the fact that librados doesn't
have a generic API for applying a given flag globally like it does for
applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya
Hi,
I am planning to support read localise feature for RGW servers similar to what RBD
volumes support. From code reading, it looks to pass
"librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I
have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this
will be per server config option. I don't know about any other technical hurdles in
implementing this feature. Please share your thoughts on the same.
Thanks and regards,
Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many
different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados
setting. is there a reason that rgw would only want *some* of its
reads localized? are there any implications for read-after-write
consistency that we need to worry about here? 
18 Jun
18 Jun
1:33 a.m.
One more point - from a business perspective, this flag is per pool and not
global so the API should be per pool and not global.
Regards,
Josh
On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov(a)gmail.com> wrote:
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley
<cbodley(a)redhat.com> wrote:
Hi Casey,
I'm not aware of any read-after-write or other consistency
implications. Read-from-replica is safe for general use since Octopus
(OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global
librados setting for BALANCE_READS and LOCALIZE_READS seems to be
a combination of a) the fact that these flags could only be used in
special circumstances in the past and b) the fact that librados doesn't
have a generic API for applying a given flag globally like it does for
applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan(a)redhat.com>
wrote:
>
> Hi,
>
> I am planning to support read localise feature for RGW servers similar
to
what RBD volumes support. From code reading, it looks to pass
"librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS.
I have created a tracker issue for this feature
https://tracker.ceph.com/issues/61701, this will be per server config
option. I don't know about any other technical hurdles in implementing this
feature. Please share your thoughts on the same.
Thanks and regards,
Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many
different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados
setting. is there a reason that rgw would only want *some* of its
reads localized? are there any implications for read-after-write
consistency that we need to worry about here? 
27 Jun
27 Jun
4:11 p.m.
It would probably make sense to add support in the objecter for
picking up certain flags (such as LOCALIZE_READS) from the pool
config.
-Sam
On Sun, Jun 18, 2023 at 1:34 AM Josh Salomon <jsalomon(a)redhat.com> wrote:
One more point - from a business perspective, this flag is per pool and not global so the
API should be per pool and not global.
Regards,
Josh
On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov(a)gmail.com> wrote:
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley(a)redhat.com> wrote:
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan(a)redhat.com> wrote:
Hi Casey,
I'm not aware of any read-after-write or other consistency
implications. Read-from-replica is safe for general use since Octopus
(OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global
librados setting for BALANCE_READS and LOCALIZE_READS seems to be
a combination of a) the fact that these flags could only be used in
special circumstances in the past and b) the fact that librados doesn't
have a generic API for applying a given flag globally like it does for
applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
Hi,
I am planning to support read localise feature for RGW servers similar to what RBD
volumes support. From code reading, it looks to pass
"librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I
have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this
will be per server config option. I don't know about any other technical hurdles in
implementing this feature. Please share your thoughts on the same.
Thanks and regards,
Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many
different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados
setting. is there a reason that rgw would only want *some* of its
reads localized? are there any implications for read-after-write
consistency that we need to worry about here?
3 Jul
3 Jul
9:24 a.m.
On Tue, Jun 27, 2023 at 7:12 PM Sam Just <sjust(a)redhat.com> wrote:
It would probably make sense to add support in the objecter for
picking up certain flags (such as LOCALIZE_READS) from the pool
config.
-Sam
thanks Sam. Jiffin, would you be willing to look into this Objecter
part? i think it would make your project easier if the feature was
transparent to rgw
On Sun, Jun 18, 2023 at 1:34 AM Josh Salomon <jsalomon(a)redhat.com> wrote:
One more point - from a business perspective, this flag is per pool and not global so the
API should be per pool and not global.
Regards,
Josh
On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov(a)gmail.com> wrote:
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley(a)redhat.com> wrote:
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan(a)redhat.com> wrote:
>
> Hi,
>
> I am planning to support read localise feature for RGW servers similar to what RBD
volumes support. From code reading, it looks to pass
"librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS. I
have created a tracker issue for this feature https://tracker.ceph.com/issues/61701, this
will be per server config option. I don't know about any other technical hurdles in
implementing this feature. Please share your thoughts on the same.
>
> Thanks and regards,
> Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many
different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados
setting. is there a reason that rgw would only want *some* of its
reads localized? are there any implications for read-after-write
consistency that we need to worry about here?
Hi Casey,
I'm not aware of any read-after-write or other consistency
implications. Read-from-replica is safe for general use since Octopus
(OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global
librados setting for BALANCE_READS and LOCALIZE_READS seems to be
a combination of a) the fact that these flags could only be used in
special circumstances in the past and b) the fact that librados doesn't
have a generic API for applying a given flag globally like it does for
applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io 
11:36 a.m.
is there a global/cluster-wide version of such a flag, or to automate
applying it? it seems like the scenarios I hear about are stretched
clusters, and that it would be attractive to not have to manage that by
exception?
Matt
On Mon, Jul 3, 2023 at 12:25 PM Casey Bodley <cbodley(a)redhat.com> wrote:
On Tue, Jun 27, 2023 at 7:12 PM Sam Just
<sjust(a)redhat.com> wrote:
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
It would probably make sense to add support in the objecter for
picking up certain flags (such as LOCALIZE_READS) from the pool
config.
-Sam
thanks Sam. Jiffin, would you be willing to look into this Objecter
part? i think it would make your project easier if the feature was
transparent to rgw
On Sun, Jun 18, 2023 at 1:34 AM Josh Salomon <jsalomon(a)redhat.com>
wrote:
>
> One more point - from a business perspective, this flag is per pool
and not
global so the API should be per pool and not global.
> Regards,
>
> Josh
>
>
> On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov(a)gmail.com>
wrote:
>>
>> On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
>> >
>> > On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan(a)redhat.com>
wrote:
>> > >
>> > > Hi,
>> > >
>> > > I am planning to support read localise feature for RGW servers
similar to what RBD volumes support. From code reading, it looks to pass
"librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS.
I have created a tracker issue for this feature
https://tracker.ceph.com/issues/61701, this will be per server config
option. I don't know about any other technical hurdles in implementing this
feature. Please share your thoughts on the same.
>> > >
>> > > Thanks and regards,
>> > > Jiffin
>> >
>> > thanks Jiffin,
>> >
>> > that sounds simple enough, though rgw issues librados reads in many
>> > different places that would need to manage this flag
>> >
>> > i'm curious why this is a per-op flag, rather than a global librados
>> > setting. is there a reason that rgw would only want *some* of its
>> > reads localized? are there any implications for read-after-write
>> > consistency that we need to worry about here?
>>
>> Hi Casey,
>>
>> I'm not aware of any read-after-write or other consistency
>> implications. Read-from-replica is safe for general use since Octopus
>> (OSD-side issues were fixed in [1]).
>>
>> Ultimately all flags are per-op. The reason that there is no global
>> librados setting for BALANCE_READS and LOCALIZE_READS seems to be
>> a combination of a) the fact that these flags could only be used in
>> special circumstances in the past and b) the fact that librados
doesn't
have a generic API for applying a given flag globally
like it does for
applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io 
11:58 a.m.
For userspace clients I think we can implement a normal config option
(rados_localize_reads) and set it to true in the mon config store, and then
clients will pick it up. But that would be global to all ops — for it to be
pool-based, we’d have to extend the OSDMap with client information and make
some more extensive Objecter changes (and kclient ones). That’s not
impossible but feels a bit weird. :/
-Greg
On Mon, Jul 3, 2023 at 11:36 AM Matt Benjamin <mbenjami(a)redhat.com> wrote:
is there a global/cluster-wide version of such a flag,
or to automate
applying it? it seems like the scenarios I hear about are stretched
clusters, and that it would be attractive to not have to manage that by
exception?
Matt
On Mon, Jul 3, 2023 at 12:25 PM Casey Bodley <cbodley(a)redhat.com> wrote:
On Tue, Jun 27, 2023 at 7:12 PM Sam Just
<sjust(a)redhat.com> wrote:
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
<https://www.google.com/maps/search/315+West+Huron+Street,+Suite+140A+Ann+Arbor,+Michigan+48103?entry=gmail&source=g>
Ann Arbor, Michigan 48103
<https://www.google.com/maps/search/315+West+Huron+Street,+Suite+140A+Ann+Arbor,+Michigan+48103?entry=gmail&source=g>
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
It would probably make sense to add support in the objecter for
picking up certain flags (such as LOCALIZE_READS) from the pool
config.
-Sam
thanks Sam. Jiffin, would you be willing to look into this Objecter
part? i think it would make your project easier if the feature was
transparent to rgw
On Sun, Jun 18, 2023 at 1:34 AM Josh Salomon <jsalomon(a)redhat.com>
wrote:
>
> One more point - from a business perspective, this flag is per pool
and not
global so the API should be per pool and not global.
> Regards,
>
> Josh
>
>
> On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov(a)gmail.com>
wrote:
>>
>> On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
>> >
>> > On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <
jthottan(a)redhat.com> wrote:
>> > >
>> > > Hi,
>> > >
>> > > I am planning to support read localise feature for RGW servers
similar to what RBD volumes support. From code reading, it looks to pass
"librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS.
I have created a tracker issue for this feature
https://tracker.ceph.com/issues/61701, this will be per server config
option. I don't know about any other technical hurdles in implementing this
feature. Please share your thoughts on the same.
>> > >
>> > > Thanks and regards,
>> > > Jiffin
>> >
>> > thanks Jiffin,
>> >
>> > that sounds simple enough, though rgw issues librados reads in many
>> > different places that would need to manage this flag
>> >
>> > i'm curious why this is a per-op flag, rather than a global
librados
>> > setting. is there a reason that rgw
would only want *some* of its
>> > reads localized? are there any implications for read-after-write
>> > consistency that we need to worry about here?
>>
>> Hi Casey,
>>
>> I'm not aware of any read-after-write or other consistency
>> implications. Read-from-replica is safe for general use since
Octopus
>> (OSD-side issues were fixed in [1]).
>>
>> Ultimately all flags are per-op. The reason that there is no global
>> librados setting for BALANCE_READS and LOCALIZE_READS seems to be
>> a combination of a) the fact that these flags could only be used in
>> special circumstances in the past and b) the fact that librados
doesn't
>> have a generic API for applying a given
flag globally like it does
for
>
applying a given snapshot ID/context to all reads/writes.
>
> [1] https://github.com/ceph/ceph/pull/32381
>
> Thanks,
>
> Ilya
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
12:15 p.m.
Josh,
I don't follow. First, are you saying that there actually is already a per
pool flag, or that the use case "should be" per pool ("business
perspective")? If the latter, can you explain why the use case wants to be
per pool?
Matt
On Sun, Jun 18, 2023 at 4:33 AM Josh Salomon <jsalomon(a)redhat.com> wrote:
One more point - from a business perspective, this
flag is per pool and
not global so the API should be per pool and not global.
Regards,
Josh
On Sat, Jun 17, 2023 at 4:22 PM Ilya Dryomov <idryomov(a)gmail.com> wrote:
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
On Fri, Jun 16, 2023 at 5:13 PM Casey Bodley
<cbodley(a)redhat.com> wrote:
Hi Casey,
I'm not aware of any read-after-write or other consistency
implications. Read-from-replica is safe for general use since Octopus
(OSD-side issues were fixed in [1]).
Ultimately all flags are per-op. The reason that there is no global
librados setting for BALANCE_READS and LOCALIZE_READS seems to be
a combination of a) the fact that these flags could only be used in
special circumstances in the past and b) the fact that librados doesn't
have a generic API for applying a given flag globally like it does for
applying a given snapshot ID/context to all reads/writes.
[1] https://github.com/ceph/ceph/pull/32381
Thanks,
Ilya
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
On Fri, Jun 16, 2023 at 3:21 AM Jiffin Thottan <jthottan(a)redhat.com>
wrote:
>
> Hi,
>
> I am planning to support read localise feature for RGW servers
similar to
what RBD volumes support. From code reading, it looks to pass
"librados::OPERATION_LOCALIZE_READS" before sending the request to RADOS.
I have created a tracker issue for this feature
https://tracker.ceph.com/issues/61701, this will be per server config
option. I don't know about any other technical hurdles in implementing this
feature. Please share your thoughts on the same.
Thanks and regards,
Jiffin
thanks Jiffin,
that sounds simple enough, though rgw issues librados reads in many
different places that would need to manage this flag
i'm curious why this is a per-op flag, rather than a global librados
setting. is there a reason that rgw would only want *some* of its
reads localized? are there any implications for read-after-write
consistency that we need to worry about here?
310
days inactive
327
days old
8 comments
7 participants
participants (7)
-
Casey Bodley -
Gregory Farnum -
Ilya Dryomov -
Jiffin Thottan -
Josh Salomon -
Matt Benjamin -
Sam Just