16 Feb
2021
16 Feb
'21
11:36 p.m.
Hello,
We have functional ceph swarm with a pair of S3 rgw in front that uses
A.B.C.D domain to be accessed.
Now a new client asks to have access using the domain : E.C.D, but to
already existing buckets. This is not a scenario discussed in the docs.
Apparently, looking at the code and by trying it, rgw does not support
multiple domains for the variable rgw_dns_name.
But reading through parts of the code, I am no dev, and my c++ is 25 years
rusty, I get the impression that maybe we could just add a second pair of
rgw S3 servers that would give service to the same buckets, but using a
different domain.
Am I wrong ? Let's say this works, is this an unconscious behaviour that
the ceph team would remove down the road ?
Is there another solution that I might have missed ? We do not have
multi-zone and there are no plans for it. And Cname (rgw_resolve_cname)
seems to only be of use when using static sites (again, from my poor code
reading abilities).
Thank you
--
**AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont
destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et
peuvent contenir de l’information privilégiée ou confidentielle. Si vous
avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez
le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que
les pièces jointes, le cas échéant. La copie ou la redistribution non
autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne
peut être interprété qu’en conformité avec les lois et règlements qui
régissent les pouvoirs des diverses instances décisionnelles compétentes de
la Ville de Montréal.
22 Feb
22 Feb
7:29 p.m.
You need to enable users with tenants …
https://docs.ceph.com/en/latest/radosgw/multitenancy/
From: Simon Pierre DESROSIERS <simonpierre.desrosiers(a)montreal.ca>
Date: Monday, February 22, 2021 at 7:27 AM
To: ceph-users(a)ceph.io <ceph-users(a)ceph.io>
Subject: [ceph-users] multiple-domain for S3 on rgws with same ceph backend on one zone
Hello,
We have functional ceph swarm with a pair of S3 rgw in front that uses
A.B.C.D domain to be accessed.
Now a new client asks to have access using the domain : E.C.D, but to
already existing buckets. This is not a scenario discussed in the docs.
Apparently, looking at the code and by trying it, rgw does not support
multiple domains for the variable rgw_dns_name.
But reading through parts of the code, I am no dev, and my c++ is 25 years
rusty, I get the impression that maybe we could just add a second pair of
rgw S3 servers that would give service to the same buckets, but using a
different domain.
Am I wrong ? Let's say this works, is this an unconscious behaviour that
the ceph team would remove down the road ?
Is there another solution that I might have missed ? We do not have
multi-zone and there are no plans for it. And Cname (rgw_resolve_cname)
seems to only be of use when using static sites (again, from my poor code
reading abilities).
Thank you
--
**AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont
destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et
peuvent contenir de l’information privilégiée ou confidentielle. Si vous
avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez
le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que
les pièces jointes, le cas échéant. La copie ou la redistribution non
autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne
peut être interprété qu’en conformité avec les lois et règlements qui
régissent les pouvoirs des diverses instances décisionnelles compétentes de
la Ville de Montréal.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 Feb
23 Feb
12:49 a.m.
I'm not sure that the tenant solution is what the OP wants - my reading
is that running under a different tenant allows you have different
tenants use the same bucket and user names but still be distinct, which
wasn't what I thought was meant.
You can however get RGW to accept a list of host names regardless of
domain. Assuming you are using the default zone (adjust if otherwise):
Extract the default zonegroup settings:
radosgw-admin zonegroup get > zg-default
Edit zg-default looking for "hostnames" and adjust to be a list of the
required ones (short and/or FQDN according to what the clients use).
Then install the updated settings:
radosgw-admin zonegroup set --infile zg-default
and restart all the RGW instances.
We do this when fronting RGW with HAProxy, to include the individual
hostnames used by the HAProxy backend.
Regards, Chris
On 22/02/2021 14:29, Freddy Andersen wrote:
> You need to enable users with tenants …
https://docs.ceph.com/en/latest/radosgw/multitenancy/
>
> From: Simon Pierre DESROSIERS <simonpierre.desrosiers(a)montreal.ca>
> Date: Monday, February 22, 2021 at 7:27 AM
> To: ceph-users(a)ceph.io <ceph-users(a)ceph.io>
> Subject: [ceph-users] multiple-domain for S3 on rgws with same ceph backend on one
zone
> Hello,
>
> We have functional ceph swarm with a pair of S3 rgw in front that uses
> A.B.C.D domain to be accessed.
>
> Now a new client asks to have access using the domain : E.C.D, but to
> already existing buckets. This is not a scenario discussed in the docs.
> Apparently, looking at the code and by trying it, rgw does not support
> multiple domains for the variable rgw_dns_name.
>
>
> But reading through parts of the code, I am no dev, and my c++ is 25 years
> rusty, I get the impression that maybe we could just add a second pair of
> rgw S3 servers that would give service to the same buckets, but using a
> different domain.
>
> Am I wrong ? Let's say this works, is this an unconscious behaviour that
> the ceph team would remove down the road ?
>
> Is there another solution that I might have missed ? We do not have
> multi-zone and there are no plans for it. And Cname (rgw_resolve_cname)
> seems to only be of use when using static sites (again, from my poor code
> reading abilities).
>
> Thank you
>
> --
> **AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont
> destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et
> peuvent contenir de l’information privilégiée ou confidentielle. Si vous
> avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez
> le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que
> les pièces jointes, le cas échéant. La copie ou la redistribution non
> autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne
> peut être interprété qu’en conformité avec les lois et règlements qui
> régissent les pouvoirs des diverses instances décisionnelles compétentes de
> la Ville de Montréal.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
24 Feb
24 Feb
6:22 p.m.
Le lun. 22 févr. 2021, à 14 h 50, Chris Palmer <chris.palmer(a)idnet.com> a
écrit :
I'm not sure that the tenant solution is what the
OP wants - my reading
is that running under a different tenant allows you have different
tenants use the same bucket and user names but still be distinct, which
wasn't what I thought was meant.
Our need is for the same user/tenant to use the Bucket using two different
domains. We deployed Ceph's S3 internally with internal domains not
accessible to the outside of our organization with plan to present S3
externally using a homemade api (we have our reasons). Of course, all good
plans have a need for exceptions, hence my question.
You can however get RGW to accept a list of host names regardless of
domain. Assuming you are using the default zone (adjust if otherwise):
Extract the default zonegroup settings:
radosgw-admin zonegroup get > zg-default
Edit zg-default looking for "hostnames" and adjust to be a list of the
required ones (short and/or FQDN according to what the clients use).
Then install the updated settings:
radosgw-admin zonegroup set --infile zg-default
and restart all the RGW instances.
We do this when fronting RGW with HAProxy, to include the individual
hostnames used by the HAProxy backend.
Regards, Chris
Thank you very much, that worked out beautifully, we shall go with that
solution.
On 22/02/2021 14:29, Freddy Andersen wrote:
--
**AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont
destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et
peuvent contenir de l’information privilégiée ou confidentielle. Si vous
avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez
le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que
les pièces jointes, le cas échéant. La copie ou la redistribution non
autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne
peut être interprété qu’en conformité avec les lois et règlements qui
régissent les pouvoirs des diverses instances décisionnelles compétentes de
la Ville de Montréal.
You need to enable users with tenants …
https://docs.ceph.com/en/latest/radosgw/multitenancy/
From: Simon Pierre DESROSIERS <simonpierre.desrosiers(a)montreal.ca>
Date: Monday, February 22, 2021 at 7:27 AM
To: ceph-users(a)ceph.io <ceph-users(a)ceph.io>
Subject: [ceph-users] multiple-domain for S3 on rgws with same ceph
backend on one
zone
Hello,
We have functional ceph swarm with a pair of S3 rgw in front that uses
A.B.C.D domain to be accessed.
Now a new client asks to have access using the domain : E.C.D, but to
already existing buckets. This is not a scenario discussed in the docs.
Apparently, looking at the code and by trying it, rgw does not support
multiple domains for the variable rgw_dns_name.
But reading through parts of the code, I am no dev, and my c++ is 25
years
rusty, I get the impression that maybe we could
just add a second pair of
rgw S3 servers that would give service to the same buckets, but using a
different domain.
Am I wrong ? Let's say this works, is this an unconscious behaviour that
the ceph team would remove down the road ?
Is there another solution that I might have missed ? We do not have
multi-zone and there are no plans for it. And Cname (rgw_resolve_cname)
seems to only be of use when using static sites (again, from my poor code
reading abilities).
Thank you
--
**AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont
destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et
peuvent contenir de l’information privilégiée ou confidentielle. Si vous
avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné,
veuillez
le mentionner immédiatement à l’expéditeur et
effacer ce courriel ainsi
que
les pièces jointes, le cas échéant. La copie ou
la redistribution non
autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne
peut être interprété qu’en conformité avec les lois et règlements qui
régissent les pouvoirs des diverses instances décisionnelles compétentes
de
la Ville de Montréal.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
22 Feb
22 Feb
8:34 p.m.
Den mån 22 feb. 2021 kl 15:27 skrev Simon Pierre DESROSIERS <
simonpierre.desrosiers(a)montreal.ca>gt;:
Hello,
We have functional ceph swarm with a pair of S3 rgw in front that uses
A.B.C.D domain to be accessed.
Now a new client asks to have access using the domain : E.C.D, but to
already existing buckets. This is not a scenario discussed in the docs.
Apparently, looking at the code and by trying it, rgw does not support
multiple domains for the variable rgw_dns_name.
But reading through parts of the code, I am no dev, and my c++ is 25 years
rusty, I get the impression that maybe we could just add a second pair of
rgw S3 servers that would give service to the same buckets, but using a
different domain.
Am I wrong ? Let's say this works, is this an unconscious behaviour that
the ceph team would remove down the road ?
We run this, a LB sends to one pool for one DNS name and to another pool
for a different DNS name, and both rgws serve the "same" buckets.
Since S3 auth v4 the dns name is very much a part of the hash to make your
access work, so whatever the client thinks is the DNS name is what it will
use to make the hash-of-hash-of-hash* combination to auth itself.
We haven't made a huge attempt to break it by doing wacky parallel accesses
from both directions, but it seems to work to move off clients from old name
to new name and the stragglers that will never change will get the old small
LB pool and the clients with a decent config get better service.
In our case the domains are completely different, so not A.B.C.D vs B.C.D
but
rather F.G.H.I instead.
*) SIGNATURE=$(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h
$(HMAC-SHA256s $AWS4SECRET $REQUEST_DATE ) $REQUEST_REGION) $REQUEST_SERVICE)
"aws4_request") $UPLOAD_REQUEST)
--
May the most significant bit of your life be positive.
23 Feb
23 Feb
12:02 a.m.
Le lun. 22 févr. 2021, à 10 h 34, Janne Johansson <icepic.dz(a)gmail.com> a
écrit :
Den mån 22 feb. 2021 kl 15:27 skrev Simon Pierre
DESROSIERS <
simonpierre.desrosiers(a)montreal.ca>gt;:
How can they serve the "same" buckets if they are in different ceph pools
? Am I understanding you correctly ? To me, same bucket means same
objects.
So if I were to deploy a new pair of RGWS with the new domain, would it
create a bunch of new pools in ceph to store its objects or reuse the
preexisting ones ?
Hello,
We have functional ceph swarm with a pair of S3 rgw in front that uses
A.B.C.D domain to be accessed.
Now a new client asks to have access using the domain : E.C.D, but to
already existing buckets. This is not a scenario discussed in the docs.
Apparently, looking at the code and by trying it, rgw does not support
multiple domains for the variable rgw_dns_name.
But reading through parts of the code, I am no dev, and my c++ is 25 years
rusty, I get the impression that maybe we could just add a second pair of
rgw S3 servers that would give service to the same buckets, but using a
different domain.
Am I wrong ? Let's say this works, is this an unconscious behaviour that
the ceph team would remove down the road ?
We run this, a LB sends to one pool for one DNS name and to another pool
for a different DNS name, and both rgws serve the "same" buckets.
Since S3 auth v4 the dns name is very much a part of
the hash to make your
access work, so whatever the client thinks is the DNS name is what it will
use to make the hash-of-hash-of-hash* combination to auth itself.
We haven't made a huge attempt to break it by doing wacky parallel accesses
from both directions, but it seems to work to move off clients from old
name
to new name and the stragglers that will never change will get the old
small
LB pool and the clients with a decent config get better service.
I have a need for parallel access, have you tried it ?
In our case the domains are completely different, so
not A.B.C.D vs B.C.D
but
rather F.G.H.I instead.
*) SIGNATURE=$(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h
$(HMAC-SHA256s $AWS4SECRET $REQUEST_DATE ) $REQUEST_REGION)
$REQUEST_SERVICE) "aws4_request") $UPLOAD_REQUEST)
--
May the most significant bit of your life be positive.
--
**AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont
destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et
peuvent contenir de l’information privilégiée ou confidentielle. Si vous
avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez
le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que
les pièces jointes, le cas échéant. La copie ou la redistribution non
autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne
peut être interprété qu’en conformité avec les lois et règlements qui
régissent les pouvoirs des diverses instances décisionnelles compétentes de
la Ville de Montréal.
1:07 p.m.
Hello,
We have functional ceph swarm with a pair of S3 rgw in front that uses
A.B.C.D domain to be accessed.
Now a new client asks to have access using the domain : E.C.D, but to
already existing buckets. This is not a scenario discussed in the docs.
Apparently, looking at the code and by trying it, rgw does not support
multiple domains for the variable rgw_dns_name.
But reading through parts of the code, I am no dev, and my c++ is 25 years
rusty, I get the impression that maybe we could just add a second pair of
rgw S3 servers that would give service to the same buckets, but using a
different domain.
Am I wrong ? Let's say this works, is this an unconscious behaviour that
the ceph team would remove down the road ?
We run this, a LB sends to one pool for one DNS name and to another pool
for a different DNS name, and both rgws serve the "same" buckets. So if I were to deploy a new pair of RGWS with the new
domain, would it create a bunch of new pools in ceph to store its objects or reuse the
preexisting ones ?
It reuses the old pools. The pool names are not tied to the DNS name
the rgw is using, so it starts looking for .rgw.root and from there
divines which zones and zonegroups exist and (in our case) that the
pools are default.rgw.buckets.index and so on, which is true for both
sets of rgws.
Since S3 auth
v4 the dns name is very much a part of the hash to make your
access work, so whatever the client thinks is the DNS name is what it will
use to make the hash-of-hash-of-hash* combination to auth itself.
We haven't made a huge attempt to break it by doing wacky parallel accesses
from both directions, but it seems to work to move off clients from old name
to new name and the stragglers that will never change will get the old small
LB pool and the clients with a decent config get better service.
I have a need for parallel access, have you tried it ?
24 Feb
24 Feb
6:28 p.m.
Le mar. 23 févr. 2021, à 03 h 07, Janne Johansson <icepic.dz(a)gmail.com> a
écrit :
How can they serve the "same" buckets if they are in different ceph
pools ? Am I understanding you correctly ? To me, same bucket means same
objects.
I mean that a user can go via either one, and it works.
And no, it is not different ceph pools, it is the same ceph pools
underneath, only the rgw name in the conf differs.
domain is
quite important and should be better documented on the various
ways to do it using ceph. This seems to be a distinct advantage of ceph
compared to amazon's S3.
We have not tried since we see it as either you have moved to the new
name or you haven't.
I don't expect this to be a showstopper, since having N+1 rgws in all
other cases is equally susceptible to races regardless of the DNS name
the client used to reach an rgw.
After auth is done, I expect it to be quite similar if your client and
my client ends up on different rgw daemons.
Since using N+1 rgw daemons is used in many many installations, I
consider that use-case tested well enough.
--
May the most significant bit of your life be positive.
--
Simon Pierre Desrosiers
Architecte de Solutions
Direction technologies, architecture, innovation et sécurité
Service des technologies de l'information
___________________________________________
801 rue Brennan, 10e étage
Montréal (Québec)
H3C 0G4
Cellullaire : 514-513-8551
sp.desrosiers(a)ville.montreal.qc.ca
--
**AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont
destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et
peuvent contenir de l’information privilégiée ou confidentielle. Si vous
avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez
le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que
les pièces jointes, le cas échéant. La copie ou la redistribution non
autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne
peut être interprété qu’en conformité avec les lois et règlements qui
régissent les pouvoirs des diverses instances décisionnelles compétentes de
la Ville de Montréal.
>>
Hello,
>> We have functional ceph swarm with a pair of S3 rgw in front that uses
>> A.B.C.D domain to be accessed.
>>
>> Now a new client asks to have access using the domain : E.C.D, but to
>> already existing buckets. This is not a scenario discussed in the
docs.
>> Apparently, looking at the code and by
trying it, rgw does not support
>> multiple domains for the variable rgw_dns_name.
>>
>> But reading through parts of the code, I am no dev, and my c++ is 25
years
>> rusty, I get the impression that maybe we
could just add a second pair
of
>> rgw S3 servers that would give service to
the same buckets, but using a
>> different domain.
>>
>> Am I wrong ? Let's say this works, is this an unconscious behaviour
that
the ceph team would remove down the road ?
We run this, a LB sends to one pool for one DNS name and to another pool
for a different DNS name, and both rgws serve the "same" buckets. So if I were to deploy a new pair of RGWS with
the new domain, would it
create a bunch of new pools in ceph to store its objects
or reuse the
preexisting ones ?
It reuses the old pools. The pool names are not tied to the DNS name
the rgw is using, so it starts looking for .rgw.root and from there
divines which zones and zonegroups exist and (in our case) that the
pools are default.rgw.buckets.index and so on, which is true for both
sets of rgws.
Thank you, it confirms what I thought. In my modest opinion, this multi Since S3 auth
v4 the dns name is very much a part of the hash to make
your
> access work, so whatever the client thinks is
the DNS name is what it
will
> use to make the hash-of-hash-of-hash*
combination to auth itself.
>
> We haven't made a huge attempt to break it by doing wacky parallel
accesses
> from both directions, but it seems to work to
move off clients from old
name
> to new name and the stragglers that will
never change will get the old
small
LB pool
and the clients with a decent config get better service.
I have a need for parallel access, have you tried it ?
1150
days inactive
1158
days old
7 comments
4 participants
participants (4)
-
Chris Palmer -
Freddy Andersen -
Janne Johansson -
Simon Pierre DESROSIERS