to anyone interested, I have a PR for review: https://github.com/ceph/ceph/pull/30600 once this is merged, more extensions (exiting or new ones) could be added to the same place. On Wed, Sep 18, 2019 at 6:21 PM Casey Bodley <cbodley@redhat.com> wrote:
I managed to hack up support for our ?allow-unordered extension to s3 object listing, by putting this file [1] in ~/.aws/models/s3/2006-03-01/service-2.sdk-extras.json. With that, a boto3 script [2] was able to call client.list_objects(Bucket=bucketname, AllowUnordered=True), and the ?allow-unordered=true param shows up in the radosgw log!
It looks like the name 'service-2.sdk-extras.json' is the only one that will work automatically, so while we -can- separate our extensions from the upstream boto project, we do still need to bundle them all in the same file. And, I guess, hope that botocore doesn't add a conflicting service-2.sdk-extras.json file for s3 at some point in the future.
[1] https://gist.github.com/cbodley/eaf483f1e372224b130e15757d9e5634
[2] https://gist.github.com/cbodley/7ecbbaf4d220a120b2a893a4abf7ba0d
On 9/18/19 10:20 AM, Casey Bodley wrote:
On 9/18/19 4:27 AM, Yuval Lifshitz wrote:
On Mon, Sep 16, 2019 at 6:56 PM Casey Bodley <cbodley@redhat.com <mailto:cbodley@redhat.com>> wrote:
That's so cool! We've been struggling to find ways to expose s3 extensions, and the client-side validation of clients like boto has made this really difficult.
I'm curious how granular we can make these extensions. For example, if we wanted to expose our ?allow-unordered extension to the ListObjects op, would we have to replace the entire botocore/data/s3/2006-03-01/service-2.json file? Or could we provide a new json file that only specifies a modified ListObjectsRequest structure? If we can make them that granular, it would limit conflicts with upstream boto and be much easier to maintain. It could also make it easy for our users to opt-in to certain extensions.
- tried adding a different file, only with the changed structures, but boto3 ignored it
The comment about 'sdk-extras' in
https://github.com/boto/botocore/blob/develop/botocore/loaders.py#L98-L102
sounds promising for this.
For example, I see a botocore/data/neptune/2014-10-31/service-2.json and a service-2.sdk-extras.json that 'merges' an extra SourceRegion string into two message types defined in service-2.json. That sounds like exactly what we want; it may just be a matter of naming the files correctly, ie 'service-2.*.json'.
- tried using the same file, but including only the modified structures - in this case boto3 took it but failed to find all the other definitions
I think the best option would be to have our own version of the file, and regularly rebase it from botocore. This services-2.json file changes about once a month (see [3]), so, it should not be too bad to track it. Another option would be to change the loaders code to allow for merging the content from multiple json files, but that would be a bigger task, IMO.
Exploring this for s3tests/pubsub in teuthology sounds like a good first step!
where do you think we should put that file?
If you want to start by testing extensions to s3 notifications, the pubsub test driver in qa/tasks/rgw_multisite_tests.py could install a file under /home/ubuntu/.aws/models/.
On 9/16/19 11:16 AM, Yuval Lifshitz wrote: > Hi All, > Recently I added a support for bucket notifications, as part of the > feature the standard (s3) way for filtering notification was extended > with new fields that are not supported by AWS. This poses 2 issues: > - how to test it > - how users should use it > Crafting these messages "by hand" (e.g. by using curl) is not ideal > for both usecases. > > However, according to this [1], there is an easy way to extend the > standard by changing a json file and then placing it under > ~/.aws/models/s3/... > Or, placing it anywhere else, and then setting its path first in > the AWS_DATA_PATH search path (I tried the 1st options and it worked!). > > I guess that this is not the only case where we want to extend/modify > what ASW/boto3 supports. > Would suggest that we add our own versions of the relevant files > (e.g.[2]) into the ceph repo (not sure where?). > This could be used for: > - teuthology infra can take these files and use them, so that the > clients could test our extension natively with boto3 > - we can add documentation around these files, so they are available > to users (maybe deliver them in the installation packages) > > Of course, we will have to track changes to the original file, and > merge from time top time... > > Yuval > > [1] https://github.com/boto/botocore/blob/develop/botocore/loaders.py#L33 > [2] >
https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/se...
> > > _______________________________________________ > Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> > To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io> _______________________________________________ Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io>
[3]
https://github.com/boto/botocore/commits/develop/botocore/data/s3/2006-03-01...