On Mon, Sep 16, 2019 at 6:56 PM Casey Bodley <cbodley@redhat.com> wrote:
That's so cool! We've been struggling to find ways to expose s3 extensions, and the client-side validation of clients like boto has made this really difficult.
I'm curious how granular we can make these extensions. For example, if we wanted to expose our ?allow-unordered extension to the ListObjects op, would we have to replace the entire botocore/data/s3/2006-03-01/service-2.json file? Or could we provide a new json file that only specifies a modified ListObjectsRequest structure? If we can make them that granular, it would limit conflicts with upstream boto and be much easier to maintain. It could also make it easy for our users to opt-in to certain extensions.
- tried adding a different file, only with the changed structures, but boto3 ignored it - tried using the same file, but including only the modified structures - in this case boto3 took it but failed to find all the other definitions I think the best option would be to have our own version of the file, and regularly rebase it from botocore. This services-2.json file changes about once a month (see [3]), so, it should not be too bad to track it. Another option would be to change the loaders code to allow for merging the content from multiple json files, but that would be a bigger task, IMO.
Exploring this for s3tests/pubsub in teuthology sounds like a good first step!
where do you think we should put that file?
On 9/16/19 11:16 AM, Yuval Lifshitz wrote:
Hi All, Recently I added a support for bucket notifications, as part of the feature the standard (s3) way for filtering notification was extended with new fields that are not supported by AWS. This poses 2 issues: - how to test it - how users should use it Crafting these messages "by hand" (e.g. by using curl) is not ideal for both usecases.
However, according to this [1], there is an easy way to extend the standard by changing a json file and then placing it under ~/.aws/models/s3/... Or, placing it anywhere else, and then setting its path first in the AWS_DATA_PATH search path (I tried the 1st options and it worked!).
I guess that this is not the only case where we want to extend/modify what ASW/boto3 supports. Would suggest that we add our own versions of the relevant files (e.g.[2]) into the ceph repo (not sure where?). This could be used for: - teuthology infra can take these files and use them, so that the clients could test our extension natively with boto3 - we can add documentation around these files, so they are available to users (maybe deliver them in the installation packages)
Of course, we will have to track changes to the original file, and merge from time top time...
Yuval
[1] https://github.com/boto/botocore/blob/develop/botocore/loaders.py#L33 [2]
https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/se...
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
[3] https://github.com/boto/botocore/commits/develop/botocore/data/s3/2006-03-01...