s3-tests and py3
I'm working on getting the test suite to run on the py3 branch, starting with rados. One of the failures I've run into is that s3-tests runs with python 2 instead of 3. This needs to be converted asap in order for us to make the jump to python 3. I started with a few trivial changes here https://github.com/ceph/s3-tests/pull/333 but I don't really know what I'm doing. Can someone more familiar with s3-tests and/or python take over? One question I had was around the s3tests vs s3tests_boto3 directories... which is the one that matters, and why is the old one still around? Or are they both still used and maintained? My other question is how the branches work. I'm guessing we should actually make the change to teh master branch and then cherry-pick it to ceph-master? I'm surprised how far apart master and ceph-master are. Thanks! sage
The py3 battle goes well. A bunch of tests have been pinned to ubuntu nodes (because they have both py2 and py3). Including s3-tests. However... there are a bunch of rgw tests combine s3-tests (needs ubuntu for py2) and vlagrind (needs centos for valgrind notcmalloc build). We could 1- disable these tests 2- add a ubuntu notcmalloc builder and see if valgrind is better in bionic than it was in the xenial and earlier 3- convert s3-tests to py3 already I'm hoping/begging/pleading that someone will take a serious look at s3-tests and tell me how hard it is. I would love to merge the py3 change tomorrow (Tuesday). Thanks! sage On Mon, 16 Dec 2019, Sage Weil wrote:
I'm working on getting the test suite to run on the py3 branch, starting with rados. One of the failures I've run into is that s3-tests runs with python 2 instead of 3. This needs to be converted asap in order for us to make the jump to python 3.
I started with a few trivial changes here
https://github.com/ceph/s3-tests/pull/333
but I don't really know what I'm doing. Can someone more familiar with s3-tests and/or python take over?
One question I had was around the s3tests vs s3tests_boto3 directories... which is the one that matters, and why is the old one still around? Or are they both still used and maintained?
My other question is how the branches work. I'm guessing we should actually make the change to teh master branch and then cherry-pick it to ceph-master? I'm surprised how far apart master and ceph-master are.
Thanks! sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Mon, Dec 16, 2019 at 6:37 PM Sage Weil <sweil@redhat.com> wrote:
The py3 battle goes well. A bunch of tests have been pinned to ubuntu nodes (because they have both py2 and py3). Including s3-tests.
However... there are a bunch of rgw tests combine s3-tests (needs ubuntu for py2) and vlagrind (needs centos for valgrind notcmalloc build). We could
1- disable these tests 2- add a ubuntu notcmalloc builder and see if valgrind is better in bionic than it was in the xenial and earlier 3- convert s3-tests to py3 already
I'm hoping/begging/pleading that someone will take a serious look at s3-tests and tell me how hard it is.
I would love to merge the py3 change tomorrow (Tuesday).
If we want to merge soon, then ignoring these until they are fixed sounds like the best past forward. Ali mentioned he had these almost done for py3 - I wonder how much is left, I can help out there if needed.
Thanks! sage
On Mon, 16 Dec 2019, Sage Weil wrote:
I'm working on getting the test suite to run on the py3 branch, starting with rados. One of the failures I've run into is that s3-tests runs with python 2 instead of 3. This needs to be converted asap in order for us to make the jump to python 3.
I started with a few trivial changes here
https://github.com/ceph/s3-tests/pull/333
but I don't really know what I'm doing. Can someone more familiar with s3-tests and/or python take over?
One question I had was around the s3tests vs s3tests_boto3 directories... which is the one that matters, and why is the old one still around? Or are they both still used and maintained?
My other question is how the branches work. I'm guessing we should actually make the change to teh master branch and then cherry-pick it to ceph-master? I'm surprised how far apart master and ceph-master are.
Thanks! sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Mon, Dec 16, 2019 at 6:41 PM Alfredo Deza <adeza@redhat.com> wrote:
On Mon, Dec 16, 2019 at 6:37 PM Sage Weil <sweil@redhat.com> wrote:
The py3 battle goes well. A bunch of tests have been pinned to ubuntu nodes (because they have both py2 and py3). Including s3-tests.
However... there are a bunch of rgw tests combine s3-tests (needs ubuntu for py2) and vlagrind (needs centos for valgrind notcmalloc build). We could
1- disable these tests 2- add a ubuntu notcmalloc builder and see if valgrind is better in bionic than it was in the xenial and earlier 3- convert s3-tests to py3 already
I'm hoping/begging/pleading that someone will take a serious look at s3-tests and tell me how hard it is.
I would love to merge the py3 change tomorrow (Tuesday).
If we want to merge soon, then ignoring these until they are fixed sounds like the best past forward.
Ali mentioned he had these almost done for py3 - I wonder how much is left, I can help out there if needed.
Ali, I was looking at the python3 work in https://github.com/ceph/s3-tests/commits/wip-python-3-port How much more do you think we need there? Let me know if I can help out.
Thanks! sage
On Mon, 16 Dec 2019, Sage Weil wrote:
I'm working on getting the test suite to run on the py3 branch, starting with rados. One of the failures I've run into is that s3-tests runs with python 2 instead of 3. This needs to be converted asap in order for us to make the jump to python 3.
I started with a few trivial changes here
https://github.com/ceph/s3-tests/pull/333
but I don't really know what I'm doing. Can someone more familiar with s3-tests and/or python take over?
One question I had was around the s3tests vs s3tests_boto3 directories... which is the one that matters, and why is the old one still around? Or are they both still used and maintained?
My other question is how the branches work. I'm guessing we should actually make the change to teh master branch and then cherry-pick it to ceph-master? I'm surprised how far apart master and ceph-master are.
Thanks! sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 12/16/19 6:41 PM, Alfredo Deza wrote:
On Mon, Dec 16, 2019 at 6:37 PM Sage Weil <sweil@redhat.com> wrote:
The py3 battle goes well. A bunch of tests have been pinned to ubuntu nodes (because they have both py2 and py3). Including s3-tests.
However... there are a bunch of rgw tests combine s3-tests (needs ubuntu for py2) and vlagrind (needs centos for valgrind notcmalloc build). We could
1- disable these tests 2- add a ubuntu notcmalloc builder and see if valgrind is better in bionic than it was in the xenial and earlier 3- convert s3-tests to py3 already
I'm hoping/begging/pleading that someone will take a serious look at s3-tests and tell me how hard it is.
I would love to merge the py3 change tomorrow (Tuesday). If we want to merge soon, then ignoring these until they are fixed sounds like the best past forward.
+1 It looks like we'll also need to look into the ragweed and swift tests, which run in the same rgw/verify jobs with s3tests. Ragweed is pretty small so it shouldn't take much effort, but our swift tests are from a 2011 fork of the openstack/swift repo and I don't think anyone's looked at running those with python3. We'll probably need to pull up our fork, but that means figuring out how to run the tests (I believe they need tox instead of nose?), then auditing all the new test failures. That's a project that we've long put off, and I don't think it's reasonable for that to block your progress here.
Ali mentioned he had these almost done for py3 - I wonder how much is left, I can help out there if needed.
Thanks! sage
On Mon, 16 Dec 2019, Sage Weil wrote:
I'm working on getting the test suite to run on the py3 branch, starting with rados. One of the failures I've run into is that s3-tests runs with python 2 instead of 3. This needs to be converted asap in order for us to make the jump to python 3.
I started with a few trivial changes here
https://github.com/ceph/s3-tests/pull/333
but I don't really know what I'm doing. Can someone more familiar with s3-tests and/or python take over?
One question I had was around the s3tests vs s3tests_boto3 directories... which is the one that matters, and why is the old one still around? Or are they both still used and maintained?
My other question is how the branches work. I'm guessing we should actually make the change to teh master branch and then cherry-pick it to ceph-master? I'm surprised how far apart master and ceph-master are.
Thanks! sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hey everyone, The python 3 port (https://github.com/ceph/s3-tests/pull/337) is working on Ubuntu for both the s3-tests and the s3-tests-website task. Additionally, the RGW team has decided to remove the readwrite and roundtrip s3-test tests. The only thing left to do is to make sure everything works on Centos 8 in teuthology. Best, Ali On Tue, Dec 17, 2019 at 8:40 AM Casey Bodley <cbodley@redhat.com> wrote:
On 12/16/19 6:41 PM, Alfredo Deza wrote:
On Mon, Dec 16, 2019 at 6:37 PM Sage Weil <sweil@redhat.com> wrote:
The py3 battle goes well. A bunch of tests have been pinned to ubuntu nodes (because they have both py2 and py3). Including s3-tests.
However... there are a bunch of rgw tests combine s3-tests (needs ubuntu for py2) and vlagrind (needs centos for valgrind notcmalloc build). We could
1- disable these tests 2- add a ubuntu notcmalloc builder and see if valgrind is better in bionic than it was in the xenial and earlier 3- convert s3-tests to py3 already
I'm hoping/begging/pleading that someone will take a serious look at s3-tests and tell me how hard it is.
I would love to merge the py3 change tomorrow (Tuesday). If we want to merge soon, then ignoring these until they are fixed sounds like the best past forward.
+1
It looks like we'll also need to look into the ragweed and swift tests, which run in the same rgw/verify jobs with s3tests. Ragweed is pretty small so it shouldn't take much effort, but our swift tests are from a 2011 fork of the openstack/swift repo and I don't think anyone's looked at running those with python3. We'll probably need to pull up our fork, but that means figuring out how to run the tests (I believe they need tox instead of nose?), then auditing all the new test failures. That's a project that we've long put off, and I don't think it's reasonable for that to block your progress here.
Ali mentioned he had these almost done for py3 - I wonder how much is left, I can help out there if needed.
Thanks! sage
On Mon, 16 Dec 2019, Sage Weil wrote:
I'm working on getting the test suite to run on the py3 branch,
with rados. One of the failures I've run into is that s3-tests runs with python 2 instead of 3. This needs to be converted asap in order for us to make the jump to python 3.
I started with a few trivial changes here
https://github.com/ceph/s3-tests/pull/333
but I don't really know what I'm doing. Can someone more familiar with s3-tests and/or python take over?
One question I had was around the s3tests vs s3tests_boto3
starting directories...
which is the one that matters, and why is the old one still around? Or are they both still used and maintained?
My other question is how the branches work. I'm guessing we should actually make the change to teh master branch and then cherry-pick it to ceph-master? I'm surprised how far apart master and ceph-master are.
Thanks! sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Just an update, the previously mentioned PR has been merged and we are going to move the s3-tests back to running on Centos in teuthology. Best, Ali On Thu, Jan 9, 2020 at 5:59 PM Ali Maredia <amaredia@redhat.com> wrote:
Hey everyone,
The python 3 port (https://github.com/ceph/s3-tests/pull/337) is working on Ubuntu for both the s3-tests and the s3-tests-website task.
Additionally, the RGW team has decided to remove the readwrite and roundtrip s3-test tests.
The only thing left to do is to make sure everything works on Centos 8 in teuthology.
Best, Ali
On Tue, Dec 17, 2019 at 8:40 AM Casey Bodley <cbodley@redhat.com> wrote:
On 12/16/19 6:41 PM, Alfredo Deza wrote:
On Mon, Dec 16, 2019 at 6:37 PM Sage Weil <sweil@redhat.com> wrote:
The py3 battle goes well. A bunch of tests have been pinned to ubuntu nodes (because they have both py2 and py3). Including s3-tests.
However... there are a bunch of rgw tests combine s3-tests (needs ubuntu for py2) and vlagrind (needs centos for valgrind notcmalloc build). We could
1- disable these tests 2- add a ubuntu notcmalloc builder and see if valgrind is better in bionic than it was in the xenial and earlier 3- convert s3-tests to py3 already
I'm hoping/begging/pleading that someone will take a serious look at s3-tests and tell me how hard it is.
I would love to merge the py3 change tomorrow (Tuesday). If we want to merge soon, then ignoring these until they are fixed sounds like the best past forward.
+1
It looks like we'll also need to look into the ragweed and swift tests, which run in the same rgw/verify jobs with s3tests. Ragweed is pretty small so it shouldn't take much effort, but our swift tests are from a 2011 fork of the openstack/swift repo and I don't think anyone's looked at running those with python3. We'll probably need to pull up our fork, but that means figuring out how to run the tests (I believe they need tox instead of nose?), then auditing all the new test failures. That's a project that we've long put off, and I don't think it's reasonable for that to block your progress here.
Ali mentioned he had these almost done for py3 - I wonder how much is left, I can help out there if needed.
Thanks! sage
On Mon, 16 Dec 2019, Sage Weil wrote:
I'm working on getting the test suite to run on the py3 branch,
with rados. One of the failures I've run into is that s3-tests runs with python 2 instead of 3. This needs to be converted asap in order for us to make the jump to python 3.
I started with a few trivial changes here
https://github.com/ceph/s3-tests/pull/333
but I don't really know what I'm doing. Can someone more familiar with s3-tests and/or python take over?
One question I had was around the s3tests vs s3tests_boto3
starting directories...
which is the one that matters, and why is the old one still around? Or are they both still used and maintained?
My other question is how the branches work. I'm guessing we should actually make the change to teh master branch and then cherry-pick it to ceph-master? I'm surprised how far apart master and ceph-master are.
Thanks! sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (4)
-
Alfredo Deza
-
Ali Maredia
-
Casey Bodley
-
Sage Weil