Documentation for S3-ceph tests
Hi all, I was wondering if anyone knew of documentation regarding the s3-test suite (https://github.com/ceph/s3-tests <https://github.com/ceph/s3-tests>). We’re trying to implement some multi-cluster functionality with s3 and part of it is validation testing to make sure everything works as it should. I’m assuming that a lot of what we’re trying to do is already covered by the s3-test suite. I’m just looking for more documentation on what exactly is covered by the tests, and what functionality is tested. I’ve been able to find docs relating to running the tests on a deployed cluster (https://docs.ceph.com/en/latest/dev/developer_guide/running-tests-locally/ <https://docs.ceph.com/en/latest/dev/developer_guide/running-tests-locally/>). But no documentation beyond that about the tests themselves. Just from looking at the code there’s organization by feature using the attribution feature in nose, so I’m wondering if there’s existing knowledge that’s recorded but that I’m missing. Something that breaks down coverage of each attribute, or a list of all the tests being run along with a quick description. - Gavin
On Fri, Mar 12, 2021 at 04:07:22PM -0500, Gavin Chen wrote:
I’ve been able to find docs relating to running the tests on a deployed cluster (https://docs.ceph.com/en/latest/dev/developer_guide/running-tests-locally/ <https://docs.ceph.com/en/latest/dev/developer_guide/running-tests-locally/>). But no documentation beyond that about the tests themselves. Just from looking at the code there’s organization by feature using the attribution feature in nose, so I’m wondering if there’s existing knowledge that’s recorded but that I’m missing. Something that breaks down coverage of each attribute, or a list of all the tests being run along with a quick description. Hi!
I answered you on IRC, but I don't know if you saw it. There isn't a good list of what's covered, and that's one of the goals I have in mind when I included a project in the list of potential GSOC projects for this year: "RGW: S3 SDK COMPATABILITY" on https://ceph.io/contribute/google-summer-of-code/ I was hoping to instrument coverage of Boto, per each test within s3-tests, and that would enable us to improve tags on each test within s3-tests with the S3 API calls that it does consume, and see if that matches what we want to have tested. This would then also see what parts of the S3 API are NOT presently tested by s3-tests. The recent bugs of s3website error handling w/ conditional HTTP requests clearly show there are cases there not properly covered in tests today. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Hi Robin. I’m the Ceph Upstream Docs guy. More detailed Teuthology documentation is likely to be developed later this year. If you have specific requests, let me know and I’ll try to shape the docs to accommodate them. Zac Dover Upstream Docs Ceph On Sun, 14 Mar 2021 at 5:55 am, Robin H. Johnson <robbat2@gentoo.org> wrote:
On Fri, Mar 12, 2021 at 04:07:22PM -0500, Gavin Chen wrote:
I’ve been able to find docs relating to running the tests on a deployed cluster ( https://docs.ceph.com/en/latest/dev/developer_guide/running-tests-locally/ < https://docs.ceph.com/en/latest/dev/developer_guide/running-tests-locally/ ). But no documentation beyond that about the tests themselves. Just from looking at the code there’s organization by feature using the attribution feature in nose, so I’m wondering if there’s existing knowledge that’s recorded but that I’m missing. Something that breaks down coverage of each attribute, or a list of all the tests being run along with a quick description. Hi!
I answered you on IRC, but I don't know if you saw it.
There isn't a good list of what's covered, and that's one of the goals I have in mind when I included a project in the list of potential GSOC projects for this year: "RGW: S3 SDK COMPATABILITY" on https://ceph.io/contribute/google-summer-of-code/
I was hoping to instrument coverage of Boto, per each test within s3-tests, and that would enable us to improve tags on each test within s3-tests with the S3 API calls that it does consume, and see if that matches what we want to have tested. This would then also see what parts of the S3 API are NOT presently tested by s3-tests. The recent bugs of s3website error handling w/ conditional HTTP requests clearly show there are cases there not properly covered in tests today.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Mon, Mar 15, 2021 at 03:22:14PM +1000, John Zachary Dover wrote:
Hi Robin.
I’m the Ceph Upstream Docs guy. We've chatted ;-)
More detailed Teuthology documentation is likely to be developed later this year.
If you have specific requests, let me know and I’ll try to shape the docs to accommodate them. I feel the following tries to hammer Gavin's ask into specific requests.
1. Document individual s3-tests suites/tests outside of the code somehow? 2. Correlate individual s3-tests with what parts of the S3 API they touch. 3. Better cover how Teuthology pieces kick off parts of s3-tests or use s3-tests as a dependency of a further test! qa/tasks/rgw_logsocket.py qa/tasks/s3readwrite.py qa/tasks/s3roundtrip.py qa/tasks/s3tests.py qa/workunits/rgw/run-s3tests.sh #2 in that list ties to the GSOC project I intend to mentor if we get a good student fit. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Hi Robin and John, Thanks for responding on IRC Robin, sorry I seemed to have missed your response! It’s good to hear there are already some other eyes on this and a plan to begin addressing it. I second your list of requests as someone currently going through the tests and trying to catalog them from a high level view. An external source of info documenting individual s3 tests would make the tests much more understandable over the current method of having to read through over all the code itself. Our current plan is to take a catalog of the existing tests to see if it covers some of the tests we’ll need to verify our ceph implementation. This will likely involve at first cataloging the existing tests in some way. Excited to hear about the GSOC project idea! Look forward to it hopefully working out. If there’s anything I can help to contribute towards I’d be happy to help. John, as it stands I would be glad to help contribute towards a summative catalog of the tests if such a place exists in the documentation. Just let me know if there’s a process in place to add in what we come up with. Thanks, - Gavin (831) 515-6028 gchen@linode.com
On Mar 15, 2021, at 2:34 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
On Mon, Mar 15, 2021 at 03:22:14PM +1000, John Zachary Dover wrote:
Hi Robin.
I’m the Ceph Upstream Docs guy. We've chatted ;-)
More detailed Teuthology documentation is likely to be developed later this year.
If you have specific requests, let me know and I’ll try to shape the docs to accommodate them. I feel the following tries to hammer Gavin's ask into specific requests.
1. Document individual s3-tests suites/tests outside of the code somehow? 2. Correlate individual s3-tests with what parts of the S3 API they touch. 3. Better cover how Teuthology pieces kick off parts of s3-tests or use s3-tests as a dependency of a further test! qa/tasks/rgw_logsocket.py qa/tasks/s3readwrite.py qa/tasks/s3roundtrip.py qa/tasks/s3tests.py qa/workunits/rgw/run-s3tests.sh
#2 in that list ties to the GSOC project I intend to mentor if we get a good student fit.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (3)
-
Gavin Chen
-
John Zachary Dover
-
Robin H. Johnson