For the benefit of our new folks and for posterity:
Many of our QA tests for CephFS are located in qa/tasks/cephfs/*.
These get run in teuthology with various cluster configurations. What
everyone will need to be able to do is develop these tests locally
without waiting for teuthology so you can rapidly find errors in your
test cases and development builds.
To do this, you need to use the qa/tasks/vstart_runner.py script. This
allows you to use a vstart cluster to execute your tests by providing
the necessary frameworks the tests expect.
On a development box*, build ceph. If you're just testing CephFS, you
can usually get away with a smaller build without rbd/rgw:
/do_cmake.sh -DWITH_PYTHON3:BOOL=ON -DWITH_BABELTRACE=OFF
-DWITH_MANPAGE=OFF -DWITH_RBD=OFF -DWITH_RADOSGW=OFF && time (cd build
&& make -j24 CMAKE_BUILD_TYPE=Debug -k)
Next, build teuthology:
git clone https://github.com/ceph/teuthology.git && cd teuthology &&
virtualenv ./venv && source venv/bin/activate && pip install --upgrade
pip && pip install -r requirements.txt && python setup.py develop
Next, start a vstart cluster:
cd ceph/build && env MDS=3 ../src/vstart.sh -d -b -l -n --without-dashboard
Finally, run vstart_runner:
python2 ../qa/tasks/vstart_runner.py --interactive
tasks.cephfs.test_snapshots.TestSnapshots
^ That's an example test. The format is based on the directory
structure of qa/tasks/cephfs/test_snapshots. The final part is the
class we're testing, TestSnapshots. This invocation of
vstart_runner.py will run every test in TestSnapshots, methods
beginning with "test_". If you want to run a specific test, then we
could do:
python2 ../qa/tasks/vstart_runner.py --interactive
tasks.cephfs.test_snapshots.TestSnapshots.test_snapclient_cache
Please give the above a try sometime soon so you know how to do it and
we can resolve any problems. This is an important skill to have for
developing CephFS.
* Hopefully you're using one of the beefy development boxes that make
compiling Ceph fast. I recommend one of the senta boxes like
senta03.front.sepia.ceph.com.
--
Patrick Donnelly
+ceph-fs(a)ceph.io for posterity and knowledge sharing
Once you get access to the teuthology box:
https://wiki.sepia.ceph.com/doku.php?id=vpnaccess
... you can get started with QA testing. Review these docs:
http://docs.ceph.com/teuthology/docs/
and
https://github.com/ceph/teuthology/tree/master/docs
This is also a great recent presentation done by Greg Farnum on the
history of teuthology and how it works:
https://www.youtube.com/watch?v=gj1OXrKdSrs
You will need to `git clone` the teuthology repository on the
teuthology node to launch tests. Here's what I run to set that up:
cd ~/teuthology && (virtualenv ./venv && source venv/bin/activate &&
pip install --upgrade pip && pip install -r requirements.txt && python
setup.py develop)
Then you should be able to run tests by doing:
pdonnell@teuthology ~$ source ~/teuthology/venv/bin/activate
(venv) pdonnell@teuthology ~$ teuthology-suite --machine-type smithi
--email pdonnell(a)redhat.com -p 9 --suite fs --ceph master --ceph-repo
https://github.com/ceph/ceph.git --suite-branch master --suite-repo
https://github.com/ceph/ceph.git --filter cephfs-shell
2019-03-18 15:08:44,206.206 INFO:teuthology.suite:Using random seed=3675
2019-03-18 15:08:44,207.207 INFO:teuthology.suite.run:kernel sha1: distro
2019-03-18 15:08:44,626.626 INFO:teuthology.suite.run:ceph sha1:
9c024bafa3570dee401d3fcaf04754376f285eca
2019-03-18 15:08:44,812.812 INFO:teuthology.suite.run:ceph version:
14.1.1-196.g9c024ba
2019-03-18 15:08:44,812.812 INFO:teuthology.suite.run:teuthology branch: master
2019-03-18 15:08:45,227.227 INFO:teuthology.suite.run:ceph branch:
master 9c024bafa3570dee401d3fcaf04754376f285eca
2019-03-18 15:08:45,229.229
INFO:teuthology.repo_utils:/home/pdonnell/src/github.com_ceph_ceph_master
was just updated; assuming it is current
2019-03-18 15:08:45,229.229 INFO:teuthology.repo_utils:Resetting repo
at /home/pdonnell/src/github.com_ceph_ceph_master to branch master
2019-03-18 15:08:45,453.453 INFO:teuthology.suite.run:Suite fs in
/home/pdonnell/src/github.com_ceph_ceph_master/qa/suites/fs generated
546 jobs (not yet filtered)
Job scheduled with name
pdonnell-2019-03-18_15:08:44-fs-master-distro-basic-smithi and ID
3744281
2019-03-18 15:08:46,372.372 INFO:teuthology.suite.run:Scheduling
fs/basic_functional/{begin.yaml clusters/1-mds-4-client-coloc.yaml
conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml
objectstore/bluestore-ec-root.yaml overrides/{frag_enable.yaml
no_client_pidfile.yaml whitelist_health.yaml
whitelist_wrongly_marked_down.yaml}
supported-random-distros$/{rhel_latest.yaml}
tasks/cephfs-shell/{py-3.yaml simple.yaml test.yaml}}
Job scheduled with name
pdonnell-2019-03-18_15:08:44-fs-master-distro-basic-smithi and ID
3744282
2019-03-18 15:08:47,116.116 INFO:teuthology.suite.run:Scheduling
fs/basic_functional/{begin.yaml clusters/1-mds-4-client-coloc.yaml
conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml
objectstore/bluestore-bitmap.yaml overrides/{frag_enable.yaml
no_client_pidfile.yaml whitelist_health.yaml
whitelist_wrongly_marked_down.yaml}
supported-random-distros$/{ubuntu_16.04.yaml}
tasks/cephfs-shell/{py-3.yaml simple.yaml test.yaml}}
Job scheduled with name
pdonnell-2019-03-18_15:08:44-fs-master-distro-basic-smithi and ID
3744283
2019-03-18 15:08:47,867.867 INFO:teuthology.suite.run:Suite fs in
/home/pdonnell/src/github.com_ceph_ceph_master/qa/suites/fs scheduled
2 jobs.
2019-03-18 15:08:47,867.867 INFO:teuthology.suite.run:544/546 jobs
were filtered out.
Job scheduled with name
pdonnell-2019-03-18_15:08:44-fs-master-distro-basic-smithi and ID
3744284
2019-03-18 15:08:48,584.584 INFO:teuthology.suite.run:Test results
viewable at http://pulpito.front.sepia.ceph.com:80/pdonnell-2019-03-18_15:08:44-fs-mast…
You can see the run here:
http://pulpito.ceph.com/pdonnell-2019-03-18_15:08:44-fs-master-distro-basic…
Few things to note about the `teuthology-suite` command above:
- I used "-p 9" to set a very high priority (lower -p is higher
priority) since I know I'm only running ~2 jobs and I want the results
quickly. If you do testing with a lot of jobs (~50+), use something
like "-p 100".
- The fs suite is located in qa/suites/fs/. As you probably learned
from the docs above, this is just a collection of yaml fragments that
are combined to create a number of test configurations. When we use
"--filter cephfs-shell", we're only selecting test configurations that
include the "tasks/cephfs-shell/..." fragments.
- We use "--ceph master --ceph-repo https://github.com/ceph/ceph.git"
when we're doing tests against a release branch (e.g. nautilus) or
master. Otherwise, the repo defaults to
https://github.com/ceph/ceph-ci.git. ceph-ci exists as a dumping
ground for development branches but also let's us protect the ceph.git
repo from experiencing accidents with stronger permissions. When you
make changes to ceph, you will be pushing your development branches to
ceph-ci.git where they will automatically get built by shaman:
https://shaman.ceph.com/builds/ceph/
- When you do testing with cephfs-shell, you'll probably be making
changes to the qa/ files. You can actually rapidly test those without
waiting for a 2 hour rebuild of ceph by pushing the branch to your
github fork of ceph and then doing:
teuthology-suite --machine-type smithi --email pdonnell(a)redhat.com -p
9 --suite fs --ceph master --ceph-repo
https://github.com/ceph/ceph.git --suite-branch your-dev-branch
--suite-repo https://github.com/yourusername/ceph.git --filter
cephfs-shell
Then your qa changes will be used with the current master branch of
Ceph. If you need to actually change src/tools/cephfs/cephfs-shell,
you will need to rebuild ceph.
--
Patrick Donnelly
I need to do a bit of testing with cephfs and would like to avoid having
to use FUSE or write C calls into libcephfs. cephfs-shell seems perfect
for this, but I haven't quite figured out how to run it out of the build
directory.
Is there a way to run cephfs-shell from a ceph build directory (i.e. w/o
make installing it)? Are there env vars that can be set to make it find
the right bits in the right places?
Thanks,
--
Jeff Layton <jlayton(a)redhat.com>
