For developers submitting jobs using teuthology, we now have
recommendations on what priority level to use:
https://docs.ceph.com/docs/master/dev/developer_guide/#testing-priority
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
hi folks,
we just migrated ceph:teuthology and all tests under qa/ in ceph:ceph
to python3. and from now on, the teuthology-worker runs in a python3
environment by default unless specified otherwise using
"--teuthology-branch py2".
which means:
- we need to write tests in python3 in master now
- teuthology should be python3 compatible.
- teuthology bug fixes should be backported to "py2" branch.
if you run into any issues related to python3 due to the above
changes, please let me know. and i will try to fix it ASAP.
currently, the tests under qa/ directories in ceph:ceph master branch
are python2 and python3 compatible. but since we've moved to python3,
there is no need to be python2 compatible anymore. since the sepia lab
is still using ubuntu xenial, we cannot use features offered by
python3.6 at this moment yet. but we do plan to upgrade the OS to
bionic soon. before that happens, the tests need to be compatible with
Python3.5.
the next step is to
- drop python2 support in ceph:ceph master branch, and
- drop python2 support in ceph:teuthology master.
- backport python3 compatible changes to octopus and nautilus to ease
the pain of backport
--
Regards
Kefu Chai
Adding the right dev list.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Wed, May 20, 2020 at 12:40 AM Robert LeBlanc <robert(a)leblancnet.us> wrote:
>
> We upgraded our Jewel cluster to Nautilus a few months ago and I've noticed that op behavior has changed. This is an HDD cluster (NVMe journals and NVMe CephFS metadata pool) with about 800 OSDs. When on Jewel and running WPQ with the high cut-off, it was rock solid. When we had recoveries going on it barely dented the client ops and when the client ops on the cluster went down the backfills would run as fast as the cluster could go. I could have max_backfills set to 10 and the cluster performed admirably.
> After upgrading to Nautilus the cluster struggles with any kind of recovery and if there is any significant client write load the cluster can get into a death spiral. Even heavy client write bandwidth (3-4 GB/s) can cause the heartbeat checks to raise, blocked IO and even OSDs becoming unresponsive.
> As the person who wrote the WPQ code initially, I know that it was fair and proportional to the op priority and in Jewel it worked. It's not working in Nautilus. I've tweaked a lot of things trying to troubleshoot the issue and setting the recovery priority to 1 or zero barely makes any difference. My best estimation is that the op priority is getting lost before reaching the WPQ scheduler and is thus not prioritizing and dispatching ops correctly. It's almost as if all ops are being treated the same and there is no priority at all.
> Unfortunately, I do not have the time to set up the dev/testing environment to track this down and we will be moving away from Ceph. But I really like Ceph and want to see it succeed. I strongly suggest that someone look into this because I think it will resolve a lot of problems people have had on the mailing list. I'm not sure if a bug was introduced with the other queues that touches more of the op path or if something in the op path restructuring that changed how things work (I know that was being discussed around the time that Jewel was released). But my guess is that it is somewhere between the op being created and being received into the queue.
> I really hope that this helps in the search for this regression. I spent a lot of time studying the issue to come up with WPQ and saw it work great when I switched this cluster from PRIO to WPQ. I've also spent countless hours studying how it's changed in Nautilus.
>
> Thank you,
> Robert LeBlanc
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Hi,
(In nautilus) I noticed that the acting set for a pg in
acting+remapped+backfilling (or backfill_wait) can violate the failure
domain rule.
We have a 3x pool replicated across racks. Following some host outage
I noticed several PGs like:
75.200 7570 0 7570 0 31583745536 0
0 3058 active+remapped+backfill_wait 55m
3208334'13480873 3208334:23611310 [596,502,717]p596
[596,502,424]p596 2020-05-28 05:44:42.604307 2020-05-28
05:44:42.604307
Checking those up and acting sets, I have:
OK: PG 75.200 has no failure domain problem in up set [596, 502,
717] with racks ['BA09', 'BA10', 'BA12']
WARN: PG 75.200 has a failure domain problem in acting set [596, 502,
424] with racks ['BA09', 'BA10', 'BA09']
If BA09 were to go down, the pg would drop below min_size.
(The full pg query output is at https://pastebin.com/rrEQMnyC)
Is this https://tracker.ceph.com/issues/3360 ?
# ceph osd dump | grep 75.200
pg_temp 75.200 [596,502,424]
Shouldn't we mark a PG degraded or similar to raise HEALTH_WARN if the
failure domain is violated?
Cheers, Dan
In the services class svc_bi_rados.cc functions like read_stats do these
functions run in parellel.
Im setting a break point at this read_stats function then run my req, my
req gets finished and I got a successful res, but after 2 or 3 sec my gdb
hits the break point, Is the thread running in detached state from main
program.
Is there any work around if I want to check whether the function is being
called after finishing the req or during its execution?
We’re happy to announce the availability of the third Octopus stable
release series. This release mainly is a workaround for a potential OSD
corruption in v15.2.2. We advise users to upgrade to v15.2.3 directly.
For users running v15.2.2 please execute the following::
ceph config set osd bluefs_preextend_wal_files false
Changelog
~~~~~~~~~
* bluestore: common/options.cc: disable bluefs_preextend_wal_files
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-15.2.3.tar.gz
* For packages, see
http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: d289bbdec69ed7c1f516e0a093594580a76b78d0
Yeah, we'll make sure the container images are built before announcing it.
On 5/28/20 1:30 PM, David Orman wrote:
> Due to the impact/severity of this issue, can we make sure the docker
> images are pushed simultaneously for those of us using
> cephadm/containers (with the last release, there was a significant
> delay)? I'm glad the tempfix is being put into place in short-order,
> thank you for the expedient turnaround and understanding.
>
> On Thu, May 28, 2020 at 3:03 PM Josh Durgin <jdurgin(a)redhat.com
> <mailto:jdurgin@redhat.com>> wrote:
>
> Hi Paul, we're planning to release 15.2.3 with the workaround [0]
> tomorrow, so folks don't have to worry as we work on a more complete
> fix.
>
> Josh
>
> [0] https://github.com/ceph/ceph/pull/35293
>
> On 5/27/20 6:27 AM, Paul Emmerich wrote:
> > Hi,
> >
> > since this bug may lead to data loss when several OSDs crash at
> the same
> > time (e.g., after a power outage): can we pull the release from the
> > mirrors and docker hub?
> >
> > Paul
> >
> > --
> > Paul Emmerich
> >
> > Looking for help with your Ceph cluster? Contact us at
> https://croit.io
> >
> > croit GmbH
> > Freseniusstr. 31h
> > 81247 München
> > www.croit.io <http://www.croit.io> <http://www.croit.io>
> > Tel: +49 89 1896585 90
> >
> >
> > On Wed, May 20, 2020 at 7:18 PM Josh Durgin <jdurgin(a)redhat.com
> <mailto:jdurgin@redhat.com>
> > <mailto:jdurgin@redhat.com <mailto:jdurgin@redhat.com>>> wrote:
> >
> > Hi folks, at this time we recommend pausing OSD upgrades to
> 15.2.2.
> >
> > There have been a couple reports of OSDs crashing due to rocksdb
> > corruption after upgrading to 15.2.2 [1] [2]. It's safe to
> upgrade
> > monitors and mgr, but OSDs and everything else should wait.
> >
> > We're investigating and will get a fix out as soon as we can. You
> > can follow progress on this tracker:
> >
> > https://tracker.ceph.com/issues/45613
> >
> > Josh
> >
> > [1]
> >
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/CX5PRFGL6UB…
> > [2]
> >
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/CWN7BNPGSRB…
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> <mailto:ceph-users@ceph.io>
> > <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>>
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> <mailto:ceph-users-leave@ceph.io>
> > <mailto:ceph-users-leave@ceph.io
> <mailto:ceph-users-leave@ceph.io>>
> >
> >
> > _______________________________________________
> > Dev mailing list -- dev(a)ceph.io <mailto:dev@ceph.io>
> > To unsubscribe send an email to dev-leave(a)ceph.io
> <mailto:dev-leave@ceph.io>
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> <mailto:ceph-users@ceph.io>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> <mailto:ceph-users-leave@ceph.io>
>
Hi all,
Due to rate limiting we're seeing from docker.io and quay.io's downtime
this week, I created a container mirror and a quay registry for our
Sepia lab and CI to use.
See https://tracker.ceph.com/issues/45343#note-8
--
David Galloway
Systems Administrator, RDU
Ceph Engineering
IRC: dgalloway
