On Tue, May 4, 2021 at 4:21 PM Janne Johansson <icepic.dz@gmail.com> wrote:

Den tis 4 maj 2021 kl 16:10 skrev Rainer Krienke <krienke@uni-koblenz.de>:

...

Hello, I am playing around with a test ceph 14.2.20 cluster. The cluster consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1, vceph2 and vceph3 are monitors. vceph1 is also mgr. What I did was quite simple. The cluster is in the state HEALTHY: vceph2: systemctl stop ceph-osd@2 # let ceph repair until ceph -s reports cluster is healthy again vceph2: systemctl start ceph-osd@2 # @ 15:39:15, for the logs # cluster reports in cephs -s that 8 OSDs are up and in, then # starts rebalance osd.2 vceph2: ceph -s # hangs forever also if executed on vceph3 or 4 # mon on vceph1 eats 100% CPU permanently, the other mons ~0 %CPU

vceph1: systemctl stop ceph-mon@vceph1 # wait ~30 sec to terminate vceph1: systemctl start ceph-mon@vceph1 # Everything is OK again

I posted the mon-log to: https://cloud.uni-koblenz.de/s/t8tWjWFAobZb5Hy

Strange enough if I set "debug mon 20" before starting the experiment this bug does not show up. I also tried the very same procedure on the same cluster updated to 15.2.11 but I was unable to reproduce this bug in this ceph version.

I might have run into the same issue recently, except not in a test but on a live system, also running 14.2.20 like you. We have (for other reasons) some flapping OSDs, and repairs/backfills take a lot of time, and while we might have had slightly less memory on the mons than we should have, they didn't OOM or anything, but we found ourselves in the situation where one mon would eat 100% cpu, not log anything of value at all, and the two others would be all but idling.

Restarting the 100%-using mon would finally allow us to get back into the rest of the recovery.

Same question as above -- does your mgr log negative progress at level 4 ? BTW, if you find that this is indeed what's blocking your mons, you can workaround by setting `ceph progress off` until the fixes are released. -- Dan

On Tue, May 4, 2021 at 4:34 PM Janne Johansson <icepic.dz@gmail.com> wrote:

Den tis 4 maj 2021 kl 16:29 skrev Dan van der Ster <dan@vanderster.com>:

...

BTW, if you find that this is indeed what's blocking your mons, you can workaround by setting `ceph progress off` until the fixes are released.

Most ceph commands (and a few of the ceph daemon commands) would just block, so I guess one would have to have it in previous to getting into this loop.

The negative progress bug will effectively lock up any mon which is preparing a response to `ceph status`. So yeah, I mean, turn `ceph progress off` to avoid hitting this in future. But in a live outage, I guess that one would need to ctrl-c all `ceph status` clients then restart mons or perhaps just wait. -- dan

Hello Dan, I checked if I see the negative "Updated progress" messages and I actually do. At 07:32:00 I started osd.2 again and then ran some ceph -s until rebalance started and then ceph -s was finally hanging. In the mgr log I see this: https://cloud.uni-koblenz.de/s/CegqBT7pi9nobk4 At the end of this rather short log at 07:32:44 the negative Updated progress is visible. So its the "negative progress bug" you described. Thanks Dan and tjanks to all who answered Rainer Am 04.05.21 um 16:21 schrieb Dan van der Ster:

Hi,

This sounds a lot like the negative progress bug we just found last week: https://tracker.ceph.com/issues/50591

That bug makes the mon enter a very long loop rendering a progress bar if the mgr incorrectly sends a message to the mon that the progress is negative. Octopus and later don't have this loop so don't have this bug.

Could you set debug_mgr = 4/5 then check the mgr log for something like this?

mgr[progress] Updated progress to -0.333333333333 (Rebalancing after osd... marked in)

Cheers, Dan

On Tue, May 4, 2021 at 4:10 PM Rainer Krienke <krienke@uni-koblenz.de> wrote:

...

Hello,

I am playing around with a test ceph 14.2.20 cluster. The cluster consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1, vceph2 and vceph3 are monitors. vceph1 is also mgr.

What I did was quite simple. The cluster is in the state HEALTHY:

vceph2: systemctl stop ceph-osd@2 # let ceph repair until ceph -s reports cluster is healthy again

vceph2: systemctl start ceph-osd@2 # @ 15:39:15, for the logs # cluster reports in cephs -s that 8 OSDs are up and in, then # starts rebalance osd.2

vceph2: ceph -s # hangs forever also if executed on vceph3 or 4 # mon on vceph1 eats 100% CPU permanently, the other mons ~0 %CPU

vceph1: systemctl stop ceph-mon@vceph1 # wait ~30 sec to terminate vceph1: systemctl start ceph-mon@vceph1 # Everything is OK again

I posted the mon-log to: https://cloud.uni-koblenz.de/s/t8tWjWFAobZb5Hy

Strange enough if I set "debug mon 20" before starting the experiment this bug does not show up. I also tried the very same procedure on the same cluster updated to 15.2.11 but I was unable to reproduce this bug in this ceph version.

Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312