Hello Dan, I checked if I see the negative "Updated progress" messages and I actually do. At 07:32:00 I started osd.2 again and then ran some ceph -s until rebalance started and then ceph -s was finally hanging. In the mgr log I see this: https://cloud.uni-koblenz.de/s/CegqBT7pi9nobk4 At the end of this rather short log at 07:32:44 the negative Updated progress is visible. So its the "negative progress bug" you described. Thanks Dan and tjanks to all who answered Rainer Am 04.05.21 um 16:21 schrieb Dan van der Ster:
Hi,
This sounds a lot like the negative progress bug we just found last week: https://tracker.ceph.com/issues/50591
That bug makes the mon enter a very long loop rendering a progress bar if the mgr incorrectly sends a message to the mon that the progress is negative. Octopus and later don't have this loop so don't have this bug.
Could you set debug_mgr = 4/5 then check the mgr log for something like this?
mgr[progress] Updated progress to -0.333333333333 (Rebalancing after osd... marked in)
Cheers, Dan
On Tue, May 4, 2021 at 4:10 PM Rainer Krienke <krienke@uni-koblenz.de> wrote:
Hello,
I am playing around with a test ceph 14.2.20 cluster. The cluster consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1, vceph2 and vceph3 are monitors. vceph1 is also mgr.
What I did was quite simple. The cluster is in the state HEALTHY:
vceph2: systemctl stop ceph-osd@2 # let ceph repair until ceph -s reports cluster is healthy again
vceph2: systemctl start ceph-osd@2 # @ 15:39:15, for the logs # cluster reports in cephs -s that 8 OSDs are up and in, then # starts rebalance osd.2
vceph2: ceph -s # hangs forever also if executed on vceph3 or 4 # mon on vceph1 eats 100% CPU permanently, the other mons ~0 %CPU
vceph1: systemctl stop ceph-mon@vceph1 # wait ~30 sec to terminate vceph1: systemctl start ceph-mon@vceph1 # Everything is OK again
I posted the mon-log to: https://cloud.uni-koblenz.de/s/t8tWjWFAobZb5Hy
Strange enough if I set "debug mon 20" before starting the experiment this bug does not show up. I also tried the very same procedure on the same cluster updated to 15.2.11 but I was unable to reproduce this bug in this ceph version.
Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312