4 May
2021
4 May
'21
2:10 p.m.
Hello,
I am playing around with a test ceph 14.2.20 cluster. The cluster
consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1,
vceph2 and vceph3 are monitors. vceph1 is also mgr.
What I did was quite simple. The cluster is in the state HEALTHY:
vceph2: systemctl stop ceph-osd@2
# let ceph repair until ceph -s reports cluster is healthy again
vceph2: systemctl start ceph-osd@2 # @ 15:39:15, for the logs
# cluster reports in cephs -s that 8 OSDs are up and in, then
# starts rebalance osd.2
vceph2: ceph -s # hangs forever also if executed on vceph3 or 4
# mon on vceph1 eats 100% CPU permanently, the other mons ~0 %CPU
vceph1: systemctl stop ceph-mon@vceph1 # wait ~30 sec to terminate
vceph1: systemctl start ceph-mon@vceph1 # Everything is OK again
I posted the mon-log to: https://cloud.uni-koblenz.de/s/t8tWjWFAobZb5Hy
Strange enough if I set "debug mon 20" before starting the experiment
this bug does not show up. I also tried the very same procedure on the
same cluster updated to 15.2.11 but I was unable to reproduce this bug
in this ceph version.
Thanks
Rainer
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
4 May
4 May
2:20 p.m.
Den tis 4 maj 2021 kl 16:10 skrev Rainer Krienke <krienke(a)uni-koblenz.de>de>:
Hello,
I am playing around with a test ceph 14.2.20 cluster. The cluster
consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1,
vceph2 and vceph3 are monitors. vceph1 is also mgr.
What I did was quite simple. The cluster is in the state HEALTHY:
vceph2: systemctl stop ceph-osd@2
# let ceph repair until ceph -s reports cluster is healthy again
vceph2: systemctl start ceph-osd@2 # @ 15:39:15, for the logs
# cluster reports in cephs -s that 8 OSDs are up and in, then
# starts rebalance osd.2
vceph2: ceph -s # hangs forever also if executed on vceph3 or 4
# mon on vceph1 eats 100% CPU permanently, the other mons ~0 %CPU
vceph1: systemctl stop ceph-mon@vceph1 # wait ~30 sec to terminate
vceph1: systemctl start ceph-mon@vceph1 # Everything is OK again
I posted the mon-log to: https://cloud.uni-koblenz.de/s/t8tWjWFAobZb5Hy
Strange enough if I set "debug mon 20" before starting the experiment
this bug does not show up. I also tried the very same procedure on the
same cluster updated to 15.2.11 but I was unable to reproduce this bug
in this ceph version.
I might have run into the same issue recently, except not in a test
but on a live system,
also running 14.2.20 like you. We have (for other reasons) some flapping OSDs,
and repairs/backfills take a lot of time, and while we might have had slightly
less memory on the mons than we should have, they didn't OOM or anything,
but we found ourselves in the situation where one mon would eat 100% cpu,
not log anything of value at all, and the two others would be all but idling.
Restarting the 100%-using mon would finally allow us to get back into the rest
of the recovery.
--
May the most significant bit of your life be positive.
2:21 p.m.
Hi,
This sounds a lot like the negative progress bug we just found last
week: https://tracker.ceph.com/issues/50591
That bug makes the mon enter a very long loop rendering a progress bar
if the mgr incorrectly sends a message to the mon that the progress is
negative.
Octopus and later don't have this loop so don't have this bug.
Could you set debug_mgr = 4/5 then check the mgr log for something like this?
mgr[progress] Updated progress to -0.333333333333 (Rebalancing
after osd... marked in)
Cheers, Dan
On Tue, May 4, 2021 at 4:10 PM Rainer Krienke <krienke(a)uni-koblenz.de> wrote:
>
> Hello,
>
> I am playing around with a test ceph 14.2.20 cluster. The cluster
> consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1,
> vceph2 and vceph3 are monitors. vceph1 is also mgr.
>
> What I did was quite simple. The cluster is in the state HEALTHY:
>
> vceph2: systemctl stop ceph-osd@2
> # let ceph repair until ceph -s reports cluster is healthy again
>
> vceph2: systemctl start ceph-osd@2 # @ 15:39:15, for the logs
> # cluster reports in cephs -s that 8 OSDs are up and in, then
> # starts rebalance osd.2
>
> vceph2: ceph -s # hangs forever also if executed on vceph3 or 4
> # mon on vceph1 eats 100% CPU permanently, the other mons ~0 %CPU
>
> vceph1: systemctl stop ceph-mon@vceph1 # wait ~30 sec to terminate
> vceph1: systemctl start ceph-mon@vceph1 # Everything is OK again
>
> I posted the mon-log to: https://cloud.uni-koblenz.de/s/t8tWjWFAobZb5Hy
>
> Strange enough if I set "debug mon 20" before starting the experiment
> this bug does not show up. I also tried the very same procedure on the
> same cluster updated to 15.2.11 but I was unable to reproduce this bug
> in this ceph version.
>
> Thanks
> Rainer
> --
> Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
> 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
> PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
> 1001312
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:28 p.m.
On Tue, May 4, 2021 at 4:21 PM Janne Johansson <icepic.dz(a)gmail.com> wrote:
Den tis 4 maj 2021 kl 16:10 skrev Rainer Krienke <krienke(a)uni-koblenz.de>de>:
Same question as above -- does your mgr log negative progress at level 4 ?
BTW, if you find that this is indeed what's blocking your mons, you
can workaround by setting `ceph progress off` until the fixes are
released.
-- Dan
Hello,
I am playing around with a test ceph 14.2.20 cluster. The cluster
consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1,
vceph2 and vceph3 are monitors. vceph1 is also mgr.
What I did was quite simple. The cluster is in the state HEALTHY:
vceph2: systemctl stop ceph-osd@2
# let ceph repair until ceph -s reports cluster is healthy again
vceph2: systemctl start ceph-osd@2 # @ 15:39:15, for the logs
# cluster reports in cephs -s that 8 OSDs are up and in, then
# starts rebalance osd.2
vceph2: ceph -s # hangs forever also if executed on vceph3 or 4
# mon on vceph1 eats 100% CPU permanently, the other mons ~0 %CPU
vceph1: systemctl stop ceph-mon@vceph1 # wait ~30 sec to terminate
vceph1: systemctl start ceph-mon@vceph1 # Everything is OK again
I posted the mon-log to: https://cloud.uni-koblenz.de/s/t8tWjWFAobZb5Hy
Strange enough if I set "debug mon 20" before starting the experiment
this bug does not show up. I also tried the very same procedure on the
same cluster updated to 15.2.11 but I was unable to reproduce this bug
in this ceph version.
I might have run into the same issue recently, except not in a test
but on a live system,
also running 14.2.20 like you. We have (for other reasons) some flapping OSDs,
and repairs/backfills take a lot of time, and while we might have had slightly
less memory on the mons than we should have, they didn't OOM or anything,
but we found ourselves in the situation where one mon would eat 100% cpu,
not log anything of value at all, and the two others would be all but idling.
Restarting the 100%-using mon would finally allow us to get back into the rest
of the recovery.
2:34 p.m.
Den tis 4 maj 2021 kl 16:29 skrev Dan van der Ster <dan(a)vanderster.com>om>:
BTW, if you find that this is indeed what's
blocking your mons, you
can workaround by setting `ceph progress off` until the fixes are
released.
Most ceph commands (and a few of the ceph daemon commands) would just
block, so I guess one would have to have it in previous to getting
into this loop.
--
May the most significant bit of your life be positive.
2:48 p.m.
On Tue, May 4, 2021 at 4:34 PM Janne Johansson <icepic.dz(a)gmail.com> wrote:
Den tis 4 maj 2021 kl 16:29 skrev Dan van der Ster <dan(a)vanderster.com>om>:
The negative progress bug will effectively lock up any mon which is
preparing a response to `ceph status`.
So yeah, I mean, turn `ceph progress off` to avoid hitting this in
future. But in a live outage, I guess that one would need to ctrl-c
all `ceph status` clients then restart mons or perhaps just wait.
-- dan
BTW, if you find that this is indeed what's
blocking your mons, you
can workaround by setting `ceph progress off` until the fixes are
released.
Most ceph commands (and a few of the ceph daemon commands) would just
block, so I guess one would have to have it in previous to getting
into this loop.
5 May
5 May
6:06 a.m.
Hello Dan,
I checked if I see the negative "Updated progress" messages and I
actually do. At 07:32:00 I started osd.2 again and then ran some ceph -s
until rebalance started and then ceph -s was finally hanging. In the mgr
log I see this:
https://cloud.uni-koblenz.de/s/CegqBT7pi9nobk4
At the end of this rather short log at 07:32:44 the negative Updated
progress is visible. So its the "negative progress bug" you described.
Thanks Dan and tjanks to all who answered
Rainer
Am 04.05.21 um 16:21 schrieb Dan van der Ster:
Hi,
This sounds a lot like the negative progress bug we just found last
week: https://tracker.ceph.com/issues/50591
That bug makes the mon enter a very long loop rendering a progress bar
if the mgr incorrectly sends a message to the mon that the progress is
negative.
Octopus and later don't have this loop so don't have this bug.
Could you set debug_mgr = 4/5 then check the mgr log for something like this?
mgr[progress] Updated progress to -0.333333333333 (Rebalancing
after osd... marked in)
Cheers, Dan
On Tue, May 4, 2021 at 4:10 PM Rainer Krienke <krienke(a)uni-koblenz.de> wrote:
>
> Hello,
>
> I am playing around with a test ceph 14.2.20 cluster. The cluster
> consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1,
> vceph2 and vceph3 are monitors. vceph1 is also mgr.
>
> What I did was quite simple. The cluster is in the state HEALTHY:
>
> vceph2: systemctl stop ceph-osd@2
> # let ceph repair until ceph -s reports cluster is healthy again
>
> vceph2: systemctl start ceph-osd@2 # @ 15:39:15, for the logs
> # cluster reports in cephs -s that 8 OSDs are up and in, then
> # starts rebalance osd.2
>
> vceph2: ceph -s # hangs forever also if executed on vceph3 or 4
> # mon on vceph1 eats 100% CPU permanently, the other mons ~0 %CPU
>
> vceph1: systemctl stop ceph-mon@vceph1 # wait ~30 sec to terminate
> vceph1: systemctl start ceph-mon@vceph1 # Everything is OK again
>
> I posted the mon-log to: https://cloud.uni-koblenz.de/s/t8tWjWFAobZb5Hy
>
> Strange enough if I set "debug mon 20" before starting the experiment
> this bug does not show up. I also tried the very same procedure on the
> same cluster updated to 15.2.11 but I was unable to reproduce this bug
> in this ceph version.
>
> Thanks
> Rainer
> --
> Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
> 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
> PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
> 1001312
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
1116
days inactive
1117
days old
6 comments
3 participants
participants (3)
-
Dan van der Ster -
Janne Johansson -
Rainer Krienke