11 May
2020
11 May
'20
3:52 p.m.
Hi all,
another client-load induced meltdown. It is just starting and I hope we get it under
control. This time, its the MGRs failing under the load. It looks like thay don't
manage to get their beacons to the mons and are kicked out as unresponsive. However, the
processes are fine and up. Its just an enormous load.
I'm trying to increase
# ceph config set global mon_mgr_beacon_grace 90
but the command doesn't complete. I guess because all the MGRs are out. Is there any
way to force the MONs *not* to mark MGRs as unresponsive?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
11 May
11 May
4:03 p.m.
OK, the command finally executed and it looks like the cluster is running stable for now.
However, I'm afraid that 90s might not be sustainable.
Questions: Can I leave the beacon_grace at 90s? Is there a better parameter to set? Why is
the MGR getting overloaded on a rather small cluster with 160 OSDs? How does this scale?
Some more info:
Here is a pool stats output for the workload just after the beacon grace increase
succeeded:
pool con-fs2-meta1 id 12
client io 1007 KiB/s rd, 1.1 MiB/s wr, 4 op/s rd, 421 op/s wr
pool con-fs2-meta2 id 13
client io 0 B/s wr, 0 op/s rd, 21 op/s wr
pool con-fs2-data id 14
client io 172 MiB/s rd, 1.8 GiB/s wr, 70 op/s rd, 4.19 kop/s wr
This is well over the limit of the aggregated IOP/s for the fs data pool and might have
been even higher before I got the prompt back. The fs layout can be seen here, taken after
IO went down:
con-fs2 - 1674 clients
=======
+------+----------------+---------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+----------------+---------+---------------+-------+-------+
| 0 | active | ceph-08 | Reqs: 119 /s | 5157k | 4673k |
| 0-s | standby-replay | ceph-12 | Evts: 437 /s | 27.5k | 21.4k |
+------+----------------+---------+---------------+-------+-------+
+---------------------+----------+-------+-------+
| Pool | type | used | avail |
+---------------------+----------+-------+-------+
| con-fs2-meta1 | metadata | 175M | 954G |
| con-fs2-meta2 | data | 0 | 954G |
| con-fs2-data | data | 131T | 858T |
| con-fs2-data-ec-ssd | data | 177G | 2289G |
+---------------------+----------+-------+-------+
con-fs2-meta2 is the default data pool not used for storing anything.
Best regads and thanks for any pointers.
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: 11 May 2020 14:52:05
To: ceph-users
Subject: [ceph-users] Yet another meltdown starting
Hi all,
another client-load induced meltdown. It is just starting and I hope we get it under
control. This time, its the MGRs failing under the load. It looks like thay don't
manage to get their beacons to the mons and are kicked out as unresponsive. However, the
processes are fine and up. Its just an enormous load.
I'm trying to increase
# ceph config set global mon_mgr_beacon_grace 90
but the command doesn't complete. I guess because all the MGRs are out. Is there any
way to force the MONs *not* to mark MGRs as unresponsive?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:50 p.m.
Hi Frank,
On 5/11/20 3:03 PM, Frank Schilder wrote:
OK, the command finally executed and it looks like the
cluster is
running stable for now. However, I'm afraid that 90s might not be
sustainable.
Questions: Can I leave the beacon_grace at 90s? Is there a better
parameter to set? Why is the MGR getting overloaded on a rather small
cluster with 160 OSDs? How does this scale?
I wonder if https://tracker.ceph.com/issues/45439 might be related to
what you're observing here?
In this issue, Andras suggests: "Increasing mgr_stats_period to 15
seconds reduces the load and brings ceph-mgr back to responsive again."
Maybe that helps?
Lenz
--
SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg
GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)
8:43 p.m.
For everyone who does not want to read the details below: I run now with (dramatically?)
increased beacon grace periods for OSD (3600s) and MGR (90s) beacons and am wondering what
the downside of this is and if there are better tuning parameters for my issues.
---
Hi Lenz,
I'm wondering about this as well. I was following this and other MGR threads about
dashboard crashes with great interest.
It is not exactly the same issue, we are still on mimic and under normal circumstances,
all queues are empty. However, I also have the feeling we are hitting a very specific
piece of code that has comparably large execution time for little input. I was actually
surprised to read that python code is involved in processing high-frequency events in a
non-distributed way.
It is said that an MGR is not a single point of failure, but this seems not to be true in
the full meaning. If some workload is not distributed, but processed by only one instance
(in active-passive way), then
- it does not scale,
- it becomes an effective single point of failure as every instance suffers from the same
restriction and
- fail-over will not help as we see a failure of a healthy instance due to load.
This seems to be exactly what I'm observing. What started was a loop of doom:
- active MGR fails to send beacon in time,
- MON marks MGR out and elects next available standby
- new MGR takes over, is hit by the same problem and does not send its beacon in time,
- the other MGR in the meantime reconnects, but slower than the next MGR is marked out.
After a while, the MONs were cyclically kicking out the active MGR. Each MGR stayed only
active for the beacon grace period and was then thrown out. Note that none of the MGR
processes crashed or died. Everything was up and running. I observed a client-load induced
evict-reconnect cycle.
What is suspicious is that I cannot see any significant increase of load or network
traffic on the MGR node during the critical time before the incident (there was, of
course, a huge increase of client traffic to the limit of the hardware). It looks like
something completely under radar, a tiny bit of some very specific processing has a huge
impact under high client load, like the difference between compiled and interpreted code
in the issue you mentioned.
There also seem to be accumulative issues like memory leaks. I observe regular crashes of
the dashboard and the dashboard creates quite a large load for not much as well.
I will put this case on the list of reference of a future thread "Cluster outage due
to client IO" I'm preparing. All the major issues I was observing lately have to
do specifically with beacons sent to the MONs. I did not see any heartbeat failures. The
cluster was physically healthy (everything up and running) but not logically (MONs did not
get required info in time) and increasing the beacon grace periods immediately restored
logical cluster health. It looks like beacons are processed in a different way than
heartbeats and that there is a critical bottleneck somewhere.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Lenz Grimmer <lgrimmer(a)suse.com>
Sent: 11 May 2020 17:50:34
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Yet another meltdown starting
Hi Frank,
On 5/11/20 3:03 PM, Frank Schilder wrote:
OK, the command finally executed and it looks like the
cluster is
running stable for now. However, I'm afraid that 90s might not be
sustainable.
Questions: Can I leave the beacon_grace at 90s? Is there a better
parameter to set? Why is the MGR getting overloaded on a rather small
cluster with 160 OSDs? How does this scale?
I wonder if https://tracker.ceph.com/issues/45439 might be related to
what you're observing here?
In this issue, Andras suggests: "Increasing mgr_stats_period to 15
seconds reduces the load and brings ceph-mgr back to responsive again."
Maybe that helps?
Lenz
--
SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg
GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)
9 p.m.
For assessing the criticality of the MGR beacon loop of doom outage, during my somewhat
desperate attempts to get this under control, I saw this here:
-------------------------------------
[root@gnosis ~]# ceph status
cluster:
id: ---
health: HEALTH_WARN
no active mgr
Reduced data availability: 2545 pgs inactive
too few PGs per OSD (9 < min 30)
services:
mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03
mgr: no daemons active
mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay
osd: 288 osds: 268 up, 268 in
data:
pools: 10 pools, 2545 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs: 100.000% pgs unknown
2545 unknown
-------------------------------------
Is an active MGR actually required for I/O or is this output just due to not having an MGR
deliver the numbers? Well, is says HEALTH_WARN, so I really hope this is just missing
stats and not complete service outage.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: 11 May 2020 19:43:24
To: Lenz Grimmer; ceph-users(a)ceph.io
Subject: [ceph-users] Re: Yet another meltdown starting
For everyone who does not want to read the details below: I run now with (dramatically?)
increased beacon grace periods for OSD (3600s) and MGR (90s) beacons and am wondering what
the downside of this is and if there are better tuning parameters for my issues.
---
Hi Lenz,
I'm wondering about this as well. I was following this and other MGR threads about
dashboard crashes with great interest.
It is not exactly the same issue, we are still on mimic and under normal circumstances,
all queues are empty. However, I also have the feeling we are hitting a very specific
piece of code that has comparably large execution time for little input. I was actually
surprised to read that python code is involved in processing high-frequency events in a
non-distributed way.
It is said that an MGR is not a single point of failure, but this seems not to be true in
the full meaning. If some workload is not distributed, but processed by only one instance
(in active-passive way), then
- it does not scale,
- it becomes an effective single point of failure as every instance suffers from the same
restriction and
- fail-over will not help as we see a failure of a healthy instance due to load.
This seems to be exactly what I'm observing. What started was a loop of doom:
- active MGR fails to send beacon in time,
- MON marks MGR out and elects next available standby
- new MGR takes over, is hit by the same problem and does not send its beacon in time,
- the other MGR in the meantime reconnects, but slower than the next MGR is marked out.
After a while, the MONs were cyclically kicking out the active MGR. Each MGR stayed only
active for the beacon grace period and was then thrown out. Note that none of the MGR
processes crashed or died. Everything was up and running. I observed a client-load induced
evict-reconnect cycle.
What is suspicious is that I cannot see any significant increase of load or network
traffic on the MGR node during the critical time before the incident (there was, of
course, a huge increase of client traffic to the limit of the hardware). It looks like
something completely under radar, a tiny bit of some very specific processing has a huge
impact under high client load, like the difference between compiled and interpreted code
in the issue you mentioned.
There also seem to be accumulative issues like memory leaks. I observe regular crashes of
the dashboard and the dashboard creates quite a large load for not much as well.
I will put this case on the list of reference of a future thread "Cluster outage due
to client IO" I'm preparing. All the major issues I was observing lately have to
do specifically with beacons sent to the MONs. I did not see any heartbeat failures. The
cluster was physically healthy (everything up and running) but not logically (MONs did not
get required info in time) and increasing the beacon grace periods immediately restored
logical cluster health. It looks like beacons are processed in a different way than
heartbeats and that there is a critical bottleneck somewhere.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Lenz Grimmer <lgrimmer(a)suse.com>
Sent: 11 May 2020 17:50:34
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Yet another meltdown starting
Hi Frank,
On 5/11/20 3:03 PM, Frank Schilder wrote:
OK, the command finally executed and it looks like the
cluster is
running stable for now. However, I'm afraid that 90s might not be
sustainable.
Questions: Can I leave the beacon_grace at 90s? Is there a better
parameter to set? Why is the MGR getting overloaded on a rather small
cluster with 160 OSDs? How does this scale?
I wonder if https://tracker.ceph.com/issues/45439 might be related to
what you're observing here?
In this issue, Andras suggests: "Increasing mgr_stats_period to 15
seconds reduces the load and brings ceph-mgr back to responsive again."
Maybe that helps?
Lenz
--
SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg
GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1453
days inactive
1453
days old
4 comments
2 participants
participants (2)
-
Frank Schilder -
Lenz Grimmer