Full cluster outage when ECONNREFUSED is triggered
Hi We’ve recently had a serious outage at work, after a host had a network problem: - We rebooted a single host in a cluster of fifteen hosts across three racks. - The single host had a bad network configuration after booting, causing it to send some packets to the wrong network. - One network still worked and offered a connection to the mons. - The other network connection was bad. Packets were refused, not dropped. - Due to osd_fast_fail_on_connection_refused=true, the broken host forced the mons to take all other OSDs down (immediate failure). - Only after shutting down the faulty host, was it possible to start the shut down OSDs, to restore the cluster. We have since solved the problem by removing the default route that caused the packets to end up in the wrong network, where they were summarily rejected by a firewall. That is, we made sure that packets would be dropped in the future, not rejected. Still, I figured I’ll send this experience of ours to this mailing list, as this seems to be something others might encounter as well. In the following PR, that introduced osd_fast_fail_on_connection_refused, there’s this description:
This changeset adds additional handler (handle_refused()) to the dispatchers and code that detects when connection attempt fails with ECONNREFUSED error (connection refused) which is a clear indication that host is alive, but daemon isn't, so daemons can instantly mark the other side as undoubtly downed without the need for grace timer.
And this comment:
As for flapping, we discussed it on ceph-devel ml and came to conclusion that it requires either broken firewall or network configuration to cause this, and these are more serious issues that should be resolved first before worrying about OSDs flapping (either way, flapping OSDs could be good for getting someone's attention).
https://github.com/ceph/ceph/pull/8558https://github.com/ceph/ceph/pull/8558 It has left us wondering if these are the right assumptions. An ECONNREFUSED condition can bring down a whole cluster, and I wonder if there should be some kind of safe-guard to ensure that this is avoided. One badly configured host should generally not be able do that, and if the packets are dropped, instead of refused, the cluster notices that the OSD down reports come only from one host, and acts accordingly. What do you think? Does this warrant a change in Ceph? I’m happy to provide details and create a ticket. Cheers, Denis
Hi Denis, I would agree with you that a single misconfigured host should not take out healthy hosts under any circumstances. I'm not sure if your incident is actually covered by the devs comments, it is quite possible that you observed an unintended side effect that is a bug in handling the connection error. I think the intention is to shut down fast the OSDs with connection refused (where timeouts are not required) and not other OSDs. A bug report with tracker seems warranted. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Denis Krienbühl <denis@href.ch> Sent: Friday, November 24, 2023 9:01 AM To: ceph-users Subject: [ceph-users] Full cluster outage when ECONNREFUSED is triggered Hi We’ve recently had a serious outage at work, after a host had a network problem: - We rebooted a single host in a cluster of fifteen hosts across three racks. - The single host had a bad network configuration after booting, causing it to send some packets to the wrong network. - One network still worked and offered a connection to the mons. - The other network connection was bad. Packets were refused, not dropped. - Due to osd_fast_fail_on_connection_refused=true, the broken host forced the mons to take all other OSDs down (immediate failure). - Only after shutting down the faulty host, was it possible to start the shut down OSDs, to restore the cluster. We have since solved the problem by removing the default route that caused the packets to end up in the wrong network, where they were summarily rejected by a firewall. That is, we made sure that packets would be dropped in the future, not rejected. Still, I figured I’ll send this experience of ours to this mailing list, as this seems to be something others might encounter as well. In the following PR, that introduced osd_fast_fail_on_connection_refused, there’s this description:
This changeset adds additional handler (handle_refused()) to the dispatchers and code that detects when connection attempt fails with ECONNREFUSED error (connection refused) which is a clear indication that host is alive, but daemon isn't, so daemons can instantly mark the other side as undoubtly downed without the need for grace timer.
And this comment:
As for flapping, we discussed it on ceph-devel ml and came to conclusion that it requires either broken firewall or network configuration to cause this, and these are more serious issues that should be resolved first before worrying about OSDs flapping (either way, flapping OSDs could be good for getting someone's attention).
https://github.com/ceph/ceph/pull/8558https://github.com/ceph/ceph/pull/8558 It has left us wondering if these are the right assumptions. An ECONNREFUSED condition can bring down a whole cluster, and I wonder if there should be some kind of safe-guard to ensure that this is avoided. One badly configured host should generally not be able do that, and if the packets are dropped, instead of refused, the cluster notices that the OSD down reports come only from one host, and acts accordingly. What do you think? Does this warrant a change in Ceph? I’m happy to provide details and create a ticket. Cheers, Denis _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Frank. I see it the same way. I’ll be sure to create a ticket with all the details and steps to reproduce the issue. Denis
On 24 Nov 2023, at 10:24, Frank Schilder <frans@dtu.dk> wrote:
Hi Denis,
I would agree with you that a single misconfigured host should not take out healthy hosts under any circumstances. I'm not sure if your incident is actually covered by the devs comments, it is quite possible that you observed an unintended side effect that is a bug in handling the connection error. I think the intention is to shut down fast the OSDs with connection refused (where timeouts are not required) and not other OSDs.
A bug report with tracker seems warranted.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Denis Krienbühl <denis@href.ch> Sent: Friday, November 24, 2023 9:01 AM To: ceph-users Subject: [ceph-users] Full cluster outage when ECONNREFUSED is triggered
Hi
We’ve recently had a serious outage at work, after a host had a network problem:
- We rebooted a single host in a cluster of fifteen hosts across three racks. - The single host had a bad network configuration after booting, causing it to send some packets to the wrong network. - One network still worked and offered a connection to the mons. - The other network connection was bad. Packets were refused, not dropped. - Due to osd_fast_fail_on_connection_refused=true, the broken host forced the mons to take all other OSDs down (immediate failure). - Only after shutting down the faulty host, was it possible to start the shut down OSDs, to restore the cluster.
We have since solved the problem by removing the default route that caused the packets to end up in the wrong network, where they were summarily rejected by a firewall. That is, we made sure that packets would be dropped in the future, not rejected.
Still, I figured I’ll send this experience of ours to this mailing list, as this seems to be something others might encounter as well.
In the following PR, that introduced osd_fast_fail_on_connection_refused, there’s this description:
This changeset adds additional handler (handle_refused()) to the dispatchers and code that detects when connection attempt fails with ECONNREFUSED error (connection refused) which is a clear indication that host is alive, but daemon isn't, so daemons can instantly mark the other side as undoubtly downed without the need for grace timer.
And this comment:
As for flapping, we discussed it on ceph-devel ml and came to conclusion that it requires either broken firewall or network configuration to cause this, and these are more serious issues that should be resolved first before worrying about OSDs flapping (either way, flapping OSDs could be good for getting someone's attention).
https://github.com/ceph/ceph/pull/8558https://github.com/ceph/ceph/pull/8558
It has left us wondering if these are the right assumptions. An ECONNREFUSED condition can bring down a whole cluster, and I wonder if there should be some kind of safe-guard to ensure that this is avoided. One badly configured host should generally not be able do that, and if the packets are dropped, instead of refused, the cluster notices that the OSD down reports come only from one host, and acts accordingly.
What do you think? Does this warrant a change in Ceph? I’m happy to provide details and create a ticket.
Cheers,
Denis _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den fre 24 nov. 2023 kl 10:25 skrev Frank Schilder <frans@dtu.dk>:
Hi Denis,
I would agree with you that a single misconfigured host should not take out healthy hosts under any circumstances. I'm not sure if your incident is actually covered by the devs comments, it is quite possible that you observed an unintended side effect that is a bug in handling the connection error. I think the intention is to shut down fast the OSDs with connection refused (where timeouts are not required) and not other OSDs.
No, this has been true for a long while and has happened to me multiple times. Any kind of fault where an OSD can talk to at least one mon, and then for any of multiple reasons do not respond or connect to other OSDs lead to this. The OSD comes up, can hold 0 data and have a lifetime of 5 seconds, and then it claims it can't talk to some 5-10-15 other OSDs, and rats on them to the mon which then proceeds to believe this new OSD and flaps those other OSDs, which then get to reconnect and the mon logs that "osdmap says I am down but I am not". This goes on until you kill the bad OSD or fix the network. If you boot up a host with many OSDs, they all pick 5-10-15 OSDs to shoot down, so you quickly get annoying errors at a large scale if you ever misconfigure a new host in the least possible way. There should be some better kind of validation that the new (and in my case at least, often empty) OSD is not at fault and that the other hundreds of working OSDs are in fact not gone at all before causing this kind of confusion. -- May the most significant bit of your life be positive.
Hi, I think this is why the mon-osd interaction requires a certain number of osd to report another osd as down/unavailable: https://docs.ceph.com/en/latest/rados/configuration/mon-osd-interaction/#osd... The default value for mon_osd_reporter_subtree_level is host, and the default value for mon_osd_min_down_reporters is 2 (values taken from one of our cluster, no override in ceph config). So it requires reports from two osds in different hosts to consider another osd as down. This should not be case in the reported situation unless setting osd_fast_fail_on_connection_refused<https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval-osd_fast_fail_on_connection_refused>=true changes this behaviour. Best regards, Burkhard Linke On 24.11.23 11:09, Janne Johansson wrote:
Hi Denis,
I would agree with you that a single misconfigured host should not take out healthy hosts under any circumstances. I'm not sure if your incident is actually covered by the devs comments, it is quite possible that you observed an unintended side effect that is a bug in handling the connection error. I think the intention is to shut down fast the OSDs with connection refused (where timeouts are not required) and not other OSDs. No, this has been true for a long while and has happened to me multiple times. Any kind of fault where an OSD can talk to at least one mon, and then for any of multiple reasons do not respond or connect to other OSDs lead to this. The OSD comes up, can hold 0 data and have a lifetime of 5 seconds, and then it claims it can't talk to some 5-10-15 other OSDs, and rats on them to the mon which then
Den fre 24 nov. 2023 kl 10:25 skrev Frank Schilder<frans@dtu.dk>: proceeds to believe this new OSD and flaps those other OSDs, which then get to reconnect and the mon logs that "osdmap says I am down but I am not". This goes on until you kill the bad OSD or fix the network. If you boot up a host with many OSDs, they all pick 5-10-15 OSDs to shoot down, so you quickly get annoying errors at a large scale if you ever misconfigure a new host in the least possible way.
There should be some better kind of validation that the new (and in my case at least, often empty) OSD is not at fault and that the other hundreds of working OSDs are in fact not gone at all before causing this kind of confusion.
On 24 Nov 2023, at 11:49, Burkhard Linke <Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
This should not be case in the reported situation unless setting osd_fast_fail_on_connection_refused<https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval-osd_fast_fail_on_connection_refused>=true changes this behaviour.
In our tests it does change the behavior. Usually the mons take mon_osd_reporter_subtree_level and mon_osd_min_down_reporters into account. In our tests, this is the case if an OSD heartbeat is dropped and the OSD is still able to talk to the mons. However, if the OSD heartbeat is rejected, in our case because of an unrelated firewall change, the OSD sends an immediate failure to the mon: https://github.com/ceph/ceph/blob/febfdd83a7838338033486826ef1fc9a5e8d588e/src/osd/OSD.cc#L6434 ceph/src/osd/OSD.cc at febfdd83a7838338033486826ef1fc9a5e8d588e · ceph/ceph github.com The mon then propagates that failure, without taking any other reports into consideration: https://github.com/ceph/ceph/blob/febfdd83a7838338033486826ef1fc9a5e8d588e/src/mon/OSDMonitor.cc#L3367 ceph/src/mon/OSDMonitor.cc at febfdd83a7838338033486826ef1fc9a5e8d588e · ceph/ceph github.com This is fine when a single OSD goes down and everything else is okay. It then has the intended effect of getting rid of the OSD fast. The assumption presumably being: If a host can answer with a rejection to the OSD heartbeat, it is only the OSD that is affected. In our case however, a network change caused rejections from an entirely different host (a gateway), while a network path to the mons was still available. In this case, Ceph does not apply the safe-guards it usually does.
Hi Denis.
The mon then propagates that failure, without taking any other reports into consideration:
Exactly. I cannot imagine that this change of behavior is intended. The configs on OSD down reporting ought to be honored in any failure situation. Since you already investigated the relevant code lines, please update/create the tracker with your findings. Hope a dev looks at this. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Denis Krienbühl <denis@href.ch> Sent: Friday, November 24, 2023 12:04 PM To: Burkhard Linke Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Full cluster outage when ECONNREFUSED is triggered
On 24 Nov 2023, at 11:49, Burkhard Linke <Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
This should not be case in the reported situation unless setting osd_fast_fail_on_connection_refused<https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval-osd_fast_fail_on_connection_refused>=true changes this behaviour.
In our tests it does change the behavior. Usually the mons take mon_osd_reporter_subtree_level and mon_osd_min_down_reporters into account. In our tests, this is the case if an OSD heartbeat is dropped and the OSD is still able to talk to the mons. However, if the OSD heartbeat is rejected, in our case because of an unrelated firewall change, the OSD sends an immediate failure to the mon: https://github.com/ceph/ceph/blob/febfdd83a7838338033486826ef1fc9a5e8d588e/src/osd/OSD.cc#L6434 ceph/src/osd/OSD.cc at febfdd83a7838338033486826ef1fc9a5e8d588e · ceph/ceph github.com The mon then propagates that failure, without taking any other reports into consideration: https://github.com/ceph/ceph/blob/febfdd83a7838338033486826ef1fc9a5e8d588e/src/mon/OSDMonitor.cc#L3367 ceph/src/mon/OSDMonitor.cc at febfdd83a7838338033486826ef1fc9a5e8d588e · ceph/ceph github.com This is fine when a single OSD goes down and everything else is okay. It then has the intended effect of getting rid of the OSD fast. The assumption presumably being: If a host can answer with a rejection to the OSD heartbeat, it is only the OSD that is affected. In our case however, a network change caused rejections from an entirely different host (a gateway), while a network path to the mons was still available. In this case, Ceph does not apply the safe-guards it usually does. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dennis, I have to ask a clarifying question. If I understand the intend of osd_fast_fail_on_connection_refused correctly, an OSD that receives a connection_refused should get marked down fast to avoid unnecessarily long wait times. And *only* OSDs that receive connection refused. In your case, did booting up the server actually create a network route for all other OSDs to the wrong network as well? In other words, did it act as a gateway and all OSDs received connection refused messages and not just the ones on the critical host? If so, your observation would be expected. If not, then there is something wrong with the down reporting that should be looked at. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, November 24, 2023 1:20 PM To: Denis Krienbühl; Burkhard Linke Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Full cluster outage when ECONNREFUSED is triggered Hi Denis.
The mon then propagates that failure, without taking any other reports into consideration:
Exactly. I cannot imagine that this change of behavior is intended. The configs on OSD down reporting ought to be honored in any failure situation. Since you already investigated the relevant code lines, please update/create the tracker with your findings. Hope a dev looks at this. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Denis Krienbühl <denis@href.ch> Sent: Friday, November 24, 2023 12:04 PM To: Burkhard Linke Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Full cluster outage when ECONNREFUSED is triggered
On 24 Nov 2023, at 11:49, Burkhard Linke <Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
This should not be case in the reported situation unless setting osd_fast_fail_on_connection_refused<https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval-osd_fast_fail_on_connection_refused>=true changes this behaviour.
In our tests it does change the behavior. Usually the mons take mon_osd_reporter_subtree_level and mon_osd_min_down_reporters into account. In our tests, this is the case if an OSD heartbeat is dropped and the OSD is still able to talk to the mons. However, if the OSD heartbeat is rejected, in our case because of an unrelated firewall change, the OSD sends an immediate failure to the mon: https://github.com/ceph/ceph/blob/febfdd83a7838338033486826ef1fc9a5e8d588e/src/osd/OSD.cc#L6434 ceph/src/osd/OSD.cc at febfdd83a7838338033486826ef1fc9a5e8d588e · ceph/ceph github.com The mon then propagates that failure, without taking any other reports into consideration: https://github.com/ceph/ceph/blob/febfdd83a7838338033486826ef1fc9a5e8d588e/src/mon/OSDMonitor.cc#L3367 ceph/src/mon/OSDMonitor.cc at febfdd83a7838338033486826ef1fc9a5e8d588e · ceph/ceph github.com This is fine when a single OSD goes down and everything else is okay. It then has the intended effect of getting rid of the OSD fast. The assumption presumably being: If a host can answer with a rejection to the OSD heartbeat, it is only the OSD that is affected. In our case however, a network change caused rejections from an entirely different host (a gateway), while a network path to the mons was still available. In this case, Ceph does not apply the safe-guards it usually does. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank.
On 24 Nov 2023, at 14:27, Frank Schilder <frans@dtu.dk> wrote:
I have to ask a clarifying question. If I understand the intend of osd_fast_fail_on_connection_refused correctly, an OSD that receives a connection_refused should get marked down fast to avoid unnecessarily long wait times. And *only* OSDs that receive connection refused.
In your case, did booting up the server actually create a network route for all other OSDs to the wrong network as well? In other words, did it act as a gateway and all OSDs received connection refused messages and not just the ones on the critical host? If so, your observation would be expected. If not, then there is something wrong with the down reporting that should be looked at.
No, the server has two networks through which to reach OSDs and mons. Say north and south. South was down and the traffic destined to it made it through the default gateway to an unrelated host that would bounce everything with “connection refused”. North was still up, and through it the other OSDs and mons could also be reached. So the host that was bootet had the wrong configuration. The packets on the other hosts of the cluster were unaffected and all their network configuration remained as is, though they would not have reached the OSDs on the booted host via south anymore. Those would have been dropped by my understanding. I’ll be sure to create a detailed ticket and to post it to this thread, I’m just not sure I’ll be able to do it today, but after what I’ve heard I think this should at least be looked at in detail and I’ll be sure to provide as much info as I can. Denis
Thanks everyone for your feedback. I created a ticket and added most of our internal post-mortem research: https://tracker.ceph.com/issues/63636 Cheers, Denis
On 24 Nov 2023, at 09:01, Denis Krienbühl <denis@href.ch> wrote:
Hi
We’ve recently had a serious outage at work, after a host had a network problem:
- We rebooted a single host in a cluster of fifteen hosts across three racks. - The single host had a bad network configuration after booting, causing it to send some packets to the wrong network. - One network still worked and offered a connection to the mons. - The other network connection was bad. Packets were refused, not dropped. - Due to osd_fast_fail_on_connection_refused=true, the broken host forced the mons to take all other OSDs down (immediate failure). - Only after shutting down the faulty host, was it possible to start the shut down OSDs, to restore the cluster.
We have since solved the problem by removing the default route that caused the packets to end up in the wrong network, where they were summarily rejected by a firewall. That is, we made sure that packets would be dropped in the future, not rejected.
Still, I figured I’ll send this experience of ours to this mailing list, as this seems to be something others might encounter as well.
In the following PR, that introduced osd_fast_fail_on_connection_refused, there’s this description:
This changeset adds additional handler (handle_refused()) to the dispatchers and code that detects when connection attempt fails with ECONNREFUSED error (connection refused) which is a clear indication that host is alive, but daemon isn't, so daemons can instantly mark the other side as undoubtly downed without the need for grace timer.
And this comment:
As for flapping, we discussed it on ceph-devel ml and came to conclusion that it requires either broken firewall or network configuration to cause this, and these are more serious issues that should be resolved first before worrying about OSDs flapping (either way, flapping OSDs could be good for getting someone's attention).
https://github.com/ceph/ceph/pull/8558https://github.com/ceph/ceph/pull/8558
It has left us wondering if these are the right assumptions. An ECONNREFUSED condition can bring down a whole cluster, and I wonder if there should be some kind of safe-guard to ensure that this is avoided. One badly configured host should generally not be able do that, and if the packets are dropped, instead of refused, the cluster notices that the OSD down reports come only from one host, and acts accordingly.
What do you think? Does this warrant a change in Ceph? I’m happy to provide details and create a ticket.
Cheers,
Denis _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Burkhard Linke
-
Denis Krienbühl
-
Frank Schilder
-
Janne Johansson