Failing heartbeats when no backfill is running
Dear ceph-users, I'm having trouble with heartbeats, there are a lot of "heartbeat_check: no reply from..."-messages in my logs when there is no backfilling or repairing running (yes, it's failing when all PGs are active+clean). Only a few OSDs are failing, even when there are several OSDs on the same host. Doesn't look like a network issue to me. When I set the flags "nobackfill" and "norecover" there are no heartbeat issues. My cluster is kind of heterogenous: it's ARMv7 and x86_64, connected mostly via VPN. Some hosts are Debian Stretch, so I'm still using Ceph Luminous (12.2.12). Is there someone having the same issue? What could be the next steps to debug? Any ideas? Thanks for any help! Lorenz
MTU issues due to the VPN connection? Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, Aug 14, 2019 at 1:48 PM Lorenz Kiefner <root+cephusers@deinadmin.de> wrote:
Dear ceph-users,
I'm having trouble with heartbeats, there are a lot of "heartbeat_check: no reply from..."-messages in my logs when there is no backfilling or repairing running (yes, it's failing when all PGs are active+clean). Only a few OSDs are failing, even when there are several OSDs on the same host. Doesn't look like a network issue to me.
When I set the flags "nobackfill" and "norecover" there are no heartbeat issues.
My cluster is kind of heterogenous: it's ARMv7 and x86_64, connected mostly via VPN. Some hosts are Debian Stretch, so I'm still using Ceph Luminous (12.2.12).
Is there someone having the same issue? What could be the next steps to debug? Any ideas?
Thanks for any help!
Lorenz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, this was the first thing I was thinking about (and yes, there had been some issues, but they are resolved - double checked!). MTU is consistent throughout the whole net and pings in all sizes are handled well. And MTU problems wouldn't probably make a difference between backfills and normal operations, I assume (I use a single network design). And in case of a MTU problem I would suspect that all OSDs on a wrong configured host would go down, or am I wrong? Is ceph sensitive to packet loss? On some VPN links I have up to 20% packet loss on 64k packets but less than 3% on 5k packets in the evenings. Perhaps TCP retransmits trigger some timeouts. Any suggestions how to tune this? I tried setting osd_heartbeat_grace to 120, but this doesn't help... Thanks for any ideas! Lorenz Am 14.08.19 um 13:56 schrieb Paul Emmerich:
MTU issues due to the VPN connection?
Paul
On 8/14/19 5:46 PM, Lorenz Kiefner wrote:
Hi,
this was the first thing I was thinking about (and yes, there had been some issues, but they are resolved - double checked!).
MTU is consistent throughout the whole net and pings in all sizes are handled well. And MTU problems wouldn't probably make a difference between backfills and normal operations, I assume (I use a single network design). And in case of a MTU problem I would suspect that all OSDs on a wrong configured host would go down, or am I wrong?
Is ceph sensitive to packet loss? On some VPN links I have up to 20% packet loss on 64k packets but less than 3% on 5k packets in the evenings.
Yes, it is! 20%, but even 3% is a very, very high number when it comes to packet loss. This should be almost 0%. Ceph is designed to work on low latency networks usually within one datacenter.
Perhaps TCP retransmits trigger some timeouts. Any suggestions how to tune this? I tried setting osd_heartbeat_grace to 120, but this doesn't help...
Thanks for any ideas!
Lorenz
Am 14.08.19 um 13:56 schrieb Paul Emmerich:
MTU issues due to the VPN connection?
Paul
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ok, then Ceph probably doesn't fit for me. I wanted to provide a backup platform for me, my family and my friends. Speed is not relevant, but long-term reliability. So I'm depending on home internet connections and VPN. At the moment I'm mostly using wireguard, but I could switch to openvpn to reduce packet loss (but this will increase latency). But I still don't understand why ceph runs rock-solid as long as there are backfills and recoveries? Lorenz Am 14.08.19 um 17:48 schrieb Wido den Hollander:
On 8/14/19 5:46 PM, Lorenz Kiefner wrote:
Hi,
this was the first thing I was thinking about (and yes, there had been some issues, but they are resolved - double checked!).
MTU is consistent throughout the whole net and pings in all sizes are handled well. And MTU problems wouldn't probably make a difference between backfills and normal operations, I assume (I use a single network design). And in case of a MTU problem I would suspect that all OSDs on a wrong configured host would go down, or am I wrong?
Is ceph sensitive to packet loss? On some VPN links I have up to 20% packet loss on 64k packets but less than 3% on 5k packets in the evenings.
Yes, it is! 20%, but even 3% is a very, very high number when it comes to packet loss. This should be almost 0%.
Ceph is designed to work on low latency networks usually within one datacenter.
Perhaps TCP retransmits trigger some timeouts. Any suggestions how to tune this? I tried setting osd_heartbeat_grace to 120, but this doesn't help...
Thanks for any ideas!
Lorenz
Am 14.08.19 um 13:56 schrieb Paul Emmerich:
MTU issues due to the VPN connection?
Paul
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Lorenz Kiefner
-
Paul Emmerich
-
Wido den Hollander