Dear all, We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues. Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU. When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes. We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline. We would greatly appreciate any assistance or insights you could provide on these matters. Thanks,
On April 25, 2023 9:03 pm, Peter wrote:
Dear all,
We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues.
Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU.
When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes.
We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline.
We would greatly appreciate any assistance or insights you could provide on these matters. Thanks,
are you using separate (physical) links for Corosync and Ceph traffic? if not, they will step on each others toes and cause problems. Corosync is very latency sensitive. https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_cluster_network_re...
Hi Peter, 2% packet loss is a lot, specifically on such expensive hardware. We observed the problems you describe with defective networking hardware with NIC/switch ports in active-active LACP bonding mode. We had periodically failing transceivers and these fails are not immediately detected by the host/switch. Only if such a fail sustained over a longer period of time would the kernel finally report a port as down. A failing transceiver on the switch side often went entirely undetected. Ceph will report slow ping times but nothing (host, osd) down, because packets still go through one of the ports. Its only part of the traffic that disappears and often small packets go through while large ones disappear. This proved very difficult to pin down. With later kernel versions such behavior started to show up with ifconfig as send-receive errors. After replacing a number of transceivers we don't see this happening any more. Things to check: - MTU over all components the same - monitor link utilization to exclude bottlenecks (see Fabian's reply) - check NIC port error counters on all hosts, they should be close to 0 - during a window you see long ping times, look at which hosts show up more often in the network report (ceph daemon mgr.HOST dump_osd_network) and bring interfaces down/up to see if the situation changes Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Fabian Grünbichler <f.gruenbichler@proxmox.com> Sent: Wednesday, April 26, 2023 9:42 AM To: ceph-users@ceph.io; Peter Subject: [ceph-users] Re: PVE CEPH OSD heartbeat show On April 25, 2023 9:03 pm, Peter wrote:
Dear all,
We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues.
Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU.
When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes.
We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline.
We would greatly appreciate any assistance or insights you could provide on these matters. Thanks,
are you using separate (physical) links for Corosync and Ceph traffic? if not, they will step on each others toes and cause problems. Corosync is very latency sensitive. https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_cluster_network_re... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Fabian, Thank you for your prompt response. It's crucial to understand how things work, and I appreciate your assistance. After replacing the switch for our Ceph environment, we experienced three days of normalcy before the issue recurred this morning. I noticed that the TCP in/out became unstable, and TCP errors occurred simultaneously. The UDP in/out values were 70K and 150K, respectively, while the errors peaked at around 50K per second. I reviewed the Proxmox documentation and found that it is recommended to separate the cluster network and storage network. Currently, we have more than 20 Ceph nodes across five different locations, and only one location has experienced this issue. We are fortunate that it has not happened in other areas. While we plan to separate the network soon, I was wondering if there are any temporary solutions or configurations that could limit the UDP triggering and resolve the "corosync" issue. I appreciate your help in this matter and look forward to your response. Peter -----Original Message----- From: Fabian Grünbichler <f.gruenbichler@proxmox.com> Sent: Wednesday, April 26, 2023 12:42 AM To: ceph-users@ceph.io; Peter <petersun@raksmart.com> Subject: Re: [ceph-users] PVE CEPH OSD heartbeat show On April 25, 2023 9:03 pm, Peter wrote:
Dear all,
We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues.
Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU.
When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes.
We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline.
We would greatly appreciate any assistance or insights you could provide on these matters. Thanks,
are you using separate (physical) links for Corosync and Ceph traffic? if not, they will step on each others toes and cause problems. Corosync is very latency sensitive. https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_cluster_network_re...
On May 1, 2023 9:30 pm, Peter wrote:
Hi Fabian,
Thank you for your prompt response. It's crucial to understand how things work, and I appreciate your assistance.
After replacing the switch for our Ceph environment, we experienced three days of normalcy before the issue recurred this morning. I noticed that the TCP in/out became unstable, and TCP errors occurred simultaneously. The UDP in/out values were 70K and 150K, respectively, while the errors peaked at around 50K per second.
I reviewed the Proxmox documentation and found that it is recommended to separate the cluster network and storage network. Currently, we have more than 20 Ceph nodes across five different locations, and only one location has experienced this issue. We are fortunate that it has not happened in other areas. While we plan to separate the network soon, I was wondering if there are any temporary solutions or configurations that could limit the UDP triggering and resolve the "corosync" issue.
the only real solution is separating the links. you can try to prioritize Corosync traffic (UDP on ports 540X) on your switches to avoid the links going over the threshold where Corosync marks them as down. links going down could cause them to start flapping (if they are not really down, but just the Corosync heartbeat timing out occasionally) and trigger an increased amount of traffic cause of retransmits and resync operations trying to reestablish the cluster membership, that could then in turn also affect other traffic going over the same links.
I appreciate your help in this matter and look forward to your response.
Peter
participants (3)
-
Fabian Grünbichler
-
Frank Schilder
-
Peter