After having upgraded my ceph cluster from Luminous to Nautilus 14.2.6 , from time to time "ceph health detail" claims about some"Long heartbeat ping times on front/back interface seen". As far as I can understand (after having read https://docs.ceph.com/docs/nautilus/rados/operations/monitoring/), this means that the ping from one OSD to another one exceeded 1 s. I have some questions on these network performance checks 1) What is meant exactly with front and back interface ? 2) I can see the involved OSDs only in the output of "ceph health detail" (when there is the problem) but I can't find this information in the log files. In the mon log file I can only see messages such as: 2020-01-28 11:14:07.641 7f618e644700 0 log_channel(cluster) log [WRN] : Health check failed: Long heartbeat ping times on back interface seen, longest is 1416.618 msec (OSD_SLOW_PING_TIME_BACK) but the involved OSDs are not reported in this log. Do I just need to increase the verbosity of the mon log ? 3) Is 1 s a reasonable value for this threshold ? How could this value be changed ? What is the relevant configuration variable ? 4) https://docs.ceph.com/docs/nautilus/rados/operations/monitoring/ suggests to use the dump_osd_network command. I think there is an error in that page: it says that the command should be issued on ceph-mgr.x.asok, while I think that instead the ceph-osd-x.asok should be used I have an other ceph cluster (running nautilus 14.2.6 as well) where there aren't OSD_SLOW_PING_* error messages in the mon logs, but: ceph daemon /var/run/ceph/ceph-osd..asok dump_osd_network 1 reports a lot of entries (i.e. pings exceeded 1 s). How can this be explained ? Thanks, Massimo
Quoting Massimo Sgaravatto (massimo.sgaravatto@gmail.com):
After having upgraded my ceph cluster from Luminous to Nautilus 14.2.6 , from time to time "ceph health detail" claims about some"Long heartbeat ping times on front/back interface seen".
As far as I can understand (after having read https://docs.ceph.com/docs/nautilus/rados/operations/monitoring/), this means that the ping from one OSD to another one exceeded 1 s.
I have some questions on these network performance checks
1) What is meant exactly with front and back interface ?
Do you have a "public" and a "cluster" network? I would expect that the "back" interface is a "cluster" network interface.
2) I can see the involved OSDs only in the output of "ceph health detail" (when there is the problem) but I can't find this information in the log files. In the mon log file I can only see messages such as:
2020-01-28 11:14:07.641 7f618e644700 0 log_channel(cluster) log [WRN] : Health check failed: Long heartbeat ping times on back interface seen, longest is 1416.618 msec (OSD_SLOW_PING_TIME_BACK)
but the involved OSDs are not reported in this log. Do I just need to increase the verbosity of the mon log ?
3) Is 1 s a reasonable value for this threshold ? How could this value be changed ? What is the relevant configuration variable ?
Not sure how much priority Ceph gives to this ping check. But if you're on a 10 Gb/s network I would start complaining when things take longer than 1 ms ... a ping should not take much longer than 0.05 ms so if it would take an order of magnitude longer than expected latency is not optimal. For Gigabit networks I would bump above values by an order of magnitude. Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Thanks for your answer MON-MGR hosts have a mgmt network and a public network. OSD nodes have instead a mgmt network, a public network. and a cluster network This is what I have in ceph.conf: public network = 192.168.61.0/24 cluster network = 192.168.222.0/24 public and cluster networks are 10 Gbps networks (actually there is a single 10 Gbps NIC on each node used for both the public and the cluster networks). The mgmt network is a 1 Gbps network, but this one shouldn't be used for such pings among the OSDs ... Cheers, Massimo On Thu, Jan 30, 2020 at 9:26 AM Stefan Kooman <stefan@bit.nl> wrote:
Quoting Massimo Sgaravatto (massimo.sgaravatto@gmail.com):
After having upgraded my ceph cluster from Luminous to Nautilus 14.2.6 , from time to time "ceph health detail" claims about some"Long heartbeat ping times on front/back interface seen".
As far as I can understand (after having read https://docs.ceph.com/docs/nautilus/rados/operations/monitoring/), this means that the ping from one OSD to another one exceeded 1 s.
I have some questions on these network performance checks
1) What is meant exactly with front and back interface ?
Do you have a "public" and a "cluster" network? I would expect that the "back" interface is a "cluster" network interface.
2) I can see the involved OSDs only in the output of "ceph health detail" (when there is the problem) but I can't find this information in the log files. In the mon log file I can only see messages such as:
2020-01-28 11:14:07.641 7f618e644700 0 log_channel(cluster) log [WRN] : Health check failed: Long heartbeat ping times on back interface seen, longest is 1416.618 msec (OSD_SLOW_PING_TIME_BACK)
but the involved OSDs are not reported in this log. Do I just need to increase the verbosity of the mon log ?
3) Is 1 s a reasonable value for this threshold ? How could this value be changed ? What is the relevant configuration variable ?
Not sure how much priority Ceph gives to this ping check. But if you're on a 10 Gb/s network I would start complaining when things take longer than 1 ms ... a ping should not take much longer than 0.05 ms so if it would take an order of magnitude longer than expected latency is not optimal.
For Gigabit networks I would bump above values by an order of magnitude.
Gr. Stefan
-- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Hi, Quoting Massimo Sgaravatto (massimo.sgaravatto@gmail.com):
Thanks for your answer
MON-MGR hosts have a mgmt network and a public network. OSD nodes have instead a mgmt network, a public network. and a cluster network This is what I have in ceph.conf:
public network = 192.168.61.0/24 cluster network = 192.168.222.0/24
public and cluster networks are 10 Gbps networks (actually there is a single 10 Gbps NIC on each node used for both the public and the cluster networks).
In that case there is no advantage of using a seperate cluster network. As it would only be beneficial when replication data between OSDs is on a seperate interface. Is the cluster heavily loaded? Do you have metrics on bandwith usage / switch port statistics? If you have many "discards" (and / or errors) this might impact the ping times as well.
The mgmt network is a 1 Gbps network, but this one shouldn't be used for such pings among the OSDs ...
I doubt Ceph will use the mgmt network, but not sure if 't doing a lookup on hostname which might use mgmt network in your case or if it's using configured IPs for ceph You can dump osd network info per OSD on the storage nodes themselves by this command: ceph daemon osd.$id dump_osd_network You would have to do that for every OSD and see which ones report "entries". Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
I am seeing very few of such error messages in the mon logs (~ a couple per day) If I issue on every OSD the command "ceph daemon osd.$id dump_osd_network" with the default 1000 ms threshold, I can't see entries. I guess this is because that command considers only the last (15 ?) minutes. Am I supposed to see in some log files which are the problematic OSDs ? Thanks, Massimo On Thu, Jan 30, 2020 at 11:13 AM Stefan Kooman <stefan@bit.nl> wrote:
Hi,
Quoting Massimo Sgaravatto (massimo.sgaravatto@gmail.com):
Thanks for your answer
MON-MGR hosts have a mgmt network and a public network. OSD nodes have instead a mgmt network, a public network. and a cluster network This is what I have in ceph.conf:
public network = 192.168.61.0/24 cluster network = 192.168.222.0/24
public and cluster networks are 10 Gbps networks (actually there is a single 10 Gbps NIC on each node used for both the public and the cluster networks).
In that case there is no advantage of using a seperate cluster network. As it would only be beneficial when replication data between OSDs is on a seperate interface. Is the cluster heavily loaded? Do you have metrics on bandwith usage / switch port statistics? If you have many "discards" (and / or errors) this might impact the ping times as well.
The mgmt network is a 1 Gbps network, but this one shouldn't be used for such pings among the OSDs ...
I doubt Ceph will use the mgmt network, but not sure if 't doing a lookup on hostname which might use mgmt network in your case or if it's using configured IPs for ceph
You can dump osd network info per OSD on the storage nodes themselves by this command:
ceph daemon osd.$id dump_osd_network
You would have to do that for every OSD and see which ones report "entries".
Gr. Stefan
-- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
participants (2)
-
Massimo Sgaravatto
-
Stefan Kooman