Flapping OSDs on pacific 16.2.10
Hi, We have a full SSD production cluster running on Pacific 16.2.10 and deployed with cephadm that is experiencing OSD flapping issues. Essentially, random OSDs will get kicked out of the cluster and then automatically brought back in a few times a day. As an example, let's take the case of OSD.184 : -It flapped 9 times between January 15th and 17th with the following log message each time : 2023-01-15T16:33:19.903+0000 prepare_failure osd.184 from osd.49 is reporting failure:1 -On January 17th, it complains that there are slow ops and spam its logs with the following line : heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after 15.000000954s The storage node itself has over 30 GB of ram still available in cache and the drives themselves only seldom peak at 100% usage and that never lasts more than a few seconds. CPU usage is also constantly around 5%. Considering there is no other error messages in any of the regular logs, including the systemd logs, why would this OSD not reply to heartbeats? -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
Do you have any network congestion or packet loss on the replication network? are you sharing nics between public / replication? That is another metric that needs looking into. ________________________________ From: J-P Methot <jp.methot@planethoster.info> Sent: 18 January 2023 12:42 To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Flapping OSDs on pacific 16.2.10 CAUTION: This email originates from outside THG Hi, We have a full SSD production cluster running on Pacific 16.2.10 and deployed with cephadm that is experiencing OSD flapping issues. Essentially, random OSDs will get kicked out of the cluster and then automatically brought back in a few times a day. As an example, let's take the case of OSD.184 : -It flapped 9 times between January 15th and 17th with the following log message each time : 2023-01-15T16:33:19.903+0000 prepare_failure osd.184 from osd.49 is reporting failure:1 -On January 17th, it complains that there are slow ops and spam its logs with the following line : heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after 15.000000954s The storage node itself has over 30 GB of ram still available in cache and the drives themselves only seldom peak at 100% usage and that never lasts more than a few seconds. CPU usage is also constantly around 5%. Considering there is no other error messages in any of the regular logs, including the systemd logs, why would this OSD not reply to heartbeats? -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Danny Webb Principal OpenStack Engineer Danny.Webb@thehutgroup.com [THG Ingenuity Logo] www.thg.com<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thg-ingenuity/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgingenuity?lang=en>
This was my first thought as well, especially if the OSDs log something like “wrongly marked down”. It’s one of the reasons why I favor not having a replication network.
On Jan 18, 2023, at 8:28 AM, Danny Webb <Danny.Webb@thehutgroup.com> wrote:
Do you have any network congestion or packet loss on the replication network? are you sharing nics between public / replication? That is another metric that needs looking into.
At the network level we're using bonds (802.3ad). There are 2 nics, each with two 25gbps port. 1 port per nic is used for the public network, the other for the replication network. That suggests a network bandwidth of 50gbps (in theory) for each network load. The network graph is showing me loads of around 100MB/sec on the public network interface, less on the replication network. No dropped packets or network errors reported. AFAIK, this is not getting overloaded. On 1/18/23 08:28, Danny Webb wrote:
Do you have any network congestion or packet loss on the replication network? are you sharing nics between public / replication? That is another metric that needs looking into. ------------------------------------------------------------------------ *From:* J-P Methot <jp.methot@planethoster.info> *Sent:* 18 January 2023 12:42 *To:* ceph-users <ceph-users@ceph.io> *Subject:* [ceph-users] Flapping OSDs on pacific 16.2.10 CAUTION: This email originates from outside THG
Hi,
We have a full SSD production cluster running on Pacific 16.2.10 and deployed with cephadm that is experiencing OSD flapping issues. Essentially, random OSDs will get kicked out of the cluster and then automatically brought back in a few times a day. As an example, let's take the case of OSD.184 :
-It flapped 9 times between January 15th and 17th with the following log message each time : 2023-01-15T16:33:19.903+0000 prepare_failure osd.184 from osd.49 is reporting failure:1
-On January 17th, it complains that there are slow ops and spam its logs with the following line : heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after 15.000000954s
The storage node itself has over 30 GB of ram still available in cache and the drives themselves only seldom peak at 100% usage and that never lasts more than a few seconds. CPU usage is also constantly around 5%. Considering there is no other error messages in any of the regular logs, including the systemd logs, why would this OSD not reply to heartbeats?
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
*Danny Webb* Principal OpenStack Engineer Danny.Webb@thehutgroup.com
THG Ingenuity Logo www.thg.com <https://www.thg.com> <https://www.linkedin.com/company/thg-ingenuity/?originalSubdomain=uk> <https://twitter.com/thgingenuity?lang=en>
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
Do you have CPU soft lock-ups around these times? We had these timeouts due to using the cfq/bfq disk schedulers with SSDs. The osd_op_tp thread timeout is typical when CPU lockups happen. Could be a sporadic problem with the disk IO path. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: J-P Methot <jp.methot@planethoster.info> Sent: 18 January 2023 14:49:54 To: Danny Webb; ceph-users Subject: [ceph-users] Re: Flapping OSDs on pacific 16.2.10 At the network level we're using bonds (802.3ad). There are 2 nics, each with two 25gbps port. 1 port per nic is used for the public network, the other for the replication network. That suggests a network bandwidth of 50gbps (in theory) for each network load. The network graph is showing me loads of around 100MB/sec on the public network interface, less on the replication network. No dropped packets or network errors reported. AFAIK, this is not getting overloaded. On 1/18/23 08:28, Danny Webb wrote:
Do you have any network congestion or packet loss on the replication network? are you sharing nics between public / replication? That is another metric that needs looking into. ------------------------------------------------------------------------ *From:* J-P Methot <jp.methot@planethoster.info> *Sent:* 18 January 2023 12:42 *To:* ceph-users <ceph-users@ceph.io> *Subject:* [ceph-users] Flapping OSDs on pacific 16.2.10 CAUTION: This email originates from outside THG
Hi,
We have a full SSD production cluster running on Pacific 16.2.10 and deployed with cephadm that is experiencing OSD flapping issues. Essentially, random OSDs will get kicked out of the cluster and then automatically brought back in a few times a day. As an example, let's take the case of OSD.184 :
-It flapped 9 times between January 15th and 17th with the following log message each time : 2023-01-15T16:33:19.903+0000 prepare_failure osd.184 from osd.49 is reporting failure:1
-On January 17th, it complains that there are slow ops and spam its logs with the following line : heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after 15.000000954s
The storage node itself has over 30 GB of ram still available in cache and the drives themselves only seldom peak at 100% usage and that never lasts more than a few seconds. CPU usage is also constantly around 5%. Considering there is no other error messages in any of the regular logs, including the systemd logs, why would this OSD not reply to heartbeats?
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
*Danny Webb* Principal OpenStack Engineer Danny.Webb@thehutgroup.com
THG Ingenuity Logo www.thg.com <https://www.thg.com> <https://www.linkedin.com/company/thg-ingenuity/?originalSubdomain=uk> <https://twitter.com/thgingenuity?lang=en>
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There's nothing in the CPU graph that suggests soft lock-ups at these times. However, thank you for pointing out that the disk io scheduler could have an impact. Ubuntu seems to be on mq-deadline by default, so we just switched to none, as it fits our workload best I believe. I don't know if this will fix our issue, but I think it's worth testing. On 1/18/23 11:17, Frank Schilder wrote:
Do you have CPU soft lock-ups around these times? We had these timeouts due to using the cfq/bfq disk schedulers with SSDs. The osd_op_tp thread timeout is typical when CPU lockups happen. Could be a sporadic problem with the disk IO path.
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
I'm not sure what you look for in the CPU graph. If its load or a similar metric you will not see these lock-ups. You need to look into the syslog and search for it. If these warnings are there, it might give give a clue as to what hardware component is causing it. They look something like "BUG: soft lockup - CPU#X stuck for ..." Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: J-P Methot <jp.methot@planethoster.info> Sent: 18 January 2023 17:38:28 To: Frank Schilder; ceph-users Subject: Re: [ceph-users] Re: Flapping OSDs on pacific 16.2.10 There's nothing in the CPU graph that suggests soft lock-ups at these times. However, thank you for pointing out that the disk io scheduler could have an impact. Ubuntu seems to be on mq-deadline by default, so we just switched to none, as it fits our workload best I believe. I don't know if this will fix our issue, but I think it's worth testing. On 1/18/23 11:17, Frank Schilder wrote:
Do you have CPU soft lock-ups around these times? We had these timeouts due to using the cfq/bfq disk schedulers with SSDs. The osd_op_tp thread timeout is typical when CPU lockups happen. Could be a sporadic problem with the disk IO path.
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
participants (4)
-
Anthony D'Atri
-
Danny Webb
-
Frank Schilder
-
J-P Methot