OSDs flapping since upgrade to 14.2.10
Hi list, since our upgrade 14.2.9 -> 14.2.10 we observe flapping OSDs: * The mons claim every few minutes: 2020-08-07 09:49:09.783648 osd.243 (osd.243) 246 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.753704 osd.243 (osd.243) 248 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:07:21.187945 osd.253 (osd.253) 469 : cluster [WRN] Monitor daemon marked osd.253 down, but it is still running 2020-08-07 10:04:35.440547 mon.cephmon01 (mon.0) 390132 : cluster [DBG] osd.243 reported failed by osd.33 2020-08-07 10:04:35.508412 mon.cephmon01 (mon.0) 390133 : cluster [DBG] osd.243 reported failed by osd.187 2020-08-07 10:04:35.508529 mon.cephmon01 (mon.0) 390134 : cluster [INF] osd.243 failed (root=default,datacenter=of,row=row-of-02,host=cephosd16) (2 reporters from different host after 44.000150 >= grace 25.935545) 2020-08-07 10:04:35.695171 mon.cephmon01 (mon.0) 390135 : cluster [DBG] osd.243 reported failed by osd.203 2020-08-07 10:04:35.771704 mon.cephmon01 (mon.0) 390136 : cluster [DBG] osd.243 reported failed by osd.163 2020-08-07 10:04:41.588530 mon.cephmon01 (mon.0) 390148 : cluster [INF] osd.243 [v2:10.198.10.16:6882/6611,v1:10.198.10.16:6885/6611] boot 2020-08-07 10:04:40.753704 osd.243 (osd.243) 248 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.753712 osd.243 (osd.243) 249 : cluster [DBG] map e2683535 wrongly marked me down at e2683534 osd.33 says: 2020-08-07 10:04:35.437 7fcaaa4f3700 -1 osd.33 2683533 heartbeat_check: no reply from 10.198.10.16:6802 osd.243 since back 2020-08-07 10:03:51.223911 front 2020-08-07 10:03:51.224322 (oldest deadline 2020-08-07 10:04:35.322704) osd.243 says: 2020-08-07 10:03:55.065 7f0d33911700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:03:55.065 7f0d34112700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 [.. ~3000(!) Lines ..] 2020-08-07 10:04:33.644 7f0d33110700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:04:33.688 7f0d13acb700 0 bluestore(/var/lib/ceph/osd/ceph-243) log_latency_fn slow operation observed for upper_bound, latency = 20.9013s, after = omap_iterator(cid = 19.58a_head, oid = #19:51a21a27::: .dir.default.223091333.1.3:head#) 2020-08-07 10:04:33.688 7f0d13acb700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:04:40.748 7f0d2279b700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.748 7f0d2279b700 0 log_channel(cluster) log [DBG] : map e2683535 wrongly marked me down at e2683534 * as a consequence, old deep-scrubs did not finish, because they would be interrupted -> ' pgs not deep-scrubbed in time' for the latter, I increased the op-thread-timeout back to the pre 12(!).2.11 value of 30 i`m am not sure, if we really have a problem, but it does not look healthy. Any ideas, thoughts? regards, Ingo -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001
On 2020-08-07 12:07, Ingo Reimann wrote:
i`m am not sure, if we really have a problem, but it does not look healthy.
It might be related to the change that is mentioned in another thread: "block.db/block.wal device performance dropped after upgrade to 14.2.10" TL;DR: bluefs_buffered_io has been changed to "false" in 14.2.10. It doesn't use buffer cache in that case, and in certain workloads (i.e. snap trimming) this seem to have a big impact, even for environments that have large osd_memory_target. I would change that back to "true" ("ceph config set osd bluefs_buffered_io true" should do the trick). Not sure if the OSDs need a restart afterwards, as the config change seem to be effectve immediately for running daemons. Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Hi Stefan, Hi Manuel, thanks for your quick advices. In fact, since i set "ceph config set osd bluefs_buffered_io true", the problems disappeared. We have lots of RAM in our osd hosts, so buffering is ok. I`ll trak this issue down further after the weekend! best regards, Ingo ----- Ursprüngliche Mail ----- Von: "Stefan Kooman" <stefan@bit.nl> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 12:24:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10 On 2020-08-07 12:07, Ingo Reimann wrote:
i`m am not sure, if we really have a problem, but it does not look healthy.
It might be related to the change that is mentioned in another thread: "block.db/block.wal device performance dropped after upgrade to 14.2.10" TL;DR: bluefs_buffered_io has been changed to "false" in 14.2.10. It doesn't use buffer cache in that case, and in certain workloads (i.e. snap trimming) this seem to have a big impact, even for environments that have large osd_memory_target. I would change that back to "true" ("ceph config set osd bluefs_buffered_io true" should do the trick). Not sure if the OSDs need a restart afterwards, as the config change seem to be effectve immediately for running daemons. Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001
Hi Ingo, If you are able and have lots of available memory, could you also try setting it to false but increasing the osd_memory_target size? I'd like to understand a little bit deeper what's going on here. Ultimately I don't want our only line of defense against slow snap trimming to be having page cache available! Mark On 8/7/20 6:51 AM, Ingo Reimann wrote:
Hi Stefan, Hi Manuel,
thanks for your quick advices.
In fact, since i set "ceph config set osd bluefs_buffered_io true", the problems disappeared. We have lots of RAM in our osd hosts, so buffering is ok. I`ll trak this issue down further after the weekend!
best regards, Ingo
----- Ursprüngliche Mail ----- Von: "Stefan Kooman" <stefan@bit.nl> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 12:24:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10
On 2020-08-07 12:07, Ingo Reimann wrote:
i`m am not sure, if we really have a problem, but it does not look healthy. It might be related to the change that is mentioned in another thread: "block.db/block.wal device performance dropped after upgrade to 14.2.10"
TL;DR: bluefs_buffered_io has been changed to "false" in 14.2.10. It doesn't use buffer cache in that case, and in certain workloads (i.e. snap trimming) this seem to have a big impact, even for environments that have large osd_memory_target.
I would change that back to "true" ("ceph config set osd bluefs_buffered_io true" should do the trick). Not sure if the OSDs need a restart afterwards, as the config change seem to be effectve immediately for running daemons.
Gr. Stefan
Hi Mark, i`ll check that after the weekend! Ingo ----- Ursprüngliche Mail ----- Von: "Mark Nelson" <mnelson@redhat.com> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 15:15:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10 Hi Ingo, If you are able and have lots of available memory, could you also try setting it to false but increasing the osd_memory_target size? I'd like to understand a little bit deeper what's going on here. Ultimately I don't want our only line of defense against slow snap trimming to be having page cache available! Mark On 8/7/20 6:51 AM, Ingo Reimann wrote:
Hi Stefan, Hi Manuel,
thanks for your quick advices.
In fact, since i set "ceph config set osd bluefs_buffered_io true", the problems disappeared. We have lots of RAM in our osd hosts, so buffering is ok. I`ll trak this issue down further after the weekend!
best regards, Ingo
----- Ursprüngliche Mail ----- Von: "Stefan Kooman" <stefan@bit.nl> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 12:24:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10
On 2020-08-07 12:07, Ingo Reimann wrote:
i`m am not sure, if we really have a problem, but it does not look healthy. It might be related to the change that is mentioned in another thread: "block.db/block.wal device performance dropped after upgrade to 14.2.10"
TL;DR: bluefs_buffered_io has been changed to "false" in 14.2.10. It doesn't use buffer cache in that case, and in certain workloads (i.e. snap trimming) this seem to have a big impact, even for environments that have large osd_memory_target.
I would change that back to "true" ("ceph config set osd bluefs_buffered_io true" should do the trick). Not sure if the OSDs need a restart afterwards, as the config change seem to be effectve immediately for running daemons.
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001
Hi Mark, i raised the osd_memory_target from 4G to 6G and set bluefs_buffered_io back to false. 10 Minutes later, i got the first 'Monitor daemon marked osd.X down, but it is still running', after additional 5, the second event. I tried to raise the memory_target to 10G, but this didn`t help, so i switched back buffered_io to true. We have a heterogenous cluster with different amounts of OSDs per host. Maximum is 37 OSDs with 256GB RAM, so i set the limit of 6G, to save that machine. All OSDs are spinners with 2GB on (internal) journal and using bluestore. The 2G are a relict of old times. I think for new OSDs, we should use the new(?) default of 5G or go even larger with our 10TB+ Disks. We use rdb and rgw, but no rbd snapshots! Does this help, or do you need some more information? best regards, Ingo ----- Ursprüngliche Mail ----- Von: "Ingo Reimann" <ireimann@dunkel.de> An: "Mark Nelson" <mnelson@redhat.com> CC: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 15:29:07 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10 Hi Mark, i`ll check that after the weekend! Ingo ----- Ursprüngliche Mail ----- Von: "Mark Nelson" <mnelson@redhat.com> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 15:15:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10 Hi Ingo, If you are able and have lots of available memory, could you also try setting it to false but increasing the osd_memory_target size? I'd like to understand a little bit deeper what's going on here. Ultimately I don't want our only line of defense against slow snap trimming to be having page cache available! Mark On 8/7/20 6:51 AM, Ingo Reimann wrote:
Hi Stefan, Hi Manuel,
thanks for your quick advices.
In fact, since i set "ceph config set osd bluefs_buffered_io true", the problems disappeared. We have lots of RAM in our osd hosts, so buffering is ok. I`ll trak this issue down further after the weekend!
best regards, Ingo
----- Ursprüngliche Mail ----- Von: "Stefan Kooman" <stefan@bit.nl> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 12:24:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10
On 2020-08-07 12:07, Ingo Reimann wrote:
i`m am not sure, if we really have a problem, but it does not look healthy. It might be related to the change that is mentioned in another thread: "block.db/block.wal device performance dropped after upgrade to 14.2.10"
TL;DR: bluefs_buffered_io has been changed to "false" in 14.2.10. It doesn't use buffer cache in that case, and in certain workloads (i.e. snap trimming) this seem to have a big impact, even for environments that have large osd_memory_target.
I would change that back to "true" ("ceph config set osd bluefs_buffered_io true" should do the trick). Not sure if the OSDs need a restart afterwards, as the config change seem to be effectve immediately for running daemons.
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001
Hi, Maybe this help, You can increase the osd_op_tp thread in ceph conf to something similar to: [osd] osd_op_thread_suicide_timeout = 900 osd_op_thread_timeout = 300 osd_recovery_thread_timeout = 300 Regards -----Mensaje original----- De: Ingo Reimann <ireimann@dunkel.de> Enviado el: viernes, 7 de agosto de 2020 12:08 Para: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] OSDs flapping since upgrade to 14.2.10 Hi list, since our upgrade 14.2.9 -> 14.2.10 we observe flapping OSDs: * The mons claim every few minutes: 2020-08-07 09:49:09.783648 osd.243 (osd.243) 246 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.753704 osd.243 (osd.243) 248 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:07:21.187945 osd.253 (osd.253) 469 : cluster [WRN] Monitor daemon marked osd.253 down, but it is still running 2020-08-07 10:04:35.440547 mon.cephmon01 (mon.0) 390132 : cluster [DBG] osd.243 reported failed by osd.33 2020-08-07 10:04:35.508412 mon.cephmon01 (mon.0) 390133 : cluster [DBG] osd.243 reported failed by osd.187 2020-08-07 10:04:35.508529 mon.cephmon01 (mon.0) 390134 : cluster [INF] osd.243 failed (root=default,datacenter=of,row=row-of-02,host=cephosd16) (2 reporters from different host after 44.000150 >= grace 25.935545) 2020-08-07 10:04:35.695171 mon.cephmon01 (mon.0) 390135 : cluster [DBG] osd.243 reported failed by osd.203 2020-08-07 10:04:35.771704 mon.cephmon01 (mon.0) 390136 : cluster [DBG] osd.243 reported failed by osd.163 2020-08-07 10:04:41.588530 mon.cephmon01 (mon.0) 390148 : cluster [INF] osd.243 [v2:10.198.10.16:6882/6611,v1:10.198.10.16:6885/6611] boot 2020-08-07 10:04:40.753704 osd.243 (osd.243) 248 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.753712 osd.243 (osd.243) 249 : cluster [DBG] map e2683535 wrongly marked me down at e2683534 osd.33 says: 2020-08-07 10:04:35.437 7fcaaa4f3700 -1 osd.33 2683533 heartbeat_check: no reply from 10.198.10.16:6802 osd.243 since back 2020-08-07 10:03:51.223911 front 2020-08-07 10:03:51.224322 (oldest deadline 2020-08-07 10:04:35.322704) osd.243 says: 2020-08-07 10:03:55.065 7f0d33911700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:03:55.065 7f0d34112700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 [.. ~3000(!) Lines ..] 2020-08-07 10:04:33.644 7f0d33110700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:04:33.688 7f0d13acb700 0 bluestore(/var/lib/ceph/osd/ceph-243) log_latency_fn slow operation observed for upper_bound, latency = 20.9013s, after = omap_iterator(cid = 19.58a_head, oid = #19:51a21a27::: .dir.default.223091333.1.3:head#) 2020-08-07 10:04:33.688 7f0d13acb700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:04:40.748 7f0d2279b700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.748 7f0d2279b700 0 log_channel(cluster) log [DBG] : map e2683535 wrongly marked me down at e2683534 * as a consequence, old deep-scrubs did not finish, because they would be interrupted -> ' pgs not deep-scrubbed in time' for the latter, I increased the op-thread-timeout back to the pre 12(!).2.11 value of 30 i`m am not sure, if we really have a problem, but it does not look healthy. Any ideas, thoughts? regards, Ingo -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
EDH - Manuel Rios
-
Ingo Reimann
-
Mark Nelson
-
Stefan Kooman