Hi Mark, i raised the osd_memory_target from 4G to 6G and set bluefs_buffered_io back to false. 10 Minutes later, i got the first 'Monitor daemon marked osd.X down, but it is still running', after additional 5, the second event. I tried to raise the memory_target to 10G, but this didn`t help, so i switched back buffered_io to true. We have a heterogenous cluster with different amounts of OSDs per host. Maximum is 37 OSDs with 256GB RAM, so i set the limit of 6G, to save that machine. All OSDs are spinners with 2GB on (internal) journal and using bluestore. The 2G are a relict of old times. I think for new OSDs, we should use the new(?) default of 5G or go even larger with our 10TB+ Disks. We use rdb and rgw, but no rbd snapshots! Does this help, or do you need some more information? best regards, Ingo ----- Ursprüngliche Mail ----- Von: "Ingo Reimann" <ireimann@dunkel.de> An: "Mark Nelson" <mnelson@redhat.com> CC: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 15:29:07 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10 Hi Mark, i`ll check that after the weekend! Ingo ----- Ursprüngliche Mail ----- Von: "Mark Nelson" <mnelson@redhat.com> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 15:15:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10 Hi Ingo, If you are able and have lots of available memory, could you also try setting it to false but increasing the osd_memory_target size? I'd like to understand a little bit deeper what's going on here. Ultimately I don't want our only line of defense against slow snap trimming to be having page cache available! Mark On 8/7/20 6:51 AM, Ingo Reimann wrote:
Hi Stefan, Hi Manuel,
thanks for your quick advices.
In fact, since i set "ceph config set osd bluefs_buffered_io true", the problems disappeared. We have lots of RAM in our osd hosts, so buffering is ok. I`ll trak this issue down further after the weekend!
best regards, Ingo
----- Ursprüngliche Mail ----- Von: "Stefan Kooman" <stefan@bit.nl> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 12:24:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10
On 2020-08-07 12:07, Ingo Reimann wrote:
i`m am not sure, if we really have a problem, but it does not look healthy. It might be related to the change that is mentioned in another thread: "block.db/block.wal device performance dropped after upgrade to 14.2.10"
TL;DR: bluefs_buffered_io has been changed to "false" in 14.2.10. It doesn't use buffer cache in that case, and in certain workloads (i.e. snap trimming) this seem to have a big impact, even for environments that have large osd_memory_target.
I would change that back to "true" ("ceph config set osd bluefs_buffered_io true" should do the trick). Not sure if the OSDs need a restart afterwards, as the config change seem to be effectve immediately for running daemons.
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001