Hi Mark, i`ll check that after the weekend! Ingo ----- Ursprüngliche Mail ----- Von: "Mark Nelson" <mnelson@redhat.com> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 15:15:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10 Hi Ingo, If you are able and have lots of available memory, could you also try setting it to false but increasing the osd_memory_target size? I'd like to understand a little bit deeper what's going on here. Ultimately I don't want our only line of defense against slow snap trimming to be having page cache available! Mark On 8/7/20 6:51 AM, Ingo Reimann wrote:
Hi Stefan, Hi Manuel,
thanks for your quick advices.
In fact, since i set "ceph config set osd bluefs_buffered_io true", the problems disappeared. We have lots of RAM in our osd hosts, so buffering is ok. I`ll trak this issue down further after the weekend!
best regards, Ingo
----- Ursprüngliche Mail ----- Von: "Stefan Kooman" <stefan@bit.nl> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Freitag, 7. August 2020 12:24:08 Betreff: [ceph-users] Re: OSDs flapping since upgrade to 14.2.10
On 2020-08-07 12:07, Ingo Reimann wrote:
i`m am not sure, if we really have a problem, but it does not look healthy. It might be related to the change that is mentioned in another thread: "block.db/block.wal device performance dropped after upgrade to 14.2.10"
TL;DR: bluefs_buffered_io has been changed to "false" in 14.2.10. It doesn't use buffer cache in that case, and in certain workloads (i.e. snap trimming) this seem to have a big impact, even for environments that have large osd_memory_target.
I would change that back to "true" ("ceph config set osd bluefs_buffered_io true" should do the trick). Not sure if the OSDs need a restart afterwards, as the config change seem to be effectve immediately for running daemons.
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support@dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001