Bluestore does not defer writes
Hi, today I recognized bad performance in our cluster. Running "watch ceph osd perf |sort -hk 2 -r" I found that all bluestore OSDs are slow on commit and that the commit timings are equal to their apply timings: For example Every 2.0s: ceph osd perf |sort -hk 2 -r 440 82 82 430 58 58 435 56 56 449 53 53 442 40 40 441 30 30 439 27 27 99 0 1 98 0 0 97 0 2 96 0 6 95 0 2 94 0 6 93 0 13 The once with zero commit timings are filestore and the others are bluestore osds. I did not see this after installing the new bluestore osds (maybe this occured later). Both types of osds have nvmes as journal/db. Servers have equal cpus/ram etc. The only tuning regarding bluestore is: bluestore_block_db_size = 69793218560 bluestore_prefer_deferred_size_hdd = 524288 In order to make a filestore like behavior, but that does not seem to work. Any tips? Regards Dennis
On 31/08/2020 11:00, Dennis Benndorf wrote:
Hi,
today I recognized bad performance in our cluster. Running "watch ceph osd perf |sort -hk 2 -r" I found that all bluestore OSDs are slow on commit and that the commit timings are equal to their apply timings:
For example Every 2.0s: ceph osd perf |sort -hk 2 -r
440 82 82 430 58 58 435 56 56 449 53 53 442 40 40 441 30 30 439 27 27 99 0 1 98 0 0 97 0 2 96 0 6 95 0 2 94 0 6 93 0 13
The once with zero commit timings are filestore and the others are bluestore osds. I did not see this after installing the new bluestore osds (maybe this occured later). Both types of osds have nvmes as journal/db. Servers have equal cpus/ram etc.
The only tuning regarding bluestore is: bluestore_block_db_size = 69793218560 bluestore_prefer_deferred_size_hdd = 524288 In order to make a filestore like behavior, but that does not seem to work.
As far as I know is that with BlueStore apply and commit latencies are equal. Where did you get the idea that you could influence this with these settings? Wido
Any tips?
Regards Dennis _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Wido, as bluestore_prefer_deferred_size is the parameter size of IOs to defer writes. There was a bug report involving Igor but dont find it. There was discussed that this value must not exceed 512KB... Regarding osd perf it is just my assumption that when commit and apply timing are equal that there is no deferring at all.Looking at iostat the iosizes to spinning disks are now a lot smaller (~ 4-80KB) than before(~>300KB). I could imagin I hitted some bug or filled my wal and now the osds are not deferring anymore. I remember that someone wrong a blog about bluestore mentioning that bluestore "journaling" does only work until a specific point. If that is reached deferring is deactivated. Could someone bring a bit of light in this? At the moment I get the write latency of my hdds. Regards,Dennis -------- Weitergeleitete Nachricht --------Von: Wido den Hollander < wido@42on.com>An: Dennis Benndorf <dennis.benndorf@googlemail.com>, ceph-users@ceph.ioBetreff: Re: [ceph-users] Bluestore does not defer writesDatum: Mon, 31 Aug 2020 16:06:37 +0200 On 31/08/2020 11:00, Dennis Benndorf wrote: Hi, today I recognized bad performance in our cluster. Running "watch cephosd perf |sort -hk 2 -r" I found that all bluestore OSDs are slow oncommit and that the commit timings are equal to their apply timings: For exampleEvery 2.0s: ceph osd perf |sort -hk 2- r 440 82 82430 58 58435 56 56449 53 5344 2 40 40441 30 30439 27 27 99 0 1 98 0 0 97 0 2 96 0 6 95 0 2 94 0 6 93 0 13 The once with zero commit timings are filestore and the others arebluestore osds.I did not see this after installing the new bluestore osds (maybe thisoccured later).Both types of osds have nvmes as journal/db. Servers have equalcpus/ram etc. The only tuning regarding bluestore is: bluestore_block_db_size = 69793218560 bluestore_prefer_deferred_size_hdd = 524288In order to make a filestore like behavior, but that does not seem towork. As far as I know is that with BlueStore apply and commit latencies are equal. Where did you get the idea that you could influence this with these settings? Wido Any tips? Regards Dennis_______________________________________________ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-leave@ceph.io
Here the blog I wrote about: https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktopHDD for data + SSD for journal Filestore writes everything to the journal and only starts to flush it to the data device when the journal fills up to the configured percent. This is very convenient because it makes journal act as a «temporary buffer» that absorbs random write bursts. Bluestore can’t do the same even when you put its WAL+DB on SSD. It also has sort of a «journal» which is called «deferred write queue», but it’s very small (only 64 requests) and it lacks any kind of background flush threads. So you actually can increase the maximum number of deferred requests, but after the queue fills up the performance will drop until OSD restarts. Maybe I have been hitten by that. Is there any change planned or done on this? Cant imagine users regulary restart their osds to get performance back.... Regards, Dennis -------- Weitergeleitete Nachricht --------Von: Wido den Hollander < wido@42on.com>An: Dennis Benndorf <dennis.benndorf@googlemail.com>, ceph-users@ceph.ioBetreff: Re: [ceph-users] Bluestore does not defer writesDatum: Mon, 31 Aug 2020 16:06:37 +0200 On 31/08/2020 11:00, Dennis Benndorf wrote: Hi, today I recognized bad performance in our cluster. Running "watch cephosd perf |sort -hk 2 -r" I found that all bluestore OSDs are slow oncommit and that the commit timings are equal to their apply timings: For exampleEvery 2.0s: ceph osd perf |sort -hk 2- r 440 82 82430 58 58435 56 56449 53 5344 2 40 40441 30 30439 27 27 99 0 1 98 0 0 97 0 2 96 0 6 95 0 2 94 0 6 93 0 13 The once with zero commit timings are filestore and the others arebluestore osds.I did not see this after installing the new bluestore osds (maybe thisoccured later).Both types of osds have nvmes as journal/db. Servers have equalcpus/ram etc. The only tuning regarding bluestore is: bluestore_block_db_size = 69793218560 bluestore_prefer_deferred_size_hdd = 524288In order to make a filestore like behavior, but that does not seem towork. As far as I know is that with BlueStore apply and commit latencies are equal. Where did you get the idea that you could influence this with these settings? Wido Any tips? Regards Dennis_______________________________________________ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Dennis Benndorf
-
Wido den Hollander