avg apply latency went up after update from octopus to pacific
Hi, today I did the first update from octopus to pacific, and it looks like the avg apply latency went up from 1ms to 2ms. All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value? Cheers Boris
Hi Boris, Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)? Josh On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks like the avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload. Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen < jbaergen@digitalocean.com>:
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks like
the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Hi Boris, OK, what I'm wondering is whether https://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1. 16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you. There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider is https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches, as sometimes disabling these write caches can improve the IOPS performance of SSDs. Josh On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen <jbaergen@digitalocean.com>:
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks like the avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io. It doesn't sound that I can mitigate the problem with more SSDs. Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen < jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whether https://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider is
https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen <
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
jbaergen@digitalocean.com>: like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
One thing to watch out for with bluefs_buffered_io is that disabling it can greatly impact certain rocksdb workloads. From what I remember it was a huge problem during certain iteration workloads for things like collection listing. I think the block cache was being invalidated or simply never cached the data properly, but the underlying code has changed quite a bit so it was tough to track down exactly how it worked in different versions of RocksDB. We got bitten by this a couple years ago when we switch to direct IO and it caused a lot of people trouble. We ended up having to turn it back on after lots of frustration and digging Basically the linux page cache is saving the day even though it really shouldn't be necessary. It's irritating because bluestore is otherwise faster in many scenarios (at least with NVMe drives) when bluefs uses direct IO. Mark On 2/28/23 15:46, Boris Behrens wrote:
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io.
It doesn't sound that I can mitigate the problem with more SSDs.
Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen < jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whether https://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider is
https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen <
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
jbaergen@digitalocean.com>: like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Disabling the write cache and the bluefs_buffered_io did not change anything. What we see is that larger disks seem to be the leader in therms of slowness (we have 70% 2TB, 20% 4TB and 10% 8TB SSDs in the cluster), but removing some of the 8TB disks and replace them with 2TB (because it's by far the majority and we have a lot of them) disks did also not change anything. Are there any other ideas I could try. Customer start to complain about the slower performance and our k8s team mentions problems with ETCD because the latency is too high. Would it be an option to recreate every OSD? Cheers Boris Am Di., 28. Feb. 2023 um 22:46 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io.
It doesn't sound that I can mitigate the problem with more SSDs.
Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen < jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whether https://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider is
https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen <
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
jbaergen@digitalocean.com>: like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Hi Boris, additionally you might want to manually compact RocksDB for every OSD. Thanks, Igor On 3/21/2023 12:22 PM, Boris Behrens wrote:
Disabling the write cache and the bluefs_buffered_io did not change anything. What we see is that larger disks seem to be the leader in therms of slowness (we have 70% 2TB, 20% 4TB and 10% 8TB SSDs in the cluster), but removing some of the 8TB disks and replace them with 2TB (because it's by far the majority and we have a lot of them) disks did also not change anything.
Are there any other ideas I could try. Customer start to complain about the slower performance and our k8s team mentions problems with ETCD because the latency is too high.
Would it be an option to recreate every OSD?
Cheers Boris
Am Di., 28. Feb. 2023 um 22:46 Uhr schrieb Boris Behrens<bb@kervyn.de>:
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io.
It doesn't sound that I can mitigate the problem with more SSDs.
Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen < jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whether https://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider is
https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen < jbaergen@digitalocean.com>:
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens<bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens<bb@kervyn.de> wrote: like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value? Cheers Boris _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>! <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>
Hi Igor, i've offline compacted all the OSDs and reenabled the bluefs_buffered_io It didn't change anything and the commit and apply latencies are around 5-10 times higher than on our nautlus cluster. The pacific cluster got a 5 minute mean over all OSDs 2.2ms, while the nautilus cluster is around 0.2 - 0.7 ms. I also see these kind of logs. Google didn't really help: 2023-03-21T14:08:22.089+0000 7efe7b911700 3 rocksdb: [le/block_based/filter_policy.cc:579] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. Am Di., 21. März 2023 um 10:46 Uhr schrieb Igor Fedotov < igor.fedotov@croit.io>:
Hi Boris,
additionally you might want to manually compact RocksDB for every OSD.
Thanks,
Igor On 3/21/2023 12:22 PM, Boris Behrens wrote:
Disabling the write cache and the bluefs_buffered_io did not change anything. What we see is that larger disks seem to be the leader in therms of slowness (we have 70% 2TB, 20% 4TB and 10% 8TB SSDs in the cluster), but removing some of the 8TB disks and replace them with 2TB (because it's by far the majority and we have a lot of them) disks did also not change anything.
Are there any other ideas I could try. Customer start to complain about the slower performance and our k8s team mentions problems with ETCD because the latency is too high.
Would it be an option to recreate every OSD?
Cheers Boris
Am Di., 28. Feb. 2023 um 22:46 Uhr schrieb Boris Behrens <bb@kervyn.de> <bb@kervyn.de>:
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io.
It doesn't sound that I can mitigate the problem with more SSDs.
Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen <jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whetherhttps://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider is https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen <
jbaergen@digitalocean.com>:
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config
value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> !
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Cheers everybody. What we've tried yesterday: - compact all OSDs - change disk scheduler from mq-deadline to none - disable kernel mitigation for cpu vulnerabilities none of these settings really brought the numbers close to the octopus. Next thing I could try is to use the alternate tuning of this blog post: https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive/ Are there any other ides? This is the graph for the avg commit latency (resolution 1d) for the last month. Green is pacific, yellow is nautilus. We've update from octopus to pacific where the first high peak is. [image: grafik.png] same data for the last two days with 1h resolution [image: grafik.png] Am Di., 21. März 2023 um 15:31 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi Igor, i've offline compacted all the OSDs and reenabled the bluefs_buffered_io
It didn't change anything and the commit and apply latencies are around 5-10 times higher than on our nautlus cluster. The pacific cluster got a 5 minute mean over all OSDs 2.2ms, while the nautilus cluster is around 0.2 - 0.7 ms.
I also see these kind of logs. Google didn't really help: 2023-03-21T14:08:22.089+0000 7efe7b911700 3 rocksdb: [le/block_based/filter_policy.cc:579] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
Am Di., 21. März 2023 um 10:46 Uhr schrieb Igor Fedotov < igor.fedotov@croit.io>:
Hi Boris,
additionally you might want to manually compact RocksDB for every OSD.
Thanks,
Igor On 3/21/2023 12:22 PM, Boris Behrens wrote:
Disabling the write cache and the bluefs_buffered_io did not change anything. What we see is that larger disks seem to be the leader in therms of slowness (we have 70% 2TB, 20% 4TB and 10% 8TB SSDs in the cluster), but removing some of the 8TB disks and replace them with 2TB (because it's by far the majority and we have a lot of them) disks did also not change anything.
Are there any other ideas I could try. Customer start to complain about the slower performance and our k8s team mentions problems with ETCD because the latency is too high.
Would it be an option to recreate every OSD?
Cheers Boris
Am Di., 28. Feb. 2023 um 22:46 Uhr schrieb Boris Behrens <bb@kervyn.de> <bb@kervyn.de>:
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io.
It doesn't sound that I can mitigate the problem with more SSDs.
Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen <jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whetherhttps://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider is https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen <
jbaergen@digitalocean.com>:
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config
value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> !
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Hi, Maybe [1] ? [1] https://tracker.ceph.com/issues/58530 k
On 22 Mar 2023, at 16:20, Boris Behrens <bb@kervyn.de> wrote:
Are there any other ides?
Might be. Josh also pointed in that direction. I currently search for ways to mitigate it. Am Mi., 22. März 2023 um 10:30 Uhr schrieb Konstantin Shalygin < k0ste@k0ste.ru>:
Hi,
Maybe [1] ?
[1] https://tracker.ceph.com/issues/58530 k
On 22 Mar 2023, at 16:20, Boris Behrens <bb@kervyn.de> wrote:
Are there any other ides?
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Hi Boris, first of all I'm not sure if it's valid to compare two different clusters (pacific vs . nautilus, C1 vs. C2 respectively). The perf numbers difference might be caused by a bunch of other factors: different H/W, user load, network etc... I can see that you got ~2x latency increase after Octopus to Pacific upgrade at C1 but Octopus numbers had been much above Nautilus at C2 before the upgrade. Did you observe even lower numbers at C1 when it was running Nautilus if any? You might want to try "ceph tell osd.N bench" to compare OSDs performance for both C1 and C2. Would it be that different? Then redeploy a single OSD at C1, wait till rebalance completion and benchmark it again. What would be the new numbers? Please also collect perf counters from the to-be-redeployed OSD beforehand. W.r.t. rocksdb warning - I presume this might be caused by newer RocksDB version running on top of DB with a legacy format.. Perhaps redeployment would fix that... Thanks, Igor On 3/21/2023 5:31 PM, Boris Behrens wrote:
Hi Igor, i've offline compacted all the OSDs and reenabled the bluefs_buffered_io
It didn't change anything and the commit and apply latencies are around 5-10 times higher than on our nautlus cluster. The pacific cluster got a 5 minute mean over all OSDs 2.2ms, while the nautilus cluster is around 0.2 - 0.7 ms.
I also see these kind of logs. Google didn't really help: 2023-03-21T14:08:22.089+0000 7efe7b911700 3 rocksdb: [le/block_based/filter_policy.cc:579] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
Am Di., 21. März 2023 um 10:46 Uhr schrieb Igor Fedotov < igor.fedotov@croit.io>:
Hi Boris,
additionally you might want to manually compact RocksDB for every OSD.
Thanks,
Igor On 3/21/2023 12:22 PM, Boris Behrens wrote:
Disabling the write cache and the bluefs_buffered_io did not change anything. What we see is that larger disks seem to be the leader in therms of slowness (we have 70% 2TB, 20% 4TB and 10% 8TB SSDs in the cluster), but removing some of the 8TB disks and replace them with 2TB (because it's by far the majority and we have a lot of them) disks did also not change anything.
Are there any other ideas I could try. Customer start to complain about the slower performance and our k8s team mentions problems with ETCD because the latency is too high.
Would it be an option to recreate every OSD?
Cheers Boris
Am Di., 28. Feb. 2023 um 22:46 Uhr schrieb Boris Behrens<bb@kervyn.de> <bb@kervyn.de>:
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io.
It doesn't sound that I can mitigate the problem with more SSDs.
Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen<jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whetherhttps://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider is https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens<bb@kervyn.de> <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen <
jbaergen@digitalocean.com>:
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens<bb@kervyn.de> <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config
value?
Cheers Boris _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web<https://croit.io/> | LinkedIn<http://linkedin.com/company/croit> | Youtube<https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter<https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more<https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> !
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>! <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>
Hey Igor, sadly we do not have the data from the time where c1 was on nautilus. The RocksDB warning persisted the recreation. Here are the measurements. I've picked the same SSD models from the clusters to have some comparablity. For the 8TB disks it's even the same chassis configuration (CPU/Memory/Board/Network) The IOPS seem VERY low for me. Or are these normal values for SSDs? After recreation the IOPS are a lot better on the pacific cluster. I also blkdiscarded the SSDs before recreating them. Nautilus Cluster osd.22 = 8TB osd.343 = 2TB https://pastebin.com/EfSSLmYS Pacific Cluster before recreating OSDs osd.40 = 8TB osd.162 = 2TB https://pastebin.com/wKMmSW9T Pacific Cluster after recreation OSDs osd.40 = 8TB osd.162 = 2TB https://pastebin.com/80eMwwBW Am Mi., 22. März 2023 um 11:09 Uhr schrieb Igor Fedotov < igor.fedotov@croit.io>:
Hi Boris,
first of all I'm not sure if it's valid to compare two different clusters (pacific vs . nautilus, C1 vs. C2 respectively). The perf numbers difference might be caused by a bunch of other factors: different H/W, user load, network etc... I can see that you got ~2x latency increase after Octopus to Pacific upgrade at C1 but Octopus numbers had been much above Nautilus at C2 before the upgrade. Did you observe even lower numbers at C1 when it was running Nautilus if any?
You might want to try "ceph tell osd.N bench" to compare OSDs performance for both C1 and C2. Would it be that different?
Then redeploy a single OSD at C1, wait till rebalance completion and benchmark it again. What would be the new numbers? Please also collect perf counters from the to-be-redeployed OSD beforehand.
W.r.t. rocksdb warning - I presume this might be caused by newer RocksDB version running on top of DB with a legacy format.. Perhaps redeployment would fix that...
Thanks,
Igor On 3/21/2023 5:31 PM, Boris Behrens wrote:
Hi Igor, i've offline compacted all the OSDs and reenabled the bluefs_buffered_io
It didn't change anything and the commit and apply latencies are around 5-10 times higher than on our nautlus cluster. The pacific cluster got a 5 minute mean over all OSDs 2.2ms, while the nautilus cluster is around 0.2 - 0.7 ms.
I also see these kind of logs. Google didn't really help: 2023-03-21T14:08:22.089+0000 7efe7b911700 3 rocksdb: [le/block_based/filter_policy.cc:579] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
Am Di., 21. März 2023 um 10:46 Uhr schrieb Igor Fedotov <igor.fedotov@croit.io>:
Hi Boris,
additionally you might want to manually compact RocksDB for every OSD.
Thanks,
Igor On 3/21/2023 12:22 PM, Boris Behrens wrote:
Disabling the write cache and the bluefs_buffered_io did not change anything. What we see is that larger disks seem to be the leader in therms of slowness (we have 70% 2TB, 20% 4TB and 10% 8TB SSDs in the cluster), but removing some of the 8TB disks and replace them with 2TB (because it's by far the majority and we have a lot of them) disks did also not change anything.
Are there any other ideas I could try. Customer start to complain about the slower performance and our k8s team mentions problems with ETCD because the latency is too high.
Would it be an option to recreate every OSD?
Cheers Boris
Am Di., 28. Feb. 2023 um 22:46 Uhr schrieb Boris Behrens <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de>:
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io.
It doesn't sound that I can mitigate the problem with more SSDs.
Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen <jbaergen@digitalocean.com> <jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whetherhttps://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider ishttps://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen < jbaergen@digitalocean.com>:
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config
value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> <https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte<https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> !
<https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> !
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
sadly we do not have the data from the time where c1 was on nautilus. The RocksDB warning persisted the recreation.
Hi Boris, I was monitoring this thread a bit because I also still need to update from Nautilus, and am interested in this performance degradation. I am happy to provide with some info from my 4 node Nautilus cluster. I have a bit mixed setup with the hdd's, slowly migrating to 8TB 7200rpm sas drives.
Hello together, I've redeployed all OSDs in the cluster and did a blkdiscard before deploying them again. It looks now a lot better, even better before the octopus. I am waiting for confirmation from the dev and customer teams as the value over all OSDs can be misleading, and we still have some OSDs that have a 5 minute mean between 1-2 ms. What I also see is that I have three OSDs that have quite a lot of OMAP data, in compare to other OSDs (~20 time higher). I don't know if this is an issue: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME ... 91 ssd 1.74660 1.00000 1.7 TiB 1.1 TiB 1.1 TiB 26 MiB 2.9 GiB 670 GiB 62.52 1.08 59 up osd.91 92 ssd 1.74660 1.00000 1.7 TiB 1.0 TiB 1022 GiB 575 MiB 2.6 GiB 764 GiB 57.30 0.99 56 up osd.92 93 ssd 1.74660 1.00000 1.7 TiB 986 GiB 983 GiB 25 MiB 3.0 GiB 803 GiB 55.12 0.95 53 up osd.93 ... 130 ssd 1.74660 1.00000 1.7 TiB 1018 GiB 1015 GiB 25 MiB 3.1 GiB 771 GiB 56.92 0.98 53 up osd.130 131 ssd 1.74660 1.00000 1.7 TiB 1023 GiB 1019 GiB 574 MiB 2.9 GiB 766 GiB 57.17 0.98 54 up osd.131 132 ssd 1.74660 1.00000 1.7 TiB 1.1 TiB 1.1 TiB 26 MiB 3.1 GiB 675 GiB 62.26 1.07 58 up osd.132 ... 41 ssd 1.74660 1.00000 1.7 TiB 991 GiB 989 GiB 25 MiB 2.5 GiB 797 GiB 55.43 0.95 52 up osd.41 44 ssd 1.74660 1.00000 1.7 TiB 1.1 TiB 1.1 TiB 576 MiB 2.8 GiB 648 GiB 63.75 1.10 60 up osd.44 56 ssd 1.74660 1.00000 1.7 TiB 993 GiB 990 GiB 25 MiB 2.9 GiB 796 GiB 55.51 0.95 54 up osd.56 IMHO this might be due to the blkdiscard. We move a lot of 2TB disks from the nautilus cluster (c-2) to the, then octopus, pacific cluster (c-1). And we only removed the LVM data. Doing the blkdiscard took around 10 minutes on an 8TB SSD on the first run, and around 5s on the second run. I could imagine, that this might be a problem with SSDs in combination with bluestore, because there is trimable FS and the information on what the OSD thinks is free vs the disk controller thinks is free might deviate. But I am not really deep into storage mechanics so this is just a wild guess. Nonetheless the IOPS the bench command generates are still VERY low compared to the nautilus cluster (~150 vs ~250). But this is something I would pin to this bug: https://tracker.ceph.com/issues/58530 @Igor do you want to me to update the ticket with my findings and the logs from pastebin? @marc If I interpret the linked bug correctly, you might want to have the metadata on an SSD, because the write aplification might hit very hard on HDDs. But maybe someone else from the mailing list can say more about it. Cheers Boris Am Mi., 22. März 2023 um 22:45 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hey Igor,
sadly we do not have the data from the time where c1 was on nautilus. The RocksDB warning persisted the recreation.
Here are the measurements. I've picked the same SSD models from the clusters to have some comparablity. For the 8TB disks it's even the same chassis configuration (CPU/Memory/Board/Network)
The IOPS seem VERY low for me. Or are these normal values for SSDs? After recreation the IOPS are a lot better on the pacific cluster.
I also blkdiscarded the SSDs before recreating them.
Nautilus Cluster osd.22 = 8TB osd.343 = 2TB https://pastebin.com/EfSSLmYS
Pacific Cluster before recreating OSDs osd.40 = 8TB osd.162 = 2TB https://pastebin.com/wKMmSW9T
Pacific Cluster after recreation OSDs osd.40 = 8TB osd.162 = 2TB https://pastebin.com/80eMwwBW
Am Mi., 22. März 2023 um 11:09 Uhr schrieb Igor Fedotov < igor.fedotov@croit.io>:
Hi Boris,
first of all I'm not sure if it's valid to compare two different clusters (pacific vs . nautilus, C1 vs. C2 respectively). The perf numbers difference might be caused by a bunch of other factors: different H/W, user load, network etc... I can see that you got ~2x latency increase after Octopus to Pacific upgrade at C1 but Octopus numbers had been much above Nautilus at C2 before the upgrade. Did you observe even lower numbers at C1 when it was running Nautilus if any?
You might want to try "ceph tell osd.N bench" to compare OSDs performance for both C1 and C2. Would it be that different?
Then redeploy a single OSD at C1, wait till rebalance completion and benchmark it again. What would be the new numbers? Please also collect perf counters from the to-be-redeployed OSD beforehand.
W.r.t. rocksdb warning - I presume this might be caused by newer RocksDB version running on top of DB with a legacy format.. Perhaps redeployment would fix that...
Thanks,
Igor On 3/21/2023 5:31 PM, Boris Behrens wrote:
Hi Igor, i've offline compacted all the OSDs and reenabled the bluefs_buffered_io
It didn't change anything and the commit and apply latencies are around 5-10 times higher than on our nautlus cluster. The pacific cluster got a 5 minute mean over all OSDs 2.2ms, while the nautilus cluster is around 0.2 - 0.7 ms.
I also see these kind of logs. Google didn't really help: 2023-03-21T14:08:22.089+0000 7efe7b911700 3 rocksdb: [le/block_based/filter_policy.cc:579] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
Am Di., 21. März 2023 um 10:46 Uhr schrieb Igor Fedotov <igor.fedotov@croit.io>:
Hi Boris,
additionally you might want to manually compact RocksDB for every OSD.
Thanks,
Igor On 3/21/2023 12:22 PM, Boris Behrens wrote:
Disabling the write cache and the bluefs_buffered_io did not change anything. What we see is that larger disks seem to be the leader in therms of slowness (we have 70% 2TB, 20% 4TB and 10% 8TB SSDs in the cluster), but removing some of the 8TB disks and replace them with 2TB (because it's by far the majority and we have a lot of them) disks did also not change anything.
Are there any other ideas I could try. Customer start to complain about the slower performance and our k8s team mentions problems with ETCD because the latency is too high.
Would it be an option to recreate every OSD?
Cheers Boris
Am Di., 28. Feb. 2023 um 22:46 Uhr schrieb Boris Behrens <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de>:
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io.
It doesn't sound that I can mitigate the problem with more SSDs.
Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen <jbaergen@digitalocean.com> <jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whetherhttps://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider ishttps://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen < jbaergen@digitalocean.com>:
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config
value?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> <https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte<https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> !
<https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> !
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
What I also see is that I have three OSDs that have quite a lot of OMAP data, in compare to other OSDs (~20 time higher). I don't know if this is an issue:
I have on 2TB ssd's with 2GB - 4GB omap data, while on 8TB hdd's the omap data is only 53MB - 100MB. Should I manually clean this? (how? :))
What I also see is that I have three OSDs that have quite a lot of OMAP data, in compare to other OSDs (~20 time higher). I don't know if this is an issue:
I have on 2TB ssd's with 2GB - 4GB omap data, while on 8TB hdd's the omap data is only 53MB - 100MB. Should I manually clean this? (how? :))
The amount of omap data depends on multiple things, especially the use-case. If a given OSD is only used for RBD, it will have a different omap experience than if it were used for an RGW index pool.
What I also see is that I have three OSDs that have quite a lot of
data, in compare to other OSDs (~20 time higher). I don't know if
OMAP this
is an issue:
I have on 2TB ssd's with 2GB - 4GB omap data, while on 8TB hdd's the omap data is only 53MB - 100MB. Should I manually clean this? (how? :))
The amount of omap data depends on multiple things, especially the use- case. If a given OSD is only used for RBD, it will have a different omap experience than if it were used for an RGW index pool.
This (mine) is mostly an rbd cluster. Is it correct that compacting leveldb is addressing 'cleaning omap data'? And this can only be done by setting leveldb_compact_on_mount = true in ceph.conf and restarting the osd?
On 3/27/2023 12:19 PM, Boris Behrens wrote:
Nonetheless the IOPS the bench command generates are still VERY low compared to the nautilus cluster (~150 vs ~250). But this is something I would pin to this bug:https://tracker.ceph.com/issues/58530
I've just run "ceph tell bench" against main, octopus and nautilus branches (fresh osd deployed with vstart.sh) - I don't see any difference between releases - sata drive shows around 110 IOPs in my case.. So I suspect some difference between clusters in your case. E.g. are you sure disk caching is off for both?
@Igor do you want to me to update the ticket with my findings and the logs from pastebin? Feel free to update if you like but IMO we still lack the understanding what was the trigger for perf improvements in you case - OSD redeployment, disk trimming or both? -- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>! <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>
After some digging in the nautilus cluster I see that the disks with the exceptional high IOPS performance are actually SAS attached NVME disks (these: https://semiconductor.samsung.com/ssd/enterprise-ssd/pm1643-pm1643a/mzilt7t6... ) and these disk make around 45% of cluster capacity. Maybe this explains the very low commit latency in the nautilus cluster. I did a bench on all SATA 8TB disks (nautilus) and most all of them only have ~30-50 IOPS. After redeploying one OSD with blkdiscard the IOPS went from 48 -> 120. The IOPS from the bench in out pacific cluster are also down to 40 again for the 4/8TB disks , but the apply latency seems to stay in the same place. But I still don't understand why it is down again. Even when I synced out the OSD so it receives 0 traffic it is still slow. I am unsure how I should interpret this. It also looks like that the AVG apply latency (4h resolution) goes up again (2023-03-01 upgrade to pacific, the dip around 25th was the redeploy and now it seems to go up again) [image: image.png] Am Mo., 27. März 2023 um 17:24 Uhr schrieb Igor Fedotov < igor.fedotov@croit.io>:
On 3/27/2023 12:19 PM, Boris Behrens wrote:
Nonetheless the IOPS the bench command generates are still VERY low compared to the nautilus cluster (~150 vs ~250). But this is something I would pin to this bug: https://tracker.ceph.com/issues/58530
I've just run "ceph tell bench" against main, octopus and nautilus branches (fresh osd deployed with vstart.sh) - I don't see any difference between releases - sata drive shows around 110 IOPs in my case..
So I suspect some difference between clusters in your case. E.g. are you sure disk caching is off for both?
@Igor do you want to me to update the ticket with my findings and the logs from pastebin?
Feel free to update if you like but IMO we still lack the understanding what was the trigger for perf improvements in you case - OSD redeployment, disk trimming or both?
A short correction: The IOPS from the bench in out pacific cluster are also down to 40 again for the 4/8TB disks , but the apply latency seems to stay in the same place. But I still don't understand why it is down again. Even when I synced out the OSD so it receives 0 traffic it is still slow. After idling over night it is back up to 120 IOPS Am Do., 30. März 2023 um 09:45 Uhr schrieb Boris Behrens <bb@kervyn.de>:
After some digging in the nautilus cluster I see that the disks with the exceptional high IOPS performance are actually SAS attached NVME disks (these: https://semiconductor.samsung.com/ssd/enterprise-ssd/pm1643-pm1643a/mzilt7t6... ) and these disk make around 45% of cluster capacity. Maybe this explains the very low commit latency in the nautilus cluster.
I did a bench on all SATA 8TB disks (nautilus) and most all of them only have ~30-50 IOPS. After redeploying one OSD with blkdiscard the IOPS went from 48 -> 120.
The IOPS from the bench in out pacific cluster are also down to 40 again for the 4/8TB disks , but the apply latency seems to stay in the same place. But I still don't understand why it is down again. Even when I synced out the OSD so it receives 0 traffic it is still slow.
I am unsure how I should interpret this. It also looks like that the AVG apply latency (4h resolution) goes up again (2023-03-01 upgrade to pacific, the dip around 25th was the redeploy and now it seems to go up again) [image: image.png]
Am Mo., 27. März 2023 um 17:24 Uhr schrieb Igor Fedotov < igor.fedotov@croit.io>:
On 3/27/2023 12:19 PM, Boris Behrens wrote:
Nonetheless the IOPS the bench command generates are still VERY low compared to the nautilus cluster (~150 vs ~250). But this is something I would pin to this bug: https://tracker.ceph.com/issues/58530
I've just run "ceph tell bench" against main, octopus and nautilus branches (fresh osd deployed with vstart.sh) - I don't see any difference between releases - sata drive shows around 110 IOPs in my case..
So I suspect some difference between clusters in your case. E.g. are you sure disk caching is off for both?
@Igor do you want to me to update the ticket with my findings and the logs from pastebin?
Feel free to update if you like but IMO we still lack the understanding what was the trigger for perf improvements in you case - OSD redeployment, disk trimming or both?
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Hi Boris, I wouldn't recommend to take absolute "osd bench" numbers too seriously. It's definitely not a full-scale quality benchmark tool. The idea was just to make brief OSDs comparison from c1 and c2. And for your reference - IOPS numbers I'm getting in my lab with data/DB colocated: 1) OSD on top of Intel S4600 (SATA SSD) - ~110 IOPS 2) OSD on top of Samsung DCT 983 (M.2 NVMe) - 310 IOPS 3) OSD on top of Intel 905p (Optane NVMe) - 546 IOPS. Could you please provide a bit more info on the H/W and OSD setup? What are the disk models? NVMe or SATA? Are DB and main disk shared? Thanks, Igor On 3/23/2023 12:45 AM, Boris Behrens wrote:
Hey Igor,
sadly we do not have the data from the time where c1 was on nautilus. The RocksDB warning persisted the recreation.
Here are the measurements. I've picked the same SSD models from the clusters to have some comparablity. For the 8TB disks it's even the same chassis configuration (CPU/Memory/Board/Network)
The IOPS seem VERY low for me. Or are these normal values for SSDs? After recreation the IOPS are a lot better on the pacific cluster.
I also blkdiscarded the SSDs before recreating them.
Nautilus Cluster osd.22 = 8TB osd.343 = 2TB https://pastebin.com/EfSSLmYS
Pacific Cluster before recreating OSDs osd.40 = 8TB osd.162 = 2TB https://pastebin.com/wKMmSW9T
Pacific Cluster after recreation OSDs osd.40 = 8TB osd.162 = 2TB https://pastebin.com/80eMwwBW
Am Mi., 22. März 2023 um 11:09 Uhr schrieb Igor Fedotov < igor.fedotov@croit.io>:
Hi Boris,
first of all I'm not sure if it's valid to compare two different clusters (pacific vs . nautilus, C1 vs. C2 respectively). The perf numbers difference might be caused by a bunch of other factors: different H/W, user load, network etc... I can see that you got ~2x latency increase after Octopus to Pacific upgrade at C1 but Octopus numbers had been much above Nautilus at C2 before the upgrade. Did you observe even lower numbers at C1 when it was running Nautilus if any?
You might want to try "ceph tell osd.N bench" to compare OSDs performance for both C1 and C2. Would it be that different?
Then redeploy a single OSD at C1, wait till rebalance completion and benchmark it again. What would be the new numbers? Please also collect perf counters from the to-be-redeployed OSD beforehand.
W.r.t. rocksdb warning - I presume this might be caused by newer RocksDB version running on top of DB with a legacy format.. Perhaps redeployment would fix that...
Thanks,
Igor On 3/21/2023 5:31 PM, Boris Behrens wrote:
Hi Igor, i've offline compacted all the OSDs and reenabled the bluefs_buffered_io
It didn't change anything and the commit and apply latencies are around 5-10 times higher than on our nautlus cluster. The pacific cluster got a 5 minute mean over all OSDs 2.2ms, while the nautilus cluster is around 0.2 - 0.7 ms.
I also see these kind of logs. Google didn't really help: 2023-03-21T14:08:22.089+0000 7efe7b911700 3 rocksdb: [le/block_based/filter_policy.cc:579] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
Am Di., 21. März 2023 um 10:46 Uhr schrieb Igor Fedotov<igor.fedotov@croit.io>:
Hi Boris,
additionally you might want to manually compact RocksDB for every OSD.
Thanks,
Igor On 3/21/2023 12:22 PM, Boris Behrens wrote:
Disabling the write cache and the bluefs_buffered_io did not change anything. What we see is that larger disks seem to be the leader in therms of slowness (we have 70% 2TB, 20% 4TB and 10% 8TB SSDs in the cluster), but removing some of the 8TB disks and replace them with 2TB (because it's by far the majority and we have a lot of them) disks did also not change anything.
Are there any other ideas I could try. Customer start to complain about the slower performance and our k8s team mentions problems with ETCD because the latency is too high.
Would it be an option to recreate every OSD?
Cheers Boris
Am Di., 28. Feb. 2023 um 22:46 Uhr schrieb Boris Behrens<bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de>:
Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io.
It doesn't sound that I can mitigate the problem with more SSDs.
Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen<jbaergen@digitalocean.com> <jbaergen@digitalocean.com>:
Hi Boris,
OK, what I'm wondering is whetherhttps://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS
Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1.
16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you.
There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider ishttps://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches , as sometimes disabling these write caches can improve the IOPS performance of SSDs.
Josh
On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens<bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi Josh, we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload.
Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen < jbaergen@digitalocean.com>:
Hi Boris,
Which version did you upgrade from and to, specifically? And what workload are you running (RBD, etc.)?
Josh
On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens<bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> <bb@kervyn.de> wrote:
Hi, today I did the first update from octopus to pacific, and it looks
like the
avg apply latency went up from 1ms to 2ms.
All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config
value?
Cheers Boris _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web<https://croit.io/> <https://croit.io/> | LinkedIn<http://linkedin.com/company/croit> <http://linkedin.com/company/croit> | Youtube<https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter<https://twitter.com/croit_io> <https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more<https://croit.io/croit-sc22> <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte<https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> !
<https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web<https://croit.io/> | LinkedIn<http://linkedin.com/company/croit> | Youtube<https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter<https://twitter.com/croit_io>
Meet us at the SC22 Conference! Learn more<https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html> !
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>! <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>
Hey Igor, we are currently using these disks - all SATA attached (is it normal to have some OSDs without waer counter?): # ceph device ls | awk '{print $1}' | cut -f 1,2 -d _ | sort | uniq -c 18 SAMSUNG_MZ7KH3T8 (4TB) 126 SAMSUNG_MZ7KM1T9 (2TB) 24 SAMSUNG_MZ7L37T6 (8TB) 1 TOSHIBA_THNSN81Q (2TB) (ceph device ls shows a wear of 16% so maybe we remove this one) These are the CPUs in the storage hosts: # ceph osd metadata | grep -F '"cpu": "' | sort -u "cpu": "Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz", "cpu": "Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz", The hosts have between 128GB and 256GB memory and each got between 20 and 30 OSDs. DB and OSD are using same device, no extra device for DB/WAL. Seeing your IOPS it looks like we are around the same level. I am curious if the performance will stay at the current level or degrade over time. Am Mo., 27. März 2023 um 13:42 Uhr schrieb Igor Fedotov < igor.fedotov@croit.io>:
Hi Boris,
I wouldn't recommend to take absolute "osd bench" numbers too seriously. It's definitely not a full-scale quality benchmark tool.
The idea was just to make brief OSDs comparison from c1 and c2.
And for your reference - IOPS numbers I'm getting in my lab with data/DB colocated:
1) OSD on top of Intel S4600 (SATA SSD) - ~110 IOPS
2) OSD on top of Samsung DCT 983 (M.2 NVMe) - 310 IOPS
3) OSD on top of Intel 905p (Optane NVMe) - 546 IOPS.
Could you please provide a bit more info on the H/W and OSD setup?
What are the disk models? NVMe or SATA? Are DB and main disk shared?
Thanks,
Igor
Hi, we have a very similar situation. We updated from nautilus -> pacific (16.2.11) and saw a rapid increase in the commit_latency and op_w_latency (>10s on some OSDs) after a few hours. We also have nearly exclusive rbd workload. After deleting old snapshots we saw an improvenent, and after recreating snapshots the numbers went up again. Without snapshots the numbers are slowly getting higher but not as fast as before with existing snapshots. We also use SAS connected NVMe-SSDs. bluefs_buffered_io made no difference. We compacted the rocksdb on a single OSD yesterday, and funnily enough this is now the OSD with the highest op_w_latency. I generated a perf graph for this single OSD and can generate more, but I'm not sure how to share this data with you...? I saw in the thread that Boris redeployed all OSDs. Could that be a more permanent solution or is this also just temporarily (like deleting the snapshots)? Greetings, Jan
Hello Jan, I had the same on two cluster from nautlus to pacific. On both it did help to fire Ceph tell osd.* compact If this had not help, i would go for a recreate of the osds... Hth Mehmet Am 31. März 2023 10:56:42 MESZ schrieb j.kruse@profihost.ag:
Hi,
we have a very similar situation. We updated from nautilus -> pacific (16.2.11) and saw a rapid increase in the commit_latency and op_w_latency (>10s on some OSDs) after a few hours. We also have nearly exclusive rbd workload.
After deleting old snapshots we saw an improvenent, and after recreating snapshots the numbers went up again. Without snapshots the numbers are slowly getting higher but not as fast as before with existing snapshots. We also use SAS connected NVMe-SSDs. bluefs_buffered_io made no difference. We compacted the rocksdb on a single OSD yesterday, and funnily enough this is now the OSD with the highest op_w_latency. I generated a perf graph for this single OSD and can generate more, but I'm not sure how to share this data with you...?
I saw in the thread that Boris redeployed all OSDs. Could that be a more permanent solution or is this also just temporarily (like deleting the snapshots)?
Greetings, Jan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Mehmet, thanks for your response. I somehow mixed up the versions, we encountered the problems when updating from octopus -> pacific, not nautilus -> pacific. But I will nevertheless try out your suggestions tomorrow. As far as I can tell, without snapshots, our latencies are not optimal (op_w_latency at around 30-40ms, peaks at 1-2s sometimes), but somewhat stable. Greetings, Jan Von: ceph@elchaka.de <ceph@elchaka.de> Datum: Samstag, 1. April 2023 um 01:12 An: ceph-users@ceph.io <ceph-users@ceph.io>, Jan-Tristan Kruse <j.kruse@profihost.ag> Betreff: Re: [ceph-users] Re: avg apply latency went up after update from octopus to pacific Hello Jan, I had the same on two cluster from nautlus to pacific. On both it did help to fire Ceph tell osd.* compact If this had not help, i would go for a recreate of the osds... Hth Mehmet Am 31. März 2023 10:56:42 MESZ schrieb j.kruse@profihost.ag: Hi, we have a very similar situation. We updated from nautilus -> pacific (16.2.11) and saw a rapid increase in the commit_latency and op_w_latency (>10s on some OSDs) after a few hours. We also have nearly exclusive rbd workload. After deleting old snapshots we saw an improvenent, and after recreating snapshots the numbers went up again. Without snapshots the numbers are slowly getting higher but not as fast as before with existing snapshots. We also use SAS connected NVMe-SSDs. bluefs_buffered_io made no difference. We compacted the rocksdb on a single OSD yesterday, and funnily enough this is now the OSD with the highest op_w_latency. I generated a perf graph for this single OSD and can generate more, but I'm not sure how to share this data with you...? I saw in the thread that Boris redeployed all OSDs. Could that be a more permanent solution or is this also just temporarily (like deleting the snapshots)? Greetings, Jan ________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (10)
-
Anthony D'Atri
-
Boris Behrens
-
ceph＠elchaka.de
-
Igor Fedotov
-
j.kruse＠profihost.ag
-
Jan-Tristan Kruse
-
Josh Baergen
-
Konstantin Shalygin
-
Marc
-
Mark Nelson