Good insights from Alex. Are these clusters all new? Or have they been around a while, previously happier? One idea that comes to mind is an MTU mismatch between hosts and switches, or some manner of bonding misalignment. What does `netstat -I` show? `ethtool -S`? I’m thinking that maybe just maybe bonding (if present) is awry in some fashion such that half of packets in/out disappear into the twilight zone. Like if LACP appears up on the host but a switch issue dooms all packets on one link, in or out.

On Nov 25, 2024, at 9:45 PM, Alex Gorbachev <ag@iss-integration.com> wrote:

Hi Martin,

This is a bit of generic recommendation, but I would go down the path of reducing complexity, i.e. first test the drive locally on the OSD node and see if there's anything going on with e.g. drive firmware, cables, HBA, power.

Then do fio from another host, and this would incorporate networking.

If those look fine, I would do something crazy with Ceph, such as a huge number of PGs, or failure domain of OSD, and just deploy a handful of OSDs to see if you can bring the problem out in the open. I would use a default setup, with no tweaks to scheduler etc. Hopefully, you'll get some error messages in the logs - ceph logs, syslog, dmesg. Maybe at that point it will become more obvious, or at least some messages will come through that will make sense (to you or someone else on the list).

In other words, it seems you have to break this a bit more to get proper diagnostics. I know you guys have played with Ceph before, and can do the math of what the IOPS values should be - three clusters all seeing the same problem would most likely indicate a non-default configuration value that is not correct. -- Alex Gorbachev ISS

...

On Mon, Nov 25, 2024 at 9:34 PM Martin Gerhard Loschwitz < martin.loschwitz@true-west.com> wrote:

Folks,

I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me.

In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me.

I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results?

I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance!

Best regards Martin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Wait … 1 gigabit?? That sure isn’t doing you any favors. Remember that RADOS sends replication sub-ops over that, though you mentioned a size=1 pool. You’ll have mon <----> OSD traffic and OSD <—> OSD heartbeats going over that link as well.

On Nov 26, 2024, at 5:22 AM, Martin Gerhard Loschwitz <martin.loschwitz@true-west.com> wrote:

Hi Anthony,

I think problems have always been like this, albeit these setups are a bit older already. We’ve specifically set the MTU to 9000 on both switches and all affected machines, but MTU 1500 or MTU 9000 literally doesn’t make a difference.

Network is non-LACP on one of the test clusters (the HDD cluster with the worst hardware). It’s a single 1G link, but that should not be a problem for an idling cluster during a normal 4k IOPS test, should it?

Best regards Martin

...

Am 26.11.2024 um 04:48 schrieb Anthony D'Atri <anthony.datri@gmail.com>:

Good insights from Alex.

Are these clusters all new? Or have they been around a while, previously happier?

One idea that comes to mind is an MTU mismatch between hosts and switches, or some manner of bonding misalignment. What does `netstat -I` show? `ethtool -S`? I’m thinking that maybe just maybe bonding (if present) is awry in some fashion such that half of packets in/out disappear into the twilight zone. Like if LACP appears up on the host but a switch issue dooms all packets on one link, in or out.

...

On Nov 25, 2024, at 9:45 PM, Alex Gorbachev <ag@iss-integration.com> wrote:

Hi Martin,

This is a bit of generic recommendation, but I would go down the path of reducing complexity, i.e. first test the drive locally on the OSD node and see if there's anything going on with e.g. drive firmware, cables, HBA, power.

Then do fio from another host, and this would incorporate networking.

If those look fine, I would do something crazy with Ceph, such as a huge number of PGs, or failure domain of OSD, and just deploy a handful of OSDs to see if you can bring the problem out in the open. I would use a default setup, with no tweaks to scheduler etc. Hopefully, you'll get some error messages in the logs - ceph logs, syslog, dmesg. Maybe at that point it will become more obvious, or at least some messages will come through that will make sense (to you or someone else on the list).

In other words, it seems you have to break this a bit more to get proper diagnostics. I know you guys have played with Ceph before, and can do the math of what the IOPS values should be - three clusters all seeing the same problem would most likely indicate a non-default configuration value that is not correct. -- Alex Gorbachev ISS

...

On Mon, Nov 25, 2024 at 9:34 PM Martin Gerhard Loschwitz < martin.loschwitz@true-west.com> wrote:

Folks,

I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me.

In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me.

I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results?

I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance!

Best regards Martin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

--

Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH P +49 2433 5253130 <tel:+49 2433 5253130> M +49 176 61832178 <https://mysig.io/4ngY23j0> A Schmiedegasse 24a, 41836 Hückelhoven, Deutschland R HRB 21985, Amtsgericht Mönchengladbach <https://mysig.io/b4g0y3rz> <https://mysignature.io/editor?utm_source=expiredpixel> True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

May not apply, but usually when I have strange (and bad) behaviours like these, I double check name resolution/DNS configuration on all hosts involved in. Il 26/11/2024 11:19, Martin Gerhard Loschwitz ha scritto:

Hi Alex,

thank you for the reply. Here are all the steps we’ve done in the last weeks to reduce complexity (we’re focussing on the HDD cluster for now in which we are seeing the worst results in relation — but it also happens to be the easiest setup network-wise, despite only having a 1G link between the nodes).

* measure IOPS values per physical device (result was within the expectations for HDDs) * reinstall OS, reset BIOS, reset HBA configuration (or actually, switch Dell PERC to HBA mode)

Current setup is Ubuntu 24.04 with Linux 6.5. This yields better results than 20.04 with some 5.something kernel and Ceph 17 (65 vs. 41 IOPS), but all that is still terrible.

We’re also not seeing anything obvious in iostat. Latency is LAN latency and normal, no packet loss. MTU 1500 or MTU 9000 literally don’t make a difference.

When we disable replication in that setup (pool size=1), we get about 90 IOPS from the same pool. But there is no special network configuration in place. I am attaching a dump of historic OSD ops of an example OSD in the cluster for further reference, maybe somebody sees something obvious in there.

Best regards Martin



...

Am 26.11.2024 um 03:43 schrieb Alex Gorbachev <ag@iss-integration.com>:

Hi Martin,

This is a bit of generic recommendation, but I would go down the path of reducing complexity, i.e. first test the drive locally on the OSD node and see if there's anything going on with e.g. drive firmware, cables, HBA, power.

Then do fio from another host, and this would incorporate networking.

If those look fine, I would do something crazy with Ceph, such as a huge number of PGs, or failure domain of OSD, and just deploy a handful of OSDs to see if you can bring the problem out in the open. I would use a default setup, with no tweaks to scheduler etc. Hopefully, you'll get some error messages in the logs - ceph logs, syslog, dmesg. Maybe at that point it will become more obvious, or at least some messages will come through that will make sense (to you or someone else on the list).

In other words, it seems you have to break this a bit more to get proper diagnostics. I know you guys have played with Ceph before, and can do the math of what the IOPS values should be - three clusters all seeing the same problem would most likely indicate a non-default configuration value that is not correct. -- Alex Gorbachev ISS

On Mon, Nov 25, 2024 at 9:34 PM Martin Gerhard Loschwitz <martin.loschwitz@true-west.com <mailto:martin.loschwitz@true-west.com>> wrote:

...

Folks,

I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me.

In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me.

I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results?

I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance!

Best regards Martin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>

-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Tel +39-0116706701 Fax +39-011751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>

Martin, are MONs set up on the same hosts, or is there latency to them by any chance? -- Alex Gorbachev https://alextelescope.blogspot.com On Tue, Nov 26, 2024 at 5:20 AM Martin Gerhard Loschwitz < martin.loschwitz@true-west.com> wrote:

Hi Alex,

thank you for the reply. Here are all the steps we’ve done in the last weeks to reduce complexity (we’re focussing on the HDD cluster for now in which we are seeing the worst results in relation — but it also happens to be the easiest setup network-wise, despite only having a 1G link between the nodes).

* measure IOPS values per physical device (result was within the expectations for HDDs) * reinstall OS, reset BIOS, reset HBA configuration (or actually, switch Dell PERC to HBA mode)

Current setup is Ubuntu 24.04 with Linux 6.5. This yields better results than 20.04 with some 5.something kernel and Ceph 17 (65 vs. 41 IOPS), but all that is still terrible.

We’re also not seeing anything obvious in iostat. Latency is LAN latency and normal, no packet loss. MTU 1500 or MTU 9000 literally don’t make a difference.

When we disable replication in that setup (pool size=1), we get about 90 IOPS from the same pool. But there is no special network configuration in place. I am attaching a dump of historic OSD ops of an example OSD in the cluster for further reference, maybe somebody sees something obvious in there.

Best regards Martin

Am 26.11.2024 um 03:43 schrieb Alex Gorbachev <ag@iss-integration.com>:

Hi Martin,

This is a bit of generic recommendation, but I would go down the path of reducing complexity, i.e. first test the drive locally on the OSD node and see if there's anything going on with e.g. drive firmware, cables, HBA, power.

Then do fio from another host, and this would incorporate networking.

If those look fine, I would do something crazy with Ceph, such as a huge number of PGs, or failure domain of OSD, and just deploy a handful of OSDs to see if you can bring the problem out in the open. I would use a default setup, with no tweaks to scheduler etc. Hopefully, you'll get some error messages in the logs - ceph logs, syslog, dmesg. Maybe at that point it will become more obvious, or at least some messages will come through that will make sense (to you or someone else on the list).

In other words, it seems you have to break this a bit more to get proper diagnostics. I know you guys have played with Ceph before, and can do the math of what the IOPS values should be - three clusters all seeing the same problem would most likely indicate a non-default configuration value that is not correct. -- Alex Gorbachev ISS

On Mon, Nov 25, 2024 at 9:34 PM Martin Gerhard Loschwitz < martin.loschwitz@true-west.com> wrote:

...

Folks,

I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me.

In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me.

I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results?

I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance!

Best regards Martin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

--

[image: True West IT Services GmbH] Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH P +49 2433 5253130 <+49%202433%205253130> M +49 176 61832178 <https://mysig.io/4ngY23j0> A Schmiedegasse 24a, 41836 Hückelhoven, Deutschland R HRB 21985, Amtsgericht Mönchengladbach <https://mysig.io/b4g0y3rz> <https://mysignature.io/editor?utm_source=expiredpixel> True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

On 26/11/2024 15:09, Peter Grandi wrote:

Regardless of the specifics: 4KiB write IOPS is definitely not what Ceph was designed for. Yet so many people know better and use Ceph for VM disk images, even with logs and databases on them.

It depends...for a single thread/queue depth Ceph will give roughly 3-5k iops read, 1k iops write. Traditional SANs can give 2-5x this iops, specially for writes, so they are much better if you have a low concurrency workload...but if you have a workload that requires hundreds or thousands of concurrent threads: like many vms, each with many applications, with many databases each doing many concurrent transactions, then Ceph makes sense as it will handle this concurrency, whereas your traditional SAN will saturate quickly and will not be able to give the nice performance numbers it had with qd=1.