Performance issues
Hello from Detroit MI: I have been doing some limited benchmarking of a Squid cluster. The arrangement of the cluster: Server Function c01 MGR, MON c02 MGR, MON o01 OSD o02 OSD o03 OSD o04 OSD Each OSD has 2 x NVME disks for Ceph, each at 370 Gig The backing network is as follows: ens18 Gigabit, mon-ip (192.168.0.0/23) regular MTU (1500) ens19 2.5 Gigabit, Cluster Network (10.0.0.0/24) Jumbo MTU (9000) Behind all this is a small ProxMox cluster. All Ceph machines are running on a single node. All Ceph drives are exposed and an NFS mounted NVME drive. All Ceph OSD drives are mounted with no cache and single controller per drive. Networking bridges are all set to either MTU 9000 or MTU 1500 as appropriate. iPerf3 is showing 2.46 Gbit/sec between servers c01 and o01 on the ens19 network. Firewall is off all the way around. OS is CentOS 10. SELinux of disabled. No network enhancements have been performed (increasing send/rcv buffer size, queue length, etc). The concern given all this: rados bench can't exceed 110 MB/s in all tests. In fact if I didn't know better I would swear that the traffic is being either throttled or is somehow routing through a 1Gbit network. The numbers that are returning from rados bench are acting like saturation at Gigabit and not exhibiting any evidence of being on a 2.5 Gbit network. Monitoring at both Ceph and ProxMox consoles confirm the same. Cluster traffic is confirmed to be going out ens19 - tested via tcpdump. Typical command line used for rados bench: rados bench -p s3block 20 write What the heck am I doing wrong here? Ron Gage
As you rightly point out the 110MB/s it sounds very much like the traffic is going through the wrong interface or being limited. So am I correct in my reading of this that this a virtual Ceph environment running on Proxmox? What do you mean by this statement? " All Ceph drives are exposed and an NFS mounted NVME drive. “ Do I take this to mean that you 4 servers are all mounting the same NVME device over NFS? Just a bit confused as to the exact hardware setup here. What is the performance you can get from a single Ceph OSD? Just do a simple dd to read not write to an OSD drive. Darren
On 2 Aug 2025, at 15:25, Ron Gage <ron@rongage.org> wrote:
Hello from Detroit MI:
I have been doing some limited benchmarking of a Squid cluster. The arrangement of the cluster: Server Function c01 MGR, MON c02 MGR, MON o01 OSD o02 OSD o03 OSD o04 OSD
Each OSD has 2 x NVME disks for Ceph, each at 370 Gig
The backing network is as follows: ens18 Gigabit, mon-ip (192.168.0.0/23) regular MTU (1500) ens19 2.5 Gigabit, Cluster Network (10.0.0.0/24) Jumbo MTU (9000)
Behind all this is a small ProxMox cluster. All Ceph machines are running on a single node. All Ceph drives are exposed and an NFS mounted NVME drive. All Ceph OSD drives are mounted with no cache and single controller per drive. Networking bridges are all set to either MTU 9000 or MTU 1500 as appropriate.
iPerf3 is showing 2.46 Gbit/sec between servers c01 and o01 on the ens19 network. Firewall is off all the way around. OS is CentOS 10. SELinux of disabled. No network enhancements have been performed (increasing send/rcv buffer size, queue length, etc).
The concern given all this: rados bench can't exceed 110 MB/s in all tests. In fact if I didn't know better I would swear that the traffic is being either throttled or is somehow routing through a 1Gbit network. The numbers that are returning from rados bench are acting like saturation at Gigabit and not exhibiting any evidence of being on a 2.5 Gbit network. Monitoring at both Ceph and ProxMox consoles confirm the same. Cluster traffic is confirmed to be going out ens19 - tested via tcpdump.
Typical command line used for rados bench: rados bench -p s3block 20 write
What the heck am I doing wrong here?
Ron Gage
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I suspect that the NVMe drives are client-class located on external systems, each with a conventional filesystem with big files that are exported to VMs that mount them as block devices. A lot of layers and media that aren’t up to a sustained workload.
On Aug 2, 2025, at 12:13 PM, darren@soothill.com wrote:
As you rightly point out the 110MB/s it sounds very much like the traffic is going through the wrong interface or being limited.
So am I correct in my reading of this that this a virtual Ceph environment running on Proxmox?
What do you mean by this statement? " All Ceph drives are exposed and an NFS mounted NVME drive. “
Do I take this to mean that you 4 servers are all mounting the same NVME device over NFS? Just a bit confused as to the exact hardware setup here.
What is the performance you can get from a single Ceph OSD? Just do a simple dd to read not write to an OSD drive.
Also, `ceph tell osd.X bench`
Darren
On 2 Aug 2025, at 15:25, Ron Gage <ron@rongage.org> wrote:
Hello from Detroit MI:
I have been doing some limited benchmarking of a Squid cluster. The arrangement of the cluster: Server Function c01 MGR, MON c02 MGR, MON o01 OSD o02 OSD o03 OSD o04 OSD
Each OSD has 2 x NVME disks for Ceph, each at 370 Gig
The backing network is as follows: ens18 Gigabit, mon-ip (192.168.0.0/23) regular MTU (1500) ens19 2.5 Gigabit, Cluster Network (10.0.0.0/24) Jumbo MTU (9000)
Behind all this is a small ProxMox cluster. All Ceph machines are running on a single node. All Ceph drives are exposed and an NFS mounted NVME drive. All Ceph OSD drives are mounted with no cache and single controller per drive. Networking bridges are all set to either MTU 9000 or MTU 1500 as appropriate.
iPerf3 is showing 2.46 Gbit/sec between servers c01 and o01 on the ens19 network. Firewall is off all the way around. OS is CentOS 10. SELinux of disabled. No network enhancements have been performed (increasing send/rcv buffer size, queue length, etc).
The concern given all this: rados bench can't exceed 110 MB/s in all tests. In fact if I didn't know better I would swear that the traffic is being either throttled or is somehow routing through a 1Gbit network. The numbers that are returning from rados bench are acting like saturation at Gigabit and not exhibiting any evidence of being on a 2.5 Gbit network. Monitoring at both Ceph and ProxMox consoles confirm the same. Cluster traffic is confirmed to be going out ens19 - tested via tcpdump.
Typical command line used for rados bench: rados bench -p s3block 20 write
What the heck am I doing wrong here?
Ron Gage
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
All: On 8/2/2025 12:20 PM, Anthony D'Atri wrote:
I suspect that the NVMe drives are client-class located on external systems, each with a conventional filesystem with big files that are exported to VMs that mount them as block devices. A lot of layers and media that aren’t up to a sustained workload.
The NVME is a Samsung 990 Plus. Not exactly entrprise grade, but it should do fairly well. It is also fairly new having picked it up yesterday. It's not going to be 100 MBit - for sure. The NVME connection path: NVME -> USB C Interface -> Nas Server (ubuntu 24.04) -> 2.5 GBit Ethernet (NFS) -> ProxMox vmbr1 Raw device performance from my "nas" system: root@nas:~# hdparm -t /dev/sdb /dev/sdb: Timing buffered disk reads: 1310 MB in 3.00 seconds = 436.21 MB/sec root@nas:~#
On Aug 2, 2025, at 12:13 PM,darren@soothill.com wrote:
As you rightly point out the 110MB/s it sounds very much like the traffic is going through the wrong interface or being limited. Limited - I agree. Wrong interface is unlikely - main network (192.168.0.0/23) has no route to the ceph network and vice versa - no gateway defined on the ceph network.
So am I correct in my reading of this that this a virtual Ceph environment running on Proxmox?
What do you mean by this statement? " All Ceph drives are exposed and an NFS mounted NVME drive. “
Do I take this to mean that you 4 servers are all mounting the same NVME device over NFS? Just a bit confused as to the exact hardware setup here.
What is the performance you can get from a single Ceph OSD? Just do a simple dd to read not write to an OSD drive.
[root@o01 ~]# dd if=/dev/sdb of=10M bs=4k status=progress 5117947904 bytes (5.1 GB, 4.8 GiB) copied, 23 s, 223 MB/s 1310720+0 records in 1310720+0 records out 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 23.9114 s, 225 MB/s [root@o01 ~]#
Also, `ceph tell osd.X bench`
[root@c01 ~]# ceph tell osd.2 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 3.7980597760000001, "bytes_per_sec": 282707984.42536151, "iops": 67.402835947361353 } [root@c01 ~]#
Ron Gage
Darren
On 2 Aug 2025, at 15:25, Ron Gage<ron@rongage.org> wrote:
Hello from Detroit MI:
I have been doing some limited benchmarking of a Squid cluster. The arrangement of the cluster: Server Function c01 MGR, MON c02 MGR, MON o01 OSD o02 OSD o03 OSD o04 OSD
Each OSD has 2 x NVME disks for Ceph, each at 370 Gig
The backing network is as follows: ens18 Gigabit, mon-ip (192.168.0.0/23) regular MTU (1500) ens19 2.5 Gigabit, Cluster Network (10.0.0.0/24) Jumbo MTU (9000)
Behind all this is a small ProxMox cluster. All Ceph machines are running on a single node. All Ceph drives are exposed and an NFS mounted NVME drive. All Ceph OSD drives are mounted with no cache and single controller per drive. Networking bridges are all set to either MTU 9000 or MTU 1500 as appropriate.
iPerf3 is showing 2.46 Gbit/sec between servers c01 and o01 on the ens19 network. Firewall is off all the way around. OS is CentOS 10. SELinux of disabled. No network enhancements have been performed (increasing send/rcv buffer size, queue length, etc).
The concern given all this: rados bench can't exceed 110 MB/s in all tests. In fact if I didn't know better I would swear that the traffic is being either throttled or is somehow routing through a 1Gbit network. The numbers that are returning from rados bench are acting like saturation at Gigabit and not exhibiting any evidence of being on a 2.5 Gbit network. Monitoring at both Ceph and ProxMox consoles confirm the same. Cluster traffic is confirmed to be going out ens19 - tested via tcpdump.
Typical command line used for rados bench: rados bench -p s3block 20 write
What the heck am I doing wrong here?
Ron Gage
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Oops - not paying close enough attention here - my bad Here is the actual performance of the NFS mounted drive: [root@o01 ~]# dd if=/dev/sdc of=/dev/null bs=4k status=progress 57427484672 bytes (57 GB, 53 GiB) copied, 200 s, 287 MB/s It's still running but this is more indicative of how thing are running... Ron Gage On 8/2/2025 12:45 PM, Ron Gage wrote:
Re: [ceph-users] Performance issues
All:
On 8/2/2025 12:20 PM, Anthony D'Atri wrote:
I suspect that the NVMe drives are client-class located on external systems, each with a conventional filesystem with big files that are exported to VMs that mount them as block devices. A lot of layers and media that aren’t up to a sustained workload.
The NVME is a Samsung 990 Plus. Not exactly entrprise grade, but it should do fairly well. It is also fairly new having picked it up yesterday. It's not going to be 100 MBit - for sure.
The NVME connection path: NVME -> USB C Interface -> Nas Server (ubuntu 24.04) -> 2.5 GBit Ethernet (NFS) -> ProxMox vmbr1
Raw device performance from my "nas" system: root@nas:~# hdparm -t /dev/sdb
/dev/sdb: Timing buffered disk reads: 1310 MB in 3.00 seconds = 436.21 MB/sec root@nas:~#
On Aug 2, 2025, at 12:13 PM,darren@soothill.com wrote:
As you rightly point out the 110MB/s it sounds very much like the traffic is going through the wrong interface or being limited. Limited - I agree. Wrong interface is unlikely - main network (192.168.0.0/23) has no route to the ceph network and vice versa - no gateway defined on the ceph network. So am I correct in my reading of this that this a virtual Ceph environment running on Proxmox?
What do you mean by this statement? " All Ceph drives are exposed and an NFS mounted NVME drive. “
Do I take this to mean that you 4 servers are all mounting the same NVME device over NFS? Just a bit confused as to the exact hardware setup here.
What is the performance you can get from a single Ceph OSD? Just do a simple dd to read not write to an OSD drive.
[root@o01 ~]# dd if=/dev/sdb of=10M bs=4k status=progress 5117947904 bytes (5.1 GB, 4.8 GiB) copied, 23 s, 223 MB/s 1310720+0 records in 1310720+0 records out 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 23.9114 s, 225 MB/s [root@o01 ~]#
Also, `ceph tell osd.X bench`
[root@c01 ~]# ceph tell osd.2 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 3.7980597760000001, "bytes_per_sec": 282707984.42536151, "iops": 67.402835947361353 } [root@c01 ~]#
Ron Gage
Darren
On 2 Aug 2025, at 15:25, Ron Gage<ron@rongage.org> wrote:
Hello from Detroit MI:
I have been doing some limited benchmarking of a Squid cluster. The arrangement of the cluster: Server Function c01 MGR, MON c02 MGR, MON o01 OSD o02 OSD o03 OSD o04 OSD
Each OSD has 2 x NVME disks for Ceph, each at 370 Gig
The backing network is as follows: ens18 Gigabit, mon-ip (192.168.0.0/23) regular MTU (1500) ens19 2.5 Gigabit, Cluster Network (10.0.0.0/24) Jumbo MTU (9000)
Behind all this is a small ProxMox cluster. All Ceph machines are running on a single node. All Ceph drives are exposed and an NFS mounted NVME drive. All Ceph OSD drives are mounted with no cache and single controller per drive. Networking bridges are all set to either MTU 9000 or MTU 1500 as appropriate.
iPerf3 is showing 2.46 Gbit/sec between servers c01 and o01 on the ens19 network. Firewall is off all the way around. OS is CentOS 10. SELinux of disabled. No network enhancements have been performed (increasing send/rcv buffer size, queue length, etc).
The concern given all this: rados bench can't exceed 110 MB/s in all tests. In fact if I didn't know better I would swear that the traffic is being either throttled or is somehow routing through a 1Gbit network. The numbers that are returning from rados bench are acting like saturation at Gigabit and not exhibiting any evidence of being on a 2.5 Gbit network. Monitoring at both Ceph and ProxMox consoles confirm the same. Cluster traffic is confirmed to be going out ens19 - tested via tcpdump.
Typical command line used for rados bench: rados bench -p s3block 20 write
What the heck am I doing wrong here?
Ron Gage
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Here is the actual performance of the NFS mounted drive: [root@o01 ~]# dd if=/dev/sdc of=/dev/null bs=4k status=progress
Null writes aren’t a good test, as they may be optimized away by any layer in the stack. I suggest repeating with /dev/urandom.
The NVME is a Samsung 990 Plus. Not exactly entrprise grade, but it should do fairly well. It is also fairly new having picked it up yesterday. It's not going to be 100 MBit - for sure.
Have you applied the most recent firmware? Issues have been reported in the past. Note that in a power loss situation you may corrupt or lose data due to the apparent lack of PLP. With updated firmware this is still a 0.33 DPWD drive, fwiw.
The NVME connection path: NVME -> USB C Interface -> Nas Server (ubuntu 24.04) -> 2.5 GBit Ethernet (NFS) -> ProxMox vmbr1
Why the deep stack? Why not have the OSD drives in cluster nodes with Ceph deployed converged? That would be a lot less complicated. * Are all of the OSDs exported from the same NAS? SPoF * The USB layer introduces latency * Since you’re exporting via NFS, I assume that the USB M.2 drives have a local filesystem built on the NAS node, and large files created to export? The filesystem layer introduces additional latency. * The NFS layer introduces latency * Proxmox drive / network emulation introduces additional latency * If your cluster network is virtualized through Proxmox SDN, that’s additional latency. Remember that every write is farmed out to other OSDs and has to through all those layers. With all theses factors honestly you’re getting better perf than I would have expected.
On 8/2/2025 4:07 PM, Anthony D'Atri wrote:
Here is the actual performance of the NFS mounted drive: [root@o01 ~]# dd if=/dev/sdc of=/dev/null bs=4k status=progress Null writes aren’t a good test, as they may be optimized away by any layer in the stack. I suggest repeating with /dev/urandom.
I'm confused - why would I use urandom for testing raw read performance on a local drive at the OSD level?
The NVME is a Samsung 990 Plus. Not exactly entrprise grade, but it should do fairly well. It is also fairly new having picked it up yesterday. It's not going to be 100 MBit - for sure. Have you applied the most recent firmware? Issues have been reported in the past. Note that in a power loss situation you may corrupt or lose data due to the apparent lack of PLP. With updated firmware this is still a 0.33 DPWD drive, fwiw.
The NVME connection path: NVME -> USB C Interface -> Nas Server (ubuntu 24.04) -> 2.5 GBit Ethernet (NFS) -> ProxMox vmbr1 Why the deep stack? Why not have the OSD drives in cluster nodes with Ceph deployed converged? That would be a lot less complicated. Yes, it would be a lot less complicated if I had the physical space to
No on the recent firmware. Again, the raw read data already posted and the ceph tell bench data all support the idea that I'm getting over 2Gb on this drive at the endpoint. Also, the DWPD rating is not germane to this subject at all. Yes, it is low for a 4TB device but this is a test bench, not a enterprise SAN. put it, plus using it as an NFS export makes possible future sharing to the rest of this cluster.
* Are all of the OSDs exported from the same NAS? SPoF
TestLab
* The USB layer introduces latency * Since you’re exporting via NFS, I assume that the USB M.2 drives have a local filesystem built on the NAS node, and large files created to export? The filesystem layer introduces additional latency. * The NFS layer introduces latency * Proxmox drive / network emulation introduces additional latency * If your cluster network is virtualized through Proxmox SDN, that’s additional latency. Remember that every write is farmed out to other OSDs and has to through all those layers.
With all theses factors honestly you’re getting better perf than I would have expected.
I am getting exactly what I was expecting - near/at saturation levels on my 2.5G ethernet network. I am going to move forward with (probably mistaken) premise that the bench function is the problem here. On to other testing - S3 performance. Ron Gage
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
All: For grins - I decided to rebuild the pool as replicated x3, no compression. No discernible difference in the rados bench numbers. Bandwidth still maxing out at 111 MB/sec. The previous pool was an EC pool 6x1 with passive compression (snappy) on. The big tell here: no performance difference seen between replicated and EC. Ron Gage On 8/2/2025 12:20 PM, Anthony D'Atri wrote:
I suspect that the NVMe drives are client-class located on external systems, each with a conventional filesystem with big files that are exported to VMs that mount them as block devices. A lot of layers and media that aren’t up to a sustained workload.
On Aug 2, 2025, at 12:13 PM, darren@soothill.com wrote:
As you rightly point out the 110MB/s it sounds very much like the traffic is going through the wrong interface or being limited.
So am I correct in my reading of this that this a virtual Ceph environment running on Proxmox?
What do you mean by this statement? " All Ceph drives are exposed and an NFS mounted NVME drive. “
Do I take this to mean that you 4 servers are all mounting the same NVME device over NFS? Just a bit confused as to the exact hardware setup here.
What is the performance you can get from a single Ceph OSD? Just do a simple dd to read not write to an OSD drive. Also, `ceph tell osd.X bench`
Darren
On 2 Aug 2025, at 15:25, Ron Gage <ron@rongage.org> wrote:
Hello from Detroit MI:
I have been doing some limited benchmarking of a Squid cluster. The arrangement of the cluster: Server Function c01 MGR, MON c02 MGR, MON o01 OSD o02 OSD o03 OSD o04 OSD
Each OSD has 2 x NVME disks for Ceph, each at 370 Gig
The backing network is as follows: ens18 Gigabit, mon-ip (192.168.0.0/23) regular MTU (1500) ens19 2.5 Gigabit, Cluster Network (10.0.0.0/24) Jumbo MTU (9000)
Behind all this is a small ProxMox cluster. All Ceph machines are running on a single node. All Ceph drives are exposed and an NFS mounted NVME drive. All Ceph OSD drives are mounted with no cache and single controller per drive. Networking bridges are all set to either MTU 9000 or MTU 1500 as appropriate.
iPerf3 is showing 2.46 Gbit/sec between servers c01 and o01 on the ens19 network. Firewall is off all the way around. OS is CentOS 10. SELinux of disabled. No network enhancements have been performed (increasing send/rcv buffer size, queue length, etc).
The concern given all this: rados bench can't exceed 110 MB/s in all tests. In fact if I didn't know better I would swear that the traffic is being either throttled or is somehow routing through a 1Gbit network. The numbers that are returning from rados bench are acting like saturation at Gigabit and not exhibiting any evidence of being on a 2.5 Gbit network. Monitoring at both Ceph and ProxMox consoles confirm the same. Cluster traffic is confirmed to be going out ens19 - tested via tcpdump.
Typical command line used for rados bench: rados bench -p s3block 20 write
What the heck am I doing wrong here?
Ron Gage
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi. Ceph supports primarily two types of networks: Public network (front-side network): this network handles client traffic and communication with Ceph monitors. All clients access the Ceph cluster through this network. It is mandatory for Ceph operation. Cluster Network (back-side network): this is optional but recommended for performance and security. The cluster network is used for internal Ceph operations such as OSD heartbeat, data replication, backfilling, and recovery traffic. It is preferred that this network is separate from the public network and not reachable from the internet for added security. If your cluster network is 2.5 Gbps and your public network is 1 Gbps, your clients will be limited to 1 Gbps speeds - unless they use some kind of LAG and access multiple Ceph nodes at the same time, which is a slightly different scenario. /Z On Sat, 2 Aug 2025 at 17:26, Ron Gage <ron@rongage.org> wrote:
Hello from Detroit MI:
I have been doing some limited benchmarking of a Squid cluster. The arrangement of the cluster: Server Function c01 MGR, MON c02 MGR, MON o01 OSD o02 OSD o03 OSD o04 OSD
Each OSD has 2 x NVME disks for Ceph, each at 370 Gig
The backing network is as follows: ens18 Gigabit, mon-ip (192.168.0.0/23) regular MTU (1500) ens19 2.5 Gigabit, Cluster Network (10.0.0.0/24) Jumbo MTU (9000)
Behind all this is a small ProxMox cluster. All Ceph machines are running on a single node. All Ceph drives are exposed and an NFS mounted NVME drive. All Ceph OSD drives are mounted with no cache and single controller per drive. Networking bridges are all set to either MTU 9000 or MTU 1500 as appropriate.
iPerf3 is showing 2.46 Gbit/sec between servers c01 and o01 on the ens19 network. Firewall is off all the way around. OS is CentOS 10. SELinux of disabled. No network enhancements have been performed (increasing send/rcv buffer size, queue length, etc).
The concern given all this: rados bench can't exceed 110 MB/s in all tests. In fact if I didn't know better I would swear that the traffic is being either throttled or is somehow routing through a 1Gbit network. The numbers that are returning from rados bench are acting like saturation at Gigabit and not exhibiting any evidence of being on a 2.5 Gbit network. Monitoring at both Ceph and ProxMox consoles confirm the same. Cluster traffic is confirmed to be going out ens19 - tested via tcpdump.
Typical command line used for rados bench: rados bench -p s3block 20 write
What the heck am I doing wrong here?
Ron Gage
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
darren＠soothill.com
-
Ron Gage
-
Zakhar Kirpichenko