Hello everyone, We are running Ceph in Proxmox with a 10G network. Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link <https://forum.proxmox.com/threads/slow-performance-on-ceph-per-vm.151223/#post-685070> For this reason, we are considering purchasing a 100G switch for our servers. This raises some questions: Should I still use separate networks for VMs and Ceph with 100G? I have read that running Ceph on bridged connections is not recommended. Does anyone have experience with 100G Ceph in Proxmox? Is upgrading to 100G a good idea, or will I have 60G sitting idle? Thanks in advance! Gio
Hi Giovanna, El 11/3/25 a las 11:55, Giovanna Ratini escribió:
We are running Ceph in Proxmox with a 10G network.
Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link <https://forum.proxmox.com/threads/slow-performance-on-ceph-per-vm.151223/#post-685070>
For this reason, we are considering purchasing a 100G switch for our servers.
This raises some questions: Should I still use separate networks for VMs and Ceph with 100G? I have read that running Ceph on bridged connections is not recommended.
Does anyone have experience with 100G Ceph in Proxmox?
Is upgrading to 100G a good idea, or will I have 60G sitting idle?
I think you should give more info on your setup and those low read rates (what numbers? how do you get them?), so that community can suggest improvements. If you're getting 80MB/s the network is not your bottleneck (10G is 1GB/s) and upgrading to 100G network won't help much. Cheers Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
Agree that the net is likely not your problem, though you should use iftop et al to look for saturation. Check that you have proper smith hash policy, otherwise you may not be using both bond links. The linked thread mentions SSDPE2KE032T8 which is SATA. The replication network is always optional. Just a matter of saturation. The linked article discusses MSFT clients, is that what you have?
On Mar 11, 2025, at 7:13 AM, Eneko Lacunza <elacunza@binovo.es> wrote:
Hi Giovanna,
El 11/3/25 a las 11:55, Giovanna Ratini escribió:
We are running Ceph in Proxmox with a 10G network.
Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link <https://forum.proxmox.com/threads/slow-performance-on-ceph-per-vm.151223/#post-685070>
For this reason, we are considering purchasing a 100G switch for our servers.
This raises some questions: Should I still use separate networks for VMs and Ceph with 100G? I have read that running Ceph on bridged connections is not recommended.
Does anyone have experience with 100G Ceph in Proxmox?
Is upgrading to 100G a good idea, or will I have 60G sitting idle?
I think you should give more info on your setup and those low read rates (what numbers? how do you get them?), so that community can suggest improvements.
If you're getting 80MB/s the network is not your bottleneck (10G is 1GB/s) and upgrading to 100G network won't help much.
Cheers
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Mar 11, 2025 at 08:50:55AM -0400, Anthony D'Atri wrote:
Agree that the net is likely not your problem, though you should use iftop et al to look for saturation.
Check that you have proper smith hash policy, otherwise you may not be using both bond links.
The linked thread mentions SSDPE2KE032T8 which is SATA.
Hmm? Specs at various places for this SSD say: PCIe Gen 3.1 X4 and NVMe as the protocol. Kind regards, Alex. -- "Opportunity is missed by most people because it is dressed in overalls and looks like work." -- Thomas A. Edison
Ah, my fault. On my phone the first search hit implied that this SKU was the S4520.
On Mar 11, 2025, at 10:17 AM, Alexander Schreiber <als@thangorodrim.ch> wrote:
On Tue, Mar 11, 2025 at 08:50:55AM -0400, Anthony D'Atri wrote:
Agree that the net is likely not your problem, though you should use iftop et al to look for saturation.
Check that you have proper smith hash policy, otherwise you may not be using both bond links.
The linked thread mentions SSDPE2KE032T8 which is SATA.
Hmm? Specs at various places for this SSD say: PCIe Gen 3.1 X4 and NVMe as the protocol.
Kind regards, Alex. -- "Opportunity is missed by most people because it is dressed in overalls and looks like work." -- Thomas A. Edison
How about testing the actual network throughput with iperf? Even today there are speed/duplex mismatches on switch ports. And what everyone else said about saturation etc. We get, at absolute worst, 600 MB/s on a 10G connection. -- Alex Gorbachev https://alextelescope.blogspot.com On Tue, Mar 11, 2025 at 6:57 AM Giovanna Ratini < giovanna.ratini@uni-konstanz.de> wrote:
Hello everyone,
We are running Ceph in Proxmox with a 10G network.
Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link < https://forum.proxmox.com/threads/slow-performance-on-ceph-per-vm.151223/#po...
For this reason, we are considering purchasing a 100G switch for our servers.
This raises some questions: Should I still use separate networks for VMs and Ceph with 100G? I have read that running Ceph on bridged connections is not recommended.
Does anyone have experience with 100G Ceph in Proxmox?
Is upgrading to 100G a good idea, or will I have 60G sitting idle?
Thanks in advance!
Gio
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello 😊, Thank you very much for your response. I give you more information. I do not have MS VM. I only have Debians and Ubuntu VMs. I have a *Proxmox Cluster* with *6 hosts*. The network setup is as follows: * *10G link* for Ceph Cluster * *10G link* for Ceph public * *1G link* for Corosync * *1G IPMI* * *10G link* for VMs Each host has *2 or 3 OSDs (15TB NVMe)*. The hosts are *heterogeneous*, but all have *512GB RAM*. I do not observe any bottlenecks in *htop or iftop*, and *iostat* reports only *0.12% iowait*. However, *fio* test results are concerning. Here is the *fio* command I used: fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=4): [r(4)][100.0%][r=39.8MiB/s][r=10.2k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=231332: Sun Mar 16 22:30:24 2025 read: IOPS=10.2k, BW=39.7MiB/s (41.7MB/s)(2385MiB/60001msec) slat (usec): min=194, max=13111, avg=390.63, stdev=80.29 clat (nsec): min=910, max=190362, avg=1521.76, stdev=873.64 lat (usec): min=195, max=13114, avg=392.15, stdev=80.35 clat percentiles (nsec): | 1.00th=[ 1112], 5.00th=[ 1208], 10.00th=[ 1224], 20.00th=[ 1272], | 30.00th=[ 1288], 40.00th=[ 1320], 50.00th=[ 1352], 60.00th=[ 1400], | 70.00th=[ 1496], 80.00th=[ 1704], 90.00th=[ 1960], 95.00th=[ 2224], | 99.00th=[ 2832], 99.50th=[ 3856], 99.90th=[12096], 99.95th=[16768], | 99.99th=[26240] bw ( KiB/s): min=31984, max=43288, per=100.00%, avg=40730.22, stdev=381.52, samples=476 iops : min= 7996, max=10822, avg=10182.55, stdev=95.38, samples=476 lat (nsec) : 1000=0.02% lat (usec) : 2=91.02%, 4=8.48%, 10=0.32%, 20=0.12%, 50=0.03% lat (usec) : 100=0.01%, 250=0.01% cpu : usr=0.80%, sys=5.99%, ctx=610640, majf=0, minf=47 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=610483,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=39.7MiB/s (41.7MB/s), 39.7MiB/s-39.7MiB/s (41.7MB/s-41.7MB/s), io=2385MiB (2501MB), run=60001-60001msec Summary: *Test Results:* * *IOPS:* 10.2k * *Bandwidth:* 39.7MiB/s (41.7MB/s) * *Latency:* o Avg: *392µs* o 99.9th percentile: *12ms* * *CPU Usage:* usr=0.80%, sys=5.99% Kind regards, Gio Am 11.03.2025 um 11:55 schrieb Giovanna Ratini:
Hello everyone,
We are running Ceph in Proxmox with a 10G network.
Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link <https://forum.proxmox.com/threads/slow-performance-on-ceph-per-vm.151223/#post-685070>
For this reason, we are considering purchasing a 100G switch for our servers.
This raises some questions: Should I still use separate networks for VMs and Ceph with 100G? I have read that running Ceph on bridged connections is not recommended.
Does anyone have experience with 100G Ceph in Proxmox?
Is upgrading to 100G a good idea, or will I have 60G sitting idle?
Thanks in advance!
Gio
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So looking at this your IOPS are controlled by latency. You have 2 options to increase your IOPS reduce the latency OR increase the number of IOs in flight. You currently are running an iodepth of 1 and Ceph is designed to run many IOs in parallel so you could either increase the number of jobs or increase the IOdepth. I would be increasing the iodepth to 16 or 32 as a starting point. IOPs should then increase until the latency starts to jump up which would then mean you are finding the limits of some other component. The other thing then is to look at how you can decrease latency if you could in theory half the latency by doing direct hardware access then you would double the IOPs. But what I would say is 4 concurrent IOs is never going to stress a large number of NVMEs. Sent from my iPhone
On 16 Mar 2025, at 21:42, Giovanna Ratini <giovanna.ratini@uni-konstanz.de> wrote:
Hello 😊,
Thank you very much for your response.
I give you more information. I do not have MS VM. I only have Debians and Ubuntu VMs.
I have a *Proxmox Cluster* with *6 hosts*. The network setup is as follows:
* *10G link* for Ceph Cluster * *10G link* for Ceph public * *1G link* for Corosync * *1G IPMI* * *10G link* for VMs
Each host has *2 or 3 OSDs (15TB NVMe)*. The hosts are *heterogeneous*, but all have *512GB RAM*.
I do not observe any bottlenecks in *htop or iftop*, and *iostat* reports only *0.12% iowait*. However, *fio* test results are concerning.
Here is the *fio* command I used:
fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=4): [r(4)][100.0%][r=39.8MiB/s][r=10.2k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=231332: Sun Mar 16 22:30:24 2025 read: IOPS=10.2k, BW=39.7MiB/s (41.7MB/s)(2385MiB/60001msec) slat (usec): min=194, max=13111, avg=390.63, stdev=80.29 clat (nsec): min=910, max=190362, avg=1521.76, stdev=873.64 lat (usec): min=195, max=13114, avg=392.15, stdev=80.35 clat percentiles (nsec): | 1.00th=[ 1112], 5.00th=[ 1208], 10.00th=[ 1224], 20.00th=[ 1272], | 30.00th=[ 1288], 40.00th=[ 1320], 50.00th=[ 1352], 60.00th=[ 1400], | 70.00th=[ 1496], 80.00th=[ 1704], 90.00th=[ 1960], 95.00th=[ 2224], | 99.00th=[ 2832], 99.50th=[ 3856], 99.90th=[12096], 99.95th=[16768], | 99.99th=[26240] bw ( KiB/s): min=31984, max=43288, per=100.00%, avg=40730.22, stdev=381.52, samples=476 iops : min= 7996, max=10822, avg=10182.55, stdev=95.38, samples=476 lat (nsec) : 1000=0.02% lat (usec) : 2=91.02%, 4=8.48%, 10=0.32%, 20=0.12%, 50=0.03% lat (usec) : 100=0.01%, 250=0.01% cpu : usr=0.80%, sys=5.99%, ctx=610640, majf=0, minf=47 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=610483,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=39.7MiB/s (41.7MB/s), 39.7MiB/s-39.7MiB/s (41.7MB/s-41.7MB/s), io=2385MiB (2501MB), run=60001-60001msec
Summary:
*Test Results:*
* *IOPS:* 10.2k * *Bandwidth:* 39.7MiB/s (41.7MB/s) * *Latency:* o Avg: *392µs* o 99.9th percentile: *12ms* * *CPU Usage:* usr=0.80%, sys=5.99%
Kind regards, Gio
Am 11.03.2025 um 11:55 schrieb Giovanna Ratini: Hello everyone,
We are running Ceph in Proxmox with a 10G network.
Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link <https://forum.proxmox.com/threads/slow-performance-on-ceph-per-vm.151223/#post-685070>
For this reason, we are considering purchasing a 100G switch for our servers.
This raises some questions: Should I still use separate networks for VMs and Ceph with 100G? I have read that running Ceph on bridged connections is not recommended.
Does anyone have experience with 100G Ceph in Proxmox?
Is upgrading to 100G a good idea, or will I have 60G sitting idle?
Thanks in advance!
Gio
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Which NVMe drive SKUs specifically? Are you running a recent kernel? Have you updated firmware on the NVMe devices?
On Mar 11, 2025, at 6:55 AM, Giovanna Ratini <giovanna.ratini@uni-konstanz.de> wrote:
However, even 80 MB per second with an NVMe drive is quite disappointing.
Hello again, I tried running tests with *--iodepth=16* and *32*. The values got even worse. # *IOPS*: *8.7k* # *Bandwidth*: *34.1MiB/s (35.7MB/s)* # *Latency*: * *Avg*: *7.3ms* * *99.9th percentile*: *15.8ms* # *CPU Usage*: *usr=0.74%, sys=5.60%* The problem seems to be only inside the VMs. VMs used: scsi0: cephvm:vm-6506-disk-1,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,cache=writeback,iothread=1,size=10T,ssd=1 I try without cache, without iothread. There is no changes. Then I tested on the *Proxmox host*, and the results were significantly better. fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G --runtime=60 --group_reporting *IOPS*: *1.54M* # *Bandwidth*: *6032MiB/s (6325MB/s)* # *Latency*: * *Avg*: *39.8µs* * *99.9th percentile*: *71µs* # *CPU Usage*: *usr=22.60%, sys=77.13%* # Am 18.03.2025 um 15:27 schrieb Anthony D'Atri:
Which NVMe drive SKUs specifically?
Are you running a recent kernel?
# */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 penultimate: 6.8.12-8-pve (VM, yes)
Have you updated firmware on the NVMe devices?
No. Thanks again, best regards, Gio
Then I tested on the *Proxmox host*, and the results were significantly better.
My Proxmox prowess is limited, but from my experience with other virtualization platforms, I have to ask if there is any QoS throttling applied to VMs. With OpenStack or DO there is often IOPS and/or throughput throttling via libvirt to mitigate noisy neighbors.
fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G --runtime=60 --group_reporting
*IOPS*: *1.54M*
# *Bandwidth*: *6032MiB/s (6325MB/s)* # *Latency*:
* *Avg*: *39.8µs* * *99.9th percentile*: *71µs*
# *CPU Usage*: *usr=22.60%, sys=77.13%* #
Am 18.03.2025 um 15:27 schrieb Anthony D'Atri:
Which NVMe drive SKUs specifically?
# */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8
Kioxia CD6. If you were using client-class drives all manner of performance issues would be expected. Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may hamper these drives. Also, how many of these are in your cluster? If it’s a small number you might still benefit from chopping each into at least 2 separate OSDs. And please send `ceph osd dump | grep pool`, having too few PGs wouldn’t do you any favors.
Are you running a recent kernel? penultimate: 6.8.12-8-pve (VM, yes)
Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then NVMe issues might be expected as old kernels had rudimentary NVMe support.
Have you updated firmware on the NVMe devices?
No.
Kioxia appears to not release firmware updates publicly but your chassis brand (Dell, HP, SMCI, etc) might have an update. e.g. https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... If there is an available update I would strongly suggest applying.
Thanks again,
best regards, Gio
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Antony, no, no QoS applied to Vms. The Server has PCIe Gen 4 ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 13.04 pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 application cephfs read_balance_score 2.02 pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.42 pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.52 I think, this is the default config. 🙈 I will search for my chassies supermicro upgrade. Thank you Am 18.03.2025 um 17:57 schrieb Anthony D'Atri:
Then I tested on the *Proxmox host*, and the results were significantly better. My Proxmox prowess is limited, but from my experience with other virtualization platforms, I have to ask if there is any QoS throttling applied to VMs. With OpenStack or DO there is often IOPS and/or throughput throttling via libvirt to mitigate noisy neighbors.
fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G --runtime=60 --group_reporting
*IOPS*: *1.54M*
# *Bandwidth*: *6032MiB/s (6325MB/s)* # *Latency*:
* *Avg*: *39.8µs* * *99.9th percentile*: *71µs*
# *CPU Usage*: *usr=22.60%, sys=77.13%* #
Am 18.03.2025 um 15:27 schrieb Anthony D'Atri:
Which NVMe drive SKUs specifically? # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 Kioxia CD6. If you were using client-class drives all manner of performance issues would be expected.
Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may hamper these drives.
Also, how many of these are in your cluster? If it’s a small number you might still benefit from chopping each into at least 2 separate OSDs.
And please send `ceph osd dump | grep pool`, having too few PGs wouldn’t do you any favors.
Are you running a recent kernel? penultimate: 6.8.12-8-pve (VM, yes) Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then NVMe issues might be expected as old kernels had rudimentary NVMe support.
Have you updated firmware on the NVMe devices? No. Kioxia appears to not release firmware updates publicly but your chassis brand (Dell, HP, SMCI, etc) might have an update. e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
If there is an available update I would strongly suggest applying.
Thanks again,
best regards, Gio
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
On Mar 18, 2025, at 2:13 PM, Giovanna Ratini <giovanna.ratini@uni-konstanz.de> wrote:
Hello Antony,
no, no QoS applied to Vms.
The Server has PCIe Gen 4
ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 13.04 pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 application cephfs read_balance_score 2.02 pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.42 pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.52
I think, this is the default config. 🙈
Yes, with the autoscaler on. I suggest raising mon_target_pg_per_osd to 250. How many OSDs do you have?
I will search for my chassies supermicro upgrade.
Thank you
Am 18.03.2025 um 17:57 schrieb Anthony D'Atri:
Then I tested on the *Proxmox host*, and the results were significantly better. My Proxmox prowess is limited, but from my experience with other virtualization platforms, I have to ask if there is any QoS throttling applied to VMs. With OpenStack or DO there is often IOPS and/or throughput throttling via libvirt to mitigate noisy neighbors.
fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G --runtime=60 --group_reporting
*IOPS*: *1.54M*
# *Bandwidth*: *6032MiB/s (6325MB/s)* # *Latency*:
* *Avg*: *39.8µs* * *99.9th percentile*: *71µs*
# *CPU Usage*: *usr=22.60%, sys=77.13%* #
Am 18.03.2025 um 15:27 schrieb Anthony D'Atri:
Which NVMe drive SKUs specifically? # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 Kioxia CD6. If you were using client-class drives all manner of performance issues would be expected.
Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may hamper these drives.
Also, how many of these are in your cluster? If it’s a small number you might still benefit from chopping each into at least 2 separate OSDs.
And please send `ceph osd dump | grep pool`, having too few PGs wouldn’t do you any favors.
Are you running a recent kernel? penultimate: 6.8.12-8-pve (VM, yes) Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then NVMe issues might be expected as old kernels had rudimentary NVMe support.
Have you updated firmware on the NVMe devices? No. Kioxia appears to not release firmware updates publicly but your chassis brand (Dell, HP, SMCI, etc) might have an update. e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
If there is an available update I would strongly suggest applying.
Thanks again,
best regards, Gio
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Giovanna, Have you tried increasing iothreads option for the VM? Cheers El 18/3/25 a las 19:13, Giovanna Ratini escribió:
Hello Antony,
no, no QoS applied to Vms.
The Server has PCIe Gen 4
ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 13.04 pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 application cephfs read_balance_score 2.02 pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.42 pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.52
I think, this is the default config. 🙈
I will search for my chassies supermicro upgrade.
Thank you
Am 18.03.2025 um 17:57 schrieb Anthony D'Atri:
Then I tested on the *Proxmox host*, and the results were significantly better. My Proxmox prowess is limited, but from my experience with other virtualization platforms, I have to ask if there is any QoS throttling applied to VMs. With OpenStack or DO there is often IOPS and/or throughput throttling via libvirt to mitigate noisy neighbors.
fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G --runtime=60 --group_reporting
*IOPS*: *1.54M*
# *Bandwidth*: *6032MiB/s (6325MB/s)* # *Latency*:
* *Avg*: *39.8µs* * *99.9th percentile*: *71µs*
# *CPU Usage*: *usr=22.60%, sys=77.13%* #
Which NVMe drive SKUs specifically? # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 Kioxia CD6. If you were using client-class drives all manner of
Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: performance issues would be expected.
Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may hamper these drives.
Also, how many of these are in your cluster? If it’s a small number you might still benefit from chopping each into at least 2 separate OSDs.
And please send `ceph osd dump | grep pool`, having too few PGs wouldn’t do you any favors.
Are you running a recent kernel? penultimate: 6.8.12-8-pve (VM, yes) Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then NVMe issues might be expected as old kernels had rudimentary NVMe support.
Have you updated firmware on the NVMe devices? No. Kioxia appears to not release firmware updates publicly but your chassis brand (Dell, HP, SMCI, etc) might have an update. e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
If there is an available update I would strongly suggest applying.
Thanks again,
best regards, Gio
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
hello Eneko, Yes I did. No significant changes. :-( Cheers, Gio Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>: Hi Giovanna, Have you tried increasing iothreads option for the VM? Cheers El 18/3/25 a las 19:13, Giovanna Ratini escribió:
Hello Antony,
no, no QoS applied to Vms.
The Server has PCIe Gen 4
ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 13.04 pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 application cephfs read_balance_score 2.02 pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.42 pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.52
I think, this is the default config. 🙈
I will search for my chassies supermicro upgrade.
Thank you
Am 18.03.2025 um 17:57 schrieb Anthony D'Atri:
Then I tested on the *Proxmox host*, and the results were significantly better. My Proxmox prowess is limited, but from my experience with other virtualization platforms, I have to ask if there is any QoS throttling applied to VMs. With OpenStack or DO there is often IOPS and/or throughput throttling via libvirt to mitigate noisy neighbors.
fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G --runtime=60 --group_reporting
*IOPS*: *1.54M*
# *Bandwidth*: *6032MiB/s (6325MB/s)* # *Latency*:
* *Avg*: *39.8µs* * *99.9th percentile*: *71µs*
# *CPU Usage*: *usr=22.60%, sys=77.13%* #
Which NVMe drive SKUs specifically? # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 Kioxia CD6. If you were using client-class drives all manner of
Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: performance issues would be expected.
Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may hamper these drives.
Also, how many of these are in your cluster? If it’s a small number you might still benefit from chopping each into at least 2 separate OSDs.
And please send `ceph osd dump | grep pool`, having too few PGs wouldn’t do you any favors.
Are you running a recent kernel? penultimate: 6.8.12-8-pve (VM, yes) Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then NVMe issues might be expected as old kernels had rudimentary NVMe support.
Have you updated firmware on the NVMe devices? No. Kioxia appears to not release firmware updates publicly but your chassis brand (Dell, HP, SMCI, etc) might have an update. e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
If there is an available update I would strongly suggest applying.
Thanks again,
best regards, Gio
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Giovanna, Can you post VM's full config? Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks? Cheers El 19/3/25 a las 17:27, Giovanna Ratini escribió:
hello Eneko,
Yes I did. No significant changes. :-( Cheers,
Gio
Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>:
Hi Giovanna,
Have you tried increasing iothreads option for the VM?
Cheers
El 18/3/25 a las 19:13, Giovanna Ratini escribió:
Hello Antony,
no, no QoS applied to Vms.
The Server has PCIe Gen 4
ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 13.04 pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 application cephfs read_balance_score 2.02 pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.42 pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.52
I think, this is the default config. 🙈
I will search for my chassies supermicro upgrade.
Thank you
Am 18.03.2025 um 17:57 schrieb Anthony D'Atri:
Then I tested on the *Proxmox host*, and the results were significantly better. My Proxmox prowess is limited, but from my experience with other virtualization platforms, I have to ask if there is any QoS throttling applied to VMs. With OpenStack or DO there is often IOPS and/or throughput throttling via libvirt to mitigate noisy neighbors.
fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G --runtime=60 --group_reporting
*IOPS*: *1.54M*
# *Bandwidth*: *6032MiB/s (6325MB/s)* # *Latency*:
* *Avg*: *39.8µs* * *99.9th percentile*: *71µs*
# *CPU Usage*: *usr=22.60%, sys=77.13%* #
Which NVMe drive SKUs specifically? # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 Kioxia CD6. If you were using client-class drives all manner of
Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: performance issues would be expected.
Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may hamper these drives.
Also, how many of these are in your cluster? If it’s a small number you might still benefit from chopping each into at least 2 separate OSDs.
And please send `ceph osd dump | grep pool`, having too few PGs wouldn’t do you any favors.
Are you running a recent kernel? penultimate: 6.8.12-8-pve (VM, yes) Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then NVMe issues might be expected as old kernels had rudimentary NVMe support.
Have you updated firmware on the NVMe devices? No. Kioxia appears to not release firmware updates publicly but your chassis brand (Dell, HP, SMCI, etc) might have an update.
e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
If there is an available update I would strongly suggest applying.
Thanks again,
best regards, Gio
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
EnekoLacunza Director Técnico | Zuzendari teknikoa Binovo IT Human Project 943 569 206 <tel:943 569 206> elacunza@binovo.es <mailto:elacunza@binovo.es> binovo.es <//binovo.es> Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/>
Hello Eneko, this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most. agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16 Cheers, Gio Am 20.03.2025 um 10:23 schrieb Eneko Lacunza:
Hi Giovanna,
Can you post VM's full config?
Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks?
Cheers
El 19/3/25 a las 17:27, Giovanna Ratini escribió:
hello Eneko,
Yes I did. No significant changes. :-( Cheers,
Gio
Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>:
Hi Giovanna,
Have you tried increasing iothreads option for the VM?
Cheers
El 18/3/25 a las 19:13, Giovanna Ratini escribió:
Hello Antony,
no, no QoS applied to Vms.
The Server has PCIe Gen 4
ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 13.04 pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 application cephfs read_balance_score 2.02 pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.42 pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.52
I think, this is the default config. 🙈
I will search for my chassies supermicro upgrade.
Thank you
Am 18.03.2025 um 17:57 schrieb Anthony D'Atri:
Then I tested on the *Proxmox host*, and the results were significantly better. My Proxmox prowess is limited, but from my experience with other virtualization platforms, I have to ask if there is any QoS throttling applied to VMs. With OpenStack or DO there is often IOPS and/or throughput throttling via libvirt to mitigate noisy neighbors.
fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G --runtime=60 --group_reporting
*IOPS*: *1.54M*
# *Bandwidth*: *6032MiB/s (6325MB/s)* # *Latency*:
* *Avg*: *39.8µs* * *99.9th percentile*: *71µs*
# *CPU Usage*: *usr=22.60%, sys=77.13%* #
Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: > Which NVMe drive SKUs specifically? # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 Kioxia CD6. If you were using client-class drives all manner of performance issues would be expected.
Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may hamper these drives.
Also, how many of these are in your cluster? If it’s a small number you might still benefit from chopping each into at least 2 separate OSDs.
And please send `ceph osd dump | grep pool`, having too few PGs wouldn’t do you any favors.
> Are you running a recent kernel? penultimate: 6.8.12-8-pve (VM, yes) Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then NVMe issues might be expected as old kernels had rudimentary NVMe support.
> Have you updated firmware on the NVMe devices? No. Kioxia appears to not release firmware updates publicly but your chassis brand (Dell, HP, SMCI, etc) might have an update.
e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
If there is an available update I would strongly suggest applying.
Thanks again,
best regards, Gio
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
EnekoLacunza
Director Técnico | Zuzendari teknikoa
Binovo IT Human Project
943 569 206 <tel:943 569 206>
elacunza@binovo.es <mailto:elacunza@binovo.es>
binovo.es <//binovo.es>
Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Giovanna, I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network. VM cpu is x86_64_v3 and host CPU Ryzen 1700. I gest almost the same IOPS with --iodepth=16 . I tried moving the VM to a Ryzen 5900X and results are somewhat better: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single. My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8 I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list. Cheers El 20/3/25 a las 12:29, Giovanna Ratini escribió:
Hello Eneko,
this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most.
agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16
Cheers,
Gio
Am 20.03.2025 um 10:23 schrieb Eneko Lacunza:
Hi Giovanna,
Can you post VM's full config?
Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks?
Cheers
El 19/3/25 a las 17:27, Giovanna Ratini escribió:
hello Eneko,
Yes I did. No significant changes. :-( Cheers,
Gio
Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>:
Hi Giovanna,
Have you tried increasing iothreads option for the VM?
Cheers
Hello Antony,
no, no QoS applied to Vms.
The Server has PCIe Gen 4
ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 13.04 pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 application cephfs read_balance_score 2.02 pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.42 pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.52
I think, this is the default config. 🙈
I will search for my chassies supermicro upgrade.
Thank you
Am 18.03.2025 um 17:57 schrieb Anthony D'Atri:
> Then I tested on the *Proxmox host*, and the results were > significantly better. My Proxmox prowess is limited, but from my experience with other virtualization platforms, I have to ask if there is any QoS throttling applied to VMs. With OpenStack or DO there is often IOPS and/or throughput throttling via libvirt to mitigate noisy neighbors.
> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio > --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G > --runtime=60 --group_reporting > > *IOPS*: *1.54M* > > # *Bandwidth*: *6032MiB/s (6325MB/s)* > # *Latency*: > > * *Avg*: *39.8µs* > * *99.9th percentile*: *71µs* > > # *CPU Usage*: *usr=22.60%, sys=77.13%* > # > > Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >> Which NVMe drive SKUs specifically? > # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 > # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 > # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 Kioxia CD6. If you were using client-class drives all manner of performance issues would be expected.
Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may hamper these drives.
Also, how many of these are in your cluster? If it’s a small number you might still benefit from chopping each into at least 2 separate OSDs.
And please send `ceph osd dump | grep pool`, having too few PGs wouldn’t do you any favors.
>> Are you running a recent kernel? > penultimate: 6.8.12-8-pve (VM, yes) Groovy. If you were running like a CentOS 6 or CentOS 7 kernel
El 18/3/25 a las 19:13, Giovanna Ratini escribió: then
NVMe issues might be expected as old kernels had rudimentary NVMe support.
>> Have you updated firmware on the NVMe devices? > No. Kioxia appears to not release firmware updates publicly but your chassis brand (Dell, HP, SMCI, etc) might have an update.
e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
If there is an available update I would strongly suggest
applying.
> Thanks again, > > best regards, > Gio > > _______________________________________________ > ceph-users mailing list --ceph-users@ceph.io > To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
EnekoLacunza
Director Técnico | Zuzendari teknikoa
Binovo IT Human Project
943 569 206 <tel:943 569 206>
elacunza@binovo.es <mailto:elacunza@binovo.es>
binovo.es <//binovo.es>
Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
I just ran that command on one of my VMs. Salient details: * Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback The results are very different: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49% On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió:
Hello Eneko,
this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most.
agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16
Cheers,
Gio
Am 20.03.2025 um 10:23 schrieb Eneko Lacunza:
Hi Giovanna,
Can you post VM's full config?
Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks?
Cheers
El 19/3/25 a las 17:27, Giovanna Ratini escribió:
hello Eneko,
Yes I did. No significant changes. :-( Cheers,
Gio
Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>:
Hi Giovanna,
Have you tried increasing iothreads option for the VM?
Cheers
Hello Antony,
no, no QoS applied to Vms.
The Server has PCIe Gen 4
ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 13.04 pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 application cephfs read_balance_score 2.02 pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.42 pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.52
I think, this is the default config. 🙈
I will search for my chassies supermicro upgrade.
Thank you
Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: >> Then I tested on the *Proxmox host*, and the results were >> significantly better. > My Proxmox prowess is limited, but from my experience with other > virtualization platforms, I have to ask if there is any QoS > throttling applied to VMs. With OpenStack or DO there is often IOPS > and/or throughput throttling via libvirt to mitigate noisy neighbors. > >> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio >> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G >> --runtime=60 --group_reporting >> >> *IOPS*: *1.54M* >> >> # *Bandwidth*: *6032MiB/s (6325MB/s)* >> # *Latency*: >> >> * *Avg*: *39.8µs* >> * *99.9th percentile*: *71µs* >> >> # *CPU Usage*: *usr=22.60%, sys=77.13%* >> # >> >> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >>> Which NVMe drive SKUs specifically? >> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 >> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 >> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 > Kioxia CD6. If you were using client-class drives all manner of > performance issues would be expected. > > Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may > hamper these drives. > > Also, how many of these are in your cluster? If it’s a small number > you might still benefit from chopping each into at least 2 separate > OSDs. > > And please send `ceph osd dump | grep pool`, having too few PGs > wouldn’t do you any favors. > > >>> Are you running a recent kernel? >> penultimate: 6.8.12-8-pve (VM, yes) > Groovy. If you were running like a CentOS 6 or CentOS 7 kernel
El 18/3/25 a las 19:13, Giovanna Ratini escribió: then
> NVMe issues might be expected as old kernels had rudimentary NVMe > support. > >>> Have you updated firmware on the NVMe devices? >> No. > Kioxia appears to not release firmware updates publicly but your > chassis brand (Dell, HP, SMCI, etc) might have an update. > e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
> > > If there is an available update I would strongly suggest applying.
> >> Thanks again, >> >> best regards, >> Gio >> >> _______________________________________________ >> ceph-users mailing list --ceph-users@ceph.io >> To unsubscribe send an email toceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
EnekoLacunza
Director Técnico | Zuzendari teknikoa
Binovo IT Human Project
943 569 206 <tel:943 569 206>
elacunza@binovo.es <mailto:elacunza@binovo.es>
binovo.es <//binovo.es>
Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Chris, you tested from a container? Or how do you configure a KRBD disk for a VM? El 20/3/25 a las 15:15, Chris Palmer escribió:
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió:
Hello Eneko,
this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most.
agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16
Cheers,
Gio
Am 20.03.2025 um 10:23 schrieb Eneko Lacunza:
Hi Giovanna,
Can you post VM's full config?
Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks?
Cheers
El 19/3/25 a las 17:27, Giovanna Ratini escribió:
hello Eneko,
Yes I did. No significant changes. :-( Cheers,
Gio
Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>:
Hi Giovanna,
Have you tried increasing iothreads option for the VM?
Cheers
El 18/3/25 a las 19:13, Giovanna Ratini escribió: > Hello Antony, > > no, no QoS applied to Vms. > > The Server has PCIe Gen 4 > > ceph osd dump | grep pool > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr > read_balance_score 13.04 > pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 > application cephfs read_balance_score 2.02 > pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 > pg_num_min 16 recovery_priority 5 application cephfs > read_balance_score 2.42 > pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 > lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 > application rbd read_balance_score 1.52 > > I think, this is the default config. 🙈 > > I will search for my chassies supermicro upgrade. > > Thank you > > > Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: >>> Then I tested on the *Proxmox host*, and the results were >>> significantly better. >> My Proxmox prowess is limited, but from my experience with other >> virtualization platforms, I have to ask if there is any QoS >> throttling applied to VMs. With OpenStack or DO there is often IOPS >> and/or throughput throttling via libvirt to mitigate noisy neighbors. >> >>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio >>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G >>> --runtime=60 --group_reporting >>> >>> *IOPS*: *1.54M* >>> >>> # *Bandwidth*: *6032MiB/s (6325MB/s)* >>> # *Latency*: >>> >>> * *Avg*: *39.8µs* >>> * *99.9th percentile*: *71µs* >>> >>> # *CPU Usage*: *usr=22.60%, sys=77.13%* >>> # >>> >>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >>>> Which NVMe drive SKUs specifically? >>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 >>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 >>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 >> Kioxia CD6. If you were using client-class drives all manner of >> performance issues would be expected. >> >> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may >> hamper these drives. >> >> Also, how many of these are in your cluster? If it’s a small number >> you might still benefit from chopping each into at least 2 separate >> OSDs. >> >> And please send `ceph osd dump | grep pool`, having too few PGs >> wouldn’t do you any favors. >> >> >>>> Are you running a recent kernel? >>> penultimate: 6.8.12-8-pve (VM, yes) >> Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then >> NVMe issues might be expected as old kernels had rudimentary NVMe >> support. >> >>>> Have you updated firmware on the NVMe devices? >>> No. >> Kioxia appears to not release firmware updates publicly but your >> chassis brand (Dell, HP, SMCI, etc) might have an update. >> e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
>> >> >> If there is an available update I would strongly suggest applying. > >> >>> Thanks again, >>> >>> best regards, >>> Gio >>> >>> _______________________________________________ >>> ceph-users mailing list --ceph-users@ceph.io >>> To unsubscribe send an email toceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
EnekoLacunza
Director Técnico | Zuzendari teknikoa
Binovo IT Human Project
943 569 206 <tel:943 569 206>
elacunza@binovo.es <mailto:elacunza@binovo.es>
binovo.es <//binovo.es>
Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
HI Eneko No containers. In the Promox console go to Datacenter\Storage, click on the storage you are using, then Edit. There is a tick box KRBD. With that set, any virtual disks created in that storage will use KRBD rather than librbd. So it applies to all VMs that use that storage. Chris On 20/03/2025 15:00, Eneko Lacunza wrote:
Chris, you tested from a container? Or how do you configure a KRBD disk for a VM?
El 20/3/25 a las 15:15, Chris Palmer escribió:
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió:
Hello Eneko,
this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most.
agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16
Cheers,
Gio
Am 20.03.2025 um 10:23 schrieb Eneko Lacunza:
Hi Giovanna,
Can you post VM's full config?
Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks?
Cheers
El 19/3/25 a las 17:27, Giovanna Ratini escribió:
hello Eneko,
Yes I did. No significant changes. :-( Cheers,
Gio
Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>:
> Hi Giovanna, > > Have you tried increasing iothreads option for the VM? > > Cheers > > El 18/3/25 a las 19:13, Giovanna Ratini escribió: > > Hello Antony, > > > > no, no QoS applied to Vms. > > > > The Server has PCIe Gen 4 > > > > ceph osd dump | grep pool > > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 > object_hash > > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 > flags > > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 > application mgr > > read_balance_score 13.04 > > pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 > > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > > last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 > > application cephfs read_balance_score 2.02 > > pool 3 'cephfs_metadata' replicated size 3 min_size 2 > crush_rule 0 > > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > > last_change 50 flags hashpspool stripe_width 0 > pg_autoscale_bias 4 > > pg_num_min 16 recovery_priority 5 application cephfs > > read_balance_score 2.42 > > pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 > object_hash > > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change > 16386 > > lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 > > application rbd read_balance_score 1.52 > > > > I think, this is the default config. 🙈 > > > > I will search for my chassies supermicro upgrade. > > > > Thank you > > > > > > Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: > >>> Then I tested on the *Proxmox host*, and the results were > >>> significantly better. > >> My Proxmox prowess is limited, but from my experience with other > >> virtualization platforms, I have to ask if there is any QoS > >> throttling applied to VMs. With OpenStack or DO there is > often IOPS > >> and/or throughput throttling via libvirt to mitigate noisy > neighbors. > >> > >>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio > >>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G > >>> --runtime=60 --group_reporting > >>> > >>> *IOPS*: *1.54M* > >>> > >>> # *Bandwidth*: *6032MiB/s (6325MB/s)* > >>> # *Latency*: > >>> > >>> * *Avg*: *39.8µs* > >>> * *99.9th percentile*: *71µs* > >>> > >>> # *CPU Usage*: *usr=22.60%, sys=77.13%* > >>> # > >>> > >>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: > >>>> Which NVMe drive SKUs specifically? > >>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 > >>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 > >>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 > >> Kioxia CD6. If you were using client-class drives all manner of > >> performance issues would be expected. > >> > >> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 > that may > >> hamper these drives. > >> > >> Also, how many of these are in your cluster? If it’s a small > number > >> you might still benefit from chopping each into at least 2 > separate > >> OSDs. > >> > >> And please send `ceph osd dump | grep pool`, having too few PGs > >> wouldn’t do you any favors. > >> > >> > >>>> Are you running a recent kernel? > >>> penultimate: 6.8.12-8-pve (VM, yes) > >> Groovy. If you were running like a CentOS 6 or CentOS 7 > kernel then > >> NVMe issues might be expected as old kernels had rudimentary > NVMe > >> support. > >> > >>>> Have you updated firmware on the NVMe devices? > >>> No. > >> Kioxia appears to not release firmware updates publicly but your > >> chassis brand (Dell, HP, SMCI, etc) might have an update. > >> > e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... > > >> > >> > >> If there is an available update I would strongly suggest > applying. > > > >> > >>> Thanks again, > >>> > >>> best regards, > >>> Gio > >>> > >>> _______________________________________________ > >>> ceph-users mailing list --ceph-users@ceph.io > >>> To unsubscribe send an email toceph-users-leave@ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > > Eneko Lacunza > Zuzendari teknikoa | Director técnico > Binovo IT Human Project > > Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es > Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun > > https://www.youtube.com/user/CANALBINOVO > https://www.linkedin.com/company/37269706/ > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
EnekoLacunza
Director Técnico | Zuzendari teknikoa
Binovo IT Human Project
943 569 206 <tel:943 569 206>
elacunza@binovo.es <mailto:elacunza@binovo.es>
binovo.es <//binovo.es>
Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Chris, I tried KRBD, even with a newly created disk and after shuting down and starting VM again, but no measurable difference. Our Ceph is 18.2.4, that may be a factor to consider, but 9k -> 273k?! Maybe Giovanna can test KRBD option and report back... :) Cheers El 20/3/25 a las 16:19, Chris Palmer escribió:
HI Eneko
No containers. In the Promox console go to Datacenter\Storage, click on the storage you are using, then Edit. There is a tick box KRBD. With that set, any virtual disks created in that storage will use KRBD rather than librbd. So it applies to all VMs that use that storage.
Chris
On 20/03/2025 15:00, Eneko Lacunza wrote:
Chris, you tested from a container? Or how do you configure a KRBD disk for a VM?
El 20/3/25 a las 15:15, Chris Palmer escribió:
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió:
Hello Eneko,
this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most.
agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16
Cheers,
Gio
Am 20.03.2025 um 10:23 schrieb Eneko Lacunza:
Hi Giovanna,
Can you post VM's full config?
Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks?
Cheers
El 19/3/25 a las 17:27, Giovanna Ratini escribió: > > hello Eneko, > > Yes I did. No significant changes. :-( > Cheers, > > Gio > > > Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza > <elacunza@binovo.es>: > >> Hi Giovanna, >> >> Have you tried increasing iothreads option for the VM? >> >> Cheers >> >> El 18/3/25 a las 19:13, Giovanna Ratini escribió: >> > Hello Antony, >> > >> > no, no QoS applied to Vms. >> > >> > The Server has PCIe Gen 4 >> > >> > ceph osd dump | grep pool >> > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 >> object_hash >> > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 >> flags >> > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 >> application mgr >> > read_balance_score 13.04 >> > pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 >> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >> > last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 >> > application cephfs read_balance_score 2.02 >> > pool 3 'cephfs_metadata' replicated size 3 min_size 2 >> crush_rule 0 >> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >> > last_change 50 flags hashpspool stripe_width 0 >> pg_autoscale_bias 4 >> > pg_num_min 16 recovery_priority 5 application cephfs >> > read_balance_score 2.42 >> > pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 >> object_hash >> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change >> 16386 >> > lfor 0/644/2603 flags hashpspool,selfmanaged_snaps >> stripe_width 0 >> > application rbd read_balance_score 1.52 >> > >> > I think, this is the default config. 🙈 >> > >> > I will search for my chassies supermicro upgrade. >> > >> > Thank you >> > >> > >> > Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: >> >>> Then I tested on the *Proxmox host*, and the results were >> >>> significantly better. >> >> My Proxmox prowess is limited, but from my experience with >> other >> >> virtualization platforms, I have to ask if there is any QoS >> >> throttling applied to VMs. With OpenStack or DO there is >> often IOPS >> >> and/or throughput throttling via libvirt to mitigate noisy >> neighbors. >> >> >> >>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio >> >>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G >> >>> --runtime=60 --group_reporting >> >>> >> >>> *IOPS*: *1.54M* >> >>> >> >>> # *Bandwidth*: *6032MiB/s (6325MB/s)* >> >>> # *Latency*: >> >>> >> >>> * *Avg*: *39.8µs* >> >>> * *99.9th percentile*: *71µs* >> >>> >> >>> # *CPU Usage*: *usr=22.60%, sys=77.13%* >> >>> # >> >>> >> >>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >> >>>> Which NVMe drive SKUs specifically? >> >>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: >> 6250A02QT5A8 >> >>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: >> 42R0A036T5A8 >> >>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: >> 6250A02UT5A8 >> >> Kioxia CD6. If you were using client-class drives all >> manner of >> >> performance issues would be expected. >> >> >> >> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 >> that may >> >> hamper these drives. >> >> >> >> Also, how many of these are in your cluster? If it’s a small >> number >> >> you might still benefit from chopping each into at least 2 >> separate >> >> OSDs. >> >> >> >> And please send `ceph osd dump | grep pool`, having too few PGs >> >> wouldn’t do you any favors. >> >> >> >> >> >>>> Are you running a recent kernel? >> >>> penultimate: 6.8.12-8-pve (VM, yes) >> >> Groovy. If you were running like a CentOS 6 or CentOS 7 >> kernel then >> >> NVMe issues might be expected as old kernels had rudimentary >> NVMe >> >> support. >> >> >> >>>> Have you updated firmware on the NVMe devices? >> >>> No. >> >> Kioxia appears to not release firmware updates publicly but >> your >> >> chassis brand (Dell, HP, SMCI, etc) might have an update. >> >> >> e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... >> >> >> >> >> >> >> If there is an available update I would strongly suggest >> applying. >> > >> >> >> >>> Thanks again, >> >>> >> >>> best regards, >> >>> Gio >> >>> >> >>> _______________________________________________ >> >>> ceph-users mailing list --ceph-users@ceph.io >> >>> To unsubscribe send an email toceph-users-leave@ceph.io >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@ceph.io >> > To unsubscribe send an email to ceph-users-leave@ceph.io >> >> Eneko Lacunza >> Zuzendari teknikoa | Director técnico >> Binovo IT Human Project >> >> Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es >> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun >> >> https://www.youtube.com/user/CANALBINOVO >> https://www.linkedin.com/company/37269706/ >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >> > > >
EnekoLacunza
Director Técnico | Zuzendari teknikoa
Binovo IT Human Project
943 569 206 <tel:943 569 206>
elacunza@binovo.es <mailto:elacunza@binovo.es>
binovo.es <//binovo.es>
Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
Hello, Yes, I will test KRBD. I will be on holiday next week, so I don’t want to make any changes before then. Could you wait until 29.3? This is a production environment, and restoring a backup would take time. Or do you think there’s no risk in making the change without concern? Thank you, best Regards, Gio Am 20.03.2025 um 16:57 schrieb Eneko Lacunza:
Hi Chris,
I tried KRBD, even with a newly created disk and after shuting down and starting VM again, but no measurable difference.
Our Ceph is 18.2.4, that may be a factor to consider, but 9k -> 273k?!
Maybe Giovanna can test KRBD option and report back... :)
Cheers
El 20/3/25 a las 16:19, Chris Palmer escribió:
HI Eneko
No containers. In the Promox console go to Datacenter\Storage, click on the storage you are using, then Edit. There is a tick box KRBD. With that set, any virtual disks created in that storage will use KRBD rather than librbd. So it applies to all VMs that use that storage.
Chris
On 20/03/2025 15:00, Eneko Lacunza wrote:
Chris, you tested from a container? Or how do you configure a KRBD disk for a VM?
El 20/3/25 a las 15:15, Chris Palmer escribió:
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió:
Hello Eneko,
this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most.
agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16
Cheers,
Gio
Am 20.03.2025 um 10:23 schrieb Eneko Lacunza: > Hi Giovanna, > > Can you post VM's full config? > > Also, can you test with IO thread enabled and SCSI virtio > single, and multiple disks? > > Cheers > > El 19/3/25 a las 17:27, Giovanna Ratini escribió: >> >> hello Eneko, >> >> Yes I did. No significant changes. :-( >> Cheers, >> >> Gio >> >> >> Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza >> <elacunza@binovo.es>: >> >>> Hi Giovanna, >>> >>> Have you tried increasing iothreads option for the VM? >>> >>> Cheers >>> >>> El 18/3/25 a las 19:13, Giovanna Ratini escribió: >>> > Hello Antony, >>> > >>> > no, no QoS applied to Vms. >>> > >>> > The Server has PCIe Gen 4 >>> > >>> > ceph osd dump | grep pool >>> > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 >>> object_hash >>> > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 >>> flags >>> > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 >>> application mgr >>> > read_balance_score 13.04 >>> > pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 >>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>> > last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 >>> > application cephfs read_balance_score 2.02 >>> > pool 3 'cephfs_metadata' replicated size 3 min_size 2 >>> crush_rule 0 >>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>> > last_change 50 flags hashpspool stripe_width 0 >>> pg_autoscale_bias 4 >>> > pg_num_min 16 recovery_priority 5 application cephfs >>> > read_balance_score 2.42 >>> > pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 >>> object_hash >>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on >>> last_change 16386 >>> > lfor 0/644/2603 flags hashpspool,selfmanaged_snaps >>> stripe_width 0 >>> > application rbd read_balance_score 1.52 >>> > >>> > I think, this is the default config. 🙈 >>> > >>> > I will search for my chassies supermicro upgrade. >>> > >>> > Thank you >>> > >>> > >>> > Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: >>> >>> Then I tested on the *Proxmox host*, and the results were >>> >>> significantly better. >>> >> My Proxmox prowess is limited, but from my experience with >>> other >>> >> virtualization platforms, I have to ask if there is any QoS >>> >> throttling applied to VMs. With OpenStack or DO there is >>> often IOPS >>> >> and/or throughput throttling via libvirt to mitigate noisy >>> neighbors. >>> >> >>> >>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio >>> >>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G >>> >>> --runtime=60 --group_reporting >>> >>> >>> >>> *IOPS*: *1.54M* >>> >>> >>> >>> # *Bandwidth*: *6032MiB/s (6325MB/s)* >>> >>> # *Latency*: >>> >>> >>> >>> * *Avg*: *39.8µs* >>> >>> * *99.9th percentile*: *71µs* >>> >>> >>> >>> # *CPU Usage*: *usr=22.60%, sys=77.13%* >>> >>> # >>> >>> >>> >>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >>> >>>> Which NVMe drive SKUs specifically? >>> >>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>> 6250A02QT5A8 >>> >>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>> 42R0A036T5A8 >>> >>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>> 6250A02UT5A8 >>> >> Kioxia CD6. If you were using client-class drives all >>> manner of >>> >> performance issues would be expected. >>> >> >>> >> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 >>> that may >>> >> hamper these drives. >>> >> >>> >> Also, how many of these are in your cluster? If it’s a >>> small number >>> >> you might still benefit from chopping each into at least 2 >>> separate >>> >> OSDs. >>> >> >>> >> And please send `ceph osd dump | grep pool`, having too few >>> PGs >>> >> wouldn’t do you any favors. >>> >> >>> >> >>> >>>> Are you running a recent kernel? >>> >>> penultimate: 6.8.12-8-pve (VM, yes) >>> >> Groovy. If you were running like a CentOS 6 or CentOS 7 >>> kernel then >>> >> NVMe issues might be expected as old kernels had >>> rudimentary NVMe >>> >> support. >>> >> >>> >>>> Have you updated firmware on the NVMe devices? >>> >>> No. >>> >> Kioxia appears to not release firmware updates publicly but >>> your >>> >> chassis brand (Dell, HP, SMCI, etc) might have an update. >>> >> >>> e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... >>> >>> >> >>> >> >>> >> If there is an available update I would strongly suggest >>> applying. >>> > >>> >> >>> >>> Thanks again, >>> >>> >>> >>> best regards, >>> >>> Gio >>> >>> >>> >>> _______________________________________________ >>> >>> ceph-users mailing list --ceph-users@ceph.io >>> >>> To unsubscribe send an email toceph-users-leave@ceph.io >>> > _______________________________________________ >>> > ceph-users mailing list -- ceph-users@ceph.io >>> > To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> Eneko Lacunza >>> Zuzendari teknikoa | Director técnico >>> Binovo IT Human Project >>> >>> Tel. +34 943 569 206 <tel:+34 943 569 206> | >>> https://www.binovo.es >>> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun >>> >>> https://www.youtube.com/user/CANALBINOVO >>> https://www.linkedin.com/company/37269706/ >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >> >> >> > > EnekoLacunza > > Director Técnico | Zuzendari teknikoa > > Binovo IT Human Project > > 943 569 206 <tel:943 569 206> > > elacunza@binovo.es <mailto:elacunza@binovo.es> > > binovo.es <//binovo.es> > > Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun > > > youtube <https://www.youtube.com/user/CANALBINOVO/> > linkedin <https://www.linkedin.com/company/37269706/> > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Digging into a different direction. I have question, are the drives connected a raid array card? And how are they presented? I don't recall where did I read something about a raid card presenting drives to the kernel as SCSI instead of NVMe and the queue depth being the issue. Cheers. -- Alvaro Soto Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people On Thu, Mar 20, 2025, 2:13 PM Giovanna Ratini < giovanna.ratini@uni-konstanz.de> wrote:
Hello,
Yes, I will test KRBD. I will be on holiday next week, so I don’t want to make any changes before then.
Could you wait until 29.3?
This is a production environment, and restoring a backup would take time. Or do you think there’s no risk in making the change without concern?
Thank you,
best Regards, Gio
Am 20.03.2025 um 16:57 schrieb Eneko Lacunza:
Hi Chris,
I tried KRBD, even with a newly created disk and after shuting down and starting VM again, but no measurable difference.
Our Ceph is 18.2.4, that may be a factor to consider, but 9k -> 273k?!
Maybe Giovanna can test KRBD option and report back... :)
Cheers
El 20/3/25 a las 16:19, Chris Palmer escribió:
HI Eneko
No containers. In the Promox console go to Datacenter\Storage, click on the storage you are using, then Edit. There is a tick box KRBD. With that set, any virtual disks created in that storage will use KRBD rather than librbd. So it applies to all VMs that use that storage.
Chris
On 20/03/2025 15:00, Eneko Lacunza wrote:
Chris, you tested from a container? Or how do you configure a KRBD disk for a VM?
El 20/3/25 a las 15:15, Chris Palmer escribió:
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió: > Hello Eneko, > > this is my configuration. The performance is similar across all > VMs. I am now checking GitLab, as that is where people are > complaining the most. > > agent: 1 > balloon: 65000 > bios: ovmf > boot: order=scsi0;net0 > cores: 10 > cpu: host > efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K > memory: 130000 > meta: creation-qemu=9.0.2,ctime=1734995123 > name: gitlab02 > net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 > numa: 0 > ostype: l26 > scsi0: >
cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1
> scsi1: > cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 > scsihw: virtio-scsi-single > smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac > sockets: 2 > vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16 > > Cheers, > > Gio > > Am 20.03.2025 um 10:23 schrieb Eneko Lacunza: >> Hi Giovanna, >> >> Can you post VM's full config? >> >> Also, can you test with IO thread enabled and SCSI virtio >> single, and multiple disks? >> >> Cheers >> >> El 19/3/25 a las 17:27, Giovanna Ratini escribió: >>> >>> hello Eneko, >>> >>> Yes I did. No significant changes. :-( >>> Cheers, >>> >>> Gio >>> >>> >>> Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza >>> <elacunza@binovo.es>: >>> >>>> Hi Giovanna, >>>> >>>> Have you tried increasing iothreads option for the VM? >>>> >>>> Cheers >>>> >>>> El 18/3/25 a las 19:13, Giovanna Ratini escribió: >>>> > Hello Antony, >>>> > >>>> > no, no QoS applied to Vms. >>>> > >>>> > The Server has PCIe Gen 4 >>>> > >>>> > ceph osd dump | grep pool >>>> > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 >>>> object_hash >>>> > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 >>>> flags >>>> > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 >>>> application mgr >>>> > read_balance_score 13.04 >>>> > pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 >>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>> > last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 >>>> > application cephfs read_balance_score 2.02 >>>> > pool 3 'cephfs_metadata' replicated size 3 min_size 2 >>>> crush_rule 0 >>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>> > last_change 50 flags hashpspool stripe_width 0 >>>> pg_autoscale_bias 4 >>>> > pg_num_min 16 recovery_priority 5 application cephfs >>>> > read_balance_score 2.42 >>>> > pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 >>>> object_hash >>>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on >>>> last_change 16386 >>>> > lfor 0/644/2603 flags hashpspool,selfmanaged_snaps >>>> stripe_width 0 >>>> > application rbd read_balance_score 1.52 >>>> > >>>> > I think, this is the default config. 🙈 >>>> > >>>> > I will search for my chassies supermicro upgrade. >>>> > >>>> > Thank you >>>> > >>>> > >>>> > Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: >>>> >>> Then I tested on the *Proxmox host*, and the results were >>>> >>> significantly better. >>>> >> My Proxmox prowess is limited, but from my experience with >>>> other >>>> >> virtualization platforms, I have to ask if there is any QoS >>>> >> throttling applied to VMs. With OpenStack or DO there is >>>> often IOPS >>>> >> and/or throughput throttling via libvirt to mitigate noisy >>>> neighbors. >>>> >> >>>> >>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio >>>> >>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G >>>> >>> --runtime=60 --group_reporting >>>> >>> >>>> >>> *IOPS*: *1.54M* >>>> >>> >>>> >>> # *Bandwidth*: *6032MiB/s (6325MB/s)* >>>> >>> # *Latency*: >>>> >>> >>>> >>> * *Avg*: *39.8µs* >>>> >>> * *99.9th percentile*: *71µs* >>>> >>> >>>> >>> # *CPU Usage*: *usr=22.60%, sys=77.13%* >>>> >>> # >>>> >>> >>>> >>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >>>> >>>> Which NVMe drive SKUs specifically? >>>> >>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>>> 6250A02QT5A8 >>>> >>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>>> 42R0A036T5A8 >>>> >>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB <https://www.google.com/maps/search/CD61LUL15T3*+%E2%80%93+15.36+TB+?entry=gmail&source=g>– SN: >>>> 6250A02UT5A8 >>>> >> Kioxia CD6. If you were using client-class drives all >>>> manner of >>>> >> performance issues would be expected. >>>> >> >>>> >> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 >>>> that may >>>> >> hamper these drives. >>>> >> >>>> >> Also, how many of these are in your cluster? If it’s a >>>> small number >>>> >> you might still benefit from chopping each into at least 2 >>>> separate >>>> >> OSDs. >>>> >> >>>> >> And please send `ceph osd dump | grep pool`, having too few >>>> PGs >>>> >> wouldn’t do you any favors. >>>> >> >>>> >> >>>> >>>> Are you running a recent kernel? >>>> >>> penultimate: 6.8.12-8-pve (VM, yes) >>>> >> Groovy. If you were running like a CentOS 6 or CentOS 7 >>>> kernel then >>>> >> NVMe issues might be expected as old kernels had >>>> rudimentary NVMe >>>> >> support. >>>> >> >>>> >>>> Have you updated firmware on the NVMe <https://www.google.com/maps/search/updated+firmware+on+the+NVMe?entry=gmail&source=g> devices? >>>> >>> No. >>>> >> Kioxia appears to not release firmware updates publicly but >>>> your >>>> >> chassis brand (Dell, HP, SMCI, etc) might have an update. >>>> >> >>>> e.g. https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... >>>> >>>> >> >>>> >> >>>> >> If there is an available update I would strongly suggest >>>> applying. >>>> > >>>> >> >>>> >>> Thanks again, >>>> >>> >>>> >>> best regards, >>>> >>> Gio >>>> >>> >>>> >>> _______________________________________________ >>>> >>> ceph-users mailing list --ceph-users@ceph.io >>>> >>> To unsubscribe send an email toceph-users-leave@ceph.io >>>> > _______________________________________________ >>>> > ceph-users mailing list -- ceph-users@ceph.io >>>> > To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> Eneko Lacunza >>>> Zuzendari teknikoa | Director técnico >>>> Binovo IT Human Project >>>> >>>> Tel. +34 943 569 206 <tel:+34 943 569 206> | >>>> https://www.binovo.es >>>> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun >>>> >>>> https://www.youtube.com/user/CANALBINOVO >>>> https://www.linkedin.com/company/37269706/ >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>> >>> >>> >> >> EnekoLacunza >> >> Director Técnico | Zuzendari teknikoa >> >> Binovo IT Human Project >> >> 943 569 206 <tel:943 569 206> >> >> elacunza@binovo.es <mailto:elacunza@binovo.es> >> >> binovo.es <//binovo.es> >> >> Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun >> >> >> youtube <https://www.youtube.com/user/CANALBINOVO/> >> linkedin <https://www.linkedin.com/company/37269706/> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Alvaro, They are /not/ behind a traditional hardware RAID controller. No RAID controller is present — I'm working with native NVMe SSDs on individual PCIe lanes. Cheers, Gio Am 21.03.2025 um 09:27 schrieb Alvaro Soto:
Digging into a different direction. I have question, are the drives connected a raid array card? And how are they presented?
I don't recall where did I read something about a raid card presenting drives to the kernel as SCSI instead of NVMe and the queue depth being the issue.
Cheers. --
Alvaro Soto
Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people
On Thu, Mar 20, 2025, 2:13 PM Giovanna Ratini < giovanna.ratini@uni-konstanz.de> wrote:
Hello,
Yes, I will test KRBD. I will be on holiday next week, so I don’t want to make any changes before then.
Could you wait until 29.3?
This is a production environment, and restoring a backup would take time. Or do you think there’s no risk in making the change without concern?
Thank you,
best Regards, Gio
Am 20.03.2025 um 16:57 schrieb Eneko Lacunza:
Hi Chris,
I tried KRBD, even with a newly created disk and after shuting down and starting VM again, but no measurable difference.
Our Ceph is 18.2.4, that may be a factor to consider, but 9k -> 273k?!
Maybe Giovanna can test KRBD option and report back... :)
Cheers
El 20/3/25 a las 16:19, Chris Palmer escribió:
HI Eneko
No containers. In the Promox console go to Datacenter\Storage, click on the storage you are using, then Edit. There is a tick box KRBD. With that set, any virtual disks created in that storage will use KRBD rather than librbd. So it applies to all VMs that use that storage.
Chris
On 20/03/2025 15:00, Eneko Lacunza wrote:
Chris, you tested from a container? Or how do you configure a KRBD disk for a VM?
El 20/3/25 a las 15:15, Chris Palmer escribió:
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote: > Hi Giovanna, > > I just tested one of my VMs: > # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k > --numjobs=4 --size=1G --runtime=60 --group_reporting > registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) > 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 > registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) > 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 > ... > fio-3.33 > Starting 4 processes > registry-read: Laying out IO file (1 file / 1024MiB) > registry-read: Laying out IO file (1 file / 1024MiB) > registry-read: Laying out IO file (1 file / 1024MiB) > registry-read: Laying out IO file (1 file / 1024MiB) > Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] > registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 > 12:57:26 2025 > read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) > slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 > clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 > lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 > clat percentiles (nsec): > | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], > 20.00th=[ 1096], > | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], > 60.00th=[ 1832], > | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], > 95.00th=[ 3024], > | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], > 99.95th=[61696], > | 99.99th=[90624] > bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, > stdev=269.75, samples=476 > iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, > samples=476 > lat (nsec) : 750=0.06%, 1000=14.94% > lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% > lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% > lat (msec) : 2=0.01% > cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 > IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, > 32=0.0%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, > 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, > 64=0.0%, >=64=0.0% > issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=1 > > Run status group 0 (all jobs): > READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s > (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec > > Results are worse than yours, but this is on a production (not > very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 > disks in your case) and 10G network. > > VM cpu is x86_64_v3 and host CPU Ryzen 1700. > > I gest almost the same IOPS with --iodepth=16 . > > I tried moving the VM to a Ryzen 5900X and results are somewhat > better: > > # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k > --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 > registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) > 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 > ... > fio-3.33 > Starting 4 processes > Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] > registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 > 13:18:23 2025 > read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) > slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 > clat (nsec): min=1390, max=42395k, avg=5147009.08, > stdev=475506.40 > lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 > clat percentiles (usec): > | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], > 20.00th=[ 4948], > | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], > 60.00th=[ 5145], > | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], > 95.00th=[ 5538], > | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], > 99.95th=[12125], > | 99.99th=[24249] > bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, > stdev=300.09, samples=476 > iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, > samples=476 > lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% > lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% > cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, > 32=0.0%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, > 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, > 64=0.0%, >=64=0.0% > issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=16 > > Run status group 0 (all jobs): > READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s > (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec > > I think we're limited by the IO thread. I suggest you try multiple > disks with SCSI Virtio single. > > My VM conf: > agent: 1 > boot: order=scsi0;ide2;net0 > cores: 2 > cpu: x86-64-v3 > ide2: none,media=cdrom > memory: 2048 > meta: creation-qemu=9.0.2,ctime=1739888364 > name: elacunza-btrfs-test > net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 > numa: 0 > ostype: l26 > scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G > scsihw: virtio-scsi-single > smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 > sockets: 1 > vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8 > > I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you > can get better suggestions in pve-user mailing list. > > Cheers > > El 20/3/25 a las 12:29, Giovanna Ratini escribió: >> Hello Eneko, >> >> this is my configuration. The performance is similar across all >> VMs. I am now checking GitLab, as that is where people are >> complaining the most. >> >> agent: 1 >> balloon: 65000 >> bios: ovmf >> boot: order=scsi0;net0 >> cores: 10 >> cpu: host >> efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K >> memory: 130000 >> meta: creation-qemu=9.0.2,ctime=1734995123 >> name: gitlab02 >> net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 >> numa: 0 >> ostype: l26 >> scsi0: >> cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 >> scsi1: >> cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 >> scsihw: virtio-scsi-single >> smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac >> sockets: 2 >> vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16 >> >> Cheers, >> >> Gio >> >> Am 20.03.2025 um 10:23 schrieb Eneko Lacunza: >>> Hi Giovanna, >>> >>> Can you post VM's full config? >>> >>> Also, can you test with IO thread enabled and SCSI virtio >>> single, and multiple disks? >>> >>> Cheers >>> >>> El 19/3/25 a las 17:27, Giovanna Ratini escribió: >>>> hello Eneko, >>>> >>>> Yes I did. No significant changes. :-( >>>> Cheers, >>>> >>>> Gio >>>> >>>> >>>> Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza >>>> <elacunza@binovo.es>: >>>> >>>>> Hi Giovanna, >>>>> >>>>> Have you tried increasing iothreads option for the VM? >>>>> >>>>> Cheers >>>>> >>>>> El 18/3/25 a las 19:13, Giovanna Ratini escribió: >>>>>> Hello Antony, >>>>>> >>>>>> no, no QoS applied to Vms. >>>>>> >>>>>> The Server has PCIe Gen 4 >>>>>> >>>>>> ceph osd dump | grep pool >>>>>> pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 >>>>> object_hash >>>>>> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 >>>>> flags >>>>>> hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 >>>>> application mgr >>>>>> read_balance_score 13.04 >>>>>> pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 >>>>>> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>>>> last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 >>>>>> application cephfs read_balance_score 2.02 >>>>>> pool 3 'cephfs_metadata' replicated size 3 min_size 2 >>>>> crush_rule 0 >>>>>> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>>>> last_change 50 flags hashpspool stripe_width 0 >>>>> pg_autoscale_bias 4 >>>>>> pg_num_min 16 recovery_priority 5 application cephfs >>>>>> read_balance_score 2.42 >>>>>> pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 >>>>> object_hash >>>>>> rjenkins pg_num 128 pgp_num 128 autoscale_mode on >>>>> last_change 16386 >>>>>> lfor 0/644/2603 flags hashpspool,selfmanaged_snaps >>>>> stripe_width 0 >>>>>> application rbd read_balance_score 1.52 >>>>>> >>>>>> I think, this is the default config. 🙈 >>>>>> >>>>>> I will search for my chassies supermicro upgrade. >>>>>> >>>>>> Thank you >>>>>> >>>>>> >>>>>> Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: >>>>>>>> Then I tested on the *Proxmox host*, and the results were >>>>>>>> significantly better. >>>>>>> My Proxmox prowess is limited, but from my experience with >>>>> other >>>>>>> virtualization platforms, I have to ask if there is any QoS >>>>>>> throttling applied to VMs. With OpenStack or DO there is >>>>> often IOPS >>>>>>> and/or throughput throttling via libvirt to mitigate noisy >>>>> neighbors. >>>>>>>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio >>>>>>>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G >>>>>>>> --runtime=60 --group_reporting >>>>>>>> >>>>>>>> *IOPS*: *1.54M* >>>>>>>> >>>>>>>> # *Bandwidth*: *6032MiB/s (6325MB/s)* >>>>>>>> # *Latency*: >>>>>>>> >>>>>>>> * *Avg*: *39.8µs* >>>>>>>> * *99.9th percentile*: *71µs* >>>>>>>> >>>>>>>> # *CPU Usage*: *usr=22.60%, sys=77.13%* >>>>>>>> # >>>>>>>> >>>>>>>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >>>>>>>>> Which NVMe drive SKUs specifically? >>>>>>>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>>>> 6250A02QT5A8 >>>>>>>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>>>> 42R0A036T5A8 >>>>>>>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB <https://www.google.com/maps/search/CD61LUL15T3*+%E2%80%93+15.36+TB+?entry=gmail&source=g>– SN: >>>>> 6250A02UT5A8 >>>>>>> Kioxia CD6. If you were using client-class drives all >>>>> manner of >>>>>>> performance issues would be expected. >>>>>>> >>>>>>> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 >>>>> that may >>>>>>> hamper these drives. >>>>>>> >>>>>>> Also, how many of these are in your cluster? If it’s a >>>>> small number >>>>>>> you might still benefit from chopping each into at least 2 >>>>> separate >>>>>>> OSDs. >>>>>>> >>>>>>> And please send `ceph osd dump | grep pool`, having too few >>>>> PGs >>>>>>> wouldn’t do you any favors. >>>>>>> >>>>>>> >>>>>>>>> Are you running a recent kernel? >>>>>>>> penultimate: 6.8.12-8-pve (VM, yes) >>>>>>> Groovy. If you were running like a CentOS 6 or CentOS 7 >>>>> kernel then >>>>>>> NVMe issues might be expected as old kernels had >>>>> rudimentary NVMe >>>>>>> support. >>>>>>> >>>>>>>>> Have you updated firmware on the NVMe <https://www.google.com/maps/search/updated+firmware+on+the+NVMe?entry=gmail&source=g> devices? >>>>>>>> No. >>>>>>> Kioxia appears to not release firmware updates publicly but >>>>> your >>>>>>> chassis brand (Dell, HP, SMCI, etc) might have an update. >>>>>>> >>>>> e.g. https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... >>>>>>> >>>>>>> If there is an available update I would strongly suggest >>>>> applying. >>>>>>>> Thanks again, >>>>>>>> >>>>>>>> best regards, >>>>>>>> Gio >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list --ceph-users@ceph.io >>>>>>>> To unsubscribe send an emailtoceph-users-leave@ceph.io >>>>>> _______________________________________________ >>>>>> ceph-users mailing list --ceph-users@ceph.io >>>>>> To unsubscribe send an email toceph-users-leave@ceph.io >>>>> Eneko Lacunza >>>>> Zuzendari teknikoa | Director técnico >>>>> Binovo IT Human Project >>>>> >>>>> Tel. +34 943 569 206<tel:+34 943 569 206> | >>>>> https://www.binovo.es >>>>> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun >>>>> >>>>> https://www.youtube.com/user/CANALBINOVO >>>>> https://www.linkedin.com/company/37269706/ >>>>> _______________________________________________ >>>>> ceph-users mailing list --ceph-users@ceph.io >>>>> To unsubscribe send an email toceph-users-leave@ceph.io >>>>> >>>> >>>> >>> EnekoLacunza >>> >>> Director Técnico | Zuzendari teknikoa >>> >>> Binovo IT Human Project >>> >>> 943 569 206<tel:943 569 206> >>> >>> elacunza@binovo.es <mailto:elacunza@binovo.es> >>> >>> binovo.es <//binovo.es> >>> >>> Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun >>> >>> >>> youtube<https://www.youtube.com/user/CANALBINOVO/> >>> linkedin<https://www.linkedin.com/company/37269706/> >>> _______________________________________________ >>> ceph-users mailing list --ceph-users@ceph.io >>> To unsubscribe send an email toceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list --ceph-users@ceph.io >> To unsubscribe send an email toceph-users-leave@ceph.io > Eneko Lacunza > Zuzendari teknikoa | Director técnico > Binovo IT Human Project > > Tel. +34 943 569 206 |https://www.binovo.es > Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun > > https://www.youtube.com/user/CANALBINOVO > https://www.linkedin.com/company/37269706/ > _______________________________________________ > ceph-users mailing list --ceph-users@ceph.io > To unsubscribe send an email toceph-users-leave@ceph.io Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Hello Eneko, I switched to KRDB, and I’m seeing slightly better performance now. For Switching: https://forum.proxmox.com/threads/how-to-safely-enable-krbd-in-a-5-node-prod... NVMe performance remains disappointing, though... They went from 35MB/s to 45MB/s. I’m planning to apply the change that Anthony recommended: setting mon_target_pg_per_osd to 250 and configuring 2 osds_per_device. This will take a bit of time. ceph config set global mon_target_pg_per_osd 250 ceph config set global osds_per_device 2 To split the drives into 2 OSDs each, I’ll need to update the ceph orch ls --export OSD service spec, then zap an existing OSD, allow it to be rebuilt as two, and repeat the process for the remaining ones. We'll see if this change helps. I’ll write the results here once it's done. Cheers, Gio root@gitlab:~# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=16 --size=1G --runtime=60 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=91.7MiB/s][r=23.5k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=1): err= 0: pid=2547: Fri Apr 11 21:02:31 2025 read: IOPS=2756, BW=10.8MiB/s (11.3MB/s)(646MiB/60001msec) slat (usec): min=50, max=8619, avg=360.14, stdev=217.84 clat (usec): min=2, max=17259, avg=5441.99, stdev=1633.01 lat (usec): min=108, max=17721, avg=5802.13, stdev=1728.71 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2507], 10.00th=[ 2966], 20.00th=[ 3818], | 30.00th=[ 4621], 40.00th=[ 5342], 50.00th=[ 5932], 60.00th=[ 6259], | 70.00th=[ 6456], 80.00th=[ 6718], 90.00th=[ 6980], 95.00th=[ 7308], | 99.00th=[ 9241], 99.50th=[10290], 99.90th=[13173], 99.95th=[13698], | 99.99th=[16450] bw ( KiB/s): min= 8456, max=22296, per=24.64%, avg=10937.08, stdev=3222.24, samples=119 iops : min= 2114, max= 5574, avg=2734.27, stdev=805.56, samples=119 lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=1.33%, 4=20.70%, 10=77.32%, 20=0.65% cpu : usr=0.78%, sys=6.75%, ctx=165432, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=165408,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 registry-read: (groupid=0, jobs=1): err= 0: pid=2548: Fri Apr 11 21:02:31 2025 read: IOPS=2807, BW=11.0MiB/s (11.5MB/s)(658MiB/60001msec) slat (usec): min=50, max=8950, avg=353.61, stdev=213.68 clat (usec): min=2, max=17110, avg=5344.32, stdev=1642.90 lat (usec): min=93, max=17575, avg=5697.93, stdev=1740.41 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2409], 10.00th=[ 2868], 20.00th=[ 3687], | 30.00th=[ 4490], 40.00th=[ 5276], 50.00th=[ 5866], 60.00th=[ 6194], | 70.00th=[ 6390], 80.00th=[ 6587], 90.00th=[ 6915], 95.00th=[ 7242], | 99.00th=[ 8979], 99.50th=[10159], 99.90th=[13042], 99.95th=[13829], | 99.99th=[15926] bw ( KiB/s): min= 8536, max=23624, per=25.10%, avg=11138.08, stdev=3441.69, samples=119 iops : min= 2134, max= 5906, avg=2784.52, stdev=860.42, samples=119 lat (usec) : 4=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=1.80%, 4=22.21%, 10=75.40%, 20=0.58% cpu : usr=0.98%, sys=6.72%, ctx=168450, majf=0, minf=25 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=168432,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 registry-read: (groupid=0, jobs=1): err= 0: pid=2549: Fri Apr 11 21:02:31 2025 read: IOPS=2773, BW=10.8MiB/s (11.4MB/s)(650MiB/60001msec) slat (usec): min=46, max=8246, avg=357.89, stdev=213.33 clat (usec): min=2, max=19652, avg=5408.19, stdev=1641.03 lat (usec): min=411, max=20124, avg=5766.08, stdev=1738.36 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2474], 10.00th=[ 2933], 20.00th=[ 3752], | 30.00th=[ 4555], 40.00th=[ 5342], 50.00th=[ 5932], 60.00th=[ 6259], | 70.00th=[ 6456], 80.00th=[ 6652], 90.00th=[ 6980], 95.00th=[ 7242], | 99.00th=[ 9110], 99.50th=[10421], 99.90th=[12911], 99.95th=[14353], | 99.99th=[16909] bw ( KiB/s): min= 8432, max=22520, per=24.79%, avg=11004.77, stdev=3330.83, samples=119 iops : min= 2108, max= 5630, avg=2751.19, stdev=832.71, samples=119 lat (usec) : 4=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=1.40%, 4=21.56%, 10=76.44%, 20=0.60% cpu : usr=0.99%, sys=6.58%, ctx=166457, majf=0, minf=25 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=166442,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 registry-read: (groupid=0, jobs=1): err= 0: pid=2550: Fri Apr 11 21:02:31 2025 read: IOPS=2757, BW=10.8MiB/s (11.3MB/s)(646MiB/60001msec) slat (usec): min=49, max=7497, avg=360.11, stdev=212.22 clat (usec): min=2, max=19699, avg=5441.22, stdev=1616.73 lat (usec): min=390, max=20175, avg=5801.33, stdev=1712.21 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2540], 10.00th=[ 2999], 20.00th=[ 3818], | 30.00th=[ 4621], 40.00th=[ 5407], 50.00th=[ 5932], 60.00th=[ 6259], | 70.00th=[ 6456], 80.00th=[ 6652], 90.00th=[ 6980], 95.00th=[ 7308], | 99.00th=[ 8979], 99.50th=[10159], 99.90th=[13042], 99.95th=[13829], | 99.99th=[16057] bw ( KiB/s): min= 8512, max=23152, per=24.65%, avg=10941.71, stdev=3229.43, samples=119 iops : min= 2128, max= 5788, avg=2735.43, stdev=807.36, samples=119 lat (usec) : 4=0.01%, 500=0.01%, 1000=0.01% lat (msec) : 2=1.39%, 4=20.78%, 10=77.28%, 20=0.54% cpu : usr=0.80%, sys=6.75%, ctx=165463, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=165432,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=43.3MiB/s (45.4MB/s), 10.8MiB/s-11.0MiB/s (11.3MB/s-11.5MB/s), io=2600MiB (2727MB), run=60001-60001msec Disk stats (read/write): dm-0: ios=663651/273, merge=0/0, ticks=221100/28, in_queue=221128, util=99.88%, aggrios=666145/189, aggrmerge=202/85, aggrticks=206340/50, aggrin_queue=206423, aggrutil=66.45% sda: ios=666145/189, merge=202/85, ticks=206340/50, in_queue=206423, util=66.45% Am 20.03.2025 um 16:57 schrieb Eneko Lacunza:
Hi Chris,
I tried KRBD, even with a newly created disk and after shuting down and starting VM again, but no measurable difference.
Our Ceph is 18.2.4, that may be a factor to consider, but 9k -> 273k?!
Maybe Giovanna can test KRBD option and report back... :)
Cheers
El 20/3/25 a las 16:19, Chris Palmer escribió:
HI Eneko
No containers. In the Promox console go to Datacenter\Storage, click on the storage you are using, then Edit. There is a tick box KRBD. With that set, any virtual disks created in that storage will use KRBD rather than librbd. So it applies to all VMs that use that storage.
Chris
On 20/03/2025 15:00, Eneko Lacunza wrote:
Chris, you tested from a container? Or how do you configure a KRBD disk for a VM?
El 20/3/25 a las 15:15, Chris Palmer escribió:
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió:
Hello Eneko,
this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most.
agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16
Cheers,
Gio
Am 20.03.2025 um 10:23 schrieb Eneko Lacunza: > Hi Giovanna, > > Can you post VM's full config? > > Also, can you test with IO thread enabled and SCSI virtio > single, and multiple disks? > > Cheers > > El 19/3/25 a las 17:27, Giovanna Ratini escribió: >> >> hello Eneko, >> >> Yes I did. No significant changes. :-( >> Cheers, >> >> Gio >> >> >> Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza >> <elacunza@binovo.es>: >> >>> Hi Giovanna, >>> >>> Have you tried increasing iothreads option for the VM? >>> >>> Cheers >>> >>> El 18/3/25 a las 19:13, Giovanna Ratini escribió: >>> > Hello Antony, >>> > >>> > no, no QoS applied to Vms. >>> > >>> > The Server has PCIe Gen 4 >>> > >>> > ceph osd dump | grep pool >>> > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 >>> object_hash >>> > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 >>> flags >>> > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 >>> application mgr >>> > read_balance_score 13.04 >>> > pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 >>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>> > last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 >>> > application cephfs read_balance_score 2.02 >>> > pool 3 'cephfs_metadata' replicated size 3 min_size 2 >>> crush_rule 0 >>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>> > last_change 50 flags hashpspool stripe_width 0 >>> pg_autoscale_bias 4 >>> > pg_num_min 16 recovery_priority 5 application cephfs >>> > read_balance_score 2.42 >>> > pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 >>> object_hash >>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on >>> last_change 16386 >>> > lfor 0/644/2603 flags hashpspool,selfmanaged_snaps >>> stripe_width 0 >>> > application rbd read_balance_score 1.52 >>> > >>> > I think, this is the default config. 🙈 >>> > >>> > I will search for my chassies supermicro upgrade. >>> > >>> > Thank you >>> > >>> > >>> > Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: >>> >>> Then I tested on the *Proxmox host*, and the results were >>> >>> significantly better. >>> >> My Proxmox prowess is limited, but from my experience with >>> other >>> >> virtualization platforms, I have to ask if there is any QoS >>> >> throttling applied to VMs. With OpenStack or DO there is >>> often IOPS >>> >> and/or throughput throttling via libvirt to mitigate noisy >>> neighbors. >>> >> >>> >>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio >>> >>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G >>> >>> --runtime=60 --group_reporting >>> >>> >>> >>> *IOPS*: *1.54M* >>> >>> >>> >>> # *Bandwidth*: *6032MiB/s (6325MB/s)* >>> >>> # *Latency*: >>> >>> >>> >>> * *Avg*: *39.8µs* >>> >>> * *99.9th percentile*: *71µs* >>> >>> >>> >>> # *CPU Usage*: *usr=22.60%, sys=77.13%* >>> >>> # >>> >>> >>> >>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >>> >>>> Which NVMe drive SKUs specifically? >>> >>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>> 6250A02QT5A8 >>> >>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>> 42R0A036T5A8 >>> >>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: >>> 6250A02UT5A8 >>> >> Kioxia CD6. If you were using client-class drives all >>> manner of >>> >> performance issues would be expected. >>> >> >>> >> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 >>> that may >>> >> hamper these drives. >>> >> >>> >> Also, how many of these are in your cluster? If it’s a >>> small number >>> >> you might still benefit from chopping each into at least 2 >>> separate >>> >> OSDs. >>> >> >>> >> And please send `ceph osd dump | grep pool`, having too few >>> PGs >>> >> wouldn’t do you any favors. >>> >> >>> >> >>> >>>> Are you running a recent kernel? >>> >>> penultimate: 6.8.12-8-pve (VM, yes) >>> >> Groovy. If you were running like a CentOS 6 or CentOS 7 >>> kernel then >>> >> NVMe issues might be expected as old kernels had >>> rudimentary NVMe >>> >> support. >>> >> >>> >>>> Have you updated firmware on the NVMe devices? >>> >>> No. >>> >> Kioxia appears to not release firmware updates publicly but >>> your >>> >> chassis brand (Dell, HP, SMCI, etc) might have an update. >>> >> >>> e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... >>> >>> >> >>> >> >>> >> If there is an available update I would strongly suggest >>> applying. >>> > >>> >> >>> >>> Thanks again, >>> >>> >>> >>> best regards, >>> >>> Gio >>> >>> >>> >>> _______________________________________________ >>> >>> ceph-users mailing list --ceph-users@ceph.io >>> >>> To unsubscribe send an email toceph-users-leave@ceph.io >>> > _______________________________________________ >>> > ceph-users mailing list -- ceph-users@ceph.io >>> > To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> Eneko Lacunza >>> Zuzendari teknikoa | Director técnico >>> Binovo IT Human Project >>> >>> Tel. +34 943 569 206 <tel:+34 943 569 206> | >>> https://www.binovo.es >>> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun >>> >>> https://www.youtube.com/user/CANALBINOVO >>> https://www.linkedin.com/company/37269706/ >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >> >> >> > > EnekoLacunza > > Director Técnico | Zuzendari teknikoa > > Binovo IT Human Project > > 943 569 206 <tel:943 569 206> > > elacunza@binovo.es <mailto:elacunza@binovo.es> > > binovo.es <//binovo.es> > > Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun > > > youtube <https://www.youtube.com/user/CANALBINOVO/> > linkedin <https://www.linkedin.com/company/37269706/> > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Please do let me know if that strategy works out. When you change an osd_spec, out of an abundance of caution it won’t be retroactively applied to existing OSDs, which can be exploited for migrations.
On Apr 11, 2025, at 3:29 PM, Giovanna Ratini <giovanna.ratini@uni-konstanz.de> wrote:
Hello Eneko,
I switched to KRDB, and I’m seeing slightly better performance now.
For Switching: https://forum.proxmox.com/threads/how-to-safely-enable-krbd-in-a-5-node-prod...
NVMe performance remains disappointing, though... They went from 35MB/s to 45MB/s.
I’m planning to apply the change that Anthony recommended: setting mon_target_pg_per_osd to 250 and configuring 2 osds_per_device. This will take a bit of time. ceph config set global mon_target_pg_per_osd 250 ceph config set global osds_per_device 2
To split the drives into 2 OSDs each, I’ll need to update the ceph orch ls --export OSD service spec,
then zap an existing OSD, allow it to be rebuilt as two, and repeat the process for the remaining ones.
We'll see if this change helps. I’ll write the results here once it's done.
Cheers,
Gio
root@gitlab:~# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=16 --size=1G --runtime=60
registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=91.7MiB/s][r=23.5k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=1): err= 0: pid=2547: Fri Apr 11 21:02:31 2025 read: IOPS=2756, BW=10.8MiB/s (11.3MB/s)(646MiB/60001msec) slat (usec): min=50, max=8619, avg=360.14, stdev=217.84 clat (usec): min=2, max=17259, avg=5441.99, stdev=1633.01 lat (usec): min=108, max=17721, avg=5802.13, stdev=1728.71 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2507], 10.00th=[ 2966], 20.00th=[ 3818], | 30.00th=[ 4621], 40.00th=[ 5342], 50.00th=[ 5932], 60.00th=[ 6259], | 70.00th=[ 6456], 80.00th=[ 6718], 90.00th=[ 6980], 95.00th=[ 7308], | 99.00th=[ 9241], 99.50th=[10290], 99.90th=[13173], 99.95th=[13698], | 99.99th=[16450] bw ( KiB/s): min= 8456, max=22296, per=24.64%, avg=10937.08, stdev=3222.24, samples=119 iops : min= 2114, max= 5574, avg=2734.27, stdev=805.56, samples=119 lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=1.33%, 4=20.70%, 10=77.32%, 20=0.65% cpu : usr=0.78%, sys=6.75%, ctx=165432, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165408,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 registry-read: (groupid=0, jobs=1): err= 0: pid=2548: Fri Apr 11 21:02:31 2025 read: IOPS=2807, BW=11.0MiB/s (11.5MB/s)(658MiB/60001msec) slat (usec): min=50, max=8950, avg=353.61, stdev=213.68 clat (usec): min=2, max=17110, avg=5344.32, stdev=1642.90 lat (usec): min=93, max=17575, avg=5697.93, stdev=1740.41 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2409], 10.00th=[ 2868], 20.00th=[ 3687], | 30.00th=[ 4490], 40.00th=[ 5276], 50.00th=[ 5866], 60.00th=[ 6194], | 70.00th=[ 6390], 80.00th=[ 6587], 90.00th=[ 6915], 95.00th=[ 7242], | 99.00th=[ 8979], 99.50th=[10159], 99.90th=[13042], 99.95th=[13829], | 99.99th=[15926] bw ( KiB/s): min= 8536, max=23624, per=25.10%, avg=11138.08, stdev=3441.69, samples=119 iops : min= 2134, max= 5906, avg=2784.52, stdev=860.42, samples=119 lat (usec) : 4=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=1.80%, 4=22.21%, 10=75.40%, 20=0.58% cpu : usr=0.98%, sys=6.72%, ctx=168450, majf=0, minf=25 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=168432,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 registry-read: (groupid=0, jobs=1): err= 0: pid=2549: Fri Apr 11 21:02:31 2025 read: IOPS=2773, BW=10.8MiB/s (11.4MB/s)(650MiB/60001msec) slat (usec): min=46, max=8246, avg=357.89, stdev=213.33 clat (usec): min=2, max=19652, avg=5408.19, stdev=1641.03 lat (usec): min=411, max=20124, avg=5766.08, stdev=1738.36 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2474], 10.00th=[ 2933], 20.00th=[ 3752], | 30.00th=[ 4555], 40.00th=[ 5342], 50.00th=[ 5932], 60.00th=[ 6259], | 70.00th=[ 6456], 80.00th=[ 6652], 90.00th=[ 6980], 95.00th=[ 7242], | 99.00th=[ 9110], 99.50th=[10421], 99.90th=[12911], 99.95th=[14353], | 99.99th=[16909] bw ( KiB/s): min= 8432, max=22520, per=24.79%, avg=11004.77, stdev=3330.83, samples=119 iops : min= 2108, max= 5630, avg=2751.19, stdev=832.71, samples=119 lat (usec) : 4=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=1.40%, 4=21.56%, 10=76.44%, 20=0.60% cpu : usr=0.99%, sys=6.58%, ctx=166457, majf=0, minf=25 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=166442,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 registry-read: (groupid=0, jobs=1): err= 0: pid=2550: Fri Apr 11 21:02:31 2025 read: IOPS=2757, BW=10.8MiB/s (11.3MB/s)(646MiB/60001msec) slat (usec): min=49, max=7497, avg=360.11, stdev=212.22 clat (usec): min=2, max=19699, avg=5441.22, stdev=1616.73 lat (usec): min=390, max=20175, avg=5801.33, stdev=1712.21 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2540], 10.00th=[ 2999], 20.00th=[ 3818], | 30.00th=[ 4621], 40.00th=[ 5407], 50.00th=[ 5932], 60.00th=[ 6259], | 70.00th=[ 6456], 80.00th=[ 6652], 90.00th=[ 6980], 95.00th=[ 7308], | 99.00th=[ 8979], 99.50th=[10159], 99.90th=[13042], 99.95th=[13829], | 99.99th=[16057] bw ( KiB/s): min= 8512, max=23152, per=24.65%, avg=10941.71, stdev=3229.43, samples=119 iops : min= 2128, max= 5788, avg=2735.43, stdev=807.36, samples=119 lat (usec) : 4=0.01%, 500=0.01%, 1000=0.01% lat (msec) : 2=1.39%, 4=20.78%, 10=77.28%, 20=0.54% cpu : usr=0.80%, sys=6.75%, ctx=165463, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165432,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=43.3MiB/s (45.4MB/s), 10.8MiB/s-11.0MiB/s (11.3MB/s-11.5MB/s), io=2600MiB (2727MB), run=60001-60001msec
Disk stats (read/write): dm-0: ios=663651/273, merge=0/0, ticks=221100/28, in_queue=221128, util=99.88%, aggrios=666145/189, aggrmerge=202/85, aggrticks=206340/50, aggrin_queue=206423, aggrutil=66.45% sda: ios=666145/189, merge=202/85, ticks=206340/50, in_queue=206423, util=66.45%
Am 20.03.2025 um 16:57 schrieb Eneko Lacunza:
Hi Chris,
I tried KRBD, even with a newly created disk and after shuting down and starting VM again, but no measurable difference.
Our Ceph is 18.2.4, that may be a factor to consider, but 9k -> 273k?!
Maybe Giovanna can test KRBD option and report back... :)
Cheers
El 20/3/25 a las 16:19, Chris Palmer escribió:
HI Eneko
No containers. In the Promox console go to Datacenter\Storage, click on the storage you are using, then Edit. There is a tick box KRBD. With that set, any virtual disks created in that storage will use KRBD rather than librbd. So it applies to all VMs that use that storage.
Chris
On 20/03/2025 15:00, Eneko Lacunza wrote:
Chris, you tested from a container? Or how do you configure a KRBD disk for a VM?
El 20/3/25 a las 15:15, Chris Palmer escribió:
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió: > Hello Eneko, > > this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most. > > agent: 1 > balloon: 65000 > bios: ovmf > boot: order=scsi0;net0 > cores: 10 > cpu: host > efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K > memory: 130000 > meta: creation-qemu=9.0.2,ctime=1734995123 > name: gitlab02 > net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 > numa: 0 > ostype: l26 > scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 > scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 > scsihw: virtio-scsi-single > smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac > sockets: 2 > vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16 > > Cheers, > > Gio > > Am 20.03.2025 um 10:23 schrieb Eneko Lacunza: >> Hi Giovanna, >> >> Can you post VM's full config? >> >> Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks? >> >> Cheers >> >> El 19/3/25 a las 17:27, Giovanna Ratini escribió: >>> >>> hello Eneko, >>> >>> Yes I did. No significant changes. :-( >>> Cheers, >>> >>> Gio >>> >>> >>> Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>: >>> >>>> Hi Giovanna, >>>> >>>> Have you tried increasing iothreads option for the VM? >>>> >>>> Cheers >>>> >>>> El 18/3/25 a las 19:13, Giovanna Ratini escribió: >>>> > Hello Antony, >>>> > >>>> > no, no QoS applied to Vms. >>>> > >>>> > The Server has PCIe Gen 4 >>>> > >>>> > ceph osd dump | grep pool >>>> > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash >>>> > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags >>>> > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr >>>> > read_balance_score 13.04 >>>> > pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 >>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>> > last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 >>>> > application cephfs read_balance_score 2.02 >>>> > pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 >>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>> > last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 >>>> > pg_num_min 16 recovery_priority 5 application cephfs >>>> > read_balance_score 2.42 >>>> > pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash >>>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 >>>> > lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 >>>> > application rbd read_balance_score 1.52 >>>> > >>>> > I think, this is the default config. 🙈 >>>> > >>>> > I will search for my chassies supermicro upgrade. >>>> > >>>> > Thank you >>>> > >>>> > >>>> > Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: >>>> >>> Then I tested on the *Proxmox host*, and the results were >>>> >>> significantly better. >>>> >> My Proxmox prowess is limited, but from my experience with other >>>> >> virtualization platforms, I have to ask if there is any QoS >>>> >> throttling applied to VMs. With OpenStack or DO there is often IOPS >>>> >> and/or throughput throttling via libvirt to mitigate noisy neighbors. >>>> >> >>>> >>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio >>>> >>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G >>>> >>> --runtime=60 --group_reporting >>>> >>> >>>> >>> *IOPS*: *1.54M* >>>> >>> >>>> >>> # *Bandwidth*: *6032MiB/s (6325MB/s)* >>>> >>> # *Latency*: >>>> >>> >>>> >>> * *Avg*: *39.8µs* >>>> >>> * *99.9th percentile*: *71µs* >>>> >>> >>>> >>> # *CPU Usage*: *usr=22.60%, sys=77.13%* >>>> >>> # >>>> >>> >>>> >>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >>>> >>>> Which NVMe drive SKUs specifically? >>>> >>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 >>>> >>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 >>>> >>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 >>>> >> Kioxia CD6. If you were using client-class drives all manner of >>>> >> performance issues would be expected. >>>> >> >>>> >> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may >>>> >> hamper these drives. >>>> >> >>>> >> Also, how many of these are in your cluster? If it’s a small number >>>> >> you might still benefit from chopping each into at least 2 separate >>>> >> OSDs. >>>> >> >>>> >> And please send `ceph osd dump | grep pool`, having too few PGs >>>> >> wouldn’t do you any favors. >>>> >> >>>> >> >>>> >>>> Are you running a recent kernel? >>>> >>> penultimate: 6.8.12-8-pve (VM, yes) >>>> >> Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then >>>> >> NVMe issues might be expected as old kernels had rudimentary NVMe >>>> >> support. >>>> >> >>>> >>>> Have you updated firmware on the NVMe devices? >>>> >>> No. >>>> >> Kioxia appears to not release firmware updates publicly but your >>>> >> chassis brand (Dell, HP, SMCI, etc) might have an update. >>>> >> e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... >>>> >> >>>> >> >>>> >> If there is an available update I would strongly suggest applying. >>>> > >>>> >> >>>> >>> Thanks again, >>>> >>> >>>> >>> best regards, >>>> >>> Gio >>>> >>> >>>> >>> _______________________________________________ >>>> >>> ceph-users mailing list --ceph-users@ceph.io >>>> >>> To unsubscribe send an email toceph-users-leave@ceph.io >>>> > _______________________________________________ >>>> > ceph-users mailing list -- ceph-users@ceph.io >>>> > To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> Eneko Lacunza >>>> Zuzendari teknikoa | Director técnico >>>> Binovo IT Human Project >>>> >>>> Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es >>>> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun >>>> >>>> https://www.youtube.com/user/CANALBINOVO >>>> https://www.linkedin.com/company/37269706/ >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>> >>> >>> >> >> EnekoLacunza >> >> Director Técnico | Zuzendari teknikoa >> >> Binovo IT Human Project >> >> 943 569 206 <tel:943 569 206> >> >> elacunza@binovo.es <mailto:elacunza@binovo.es> >> >> binovo.es <//binovo.es> >> >> Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun >> >> >> youtube <https://www.youtube.com/user/CANALBINOVO/> >> linkedin <https://www.linkedin.com/company/37269706/> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, these are very impressive results! On HDD even!! Here are results on my cluster: | | no cache | writeback | unsafe | | ---- | -------- | --------- | ------- | | RBD | 40MB/s | 40MB/s | ? | | KRBD | 40MB/s | 245MB/s | 245MB/s | cluster: 8 proxmox nodes, 6 of them hosting ceph OSDs, 17 NVMe OSDs, dual 25G connections, running ceph 18.2.4, pve 8.2.7. VM with 4 vcpu, 8GB ram, using krbd, cache writeback, direct ext4 on virtio-scsi single controller, iothread=1 40MB/s is about 10k IOPS, 10ms typical latency 245MB/s is about 55k IOPS, 1ms typical latency It also depends on the cache in the VM: freshly after a reboot, the disk using krbd, writeback was at 56MB/s the first time, then 300MB/s when re-running. It performed better than prior results given in the table above, where I run all tests in sequence. I suspect because no cache eviction had to be done. Best, =============== KRBD_WRITEBACK ============ registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=204MiB/s][r=52.3k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=1083: Fri Mar 21 18:03:09 2025 read: IOPS=13.6k, BW=53.2MiB/s (55.8MB/s)(3195MiB/60001msec) slat (usec): min=27, max=24503, avg=291.01, stdev=270.71 clat (usec): min=2, max=30294, avg=4402.54, stdev=2199.54 lat (usec): min=52, max=30621, avg=4693.55, stdev=2333.21 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 914], 10.00th=[ 1254], 20.00th=[ 1926], | 30.00th=[ 2868], 40.00th=[ 3949], 50.00th=[ 4883], 60.00th=[ 5538], | 70.00th=[ 5932], 80.00th=[ 6259], 90.00th=[ 6652], 95.00th=[ 7242], | 99.00th=[ 9503], 99.50th=[10552], 99.90th=[13304], 99.95th=[14746], | 99.99th=[19006] bw ( KiB/s): min=30440, max=209488, per=97.68%, avg=53257.61, stdev=8053.45, samples=476 iops : min= 7610, max=52372, avg=13314.40, stdev=2013.36, samples=476 lat (usec) : 4=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.20% lat (usec) : 1000=5.87% lat (msec) : 2=14.87%, 4=19.56%, 10=58.78%, 20=0.71%, 50=0.01% cpu : usr=1.05%, sys=7.49%, ctx=817981, majf=0, minf=103 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=817858,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=53.2MiB/s (55.8MB/s), 53.2MiB/s-53.2MiB/s (55.8MB/s-55.8MB/s), io=3195MiB (3350MB), run=60001-60001msec Disk stats (read/write): sdb: ios=811605/4, merge=0/1, ticks=226346/3, in_queue=226350, util=100.00% ======== RUN 2 ========= registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=277MiB/s][r=70.9k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=1094: Fri Mar 21 18:03:23 2025 read: IOPS=72.9k, BW=285MiB/s (299MB/s)(4096MiB/14384msec) slat (usec): min=20, max=3941, avg=51.95, stdev=24.49 clat (usec): min=2, max=5271, avg=810.08, stdev=117.76 lat (usec): min=45, max=5334, avg=862.03, stdev=122.81 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 709], 10.00th=[ 725], 20.00th=[ 750], | 30.00th=[ 766], 40.00th=[ 783], 50.00th=[ 791], 60.00th=[ 807], | 70.00th=[ 816], 80.00th=[ 840], 90.00th=[ 881], 95.00th=[ 955], | 99.00th=[ 1336], 99.50th=[ 1483], 99.90th=[ 1860], 99.95th=[ 2147], | 99.99th=[ 2900] bw ( KiB/s): min=196264, max=315960, per=100.00%, avg=293362.00, stdev=6237.12, samples=112 iops : min=49066, max=78990, avg=73340.50, stdev=1559.28, samples=112 lat (usec) : 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=19.55%, 1000=76.53% lat (msec) : 2=3.85%, 4=0.06%, 10=0.01% cpu : usr=4.22%, sys=36.79%, ctx=1048623, majf=0, minf=110 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=285MiB/s (299MB/s), 285MiB/s-285MiB/s (299MB/s-299MB/s), io=4096MiB (4295MB), run=14384-14384msec Disk stats (read/write): sdb: ios=1036222/0, merge=0/0, ticks=41377/0, in_queue=41378, util=96.85% # free -h total used free shared buff/cache available Mem: 7.7Gi 387Mi 3.3Gi 628Ki 4.3Gi 7.4Gi Swap: 0B 0B 0B =============== RBD_NOCACHE ============ registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=38.6MiB/s][r=9890 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=1822: Fri Mar 21 17:40:55 2025 read: IOPS=10.2k, BW=39.9MiB/s (41.9MB/s)(2396MiB/60001msec) slat (usec): min=146, max=10638, avg=388.14, stdev=174.69 clat (usec): min=2, max=21535, avg=5858.97, stdev=832.41 lat (usec): min=362, max=21954, avg=6247.12, stdev=868.24 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4948], 10.00th=[ 5080], 20.00th=[ 5276], | 30.00th=[ 5473], 40.00th=[ 5604], 50.00th=[ 5735], 60.00th=[ 5866], | 70.00th=[ 6063], 80.00th=[ 6259], 90.00th=[ 6587], 95.00th=[ 7111], | 99.00th=[ 9241], 99.50th=[10159], 99.90th=[12911], 99.95th=[14091], | 99.99th=[16319] bw ( KiB/s): min=27920, max=45992, per=100.00%, avg=40919.80, stdev=703.00, samples=476 iops : min= 6980, max=11498, avg=10229.95, stdev=175.75, samples=476 lat (usec) : 4=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.42%, 20=0.57%, 50=0.01% cpu : usr=0.74%, sys=5.96%, ctx=613573, majf=0, minf=105 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=613414,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=39.9MiB/s (41.9MB/s), 39.9MiB/s-39.9MiB/s (41.9MB/s-41.9MB/s), io=2396MiB (2513MB), run=60001-60001msec Disk stats (read/write): sdc: ios=612284/0, merge=0/0, ticks=229266/0, in_queue=229266, util=99.70% Le 20/03/2025 à 15:15, Chris Palmer a écrit :
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió:
Hello Eneko,
this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most.
agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1
scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1
scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16
Cheers,
Gio
Am 20.03.2025 um 10:23 schrieb Eneko Lacunza:
Hi Giovanna,
Can you post VM's full config?
Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks?
Cheers
El 19/3/25 a las 17:27, Giovanna Ratini escribió:
hello Eneko,
Yes I did. No significant changes. :-( Cheers,
Gio
Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>:
Hi Giovanna,
Have you tried increasing iothreads option for the VM?
Cheers
El 18/3/25 a las 19:13, Giovanna Ratini escribió: > Hello Antony, > > no, no QoS applied to Vms. > > The Server has PCIe Gen 4 > > ceph osd dump | grep pool > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr > read_balance_score 13.04 > pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 > application cephfs read_balance_score 2.02 > pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > last_change 50 flags hashpspool stripe_width 0 pg_autoscale_bias 4 > pg_num_min 16 recovery_priority 5 application cephfs > read_balance_score 2.42 > pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 16386 > lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 > application rbd read_balance_score 1.52 > > I think, this is the default config. 🙈 > > I will search for my chassies supermicro upgrade. > > Thank you > > > Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: >>> Then I tested on the *Proxmox host*, and the results were >>> significantly better. >> My Proxmox prowess is limited, but from my experience with other >> virtualization platforms, I have to ask if there is any QoS >> throttling applied to VMs. With OpenStack or DO there is often IOPS >> and/or throughput throttling via libvirt to mitigate noisy neighbors. >> >>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio >>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G >>> --runtime=60 --group_reporting >>> >>> *IOPS*: *1.54M* >>> >>> # *Bandwidth*: *6032MiB/s (6325MB/s)* >>> # *Latency*: >>> >>> * *Avg*: *39.8µs* >>> * *99.9th percentile*: *71µs* >>> >>> # *CPU Usage*: *usr=22.60%, sys=77.13%* >>> # >>> >>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: >>>> Which NVMe drive SKUs specifically? >>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 >>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 >>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 >> Kioxia CD6. If you were using client-class drives all manner of >> performance issues would be expected. >> >> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may >> hamper these drives. >> >> Also, how many of these are in your cluster? If it’s a small number >> you might still benefit from chopping each into at least 2 separate >> OSDs. >> >> And please send `ceph osd dump | grep pool`, having too few PGs >> wouldn’t do you any favors. >> >> >>>> Are you running a recent kernel? >>> penultimate: 6.8.12-8-pve (VM, yes) >> Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then >> NVMe issues might be expected as old kernels had rudimentary NVMe >> support. >> >>>> Have you updated firmware on the NVMe devices? >>> No. >> Kioxia appears to not release firmware updates publicly but your >> chassis brand (Dell, HP, SMCI, etc) might have an update. >> e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5...
>> >> >> If there is an available update I would strongly suggest applying. > >> >>> Thanks again, >>> >>> best regards, >>> Gio >>> >>> _______________________________________________ >>> ceph-users mailing list --ceph-users@ceph.io >>> To unsubscribe send an email toceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es/ Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
EnekoLacunza
Director Técnico | Zuzendari teknikoa
Binovo IT Human Project
943 569 206 <tel:943 569 206>
elacunza@binovo.es <mailto:elacunza@binovo.es>
binovo.es <//binovo.es>
Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es/ Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 2025-03-20 15:15, schrieb Chris Palmer: Hi,
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
"cache=writeback" measures basically the memory bandwidth of the kvm-host not the Ceph performance. This gets even worse with --size=1G instead of more realistic 10G. Please remove the writeback cache and increase the filesize to at least 10G in order to measure Ceph rbd performance Last but not least in order to make the benchmark independent of previous runs you shall please drop the caches. Overview Results for blocksize=4K ================================= NVMe with cache=none (10 GB Fibre) | read: IOPS = 12.9k, BW = 50.3 MiB/s NVMe with cache=writeback (10 GB Fibre)| read: IOPS = 58.3k, BW = 228.0 MiB/s HDD with cache=none (10 GB Fibre) | read: IOPS = 0.34k, BW = 1.3 MiB/s HDD with cache=writeback (10 GB Fibre)| read: IOPS = 53.8k, BW = 210.0 MiB/s Detailed reports ================ NVME with cache=none # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=39.0MiB/s][r=9981 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=443550: Tue Mar 25 13:25:10 2025 read: IOPS=12.9k, BW=50.3MiB/s (52.7MB/s)(3017MiB/60001msec) slat (usec): min=2, max=13134, avg=307.13, stdev=154.43 clat (usec): min=2, max=28147, avg=4662.15, stdev=1048.17 lat (usec): min=384, max=28432, avg=4969.28, stdev=1106.45 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3261], 10.00th=[ 3589], 20.00th=[ 4080], | 30.00th=[ 4293], 40.00th=[ 4424], 50.00th=[ 4555], 60.00th=[ 4621], | 70.00th=[ 4817], 80.00th=[ 5080], 90.00th=[ 5932], 95.00th=[ 6325], | 99.00th=[ 8225], 99.50th=[ 9634], 99.90th=[12911], 99.95th=[14353], | 99.99th=[17957] bw ( KiB/s): min=26816, max=58136, per=100.00%, avg=51622.66, stdev=1587.61, samples=476 iops : min= 6704, max=14534, avg=12905.66, stdev=396.90, samples=476 lat (usec) : 4=0.01%, 10=0.01%, 500=0.01%, 1000=0.01% lat (msec) : 2=0.03%, 4=18.10%, 10=81.47%, 20=0.40%, 50=0.01% cpu : usr=1.52%, sys=8.43%, ctx=726528, majf=0, minf=109 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=772457,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=50.3MiB/s (52.7MB/s), 50.3MiB/s-50.3MiB/s (52.7MB/s-52.7MB/s), io=3017MiB (3164MB), run=60001-60001msec Disk stats (read/write): sdc: ios=725474/0, sectors=5803792/0, merge=0/0, ticks=223185/0, in_queue=223185, util=99.89% NVMe with cache=writeback # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 3 (f=3): [r(3),_(1)][65.9%][r=31.9MiB/s][r=8165 IOPS][eta 00m:31s] registry-read: (groupid=0, jobs=4): err= 0: pid=442595: Tue Mar 25 13:19:11 2025 read: IOPS=58.3k, BW=228MiB/s (239MB/s)(13.3GiB/60001msec) slat (usec): min=2, max=5185, avg=58.95, stdev=131.23 clat (nsec): min=1440, max=15294k, avg=908789.46, stdev=1764896.35 lat (usec): min=3, max=15367, avg=967.74, stdev=1880.61 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 58], 10.00th=[ 67], 20.00th=[ 117], | 30.00th=[ 133], 40.00th=[ 165], 50.00th=[ 196], 60.00th=[ 239], | 70.00th=[ 347], 80.00th=[ 873], 90.00th=[ 4146], 95.00th=[ 6063], | 99.00th=[ 6652], 99.50th=[ 6980], 99.90th=[ 8717], 99.95th=[ 9503], | 99.99th=[11076] bw ( KiB/s): min=203632, max=602000, per=100.00%, avg=387963.51, stdev=30229.81, samples=420 iops : min=50908, max=150500, avg=96990.60, stdev=7557.47, samples=420 lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=15.15%, 250=46.78%, 500=10.58%, 750=5.17%, 1000=6.62% lat (msec) : 2=3.90%, 4=1.46%, 10=10.29%, 20=0.03% cpu : usr=2.96%, sys=15.72%, ctx=1029435, majf=0, minf=108 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=3499461,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=13.3GiB (14.3GB), run=60001-60001msec Disk stats (read/write): sdb: ios=1028680/0, sectors=8229440/0, merge=0/0, ticks=180767/0, in_queue=180766, util=99.60% HDD with cache=none # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=1033KiB/s][r=258 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=442579: Tue Mar 25 13:15:51 2025 read: IOPS=335, BW=1340KiB/s (1373kB/s)(78.6MiB/60011msec) slat (usec): min=160, max=361240, avg=11930.20, stdev=15819.52 clat (usec): min=4, max=957176, avg=178747.09, stdev=89162.82 lat (msec): min=5, max=977, avg=190.68, stdev=93.50 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 101], 10.00th=[ 109], 20.00th=[ 121], | 30.00th=[ 130], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 163], | 70.00th=[ 180], 80.00th=[ 218], 90.00th=[ 300], 95.00th=[ 372], | 99.00th=[ 514], 99.50th=[ 558], 99.90th=[ 726], 99.95th=[ 802], | 99.99th=[ 885] bw ( KiB/s): min= 248, max= 2160, per=99.89%, avg=1339.50, stdev=117.47, samples=476 iops : min= 62, max= 540, avg=334.87, stdev=29.37, samples=476 lat (usec) : 10=0.02% lat (msec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=4.87%, 250=79.56% lat (msec) : 500=14.36%, 750=1.08%, 1000=0.08% cpu : usr=0.06%, sys=0.27%, ctx=20110, majf=0, minf=110 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=20110,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1340KiB/s (1373kB/s), 1340KiB/s-1340KiB/s (1373kB/s-1373kB/s), io=78.6MiB (82.4MB), run=60011-60011msec Disk stats (read/write): sdd: ios=20096/0, sectors=160768/0, merge=0/0, ticks=238953/0, in_queue=238952, util=99.89% HDD with cache=writeback # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=210MiB/s][r=53.7k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=943730: Thu Mar 20 14:51:33 2025 read: IOPS=53.8k, BW=210MiB/s (220MB/s)(12.3GiB/60001msec) slat (usec): min=26, max=4995, avg=71.31, stdev=21.64 clat (usec): min=3, max=8707, avg=1116.26, stdev=141.55 lat (usec): min=79, max=8769, avg=1187.56, stdev=148.56 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 979], 10.00th=[ 1004], 20.00th=[ 1029], | 30.00th=[ 1045], 40.00th=[ 1074], 50.00th=[ 1090], 60.00th=[ 1106], | 70.00th=[ 1139], 80.00th=[ 1188], 90.00th=[ 1254], 95.00th=[ 1336], | 99.00th=[ 1582], 99.50th=[ 1811], 99.90th=[ 2474], 99.95th=[ 2802], | 99.99th=[ 3982] bw ( KiB/s): min=167800, max=230352, per=100.00%, avg=215502.18, stdev=2114.67, samples=476 iops : min=41950, max=57588, avg=53875.55, stdev=528.67, samples=476 lat (usec) : 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=10.02% lat (msec) : 2=89.68%, 4=0.28%, 10=0.01% cpu : usr=4.83%, sys=37.00%, ctx=3232089, majf=0, minf=101 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=3230027,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=210MiB/s (220MB/s), 210MiB/s-210MiB/s (220MB/s-220MB/s), io=12.3GiB (13.2GB), run=60001-60001msec Disk stats (read/write): sdd: ios=3224017/2, sectors=25792136/3, merge=0/0, ticks=168114/14, in_queue=168141, util=99.03% This was hdd (3/2 replication). -- --martin konold ppa. Martin Konold -- Martin Konold - Prokurist, CTO KONSEC GmbH - make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany
I completely agree that the test I did is not suitable for testing ceph performance. I merely did the same command as the OP and obtained very different results. I suspect the performance difference is much more due to things like network, OS config, memory constraints, etc. But that needs a rigorous set of tests to be done on the infrastructure before looking at ceph itself. For example, we found that many 10g NICs don't handle full-sized jumbo frames (9000) without a performance hit. We got a huge performance improvement by finding the exact optimal MTU (5139 on a cluster I am looking at - at 5140 it falls off a cliff). Just one example. Chris On 25/03/2025 13:35, Konold, Martin wrote:
Am 2025-03-20 15:15, schrieb Chris Palmer:
Hi,
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
"cache=writeback" measures basically the memory bandwidth of the kvm-host not the Ceph performance. This gets even worse with --size=1G instead of more realistic 10G.
Please remove the writeback cache and increase the filesize to at least 10G in order to measure Ceph rbd performance
Last but not least in order to make the benchmark independent of previous runs you shall please drop the caches.
Overview Results for blocksize=4K =================================
NVMe with cache=none (10 GB Fibre) | read: IOPS = 12.9k, BW = 50.3 MiB/s NVMe with cache=writeback (10 GB Fibre)| read: IOPS = 58.3k, BW = 228.0 MiB/s HDD with cache=none (10 GB Fibre) | read: IOPS = 0.34k, BW = 1.3 MiB/s HDD with cache=writeback (10 GB Fibre)| read: IOPS = 53.8k, BW = 210.0 MiB/s
Detailed reports ================ NVME with cache=none
# echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=39.0MiB/s][r=9981 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=443550: Tue Mar 25 13:25:10 2025 read: IOPS=12.9k, BW=50.3MiB/s (52.7MB/s)(3017MiB/60001msec) slat (usec): min=2, max=13134, avg=307.13, stdev=154.43 clat (usec): min=2, max=28147, avg=4662.15, stdev=1048.17 lat (usec): min=384, max=28432, avg=4969.28, stdev=1106.45 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3261], 10.00th=[ 3589], 20.00th=[ 4080], | 30.00th=[ 4293], 40.00th=[ 4424], 50.00th=[ 4555], 60.00th=[ 4621], | 70.00th=[ 4817], 80.00th=[ 5080], 90.00th=[ 5932], 95.00th=[ 6325], | 99.00th=[ 8225], 99.50th=[ 9634], 99.90th=[12911], 99.95th=[14353], | 99.99th=[17957] bw ( KiB/s): min=26816, max=58136, per=100.00%, avg=51622.66, stdev=1587.61, samples=476 iops : min= 6704, max=14534, avg=12905.66, stdev=396.90, samples=476 lat (usec) : 4=0.01%, 10=0.01%, 500=0.01%, 1000=0.01% lat (msec) : 2=0.03%, 4=18.10%, 10=81.47%, 20=0.40%, 50=0.01% cpu : usr=1.52%, sys=8.43%, ctx=726528, majf=0, minf=109 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=772457,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=50.3MiB/s (52.7MB/s), 50.3MiB/s-50.3MiB/s (52.7MB/s-52.7MB/s), io=3017MiB (3164MB), run=60001-60001msec
Disk stats (read/write): sdc: ios=725474/0, sectors=5803792/0, merge=0/0, ticks=223185/0, in_queue=223185, util=99.89%
NVMe with cache=writeback # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 3 (f=3): [r(3),_(1)][65.9%][r=31.9MiB/s][r=8165 IOPS][eta 00m:31s] registry-read: (groupid=0, jobs=4): err= 0: pid=442595: Tue Mar 25 13:19:11 2025 read: IOPS=58.3k, BW=228MiB/s (239MB/s)(13.3GiB/60001msec) slat (usec): min=2, max=5185, avg=58.95, stdev=131.23 clat (nsec): min=1440, max=15294k, avg=908789.46, stdev=1764896.35 lat (usec): min=3, max=15367, avg=967.74, stdev=1880.61 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 58], 10.00th=[ 67], 20.00th=[ 117], | 30.00th=[ 133], 40.00th=[ 165], 50.00th=[ 196], 60.00th=[ 239], | 70.00th=[ 347], 80.00th=[ 873], 90.00th=[ 4146], 95.00th=[ 6063], | 99.00th=[ 6652], 99.50th=[ 6980], 99.90th=[ 8717], 99.95th=[ 9503], | 99.99th=[11076] bw ( KiB/s): min=203632, max=602000, per=100.00%, avg=387963.51, stdev=30229.81, samples=420 iops : min=50908, max=150500, avg=96990.60, stdev=7557.47, samples=420 lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=15.15%, 250=46.78%, 500=10.58%, 750=5.17%, 1000=6.62% lat (msec) : 2=3.90%, 4=1.46%, 10=10.29%, 20=0.03% cpu : usr=2.96%, sys=15.72%, ctx=1029435, majf=0, minf=108 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=3499461,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=13.3GiB (14.3GB), run=60001-60001msec
Disk stats (read/write): sdb: ios=1028680/0, sectors=8229440/0, merge=0/0, ticks=180767/0, in_queue=180766, util=99.60%
HDD with cache=none # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=1033KiB/s][r=258 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=442579: Tue Mar 25 13:15:51 2025 read: IOPS=335, BW=1340KiB/s (1373kB/s)(78.6MiB/60011msec) slat (usec): min=160, max=361240, avg=11930.20, stdev=15819.52 clat (usec): min=4, max=957176, avg=178747.09, stdev=89162.82 lat (msec): min=5, max=977, avg=190.68, stdev=93.50 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 101], 10.00th=[ 109], 20.00th=[ 121], | 30.00th=[ 130], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 163], | 70.00th=[ 180], 80.00th=[ 218], 90.00th=[ 300], 95.00th=[ 372], | 99.00th=[ 514], 99.50th=[ 558], 99.90th=[ 726], 99.95th=[ 802], | 99.99th=[ 885] bw ( KiB/s): min= 248, max= 2160, per=99.89%, avg=1339.50, stdev=117.47, samples=476 iops : min= 62, max= 540, avg=334.87, stdev=29.37, samples=476 lat (usec) : 10=0.02% lat (msec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=4.87%, 250=79.56% lat (msec) : 500=14.36%, 750=1.08%, 1000=0.08% cpu : usr=0.06%, sys=0.27%, ctx=20110, majf=0, minf=110 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=20110,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1340KiB/s (1373kB/s), 1340KiB/s-1340KiB/s (1373kB/s-1373kB/s), io=78.6MiB (82.4MB), run=60011-60011msec
Disk stats (read/write): sdd: ios=20096/0, sectors=160768/0, merge=0/0, ticks=238953/0, in_queue=238952, util=99.89%
HDD with cache=writeback # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=210MiB/s][r=53.7k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=943730: Thu Mar 20 14:51:33 2025 read: IOPS=53.8k, BW=210MiB/s (220MB/s)(12.3GiB/60001msec) slat (usec): min=26, max=4995, avg=71.31, stdev=21.64 clat (usec): min=3, max=8707, avg=1116.26, stdev=141.55 lat (usec): min=79, max=8769, avg=1187.56, stdev=148.56 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 979], 10.00th=[ 1004], 20.00th=[ 1029], | 30.00th=[ 1045], 40.00th=[ 1074], 50.00th=[ 1090], 60.00th=[ 1106], | 70.00th=[ 1139], 80.00th=[ 1188], 90.00th=[ 1254], 95.00th=[ 1336], | 99.00th=[ 1582], 99.50th=[ 1811], 99.90th=[ 2474], 99.95th=[ 2802], | 99.99th=[ 3982] bw ( KiB/s): min=167800, max=230352, per=100.00%, avg=215502.18, stdev=2114.67, samples=476 iops : min=41950, max=57588, avg=53875.55, stdev=528.67, samples=476 lat (usec) : 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=10.02% lat (msec) : 2=89.68%, 4=0.28%, 10=0.01% cpu : usr=4.83%, sys=37.00%, ctx=3232089, majf=0, minf=101 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=3230027,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=210MiB/s (220MB/s), 210MiB/s-210MiB/s (220MB/s-220MB/s), io=12.3GiB (13.2GB), run=60001-60001msec
Disk stats (read/write): sdd: ios=3224017/2, sectors=25792136/3, merge=0/0, ticks=168114/14, in_queue=168141, util=99.03%
This was hdd (3/2 replication).
Sounds weird to me. Don't you have some element in the network that is just limited to 5140 and above it, it starts to fix fragmentation or so. I can remember asking the data center to enable 9000 and they never did and also were experimenting with some software defined network. I will bet that peer to peer you should have no issues when testing mtu 9000.
For example, we found that many 10g NICs don't handle full-sized jumbo frames (9000) without a performance hit. We got a huge performance improvement by finding the exact optimal MTU (5139 on a cluster I am looking at - at 5140 it falls off a cliff). Just one example.
No, that was peer-to-peer, controlled testing. The results were different with different NIC chipsets, even on the same machines through the same switch. And even without a switch. I have to say some of these were cheaper NICs. With better ones there are less problems. But you don't know until you test. In our bigger production clusters everything is dual-100g without any problems at 9000. On 25/03/2025 15:20, Marc wrote:
Sounds weird to me. Don't you have some element in the network that is just limited to 5140 and above it, it starts to fix fragmentation or so. I can remember asking the data center to enable 9000 and they never did and also were experimenting with some software defined network. I will bet that peer to peer you should have no issues when testing mtu 9000.
For example, we found that many 10g NICs don't handle full-sized jumbo frames (9000) without a performance hit. We got a huge performance improvement by finding the exact optimal MTU (5139 on a cluster I am looking at - at 5140 it falls off a cliff). Just one example.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (13)
-
Alex Gorbachev
-
Alexander Schreiber
-
Alvaro Soto
-
Anthony D'Atri
-
Anthony D'Atri
-
Chris Palmer
-
darren＠soothill.com
-
Eneko Lacunza
-
Eric Le Lay
-
Giovanna Ratini
-
Konold, Martin
-
Marc
-
Martin Konold