HI Eneko No containers. In the Promox console go to Datacenter\Storage, click on the storage you are using, then Edit. There is a tick box KRBD. With that set, any virtual disks created in that storage will use KRBD rather than librbd. So it applies to all VMs that use that storage. Chris On 20/03/2025 15:00, Eneko Lacunza wrote:
Chris, you tested from a container? Or how do you configure a KRBD disk for a VM?
El 20/3/25 a las 15:15, Chris Palmer escribió:
I just ran that command on one of my VMs. Salient details:
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
The results are very different:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.37 Starting 4 processes Jobs: 4 (f=4): [r(4)][-.-%][r=1080MiB/s][r=277k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=13355: Thu Mar 20 13:57:05 2025 read: IOPS=273k, BW=1068MiB/s (1120MB/s)(4096MiB/3835msec) slat (usec): min=7, max=3802, avg=13.77, stdev= 6.41 clat (nsec): min=599, max=4395.1k, avg=215298.68, stdev=38131.71 lat (usec): min=11, max=4408, avg=229.07, stdev=40.01 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 204], | 30.00th=[ 206], 40.00th=[ 208], 50.00th=[ 210], 60.00th=[ 212], | 70.00th=[ 215], 80.00th=[ 217], 90.00th=[ 227], 95.00th=[ 243], | 99.00th=[ 367], 99.50th=[ 420], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 963] bw ( MiB/s): min= 920, max= 1118, per=100.00%, avg=1068.04, stdev=16.81, samples=28 iops : min=235566, max=286286, avg=273417.14, stdev=4303.79, samples=28 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=96.06%, 500=3.67% lat (usec) : 750=0.24%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.68%, sys=29.99%, ctx=1048987, majf=0, minf=102 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=4096MiB (4295MB), run=3835-3835msec
Disk stats (read/write): sdc: ios=999346/0, sectors=7994768/0, merge=0/0, ticks=10360/0, in_queue=10361, util=95.49%
On 20/03/2025 12:23, Eneko Lacunza wrote:
Hi Giovanna,
I just tested one of my VMs: # fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.33 Starting 4 processes registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) registry-read: Laying out IO file (1 file / 1024MiB) Jobs: 4 (f=0): [f(4)][100.0%][r=33.5MiB/s][r=8578 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24261: Thu Mar 20 12:57:26 2025 read: IOPS=8538, BW=33.4MiB/s (35.0MB/s)(2001MiB/60001msec) slat (usec): min=309, max=4928, avg=464.54, stdev=73.15 clat (nsec): min=602, max=1532.4k, avg=1999.15, stdev=3724.16 lat (usec): min=310, max=4931, avg=466.54, stdev=73.36 clat percentiles (nsec): | 1.00th=[ 812], 5.00th=[ 884], 10.00th=[ 940], 20.00th=[ 1096], | 30.00th=[ 1368], 40.00th=[ 1576], 50.00th=[ 1720], 60.00th=[ 1832], | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2480], 95.00th=[ 3024], | 99.00th=[12480], 99.50th=[15808], 99.90th=[47360], 99.95th=[61696], | 99.99th=[90624] bw ( KiB/s): min=30448, max=35868, per=100.00%, avg=34155.76, stdev=269.75, samples=476 iops : min= 7612, max= 8966, avg=8538.87, stdev=67.43, samples=476 lat (nsec) : 750=0.06%, 1000=14.94% lat (usec) : 2=59.18%, 4=23.07%, 10=1.28%, 20=1.17%, 50=0.21% lat (usec) : 100=0.08%, 250=0.01%, 500=0.01% lat (msec) : 2=0.01% cpu : usr=1.04%, sys=5.50%, ctx=537639, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512316,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2001MiB (2098MB), run=60001-60001msec
Results are worse than yours, but this is on a production (not very busy) pool with 4x3.84TB SATA disks (4 disks total vs ~15 disks in your case) and 10G network.
VM cpu is x86_64_v3 and host CPU Ryzen 1700.
I gest almost the same IOPS with --iodepth=16 .
I tried moving the VM to a Ryzen 5900X and results are somewhat better:
# fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.33 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=45.4MiB/s][r=11.6k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=24282: Thu Mar 20 13:18:23 2025 read: IOPS=11.6k, BW=45.5MiB/s (47.7MB/s)(2730MiB/60001msec) slat (usec): min=110, max=21206, avg=341.21, stdev=79.69 clat (nsec): min=1390, max=42395k, avg=5147009.08, stdev=475506.40 lat (usec): min=335, max=42779, avg=5488.22, stdev=498.03 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4752], 10.00th=[ 4817], 20.00th=[ 4948], | 30.00th=[ 5014], 40.00th=[ 5080], 50.00th=[ 5080], 60.00th=[ 5145], | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5407], 95.00th=[ 5538], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[ 9765], 99.95th=[12125], | 99.99th=[24249] bw ( KiB/s): min=36434, max=48352, per=100.00%, avg=46612.18, stdev=300.09, samples=476 iops : min= 9108, max=12088, avg=11653.04, stdev=75.03, samples=476 lat (usec) : 2=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%, 50=0.01% cpu : usr=0.98%, sys=4.18%, ctx=706399, majf=0, minf=99 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=698956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2730MiB (2863MB), run=60001-60001msec
I think we're limited by the IO thread. I suggest you try multiple disks with SCSI Virtio single.
My VM conf: agent: 1 boot: order=scsi0;ide2;net0 cores: 2 cpu: x86-64-v3 ide2: none,media=cdrom memory: 2048 meta: creation-qemu=9.0.2,ctime=1739888364 name: elacunza-btrfs-test net0: virtio=BC:24:11:47:9B:58,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: proxmox_r3_ssd2:vm-112-disk-0,discard=on,iothread=1,size=15G scsihw: virtio-scsi-single smbios1: uuid=263ab229-4379-4abf-b6bf-615b98ccd3d4 sockets: 1 vmgenid: 13b7f2a4-2a42-4600-845a-da88f96ae6e8
I think this is a KVM/QEMU issue, not a Ceph issue :) Maybe you can get better suggestions in pve-user mailing list.
Cheers
El 20/3/25 a las 12:29, Giovanna Ratini escribió:
Hello Eneko,
this is my configuration. The performance is similar across all VMs. I am now checking GitLab, as that is where people are complaining the most.
agent: 1 balloon: 65000 bios: ovmf boot: order=scsi0;net0 cores: 10 cpu: host efidisk0: cephvm:vm-6506-disk-0,efitype=4m,size=528K memory: 130000 meta: creation-qemu=9.0.2,ctime=1734995123 name: gitlab02 net0: virtio=BC:24:11:6E:28:71,bridge=vmbr1,firewall=1 numa: 0 ostype: l26 scsi0: cephvm:vm-6506-disk-1,aio=native,cache=writeback,iothread=1,size=64G,ssd=1 scsi1: cephvm:vm-6506-disk-2,aio=native,cache=writeback,iothread=1,size=10T,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=0a5294c0-c82a-40f2-aae4-f5880022a2ac sockets: 2 vmgenid: ea610fde-6c71-4b7f-9257-fa431a428e16
Cheers,
Gio
Am 20.03.2025 um 10:23 schrieb Eneko Lacunza:
Hi Giovanna,
Can you post VM's full config?
Also, can you test with IO thread enabled and SCSI virtio single, and multiple disks?
Cheers
El 19/3/25 a las 17:27, Giovanna Ratini escribió:
hello Eneko,
Yes I did. No significant changes. :-( Cheers,
Gio
Am Mittwoch, März 19, 2025 13:09 CET, schrieb Eneko Lacunza <elacunza@binovo.es>:
> Hi Giovanna, > > Have you tried increasing iothreads option for the VM? > > Cheers > > El 18/3/25 a las 19:13, Giovanna Ratini escribió: > > Hello Antony, > > > > no, no QoS applied to Vms. > > > > The Server has PCIe Gen 4 > > > > ceph osd dump | grep pool > > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 > object_hash > > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 > flags > > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 > application mgr > > read_balance_score 13.04 > > pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 > > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > > last_change 598 lfor 0/598/596 flags hashpspool stripe_width 0 > > application cephfs read_balance_score 2.02 > > pool 3 'cephfs_metadata' replicated size 3 min_size 2 > crush_rule 0 > > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > > last_change 50 flags hashpspool stripe_width 0 > pg_autoscale_bias 4 > > pg_num_min 16 recovery_priority 5 application cephfs > > read_balance_score 2.42 > > pool 4 'cephvm' replicated size 3 min_size 2 crush_rule 0 > object_hash > > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change > 16386 > > lfor 0/644/2603 flags hashpspool,selfmanaged_snaps stripe_width 0 > > application rbd read_balance_score 1.52 > > > > I think, this is the default config. 🙈 > > > > I will search for my chassies supermicro upgrade. > > > > Thank you > > > > > > Am 18.03.2025 um 17:57 schrieb Anthony D'Atri: > >>> Then I tested on the *Proxmox host*, and the results were > >>> significantly better. > >> My Proxmox prowess is limited, but from my experience with other > >> virtualization platforms, I have to ask if there is any QoS > >> throttling applied to VMs. With OpenStack or DO there is > often IOPS > >> and/or throughput throttling via libvirt to mitigate noisy > neighbors. > >> > >>> fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio > >>> --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G > >>> --runtime=60 --group_reporting > >>> > >>> *IOPS*: *1.54M* > >>> > >>> # *Bandwidth*: *6032MiB/s (6325MB/s)* > >>> # *Latency*: > >>> > >>> * *Avg*: *39.8µs* > >>> * *99.9th percentile*: *71µs* > >>> > >>> # *CPU Usage*: *usr=22.60%, sys=77.13%* > >>> # > >>> > >>> Am 18.03.2025 um 15:27 schrieb Anthony D'Atri: > >>>> Which NVMe drive SKUs specifically? > >>> # */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 > >>> # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 > >>> # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8 > >> Kioxia CD6. If you were using client-class drives all manner of > >> performance issues would be expected. > >> > >> Is your server chassis at least PCIe Gen 4? If it’s Gen 3 > that may > >> hamper these drives. > >> > >> Also, how many of these are in your cluster? If it’s a small > number > >> you might still benefit from chopping each into at least 2 > separate > >> OSDs. > >> > >> And please send `ceph osd dump | grep pool`, having too few PGs > >> wouldn’t do you any favors. > >> > >> > >>>> Are you running a recent kernel? > >>> penultimate: 6.8.12-8-pve (VM, yes) > >> Groovy. If you were running like a CentOS 6 or CentOS 7 > kernel then > >> NVMe issues might be expected as old kernels had rudimentary > NVMe > >> support. > >> > >>>> Have you updated firmware on the NVMe devices? > >>> No. > >> Kioxia appears to not release firmware updates publicly but your > >> chassis brand (Dell, HP, SMCI, etc) might have an update. > >> > e.g.https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... > > >> > >> > >> If there is an available update I would strongly suggest > applying. > > > >> > >>> Thanks again, > >>> > >>> best regards, > >>> Gio > >>> > >>> _______________________________________________ > >>> ceph-users mailing list --ceph-users@ceph.io > >>> To unsubscribe send an email toceph-users-leave@ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > > Eneko Lacunza > Zuzendari teknikoa | Director técnico > Binovo IT Human Project > > Tel. +34 943 569 206 <tel:+34 943 569 206> | https://www.binovo.es > Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun > > https://www.youtube.com/user/CANALBINOVO > https://www.linkedin.com/company/37269706/ > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
EnekoLacunza
Director Técnico | Zuzendari teknikoa
Binovo IT Human Project
943 569 206 <tel:943 569 206>
elacunza@binovo.es <mailto:elacunza@binovo.es>
binovo.es <//binovo.es>
Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io