On Wed, Jul 19, 2023 at 11:01 AM Engelmann Florian <florian.engelmann@everyware.ch> wrote:
Hi,
I noticed an incredible high performance drop with mkfs.ext4 (as well as mkfs.xfs) when setting (almost) "any" value for rbd_qos_write_bps_limit (or rbd_qos_bps_limit).
Baseline: 4TB rbd volume rbd_qos_write_bps_limit = 0 mkfs.ext4: real 0m6.688s user 0m0.000s sys 0m0.006s
50GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 53687091200 mkfs.ext4: real 1m22.217s user 0m0.009s sys 0m0.000s
5GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 5368709120 mkfs.ext4: real 13m39.770s user 0m0.008s sys 0m0.034s
500MB/s: 4TB rbd volume rbd_qos_write_bps_limit = 524288000 mkfs.ext4: test still runing... I can provide the result if needed.
The tests are running on a client vm (Ubuntu 22.04) using Qemu/libvirt.
Using the same values with Qemu/libvirt QoS does not affect mkfs performance. https://libvirt.org/formatdomain.html#block-i-o-tuning
Ceph Version: 16.2.11 Qemu: 6.2.0 Libvirt: 8.0.0 Kernel (hypervisor host): 5.19.0-35-generic librbd1 (hypervisor host): 17.2.5
Could anyone pls confirm and explain what's going on?
Hi Florian, RBD QoS write limits apply to all write-like operations, including discards. By default, both mkfs.ext4 and mkfs.xfs attempt to discard the entire partition/device and librbd QoS machinery treats that as 4TB worth of writes. RBD images are thin-provisioned, so if you are creating a filesystem on a freshly created image, you can skip discarding with "-E nodiscard" for mkfs.ext4 or "-K" for mkfs.xfs. Alternatively, you can waive QoS limits for discards (or even an arbitrary combination of operations) by setting rbd_qos_exclude_ops option [1] appropriately. [1] https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#confval-rbd_qos_exclude_... Thanks, Ilya