We’re running Quincy 17.2.7 here, And we see the iops benchmark performed on osd start: 2024-09-20T09:57:26.265+1000 7facbdc64540 1 osd.196 2879010 maybe_override_max_osd_capacity_for_qos osd bench result - bandwidth (MiB/sec): 4.369 iops: 1118.445 elapsed_sec: 2.682 2024-09-20T09:57:26.265+1000 7facbdc64540 0 log_channel(cluster) log [WRN] : OSD bench result of 1118.445008 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.196. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. Has the accuracy of this benchmark been improved in newer releases? From: Andrej Filipcic <andrej.filipcic@ijs.si> Date: Thursday, 19 September 2024 at 9:20 pm To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [EXT] [ceph-users] Re: mclock scheduler kills clients IOs External email: Please exercise caution Hi, the problem comes from older ceph releases. In our case, hdd iops were benchmarked in the range of 250 to 4000, which clearly makes no sense. At osd startup, the benchmark is skipped if that value is already in ceph config, so these initial benchmark values were never changed. To reset them, all osd.N osd_mclock_max_capacity_iops_hdd values should be removed and osds restarted. There is a safety mechanism (osd_mclock_iops_capacity_threshold_hdd) which prevents for the values to be overestimated. Best, Andrej On 19. 09. 24 11:33, Daniel Schreiber wrote:
Hi Denis,
we observed the same behaviour here. The cause was that the number of iops discovered at OSD startup was way too high. In our setup the rocksdb is on flash.
When I set osd_mclock_max_capacity_iops_hdd to a value that the HDDs could handle, the situation was resolved, clients got ther fair share of IO.
Hope this helps,
Daniel
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 ------------------------------------------------------------- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io