Considering https://github.com/ceph/ceph/blob/f6edcef6efe209e8947887752bd2b833d0ca13b7/s..., the OSD: - always sets and updates its per osd osd_mclock_max_capacity_iops_{hdd,ssd} when the benchmark occurs and its measured iops is below or equal to osd_mclock_iops_capacity_threshold_{hdd,ssd} but - doesn't remove osd_mclock_max_capacity_iops_{hdd,ssd} when the measured iops exceeds osd_mclock_iops_capacity_threshold_{hdd,ssd} (500 for HDD and 80.000 for SSD) and the current value for osd_mclock_max_capacity_iops_{hdd,ssd} is set below its default (315 for HDD and 21500 for SSD) Thus per osd osd_mclock_max_capacity_iops_hdd sometimes being set as low as 0.145327 (as per Michel's post) and never being updated afterwards leading to performance issues. The idea of a minimum threshold below which osd_mclock_iops_capacity_threshold_{hdd,ssd} should not be set seems relevant. CC'ing Sridhar to have his thoughts. Cheers, Frédéric. ----- Le 22 Mar 24, à 19:37, Kai Stian Olstad ceph+list@olstad.com a écrit :
On Fri, Mar 22, 2024 at 06:51:44PM +0100, Frédéric Nass wrote:
The OSD run bench and update osd_mclock_max_capacity_iops_{hdd,ssd} every time the OSD is started. If you check the OSD log you'll see it does the bench. Are you sure about the update on every start? Does the update happen only if the benchmark result is < 500 iops? Looks like the OSD does not remove any set configuration when the benchmark result is > 500 iops. Otherwise, the extremely low value that Michel reported earlier (less than 1 iops) would have been updated over time. I guess.
I'm not completely sure, it's a couple a month since I used mclock, have switch back to wpq because of a nasty bug in mclock that can freeze cluster I/O.
It could be because I was testing osd_mclock_force_run_benchmark_on_init. The OSD had DB on SSD and data on HDD, so the measured to about 1700 IOPS and was ignored because of the 500 limit. So only the SSD got the osd_mclock_max_capacity_iops_ssd set.
-- Kai Stian Olstad