I observed that on an otherwise idle cluster, scrubbing cannot fully utilise the speed of my HDDs.
Maybe the configured limit is set like this, because of that once (a part of) the scrubbing process is started it is not possible/easy to automatically scale down the performance to benefit client io.
`iostat` shows only 8-10 MB/s per disk, instead of the ~100 MB/s most HDDs can easily deliver.
100MB/s is sequential, your scrubbing is random. afaik everything is random.
Changing scrubbing settings does not help (see below).
I think you should be able to use the full performance of the disk when ceph tell osd.* injectargs '--osd_max_scrubs=X'. I never tried increasing the individual scrub speed, maybe such a setting is available ceph tell osd.* injectargs '--osd_recovery_sleep_hdd=0.100000' Best is to stick as much as possible to the defaults.