The question you should ask yourself, why you want to change/investigate this?
Because if scrubbing takes 10x longer thrashing seeks, my scrubs never finish in time (the default is 1 week). I end with e.g.
267 pgs not deep-scrubbed in time
On a 38 TB cluster, if you scrub 8 MB/s on 10 disks (using only numbers already divided by replication factor), you need 55 days to scrub it once.
That's 8x larger than the default scrub factor, so I'll get warnings and my risk of data degradation increases.
Also, even if I set the default scrub interval to 8x larger, it my disks will still be thrashing seeks 100% of the time, affecting the cluster's throughput and latency performance.
Oh I get it. Interesting. I think if you will expand the cluster in the future with more disks you will spread the load have more iops, this will disappear. I am not sure if you will be able to fix this other than to increase the scrub interval. If you are sure it is nothing related to hardware. For you reference I have included how my disk io / performance looks like when I issue a deep-scrub. You can see it reads 2 disks here at ~70MB/s and the atop shows it is at 100% load. Nothing more you can do here. #ceph osd pool ls detail #ceph pg ls | grep '^53' #ceph osd tree #ceph pg deep-scrub 53.38 #dstat -D sdd,sde [@~]# dstat -d -D sdd,sde,sdj --dsk/sdd-----dsk/sde-----dsk/sdj-- read writ: read writ: read writ 2493k 177k:5086k 316k:5352k 422k 70M 0 : 89M 0 : 0 68k 78M 0 : 59M 0 : 0 0 68M 0 : 68M 0 : 0 28k 90M 4096B: 90M 80k:4096B 24k 76M 0 : 78M 0 : 0 12k 66M 0 : 64M 0 : 0 12k 70M 0 : 80M 0 :4096B 52k 77M 0 : 70M 0 : 0 0 atop: | DSK | sdd | busy 97% | read 1462 | write 4 | KiB/r 469 | KiB/w 5 | MBr/s 67.0 | MBw/s 0.0 | avq 1.01 | avio 6.59 ms | DSK | sde | busy 64% | read 1472 | write 4 | KiB/r 465 | KiB/w 6 | MBr/s 67.0 | MBw/s 0.0 | avq 1.01 | avio 4.32 ms | DSK | sdb | busy 1% | read 0 | write 82 | KiB/r 0 | KiB/w 9 | MBr/s 0.0 | MBw/s 0.1 | avq 1.30 | avio 1.29 ms |