I have been running a ceph cluster for a while and one of the main things that impacts performance is deep-scrubbing. I would like to limit this as much as possible and have tried the below options to do this:
osd scrub sleep = 1 # Time to sleep before scrubbing next group of chunks
osd scrub chunk max = 1 # Maximum number of chunks to scrub during a single operation
osd scrub chunk min = 1 # Minimum number of chunks to scrub during a single operation
osd scrub max pre-emption = 30 # Maximum number of times ceph pre-empts a deep scrub due to a client operation blocking it
osd client op priority = 63 # The priority of client operations
osd requested scrub priority = 1 # The priority of administrator requested scrubs
osd scrub priority = 1 # The priority of scheduled scrubs
These options combined do slow down scrubbing and make it easier on client IO, but there is still a performance impact when they happen. One other option I have found is:
‘osd debug deep scrub sleep’
Setting this option to even 0.1 will have an immediate and large effect (I have seen nodes that were scrubbing at 50MB/s to down to <5MB/s). The effect is larger than all the above options combined and is exactly what I want, however there is no documentation about this option and I don’t know the full impact of configuring it. It also spams the logs with messages saying that the deep scrub is sleeping for x seconds. So I would like to know:
Thank you and Regards