Documented here: https://github.com/ceph/ceph/blob/9754cafc029e1da83f5ddd4332b69066fe6b3ffb/s... Introduced back here with a bunch of other scrub tweaks: https://github.com/ceph/ceph/pull/18971/files Are your OSDs HDDs? Using EC? How many deep scrubs do you have running in parallel? Assuming more than one, you could increase osd_deep_scrub_interval to spread them out over time.
On Feb 7, 2023, at 05:58, Broccoli Bob <brockolibob@yahoo.com> wrote:
I have been running a ceph cluster for a while and one of the main things that impacts performance is deep-scrubbing. I would like to limit this as much as possible and have tried the below options to do this:
osd scrub sleep = 1 # Time to sleep before scrubbing next group of chunks osd scrub chunk max = 1 # Maximum number of chunks to scrub during a single operation osd scrub chunk min = 1 # Minimum number of chunks to scrub during a single operation osd scrub max pre-emption = 30 # Maximum number of times ceph pre-empts a deep scrub due to a client operation blocking it osd client op priority = 63 # The priority of client operations osd requested scrub priority = 1 # The priority of administrator requested scrubs osd scrub priority = 1 # The priority of scheduled scrubs
These options combined do slow down scrubbing and make it easier on client IO, but there is still a performance impact when they happen. One other option I have found is: ‘osd debug deep scrub sleep’
Setting this option to even 0.1 will have an immediate and large effect (I have seen nodes that were scrubbing at 50MB/s to down to <5MB/s). The effect is larger than all the above options combined and is exactly what I want, however there is no documentation about this option and I don’t know the full impact of configuring it. It also spams the logs with messages saying that the deep scrub is sleeping for x seconds. So I would like to know: Is it safe to have the debug deep scrub option turned on? If so is there a way to stop it from spamming the logs? If it is not safe are there any other options available to us to limit the impact of deep scrubbing? Thank you and Regards
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>