Hi all, I’m running a 10-node Ceph cluster (Reef 18.2.7) that has been having persistent issues keeping PG scrubs up to date as we migrate workloads from our previous production cluster. This environment provides CephFS only. The main CephFS data pool is an EC 4+2 pool backed by HDD OSDs, and the metadata pool is a REP 3 pool on SSD OSDs. The hardware is relatively old (mostly Dell PowerEdge R630s with SAS-2 attached JBODs). This cluster was deployed as an interim platform while we rebuild our production environment following unrecoverable MDS corruption. The scrub backlog started to accumulate as soon as we began migrating data onto this interim cluster, and although I’ve been able to mitigate it at times, it has been growing again now that the majority of our clients have shifted over to it. Below is the current cluster status:

cluster: id: 150735d4-4449-11f0-a328-a0369ff049b4 health: HEALTH_WARN 126 pgs not scrubbed in time

services: mon: 5 daemons, quorum sio-nautiloid01,sio-nautiloid02,sio-nautiloid03,sio-nautiloid07,sio-nautiloid06 (age 4w) mgr: sio-nautiloid02.jbayyi(active, since 5h), standbys: sio-nautiloid01.ssmngh mds: 1/1 daemons up, 1 standby osd: 248 osds: 248 up (since 4w), 248 in (since 9w)

data: volumes: 1/1 healthy pools: 6 pools, 8417 pgs objects: 1.04G objects, 1.3 PiB usage: 1.9 PiB used, 1.7 PiB / 3.6 PiB avail pgs: 6913 active+clean 669 active+clean+scrubbing+deep 408 active+clean+scrubbing 405 active+clean+snaptrim_wait 22 active+clean+snaptrim

io: client: 37 MiB/s rd, 4.3 MiB/s wr, 1.11k op/s rd, 53 op/s wr

Current scrub-related configuration:

mgr advanced

osd_deep_scrub_interval 5184000.000000

mgr advanced osd_scrub_max_interval 2419200.000000

mgr advanced osd_scrub_min_interval 1209600.000000

osd advanced osd_deep_scrub_interval 4838400.000000

osd advanced osd_deep_scrub_randomize_ratio 0.150000

osd advanced osd_max_scrubs 3

osd advanced osd_scrub_chunk_max 25

osd advanced osd_scrub_load_threshold 0.900000

osd advanced osd_scrub_max_interval 2419200.000000

osd advanced osd_scrub_min_interval 1209600.000000

osd advanced osd_scrub_priority 10

osd advanced osd_scrub_sleep 0.100000

These intervals mirror what we had configured on our previous production cluster (Quincy 17.2.7). I introduced osd_scrub_sleep after observing high commit and apply latency in ceph osd perf. I’ve also experimented with higher scrub priority and a higher load threshold, but the number of overdue scrubs continues to grow. At this point I’m looking for guidance on: - Additional tuning options that might help scrub throughput - Whether my current interval choices are counterproductive - How to identify the primary bottlenecks (OSD load, disk latency, stride size, concurrency, etc.) Any insights would be greatly appreciated. Thanks, Zach