Hi, after upgrading one of our production clusters from Ceph 19.2.3 to 20.2.1 we observed that deep scrubbing is no longer keeping up with the configured schedule. The cluster is healthy apart from: HEALTH_WARN 400 pgs not deep-scrubbed in time The number of overdue PGs is steadily increasing (e.g. 0 -> 400 within 4 days). Cluster Ceph 20.2.1 (upgraded from 19.2.3 approximately one week ago) 18 OSDs NVMe SSDs 2208 PGs ~27 TiB user data ~64% raw utilization No recovery/backfill All PGs are active+clean 18 osds: 18 up, 18 in flags: noautoscale OSD latencies are essentially zero: ceph osd perf commit_latency = 0 ms apply_latency = 0 ms Scrub configuration The scrub window has been unchanged for a long time: osd_scrub_begin_hour = 21 osd_scrub_end_hour = 6 The cluster kept up with this configuration before the upgrade. The scheduler configuration is: osd_op_queue = mclock_scheduler osd_mclock_profile = high_recovery_ops This profile was already configured before the upgrade. Observation: I collect the ceph_pg_scrubbing Prometheus metric. Before the upgrade (19.2.3): scrubbing starts around 23:00 continues throughout the whole scrub window there are continuously 1-5 PGs being scrubbed during the night After the upgrade (20.2.1): there is a large initial burst (around 30 PGs scrubbing simultaneously) after that, scrubbing almost completely stops only occasional individual PGs are scrubbed during the rest of the scrub window As a result, the number of overdue deep scrubs increases every day. Questions: Were there any changes to scrub scheduling or mClock behavior between 19.2.3 and 20.2.1 that could explain this? Is this a known issue in 20.2.1? Is high_recovery_ops known to affect deep scrub scheduling differently in Ceph 20 compared to Ceph 19? Are there any debug commands or perf counters that would help determine why new scrub jobs are not being scheduled after the initial burst? I'd appreciate any suggestions on what to investigate next. Thanks! Kamil Madac