Hi, since some time (I think upgrade to nautilus) we get X pgs not deep scrubbed in time I deep-scrubbed the pgs when the error occurred and expected the cluster to recover over time, but no such luck. The warning comes up again and again. In our spinning rust cluster we allow deep scrubbing only from 19:00 to 06:00 and changed the deep scrub interval to 28 days (detailed osd scrub config below). Looking at the pgs which are supposedly not deep scrubbed in time, reveals that their 28 day period is not over yet, example: # date && ceph health detail | awk '$1=="pg" { print $0 }' Di 11. Aug 09:32:17 CEST 2020 pg 0.787 not deep-scrubbed since 2020-07-30 03:08:24.899264 pg 0.70c not deep-scrubbed since 2020-07-30 02:45:08.989329 pg 0.6c1 not deep-scrubbed since 2020-07-30 03:01:15.199496 pg 13.3 not deep-scrubbed since 2020-07-30 03:29:54.536825 pg 0.d9 not deep-scrubbed since 2020-07-30 03:12:34.503586 pg 0.41a not deep-scrubbed since 2020-07-30 03:01:23.514582 pg 0.490 not deep-scrubbed since 2020-07-30 03:05:45.616100 I wonder if I have missed or messed up some parameter that could cause this. # ceph daemon osd.40 config show | grep scrub "mds_max_scrub_ops_in_progress": "5", "mon_scrub_inject_crc_mismatch": "0.000000", "mon_scrub_inject_missing_keys": "0.000000", "mon_scrub_interval": "86400", "mon_scrub_max_keys": "100", "mon_scrub_timeout": "300", "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", "mon_warn_pg_not_scrubbed_ratio": "0.500000", "osd_debug_deep_scrub_sleep": "0.000000", "osd_deep_scrub_interval": "2419200.000000", "osd_deep_scrub_keys": "1024", "osd_deep_scrub_large_omap_object_key_threshold": "200000", "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", "osd_deep_scrub_randomize_ratio": "0.150000", "osd_deep_scrub_stride": "1048576", "osd_deep_scrub_update_digest_min_age": "7200", "osd_max_scrubs": "1", "osd_op_queue_mclock_scrub_lim": "0.001000", "osd_op_queue_mclock_scrub_res": "0.000000", "osd_op_queue_mclock_scrub_wgt": "1.000000", "osd_requested_scrub_priority": "120", "osd_scrub_auto_repair": "false", "osd_scrub_auto_repair_num_errors": "5", "osd_scrub_backoff_ratio": "0.660000", "osd_scrub_begin_hour": "19", "osd_scrub_begin_week_day": "0", "osd_scrub_chunk_max": "25", "osd_scrub_chunk_min": "5", "osd_scrub_cost": "52428800", "osd_scrub_during_recovery": "false", "osd_scrub_end_hour": "6", "osd_scrub_end_week_day": "7", "osd_scrub_interval_randomize_ratio": "0.500000", "osd_scrub_invalid_stats": "true", "osd_scrub_load_threshold": "0.500000", "osd_scrub_max_interval": "604800.000000", "osd_scrub_max_preemptions": "5", "osd_scrub_min_interval": "172800.000000", "osd_scrub_priority": "5", "osd_scrub_sleep": "0.100000", ceph version 14.2.10 My ad hoc helper script #!/bin/bash # gently deep scrub all pgs which failed to deep scrub in set period ceph health detail | awk '$1=="pg" { print $2 } ' | while read -r pg do ceph pg deep-scrub "$pg" sleep 60 done Cheers, Dirk