Of course I found the cause shortly after sending the message … The scrubbing parameters need to move from the [osd] section to the [global] section, see https://www.suse.com/support/kb/doc/?id=000019621 Health is back to OK after restarting osds, mons and mgrs. Cheers, Dirk On Dienstag, 11. August 2020 09:34:36 CEST Dirk Sarpe wrote:
Hi,
since some time (I think upgrade to nautilus) we get
X pgs not deep scrubbed in time
I deep-scrubbed the pgs when the error occurred and expected the cluster to recover over time, but no such luck. The warning comes up again and again.
In our spinning rust cluster we allow deep scrubbing only from 19:00 to 06:00 and changed the deep scrub interval to 28 days (detailed osd scrub config below). Looking at the pgs which are supposedly not deep scrubbed in time, reveals that their 28 day period is not over yet, example:
# date && ceph health detail | awk '$1=="pg" { print $0 }' Di 11. Aug 09:32:17 CEST 2020 pg 0.787 not deep-scrubbed since 2020-07-30 03:08:24.899264 pg 0.70c not deep-scrubbed since 2020-07-30 02:45:08.989329 pg 0.6c1 not deep-scrubbed since 2020-07-30 03:01:15.199496 pg 13.3 not deep-scrubbed since 2020-07-30 03:29:54.536825 pg 0.d9 not deep-scrubbed since 2020-07-30 03:12:34.503586 pg 0.41a not deep-scrubbed since 2020-07-30 03:01:23.514582 pg 0.490 not deep-scrubbed since 2020-07-30 03:05:45.616100
I wonder if I have missed or messed up some parameter that could cause this.
# ceph daemon osd.40 config show | grep scrub "mds_max_scrub_ops_in_progress": "5", "mon_scrub_inject_crc_mismatch": "0.000000", "mon_scrub_inject_missing_keys": "0.000000", "mon_scrub_interval": "86400", "mon_scrub_max_keys": "100", "mon_scrub_timeout": "300", "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", "mon_warn_pg_not_scrubbed_ratio": "0.500000", "osd_debug_deep_scrub_sleep": "0.000000", "osd_deep_scrub_interval": "2419200.000000", "osd_deep_scrub_keys": "1024", "osd_deep_scrub_large_omap_object_key_threshold": "200000", "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", "osd_deep_scrub_randomize_ratio": "0.150000", "osd_deep_scrub_stride": "1048576", "osd_deep_scrub_update_digest_min_age": "7200", "osd_max_scrubs": "1", "osd_op_queue_mclock_scrub_lim": "0.001000", "osd_op_queue_mclock_scrub_res": "0.000000", "osd_op_queue_mclock_scrub_wgt": "1.000000", "osd_requested_scrub_priority": "120", "osd_scrub_auto_repair": "false", "osd_scrub_auto_repair_num_errors": "5", "osd_scrub_backoff_ratio": "0.660000", "osd_scrub_begin_hour": "19", "osd_scrub_begin_week_day": "0", "osd_scrub_chunk_max": "25", "osd_scrub_chunk_min": "5", "osd_scrub_cost": "52428800", "osd_scrub_during_recovery": "false", "osd_scrub_end_hour": "6", "osd_scrub_end_week_day": "7", "osd_scrub_interval_randomize_ratio": "0.500000", "osd_scrub_invalid_stats": "true", "osd_scrub_load_threshold": "0.500000", "osd_scrub_max_interval": "604800.000000", "osd_scrub_max_preemptions": "5", "osd_scrub_min_interval": "172800.000000", "osd_scrub_priority": "5", "osd_scrub_sleep": "0.100000",
ceph version 14.2.10
My ad hoc helper script
#!/bin/bash
# gently deep scrub all pgs which failed to deep scrub in set period
ceph health detail | awk '$1=="pg" { print $2 } ' | while read -r pg do ceph pg deep-scrub "$pg" sleep 60 done
Cheers, Dirk
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- general it-support unit Phone +49 341 97-33118 Email dirk.sarpe@idiv.de <mailto:dirk.sarpe@idiv.de> German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig Deutscher Platz 5e 04103 Leipzig Germany iDiv is a research centre of the DFG - Deutsche Forschungsgemeinschaft iDiv ist ein Forschungszentrum der Deutschen Forschungsgemeinschaft (DFG). Es ist eine zentrale Einrichtung der Universität Leipzig im Sinne des § 92 Abs. 1 SächsHSFG und wird zusammen mit der Martin-Luther-Universität Halle-Wittenberg, der Friedrich-Schiller-Universität Jena sowie dem Helmholtz-Zentrum für Umweltforschung (UFZ) betrieben. Sieben außeruniversitäre Einrichtungen unterstützen iDiv finanziell sowie durch ihre Expertise: das Max-Planck-Institut für Biogeochemie (MPI BGC), das Max-Planck-Institut für chemische Ökologie (MPI CE), das Max-Planck-Institut für evolutionäre Anthropologie (MPI EVA), das Leibniz-Institut Deutsche Sammlung von Mikroorganismen und Zellkulturen (DSMZ), das Leibniz-Institut für Pflanzenbiochemie (IPB), das Leibnitz-Institut für Pflanzengenetik und Kulturpflanzenforschung (IPK) und das Leibniz-Institut Senckenberg Museum für Naturkunde Görlitz (SMNG). USt-IdNr. DE 141510383