Le 20/09/2024 à 08:00:16+0000, Eugen Block a écrit Hi,
there's some ratio involved when deep-scrubs are checked:
(mon_warn_pg_not_deep_scrubbed_ratio * deep_scrub_interval) + deep_scrub_interval
So based on the defaults, ceph would only warn if the last deep-scrub timestamp is older than:
(0.75 * 7 days) + 7 days = 12.25 days
OK. Thanks. It fit perfectly.
Is they are any way to find which pg ceph status are talking about.
'ceph health detail' will show you which PG it's warning about.
Too easy for me ;-) ;-)...Thanks ;-)
Is they are any way to see the progress or scrubbing/remapping/backfill ?
You can see when (deep-)scrubs have been started in the OSD logs or depending on your cluster log configuration:
ceph log last 1000 debug cluster | grep scrub
Thanks. So I think I get some issue with one osd 2024-09-20T08:55:59.760766+0000 osd.356 (osd.356) 84 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:03.751487+0000 osd.356 (osd.356) 85 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:05.742816+0000 osd.356 (osd.356) 86 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:07.822277+0000 osd.356 (osd.356) 87 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:08.795748+0000 osd.356 (osd.356) 88 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:09.749838+0000 osd.356 (osd.356) 89 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:10.778235+0000 osd.356 (osd.356) 90 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:14.792102+0000 osd.356 (osd.356) 91 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:15.832620+0000 osd.356 (osd.356) 92 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:16.791811+0000 osd.356 (osd.356) 93 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:17.798181+0000 osd.356 (osd.356) 94 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:19.793526+0000 osd.356 (osd.356) 95 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:20.809140+0000 osd.356 (osd.356) 96 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:21.835052+0000 osd.356 (osd.356) 97 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:22.817378+0000 osd.356 (osd.356) 98 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:28.887092+0000 osd.356 (osd.356) 99 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:32.907468+0000 osd.356 (osd.356) 100 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:33.900065+0000 osd.356 (osd.356) 101 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:37.853769+0000 osd.356 (osd.356) 102 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:38.850513+0000 osd.356 (osd.356) 103 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:46.799896+0000 osd.356 (osd.356) 104 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:50.915450+0000 osd.356 (osd.356) 105 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:53.875875+0000 osd.356 (osd.356) 106 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:55.791675+0000 osd.356 (osd.356) 107 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:57.833012+0000 osd.356 (osd.356) 108 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:59.818262+0000 osd.356 (osd.356) 109 : cluster [DBG] 4.51 deep-scrub starts but the osd are still in queued for deep scrub in ceph pg dump pgs
The (deep-)scrub duration depends on the PG sizes, so they can vary. But from experience (and older Logs) you can see if scrubbing duration has increased. I haven't checked if there's a metric for that in prometheus.
As for remapping and backfill operations, they are constantly reported in 'ceph status', it shows how many objects are degraded, how many PGs are remapped etc. If you mean something else, please clarify.
Yes. But the point is I got 7 active+remapped+backfilling 2 active+remapped+backfill_wait but I've no idea if that's the 7 (and 2 ) same pg and are stuck in the process since X days.
[0] https://docs.ceph.com/en/latest/rados/operations/health-checks/#pg-not-deep-... [1] https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-tim...
Thanks... Regards. -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 20 sept. 2024 10:55:41 CEST