Hi everyone. Few time ago I add a new node to my cluster with some HDD. Currently the cluster does the remapping and backfill. I now got a warning about HEALTH_WARN 1 pgs not deep-scrubbed in time So I check and find something a litle weird. root@cthulhu1:~# ceph config get osd osd_deep_scrub_interval 604800.000000 so that's one week. If I check the LAST DEEP SCRUB TIMESTAMP I got root@cthulhu1:~# ceph pg dump pgs | awk '{print $1" "$24}' | grep -v 2024-09-[1-2][0-9] dumped pgs PG_STAT DEEP_SCRUB_STAMP 4.63 2024-09-09T19:00:57.739975+0000 4.5a 2024-09-09T08:17:15.124704+0000 4.56 2024-09-09T21:51:07.478651+0000 4.51 2024-09-08T00:10:30.552347+0000 4.4c 2024-09-09T10:35:02.048445+0000 4.4b 2024-09-09T19:53:19.839341+0000 4.14 2024-09-08T18:36:12.025455+0000 4.c 2024-09-09T16:00:59.047968+0000 4.4 2024-09-09T00:19:07.554153+0000 4.8 2024-09-09T22:19:15.280310+0000 4.25 2024-09-09T06:45:37.258306+0000 4.30 2024-09-09T16:56:21.472410+0000 4.82 2024-09-09T21:14:09.802303+0000 4.c9 2024-09-08T17:10:56.133363+0000 4.f7 2024-09-09T08:25:40.011924+0000 If I check the status of those PG it's or active+clean+scrubbing+deep and deep scrubbing for Xs or queued for deep scrub So my questions are : Why ceph tell me «1» pg not been scrub when I see 15 ? Is they are any way to find which pg ceph status are talking about. Is they are any way to see the progress or scrubbing/remapping/backfill ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 20 sept. 2024 09:35:43 CEST
Hi, there's some ratio involved when deep-scrubs are checked: (mon_warn_pg_not_deep_scrubbed_ratio * deep_scrub_interval) + deep_scrub_interval So based on the defaults, ceph would only warn if the last deep-scrub timestamp is older than: (0.75 * 7 days) + 7 days = 12.25 days Note that the MGR has also a config for deep_scrub_interval. Check out the docs [0] or my recent blog [1] on that topic.
Why ceph tell me «1» pg not been scrub when I see 15 ?
See my reply above.
Is they are any way to find which pg ceph status are talking about.
'ceph health detail' will show you which PG it's warning about.
Is they are any way to see the progress or scrubbing/remapping/backfill ?
You can see when (deep-)scrubs have been started in the OSD logs or depending on your cluster log configuration: ceph log last 1000 debug cluster | grep scrub The (deep-)scrub duration depends on the PG sizes, so they can vary. But from experience (and older Logs) you can see if scrubbing duration has increased. I haven't checked if there's a metric for that in prometheus. As for remapping and backfill operations, they are constantly reported in 'ceph status', it shows how many objects are degraded, how many PGs are remapped etc. If you mean something else, please clarify. Regards, Eugen [0] https://docs.ceph.com/en/latest/rados/operations/health-checks/#pg-not-deep-... [1] https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-tim... Zitat von Albert Shih <Albert.Shih@obspm.fr>:
Hi everyone.
Few time ago I add a new node to my cluster with some HDD.
Currently the cluster does the remapping and backfill.
I now got a warning about
HEALTH_WARN 1 pgs not deep-scrubbed in time
So I check and find something a litle weird.
root@cthulhu1:~# ceph config get osd osd_deep_scrub_interval 604800.000000
so that's one week.
If I check the LAST DEEP SCRUB TIMESTAMP I got
root@cthulhu1:~# ceph pg dump pgs | awk '{print $1" "$24}' | grep -v 2024-09-[1-2][0-9] dumped pgs PG_STAT DEEP_SCRUB_STAMP 4.63 2024-09-09T19:00:57.739975+0000 4.5a 2024-09-09T08:17:15.124704+0000 4.56 2024-09-09T21:51:07.478651+0000 4.51 2024-09-08T00:10:30.552347+0000 4.4c 2024-09-09T10:35:02.048445+0000 4.4b 2024-09-09T19:53:19.839341+0000 4.14 2024-09-08T18:36:12.025455+0000 4.c 2024-09-09T16:00:59.047968+0000 4.4 2024-09-09T00:19:07.554153+0000 4.8 2024-09-09T22:19:15.280310+0000 4.25 2024-09-09T06:45:37.258306+0000 4.30 2024-09-09T16:56:21.472410+0000 4.82 2024-09-09T21:14:09.802303+0000 4.c9 2024-09-08T17:10:56.133363+0000 4.f7 2024-09-09T08:25:40.011924+0000
If I check the status of those PG it's or
active+clean+scrubbing+deep and deep scrubbing for Xs
or
queued for deep scrub
So my questions are :
Why ceph tell me «1» pg not been scrub when I see 15 ?
Is they are any way to find which pg ceph status are talking about.
Is they are any way to see the progress or scrubbing/remapping/backfill ?
Regards
-- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 20 sept. 2024 09:35:43 CEST _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 20/09/2024 à 08:00:16+0000, Eugen Block a écrit Hi,
there's some ratio involved when deep-scrubs are checked:
(mon_warn_pg_not_deep_scrubbed_ratio * deep_scrub_interval) + deep_scrub_interval
So based on the defaults, ceph would only warn if the last deep-scrub timestamp is older than:
(0.75 * 7 days) + 7 days = 12.25 days
OK. Thanks. It fit perfectly.
Is they are any way to find which pg ceph status are talking about.
'ceph health detail' will show you which PG it's warning about.
Too easy for me ;-) ;-)...Thanks ;-)
Is they are any way to see the progress or scrubbing/remapping/backfill ?
You can see when (deep-)scrubs have been started in the OSD logs or depending on your cluster log configuration:
ceph log last 1000 debug cluster | grep scrub
Thanks. So I think I get some issue with one osd 2024-09-20T08:55:59.760766+0000 osd.356 (osd.356) 84 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:03.751487+0000 osd.356 (osd.356) 85 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:05.742816+0000 osd.356 (osd.356) 86 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:07.822277+0000 osd.356 (osd.356) 87 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:08.795748+0000 osd.356 (osd.356) 88 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:09.749838+0000 osd.356 (osd.356) 89 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:10.778235+0000 osd.356 (osd.356) 90 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:14.792102+0000 osd.356 (osd.356) 91 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:15.832620+0000 osd.356 (osd.356) 92 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:16.791811+0000 osd.356 (osd.356) 93 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:17.798181+0000 osd.356 (osd.356) 94 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:19.793526+0000 osd.356 (osd.356) 95 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:20.809140+0000 osd.356 (osd.356) 96 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:21.835052+0000 osd.356 (osd.356) 97 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:22.817378+0000 osd.356 (osd.356) 98 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:28.887092+0000 osd.356 (osd.356) 99 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:32.907468+0000 osd.356 (osd.356) 100 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:33.900065+0000 osd.356 (osd.356) 101 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:37.853769+0000 osd.356 (osd.356) 102 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:38.850513+0000 osd.356 (osd.356) 103 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:46.799896+0000 osd.356 (osd.356) 104 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:50.915450+0000 osd.356 (osd.356) 105 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:53.875875+0000 osd.356 (osd.356) 106 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:55.791675+0000 osd.356 (osd.356) 107 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:57.833012+0000 osd.356 (osd.356) 108 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:59.818262+0000 osd.356 (osd.356) 109 : cluster [DBG] 4.51 deep-scrub starts but the osd are still in queued for deep scrub in ceph pg dump pgs
The (deep-)scrub duration depends on the PG sizes, so they can vary. But from experience (and older Logs) you can see if scrubbing duration has increased. I haven't checked if there's a metric for that in prometheus.
As for remapping and backfill operations, they are constantly reported in 'ceph status', it shows how many objects are degraded, how many PGs are remapped etc. If you mean something else, please clarify.
Yes. But the point is I got 7 active+remapped+backfilling 2 active+remapped+backfill_wait but I've no idea if that's the 7 (and 2 ) same pg and are stuck in the process since X days.
[0] https://docs.ceph.com/en/latest/rados/operations/health-checks/#pg-not-deep-... [1] https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-tim...
Thanks... Regards. -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 20 sept. 2024 10:55:41 CEST
Le 20/09/2024 à 11:01:20+0200, Albert Shih a écrit Hi,
Is they are any way to find which pg ceph status are talking about.
'ceph health detail' will show you which PG it's warning about.
Too easy for me ;-) ;-)...Thanks ;-)
Is they are any way to see the progress or scrubbing/remapping/backfill ?
You can see when (deep-)scrubs have been started in the OSD logs or depending on your cluster log configuration:
ceph log last 1000 debug cluster | grep scrub
Thanks.
So I think I get some issue with one osd
2024-09-20T08:55:59.760766+0000 osd.356 (osd.356) 84 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:03.751487+0000 osd.356 (osd.356) 85 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:05.742816+0000 osd.356 (osd.356) 86 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:07.822277+0000 osd.356 (osd.356) 87 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:08.795748+0000 osd.356 (osd.356) 88 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:09.749838+0000 osd.356 (osd.356) 89 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:10.778235+0000 osd.356 (osd.356) 90 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:14.792102+0000 osd.356 (osd.356) 91 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:15.832620+0000 osd.356 (osd.356) 92 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:16.791811+0000 osd.356 (osd.356) 93 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:17.798181+0000 osd.356 (osd.356) 94 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:19.793526+0000 osd.356 (osd.356) 95 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:20.809140+0000 osd.356 (osd.356) 96 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:21.835052+0000 osd.356 (osd.356) 97 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:22.817378+0000 osd.356 (osd.356) 98 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:28.887092+0000 osd.356 (osd.356) 99 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:32.907468+0000 osd.356 (osd.356) 100 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:33.900065+0000 osd.356 (osd.356) 101 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:37.853769+0000 osd.356 (osd.356) 102 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:38.850513+0000 osd.356 (osd.356) 103 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:46.799896+0000 osd.356 (osd.356) 104 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:50.915450+0000 osd.356 (osd.356) 105 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:53.875875+0000 osd.356 (osd.356) 106 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:55.791675+0000 osd.356 (osd.356) 107 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:57.833012+0000 osd.356 (osd.356) 108 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:59.818262+0000 osd.356 (osd.356) 109 : cluster [DBG] 4.51 deep-scrub starts
but the osd are still in
queued for deep scrub
Yeah...well after few hours the scrub eventually started. So eveything seem fine. Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 20 sept. 2024 14:13:20 CEST
I assume that the OSD maybe had some backfill going on, hence waiting for the scheduled deep-scrub to start. There are config options which would allow deep-scrubs during recovery, I believe, but if it's not a real issue, you can leave it as is. Zitat von Albert Shih <Albert.Shih@obspm.fr>:
Le 20/09/2024 à 11:01:20+0200, Albert Shih a écrit Hi,
Is they are any way to find which pg ceph status are talking about.
'ceph health detail' will show you which PG it's warning about.
Too easy for me ;-) ;-)...Thanks ;-)
Is they are any way to see the progress or
scrubbing/remapping/backfill ?
You can see when (deep-)scrubs have been started in the OSD logs or depending on your cluster log configuration:
ceph log last 1000 debug cluster | grep scrub
Thanks.
So I think I get some issue with one osd
2024-09-20T08:55:59.760766+0000 osd.356 (osd.356) 84 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:03.751487+0000 osd.356 (osd.356) 85 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:05.742816+0000 osd.356 (osd.356) 86 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:07.822277+0000 osd.356 (osd.356) 87 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:08.795748+0000 osd.356 (osd.356) 88 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:09.749838+0000 osd.356 (osd.356) 89 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:10.778235+0000 osd.356 (osd.356) 90 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:14.792102+0000 osd.356 (osd.356) 91 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:15.832620+0000 osd.356 (osd.356) 92 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:16.791811+0000 osd.356 (osd.356) 93 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:17.798181+0000 osd.356 (osd.356) 94 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:19.793526+0000 osd.356 (osd.356) 95 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:20.809140+0000 osd.356 (osd.356) 96 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:21.835052+0000 osd.356 (osd.356) 97 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:22.817378+0000 osd.356 (osd.356) 98 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:28.887092+0000 osd.356 (osd.356) 99 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:32.907468+0000 osd.356 (osd.356) 100 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:33.900065+0000 osd.356 (osd.356) 101 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:37.853769+0000 osd.356 (osd.356) 102 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:38.850513+0000 osd.356 (osd.356) 103 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:46.799896+0000 osd.356 (osd.356) 104 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:50.915450+0000 osd.356 (osd.356) 105 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:53.875875+0000 osd.356 (osd.356) 106 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:55.791675+0000 osd.356 (osd.356) 107 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:57.833012+0000 osd.356 (osd.356) 108 : cluster [DBG] 4.51 deep-scrub starts 2024-09-20T08:56:59.818262+0000 osd.356 (osd.356) 109 : cluster [DBG] 4.51 deep-scrub starts
but the osd are still in
queued for deep scrub
Yeah...well after few hours the scrub eventually started.
So eveything seem fine.
Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 20 sept. 2024 14:13:20 CEST
participants (2)
-
Albert Shih
-
Eugen Block