pgs not deep-scrubbed in time and pgs not scrubbed in time
Hello Ceph Community, My cluster was hit by a power outage some month ago. Luckily no data was destroyed and powering up the nodes and services went well. But till than some pgs are still shown as not scrubbed in time. Googling and searching the list showed some debugging hints like „ceph pg deep-scrub“ the pgs or restarting osd deamons. Nothing „solved“ that issue here. I’m on ceph version 18.2.4 now. Is there anything special what I can do to have thous pgs scrubbed? I like having the cluster health state ok not warning :) Or will time solve the problem when the pgs are in there regular cycle for being scrubbed again? Thanks for hints and suggestion . Best regards Götz
Hi Götz, usually, OSDs start (deep-)scrubbing PGs after they have been powered on. You should see PGs in (deep-)scrubbing state right now. Depending on your PG sizes, number of OSDs etc., that can take some time, of course. But eventually the number should decrease over time. If you have the defaults for deep_scrub_interval (1 week) and your cluster hasn't complained before, you'll probably get rid of the warning within a week or so. If you want to speed things up and your cluster can handle the load, you could temporarily increase osd_max_scrubs (max concurrent scrubs on a single OSD, default 1, can be updated at runtime): ceph config set osd osd_max_scrubs 2 It doesn't sound like you had this warning before, so I assume it will eventually clear. If not, you can check out the docs [0] and my recent blog post [1] about this topic. Regards, Eugen [0] https://docs.ceph.com/en/latest/rados/operations/health-checks/#pg-not-deep-... [1] https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-tim... Zitat von Götz Reinicke <goetz.reinicke@filmakademie.de>:
Hello Ceph Community,
My cluster was hit by a power outage some month ago. Luckily no data was destroyed and powering up the nodes and services went well.
But till than some pgs are still shown as not scrubbed in time. Googling and searching the list showed some debugging hints like „ceph pg deep-scrub“ the pgs or restarting osd deamons.
Nothing „solved“ that issue here. I’m on ceph version 18.2.4 now.
Is there anything special what I can do to have thous pgs scrubbed? I like having the cluster health state ok not warning :) Or will time solve the problem when the pgs are in there regular cycle for being scrubbed again?
Thanks for hints and suggestion . Best regards Götz
Hi Götz, we have solved several scenarios like this in the past. Do you recognize a trend as to whether the pg not scrub is decreasing, remaining the same or increasing? There are a few osd configuration parameters that can be used to set the scrubbing behavior of the cluster: https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/ https://docs.clyso.com/blog/ceph-blocked-requests-in-the-cluster-caused-by-d... Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Am Mi., 23. Okt. 2024 um 09:37 Uhr schrieb Eugen Block <eblock@nde.ag>:
Hi Götz,
usually, OSDs start (deep-)scrubbing PGs after they have been powered on. You should see PGs in (deep-)scrubbing state right now. Depending on your PG sizes, number of OSDs etc., that can take some time, of course. But eventually the number should decrease over time. If you have the defaults for deep_scrub_interval (1 week) and your cluster hasn't complained before, you'll probably get rid of the warning within a week or so. If you want to speed things up and your cluster can handle the load, you could temporarily increase osd_max_scrubs (max concurrent scrubs on a single OSD, default 1, can be updated at runtime):
ceph config set osd osd_max_scrubs 2
It doesn't sound like you had this warning before, so I assume it will eventually clear. If not, you can check out the docs [0] and my recent blog post [1] about this topic.
Regards, Eugen
[0]
https://docs.ceph.com/en/latest/rados/operations/health-checks/#pg-not-deep-... [1]
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-tim...
Zitat von Götz Reinicke <goetz.reinicke@filmakademie.de>:
Hello Ceph Community,
My cluster was hit by a power outage some month ago. Luckily no data was destroyed and powering up the nodes and services went well.
But till than some pgs are still shown as not scrubbed in time. Googling and searching the list showed some debugging hints like „ceph pg deep-scrub“ the pgs or restarting osd deamons.
Nothing „solved“ that issue here. I’m on ceph version 18.2.4 now.
Is there anything special what I can do to have thous pgs scrubbed? I like having the cluster health state ok not warning :) Or will time solve the problem when the pgs are in there regular cycle for being scrubbed again?
Thanks for hints and suggestion . Best regards Götz
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, you might want to take a look here: https://github.com/frans42/ceph-goodies/blob/main/doc/TuningScrub.md Don't set max_scrubs > 1 on HDD OSDs, you will almost certainly regret it like I did. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Joachim Kraftmayer <joachim.kraftmayer@clyso.com> Sent: Wednesday, October 23, 2024 10:49 AM To: Eugen Block Cc: ceph-users@ceph.io Subject: [ceph-users] Re: pgs not deep-scrubbed in time and pgs not scrubbed in time Hi Götz, we have solved several scenarios like this in the past. Do you recognize a trend as to whether the pg not scrub is decreasing, remaining the same or increasing? There are a few osd configuration parameters that can be used to set the scrubbing behavior of the cluster: https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/ https://docs.clyso.com/blog/ceph-blocked-requests-in-the-cluster-caused-by-d... Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Am Mi., 23. Okt. 2024 um 09:37 Uhr schrieb Eugen Block <eblock@nde.ag>:
Hi Götz,
usually, OSDs start (deep-)scrubbing PGs after they have been powered on. You should see PGs in (deep-)scrubbing state right now. Depending on your PG sizes, number of OSDs etc., that can take some time, of course. But eventually the number should decrease over time. If you have the defaults for deep_scrub_interval (1 week) and your cluster hasn't complained before, you'll probably get rid of the warning within a week or so. If you want to speed things up and your cluster can handle the load, you could temporarily increase osd_max_scrubs (max concurrent scrubs on a single OSD, default 1, can be updated at runtime):
ceph config set osd osd_max_scrubs 2
It doesn't sound like you had this warning before, so I assume it will eventually clear. If not, you can check out the docs [0] and my recent blog post [1] about this topic.
Regards, Eugen
[0]
https://docs.ceph.com/en/latest/rados/operations/health-checks/#pg-not-deep-... [1]
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-tim...
Zitat von Götz Reinicke <goetz.reinicke@filmakademie.de>:
Hello Ceph Community,
My cluster was hit by a power outage some month ago. Luckily no data was destroyed and powering up the nodes and services went well.
But till than some pgs are still shown as not scrubbed in time. Googling and searching the list showed some debugging hints like „ceph pg deep-scrub“ the pgs or restarting osd deamons.
Nothing „solved“ that issue here. I’m on ceph version 18.2.4 now.
Is there anything special what I can do to have thous pgs scrubbed? I like having the cluster health state ok not warning :) Or will time solve the problem when the pgs are in there regular cycle for being scrubbed again?
Thanks for hints and suggestion . Best regards Götz
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That is super useful. Thank you so much for sharing! :) Kevin ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, October 25, 2024 8:03 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: pgs not deep-scrubbed in time and pgs not scrubbed in time Check twice before you click! This email originated from outside PNNL. Hi, you might want to take a look here: https://github.com/frans42/ceph-goodies/blob/main/doc/TuningScrub.md Don't set max_scrubs > 1 on HDD OSDs, you will almost certainly regret it like I did. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Joachim Kraftmayer <joachim.kraftmayer@clyso.com> Sent: Wednesday, October 23, 2024 10:49 AM To: Eugen Block Cc: ceph-users@ceph.io Subject: [ceph-users] Re: pgs not deep-scrubbed in time and pgs not scrubbed in time Hi Götz, we have solved several scenarios like this in the past. Do you recognize a trend as to whether the pg not scrub is decreasing, remaining the same or increasing? There are a few osd configuration parameters that can be used to set the scrubbing behavior of the cluster: https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/ https://docs.clyso.com/blog/ceph-blocked-requests-in-the-cluster-caused-by-d... Regards, Joachim joachim.kraftmayer@clyso.com http://www.clyso.com/ Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Am Mi., 23. Okt. 2024 um 09:37 Uhr schrieb Eugen Block <eblock@nde.ag>:
Hi Götz,
usually, OSDs start (deep-)scrubbing PGs after they have been powered on. You should see PGs in (deep-)scrubbing state right now. Depending on your PG sizes, number of OSDs etc., that can take some time, of course. But eventually the number should decrease over time. If you have the defaults for deep_scrub_interval (1 week) and your cluster hasn't complained before, you'll probably get rid of the warning within a week or so. If you want to speed things up and your cluster can handle the load, you could temporarily increase osd_max_scrubs (max concurrent scrubs on a single OSD, default 1, can be updated at runtime):
ceph config set osd osd_max_scrubs 2
It doesn't sound like you had this warning before, so I assume it will eventually clear. If not, you can check out the docs [0] and my recent blog post [1] about this topic.
Regards, Eugen
[0]
https://docs.ceph.com/en/latest/rados/operations/health-checks/#pg-not-deep-... [1]
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-tim...
Zitat von Götz Reinicke <goetz.reinicke@filmakademie.de>:
Hello Ceph Community,
My cluster was hit by a power outage some month ago. Luckily no data was destroyed and powering up the nodes and services went well.
But till than some pgs are still shown as not scrubbed in time. Googling and searching the list showed some debugging hints like „ceph pg deep-scrub“ the pgs or restarting osd deamons.
Nothing „solved“ that issue here. I’m on ceph version 18.2.4 now.
Is there anything special what I can do to have thous pgs scrubbed? I like having the cluster health state ok not warning :) Or will time solve the problem when the pgs are in there regular cycle for being scrubbed again?
Thanks for hints and suggestion . Best regards Götz
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Eugen, dear Joachim, Thanks for your feedback and input. The amount of PGs stuck is +-1 the same around 30 in total. The most PGs are located on the same OSD from what I see in the detail. And most of them are listed in scrub and deep-scrub. Most are from not scrubbed since end of August … I’ll look into your links and see what might help. Thanks once more and regards . Götz
Am 23.10.2024 um 08:35 schrieb Götz Reinicke <goetz.reinicke@filmakademie.de>:
Hello Ceph Community,
My cluster was hit by a power outage some month ago. Luckily no data was destroyed and powering up the nodes and services went well.
But till than some pgs are still shown as not scrubbed in time. Googling and searching the list showed some debugging hints like „ceph pg deep-scrub“ the pgs or restarting osd deamons.
Nothing „solved“ that issue here. I’m on ceph version 18.2.4 now.
Is there anything special what I can do to have thous pgs scrubbed? I like having the cluster health state ok not warning :) Or will time solve the problem when the pgs are in there regular cycle for being scrubbed again?
Thanks for hints and suggestion . Best regards Götz
Most are from not scrubbed since end of August …
That is lucky! On an inherited Ceph instance I had got most of them unscrubbed for 1-2 years. :-) The usual reason for delays in scrubbing are insufficient IOPS (both) and even insufficient bandwidth (deep scrubbing). Scrubbing like balancing and replicating etc. are very heavy background "admin" operations that consume a lot of IOPS (and bandwidth) and many Ceph instances that I have seen can barely cope with the user workload so the background "admin" workload gets very much delayed as a result. One useful check is to increase the priority of background "admin" operations so they get done without delay and if users start complaining that is indeed the situation :-).
participants (6)
-
Eugen Block
-
Fox, Kevin M
-
Frank Schilder
-
Götz Reinicke
-
Joachim Kraftmayer
-
pg＠ceph.list.sabi.co.UK