That is super useful. Thank you so much for sharing! :) Kevin ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, October 25, 2024 8:03 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: pgs not deep-scrubbed in time and pgs not scrubbed in time Check twice before you click! This email originated from outside PNNL. Hi, you might want to take a look here: https://github.com/frans42/ceph-goodies/blob/main/doc/TuningScrub.md Don't set max_scrubs > 1 on HDD OSDs, you will almost certainly regret it like I did. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Joachim Kraftmayer <joachim.kraftmayer@clyso.com> Sent: Wednesday, October 23, 2024 10:49 AM To: Eugen Block Cc: ceph-users@ceph.io Subject: [ceph-users] Re: pgs not deep-scrubbed in time and pgs not scrubbed in time Hi Götz, we have solved several scenarios like this in the past. Do you recognize a trend as to whether the pg not scrub is decreasing, remaining the same or increasing? There are a few osd configuration parameters that can be used to set the scrubbing behavior of the cluster: https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/ https://docs.clyso.com/blog/ceph-blocked-requests-in-the-cluster-caused-by-d... Regards, Joachim joachim.kraftmayer@clyso.com http://www.clyso.com/ Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Am Mi., 23. Okt. 2024 um 09:37 Uhr schrieb Eugen Block <eblock@nde.ag>:
Hi Götz,
usually, OSDs start (deep-)scrubbing PGs after they have been powered on. You should see PGs in (deep-)scrubbing state right now. Depending on your PG sizes, number of OSDs etc., that can take some time, of course. But eventually the number should decrease over time. If you have the defaults for deep_scrub_interval (1 week) and your cluster hasn't complained before, you'll probably get rid of the warning within a week or so. If you want to speed things up and your cluster can handle the load, you could temporarily increase osd_max_scrubs (max concurrent scrubs on a single OSD, default 1, can be updated at runtime):
ceph config set osd osd_max_scrubs 2
It doesn't sound like you had this warning before, so I assume it will eventually clear. If not, you can check out the docs [0] and my recent blog post [1] about this topic.
Regards, Eugen
[0]
https://docs.ceph.com/en/latest/rados/operations/health-checks/#pg-not-deep-... [1]
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-tim...
Zitat von Götz Reinicke <goetz.reinicke@filmakademie.de>:
Hello Ceph Community,
My cluster was hit by a power outage some month ago. Luckily no data was destroyed and powering up the nodes and services went well.
But till than some pgs are still shown as not scrubbed in time. Googling and searching the list showed some debugging hints like „ceph pg deep-scrub“ the pgs or restarting osd deamons.
Nothing „solved“ that issue here. I’m on ceph version 18.2.4 now.
Is there anything special what I can do to have thous pgs scrubbed? I like having the cluster health state ok not warning :) Or will time solve the problem when the pgs are in there regular cycle for being scrubbed again?
Thanks for hints and suggestion . Best regards Götz
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io