Hi, we are operating a Ceph cluster with mostly replicated pools on SSDs, but one of our pools is erasure-coded on 16TB HDDs. Two weeks ago, we upgraded from Reef to Squid (19.2.3). Since then, there has been an ever growing number of PGs that have not been deep-scrubbed or scrubbed in time. All these PGs are located on the HDD pool. I would like to increase the scrubbing and deep-scrubbing intervals for the HDDs, but not for the SSDs. Is there any way to do that other than setting it on a per-OSD base, such as per device class or per sub-tree of the crush map? Thanks in advance for any advice! Ciao, Sven
Hi Sven, you can set a different interval per pool: ceph osd pool set <POOL> deep_scrub_interval 1209600 I have some more insights about "PGs not deep-scrubbed in time" summarized here: https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-tim... Rergards, Eugen Zitat von Sven Grottke <sfs@math.tu-berlin.de>:
Hi,
we are operating a Ceph cluster with mostly replicated pools on SSDs, but one of our pools is erasure-coded on 16TB HDDs.
Two weeks ago, we upgraded from Reef to Squid (19.2.3). Since then, there has been an ever growing number of PGs that have not been deep-scrubbed or scrubbed in time. All these PGs are located on the HDD pool.
I would like to increase the scrubbing and deep-scrubbing intervals for the HDDs, but not for the SSDs. Is there any way to do that other than setting it on a per-OSD base, such as per device class or per sub-tree of the crush map?
Thanks in advance for any advice!
Ciao, Sven _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, thanks, that did the trick. Ciao, Sven On 10/13/25 13:36, Eugen Block wrote:
Hi Sven,
you can set a different interval per pool:
ceph osd pool set <POOL> deep_scrub_interval 1209600
I have some more insights about "PGs not deep-scrubbed in time" summarized here:
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed- in-time/
Rergards, Eugen
Zitat von Sven Grottke <sfs@math.tu-berlin.de>:
Hi,
we are operating a Ceph cluster with mostly replicated pools on SSDs, but one of our pools is erasure-coded on 16TB HDDs.
Two weeks ago, we upgraded from Reef to Squid (19.2.3). Since then, there has been an ever growing number of PGs that have not been deep- scrubbed or scrubbed in time. All these PGs are located on the HDD pool.
I would like to increase the scrubbing and deep-scrubbing intervals for the HDDs, but not for the SSDs. Is there any way to do that other than setting it on a per-OSD base, such as per device class or per sub-tree of the crush map?
Thanks in advance for any advice!
Ciao, Sven _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, 13 Oct 2025 12:27:27 +0200, Sven Grottke <sfs@math.tu-berlin.de> said:
[...] but one of our pools is erasure-coded on 16TB HDDs.
This is the beginning of many sad stories :-).
Two weeks ago, we upgraded from Reef to Squid (19.2.3) [...] an ever growing number of PGs that have not been deep-scrubbed or scrubbed in time. All these PGs are located on the HDD pool.
So far it is behaving as expected. Surprising that it did not happen before. Perhaps it started to become more fully used or to have more small files in that period of time.
I would like to increase the scrubbing and deep-scrubbing intervals for the HDDs, but not for the SSDs. Is there any way to do that other than setting it on a per-OSD base [...]
The expectation for a significant Ceph setup is that its servers are managed by something like Ansible, CfEngine, etc, where configuration updates are distributed to many OSDs by that kind of tool. To some extent "orchestration" and GUI frontends can supplement that, but for finer control some kind of CM engine is usually needed.
On 10/13/25 13:44, Peter Grandi wrote:
On Mon, 13 Oct 2025 12:27:27 +0200, Sven Grottke <sfs@math.tu-berlin.de> said:
[...] but one of our pools is erasure-coded on 16TB HDDs.
This is the beginning of many sad stories :-).
The result of an unfortunate disconnect between a customer's requirements and their funding.
Two weeks ago, we upgraded from Reef to Squid (19.2.3) [...] an ever growing number of PGs that have not been deep-scrubbed or scrubbed in time. All these PGs are located on the HDD pool.
So far it is behaving as expected. Surprising that it did not happen before. Perhaps it started to become more fully used or to have more small files in that period of time.
It has been gradually filling up, so I suspect it's the former.
I would like to increase the scrubbing and deep-scrubbing intervals for the HDDs, but not for the SSDs. Is there any way to do that other than setting it on a per-OSD base [...]
The expectation for a significant Ceph setup is that its servers are managed by something like Ansible, CfEngine, etc, where configuration updates are distributed to many OSDs by that kind of tool. To some extent "orchestration" and GUI frontends can supplement that, but for finer control some kind of CM engine is usually needed.
We've been burnt in the past handing Ceph tuning over to Ansible. The solution Eugen suggested to set it on a per-pool base worked for us. Ciao, Sven
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The result of an unfortunate disconnect between a customer's requirements and their funding.
Moreover, HDDs aren't necessarily even less expensive: https://www.snia.org/forums/cmsi/programs/TCOcalc * If your chassis vendor pushes an expensive, flaky RAID HBA on you * If you pay inflated chassis vendor prices for drives * If you don't have free DC racks and RUs
participants (4)
-
Anthony D'Atri
-
Eugen Block
-
pg＠ceph.list.sabi.co.UK
-
Sven Grottke