15 Nov
2023
15 Nov
'23
10:14 a.m.
Hi folks,
I am fighting a bit with odd deep-scrub behavior on HDDs and discovered a likely cause of
why the distribution of last_deep_scrub_stamps is so weird. I wrote a small script to
extract a histogram of scrubs by "days not scrubbed" (more precisely, intervals
not scrubbed; see code) to find out how (deep-) scrub times are distributed. Output
below.
What I expected is along the lines that HDD-OSDs try to scrub every 1-3 days, while they
try to deep-scrub every 7-14 days. In other words, OSDs that have been deep-scrubbed
within the last 7 days would *never* be in scrubbing+deep state. However, what I see is
completely different. There seems to be no distinction between scrub- and deep-scrub start
times. This is really unexpected as nobody would try to deep-scrub HDDs every day. Weekly
to bi-weekly is normal, specifically for large drives.
Is there a way to configure something like osd_deep_scrub_min_interval (no, I don't
want to run cron jobs for scrubbing yet)? In the output below, I would like to be able to
configure a minimum period of 1-2 weeks before the next deep-scrub happens. How can I do
that?
The observed behavior is very unusual for RAID systems (if its not a bug in the report
script). With this behavior its not surprising that people complain about "not
deep-scrubbed in time" messages and too high deep-scrub IO load when such a large
percentage of OSDs is needlessly deep-scrubbed after 1-6 days again already.
Sample output:
# scrub-report
dumped pgs
Scrub report:
4121 PGs not scrubbed since 1 intervals (6h)
3831 PGs not scrubbed since 2 intervals (6h)
4012 PGs not scrubbed since 3 intervals (6h)
3986 PGs not scrubbed since 4 intervals (6h)
2998 PGs not scrubbed since 5 intervals (6h)
1488 PGs not scrubbed since 6 intervals (6h)
909 PGs not scrubbed since 7 intervals (6h)
771 PGs not scrubbed since 8 intervals (6h)
582 PGs not scrubbed since 9 intervals (6h) 2 scrubbing
431 PGs not scrubbed since 10 intervals (6h)
333 PGs not scrubbed since 11 intervals (6h) 1 scrubbing
265 PGs not scrubbed since 12 intervals (6h)
195 PGs not scrubbed since 13 intervals (6h)
116 PGs not scrubbed since 14 intervals (6h)
78 PGs not scrubbed since 15 intervals (6h) 1 scrubbing
72 PGs not scrubbed since 16 intervals (6h)
37 PGs not scrubbed since 17 intervals (6h)
5 PGs not scrubbed since 18 intervals (6h) 14.237* 19.5cd* 19.12cc* 19.1233*
14.40e*
33 PGs not scrubbed since 20 intervals (6h)
23 PGs not scrubbed since 21 intervals (6h)
16 PGs not scrubbed since 22 intervals (6h)
12 PGs not scrubbed since 23 intervals (6h)
8 PGs not scrubbed since 24 intervals (6h)
2 PGs not scrubbed since 25 intervals (6h) 19.eef* 19.bb3*
4 PGs not scrubbed since 26 intervals (6h) 19.b4c* 19.10b8* 19.f13* 14.1ed*
5 PGs not scrubbed since 27 intervals (6h) 19.43f* 19.231* 19.1dbe* 19.1788*
19.16c0*
6 PGs not scrubbed since 28 intervals (6h)
2 PGs not scrubbed since 30 intervals (6h) 19.10f6* 14.9d*
3 PGs not scrubbed since 31 intervals (6h) 19.1322* 19.1318* 8.a*
1 PGs not scrubbed since 32 intervals (6h) 19.133f*
1 PGs not scrubbed since 33 intervals (6h) 19.1103*
3 PGs not scrubbed since 36 intervals (6h) 19.19cc* 19.12f4* 19.248*
1 PGs not scrubbed since 39 intervals (6h) 19.1984*
1 PGs not scrubbed since 41 intervals (6h) 14.449*
1 PGs not scrubbed since 44 intervals (6h) 19.179f*
Deep-scrub report:
3723 PGs not deep-scrubbed since 1 intervals (24h)
4621 PGs not deep-scrubbed since 2 intervals (24h) 8 scrubbing+deep
3588 PGs not deep-scrubbed since 3 intervals (24h) 8 scrubbing+deep
2929 PGs not deep-scrubbed since 4 intervals (24h) 3 scrubbing+deep
1705 PGs not deep-scrubbed since 5 intervals (24h) 4 scrubbing+deep
1904 PGs not deep-scrubbed since 6 intervals (24h) 5 scrubbing+deep
1540 PGs not deep-scrubbed since 7 intervals (24h) 7 scrubbing+deep
1304 PGs not deep-scrubbed since 8 intervals (24h) 7 scrubbing+deep
923 PGs not deep-scrubbed since 9 intervals (24h) 5 scrubbing+deep
557 PGs not deep-scrubbed since 10 intervals (24h) 7 scrubbing+deep
501 PGs not deep-scrubbed since 11 intervals (24h) 2 scrubbing+deep
363 PGs not deep-scrubbed since 12 intervals (24h) 2 scrubbing+deep
377 PGs not deep-scrubbed since 13 intervals (24h) 1 scrubbing+deep
383 PGs not deep-scrubbed since 14 intervals (24h) 2 scrubbing+deep
252 PGs not deep-scrubbed since 15 intervals (24h) 2 scrubbing+deep
116 PGs not deep-scrubbed since 16 intervals (24h) 5 scrubbing+deep
47 PGs not deep-scrubbed since 17 intervals (24h) 2 scrubbing+deep
10 PGs not deep-scrubbed since 18 intervals (24h)
2 PGs not deep-scrubbed since 19 intervals (24h) 19.1c6c* 19.a01*
1 PGs not deep-scrubbed since 20 intervals (24h) 14.1ed*
2 PGs not deep-scrubbed since 21 intervals (24h) 19.1322* 19.10f6*
1 PGs not deep-scrubbed since 23 intervals (24h) 19.19cc*
1 PGs not deep-scrubbed since 24 intervals (24h) 19.179f*
PGs marked with a * are on busy OSDs and not eligible for scrubbing.
The script (pasted here because attaching doesn't work):
# cat bin/scrub-report
#!/bin/bash
# Compute last scrub interval count. Scrub interval 6h, deep-scrub interval 24h.
# Print how many PGs have not been (deep-)scrubbed since #intervals.
ceph -f json pg dump pgs 2>&1 > /root/.cache/ceph/pgs_dump.json
echo ""
T0="$(date +%s)"
scrub_info="$(jq --arg T0 "$T0" -rc '.pg_stats[] | [
.pgid,
(.last_scrub_stamp[:19]+"Z" | (($T0|tonumber) -
fromdateiso8601)/(60*60*6)|ceil),
(.last_deep_scrub_stamp[:19]+"Z" | (($T0|tonumber) -
fromdateiso8601)/(60*60*24)|ceil),
.state,
(.acting | join(" "))
] | @tsv
' /root/.cache/ceph/pgs_dump.json)"
# less <<<"$scrub_info"
# 1 2 3 4 5..NF
# pg_id scrub-ints deep-scrub-ints status acting[]
awk <<<"$scrub_info" '{
for(i=5; i<=NF; ++i) pg_osds[$1]=pg_osds[$1] " " $i
if($4 == "active+clean") {
si_mx=si_mx<$2 ? $2 : si_mx
dsi_mx=dsi_mx<$3 ? $3 : dsi_mx
pg_sn[$2]++
pg_sn_ids[$2]=pg_sn_ids[$2] " " $1
pg_dsn[$3]++
pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1
} else if($4 ~ /scrubbing\+deep/) {
deep_scrubbing[$3]++
for(i=5; i<=NF; ++i) osd[$i]="busy"
} else if($4 ~ /scrubbing/) {
scrubbing[$2]++
for(i=5; i<=NF; ++i) osd[$i]="busy"
} else {
unclean[$2]++
unclean_d[$3]++
si_mx=si_mx<$2 ? $2 : si_mx
dsi_mx=dsi_mx<$3 ? $3 : dsi_mx
pg_sn[$2]++
pg_sn_ids[$2]=pg_sn_ids[$2] " " $1
pg_dsn[$3]++
pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1
for(i=5; i<=NF; ++i) osd[$i]="busy"
}
}
END {
print "Scrub report:"
for(si=1; si<=si_mx; ++si) {
if(pg_sn[si]==0 && scrubbing[si]==0 && unclean[si]==0) continue;
printf("%7d PGs not scrubbed since %2d intervals (6h)", pg_sn[si], si)
if(scrubbing[si]) printf(" %d scrubbing", scrubbing[si])
if(unclean[si]) printf(" %d unclean", unclean[si])
if(pg_sn[si]<=5) {
split(pg_sn_ids[si], pgs)
osds_busy=0
for(pg in pgs) {
split(pg_osds[pgs[pg]], osds)
for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1
if(osds_busy) printf(" %s*", pgs[pg])
if(!osds_busy) printf(" %s", pgs[pg])
}
}
printf("\n")
}
print ""
print "Deep-scrub report:"
for(dsi=1; dsi<=dsi_mx; ++dsi) {
if(pg_dsn[dsi]==0 && deep_scrubbing[dsi]==0 && unclean_d[dsi]==0)
continue;
printf("%7d PGs not deep-scrubbed since %2d intervals (24h)", pg_dsn[dsi],
dsi)
if(deep_scrubbing[dsi]) printf(" %d scrubbing+deep", deep_scrubbing[dsi])
if(unclean_d[dsi]) printf(" %d unclean", unclean_d[dsi])
if(pg_dsn[dsi]<=5) {
split(pg_dsn_ids[dsi], pgs)
osds_busy=0
for(pg in pgs) {
split(pg_osds[pgs[pg]], osds)
for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1
if(osds_busy) printf(" %s*", pgs[pg])
if(!osds_busy) printf(" %s", pgs[pg])
}
}
printf("\n")
}
print ""
print "PGs marked with a * are on busy OSDs and not eligible for scrubbing."
}
'
Don't forget the last "'" when copy-pasting.
Thanks for any pointers.
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
16 Nov
16 Nov
noon
Partially answering my own question. I think it is possible to tweak the existing
parameters to achieve what I'm looking for on average. The main reason I want to use
the internal scheduler is the high number of PGs on some pools, which I actually intend to
increase even further. For such pools a simple calculation shows that it is impractical to
do manual scrubbing with cron, I simply cannot execute cron jobs often enough to achieve a
reasonable scrub distribution (looking at a script like
https://gist.github.com/ethaniel/5db696d9c78516308b235b0cb904e4ad).
Looking at scrub date stamp distributions for specific pools, PGs with old deep-scrub time
stamps tend to correlate also with PGs with old scrub date stamps. The idea now id to
tweak scrub_min_interval such that the scrub scheduler is forced to select PGs out of the
20-30% with the oldest scrub stamps. This should imply that, after a reasonable time
interval, the age of the oldest deep-scrub stamps is reduced as long not deep-scrubbed PGs
become much more likely to be scheduled for deep-scrub.
This adjustment is done together with making osd_deep_scrub_randomize_ratio and
osd_scrub_backoff_ratio more aggressive to cycle more frequently through the small list of
PGs eligible for scrubbing. This is a bit like the reverse calculation for achieving the
effect of a not implemented deep_scrub_min_interval.
I made the following global changes (and hope the parameters do something like what their
documentation says):
global advanced osd_deep_scrub_randomize_ratio 0.330000
global dev osd_scrub_backoff_ratio 0.500000
With this setting, about 33% of all scrub should be deep-scrubs, meaning that on average
after 3 scrub events a PG is also deep-scrubbed. This leads to this estimate for expected
deep-scrub intervals:
given scrub-interval is scrub_min_interval*[1, 1+osd_scrub_interval_randomize_ratio] =
scrub_min_interval*[1, 1.5], the expected deep-scrub-interval is (assuming worst-case
realisation of randomize_ratio for upper value):
scrub_min_interval*[1, 1.5]*3 = scrub_min_interval*[3, 4.5]
I can't calculate how the tail will look like, but I hope its not a fat tail. I will
report back what I observe; see below.
The idea now is to tune scrub_min_interval per pool such that only about 20-30% of PGs
have a scrub-stamp older than scrub_min_interval. The scheduler will cycle only through
these and a bit faster than default. As the stamp histograms included below indicate, the
distribution is probably very sensitive to changes of this interval. I now changed these
values on some pools and already see that PGs with much older deep-scrub stamps are now
selected for deep-scrubbing. I will observe what these settings converge to and report
back. It seems that it will lead to an improved stamp distribution and one only needs to
issue manual deep-scrubs for very few PGs that are outliers of the random number generator
(that's the tail I talked about above). My goal is to have a script schedule a
deep-scrub on the outliers no more often than daily.
The reports below have been pulled after changes to settings were applied for about 2-3h.
There was already improvement in the right direction, but the original distribution issue
is still very pronounced.
Here two per-pool scrub stamp distributions for an SSD pool and an HDD pool, both with
large number of PGs per OSD:
=== SSD pool:
Scrub info for pool sr-rbd-data-one (id=2): dumped pgs
Scrub report:
22% 941 PGs not scrubbed since 1 intervals ( 6h)
42% 795 PGs not scrubbed since 2 intervals ( 12h)
62% 829 PGs not scrubbed since 3 intervals ( 18h)
82% 823 PGs not scrubbed since 4 intervals ( 24h)
96% 576 PGs not scrubbed since 5 intervals ( 30h)
100% 132 PGs not scrubbed since 6 intervals ( 36h)
4096 PGs out of 4096 reported, 0 missing.
Deep-scrub report:
13% 545 PGs not deep-scrubbed since 1 intervals ( 24h)
25% 508 PGs not deep-scrubbed since 2 intervals ( 48h) 1 scrubbing+deep
45% 797 PGs not deep-scrubbed since 3 intervals ( 72h) 1 scrubbing+deep
59% 587 PGs not deep-scrubbed since 4 intervals ( 96h)
70% 463 PGs not deep-scrubbed since 5 intervals (120h)
78% 312 PGs not deep-scrubbed since 6 intervals (144h)
84% 263 PGs not deep-scrubbed since 7 intervals (168h)
89% 173 PGs not deep-scrubbed since 8 intervals (192h)
92% 151 PGs not deep-scrubbed since 9 intervals (216h)
95% 106 PGs not deep-scrubbed since 10 intervals (240h)
96% 55 PGs not deep-scrubbed since 11 intervals (264h)
97% 50 PGs not deep-scrubbed since 12 intervals (288h)
98% 44 PGs not deep-scrubbed since 13 intervals (312h)
99% 24 PGs not deep-scrubbed since 14 intervals (336h)
100% 18 PGs not deep-scrubbed since 15 intervals (360h)
4096 PGs out of 4096 reported, 0 missing.
PGs marked with a * are on busy OSDs and not eligible for scrubbing.
sr-rbd-data-one scrub_min_interval=0h
sr-rbd-data-one scrub_max_interval=0h
sr-rbd-data-one deep_scrub_interval=0h
===
Here we see that after 24h 82% of PGs are scrubbed, but we have quite a tail of not
deep-scrubbed PGs. Its long enough to trigger a warning with default parameters. In this
case, reducing scrub_min_interval to a value around 18-20h could reduce the tail enough.
The alternative is simply to schedule a deep-scrub on the oldest PGs manually (cron). This
would start immediately since no OSDs are allocated to scrubbing/recovery.
=== HDD pool:
Scrub info for pool con-fs2-data2 (id=19): dumped pgs
Scrub report:
11% 939 PGs not scrubbed since 1 intervals ( 6h)
22% 936 PGs not scrubbed since 2 intervals ( 12h)
33% 874 PGs not scrubbed since 3 intervals ( 18h)
43% 821 PGs not scrubbed since 4 intervals ( 24h)
54% 931 PGs not scrubbed since 5 intervals ( 30h)
64% 766 PGs not scrubbed since 6 intervals ( 36h)
72% 646 PGs not scrubbed since 7 intervals ( 42h)
79% 559 PGs not scrubbed since 8 intervals ( 48h)
84% 411 PGs not scrubbed since 9 intervals ( 54h)
88% 346 PGs not scrubbed since 10 intervals ( 60h)
90% 222 PGs not scrubbed since 11 intervals ( 66h)
93% 213 PGs not scrubbed since 12 intervals ( 72h)
95% 160 PGs not scrubbed since 13 intervals ( 78h)
96% 87 PGs not scrubbed since 14 intervals ( 84h)
97% 77 PGs not scrubbed since 15 intervals ( 90h)
98% 57 PGs not scrubbed since 16 intervals ( 96h) 1 scrubbing
98% 42 PGs not scrubbed since 17 intervals (102h)
99% 32 PGs not scrubbed since 18 intervals (108h)
99% 19 PGs not scrubbed since 19 intervals (114h)
99% 19 PGs not scrubbed since 20 intervals (120h)
99% 10 PGs not scrubbed since 21 intervals (126h)
99% 1 PGs not scrubbed since 22 intervals (132h) 19.165f*
99% 5 PGs not scrubbed since 24 intervals (138h) 19.412* 19.75c* 19.140f* 19.134c*
19.fb7*
99% 5 PGs not scrubbed since 25 intervals (144h) 19.1714* 19.148d* 19.1fa9*
19.1f05* 19.1cda*
99% 1 PGs not scrubbed since 26 intervals (150h) 19.a3f*
99% 1 PGs not scrubbed since 27 intervals (156h) 19.a01*
99% 3 PGs not scrubbed since 28 intervals (162h) 19.12f2* 19.1284* 19.c90*
99% 1 PGs not scrubbed since 29 intervals (168h)
99% 1 PGs not scrubbed since 30 intervals (174h) 19.f13*
99% 2 PGs not scrubbed since 32 intervals (180h) 19.1f87* 19.67b*
99% 2 PGs not scrubbed since 36 intervals (186h) 19.133f* 19.1318*
99% 2 PGs not scrubbed since 40 intervals (192h) 19.12f4* 19.248*
100% 1 PGs not scrubbed since 43 intervals (198h) 19.1984*
8192 PGs out of 8192 reported, 0 missing.
Deep-scrub report:
14% 1210 PGs not deep-scrubbed since 1 intervals ( 24h)
28% 1136 PGs not deep-scrubbed since 2 intervals ( 48h) 1 scrubbing+deep
40% 985 PGs not deep-scrubbed since 3 intervals ( 72h) 4 scrubbing+deep
51% 851 PGs not deep-scrubbed since 4 intervals ( 96h) 5 scrubbing+deep
59% 713 PGs not deep-scrubbed since 5 intervals (120h) 4 scrubbing+deep
63% 276 PGs not deep-scrubbed since 6 intervals (144h) 2 scrubbing+deep
70% 566 PGs not deep-scrubbed since 7 intervals (168h) 1 scrubbing+deep
76% 534 PGs not deep-scrubbed since 8 intervals (192h) 2 scrubbing+deep
82% 480 PGs not deep-scrubbed since 9 intervals (216h) 2 scrubbing+deep
87% 381 PGs not deep-scrubbed since 10 intervals (240h) 2 scrubbing+deep
90% 253 PGs not deep-scrubbed since 11 intervals (264h) 1 scrubbing+deep
92% 222 PGs not deep-scrubbed since 12 intervals (288h) 1 scrubbing+deep
94% 136 PGs not deep-scrubbed since 13 intervals (312h) 1 scrubbing+deep
96% 179 PGs not deep-scrubbed since 14 intervals (336h) 3 scrubbing+deep
98% 156 PGs not deep-scrubbed since 15 intervals (360h) 6 scrubbing+deep
99% 65 PGs not deep-scrubbed since 16 intervals (384h) 3 scrubbing+deep
99% 31 PGs not deep-scrubbed since 17 intervals (408h) 4 scrubbing+deep
99% 14 PGs not deep-scrubbed since 18 intervals (432h) 3 scrubbing+deep
99% 3 PGs not deep-scrubbed since 19 intervals (456h) 19.1d89* 19.fb7* 19.807*
100% 1 PGs not deep-scrubbed since 21 intervals (480h) 19.a01*
8192 PGs out of 8192 reported, 0 missing.
PGs marked with a * are on busy OSDs and not eligible for scrubbing.
con-fs2-data2 scrub_min_interval=42h
con-fs2-data2 scrub_max_interval=0h
con-fs2-data2 deep_scrub_interval=0h
===
This is the real deal, the pool I'm fighting with at the moment. I made a small change
in scrub_min_interval (pool setting) from 24h to 42h, which resulted in the very good
deep-scrub state allocation of the PGs in the pool. With scrub_min_interval=24h basically
all scrubbing happened on PGs not deep-scrubbed within 1-6 days. After increasing this
value to the time interval for which about 70% of PGs were scrubbed (leaving 30%
eligible), the allocation of deep-scrub states is much much better. I expect both tails to
get shorter and the overall deep-scrub load to go down as well. I hope to reach a state
where I only need to issue a few deep-scrubs manually per day to get everything scrubbed
within 1 week and deep-scrubbed within 3-4 weeks.
For now I will wait what effect the global settings have on the SSD pools and what the HDD
pool converges to. This will need 1-2 months observations and I will report back when
significant changes show up.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: Wednesday, November 15, 2023 11:14 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] How to configure something like osd_deep_scrub_min_interval?
Hi folks,
I am fighting a bit with odd deep-scrub behavior on HDDs and discovered a likely cause of
why the distribution of last_deep_scrub_stamps is so weird. I wrote a small script to
extract a histogram of scrubs by "days not scrubbed" (more precisely, intervals
not scrubbed; see code) to find out how (deep-) scrub times are distributed. Output
below.
What I expected is along the lines that HDD-OSDs try to scrub every 1-3 days, while they
try to deep-scrub every 7-14 days. In other words, OSDs that have been deep-scrubbed
within the last 7 days would *never* be in scrubbing+deep state. However, what I see is
completely different. There seems to be no distinction between scrub- and deep-scrub start
times. This is really unexpected as nobody would try to deep-scrub HDDs every day. Weekly
to bi-weekly is normal, specifically for large drives.
Is there a way to configure something like osd_deep_scrub_min_interval (no, I don't
want to run cron jobs for scrubbing yet)? In the output below, I would like to be able to
configure a minimum period of 1-2 weeks before the next deep-scrub happens. How can I do
that?
The observed behavior is very unusual for RAID systems (if its not a bug in the report
script). With this behavior its not surprising that people complain about "not
deep-scrubbed in time" messages and too high deep-scrub IO load when such a large
percentage of OSDs is needlessly deep-scrubbed after 1-6 days again already.
Sample output:
# scrub-report
dumped pgs
Scrub report:
4121 PGs not scrubbed since 1 intervals (6h)
3831 PGs not scrubbed since 2 intervals (6h)
4012 PGs not scrubbed since 3 intervals (6h)
3986 PGs not scrubbed since 4 intervals (6h)
2998 PGs not scrubbed since 5 intervals (6h)
1488 PGs not scrubbed since 6 intervals (6h)
909 PGs not scrubbed since 7 intervals (6h)
771 PGs not scrubbed since 8 intervals (6h)
582 PGs not scrubbed since 9 intervals (6h) 2 scrubbing
431 PGs not scrubbed since 10 intervals (6h)
333 PGs not scrubbed since 11 intervals (6h) 1 scrubbing
265 PGs not scrubbed since 12 intervals (6h)
195 PGs not scrubbed since 13 intervals (6h)
116 PGs not scrubbed since 14 intervals (6h)
78 PGs not scrubbed since 15 intervals (6h) 1 scrubbing
72 PGs not scrubbed since 16 intervals (6h)
37 PGs not scrubbed since 17 intervals (6h)
5 PGs not scrubbed since 18 intervals (6h) 14.237* 19.5cd* 19.12cc* 19.1233*
14.40e*
33 PGs not scrubbed since 20 intervals (6h)
23 PGs not scrubbed since 21 intervals (6h)
16 PGs not scrubbed since 22 intervals (6h)
12 PGs not scrubbed since 23 intervals (6h)
8 PGs not scrubbed since 24 intervals (6h)
2 PGs not scrubbed since 25 intervals (6h) 19.eef* 19.bb3*
4 PGs not scrubbed since 26 intervals (6h) 19.b4c* 19.10b8* 19.f13* 14.1ed*
5 PGs not scrubbed since 27 intervals (6h) 19.43f* 19.231* 19.1dbe* 19.1788*
19.16c0*
6 PGs not scrubbed since 28 intervals (6h)
2 PGs not scrubbed since 30 intervals (6h) 19.10f6* 14.9d*
3 PGs not scrubbed since 31 intervals (6h) 19.1322* 19.1318* 8.a*
1 PGs not scrubbed since 32 intervals (6h) 19.133f*
1 PGs not scrubbed since 33 intervals (6h) 19.1103*
3 PGs not scrubbed since 36 intervals (6h) 19.19cc* 19.12f4* 19.248*
1 PGs not scrubbed since 39 intervals (6h) 19.1984*
1 PGs not scrubbed since 41 intervals (6h) 14.449*
1 PGs not scrubbed since 44 intervals (6h) 19.179f*
Deep-scrub report:
3723 PGs not deep-scrubbed since 1 intervals (24h)
4621 PGs not deep-scrubbed since 2 intervals (24h) 8 scrubbing+deep
3588 PGs not deep-scrubbed since 3 intervals (24h) 8 scrubbing+deep
2929 PGs not deep-scrubbed since 4 intervals (24h) 3 scrubbing+deep
1705 PGs not deep-scrubbed since 5 intervals (24h) 4 scrubbing+deep
1904 PGs not deep-scrubbed since 6 intervals (24h) 5 scrubbing+deep
1540 PGs not deep-scrubbed since 7 intervals (24h) 7 scrubbing+deep
1304 PGs not deep-scrubbed since 8 intervals (24h) 7 scrubbing+deep
923 PGs not deep-scrubbed since 9 intervals (24h) 5 scrubbing+deep
557 PGs not deep-scrubbed since 10 intervals (24h) 7 scrubbing+deep
501 PGs not deep-scrubbed since 11 intervals (24h) 2 scrubbing+deep
363 PGs not deep-scrubbed since 12 intervals (24h) 2 scrubbing+deep
377 PGs not deep-scrubbed since 13 intervals (24h) 1 scrubbing+deep
383 PGs not deep-scrubbed since 14 intervals (24h) 2 scrubbing+deep
252 PGs not deep-scrubbed since 15 intervals (24h) 2 scrubbing+deep
116 PGs not deep-scrubbed since 16 intervals (24h) 5 scrubbing+deep
47 PGs not deep-scrubbed since 17 intervals (24h) 2 scrubbing+deep
10 PGs not deep-scrubbed since 18 intervals (24h)
2 PGs not deep-scrubbed since 19 intervals (24h) 19.1c6c* 19.a01*
1 PGs not deep-scrubbed since 20 intervals (24h) 14.1ed*
2 PGs not deep-scrubbed since 21 intervals (24h) 19.1322* 19.10f6*
1 PGs not deep-scrubbed since 23 intervals (24h) 19.19cc*
1 PGs not deep-scrubbed since 24 intervals (24h) 19.179f*
PGs marked with a * are on busy OSDs and not eligible for scrubbing.
The script (pasted here because attaching doesn't work):
# cat bin/scrub-report
#!/bin/bash
# Compute last scrub interval count. Scrub interval 6h, deep-scrub interval 24h.
# Print how many PGs have not been (deep-)scrubbed since #intervals.
ceph -f json pg dump pgs 2>&1 > /root/.cache/ceph/pgs_dump.json
echo ""
T0="$(date +%s)"
scrub_info="$(jq --arg T0 "$T0" -rc '.pg_stats[] | [
.pgid,
(.last_scrub_stamp[:19]+"Z" | (($T0|tonumber) -
fromdateiso8601)/(60*60*6)|ceil),
(.last_deep_scrub_stamp[:19]+"Z" | (($T0|tonumber) -
fromdateiso8601)/(60*60*24)|ceil),
.state,
(.acting | join(" "))
] | @tsv
' /root/.cache/ceph/pgs_dump.json)"
# less <<<"$scrub_info"
# 1 2 3 4 5..NF
# pg_id scrub-ints deep-scrub-ints status acting[]
awk <<<"$scrub_info" '{
for(i=5; i<=NF; ++i) pg_osds[$1]=pg_osds[$1] " " $i
if($4 == "active+clean") {
si_mx=si_mx<$2 ? $2 : si_mx
dsi_mx=dsi_mx<$3 ? $3 : dsi_mx
pg_sn[$2]++
pg_sn_ids[$2]=pg_sn_ids[$2] " " $1
pg_dsn[$3]++
pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1
} else if($4 ~ /scrubbing\+deep/) {
deep_scrubbing[$3]++
for(i=5; i<=NF; ++i) osd[$i]="busy"
} else if($4 ~ /scrubbing/) {
scrubbing[$2]++
for(i=5; i<=NF; ++i) osd[$i]="busy"
} else {
unclean[$2]++
unclean_d[$3]++
si_mx=si_mx<$2 ? $2 : si_mx
dsi_mx=dsi_mx<$3 ? $3 : dsi_mx
pg_sn[$2]++
pg_sn_ids[$2]=pg_sn_ids[$2] " " $1
pg_dsn[$3]++
pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1
for(i=5; i<=NF; ++i) osd[$i]="busy"
}
}
END {
print "Scrub report:"
for(si=1; si<=si_mx; ++si) {
if(pg_sn[si]==0 && scrubbing[si]==0 && unclean[si]==0)
continue;
printf("%7d PGs not scrubbed since %2d intervals (6h)",
pg_sn[si], si)
if(scrubbing[si]) printf(" %d scrubbing", scrubbing[si])
if(unclean[si]) printf(" %d unclean", unclean[si])
if(pg_sn[si]<=5) {
split(pg_sn_ids[si], pgs)
osds_busy=0
for(pg in pgs) {
split(pg_osds[pgs[pg]], osds)
for(o in osds) if(osd[osds[o]]=="busy")
osds_busy=1
if(osds_busy) printf(" %s*", pgs[pg])
if(!osds_busy) printf(" %s", pgs[pg])
}
}
printf("\n")
}
print ""
print "Deep-scrub report:"
for(dsi=1; dsi<=dsi_mx; ++dsi) {
if(pg_dsn[dsi]==0 && deep_scrubbing[dsi]==0 &&
unclean_d[dsi]==0) continue;
printf("%7d PGs not deep-scrubbed since %2d intervals (24h)",
pg_dsn[dsi], dsi)
if(deep_scrubbing[dsi]) printf(" %d scrubbing+deep",
deep_scrubbing[dsi])
if(unclean_d[dsi]) printf(" %d unclean", unclean_d[dsi])
if(pg_dsn[dsi]<=5) {
split(pg_dsn_ids[dsi], pgs)
osds_busy=0
for(pg in pgs) {
split(pg_osds[pgs[pg]], osds)
for(o in osds) if(osd[osds[o]]=="busy")
osds_busy=1
if(osds_busy) printf(" %s*", pgs[pg])
if(!osds_busy) printf(" %s", pgs[pg])
}
}
printf("\n")
}
print ""
print "PGs marked with a * are on busy OSDs and not eligible for
scrubbing."
}
'
Don't forget the last "'" when copy-pasting.
Thanks for any pointers.
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12 Dec
12 Dec
12:33 p.m.
Hi all,
a little gem for Christmas. After going through the OSD code, scratching my had and doing
a bit of maths, I seem to have found a way to tune the built-in scrub machine to work
perfectly. Its only few knobs to turn, but its difficult to find out, because the
documentation is misleading to incorrect or even missing entirely. I plan to write a bit
more in the documentation to this script
(https://github.com/frans42/ceph-goodies/blob/5e2016f0b00f8dbc3e51c7e9904a73…),
so here only the executive summary.
- global, set osd_max_scrubs=1, higher values do not have any effect except making your
users angry
- global, set osd_deep_scrub_randomize_ratio=0, this parameter is unnecessary for
distributing deep scrubs and its only effect in the current implementation is to trigger a
large amount of too early deep-scrubs increasing the overall deep-scrub load significantly
without a useful effect
- on the pools, set deep_scrub_interval according to needs and performance, scrubs will
turn into deep-scrubs for every PG with a deep-scrub stamp older than deep_scrub_interval,
here you will need to do some calculations what your hardware can do and how much average
load you can tolerate
- on the pools, also set scrub_min_interval and scrub_max_interval so that its only one
place to look for these settings
- per OSD device class, set osd_scrub_interval_randomize_ratio such that scrubs start
within a reasonable window after scrub_min_interval. This parameter is very important for
distributing scrubs as evenly as possible over time. The default of 0.5 is good for most
cases. For the HDD pool used below I reduced it a bit, because scrub_min_interval is set
to 66h and here 0.5 leads to a slightly too large start-interval.
- per OSD device class, set osd_scrub_backoff_ratio to a value close to but not higher
than 1-1/(largest replication factor [=size] of pools on this device class). This
parameter is labelled dev, but is really important for effective scrub scheduling. OSDs
need to allocate scrub reservations and this process is extremely racy specifically for EC
pools with high replication factor. The default 0.66 probably has 3-times replicated pools
in mind, but it triggers way too many attempts to allocate scrub reservations for pools
with larger replication factor, causing dead-locks and blocking scrubs being executed even
if plenty of OSDs are idle. I found that 1-0.75/max_size_on_device_class works very well.
After having found out about what the parameters really do and adjusting them for my
pools, I passed through a valley of tears and arrived now at the beautiful distribution of
(deep-)scrub stamps for a pool on 16TB HDDs shown at the end. Everything gets scrubbed
every 3-4 days and deep-scrubs start no earlier than 14 days after the last deep-scrub.
The overall (deep)-scrub load is now half of what it was before the changes and I
don't have the dreaded "PGs not (deep-)scrubbed in time" warnings any more.
I calculated the (deep-)scrub time window configs such that about 30% of OSDs will be
continuously busy when the disks reach 70% utilization (currently ca. 45%). No user will
complain about that and there is enough spare performance left to catch up after
high-load- or recovery episodes without having to do anything.
Here is the scrub report generated for the pool I was looking at for weeks now, its
exactly as I wanted it and I don't have to run cron jobs, it just works:
# pool-scrub-report con-fs2-data2
Scrub info for pool con-fs2-data2 (id=19): dumped pgs
Scrub report:
6% 566 PGs not scrubbed since 1 intervals ( 6h)
13% 528 PGs not scrubbed since 2 intervals ( 12h)
21% 640 PGs not scrubbed since 3 intervals ( 18h)
29% 668 PGs not scrubbed since 4 intervals ( 24h)
37% 677 PGs not scrubbed since 5 intervals ( 30h)
46% 729 PGs not scrubbed since 6 intervals ( 36h)
54% 631 PGs not scrubbed since 7 intervals ( 42h)
62% 662 PGs not scrubbed since 8 intervals ( 48h)
70% 663 PGs not scrubbed since 9 intervals ( 54h)
78% 660 PGs not scrubbed since 10 intervals ( 60h)
85% 571 PGs not scrubbed since 11 intervals ( 66h)
92% 585 PGs not scrubbed since 12 intervals ( 72h) [74 idle] 1 scrubbing
96% 358 PGs not scrubbed since 13 intervals ( 78h) [34 idle] [3 scrubbing+deep] 2
scrubbing
99% 181 PGs not scrubbed since 14 intervals ( 84h) [23 idle] [3 scrubbing+deep]
99% 70 PGs not scrubbed since 15 intervals ( 90h) [9 idle] 1 scrubbing
100% 3 PGs not scrubbed since 16 intervals ( 96h) [1 scrubbing+deep]
8192 PGs out of 8192 reported, 0 missing, 4 scrubbing, 140 idle, 0 unclean.
Deep-scrub report:
3% 295 PGs not deep-scrubbed since 1 intervals ( 24h)
9% 461 PGs not deep-scrubbed since 2 intervals ( 48h)
16% 558 PGs not deep-scrubbed since 3 intervals ( 72h)
23% 613 PGs not deep-scrubbed since 4 intervals ( 96h) [1 scrubbing]
31% 619 PGs not deep-scrubbed since 5 intervals (120h)
39% 660 PGs not deep-scrubbed since 6 intervals (144h)
47% 726 PGs not deep-scrubbed since 7 intervals (168h) [1 scrubbing]
57% 743 PGs not deep-scrubbed since 8 intervals (192h)
65% 727 PGs not deep-scrubbed since 9 intervals (216h) [1 scrubbing]
73% 656 PGs not deep-scrubbed since 10 intervals (240h)
75% 107 PGs not deep-scrubbed since 11 intervals (264h)
82% 626 PGs not deep-scrubbed since 12 intervals (288h)
90% 588 PGs not deep-scrubbed since 13 intervals (312h)
94% 388 PGs not deep-scrubbed since 14 intervals (336h) [1 scrubbing]
96% 129 PGs not deep-scrubbed since 15 intervals (360h) 2 scrubbing+deep
98% 207 PGs not deep-scrubbed since 16 intervals (384h) 2 scrubbing+deep
99% 79 PGs not deep-scrubbed since 17 intervals (408h) 1 scrubbing+deep
100% 10 PGs not deep-scrubbed since 18 intervals (432h) 2 scrubbing+deep
8192 PGs out of 8192 reported, 0 missing, 7 scrubbing+deep, 0 unclean.
con-fs2-data2 scrub_min_interval=66h (11i/84%/625PGs÷i)
con-fs2-data2 scrub_max_interval=168h (7d)
con-fs2-data2 deep_scrub_interval=336h (14d/~89%/~520PGs÷d)
osd.338 osd_scrub_interval_randomize_ratio=0.363636 scrubs start after: 66h..90h
osd.338 osd_deep_scrub_randomize_ratio=0.000000
osd.338 osd_max_scrubs=1
osd.338 osd_scrub_backoff_ratio=0.931900 rec. this pool: .9319 (class hdd, size 11)
mon.ceph-01 mon_warn_pg_not_scrubbed_ratio=0.500000 warn: 10.5d (42.0i)
mon.ceph-01 mon_warn_pg_not_deep_scrubbed_ratio=0.750000 warn: 24.5d
Best regards, merry Christmas and a happy new year to everyone!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
13 Dec
13 Dec
12:36 p.m.
Hi all,
since there seems to be some interest, here some additional notes.
1) The script is tested on octopus. It seems that there was a change in the output of ceph
commands used and it might need some tweaking to get it to work on other versions.
2) If you want to give my findings a shot, you can do so in a gradual way. The most
important change is setting osd_deep_scrub_randomize_ratio=0 (with osd_max_scrubs=1), this
will make osd_deep_scrub_interval work exactly as the requested
osd_deep_scrub_min_interval setting, PGs with a deep-scrub stamp younger than
osd_deep_scrub_interval will *not* be deep-scrubbed. This is the one change to test, all
other settings have less impact. The script will not report some numbers at the end, but
the histogram will be correct. Let it run a few deep-scrub-interval rounds until the
histogram is evened out.
If you start your test after using osd_max_scrubs>1 for a while -as I did - you will
need a lot of patience and might need to mute some scrub warnings for a while.
3) The changes are mostly relevant for large HDDs that take a long time to deep-scrub
(many small objects). The overall load reduction, however, is useful in general.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
15 Dec
15 Dec
9:11 a.m.
Hi all,
another quick update: please use this link to download the script:
https://github.com/frans42/ceph-goodies/blob/main/scripts/pool-scrub-report
The one I sent originally does not follow latest.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8 Jan
8 Jan
3:21 p.m.
Hallo Frank,
just found this post, thank you! I have also been puzzled/struggling
with scrub/deep-scrub and found your post very useful: will give this a
try, soon.
One thing, first: I am using Octopus, too, but I cannot find any
documentation about osd_deep_scrub_randomize_ratio. I do see that in
past releases, but not on Octopus: is it still a valid parameter?
Let me check whether I understood your procedure: you optimize scrub
time distribution essentially by playing with osd_scrub_min_interval,
thus "forcing" the automated algorithm to preferentially select
older-scrubbed PGs, am I correct?
Another small question: you opt for osd_max_scrubs=1 just to make sure
your I/O is not adversely affected by scrubbing, or is there a more
profound reason for that?
Thanks!
Fulvio
On 12/13/23 13:36, Frank Schilder wrote:
Hi all,
since there seems to be some interest, here some additional notes.
1) The script is tested on octopus. It seems that there was a change in the output of
ceph commands used and it might need some tweaking to get it to work on other versions.
2) If you want to give my findings a shot, you can do so in a gradual way. The most
important change is setting osd_deep_scrub_randomize_ratio=0 (with osd_max_scrubs=1), this
will make osd_deep_scrub_interval work exactly as the requested
osd_deep_scrub_min_interval setting, PGs with a deep-scrub stamp younger than
osd_deep_scrub_interval will *not* be deep-scrubbed. This is the one change to test, all
other settings have less impact. The script will not report some numbers at the end, but
the histogram will be correct. Let it run a few deep-scrub-interval rounds until the
histogram is evened out.
If you start your test after using osd_max_scrubs>1 for a while -as I did - you will
need a lot of patience and might need to mute some scrub warnings for a while.
3) The changes are mostly relevant for large HDDs that take a long time to deep-scrub
(many small objects). The overall load reduction, however, is useful in general.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Fulvio Galeazzi
GARR-Net Department
tel.: +39-334-6533-250
skype: fgaleazzi70
9 Jan
9 Jan
4:05 p.m.
Quick answers:
* ... osd_deep_scrub_randomize_ratio ... but not on Octopus: is it still a valid
parameter?
Yes, this parameter exists and can be used to prevent premature deep-scrubs. The effect is
dramatic.
* ... essentially by playing with osd_scrub_min_interval,...
The main parameter is actually osd_deep_scrub_randomize_ratio, all other parameters have
less effect in terms of scrub load. osd_scrub_min_interval is the second most important
parameter and needs increasing for large SATA-/NL-SAS-HDDs. For sufficiently fast drives
the default of 24h is good (although might be a bit aggressive/paranoid).
* Another small question: you opt for osd_max_scrubs=1 just to make sure
your I/O is not adversely affected by scrubbing, or is there a more
profound reason for that?
Well, not affecting user-IO too much is a quite profound reason and many admins try to
avoid scrubbing at all when users are on the system. It makes IO somewhat unpredictable
and can trigger user complaints.
However, there is another profound reason: for HDDs it increases deep-scrub load (that is,
interference with user IO) a lot while it actually slows down the deep-scrubbing. HDDs
can't handle the implied random IO of concurrent deep-scrubs well. On my system I saw
that with osd_max_scrubs=2 the scrub time for a PG increased a bit more than double. In
other words: more scrub load, less scrub progress = useless, do not do this.
I plan to document the script a bit more and am waiting for some deep-scrub histograms to
converge to equilibrium. This takes months for our large pools, but I would like to have
the numbers for an example of how it should look like.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________
From: Fulvio Galeazzi
Sent: Monday, January 8, 2024 4:21 PM
To: Frank Schilder; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: How to configure something like
osd_deep_scrub_min_interval?
Hallo Frank,
just found this post, thank you! I have also been puzzled/struggling
with scrub/deep-scrub and found your post very useful: will give this a
try, soon.
One thing, first: I am using Octopus, too, but I cannot find any
documentation about osd_deep_scrub_randomize_ratio. I do see that in
past releases, but not on Octopus: is it still a valid parameter?
Let me check whether I understood your procedure: you optimize scrub
time distribution essentially by playing with osd_scrub_min_interval,
thus "forcing" the automated algorithm to preferentially select
older-scrubbed PGs, am I correct?
Another small question: you opt for osd_max_scrubs=1 just to make sure
your I/O is not adversely affected by scrubbing, or is there a more
profound reason for that?
Thanks!
Fulvio
On 12/13/23 13:36, Frank Schilder wrote:
Hi all,
since there seems to be some interest, here some additional notes.
1) The script is tested on octopus. It seems that there was a change in the output of
ceph commands used and it might need some tweaking to get it to work on other versions.
2) If you want to give my findings a shot, you can do so in a gradual way. The most
important change is setting osd_deep_scrub_randomize_ratio=0 (with osd_max_scrubs=1), this
will make osd_deep_scrub_interval work exactly as the requested
osd_deep_scrub_min_interval setting, PGs with a deep-scrub stamp younger than
osd_deep_scrub_interval will *not* be deep-scrubbed. This is the one change to test, all
other settings have less impact. The script will not report some numbers at the end, but
the histogram will be correct. Let it run a few deep-scrub-interval rounds until the
histogram is evened out.
If you start your test after using osd_max_scrubs>1 for a while -as I did - you will
need a lot of patience and might need to mute some scrub warnings for a while.
3) The changes are mostly relevant for large HDDs that take a long time to deep-scrub
(many small objects). The overall load reduction, however, is useful in general.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Fulvio Galeazzi
GARR-Net Department
tel.: +39-334-6533-250
skype: fgaleazzi70
130
days inactive
185
days old
6 comments
2 participants
participants (2)
-
Frank Schilder -
Fulvio Galeazzi