How to configure something like osd_deep_scrub_min_interval?
Hi folks, I am fighting a bit with odd deep-scrub behavior on HDDs and discovered a likely cause of why the distribution of last_deep_scrub_stamps is so weird. I wrote a small script to extract a histogram of scrubs by "days not scrubbed" (more precisely, intervals not scrubbed; see code) to find out how (deep-) scrub times are distributed. Output below. What I expected is along the lines that HDD-OSDs try to scrub every 1-3 days, while they try to deep-scrub every 7-14 days. In other words, OSDs that have been deep-scrubbed within the last 7 days would *never* be in scrubbing+deep state. However, what I see is completely different. There seems to be no distinction between scrub- and deep-scrub start times. This is really unexpected as nobody would try to deep-scrub HDDs every day. Weekly to bi-weekly is normal, specifically for large drives. Is there a way to configure something like osd_deep_scrub_min_interval (no, I don't want to run cron jobs for scrubbing yet)? In the output below, I would like to be able to configure a minimum period of 1-2 weeks before the next deep-scrub happens. How can I do that? The observed behavior is very unusual for RAID systems (if its not a bug in the report script). With this behavior its not surprising that people complain about "not deep-scrubbed in time" messages and too high deep-scrub IO load when such a large percentage of OSDs is needlessly deep-scrubbed after 1-6 days again already. Sample output: # scrub-report dumped pgs Scrub report: 4121 PGs not scrubbed since 1 intervals (6h) 3831 PGs not scrubbed since 2 intervals (6h) 4012 PGs not scrubbed since 3 intervals (6h) 3986 PGs not scrubbed since 4 intervals (6h) 2998 PGs not scrubbed since 5 intervals (6h) 1488 PGs not scrubbed since 6 intervals (6h) 909 PGs not scrubbed since 7 intervals (6h) 771 PGs not scrubbed since 8 intervals (6h) 582 PGs not scrubbed since 9 intervals (6h) 2 scrubbing 431 PGs not scrubbed since 10 intervals (6h) 333 PGs not scrubbed since 11 intervals (6h) 1 scrubbing 265 PGs not scrubbed since 12 intervals (6h) 195 PGs not scrubbed since 13 intervals (6h) 116 PGs not scrubbed since 14 intervals (6h) 78 PGs not scrubbed since 15 intervals (6h) 1 scrubbing 72 PGs not scrubbed since 16 intervals (6h) 37 PGs not scrubbed since 17 intervals (6h) 5 PGs not scrubbed since 18 intervals (6h) 14.237* 19.5cd* 19.12cc* 19.1233* 14.40e* 33 PGs not scrubbed since 20 intervals (6h) 23 PGs not scrubbed since 21 intervals (6h) 16 PGs not scrubbed since 22 intervals (6h) 12 PGs not scrubbed since 23 intervals (6h) 8 PGs not scrubbed since 24 intervals (6h) 2 PGs not scrubbed since 25 intervals (6h) 19.eef* 19.bb3* 4 PGs not scrubbed since 26 intervals (6h) 19.b4c* 19.10b8* 19.f13* 14.1ed* 5 PGs not scrubbed since 27 intervals (6h) 19.43f* 19.231* 19.1dbe* 19.1788* 19.16c0* 6 PGs not scrubbed since 28 intervals (6h) 2 PGs not scrubbed since 30 intervals (6h) 19.10f6* 14.9d* 3 PGs not scrubbed since 31 intervals (6h) 19.1322* 19.1318* 8.a* 1 PGs not scrubbed since 32 intervals (6h) 19.133f* 1 PGs not scrubbed since 33 intervals (6h) 19.1103* 3 PGs not scrubbed since 36 intervals (6h) 19.19cc* 19.12f4* 19.248* 1 PGs not scrubbed since 39 intervals (6h) 19.1984* 1 PGs not scrubbed since 41 intervals (6h) 14.449* 1 PGs not scrubbed since 44 intervals (6h) 19.179f* Deep-scrub report: 3723 PGs not deep-scrubbed since 1 intervals (24h) 4621 PGs not deep-scrubbed since 2 intervals (24h) 8 scrubbing+deep 3588 PGs not deep-scrubbed since 3 intervals (24h) 8 scrubbing+deep 2929 PGs not deep-scrubbed since 4 intervals (24h) 3 scrubbing+deep 1705 PGs not deep-scrubbed since 5 intervals (24h) 4 scrubbing+deep 1904 PGs not deep-scrubbed since 6 intervals (24h) 5 scrubbing+deep 1540 PGs not deep-scrubbed since 7 intervals (24h) 7 scrubbing+deep 1304 PGs not deep-scrubbed since 8 intervals (24h) 7 scrubbing+deep 923 PGs not deep-scrubbed since 9 intervals (24h) 5 scrubbing+deep 557 PGs not deep-scrubbed since 10 intervals (24h) 7 scrubbing+deep 501 PGs not deep-scrubbed since 11 intervals (24h) 2 scrubbing+deep 363 PGs not deep-scrubbed since 12 intervals (24h) 2 scrubbing+deep 377 PGs not deep-scrubbed since 13 intervals (24h) 1 scrubbing+deep 383 PGs not deep-scrubbed since 14 intervals (24h) 2 scrubbing+deep 252 PGs not deep-scrubbed since 15 intervals (24h) 2 scrubbing+deep 116 PGs not deep-scrubbed since 16 intervals (24h) 5 scrubbing+deep 47 PGs not deep-scrubbed since 17 intervals (24h) 2 scrubbing+deep 10 PGs not deep-scrubbed since 18 intervals (24h) 2 PGs not deep-scrubbed since 19 intervals (24h) 19.1c6c* 19.a01* 1 PGs not deep-scrubbed since 20 intervals (24h) 14.1ed* 2 PGs not deep-scrubbed since 21 intervals (24h) 19.1322* 19.10f6* 1 PGs not deep-scrubbed since 23 intervals (24h) 19.19cc* 1 PGs not deep-scrubbed since 24 intervals (24h) 19.179f* PGs marked with a * are on busy OSDs and not eligible for scrubbing. The script (pasted here because attaching doesn't work): # cat bin/scrub-report #!/bin/bash # Compute last scrub interval count. Scrub interval 6h, deep-scrub interval 24h. # Print how many PGs have not been (deep-)scrubbed since #intervals. ceph -f json pg dump pgs 2>&1 > /root/.cache/ceph/pgs_dump.json echo "" T0="$(date +%s)" scrub_info="$(jq --arg T0 "$T0" -rc '.pg_stats[] | [ .pgid, (.last_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*6)|ceil), (.last_deep_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*24)|ceil), .state, (.acting | join(" ")) ] | @tsv ' /root/.cache/ceph/pgs_dump.json)" # less <<<"$scrub_info" # 1 2 3 4 5..NF # pg_id scrub-ints deep-scrub-ints status acting[] awk <<<"$scrub_info" '{ for(i=5; i<=NF; ++i) pg_osds[$1]=pg_osds[$1] " " $i if($4 == "active+clean") { si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 } else if($4 ~ /scrubbing\+deep/) { deep_scrubbing[$3]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else if($4 ~ /scrubbing/) { scrubbing[$2]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else { unclean[$2]++ unclean_d[$3]++ si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 for(i=5; i<=NF; ++i) osd[$i]="busy" } } END { print "Scrub report:" for(si=1; si<=si_mx; ++si) { if(pg_sn[si]==0 && scrubbing[si]==0 && unclean[si]==0) continue; printf("%7d PGs not scrubbed since %2d intervals (6h)", pg_sn[si], si) if(scrubbing[si]) printf(" %d scrubbing", scrubbing[si]) if(unclean[si]) printf(" %d unclean", unclean[si]) if(pg_sn[si]<=5) { split(pg_sn_ids[si], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "Deep-scrub report:" for(dsi=1; dsi<=dsi_mx; ++dsi) { if(pg_dsn[dsi]==0 && deep_scrubbing[dsi]==0 && unclean_d[dsi]==0) continue; printf("%7d PGs not deep-scrubbed since %2d intervals (24h)", pg_dsn[dsi], dsi) if(deep_scrubbing[dsi]) printf(" %d scrubbing+deep", deep_scrubbing[dsi]) if(unclean_d[dsi]) printf(" %d unclean", unclean_d[dsi]) if(pg_dsn[dsi]<=5) { split(pg_dsn_ids[dsi], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "PGs marked with a * are on busy OSDs and not eligible for scrubbing." } ' Don't forget the last "'" when copy-pasting. Thanks for any pointers. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Partially answering my own question. I think it is possible to tweak the existing parameters to achieve what I'm looking for on average. The main reason I want to use the internal scheduler is the high number of PGs on some pools, which I actually intend to increase even further. For such pools a simple calculation shows that it is impractical to do manual scrubbing with cron, I simply cannot execute cron jobs often enough to achieve a reasonable scrub distribution (looking at a script like https://gist.github.com/ethaniel/5db696d9c78516308b235b0cb904e4ad). Looking at scrub date stamp distributions for specific pools, PGs with old deep-scrub time stamps tend to correlate also with PGs with old scrub date stamps. The idea now id to tweak scrub_min_interval such that the scrub scheduler is forced to select PGs out of the 20-30% with the oldest scrub stamps. This should imply that, after a reasonable time interval, the age of the oldest deep-scrub stamps is reduced as long not deep-scrubbed PGs become much more likely to be scheduled for deep-scrub. This adjustment is done together with making osd_deep_scrub_randomize_ratio and osd_scrub_backoff_ratio more aggressive to cycle more frequently through the small list of PGs eligible for scrubbing. This is a bit like the reverse calculation for achieving the effect of a not implemented deep_scrub_min_interval. I made the following global changes (and hope the parameters do something like what their documentation says): global advanced osd_deep_scrub_randomize_ratio 0.330000 global dev osd_scrub_backoff_ratio 0.500000 With this setting, about 33% of all scrub should be deep-scrubs, meaning that on average after 3 scrub events a PG is also deep-scrubbed. This leads to this estimate for expected deep-scrub intervals: given scrub-interval is scrub_min_interval*[1, 1+osd_scrub_interval_randomize_ratio] = scrub_min_interval*[1, 1.5], the expected deep-scrub-interval is (assuming worst-case realisation of randomize_ratio for upper value): scrub_min_interval*[1, 1.5]*3 = scrub_min_interval*[3, 4.5] I can't calculate how the tail will look like, but I hope its not a fat tail. I will report back what I observe; see below. The idea now is to tune scrub_min_interval per pool such that only about 20-30% of PGs have a scrub-stamp older than scrub_min_interval. The scheduler will cycle only through these and a bit faster than default. As the stamp histograms included below indicate, the distribution is probably very sensitive to changes of this interval. I now changed these values on some pools and already see that PGs with much older deep-scrub stamps are now selected for deep-scrubbing. I will observe what these settings converge to and report back. It seems that it will lead to an improved stamp distribution and one only needs to issue manual deep-scrubs for very few PGs that are outliers of the random number generator (that's the tail I talked about above). My goal is to have a script schedule a deep-scrub on the outliers no more often than daily. The reports below have been pulled after changes to settings were applied for about 2-3h. There was already improvement in the right direction, but the original distribution issue is still very pronounced. Here two per-pool scrub stamp distributions for an SSD pool and an HDD pool, both with large number of PGs per OSD: === SSD pool: Scrub info for pool sr-rbd-data-one (id=2): dumped pgs Scrub report: 22% 941 PGs not scrubbed since 1 intervals ( 6h) 42% 795 PGs not scrubbed since 2 intervals ( 12h) 62% 829 PGs not scrubbed since 3 intervals ( 18h) 82% 823 PGs not scrubbed since 4 intervals ( 24h) 96% 576 PGs not scrubbed since 5 intervals ( 30h) 100% 132 PGs not scrubbed since 6 intervals ( 36h) 4096 PGs out of 4096 reported, 0 missing. Deep-scrub report: 13% 545 PGs not deep-scrubbed since 1 intervals ( 24h) 25% 508 PGs not deep-scrubbed since 2 intervals ( 48h) 1 scrubbing+deep 45% 797 PGs not deep-scrubbed since 3 intervals ( 72h) 1 scrubbing+deep 59% 587 PGs not deep-scrubbed since 4 intervals ( 96h) 70% 463 PGs not deep-scrubbed since 5 intervals (120h) 78% 312 PGs not deep-scrubbed since 6 intervals (144h) 84% 263 PGs not deep-scrubbed since 7 intervals (168h) 89% 173 PGs not deep-scrubbed since 8 intervals (192h) 92% 151 PGs not deep-scrubbed since 9 intervals (216h) 95% 106 PGs not deep-scrubbed since 10 intervals (240h) 96% 55 PGs not deep-scrubbed since 11 intervals (264h) 97% 50 PGs not deep-scrubbed since 12 intervals (288h) 98% 44 PGs not deep-scrubbed since 13 intervals (312h) 99% 24 PGs not deep-scrubbed since 14 intervals (336h) 100% 18 PGs not deep-scrubbed since 15 intervals (360h) 4096 PGs out of 4096 reported, 0 missing. PGs marked with a * are on busy OSDs and not eligible for scrubbing. sr-rbd-data-one scrub_min_interval=0h sr-rbd-data-one scrub_max_interval=0h sr-rbd-data-one deep_scrub_interval=0h === Here we see that after 24h 82% of PGs are scrubbed, but we have quite a tail of not deep-scrubbed PGs. Its long enough to trigger a warning with default parameters. In this case, reducing scrub_min_interval to a value around 18-20h could reduce the tail enough. The alternative is simply to schedule a deep-scrub on the oldest PGs manually (cron). This would start immediately since no OSDs are allocated to scrubbing/recovery. === HDD pool: Scrub info for pool con-fs2-data2 (id=19): dumped pgs Scrub report: 11% 939 PGs not scrubbed since 1 intervals ( 6h) 22% 936 PGs not scrubbed since 2 intervals ( 12h) 33% 874 PGs not scrubbed since 3 intervals ( 18h) 43% 821 PGs not scrubbed since 4 intervals ( 24h) 54% 931 PGs not scrubbed since 5 intervals ( 30h) 64% 766 PGs not scrubbed since 6 intervals ( 36h) 72% 646 PGs not scrubbed since 7 intervals ( 42h) 79% 559 PGs not scrubbed since 8 intervals ( 48h) 84% 411 PGs not scrubbed since 9 intervals ( 54h) 88% 346 PGs not scrubbed since 10 intervals ( 60h) 90% 222 PGs not scrubbed since 11 intervals ( 66h) 93% 213 PGs not scrubbed since 12 intervals ( 72h) 95% 160 PGs not scrubbed since 13 intervals ( 78h) 96% 87 PGs not scrubbed since 14 intervals ( 84h) 97% 77 PGs not scrubbed since 15 intervals ( 90h) 98% 57 PGs not scrubbed since 16 intervals ( 96h) 1 scrubbing 98% 42 PGs not scrubbed since 17 intervals (102h) 99% 32 PGs not scrubbed since 18 intervals (108h) 99% 19 PGs not scrubbed since 19 intervals (114h) 99% 19 PGs not scrubbed since 20 intervals (120h) 99% 10 PGs not scrubbed since 21 intervals (126h) 99% 1 PGs not scrubbed since 22 intervals (132h) 19.165f* 99% 5 PGs not scrubbed since 24 intervals (138h) 19.412* 19.75c* 19.140f* 19.134c* 19.fb7* 99% 5 PGs not scrubbed since 25 intervals (144h) 19.1714* 19.148d* 19.1fa9* 19.1f05* 19.1cda* 99% 1 PGs not scrubbed since 26 intervals (150h) 19.a3f* 99% 1 PGs not scrubbed since 27 intervals (156h) 19.a01* 99% 3 PGs not scrubbed since 28 intervals (162h) 19.12f2* 19.1284* 19.c90* 99% 1 PGs not scrubbed since 29 intervals (168h) 99% 1 PGs not scrubbed since 30 intervals (174h) 19.f13* 99% 2 PGs not scrubbed since 32 intervals (180h) 19.1f87* 19.67b* 99% 2 PGs not scrubbed since 36 intervals (186h) 19.133f* 19.1318* 99% 2 PGs not scrubbed since 40 intervals (192h) 19.12f4* 19.248* 100% 1 PGs not scrubbed since 43 intervals (198h) 19.1984* 8192 PGs out of 8192 reported, 0 missing. Deep-scrub report: 14% 1210 PGs not deep-scrubbed since 1 intervals ( 24h) 28% 1136 PGs not deep-scrubbed since 2 intervals ( 48h) 1 scrubbing+deep 40% 985 PGs not deep-scrubbed since 3 intervals ( 72h) 4 scrubbing+deep 51% 851 PGs not deep-scrubbed since 4 intervals ( 96h) 5 scrubbing+deep 59% 713 PGs not deep-scrubbed since 5 intervals (120h) 4 scrubbing+deep 63% 276 PGs not deep-scrubbed since 6 intervals (144h) 2 scrubbing+deep 70% 566 PGs not deep-scrubbed since 7 intervals (168h) 1 scrubbing+deep 76% 534 PGs not deep-scrubbed since 8 intervals (192h) 2 scrubbing+deep 82% 480 PGs not deep-scrubbed since 9 intervals (216h) 2 scrubbing+deep 87% 381 PGs not deep-scrubbed since 10 intervals (240h) 2 scrubbing+deep 90% 253 PGs not deep-scrubbed since 11 intervals (264h) 1 scrubbing+deep 92% 222 PGs not deep-scrubbed since 12 intervals (288h) 1 scrubbing+deep 94% 136 PGs not deep-scrubbed since 13 intervals (312h) 1 scrubbing+deep 96% 179 PGs not deep-scrubbed since 14 intervals (336h) 3 scrubbing+deep 98% 156 PGs not deep-scrubbed since 15 intervals (360h) 6 scrubbing+deep 99% 65 PGs not deep-scrubbed since 16 intervals (384h) 3 scrubbing+deep 99% 31 PGs not deep-scrubbed since 17 intervals (408h) 4 scrubbing+deep 99% 14 PGs not deep-scrubbed since 18 intervals (432h) 3 scrubbing+deep 99% 3 PGs not deep-scrubbed since 19 intervals (456h) 19.1d89* 19.fb7* 19.807* 100% 1 PGs not deep-scrubbed since 21 intervals (480h) 19.a01* 8192 PGs out of 8192 reported, 0 missing. PGs marked with a * are on busy OSDs and not eligible for scrubbing. con-fs2-data2 scrub_min_interval=42h con-fs2-data2 scrub_max_interval=0h con-fs2-data2 deep_scrub_interval=0h === This is the real deal, the pool I'm fighting with at the moment. I made a small change in scrub_min_interval (pool setting) from 24h to 42h, which resulted in the very good deep-scrub state allocation of the PGs in the pool. With scrub_min_interval=24h basically all scrubbing happened on PGs not deep-scrubbed within 1-6 days. After increasing this value to the time interval for which about 70% of PGs were scrubbed (leaving 30% eligible), the allocation of deep-scrub states is much much better. I expect both tails to get shorter and the overall deep-scrub load to go down as well. I hope to reach a state where I only need to issue a few deep-scrubs manually per day to get everything scrubbed within 1 week and deep-scrubbed within 3-4 weeks. For now I will wait what effect the global settings have on the SSD pools and what the HDD pool converges to. This will need 1-2 months observations and I will report back when significant changes show up. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Wednesday, November 15, 2023 11:14 AM To: ceph-users@ceph.io Subject: [ceph-users] How to configure something like osd_deep_scrub_min_interval? Hi folks, I am fighting a bit with odd deep-scrub behavior on HDDs and discovered a likely cause of why the distribution of last_deep_scrub_stamps is so weird. I wrote a small script to extract a histogram of scrubs by "days not scrubbed" (more precisely, intervals not scrubbed; see code) to find out how (deep-) scrub times are distributed. Output below. What I expected is along the lines that HDD-OSDs try to scrub every 1-3 days, while they try to deep-scrub every 7-14 days. In other words, OSDs that have been deep-scrubbed within the last 7 days would *never* be in scrubbing+deep state. However, what I see is completely different. There seems to be no distinction between scrub- and deep-scrub start times. This is really unexpected as nobody would try to deep-scrub HDDs every day. Weekly to bi-weekly is normal, specifically for large drives. Is there a way to configure something like osd_deep_scrub_min_interval (no, I don't want to run cron jobs for scrubbing yet)? In the output below, I would like to be able to configure a minimum period of 1-2 weeks before the next deep-scrub happens. How can I do that? The observed behavior is very unusual for RAID systems (if its not a bug in the report script). With this behavior its not surprising that people complain about "not deep-scrubbed in time" messages and too high deep-scrub IO load when such a large percentage of OSDs is needlessly deep-scrubbed after 1-6 days again already. Sample output: # scrub-report dumped pgs Scrub report: 4121 PGs not scrubbed since 1 intervals (6h) 3831 PGs not scrubbed since 2 intervals (6h) 4012 PGs not scrubbed since 3 intervals (6h) 3986 PGs not scrubbed since 4 intervals (6h) 2998 PGs not scrubbed since 5 intervals (6h) 1488 PGs not scrubbed since 6 intervals (6h) 909 PGs not scrubbed since 7 intervals (6h) 771 PGs not scrubbed since 8 intervals (6h) 582 PGs not scrubbed since 9 intervals (6h) 2 scrubbing 431 PGs not scrubbed since 10 intervals (6h) 333 PGs not scrubbed since 11 intervals (6h) 1 scrubbing 265 PGs not scrubbed since 12 intervals (6h) 195 PGs not scrubbed since 13 intervals (6h) 116 PGs not scrubbed since 14 intervals (6h) 78 PGs not scrubbed since 15 intervals (6h) 1 scrubbing 72 PGs not scrubbed since 16 intervals (6h) 37 PGs not scrubbed since 17 intervals (6h) 5 PGs not scrubbed since 18 intervals (6h) 14.237* 19.5cd* 19.12cc* 19.1233* 14.40e* 33 PGs not scrubbed since 20 intervals (6h) 23 PGs not scrubbed since 21 intervals (6h) 16 PGs not scrubbed since 22 intervals (6h) 12 PGs not scrubbed since 23 intervals (6h) 8 PGs not scrubbed since 24 intervals (6h) 2 PGs not scrubbed since 25 intervals (6h) 19.eef* 19.bb3* 4 PGs not scrubbed since 26 intervals (6h) 19.b4c* 19.10b8* 19.f13* 14.1ed* 5 PGs not scrubbed since 27 intervals (6h) 19.43f* 19.231* 19.1dbe* 19.1788* 19.16c0* 6 PGs not scrubbed since 28 intervals (6h) 2 PGs not scrubbed since 30 intervals (6h) 19.10f6* 14.9d* 3 PGs not scrubbed since 31 intervals (6h) 19.1322* 19.1318* 8.a* 1 PGs not scrubbed since 32 intervals (6h) 19.133f* 1 PGs not scrubbed since 33 intervals (6h) 19.1103* 3 PGs not scrubbed since 36 intervals (6h) 19.19cc* 19.12f4* 19.248* 1 PGs not scrubbed since 39 intervals (6h) 19.1984* 1 PGs not scrubbed since 41 intervals (6h) 14.449* 1 PGs not scrubbed since 44 intervals (6h) 19.179f* Deep-scrub report: 3723 PGs not deep-scrubbed since 1 intervals (24h) 4621 PGs not deep-scrubbed since 2 intervals (24h) 8 scrubbing+deep 3588 PGs not deep-scrubbed since 3 intervals (24h) 8 scrubbing+deep 2929 PGs not deep-scrubbed since 4 intervals (24h) 3 scrubbing+deep 1705 PGs not deep-scrubbed since 5 intervals (24h) 4 scrubbing+deep 1904 PGs not deep-scrubbed since 6 intervals (24h) 5 scrubbing+deep 1540 PGs not deep-scrubbed since 7 intervals (24h) 7 scrubbing+deep 1304 PGs not deep-scrubbed since 8 intervals (24h) 7 scrubbing+deep 923 PGs not deep-scrubbed since 9 intervals (24h) 5 scrubbing+deep 557 PGs not deep-scrubbed since 10 intervals (24h) 7 scrubbing+deep 501 PGs not deep-scrubbed since 11 intervals (24h) 2 scrubbing+deep 363 PGs not deep-scrubbed since 12 intervals (24h) 2 scrubbing+deep 377 PGs not deep-scrubbed since 13 intervals (24h) 1 scrubbing+deep 383 PGs not deep-scrubbed since 14 intervals (24h) 2 scrubbing+deep 252 PGs not deep-scrubbed since 15 intervals (24h) 2 scrubbing+deep 116 PGs not deep-scrubbed since 16 intervals (24h) 5 scrubbing+deep 47 PGs not deep-scrubbed since 17 intervals (24h) 2 scrubbing+deep 10 PGs not deep-scrubbed since 18 intervals (24h) 2 PGs not deep-scrubbed since 19 intervals (24h) 19.1c6c* 19.a01* 1 PGs not deep-scrubbed since 20 intervals (24h) 14.1ed* 2 PGs not deep-scrubbed since 21 intervals (24h) 19.1322* 19.10f6* 1 PGs not deep-scrubbed since 23 intervals (24h) 19.19cc* 1 PGs not deep-scrubbed since 24 intervals (24h) 19.179f* PGs marked with a * are on busy OSDs and not eligible for scrubbing. The script (pasted here because attaching doesn't work): # cat bin/scrub-report #!/bin/bash # Compute last scrub interval count. Scrub interval 6h, deep-scrub interval 24h. # Print how many PGs have not been (deep-)scrubbed since #intervals. ceph -f json pg dump pgs 2>&1 > /root/.cache/ceph/pgs_dump.json echo "" T0="$(date +%s)" scrub_info="$(jq --arg T0 "$T0" -rc '.pg_stats[] | [ .pgid, (.last_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*6)|ceil), (.last_deep_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*24)|ceil), .state, (.acting | join(" ")) ] | @tsv ' /root/.cache/ceph/pgs_dump.json)" # less <<<"$scrub_info" # 1 2 3 4 5..NF # pg_id scrub-ints deep-scrub-ints status acting[] awk <<<"$scrub_info" '{ for(i=5; i<=NF; ++i) pg_osds[$1]=pg_osds[$1] " " $i if($4 == "active+clean") { si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 } else if($4 ~ /scrubbing\+deep/) { deep_scrubbing[$3]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else if($4 ~ /scrubbing/) { scrubbing[$2]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else { unclean[$2]++ unclean_d[$3]++ si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 for(i=5; i<=NF; ++i) osd[$i]="busy" } } END { print "Scrub report:" for(si=1; si<=si_mx; ++si) { if(pg_sn[si]==0 && scrubbing[si]==0 && unclean[si]==0) continue; printf("%7d PGs not scrubbed since %2d intervals (6h)", pg_sn[si], si) if(scrubbing[si]) printf(" %d scrubbing", scrubbing[si]) if(unclean[si]) printf(" %d unclean", unclean[si]) if(pg_sn[si]<=5) { split(pg_sn_ids[si], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "Deep-scrub report:" for(dsi=1; dsi<=dsi_mx; ++dsi) { if(pg_dsn[dsi]==0 && deep_scrubbing[dsi]==0 && unclean_d[dsi]==0) continue; printf("%7d PGs not deep-scrubbed since %2d intervals (24h)", pg_dsn[dsi], dsi) if(deep_scrubbing[dsi]) printf(" %d scrubbing+deep", deep_scrubbing[dsi]) if(unclean_d[dsi]) printf(" %d unclean", unclean_d[dsi]) if(pg_dsn[dsi]<=5) { split(pg_dsn_ids[dsi], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "PGs marked with a * are on busy OSDs and not eligible for scrubbing." } ' Don't forget the last "'" when copy-pasting. Thanks for any pointers. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, a little gem for Christmas. After going through the OSD code, scratching my had and doing a bit of maths, I seem to have found a way to tune the built-in scrub machine to work perfectly. Its only few knobs to turn, but its difficult to find out, because the documentation is misleading to incorrect or even missing entirely. I plan to write a bit more in the documentation to this script (https://github.com/frans42/ceph-goodies/blob/5e2016f0b00f8dbc3e51c7e9904a738...), so here only the executive summary. - global, set osd_max_scrubs=1, higher values do not have any effect except making your users angry - global, set osd_deep_scrub_randomize_ratio=0, this parameter is unnecessary for distributing deep scrubs and its only effect in the current implementation is to trigger a large amount of too early deep-scrubs increasing the overall deep-scrub load significantly without a useful effect - on the pools, set deep_scrub_interval according to needs and performance, scrubs will turn into deep-scrubs for every PG with a deep-scrub stamp older than deep_scrub_interval, here you will need to do some calculations what your hardware can do and how much average load you can tolerate - on the pools, also set scrub_min_interval and scrub_max_interval so that its only one place to look for these settings - per OSD device class, set osd_scrub_interval_randomize_ratio such that scrubs start within a reasonable window after scrub_min_interval. This parameter is very important for distributing scrubs as evenly as possible over time. The default of 0.5 is good for most cases. For the HDD pool used below I reduced it a bit, because scrub_min_interval is set to 66h and here 0.5 leads to a slightly too large start-interval. - per OSD device class, set osd_scrub_backoff_ratio to a value close to but not higher than 1-1/(largest replication factor [=size] of pools on this device class). This parameter is labelled dev, but is really important for effective scrub scheduling. OSDs need to allocate scrub reservations and this process is extremely racy specifically for EC pools with high replication factor. The default 0.66 probably has 3-times replicated pools in mind, but it triggers way too many attempts to allocate scrub reservations for pools with larger replication factor, causing dead-locks and blocking scrubs being executed even if plenty of OSDs are idle. I found that 1-0.75/max_size_on_device_class works very well. After having found out about what the parameters really do and adjusting them for my pools, I passed through a valley of tears and arrived now at the beautiful distribution of (deep-)scrub stamps for a pool on 16TB HDDs shown at the end. Everything gets scrubbed every 3-4 days and deep-scrubs start no earlier than 14 days after the last deep-scrub. The overall (deep)-scrub load is now half of what it was before the changes and I don't have the dreaded "PGs not (deep-)scrubbed in time" warnings any more. I calculated the (deep-)scrub time window configs such that about 30% of OSDs will be continuously busy when the disks reach 70% utilization (currently ca. 45%). No user will complain about that and there is enough spare performance left to catch up after high-load- or recovery episodes without having to do anything. Here is the scrub report generated for the pool I was looking at for weeks now, its exactly as I wanted it and I don't have to run cron jobs, it just works: # pool-scrub-report con-fs2-data2 Scrub info for pool con-fs2-data2 (id=19): dumped pgs Scrub report: 6% 566 PGs not scrubbed since 1 intervals ( 6h) 13% 528 PGs not scrubbed since 2 intervals ( 12h) 21% 640 PGs not scrubbed since 3 intervals ( 18h) 29% 668 PGs not scrubbed since 4 intervals ( 24h) 37% 677 PGs not scrubbed since 5 intervals ( 30h) 46% 729 PGs not scrubbed since 6 intervals ( 36h) 54% 631 PGs not scrubbed since 7 intervals ( 42h) 62% 662 PGs not scrubbed since 8 intervals ( 48h) 70% 663 PGs not scrubbed since 9 intervals ( 54h) 78% 660 PGs not scrubbed since 10 intervals ( 60h) 85% 571 PGs not scrubbed since 11 intervals ( 66h) 92% 585 PGs not scrubbed since 12 intervals ( 72h) [74 idle] 1 scrubbing 96% 358 PGs not scrubbed since 13 intervals ( 78h) [34 idle] [3 scrubbing+deep] 2 scrubbing 99% 181 PGs not scrubbed since 14 intervals ( 84h) [23 idle] [3 scrubbing+deep] 99% 70 PGs not scrubbed since 15 intervals ( 90h) [9 idle] 1 scrubbing 100% 3 PGs not scrubbed since 16 intervals ( 96h) [1 scrubbing+deep] 8192 PGs out of 8192 reported, 0 missing, 4 scrubbing, 140 idle, 0 unclean. Deep-scrub report: 3% 295 PGs not deep-scrubbed since 1 intervals ( 24h) 9% 461 PGs not deep-scrubbed since 2 intervals ( 48h) 16% 558 PGs not deep-scrubbed since 3 intervals ( 72h) 23% 613 PGs not deep-scrubbed since 4 intervals ( 96h) [1 scrubbing] 31% 619 PGs not deep-scrubbed since 5 intervals (120h) 39% 660 PGs not deep-scrubbed since 6 intervals (144h) 47% 726 PGs not deep-scrubbed since 7 intervals (168h) [1 scrubbing] 57% 743 PGs not deep-scrubbed since 8 intervals (192h) 65% 727 PGs not deep-scrubbed since 9 intervals (216h) [1 scrubbing] 73% 656 PGs not deep-scrubbed since 10 intervals (240h) 75% 107 PGs not deep-scrubbed since 11 intervals (264h) 82% 626 PGs not deep-scrubbed since 12 intervals (288h) 90% 588 PGs not deep-scrubbed since 13 intervals (312h) 94% 388 PGs not deep-scrubbed since 14 intervals (336h) [1 scrubbing] 96% 129 PGs not deep-scrubbed since 15 intervals (360h) 2 scrubbing+deep 98% 207 PGs not deep-scrubbed since 16 intervals (384h) 2 scrubbing+deep 99% 79 PGs not deep-scrubbed since 17 intervals (408h) 1 scrubbing+deep 100% 10 PGs not deep-scrubbed since 18 intervals (432h) 2 scrubbing+deep 8192 PGs out of 8192 reported, 0 missing, 7 scrubbing+deep, 0 unclean. con-fs2-data2 scrub_min_interval=66h (11i/84%/625PGs÷i) con-fs2-data2 scrub_max_interval=168h (7d) con-fs2-data2 deep_scrub_interval=336h (14d/~89%/~520PGs÷d) osd.338 osd_scrub_interval_randomize_ratio=0.363636 scrubs start after: 66h..90h osd.338 osd_deep_scrub_randomize_ratio=0.000000 osd.338 osd_max_scrubs=1 osd.338 osd_scrub_backoff_ratio=0.931900 rec. this pool: .9319 (class hdd, size 11) mon.ceph-01 mon_warn_pg_not_scrubbed_ratio=0.500000 warn: 10.5d (42.0i) mon.ceph-01 mon_warn_pg_not_deep_scrubbed_ratio=0.750000 warn: 24.5d Best regards, merry Christmas and a happy new year to everyone! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, since there seems to be some interest, here some additional notes. 1) The script is tested on octopus. It seems that there was a change in the output of ceph commands used and it might need some tweaking to get it to work on other versions. 2) If you want to give my findings a shot, you can do so in a gradual way. The most important change is setting osd_deep_scrub_randomize_ratio=0 (with osd_max_scrubs=1), this will make osd_deep_scrub_interval work exactly as the requested osd_deep_scrub_min_interval setting, PGs with a deep-scrub stamp younger than osd_deep_scrub_interval will *not* be deep-scrubbed. This is the one change to test, all other settings have less impact. The script will not report some numbers at the end, but the histogram will be correct. Let it run a few deep-scrub-interval rounds until the histogram is evened out. If you start your test after using osd_max_scrubs>1 for a while -as I did - you will need a lot of patience and might need to mute some scrub warnings for a while. 3) The changes are mostly relevant for large HDDs that take a long time to deep-scrub (many small objects). The overall load reduction, however, is useful in general. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, another quick update: please use this link to download the script: https://github.com/frans42/ceph-goodies/blob/main/scripts/pool-scrub-report The one I sent originally does not follow latest. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hallo Frank, just found this post, thank you! I have also been puzzled/struggling with scrub/deep-scrub and found your post very useful: will give this a try, soon. One thing, first: I am using Octopus, too, but I cannot find any documentation about osd_deep_scrub_randomize_ratio. I do see that in past releases, but not on Octopus: is it still a valid parameter? Let me check whether I understood your procedure: you optimize scrub time distribution essentially by playing with osd_scrub_min_interval, thus "forcing" the automated algorithm to preferentially select older-scrubbed PGs, am I correct? Another small question: you opt for osd_max_scrubs=1 just to make sure your I/O is not adversely affected by scrubbing, or is there a more profound reason for that? Thanks! Fulvio On 12/13/23 13:36, Frank Schilder wrote:
Hi all,
since there seems to be some interest, here some additional notes.
1) The script is tested on octopus. It seems that there was a change in the output of ceph commands used and it might need some tweaking to get it to work on other versions.
2) If you want to give my findings a shot, you can do so in a gradual way. The most important change is setting osd_deep_scrub_randomize_ratio=0 (with osd_max_scrubs=1), this will make osd_deep_scrub_interval work exactly as the requested osd_deep_scrub_min_interval setting, PGs with a deep-scrub stamp younger than osd_deep_scrub_interval will *not* be deep-scrubbed. This is the one change to test, all other settings have less impact. The script will not report some numbers at the end, but the histogram will be correct. Let it run a few deep-scrub-interval rounds until the histogram is evened out.
If you start your test after using osd_max_scrubs>1 for a while -as I did - you will need a lot of patience and might need to mute some scrub warnings for a while.
3) The changes are mostly relevant for large HDDs that take a long time to deep-scrub (many small objects). The overall load reduction, however, is useful in general.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Fulvio Galeazzi GARR-Net Department tel.: +39-334-6533-250 skype: fgaleazzi70
Quick answers: * ... osd_deep_scrub_randomize_ratio ... but not on Octopus: is it still a valid parameter? Yes, this parameter exists and can be used to prevent premature deep-scrubs. The effect is dramatic. * ... essentially by playing with osd_scrub_min_interval,... The main parameter is actually osd_deep_scrub_randomize_ratio, all other parameters have less effect in terms of scrub load. osd_scrub_min_interval is the second most important parameter and needs increasing for large SATA-/NL-SAS-HDDs. For sufficiently fast drives the default of 24h is good (although might be a bit aggressive/paranoid). * Another small question: you opt for osd_max_scrubs=1 just to make sure your I/O is not adversely affected by scrubbing, or is there a more profound reason for that? Well, not affecting user-IO too much is a quite profound reason and many admins try to avoid scrubbing at all when users are on the system. It makes IO somewhat unpredictable and can trigger user complaints. However, there is another profound reason: for HDDs it increases deep-scrub load (that is, interference with user IO) a lot while it actually slows down the deep-scrubbing. HDDs can't handle the implied random IO of concurrent deep-scrubs well. On my system I saw that with osd_max_scrubs=2 the scrub time for a PG increased a bit more than double. In other words: more scrub load, less scrub progress = useless, do not do this. I plan to document the script a bit more and am waiting for some deep-scrub histograms to converge to equilibrium. This takes months for our large pools, but I would like to have the numbers for an example of how it should look like. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________ From: Fulvio Galeazzi Sent: Monday, January 8, 2024 4:21 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: How to configure something like osd_deep_scrub_min_interval? Hallo Frank, just found this post, thank you! I have also been puzzled/struggling with scrub/deep-scrub and found your post very useful: will give this a try, soon. One thing, first: I am using Octopus, too, but I cannot find any documentation about osd_deep_scrub_randomize_ratio. I do see that in past releases, but not on Octopus: is it still a valid parameter? Let me check whether I understood your procedure: you optimize scrub time distribution essentially by playing with osd_scrub_min_interval, thus "forcing" the automated algorithm to preferentially select older-scrubbed PGs, am I correct? Another small question: you opt for osd_max_scrubs=1 just to make sure your I/O is not adversely affected by scrubbing, or is there a more profound reason for that? Thanks! Fulvio On 12/13/23 13:36, Frank Schilder wrote:
Hi all,
since there seems to be some interest, here some additional notes.
1) The script is tested on octopus. It seems that there was a change in the output of ceph commands used and it might need some tweaking to get it to work on other versions.
2) If you want to give my findings a shot, you can do so in a gradual way. The most important change is setting osd_deep_scrub_randomize_ratio=0 (with osd_max_scrubs=1), this will make osd_deep_scrub_interval work exactly as the requested osd_deep_scrub_min_interval setting, PGs with a deep-scrub stamp younger than osd_deep_scrub_interval will *not* be deep-scrubbed. This is the one change to test, all other settings have less impact. The script will not report some numbers at the end, but the histogram will be correct. Let it run a few deep-scrub-interval rounds until the histogram is evened out.
If you start your test after using osd_max_scrubs>1 for a while -as I did - you will need a lot of patience and might need to mute some scrub warnings for a while.
3) The changes are mostly relevant for large HDDs that take a long time to deep-scrub (many small objects). The overall load reduction, however, is useful in general.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Fulvio Galeazzi GARR-Net Department tel.: +39-334-6533-250 skype: fgaleazzi70
participants (2)
-
Frank Schilder
-
Fulvio Galeazzi