Hi, We upgraded one of our production cluster (480 OSDs, 13,5 K PGs, most pools EC 9+6) from 18.2.7 to 19.2.2 on May 26. It was healthy when we upgraded it and remained so until last Thrusday (June 5) where, doing a `ceph -s`, I saw that there were 1 deep scrub and ~1000 scrubs late: 1 pgs not deep-scrubbed in time 1168 pgs not scrubbed in time Looking again Friday morning (~16 hours later), this number increased a lot: 27 pgs not deep-scrubbed in time 3252 pgs not scrubbed in time Checking our configuration on Friday, we found that osd_max_scrubs was set to 1 instead of using the new default since Reef which is 3 (probably a leftover of a config change after a problem 18 monts ago). We unset the specific value and this led to a reduction of these numbers in the next 24 hours (~2700 scrubs late) but since then (2 days) it remained stable initially and is now increasing slowly. This morning the situation is: 294 pgs not deep-scrubbed in time 3013 pgs not scrubbed in time My guess is that the real issue is the late scrubs that results in many OSD reaching the limit of 3 concurrent scrubs and that it has the consequences that some deep scrubs cannot run too (I've in mind that the limit applies both to shallow scubs and deep scrubs, am I right?). I checked the main cluster logs and didn't find any error or warning, related to OSDs like slow ops, slow requests...The only things we have spotted through our monitoring system is a dramatic decrease (~75%) of IOPS on each OSD server right after the 19.2.2 upgrade but it is not necessarily the sign of a problem. I guess it may in particular be a consequence of the new deep scrubs parameters, osd_shallow_scrub_chunk_min/max, which a probably intended to reduced the deep scrub IOPS load. The release notes for Squid also mention a change in osd_op_num_shards_hdd and osd_op_num_threads_per_shard_hdd, I don't know if they may also have impact. Up to now, no users has reported any issue so it seems to be a problem only with scrubs. I'm wondering where to start looking for an issue or anything related to 19.2.2 is already known. We increased the deep scrub interval from 10 to 14 days a few days before the upgrade (we saw that there was permanently 1 deep scrub late, a different PG all the time) and kept the standard 7 day interval for scrubs. Looking at the number of scrubs and deep scrubs per day, it doesn't look weird (see below). I guess that if we retart all OSDs we'll clear the problem but we'd like to understand what happened and be sure it is not something related to Squid, before upgrade our other production cluster. Any hint/advice will be highly appreciated. I took a snapshot of `ceph pg dump pgs_brief` regularly and I'll try to identify if there are some stucked scrubs and what are the OSDs involved but with 500 OSDs and 18 OSD servers, it may not be obvious... Best regards, Michel _Distribution of scrubs_ (first number is the number of scrubs during the day) (number increases on June 6 after setting osd_max_scrubs=3) 295 "2025-05-25 1578 "2025-05-26 300 "2025-05-27 392 "2025-05-28 578 "2025-05-29 707 "2025-05-30 611 "2025-05-31 819 "2025-06-01 679 "2025-06-02 724 "2025-06-03 698 "2025-06-04 726 "2025-06-05 1577 "2025-06-06 1393 "2025-06-07 1962 "2025-06-08 645 "2025-06-09 _Distribution of deep scrubs per day_ (number increases on June 6 after setting osd_max_scrubs=3 and starts to decrease again on June 8, when the number of late subs increases again, probably because we hit the limit of 3 scrubs per OSD) 22 "2025-05-12 63 "2025-05-13 101 "2025-05-14 127 "2025-05-15 173 "2025-05-16 238 "2025-05-17 305 "2025-05-18 387 "2025-05-19 450 "2025-05-20 564 "2025-05-21 675 "2025-05-22 716 "2025-05-23 871 "2025-05-24 1071 "2025-05-25 801 "2025-05-26 188 "2025-05-27 292 "2025-05-28 335 "2025-05-29 409 "2025-05-30 371 "2025-05-31 514 "2025-06-01 440 "2025-06-02 504 "2025-06-03 478 "2025-06-04 546 "2025-06-05 1132 "2025-06-06 1022 "2025-06-07 662 "2025-06-08 227 "2025-06-09