Pour info, meme si pas en rapport avec nos pbs je pense puisqu'on tourne v18... Michel -------- Message transféré -------- Sujet : [ceph-users] Re: Squid: deep scrub issues Date : Wed, 27 Nov 2024 17:15:32 +0100 (CET) De : Frédéric Nass <frederic.nass@univ-lorraine.fr> Pour : Laimis Juzeliūnas <laimis.juzeliunas@oxylabs.io> Copie à : ceph-users <ceph-users@ceph.io> Hi Laimis, Might be the result of osd_scrub_chunk_max now being 15 instead of 25 previously. See [1] and [2]. Cheers, Frédéric. [1] https://tracker.ceph.com/issues/68057 [2] https://github.com/ceph/ceph/pull/59791/commits/0841603023ba53923a986f2fb96a... ----- Le 26 Nov 24, à 23:36, Laimis Juzeliūnas laimis.juzeliunas@oxylabs.io a écrit :

Hello Ceph community,

Wanted to highlight one observation and gather any Squid users having similar experiences. Since upgrading to 19.2.0 (from 18.4.0) we have observed that pg deep scrubbing times have drastically increased. Some pgs take 2-5 days to complete deep scrubbing while others increase to 20+ days. This causes the deep scrubbing queue to fill up and the cluster almost constantly has 'pgs not deep-scrubbed in time' alerts. We have on average 67 pgs/osd: running on 15TB hdd disks this results in 200GB-ish pgs. While fairly large - these pgs did not cause such increase in deep scrubs when on Reef.

"ceph pg dump | grep 'deep scrubbing for'" will always have a few entries of quite morbid scrubs like the following: 7.3e 121289 0 0 0 0 225333247207 0 0 127 0 127 active+clean+scrubbing+deep 2024-11-13T09:37:42.549418+0000 490179'5220664 490179:23902923 [268,27,122] 268 [268,27,122] 268 483850'5203141 2024-11-02T11:33:57.835277+0000 472713'5197481 2024-10-11T04:30:00.639763+0000 0 21873 deep scrubbing for 1169147s 34.247 62618 0 0 0 0 179797964677 0 0 101 50 101 active+clean+scrubbing+deep 2024-11-05T06:27:52.288785+0000 490179'22729571 490179:80672442 [34,97,25] 34 [34,97,25] 34 481331'22436869 2024-10-23T16:06:50.092439+0000 471395'22289914 2024-10-07T19:29:26.115047+0000 0 204864 deep scrubbing for 1871733s

Not pointing any fingers but Squid release had "better scrub scheduling" announced. Though this is not scheduling directly, but maybe this change had any impact causing such behaviour?

Scrubbing configurations: ceph config get osd | grep scrub global advanced osd_deep_scrub_interval 2678400.000000 global advanced osd_deep_scrub_large_omap_object_key_threshold 500000 global advanced osd_max_scrubs 5 global advanced osd_scrub_auto_repair true global advanced osd_scrub_max_interval 2678400.000000 global advanced osd_scrub_min_interval 172800.000000

Cluster details (backfilling expected and caused by some manual reweights): cluster: id: 96df99f6-fc1a-11ea-90a4-6cb3113cb732 health: HEALTH_WARN 24 pgs not deep-scrubbed in time

services: mon: 5 daemons, quorum ceph-node004,ceph-node003,ceph-node001,ceph-node005,ceph-node002 (age 4d) mgr: ceph-node001.hgythj(active, since 11d), standbys: ceph-node002.jphtvg mds: 20/20 daemons up, 12 standby osd: 384 osds: 384 up (since 25h), 384 in (since 5d); 5 remapped pgs rbd-mirror: 2 daemons active (2 hosts) rgw: 64 daemons active (32 hosts, 1 zones)

data: volumes: 1/1 healthy pools: 14 pools, 8681 pgs objects: 758.42M objects, 1.5 PiB usage: 4.6 PiB used, 1.1 PiB / 5.7 PiB avail pgs: 275177/2275254543 objects misplaced (0.012%) 6807 active+clean 989 active+clean+scrubbing+deep 880 active+clean+scrubbing 5 active+remapped+backfilling

io: client: 37 MiB/s rd, 59 MiB/s wr, 1.72k op/s rd, 439 op/s wr recovery: 70 MiB/s, 38 objects/s

One thread of other users experiencing same 19.2.0 prolonged deep scrub issues: https://www.reddit.com/r/ceph/comments/1guynak/strange_issue_where_scrubdeep... Any hints or help would be greately appreciated!

Thanks in advance, Laimis J. laimis.juzeliunas@oxylabs.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Anthony, No we dont have any hours set - scrubbing happens at all times. The only thing we changed from default and kept was increasing osd_max_scrubs to 5 to try and catch up. Other than that it was just expanding the window of scrubbing intervals as pgs not deep-scrubbed in time alerts kept hitting us. And yes, there are some pgs taking 20+ days to complete deep scrubs - thats visible in the pg dump with entries like "deep scrubbing for 1871733s". They do complete eventually though. Most of pgs take 2 to 5-7 days for deep scrubbing to finish. I'll try reducing osd_scrub_chunk_max from 25 to 15 as suggested by Frédéric and see if that can help solving this. Thanks, Laimis J.

Hi Laimis, I apologize for not paying attention to the Reddit link/discussion in your previous message. Forget about osd_scrub_chunk_max. It's very unlikely to explain why scrubbing is so slow that it doesn't progress (if at all) for many v19.2 users. Given the number of testimonies and recent modifications in the code related to scrubbing, I would encourage you to create a bug report in the tracker so that this issue can be investigated. Cheers, Frédéric. ________________________________ De : Frédéric Nass Envoyé : mercredi 27 novembre 2024 17:17 À : Laimis Juzeliūnas Cc: ceph-users Objet : [ceph-users] Re: Squid: deep scrub issues Hi Laimis, Might be the result of osd_scrub_chunk_max now being 15 instead of 25 previously. See [1] and [2]. Cheers, Frédéric. [1] https://tracker.ceph.com/issues/68057 [2] https://github.com/ceph/ceph/pull/59791/commits/0841603023ba53923a986f2fb96a... ----- Le 26 Nov 24, à 23:36, Laimis Juzeliūnas laimis.juzeliunas@oxylabs.io a écrit :

Hello Ceph community,

Wanted to highlight one observation and gather any Squid users having similar experiences. Since upgrading to 19.2.0 (from 18.4.0) we have observed that pg deep scrubbing times have drastically increased. Some pgs take 2-5 days to complete deep scrubbing while others increase to 20+ days. This causes the deep scrubbing queue to fill up and the cluster almost constantly has 'pgs not deep-scrubbed in time' alerts. We have on average 67 pgs/osd: running on 15TB hdd disks this results in 200GB-ish pgs. While fairly large - these pgs did not cause such increase in deep scrubs when on Reef.

"ceph pg dump | grep 'deep scrubbing for'" will always have a few entries of quite morbid scrubs like the following: 7.3e      121289                   0         0          0        0  225333247207 0           0   127         0       127  active+clean+scrubbing+deep 2024-11-13T09:37:42.549418+0000     490179'5220664    490179:23902923 [268,27,122]         268   [268,27,122]             268     483850'5203141 2024-11-02T11:33:57.835277+0000     472713'5197481 2024-10-11T04:30:00.639763+0000              0                21873  deep scrubbing for 1169147s 34.247     62618                   0         0          0        0  179797964677 0           0   101        50       101  active+clean+scrubbing+deep 2024-11-05T06:27:52.288785+0000    490179'22729571    490179:80672442 [34,97,25]          34     [34,97,25]              34    481331'22436869 2024-10-23T16:06:50.092439+0000    471395'22289914 2024-10-07T19:29:26.115047+0000              0               204864  deep scrubbing for 1871733s

Not pointing any fingers but Squid release had "better scrub scheduling" announced. Though this is not scheduling directly, but maybe this change had any impact causing such behaviour?

Scrubbing configurations: ceph config get osd | grep scrub global        advanced  osd_deep_scrub_interval 2678400.000000 global        advanced  osd_deep_scrub_large_omap_object_key_threshold  500000 global        advanced  osd_max_scrubs                                  5 global        advanced  osd_scrub_auto_repair                           true global        advanced  osd_scrub_max_interval 2678400.000000 global        advanced  osd_scrub_min_interval 172800.000000

Cluster details (backfilling expected and caused by some manual reweights):   cluster:     id:     96df99f6-fc1a-11ea-90a4-6cb3113cb732     health: HEALTH_WARN             24 pgs not deep-scrubbed in time

  services:     mon:        5 daemons, quorum     ceph-node004,ceph-node003,ceph-node001,ceph-node005,ceph-node002 (age 4d)     mgr:        ceph-node001.hgythj(active, since 11d), standbys:     ceph-node002.jphtvg     mds:        20/20 daemons up, 12 standby     osd:        384 osds: 384 up (since 25h), 384 in (since 5d); 5 remapped pgs     rbd-mirror: 2 daemons active (2 hosts)     rgw:        64 daemons active (32 hosts, 1 zones)

  data:     volumes: 1/1 healthy     pools:   14 pools, 8681 pgs     objects: 758.42M objects, 1.5 PiB     usage:   4.6 PiB used, 1.1 PiB / 5.7 PiB avail     pgs:     275177/2275254543 objects misplaced (0.012%)              6807 active+clean              989  active+clean+scrubbing+deep              880  active+clean+scrubbing              5    active+remapped+backfilling

  io:     client:   37 MiB/s rd, 59 MiB/s wr, 1.72k op/s rd, 439 op/s wr     recovery: 70 MiB/s, 38 objects/s

One thread of other users experiencing same 19.2.0 prolonged deep scrub issues: https://www.reddit.com/r/ceph/comments/1guynak/strange_issue_where_scrubdeep... Any hints or help would be greately appreciated!

Thanks in advance, Laimis J. laimis.juzeliunas@oxylabs.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

So is this recommend for all new squid clusters? osd_scrub_chunk_max from 25

To clarify, Squid reduced osd_scrub_chunk_max from 25 to 15 to limit the impact on client I/Os which may had led to increased (deep)scrubbing times. My advise was to raise this value back to 25 and see the influence of this change. But clearly, this is a more serious matter.

Thank you for creating tracker [1]. I'll do my best to ensure it gets the appropriate visibility.

Cheers, Frédéric.

[1] https://tracker.ceph.com/issues/69078

----- Le 28 Nov 24, à 22:58, Laimis Juzeliūnas laimis.juzeliunas@oxylabs.io a écrit :

...

Hi all, sveikas,

Thanks everyone for the tips and trying to help out! I've eventually raised a bug tracker for the case to get more developers involved: https://tracker.ceph.com/issues/69078

We tried decreasing osd_scrub_chunk_max from 25 to 15 as per Frédéric suggestion, but unfortunately did not observe any signs relief. One Squid user in reddit community thread confirmed the same after decreasing - no results. There are more users there in the thread that tried out various cluster configuration tunings, including osd_mclock_profile with high_recovery_ops but still no one managed to get any good results.

Our scrub cycle runs 24/7 with no time windows/schedules therefore no possibilities of queue buildups due to time constrains. And yes - our longest running pg now is 23 days in the deep scrub (and still counting).

Laimis J. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi all, Just came back from this years Cephalocon and managed to get a quick chat with Ronen regarding this issue. He had a great presentation[1, 2] on the upcoming changes to scrubbing in Tentacle as well as some changes already made in Squid release. The primary suspect here is the mclock scheduler and the way replica reservations are made with since 19.2.0. Regular scrubs begin by the primary requesting all acting-set replicas to allow the scrub to continue, each replica either grants the request immediately or queues it. As I understand previous releases instead of queuing would send a simple deny on the spot in case of thinned resources (that happens when the scrub map is asked for from the acting set members, but I might be wrong). For some reason with mclock this can lead to acting sets constantly queuing these scrub requests and never actually completing. As for the configuraiton goes: in Squid osd_scrub_cost config that has been increased to 52428800 for some reason. I'm having a hard time finding previous values but [3] redhat docs have this value set at 50 << 20. Unless the whole logic/calculation has changed such an abyssmal value will simply never allow resources to be granted with mclock. Another suspect is osd_scrub_event_cost which has been set to 4096. Once again having a hard time to find any previous version values for it to compare. One thing we've found that there is now a config option osd_scrub_disable_reservation_queuing (default - false): "When set - scrub replica reservations are responded to immediately, with either success or failure (the pre-Squid version behaviour). This configuration option is introduced to support mixed-version clusters and debugging, and will be removed in the next release." My guess is that setting this to true would simply return scrubbing options back to Reef and previous releases. To keep all the work done with scrubbing changes in place we will try reducing osd_scrub_cost to a much lower value (50 or even less) and check if that helps our case. If not, we will reduce osd_scrub_event_cost as well as we're not sure at this point which one of these have the direct impact. If that wont help we will have to set osd_scrub_disable_reservation_queuing to true, but that will leave us simply with an old way scrubs are done (not cool - we want the fancy new way). If that wont help we will have to start thinking of switching to wpq instead of mclock, which is also not that cool looking into the future of Ceph. I'll keep the mailing list (and tracker) updated with our findings. Best, Laimis J. 1 - https://ceph2024.sched.com/event/1ktWh/the-scrub-type-to-limitations-matrix-... 2 - https://static.sched.com/hosted_files/ceph2024/08/ceph24_main%20%284%29.pdf 3 - https://docs.redhat.com/en/documentation/red_hat_ceph_storage/2/html/configu...