Ceph reef and (slow) backfilling - how to speed it up
Dear ceph community, I’ve a ceph cluster which got upgraded from nautilus/pacific/…to reef over time. Now I added two new nodes to an existing EC pool as I did with the previous versions of ceph. Now I face the fact, that the previous „backfilling tuning“ I’v used by increasing injectargs --osd-max-backfills=XX --osd-recovery-max-active=YY dose not work anymore. With adjusting thous parameters the backfill was running with up to 2k +- objects/s. As I’m not (yet) familiar with the reef opiont the only speed up so far I found is „ceph config set osd osd_mclock_profile high_recovery_ops“ which currently runs the backfill with up to 600 opbjects/s. My question: What is a best (simple) way to speed that backfill up ? I’v tried to understand the custom profiles (?) but without success - and did not apply anything other yet. Thanks for feedback and suggestions ! Best regards . Götz
Hi Götz, You can change the value of osd_max_backfills (for all OSDs or specific ones) using `ceph config`, but you need enable osd_mclock_override_recovery_settings. See https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/#steps... for more information. Best regards, Pierre Riteau On Sat, 27 Apr 2024 at 08:32, Götz Reinicke <goetz.reinicke@filmakademie.de> wrote:
Dear ceph community,
I’ve a ceph cluster which got upgraded from nautilus/pacific/…to reef over time. Now I added two new nodes to an existing EC pool as I did with the previous versions of ceph.
Now I face the fact, that the previous „backfilling tuning“ I’v used by increasing injectargs --osd-max-backfills=XX --osd-recovery-max-active=YY dose not work anymore.
With adjusting thous parameters the backfill was running with up to 2k +- objects/s.
As I’m not (yet) familiar with the reef opiont the only speed up so far I found is „ceph config set osd osd_mclock_profile high_recovery_ops“ which currently runs the backfill with up to 600 opbjects/s.
My question: What is a best (simple) way to speed that backfill up ?
I’v tried to understand the custom profiles (?) but without success - and did not apply anything other yet.
Thanks for feedback and suggestions ! Best regards . Götz
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
For our customers we are still disabling mclock and using wpq. Might be worth trying. Mark On 4/30/24 09:08, Pierre Riteau wrote:
Hi Götz,
You can change the value of osd_max_backfills (for all OSDs or specific ones) using `ceph config`, but you need enable osd_mclock_override_recovery_settings. See https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/#steps... for more information.
Best regards, Pierre Riteau
On Sat, 27 Apr 2024 at 08:32, Götz Reinicke <goetz.reinicke@filmakademie.de> wrote:
Dear ceph community,
I’ve a ceph cluster which got upgraded from nautilus/pacific/…to reef over time. Now I added two new nodes to an existing EC pool as I did with the previous versions of ceph.
Now I face the fact, that the previous „backfilling tuning“ I’v used by increasing injectargs --osd-max-backfills=XX --osd-recovery-max-active=YY dose not work anymore.
With adjusting thous parameters the backfill was running with up to 2k +- objects/s.
As I’m not (yet) familiar with the reef opiont the only speed up so far I found is „ceph config set osd osd_mclock_profile high_recovery_ops“ which currently runs the backfill with up to 600 opbjects/s.
My question: What is a best (simple) way to speed that backfill up ?
I’v tried to understand the custom profiles (?) but without success - and did not apply anything other yet.
Thanks for feedback and suggestions ! Best regards . Götz
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
Hi Mark, On Thu, May 2, 2024 at 3:18 AM Mark Nelson <mark.nelson@clyso.com> wrote:
For our customers we are still disabling mclock and using wpq. Might be worth trying.
Could you please elaborate a bit on the issue(s) preventing the use of mClock. Is this specific to only the slow backfill rate and/or other issue? This feedback would help prioritize the improvements in those areas. Thanks, -Sridhar
Hi Sridhar, (Very!) Slow backfill was one issue, but if I recall we hit a case where backfill wasn't completing at all until we reverted to WPQ. I was getting hammered with other stuff at the time so I don't quite remember the details, but Dan might. I think this was in Quincy after the more recent updates landed in 17.2.6+ though. Mark On 5/2/24 00:05, Sridhar Seshasayee wrote:
Hi Mark,
On Thu, May 2, 2024 at 3:18 AM Mark Nelson <mark.nelson@clyso.com> wrote:
For our customers we are still disabling mclock and using wpq. Might be worth trying.
Could you please elaborate a bit on the issue(s) preventing the use of mClock. Is this specific to only the slow backfill rate and/or other issue?
This feedback would help prioritize the improvements in those areas.
Thanks, -Sridhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
For our customers we are still disabling mclock and using wpq. Might be worth trying.
Could you please elaborate a bit on the issue(s) preventing the use of mClock. Is this specific to only the slow backfill rate and/or other issue?
This feedback would help prioritize the improvements in those areas.
Multiple people -- including me -- have also observed backfill/recovery stop completely for no apparent reason. In some cases poking the lead OSD for a PG with `ceph osd down` restores, in other cases it doesn't. Anecdotally this *may* only happen for EC pools on HDDs but that sample size is small.
Multiple people -- including me -- have also observed backfill/recovery stop completely for no apparent reason.
In some cases poking the lead OSD for a PG with `ceph osd down` restores, in other cases it doesn't.
Anecdotally this *may* only happen for EC pools on HDDs but that sample size is small.
Thanks for the information. We will try and reproduce this locally with EC pools and investigate this further. I will revert with a tracker for this.
In our case it was with a EC pool as well. I believe the PG state was degraded+recovering / recovery_wait and iirc the PGs just simply sat in the recovering state without any progress (degraded PG object count did not decline). A repeer of the PG was attempted but no success there. A restart of all the OSDs for the given PGs was attempted under mclock. That didnt work. Switching to wpq for all OSDS in the given PG did resolve the issue. This was on a 17.2.7 cluster. Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Thu, May 2, 2024 at 9:54 AM Sridhar Seshasayee <sseshasa@redhat.com> wrote:
Multiple people -- including me -- have also observed backfill/recovery stop completely for no apparent reason.
In some cases poking the lead OSD for a PG with `ceph osd down` restores, in other cases it doesn't.
Anecdotally this *may* only happen for EC pools on HDDs but that sample size is small.
Thanks for the information. We will try and reproduce this locally with EC pools and investigate this further. I will revert with a tracker for this. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We have had pgs get stuck in quincy (17.2.7). After changing to wpq, no such problems were observed. We're using a replicated (x3) pool. On 2024-05-02 10:02, Wesley Dillingham wrote:
In our case it was with a EC pool as well. I believe the PG state was degraded+recovering / recovery_wait and iirc the PGs just simply sat in the recovering state without any progress (degraded PG object count did not decline). A repeer of the PG was attempted but no success there. A restart of all the OSDs for the given PGs was attempted under mclock. That didnt work. Switching to wpq for all OSDS in the given PG did resolve the issue. This was on a 17.2.7 cluster.
Respectfully,
*Wes Dillingham* LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com
On Thu, May 2, 2024 at 9:54 AM Sridhar Seshasayee<sseshasa@redhat.com> wrote:
Multiple people -- including me -- have also observed backfill/recovery stop completely for no apparent reason.
In some cases poking the lead OSD for a PG with `ceph osd down` restores, in other cases it doesn't.
Anecdotally this *may* only happen for EC pools on HDDs but that sample size is small.
Thanks for the information. We will try and reproduce this locally with EC pools and investigate this further. I will revert with a tracker for this. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
I halfway suspect that something akin to the speculation in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7MWAHAY7NCJK... is going on. Below are reservations reported by a random OSD that serves (mostly) an EC RGW bucket pool. This is with the mclock override on and the usual three backfill/recovery tunables set to 7 (bumped to get more OSDs backfilling after I changed to rack failure domain, having 50+ % of objects remapped makes me nervous and I want convergence. 3 happens to be the value of osd_recovery_max_active_hdd , so maybe there is some interaction between EC and how osd_recovery_max_active is derived and used? Complete wild-ass speculation. Just for grins, after `ceph osd down 313` * local_reservations incremented * remote_reservations decreased somewhat * cluster aggregate recovery speed increased for at least the short term [root@rook-ceph-osd-313-6f84bc5bd5-hr825 ceph]# ceph daemon osd.313 dump_recovery_reservations { "local_reservations": { "max_allowed": 7, "min_priority": 0, "queues": [], "in_progress": [ { "item": "21.161es0", "prio": 110, "can_preempt": true }, { "item": "21.180bs0", "prio": 110, "can_preempt": true }, { "item": "21.1e0as0", "prio": 110, "can_preempt": true } ] }, "remote_reservations": { "max_allowed": 7, "min_priority": 0, "queues": [ { "priority": 110, "items": [ { "item": "21.1d18s5", "prio": 110, "can_preempt": true }, { "item": "21.7d0s2", "prio": 110, "can_preempt": true }, { "item": "21.766s5", "prio": 110, "can_preempt": true }, { "item": "21.373s1", "prio": 110, "can_preempt": true }, { "item": "21.1a8es1", "prio": 110, "can_preempt": true }, { "item": "21.2das2", "prio": 110, "can_preempt": true }, { "item": "21.14a0s2", "prio": 110, "can_preempt": true }, { "item": "21.c7fs5", "prio": 110, "can_preempt": true }, { "item": "21.18e5s5", "prio": 110, "can_preempt": true }, { "item": "21.54ds2", "prio": 110, "can_preempt": true }, { "item": "21.79bs4", "prio": 110, "can_preempt": true }, { "item": "21.15c3s2", "prio": 110, "can_preempt": true }, { "item": "21.e15s4", "prio": 110, "can_preempt": true }, { "item": "21.226s3", "prio": 110, "can_preempt": true }, { "item": "21.adfs2", "prio": 110, "can_preempt": true }, { "item": "21.184bs4", "prio": 110, "can_preempt": true }, { "item": "21.f43s3", "prio": 110, "can_preempt": true }, { "item": "21.f5cs4", "prio": 110, "can_preempt": true }, { "item": "21.1300s3", "prio": 110, "can_preempt": true }, { "item": "21.11d8s3", "prio": 110, "can_preempt": true }, { "item": "21.5cfs5", "prio": 110, "can_preempt": true }, { "item": "21.1342s2", "prio": 110, "can_preempt": true }, { "item": "21.1846s5", "prio": 110, "can_preempt": true }, { "item": "21.c60s2", "prio": 110, "can_preempt": true }, { "item": "21.f1cs3", "prio": 110, "can_preempt": true }, { "item": "21.314s5", "prio": 110, "can_preempt": true }, { "item": "21.1be3s5", "prio": 110, "can_preempt": true } ] } ], "in_progress": [ { "item": "21.2fds0", "prio": 110, "can_preempt": true }, { "item": "21.4e6s1", "prio": 110, "can_preempt": true }, { "item": "21.c4cs3", "prio": 110, "can_preempt": true }, { "item": "21.f7as1", "prio": 110, "can_preempt": true }, { "item": "21.103es3", "prio": 110, "can_preempt": true }, { "item": "21.1d09s1", "prio": 110, "can_preempt": true }, { "item": "21.1f53s3", "prio": 110, "can_preempt": true } ] } }
On May 2, 2024, at 10:02, Wesley Dillingham <wes@wesdillingham.com> wrote:
In our case it was with a EC pool as well. I believe the PG state was degraded+recovering / recovery_wait and iirc the PGs just simply sat in the recovering state without any progress (degraded PG object count did not decline). A repeer of the PG was attempted but no success there. A restart of all the OSDs for the given PGs was attempted under mclock. That didnt work. Switching to wpq for all OSDS in the given PG did resolve the issue. This was on a 17.2.7 cluster.
Respectfully,
Wes Dillingham LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com <mailto:wes@wesdillingham.com>
On Thu, May 2, 2024 at 9:54 AM Sridhar Seshasayee <sseshasa@redhat.com <mailto:sseshasa@redhat.com>> wrote:
Multiple people -- including me -- have also observed backfill/recovery stop completely for no apparent reason.
In some cases poking the lead OSD for a PG with `ceph osd down` restores, in other cases it doesn't.
Anecdotally this *may* only happen for EC pools on HDDs but that sample size is small.
Thanks for the information. We will try and reproduce this locally with EC pools and investigate this further. I will revert with a tracker for this. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hi Götz, Please see my response below. On Tue, Apr 30, 2024 at 7:39 PM Pierre Riteau <pierre@stackhpc.com> wrote:
Hi Götz,
You can change the value of osd_max_backfills (for all OSDs or specific ones) using `ceph config`, but you need enable osd_mclock_override_recovery_settings. See
https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/#steps... for more information.
Did the suggestion from Pierre help improve the backfilling rate? With the mClock scheduler, this is the correct way of modifying the value of osd_max_backfills and osd_recovery_max_active. As to the observation of slower backfills, this is expected with the 'balanced' and 'high_client_ops' mClock profiles (see allocations here <https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#built-in-profiles>). This is due to backfill operation being classified as background best-effort service and a lower priority assigned to backfill operations when compared to degraded recoveries. Degraded recovery (or background recovery service) is given higher priority as there's a higher risk of data unavailability in case other OSDs in the cluster go down. Backfill operations are assigned lower priority since it just involves data movement. If the 'high_recovery_ops' profile coupled with increasing the above config parameters is still not enough to improve the backfilling rate, then the cluster must be examined to see if there are other competing services like degraded recoveries, client ops etc. that could affect the backfilling rate. The ceph status output should give an idea about this. -Sridhar
participants (7)
-
Anthony D'Atri
-
Götz Reinicke
-
Mark Nelson
-
Matthew Darwin
-
Pierre Riteau
-
Sridhar Seshasayee
-
Wesley Dillingham