Hi. Fresh cluster - but despite setting: jskr@dkcphhpcmgt028:/$ sudo ceph config show osd.0 | grep recovery_max_active_ssd osd_recovery_max_active_ssd 50 mon default[20] jskr@dkcphhpcmgt028:/$ sudo ceph config show osd.0 | grep osd_max_backfills osd_max_backfills 100 mon default[10] I still get jskr@dkcphhpcmgt028:/$ sudo ceph status cluster: id: 5c384430-da91-11ed-af9c-c780a5227aff health: HEALTH_OK services: mon: 3 daemons, quorum dkcphhpcmgt031,dkcphhpcmgt029,dkcphhpcmgt028 (age 16h) mgr: dkcphhpcmgt031.afbgjx(active, since 33h), standbys: dkcphhpcmgt029.bnsegi, dkcphhpcmgt028.bxxkqd mds: 2/2 daemons up, 1 standby osd: 40 osds: 40 up (since 45h), 40 in (since 39h); 21 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 495 pgs objects: 24.85M objects, 60 TiB usage: 117 TiB used, 159 TiB / 276 TiB avail pgs: 10655690/145764002 objects misplaced (7.310%) 474 active+clean 15 active+remapped+backfilling 6 active+remapped+backfill_wait io: client: 0 B/s rd, 1.4 MiB/s wr, 0 op/s rd, 116 op/s wr recovery: 328 MiB/s, 108 objects/s progress: Global Recovery Event (9h) [==========================..] (remaining: 25m) With these numbers for the setting - I would expect to get more than 15 active backfilling... (and based on SSD's and 2x25gbit network, I can also spend more resources on recovery than 328 MiB/s Thanks, . -- Jesper Krogh
Hi Jesper, Indeed many users reported slow backfilling and recovery with the mclock scheduler. This is supposed to be fixed in the latest quincy but clearly something is still slowing things down. Some clusters have better luck reverting to osd_op_queue = wpq. (I'm hoping by proposing this someone who tuned mclock recently will chime in with better advice). Cheers, Dan ______________________________________________________ Clyso GmbH | Ceph Support and Consulting | https://www.clyso.com On Wed, Jul 5, 2023 at 10:28 PM Jesper Krogh <jesper@krogh.cc> wrote:
Hi.
Fresh cluster - but despite setting: jskr@dkcphhpcmgt028:/$ sudo ceph config show osd.0 | grep recovery_max_active_ssd osd_recovery_max_active_ssd 50
mon default[20] jskr@dkcphhpcmgt028:/$ sudo ceph config show osd.0 | grep osd_max_backfills osd_max_backfills 100
mon default[10]
I still get jskr@dkcphhpcmgt028:/$ sudo ceph status cluster: id: 5c384430-da91-11ed-af9c-c780a5227aff health: HEALTH_OK
services: mon: 3 daemons, quorum dkcphhpcmgt031,dkcphhpcmgt029,dkcphhpcmgt028 (age 16h) mgr: dkcphhpcmgt031.afbgjx(active, since 33h), standbys: dkcphhpcmgt029.bnsegi, dkcphhpcmgt028.bxxkqd mds: 2/2 daemons up, 1 standby osd: 40 osds: 40 up (since 45h), 40 in (since 39h); 21 remapped pgs
data: volumes: 2/2 healthy pools: 9 pools, 495 pgs objects: 24.85M objects, 60 TiB usage: 117 TiB used, 159 TiB / 276 TiB avail pgs: 10655690/145764002 objects misplaced (7.310%) 474 active+clean 15 active+remapped+backfilling 6 active+remapped+backfill_wait
io: client: 0 B/s rd, 1.4 MiB/s wr, 0 op/s rd, 116 op/s wr recovery: 328 MiB/s, 108 objects/s
progress: Global Recovery Event (9h) [==========================..] (remaining: 25m)
With these numbers for the setting - I would expect to get more than 15 active backfilling... (and based on SSD's and 2x25gbit network, I can also spend more resources on recovery than 328 MiB/s
Thanks, .
-- Jesper Krogh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I recently got mclock going literally an order of magnitude faster. I would love to claim I found all the options myself but I collected the knowledge of what knobs I needed to turn from here. Steps I took: - Cleared all osd specific osd_mclock_max_capacity_iops settings. The auto created ones were all over the place. Some rust drives claimed 200 and others well over 5000. - Set sane global osd_mclock_max_capacity_iops_hdd and osd_mclock_max_capacity_iops_ssd numbers for my average lowest performance drive performance in my environment (your numbers will be different. These are for 18t SAS seagate rust drives and micron 9100 6.4t NVMe) - osd basic osd_mclock_max_capacity_iops_hdd 375.000000 - osd basic osd_mclock_max_capacity_iops_ssd 575000.000000 - Set the profile to what I wanted my global default to be. - osd advanced osd_mclock_profile high_client_ops - Tweaked the costs of doing operations -osd dev osd_mclock_cost_per_byte_usec_hdd 1.000000 - osd dev osd_mclock_cost_per_byte_usec_ssd 0.005000 I need to revisit the cost per byte settings. Originally I was using just this knob to play with speeds but I quickly starting getting many slow ops along with faster speeds. Then I pulled the max capacity iops down from 400 and finally settled where I am now. I have room for improvement here but this is my prod cluster so.. yeah. - Next I set specific faster drives to their own specific max capacity iops (optane drives I have for the metadata tier) - e.g. osd.450 basic osd_mclock_max_capacity_iops_ssd 785000.000000 - I also set the profile to specific drives in a tier I'm migrating to new spinners to "balanced" to speed that up. - e.g. osd.789 advanced osd_mclock_profile balanced I think that's about it. I was not scientific AT ALL with this. I just kept turning knobs a little and watching the recovery throughput and healthometer. On my cold EC tier rebalance I went from something like 150MB/s 20 obj/s to 2.1GB/s 750 obj/s. I know I'm pushing these drives pretty hard because I'm watching different drives claim 0 slow ops for N seconds, then a few min later clear. My replicated tier now recovers ridiculously fast as well. I'm looking forward to pulling all of this out and having ceph DoTheRightThing(tm) with recovery speeds. We shall see. -paul -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology “End users is a description, not a goal.” ________________________________________ From: Dan van der Ster <dan.vanderster@clyso.com> Sent: Thursday, July 6, 2023 6:04 PM To: Jesper Krogh Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Cannot get backfill speed up Hi Jesper, Indeed many users reported slow backfilling and recovery with the mclock scheduler. This is supposed to be fixed in the latest quincy but clearly something is still slowing things down. Some clusters have better luck reverting to osd_op_queue = wpq. (I'm hoping by proposing this someone who tuned mclock recently will chime in with better advice). Cheers, Dan ______________________________________________________ Clyso GmbH | Ceph Support and Consulting | https://www.clyso.com On Wed, Jul 5, 2023 at 10:28 PM Jesper Krogh <jesper@krogh.cc> wrote:
Hi.
Fresh cluster - but despite setting: jskr@dkcphhpcmgt028:/$ sudo ceph config show osd.0 | grep recovery_max_active_ssd osd_recovery_max_active_ssd 50
mon default[20] jskr@dkcphhpcmgt028:/$ sudo ceph config show osd.0 | grep osd_max_backfills osd_max_backfills 100
mon default[10]
I still get jskr@dkcphhpcmgt028:/$ sudo ceph status cluster: id: 5c384430-da91-11ed-af9c-c780a5227aff health: HEALTH_OK
services: mon: 3 daemons, quorum dkcphhpcmgt031,dkcphhpcmgt029,dkcphhpcmgt028 (age 16h) mgr: dkcphhpcmgt031.afbgjx(active, since 33h), standbys: dkcphhpcmgt029.bnsegi, dkcphhpcmgt028.bxxkqd mds: 2/2 daemons up, 1 standby osd: 40 osds: 40 up (since 45h), 40 in (since 39h); 21 remapped pgs
data: volumes: 2/2 healthy pools: 9 pools, 495 pgs objects: 24.85M objects, 60 TiB usage: 117 TiB used, 159 TiB / 276 TiB avail pgs: 10655690/145764002 objects misplaced (7.310%) 474 active+clean 15 active+remapped+backfilling 6 active+remapped+backfill_wait
io: client: 0 B/s rd, 1.4 MiB/s wr, 0 op/s rd, 116 op/s wr recovery: 328 MiB/s, 108 objects/s
progress: Global Recovery Event (9h) [==========================..] (remaining: 25m)
With these numbers for the setting - I would expect to get more than 15 active backfilling... (and based on SSD's and 2x25gbit network, I can also spend more resources on recovery than 328 MiB/s
Thanks, .
-- Jesper Krogh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Paul, On Fri, Jul 7, 2023 at 5:13 PM Paul Mezzanini <pfmeec@rit.edu> wrote:
I recently got mclock going literally an order of magnitude faster. I would love to claim I found all the options myself but I collected the knowledge of what knobs I needed to turn from here.
Significant usability and design improvements have been made to the mclock scheduler in the upstream Reef release. These improvements should soon be available in Quincy as well. One of the major goals is to reduce the number of knobs to tune and achieve a more hands free operation. This was partly achieved in the existing releases to some extent by eliminating the need to tune sleep and operation specific cost options. Here are some of the major improvements (from Reef release notes) that should help: 1. The balanced profile is set as the default mClock profile because it represents a compromise between prioritizing client IO or recovery IO. Users can then choose either the high_client_ops profile to prioritize client IO or the high_recovery_ops profile to prioritize recovery IO. 2. QoS parameters like reservation and limit are now specified in terms of a fraction (range: 0.0 to 1.0) of the OSD’s IOPS capacity. 3. The cost parameters - osd_mclock_cost_per_io_usec_* and osd_mclock_cost_per_byte_usec_* have been removed. The cost of an operation is now determined internally using the random IOPS and maximum sequential bandwidth capability of the OSD’s underlying device. 4. The random IOPS capacity is determined using 'osd bench' as before, but now based on the result, unrealistic values are not considered and reasonable defaults are used if the measurement crosses a threshold governed by osd_mclock_iops_capacity_threshold_[hdd|ssd]. The default IOPS capacity can still be overridden by users if not accurate, The thresholds too are configurable. The max sequential bandwidth is defined by osd_mclock_max_sequential_bandwidth_[hdd|ssd], and are set to reasonable defaults. Again, these may be modified if not accurate. Therefore, these changes account for inaccuracies and provide good control to the user in terms of specifying accurate OSD characteristics. 5. Degraded object recovery is given higher priority when compared to misplaced object recovery because degraded objects present a data safety issue not present with objects that are merely misplaced. Therefore, backfilling operations with the balanced and high_client_ops mClock profiles may progress slower than what was seen with the WeightedPriorityQueue (WPQ) scheduler. For faster recovery and backfills, the 'high_recovery_ops' profile with modified QoS parameters would help. 6. The QoS allocations in all the mClock profiles are optimized based on the above fixes and enhancements. Please see the latest upstream documentation for more details: https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/ The recommendation is to upgrade when feasible and provide your feedback, questions and suggestions. -Sridhar
participants (4)
-
Dan van der Ster
-
Jesper Krogh
-
Paul Mezzanini
-
Sridhar Seshasayee