On 23/08/2023 08:27, Sridhar Seshasayee wrote:
This also leads me to agree with you there's 'something wrong' with the mclock scheduler. I was almost starting to suspect hardware issues or something like that, I was at my wit's end.
Could you update this thread with the exact quincy version by running:
$ ceph versions
and
$ ceph config show-with-defaults osd.N | grep osd_mclock
Please replace N with any valid OSD id.
I suspect that the quincy version you are running on doesn't have the latest changes we made to the Reef upstream release. Recent changes introduced significant improvements to the mClock profiles and address slow recovery/backfill rates. The improvements to the mClock profiles should also help throttle snaptrim operations.
Snaptrim operation with mClock currently uses a static cost as defined by osd_snap_trim_cost. There are improvements planned around this soon. For e.g., the cost must be dynamic and reflect the size of the object being trimmed. -Sridhar
Here's the requested info, even though I'm going to stay on wpq for a while. # ceph versions { "mon": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 3 }, "osd": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 117 }, "mds": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 3 }, "overall": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 126 } } I'm using the docker image registry.<company>.local/quay-proxy/ceph/ceph@sha256:673b48521fd53e1b4bc7dda96335505c4d4b2e13d7bb92bf2e7782e2083094c9. # ceph config show-with-defaults osd.0 | grep osd_mclock osd_mclock_cost_per_byte_usec 0.000000 default osd_mclock_cost_per_byte_usec_hdd 2.600000 default osd_mclock_cost_per_byte_usec_ssd 0.011000 default osd_mclock_cost_per_io_usec 0.000000 default osd_mclock_cost_per_io_usec_hdd 11400.000000 default osd_mclock_cost_per_io_usec_ssd 50.000000 default osd_mclock_force_run_benchmark_on_init false default osd_mclock_iops_capacity_threshold_hdd 500.000000 default osd_mclock_iops_capacity_threshold_ssd 80000.000000 default osd_mclock_max_capacity_iops_hdd 250.000000 mon osd_mclock_max_capacity_iops_ssd 21500.000000 default osd_mclock_override_recovery_settings true mon osd_mclock_profile high_client_ops mon osd_mclock_scheduler_anticipation_timeout 0.000000 default osd_mclock_scheduler_background_best_effort_lim 999999 default osd_mclock_scheduler_background_best_effort_res 1 default osd_mclock_scheduler_background_best_effort_wgt 1 default osd_mclock_scheduler_background_recovery_lim 999999 default osd_mclock_scheduler_background_recovery_res 1 default osd_mclock_scheduler_background_recovery_wgt 1 default osd_mclock_scheduler_client_lim 999999 default osd_mclock_scheduler_client_res 1 default osd_mclock_scheduler_client_wgt 1 default osd_mclock_skip_benchmark false default # ceph config show-with-defaults osd.0 | grep trim_cost osd_snap_trim_cost 1048576 default Angelo.