When is osd_mclock_capacity_iops_hdd/ssd updated?
Hi, We have a production cluster homogeneous in terms of HDD devices and I'm a little bit surprised that the osd_mclock_capacity_iops_hdd computed for each OSD is so different (basically a x2 between the lowest and highest values). I understand that sometimes the OSD bench is perturbated by external factors leading to an inappropriate value and if it is too low, it may impact in particular deep scrub duration. The documentation [1] explains that the OSD bench is run at OSD initialization, which is, I guess, startup time. It also suggests that it can be run manually but I guess it makes no sense if the OSD is up because of the concurrent activity. Am I right? How is the bench command supposed to be run as I guess it doesn't work if the OSD is not up? Also, the documentation explains that you can define a value that you measured and seems to suggest that once defined, it will not be updated. Am I right? If yes, does it mean that once the automatic bench has determined a value the only way to update it is to delete it from the config and restart the OSD (if you want the automatic bench to update/redefine it)? Best regards, Michel [1] https://docs.ceph.com/en/squid/rados/configuration/mclock-config-ref/
Den tis 10 juni 2025 kl 18:59 skrev Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:
a little bit surprised that the osd_mclock_capacity_iops_hdd computed for each OSD is so different (basically a x2 between the lowest and highest values).
Also, the documentation explains that you can define a value that you measured and seems to suggest that once defined, it will not be updated. Am I right? If yes, does it mean that once the automatic bench has determined a value the only way to update it is to delete it from the config and restart the OSD (if you want the automatic bench to update/redefine it)?
I think your assessment is correct on all details. I guess you would take a decent value from the high end of your range and set it on all drives, to "compensate" for the tests being done in various times. Not necessarily the exact highest, but if it was showing between 100 to 200 iops, then perhaps 150 or 175 could be reasonable for all drives, and unless it causes problems just leave it there for the hdd drives. It's hard from the outside to tell if it is worse that it becomes only 100 for one or some drives because it was tested when the system was a bit more busy than usual, and hence get less io scheduled to it (scrubs and repairs and so on), compared to how bad it would be if one drive actually only can deliver 100 for some reason and you hard code it to 150 so it is given 50% too many non-client-IO requests. -- May the most significant bit of your life be positive.
Janne, Thanks for your answer, I'll do as you suggest and see if we observe negative side effects. We are struggling with slow deep scrubs (like described in https://tracker.ceph.com/issues/69078) and I'm wondering if the OSDs with low values may contribute to the problem... Michel Le 11/06/2025 à 09:37, Janne Johansson a écrit :
Den tis 10 juni 2025 kl 18:59 skrev Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:
a little bit surprised that the osd_mclock_capacity_iops_hdd computed for each OSD is so different (basically a x2 between the lowest and highest values). Also, the documentation explains that you can define a value that you measured and seems to suggest that once defined, it will not be updated. Am I right? If yes, does it mean that once the automatic bench has determined a value the only way to update it is to delete it from the config and restart the OSD (if you want the automatic bench to update/redefine it)? I think your assessment is correct on all details. I guess you would take a decent value from the high end of your range and set it on all drives, to "compensate" for the tests being done in various times. Not necessarily the exact highest, but if it was showing between 100 to 200 iops, then perhaps 150 or 175 could be reasonable for all drives, and unless it causes problems just leave it there for the hdd drives. It's hard from the outside to tell if it is worse that it becomes only 100 for one or some drives because it was tested when the system was a bit more busy than usual, and hence get less io scheduled to it (scrubs and repairs and so on), compared to how bad it would be if one drive actually only can deliver 100 for some reason and you hard code it to 150 so it is given 50% too many non-client-IO requests.
We derive appropriate values for each model/firmware of disk using fio tests, and set them explicitly. We set osd_mclock_skip_benchmark = true to get rid of the inbuilt benchmark altogether. I don't know what the algorithm is for automatically re-running the inbuilt benchmark, but with that parameter set to true it never cuts in. Using that approach we have never had a problem with mclock. Under reef we found it pretty good; under squid it is better again. And in most cases better than wpq. Chris On 11/06/2025 09:12, Michel Jouvin wrote:
Janne,
Thanks for your answer, I'll do as you suggest and see if we observe negative side effects. We are struggling with slow deep scrubs (like described in https://tracker.ceph.com/issues/69078) and I'm wondering if the OSDs with low values may contribute to the problem...
Michel
Le 11/06/2025 à 09:37, Janne Johansson a écrit :
Den tis 10 juni 2025 kl 18:59 skrev Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:
a little bit surprised that the osd_mclock_capacity_iops_hdd computed for each OSD is so different (basically a x2 between the lowest and highest values). Also, the documentation explains that you can define a value that you measured and seems to suggest that once defined, it will not be updated. Am I right? If yes, does it mean that once the automatic bench has determined a value the only way to update it is to delete it from the config and restart the OSD (if you want the automatic bench to update/redefine it)? I think your assessment is correct on all details. I guess you would take a decent value from the high end of your range and set it on all drives, to "compensate" for the tests being done in various times. Not necessarily the exact highest, but if it was showing between 100 to 200 iops, then perhaps 150 or 175 could be reasonable for all drives, and unless it causes problems just leave it there for the hdd drives. It's hard from the outside to tell if it is worse that it becomes only 100 for one or some drives because it was tested when the system was a bit more busy than usual, and hence get less io scheduled to it (scrubs and repairs and so on), compared to how bad it would be if one drive actually only can deliver 100 for some reason and you hard code it to 150 so it is given 50% too many non-client-IO requests.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, Do manual OSD benchmark for hdd/ssd https://docs.ceph.com/en/squid/rados/configuration/mclock-config-ref/#global... Change value globaly https://docs.ceph.com/en/squid/rados/configuration/mclock-config-ref/#global... Or per OSD ceph config set osd.X osd_mclock_max_capacity_iops_hdd 900.000000 ceph config set osd.X osd_mclock_max_capacity_iops_ssd 74000.000000 Change: ceph config set osd_scrub_load_threshold 10.000000 It should help mClock work better. On Wed, 11 Jun 2025 at 11:13, Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:
Janne,
Thanks for your answer, I'll do as you suggest and see if we observe negative side effects. We are struggling with slow deep scrubs (like described in https://tracker.ceph.com/issues/69078) and I'm wondering if the OSDs with low values may contribute to the problem...
Michel
Le 11/06/2025 à 09:37, Janne Johansson a écrit :
Den tis 10 juni 2025 kl 18:59 skrev Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:
a little bit surprised that the osd_mclock_capacity_iops_hdd computed for each OSD is so different (basically a x2 between the lowest and highest values). Also, the documentation explains that you can define a value that you measured and seems to suggest that once defined, it will not be updated. Am I right? If yes, does it mean that once the automatic bench has determined a value the only way to update it is to delete it from the config and restart the OSD (if you want the automatic bench to update/redefine it)? I think your assessment is correct on all details. I guess you would take a decent value from the high end of your range and set it on all drives, to "compensate" for the tests being done in various times. Not necessarily the exact highest, but if it was showing between 100 to 200 iops, then perhaps 150 or 175 could be reasonable for all drives, and unless it causes problems just leave it there for the hdd drives. It's hard from the outside to tell if it is worse that it becomes only 100 for one or some drives because it was tested when the system was a bit more busy than usual, and hence get less io scheduled to it (scrubs and repairs and so on), compared to how bad it would be if one drive actually only can deliver 100 for some reason and you hard code it to 150 so it is given 50% too many non-client-IO requests.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Chris Palmer
-
Janne Johansson
-
Kristaps Cudars
-
Michel Jouvin