How/when the osd_mclock_max_capacity_iops is updated?
Hi, Looking at our configuration during a recovery operation (misplaced objects after adding PG in a pool) that I found a bit slow, I saw that ~30 OSD over 500 had an entry for osd_mclock_max_capacity_iops_hdd in the configuration DB (and almost all the SSD). Despite all HDD and SSD being the same models (and the server HW being the same). As this value can have a significant impact on recovery time/speed with mclock, I was wondering if this value is periodically refreshed, in case one measurement was wrong (perturbated by external factors). The documentation is not very clear about this and seems to say that the bench used to estimate the IOPS capacity is run only when the OSD is activated (which may make sense as it is before there is any other activity on it)? I'm also wondering why we observe value like 350 on some OSD as the HW is the same on all servers? Currently, I tend to remove all the osd_mclock_max_capacity_iops_xxx entries from the DB periodically but it's probably not the right approach... Cheers, Michel
I use the following method, and haven't ever noticed it being interfered with: * set osd_mclock_skip_benchmark to true * remove any iops values that may have been added * run fio benchmarks on each model of disk * run a script that, for each osd, interrogates the drive model of the block device using smartctl and sets the iops value from my benchmark Running with the mclock profile set to balanced works really well for us (on squid 19.2.1, which performs better for us than reef did). Chris On 31/03/2025 09:07, Michel Jouvin wrote:
Hi,
Looking at our configuration during a recovery operation (misplaced objects after adding PG in a pool) that I found a bit slow, I saw that ~30 OSD over 500 had an entry for osd_mclock_max_capacity_iops_hdd in the configuration DB (and almost all the SSD). Despite all HDD and SSD being the same models (and the server HW being the same). As this value can have a significant impact on recovery time/speed with mclock, I was wondering if this value is periodically refreshed, in case one measurement was wrong (perturbated by external factors). The documentation is not very clear about this and seems to say that the bench used to estimate the IOPS capacity is run only when the OSD is activated (which may make sense as it is before there is any other activity on it)? I'm also wondering why we observe value like 350 on some OSD as the HW is the same on all servers?
Currently, I tend to remove all the osd_mclock_max_capacity_iops_xxx entries from the DB periodically but it's probably not the right approach...
Cheers,
Michel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm also wondering why we observe value like 350 on some OSD as the HW is the same on all servers?
Sometimes drive firmware correlates strongly with latency flern, I’ve experienced this with multiple SSD SKUs, especially with latency being inconsistent over time. So that’s one reason that drives within a cluster might behave differently, if some are downrev.
Yes, we have found the same. We always re-benchmark iops when a firmware upgrade is done, and ensure that new/replacement drives have a known and tested model/firmware revision. On 31/03/2025 15:58, Anthony D'Atri wrote:
I'm also wondering why we observe value like 350 on some OSD as the HW is the same on all servers?
Sometimes drive firmware correlates strongly with latency flern, I’ve experienced this with multiple SSD SKUs, especially with latency being inconsistent over time. So that’s one reason that drives within a cluster might behave differently, if some are downrev.
Hi, Thanks Anthony for raising our attention on this. We'll try to do the correlation matrix. In our case, I've seen this since the creation of the cluster 18 months ago with the initial batch of servers whee I would assume that all the drives had the same hardware... Best regards, Michel Sent from my mobile Le 31 mars 2025 17:40:28 Chris Palmer <chris.palmer@idnet.com> a écrit :
Yes, we have found the same. We always re-benchmark iops when a firmware upgrade is done, and ensure that new/replacement drives have a known and tested model/firmware revision.
On 31/03/2025 15:58, Anthony D'Atri wrote:
I'm also wondering why we observe value like 350 on some OSD as the HW is the same on all servers?
Sometimes drive firmware correlates strongly with latency flern, I’ve experienced this with multiple SSD SKUs, especially with latency being inconsistent over time. So that’s one reason that drives within a cluster might behave differently, if some are downrev.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Chris, Thanks for your feedback. We may go this way as I've the feeling it has no sense to have a value specific to a drive, independly of its model. And we were discussing upgrading to 19.2.1, so good to know that you have a good experience (We are currently running 18.2.2). Apart from this problem from time to time with the iops value, we have also a very good experience with mclock balanced profile. Cheers, Michel Le 31/03/2025 à 12:01, Chris Palmer a écrit :
I use the following method, and haven't ever noticed it being interfered with:
* set osd_mclock_skip_benchmark to true * remove any iops values that may have been added * run fio benchmarks on each model of disk * run a script that, for each osd, interrogates the drive model of the block device using smartctl and sets the iops value from my benchmark
Running with the mclock profile set to balanced works really well for us (on squid 19.2.1, which performs better for us than reef did).
Chris
On 31/03/2025 09:07, Michel Jouvin wrote:
Hi,
Looking at our configuration during a recovery operation (misplaced objects after adding PG in a pool) that I found a bit slow, I saw that ~30 OSD over 500 had an entry for osd_mclock_max_capacity_iops_hdd in the configuration DB (and almost all the SSD). Despite all HDD and SSD being the same models (and the server HW being the same). As this value can have a significant impact on recovery time/speed with mclock, I was wondering if this value is periodically refreshed, in case one measurement was wrong (perturbated by external factors). The documentation is not very clear about this and seems to say that the bench used to estimate the IOPS capacity is run only when the OSD is activated (which may make sense as it is before there is any other activity on it)? I'm also wondering why we observe value like 350 on some OSD as the HW is the same on all servers?
Currently, I tend to remove all the osd_mclock_max_capacity_iops_xxx entries from the DB periodically but it's probably not the right approach...
Cheers,
Michel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your feedback. We may go this way as I've the feeling it has no sense to have a value specific to a drive, independly of its model.
I disagree, I think this could - once stabilized - grow into detection of latent or subclinical drive issues. That unlike the SMART pass/fail attribute could be actually useful.
Anthony, Good point, I had this in mind originally but what I have seen since we updated to mclock a few years ago (we have 2 other Ceph cluster) is that the way it is handled currently leads to underperforming clusters because of too low value for iops capacity. And in my experience so far it has never been correlated to drives that start to misbehave. After removing the value set for an osd, it will not be set again and that should be the case if it was related to the drive. What would be great would be to have the possibility of an "advisory mode" where you could get the information computed by Ceph without enforcing it in mclock IO scheduling... Best regards. Michel Sent from my mobile Le 31 mars 2025 18:14:49 "Anthony D'Atri" <anthony.datri@gmail.com> a écrit :
Thanks for your feedback. We may go this way as I've the feeling it has no sense to have a value specific to a drive, independly of its model.
I disagree, I think this could - once stabilized - grow into detection of latent or subclinical drive issues. That unlike the SMART pass/fail attribute could be actually useful.
participants (3)
-
Anthony D'Atri
-
Chris Palmer
-
Michel Jouvin