31 Mar
2023
31 Mar
'23
4:14 p.m.
Hi,
I am reading reading some documentation about mClock and have two questions.
First, about the IOPS. Are those IOPS disk IOPS or other kind of IOPS? And what the
assumption of those? (Like block size, sequential or random reads/writes)?
And the second question,
How mClock calculates its profiles? I have my lab cluster running Quincy, and I have this
parameters for mClock:
"osd_mclock_max_capacity_iops_hdd": "450.000000",
"osd_mclock_profile": "balanced",
According to the documentation:
https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/#bala… I am
expecting to have:
"osd_mclock_scheduler_background_best_effort_lim": "999999",
"osd_mclock_scheduler_background_best_effort_res": "90",
"osd_mclock_scheduler_background_best_effort_wgt": "2",
"osd_mclock_scheduler_background_recovery_lim": "675",
"osd_mclock_scheduler_background_recovery_res": "180",
"osd_mclock_scheduler_background_recovery_wgt": "1",
"osd_mclock_scheduler_client_lim": "450",
"osd_mclock_scheduler_client_res": "180",
"osd_mclock_scheduler_client_wgt": "1",
But what I get is:
"osd_mclock_scheduler_background_best_effort_lim": "999999",
"osd_mclock_scheduler_background_best_effort_res": "18",
"osd_mclock_scheduler_background_best_effort_wgt": "2",
"osd_mclock_scheduler_background_recovery_lim": "135",
"osd_mclock_scheduler_background_recovery_res": "36",
"osd_mclock_scheduler_background_recovery_wgt": "1",
"osd_mclock_scheduler_client_lim": "90",
"osd_mclock_scheduler_client_res": "36",
"osd_mclock_scheduler_client_wgt": "1",
Which seems very low according to what my disk seems to be able to handle.
Is this calculation the expected one? Or did I miss something on how those profiles are
populated?
Luis Domingues
Proton AG
3 Apr
3 Apr
6:43 a.m.
Hi Luis,
I am reading reading some documentation about mClock and have two questions.
First, about the IOPS. Are those IOPS disk IOPS or other kind of IOPS? And
what the assumption of those? (Like block size, sequential or random
reads/writes)?
This is the result of testing running OSD bench random writes at 4 KiB
block size.
But what I get is:
"osd_mclock_scheduler_background_best_effort_lim": "999999",
"osd_mclock_scheduler_background_best_effort_res": "18",
"osd_mclock_scheduler_background_best_effort_wgt": "2",
"osd_mclock_scheduler_background_recovery_lim": "135",
"osd_mclock_scheduler_background_recovery_res": "36",
"osd_mclock_scheduler_background_recovery_wgt": "1",
"osd_mclock_scheduler_client_lim": "90",
"osd_mclock_scheduler_client_res": "36",
"osd_mclock_scheduler_client_wgt": "1",
Which seems very low according to what my disk seems to be able to handle.
Is this calculation the expected one? Or did I miss something on how those
profiles are populated?
The above values are a result of distributing the IOPS across all the OSD
shards as defined by the
osd_op_num_shards_[hdd|ssd] option. For HDDs, this is set to 5 and
therefore the IOPS will be
distributed across the 5 shards (i.e. for e.g., 675/5 for
osd_mclock_scheduler_background_recovery_lim
and so on for other reservation and limit options).
-Sridhar
7:32 a.m.
Hi Sridhar
Thanks for the information.
The above values are a result of distributing the IOPS across all the OSD
shards as defined by the
osd_op_num_shards_[hdd|ssd] option. For HDDs, this is set to 5 and
therefore the IOPS will be
distributed across the 5 shards (i.e. for e.g., 675/5 for
osd_mclock_scheduler_background_recovery_lim
and so on for other reservation and limit options).
Why was it done that way? I do not understand the reason why distributing the IOPS accross
different disks, when the measurement we have is for one disk alone. This means with
default parameters we will always be far from reaching OSD limit right?
Luis Domingues
Proton AG
------- Original Message -------
On Monday, April 3rd, 2023 at 07:43, Sridhar Seshasayee <sseshasa(a)redhat.com>
wrote:
> Hi Luis,
>
>
> I am reading reading some documentation about mClock and have two questions.
>
> > First, about the IOPS. Are those IOPS disk IOPS or other kind of IOPS? And
> > what the assumption of those? (Like block size, sequential or random
> > reads/writes)?
>
>
> This is the result of testing running OSD bench random writes at 4 KiB
> block size.
>
> > But what I get is:
> >
> > "osd_mclock_scheduler_background_best_effort_lim":
"999999",
> > "osd_mclock_scheduler_background_best_effort_res": "18",
> > "osd_mclock_scheduler_background_best_effort_wgt": "2",
> > "osd_mclock_scheduler_background_recovery_lim": "135",
> > "osd_mclock_scheduler_background_recovery_res": "36",
> > "osd_mclock_scheduler_background_recovery_wgt": "1",
> > "osd_mclock_scheduler_client_lim": "90",
> > "osd_mclock_scheduler_client_res": "36",
> > "osd_mclock_scheduler_client_wgt": "1",
> >
> > Which seems very low according to what my disk seems to be able to handle.
> >
> > Is this calculation the expected one? Or did I miss something on how those
> > profiles are populated?
>
The above values are a result of distributing the IOPS across all the OSD
shards as defined by the
osd_op_num_shards_[hdd|ssd] option. For HDDs, this is set to 5 and
therefore the IOPS will be
distributed across the 5 shards (i.e. for e.g., 675/5 for
osd_mclock_scheduler_background_recovery_lim
and so on for other reservation and limit options).
>
> -Sridhar
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:44 a.m.
Why was it done that way? I do not understand the reason why distributing
the IOPS accross different disks, when the measurement
we have is for one
disk alone. This means with default parameters we will always be far from
reaching OSD limit right?
It's not on different disks. We distribute the IOPS across shards on a
given
OSD/disk. This is an internal implementation detail.
This means in your case, 450 IOPS is distributed across 5 shards on the
same OSD/disk. You can think of it as 5 threads
being allocated a share of the total IOPS on a given OSD.
-Sridhar
11:14 a.m.
Hi,
Thanks a lot for the information.
I have a last question. Why is the bench performed using writes of 4 KiB. Is any reason to
choose that over another another value?
On my lab, I tested with various values, and I have mainly two type of disks. Some
Seagates and Toshiba.
If I do bench with 4KiB, what I get from Seagate is a result around 2000 IOPS. While the
Toshiba is more arround 600.
If I do bench with 128KiB, I still have results arround 2000 IOPS for Seagate, but Toshiba
also bench arround 2000 IOPS. And from the rados experiment I did, having
osd_mclock_max_capacity_iops_hdd set to 2000 on that lab setup is the value I get the most
performance from my rados experiments, both with Segate and Toshiba disks.
Luis Domingues
Proton AG
------- Original Message -------
On Monday, April 3rd, 2023 at 08:44, Sridhar Seshasayee <sseshasa(a)redhat.com>
wrote:
> Why was it done that way? I do not understand the reason why distributing
>
> > the IOPS accross different disks, when the measurement we have is for one
> > disk alone. This means with default parameters we will always be far from
> > reaching OSD limit right?
> >
> > It's not on different disks. We distribute the IOPS across shards on a
>
> given OSD/disk. This is an internal implementation detail.
> This means in your case, 450 IOPS is distributed across 5 shards on the
> same OSD/disk. You can think of it as 5 threads
> being allocated a share of the total IOPS on a given OSD.
> -Sridhar
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:43 p.m.
Responses inline.
I have a last question. Why is the bench performed using writes of 4 KiB.
Is any reason to choose that over another another
value?
Yes, the mClock scheduler considers this as a baseline in order to
estimate costs
for operations involving other block sizes.
This is again an internal implementation detail.
On my lab, I tested with various values, and I have mainly two type of
disks. Some Seagates and Toshiba.
If I do bench with 4KiB, what I get from Seagate is a result around 2000
IOPS. While the Toshiba is more arround 600.
If I do bench with 128KiB, I still have results arround 2000 IOPS for
Seagate, but Toshiba also bench arround 2000 IOPS. And from the rados
experiment I did, having osd_mclock_max_capacity_iops_hdd set to 2000 on
that lab setup is the value I get the most performance from my rados
experiments, both with Segate and Toshiba disks.
I would currently suggest setting osd_mclock_max_capacity_iops_hdd to
values you
measured with fio as that is more realistic.
Like I mentioned, there are some improvements coming around this area that
would allow users to have greater control on
setting a realistic benchmark value.
-Sridhar
419
days inactive
422
days old
5 comments
2 participants
participants (2)
-
Luis Domingues -
Sridhar Seshasayee