25 Sep
2020
25 Sep
'20
12:42 p.m.
Hi Ceph maintainers and developers,
The objective of this is to discuss our work on a dmClock based client QoS management for
CephFS.
Our group at LINE maintains Ceph storage clusters such as RGW, RBD, and CephFS to
internally support OpenStack and K8S based private cloud environment for various
applications and platforms including LINE messenger. We have seen that the RGW and RBD
services can provide consistent performance to multiple active users since RGW employes
the dmClock QoS scheduler for S3 clients and hypervisors internally utilize I/O throttler
for VM block storage clients. Unfortunately, unlike RGW and RBD, CephFS clients can
directly issue metadata requests to MDSs and filedata requests OSDs as they want. This
situation occasionally (or frequently) happens and the other client performance may be
degraded by the noisy neighbor. In the end, consistent performance cannot be guaranteed
in our environment. From this observation and motivation, we are now considering the
client QoS scheduler using the dmClock library for CephFS.
A few things about how to realize the QoS scheduler.
- Per subvolume QoS management. IOPS resources are only shared among the clients that
mount the same root directory. QoS parameters can be easily configured through the
extended attributes (similar to quota). Each dmClock scheduler can manage clients'
requests using client session information.
- MDS QoS management. Client metadata requests like create, lookup, and etc. are managed
by dmClock scheduler placed between the dispatcher and the main request handler (e.g.,
Server::handle_client_request()). We have observed that two active MDSs provide
approximately 20KIOPS. As performance capacity is sometimes scarce for lots of clients,
QoS management is needed for MDS.
- OSD QoS management. We would like to reopen and improve the previous work available at
https://github.com/ceph/ceph/pull/20235.
- Client QoS management. Each client manages the dmClock tracker to keep track of both rho
and delta to be packed to client request messages.
In case of the CLI, QoS parameters are configured using the extended attributes on each
subvolume directory. Specifically, separate QoS configurations are considered for both
MDSs and OSDs.
setfattr -n ceph.dmclock.mds_reservation -v 200
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.mds_weight -v 500
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.mds_limit -v 1000
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.osd_reservation -v 500
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.osd_weight -v 1000
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.osd_limit -v 2000
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
Our QoS work has been kicked off from the previous month. Our first step is to go over the
prior work and dmClock algorithm/library. Now we are actively focusing on checking the
feasibility of our idea with some modifications to MDS and ceph-fuse. Our development is
planned as follows.
- dmClock scheduler will be integrated into MDS and ceph-fuse by December 2020.
- dmClock scheduler will be incorporated with OSD by the first half of the next year.
Does the community have any plan to develop per client QoS management? Are there any other
issues related to our QoS work? We are looking forward to hearing your valuable comments
and feedback at an early stage.
Thanks
Yongseok Oh
29 Sep
29 Sep
6:26 a.m.
Hello Yongseok,
That is a very exciting project! We are also interested in
improving QoS. For Pacific, we've been focusing on OSD-side
mclock and balancing background work vs clients.
Sridhar is doing extensive testing and fixed a few issues with
the op scheduler integration and dmclock library, e.g.:
https://github.com/ceph/ceph/pull/37031
https://github.com/ceph/ceph/pull/37431
It is looking quite promising, we think osd_op_queue =
mclock_scheduler will be in good shape for Pacific.
There are a couple outstanding settings changes to make this
work:
1) bluestore throttling
If bluestore is not throttled enough, the osd op queue is empty
and mclock has no chance to prioritize requests.
On our nvme test hardware bluestore_throttle_bytes and
bluestore_throttle_deferred_bytes set to 128K allows us to get
near full throughput with the fewest ops in flight at the
bluestore level.
2) osd op queue sharding
The mclock algorithm works well with only a single
queue (osd_op_num_shards = 1). Further testing determined that
keeping the same total amount of parallelism
(osd_op_num_threads_per_shard = 16) allows us to get the same
performance as the default of 8 shards and 2 threads per shard.
Extending this to the client side like
https://github.com/ceph/ceph/pull/20235 would be great. There may
be some challenges left there, in terms of accounting for
in-flight I/Os. Eric, do you recall if you had found a solution
for that?
CephFS is challenging with that model - if multiple clients are
accessing the same subvolume, and they are meant to be limited in
aggregate, their independent dmClock trackers will not know about
in-flight I/O from other clients. Have you considered how to
resolve this?
Patrick, Greg, any other comments from a CephFS point of view?
Josh
On 9/25/20 3:42 AM, Yongseok Oh wrote:
Hi Ceph maintainers and developers,
The objective of this is to discuss our work on a dmClock based client QoS management for
CephFS.
Our group at LINE maintains Ceph storage clusters such as RGW, RBD, and CephFS to
internally support OpenStack and K8S based private cloud environment for various
applications and platforms including LINE messenger. We have seen that the RGW and RBD
services can provide consistent performance to multiple active users since RGW employes
the dmClock QoS scheduler for S3 clients and hypervisors internally utilize I/O throttler
for VM block storage clients. Unfortunately, unlike RGW and RBD, CephFS clients can
directly issue metadata requests to MDSs and filedata requests OSDs as they want. This
situation occasionally (or frequently) happens and the other client performance may be
degraded by the noisy neighbor. In the end, consistent performance cannot be guaranteed
in our environment. From this observation and motivation, we are now considering the
client QoS scheduler using the dmClock library for CephFS.
A few things about how to realize the QoS scheduler.
- Per subvolume QoS management. IOPS resources are only shared among the clients that
mount the same root directory. QoS parameters can be easily configured through the
extended attributes (similar to quota). Each dmClock scheduler can manage clients'
requests using client session information.
- MDS QoS management. Client metadata requests like create, lookup, and etc. are managed
by dmClock scheduler placed between the dispatcher and the main request handler (e.g.,
Server::handle_client_request()). We have observed that two active MDSs provide
approximately 20KIOPS. As performance capacity is sometimes scarce for lots of clients,
QoS management is needed for MDS.
- OSD QoS management. We would like to reopen and improve the previous work available at
https://github.com/ceph/ceph/pull/20235.
- Client QoS management. Each client manages the dmClock tracker to keep track of both
rho and delta to be packed to client request messages.
In case of the CLI, QoS parameters are configured using the extended attributes on each
subvolume directory. Specifically, separate QoS configurations are considered for both
MDSs and OSDs.
setfattr -n ceph.dmclock.mds_reservation -v 200
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.mds_weight -v 500
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.mds_limit -v 1000
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.osd_reservation -v 500
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.osd_weight -v 1000
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.osd_limit -v 2000
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
Our QoS work has been kicked off from the previous month. Our first step is to go over
the prior work and dmClock algorithm/library. Now we are actively focusing on checking the
feasibility of our idea with some modifications to MDS and ceph-fuse. Our development is
planned as follows.
- dmClock scheduler will be integrated into MDS and ceph-fuse by December 2020.
- dmClock scheduler will be incorporated with OSD by the first half of the next year.
Does the community have any plan to develop per client QoS management? Are there any
other issues related to our QoS work? We are looking forward to hearing your valuable
comments and feedback at an early stage.
Thanks
Yongseok Oh
30 Sep
30 Sep
3:04 a.m.
On Mon, Sep 28, 2020 at 9:26 PM Josh Durgin <jdurgin(a)redhat.com> wrote:
Hello Yongseok,
That is a very exciting project! We are also interested in
improving QoS. For Pacific, we've been focusing on OSD-side
mclock and balancing background work vs clients.
Sridhar is doing extensive testing and fixed a few issues with
the op scheduler integration and dmclock library, e.g.:
https://github.com/ceph/ceph/pull/37031
https://github.com/ceph/ceph/pull/37431
It is looking quite promising, we think osd_op_queue =
mclock_scheduler will be in good shape for Pacific.
There are a couple outstanding settings changes to make this
work:
1) bluestore throttling
If bluestore is not throttled enough, the osd op queue is empty
and mclock has no chance to prioritize requests.
On our nvme test hardware bluestore_throttle_bytes and
bluestore_throttle_deferred_bytes set to 128K allows us to get
near full throughput with the fewest ops in flight at the
bluestore level.
2) osd op queue sharding
The mclock algorithm works well with only a single
queue (osd_op_num_shards = 1). Further testing determined that
keeping the same total amount of parallelism
(osd_op_num_threads_per_shard = 16) allows us to get the same
performance as the default of 8 shards and 2 threads per shard.
Extending this to the client side like
https://github.com/ceph/ceph/pull/20235 would be great. There may
be some challenges left there, in terms of accounting for
in-flight I/Os. Eric, do you recall if you had found a solution
for that?
CephFS is challenging with that model - if multiple clients are
accessing the same subvolume, and they are meant to be limited in
aggregate, their independent dmClock trackers will not know about
in-flight I/O from other clients. Have you considered how to
resolve this?
Patrick, Greg, any other comments from a CephFS point of view?
That's basically what I've got. CephFS is tricky because we naively
expect the clients to be scaling out as well; most single-mounter
scenarios just end up on rbd because it's simpler.
We are starting to talk about maybe being able to do QoS in the
NFS-Ganesha frontend for NFS users (apparently it recently added some
tooling for this), but even then any real deployment will have
multiple active servers and I'm not sure if we can "typically" route
to a single one to track the global state. I could envision a system
where the clients periodically share their activity with the MDS, and
the MDS partitions their total QoS limits between any clients working
on the same data based on models of their changing "heat", but it
would still be really complicated and be prone to a lot of
over-limiting or burstiness. :/
-Greg
Josh
On 9/25/20 3:42 AM, Yongseok Oh wrote:
Hi Ceph maintainers and developers,
The objective of this is to discuss our work on a dmClock based client QoS management for
CephFS.
Our group at LINE maintains Ceph storage clusters such as RGW, RBD, and CephFS to
internally support OpenStack and K8S based private cloud environment for various
applications and platforms including LINE messenger. We have seen that the RGW and RBD
services can provide consistent performance to multiple active users since RGW employes
the dmClock QoS scheduler for S3 clients and hypervisors internally utilize I/O throttler
for VM block storage clients. Unfortunately, unlike RGW and RBD, CephFS clients can
directly issue metadata requests to MDSs and filedata requests OSDs as they want. This
situation occasionally (or frequently) happens and the other client performance may be
degraded by the noisy neighbor. In the end, consistent performance cannot be guaranteed
in our environment. From this observation and motivation, we are now considering the
client QoS scheduler using the dmClock library for CephFS.
A few things about how to realize the QoS scheduler.
- Per subvolume QoS management. IOPS resources are only shared among the clients that
mount the same root directory. QoS parameters can be easily configured through the
extended attributes (similar to quota). Each dmClock scheduler can manage clients'
requests using client session information.
- MDS QoS management. Client metadata requests like create, lookup, and etc. are managed
by dmClock scheduler placed between the dispatcher and the main request handler (e.g.,
Server::handle_client_request()). We have observed that two active MDSs provide
approximately 20KIOPS. As performance capacity is sometimes scarce for lots of clients,
QoS management is needed for MDS.
- OSD QoS management. We would like to reopen and improve the previous work available at
https://github.com/ceph/ceph/pull/20235.
- Client QoS management. Each client manages the dmClock tracker to keep track of both
rho and delta to be packed to client request messages.
In case of the CLI, QoS parameters are configured using the extended attributes on each
subvolume directory. Specifically, separate QoS configurations are considered for both
MDSs and OSDs.
setfattr -n ceph.dmclock.mds_reservation -v 200
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.mds_weight -v 500
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.mds_limit -v 1000
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.osd_reservation -v 500
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.osd_weight -v 1000
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
setfattr -n ceph.dmclock.osd_limit -v 2000
/volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55
Our QoS work has been kicked off from the previous month. Our first step is to go over
the prior work and dmClock algorithm/library. Now we are actively focusing on checking the
feasibility of our idea with some modifications to MDS and ceph-fuse. Our development is
planned as follows.
- dmClock scheduler will be integrated into MDS and ceph-fuse by December 2020.
- dmClock scheduler will be incorporated with OSD by the first half of the next year.
Does the community have any plan to develop per client QoS management? Are there any
other issues related to our QoS work? We are looking forward to hearing your valuable
comments and feedback at an early stage.
Thanks
Yongseok Oh
1 Oct
1 Oct
4:49 p.m.
Hi Josh and Greg,
Let me try to recall major issues related to dmClock QoS scheduler from the previous PRs
and your helpful comments.
[MDS]
- NFS Ganesha. Exploiting NFS Ganesha is one of the possible solutions to the client QoS.
But, I think introducing a new layer has the potential to cause limitations in terms of
overall performance or scalability. It cannot also cover the CephFS native clients.
- Multiple different clients on the same subvolume. In this case, subvolume is probably
shared with multiple clients, where each client maintains their own QoS tracker, resulting
in inappropriate scheduling. In other words, another layer (or process) must monitor the
global QoS state among clients. Now we can simply devise a client group approach that each
client group constitutes numerous clients and allocated IOPS are divided equally or based
on a policy. For instance, there are four clients in a group running on the same volume
and 1000 IOPS are given per group, each client can satisfy a 250 IOPS. Another approach
(more complex) is that a client perf monitor classifies clients' workloads and
dynamically tunes and reallocates their IOPS based on workloads.
[OSD]
- multiple OP queue shards per OSD. Previously, they have mentioned [1] that it is
difficult to provide the right QoS with the less in-flight request as the number of shards
increases and requests are distributed to shards. To overcome this issue, they have
proposed two solutions. One is that the number of shards is just set to ‘1’. The other is
Outstanding I/O, namely OIO, throttler that gathers many requests as much as possible with
the insertion of short latency. As clients cannot distinguish between shards in an OSD,
they have come up with the shard identifier along with OSD ID [2]. For background
operations, normalized rho/delta values are calculated based on their average
numbers [3].
[Compatibility]
- One of the feature bits needs to be allotted to QoS for compatibility. Since the bits
are very limited resources, further discussion and confirmation by maintainers are
required [4].
[References]
[1] http://bos.itdks.com/f54f1c93811d419398edb8b8c9cb35d0.pdf
[2] https://github.com/ceph/ceph/pull/16369
[3] https://github.com/ceph/ceph/pull/18280
[4] https://github.com/ceph/ceph/pull/17450
[5] https://github.com/ceph/ceph/pull/20235
I think rebasing the PR [5] to the master has a high priority to recover the latest state
and check the feasibility.
Thanks
Yongseok
16 Oct
16 Oct
2:17 a.m.
Hi Yongseok, apologies for the delayed response.
On 10/1/20 7:49 AM, Yongseok Oh wrote:
Hi Josh and Greg,
Let me try to recall major issues related to dmClock QoS scheduler from the previous PRs
and your helpful comments.
[MDS]
- NFS Ganesha. Exploiting NFS Ganesha is one of the possible solutions to the client
QoS. But, I think introducing a new layer has the potential to cause limitations in terms
of overall performance or scalability. It cannot also cover the CephFS native clients.
- Multiple different clients on the same subvolume. In this case, subvolume is probably
shared with multiple clients, where each client maintains their own QoS tracker, resulting
in inappropriate scheduling. In other words, another layer (or process) must monitor the
global QoS state among clients. Now we can simply devise a client group approach that each
client group constitutes numerous clients and allocated IOPS are divided equally or based
on a policy. For instance, there are four clients in a group running on the same volume
and 1000 IOPS are given per group, each client can satisfy a 250 IOPS. Another approach
(more complex) is that a client perf monitor classifies clients' workloads and
dynamically tunes and reallocates their IOPS based on workloads.
This seems like a good approach.
[OSD]
- multiple OP queue shards per OSD. Previously, they have mentioned [1] that it is
difficult to provide the right QoS with the less in-flight request as the number of shards
increases and requests are distributed to shards. To overcome this issue, they have
proposed two solutions. One is that the number of shards is just set to ‘1’. The other is
Outstanding I/O, namely OIO, throttler that gathers many requests as much as possible with
the insertion of short latency. As clients cannot distinguish between shards in an OSD,
they have come up with the shard identifier along with OSD ID [2]. For background
operations, normalized rho/delta values are calculated based on their average numbers [3].
Settings shards to 1 is simpler, and has shown the same performance when
we keep the total number of threads constant (i.e. 1 shard x 16
threads, instead of today's default of 8 x 2 on ssd). We'll propose
changing the defaults to reflect this before Pacific.
[Compatibility]
- One of the feature bits needs to be allotted to QoS for compatibility. Since the bits
are very limited resources, further discussion and confirmation by maintainers are
required [4].
We may be able to piggyback on the SERVER_PACIFIC feature bit to avoid
consuming another one. Ilya, any concern about that from the kernel
client?
[References]
[1] http://bos.itdks.com/f54f1c93811d419398edb8b8c9cb35d0.pdf
[2] https://github.com/ceph/ceph/pull/16369
[3] https://github.com/ceph/ceph/pull/18280
[4] https://github.com/ceph/ceph/pull/17450
[5] https://github.com/ceph/ceph/pull/20235
I think rebasing the PR [5] to the master has a high priority to recover the latest state
and check the feasibility.
Agreed.
Regards,
Josh
2:29 p.m.
On Fri, Oct 16, 2020 at 2:17 AM Josh Durgin <jdurgin(a)redhat.com> wrote:
Hi Yongseok, apologies for the delayed response.
On 10/1/20 7:49 AM, Yongseok Oh wrote:
That depends entirely on what else is going to be covered by
SERVER_PACIFIC. Unless it's just client aware QoS, we should try
to avoid conditioning anything else that is client facing (or can
potentially become client facing in the future) on it. One example
we ran into in the past was a X+2 or X+3 version of some encoding
depending on an otherwise optional feature (i.e. no way to get the
version with a new client facing field without adding some form of
support for the feature).
Thanks,
Ilya
Hi Josh and Greg,
Let me try to recall major issues related to dmClock QoS scheduler from the previous PRs
and your helpful comments.
[MDS]
- NFS Ganesha. Exploiting NFS Ganesha is one of the possible solutions to the client
QoS. But, I think introducing a new layer has the potential to cause limitations in terms
of overall performance or scalability. It cannot also cover the CephFS native clients.
- Multiple different clients on the same subvolume. In this case, subvolume is probably
shared with multiple clients, where each client maintains their own QoS tracker, resulting
in inappropriate scheduling. In other words, another layer (or process) must monitor the
global QoS state among clients. Now we can simply devise a client group approach that each
client group constitutes numerous clients and allocated IOPS are divided equally or based
on a policy. For instance, there are four clients in a group running on the same volume
and 1000 IOPS are given per group, each client can satisfy a 250 IOPS. Another approach
(more complex) is that a client perf monitor classifies clients' workloads and
dynamically tunes and reallocates their IOPS based on workloads.
This seems like a good approach.
[OSD]
- multiple OP queue shards per OSD. Previously, they have mentioned [1] that it is
difficult to provide the right QoS with the less in-flight request as the number of shards
increases and requests are distributed to shards. To overcome this issue, they have
proposed two solutions. One is that the number of shards is just set to ‘1’. The other is
Outstanding I/O, namely OIO, throttler that gathers many requests as much as possible with
the insertion of short latency. As clients cannot distinguish between shards in an OSD,
they have come up with the shard identifier along with OSD ID [2]. For background
operations, normalized rho/delta values are calculated based on their average numbers [3].
Settings shards to 1 is simpler, and has shown the same performance when
we keep the total number of threads constant (i.e. 1 shard x 16
threads, instead of today's default of 8 x 2 on ssd). We'll propose
changing the defaults to reflect this before Pacific.
[Compatibility]
- One of the feature bits needs to be allotted to QoS for compatibility. Since the bits
are very limited resources, further discussion and confirmation by maintainers are
required [4].
We may be able to piggyback on the SERVER_PACIFIC feature bit to avoid
consuming another one. Ilya, any concern about that from the kernel
client?
26 Oct
26 Oct
12:44 p.m.
Hi Josh and maintainers,
We have confirmed again that the dmClock algorithm does not address multiple clients
running on the same subvolume. In the dmClock, the client tracker monitors how much
workload has been performed on each server and acts as a sort of scheduler through sending
rho/delta values the servers. Our simple idea was that the total QoS IOPS is divided into
multiple clients within a subvolume based on the workload dynamics or evenly manner. For
example, assuming that 1000 IOPS is allocated to a subvolume and then the value is shared
to periodically multiple clients. If 100 clients simultaneously issue requests on the same
volume, each client can consume 10 IOPS by mClock scheduler.
Like this, we can consider a client workload metric-based approach, but it is not easy to
ensure QoS stability as client workloads are dynamically changed and the time period to
obtain metrics affects the allocation accuracy. Additionally, per client session QoS can
instead be considered, but it is difficult to predict and limit the number of sessions in
the subvolume.
For this reason, instead of applying dmClock, mClock scheduler can be considered as a good
solution to the noisy neighbor problem. Expected per subvolume QoS IOPS can also be
calculated as follows. (Assuming MDS and OSD requests are almost evenly distributed and
reservation and weight values are omitted for brief exploration.)
[MDS]
- Per MDS Limit IOPS * # of MDSs (Perf MDS Limit IOPS * 1 when ephemeral random pinning is
configured.)
[OSD]
- Per OSD Limit IOPS * # of OSDs
Of course, depending on the workload or server conditions, the above equations may not be
100% satisfied. However, the QoS scheduler can be implemented without the client and
manager modifications.
There are any other comments from a CephFS point of view?
Finally, we are going to implement our prototype that the mClock scheduler is applied to
MDS and then make a pull request to share and discuss them.
Thanks
Yongseok
4:55 p.m.
Hi Yongseok,
I’m guessing you know this already, but dmClock becomes mClock if rho and delta are both
0. Within the dmClock library, the ServiceTracker class, which is run on the clients,
tracks the rho and delta values. But you don’t have to use a ServiceTracker. In fact, in
the 3 current uses of the dmClock library in ceph master (osd, crimson osd, rgw), none of
them currently use a ServiceTracker, so they’re essentially getting mClock.
The other thing that’s worth considering is that “server” and “client’ can be viewed in an
abstract sense. So in the osd, for example, the mClock “clients" are not the true
clients, but instead the different classification of operations.
One of the nice things about dmClock is that the “servers” do not need to communicate
directly among themselves in order to provide QoS. The “clients” provide extra information
to the servers that allow them to compensate for the work of the other servers.
Eric
--
J. Eric Ivancich
he / him / his
Red Hat Storage
Ann Arbor, Michigan, USA
On Oct 26, 2020, at 7:44 AM, Yongseok Oh
<yongseok.oh(a)linecorp.com> wrote:
Hi Josh and maintainers,
We have confirmed again that the dmClock algorithm does not address multiple clients
running on the same subvolume. In the dmClock, the client tracker monitors how much
workload has been performed on each server and acts as a sort of scheduler through sending
rho/delta values the servers. Our simple idea was that the total QoS IOPS is divided into
multiple clients within a subvolume based on the workload dynamics or evenly manner. For
example, assuming that 1000 IOPS is allocated to a subvolume and then the value is shared
to periodically multiple clients. If 100 clients simultaneously issue requests on the same
volume, each client can consume 10 IOPS by mClock scheduler.
Like this, we can consider a client workload metric-based approach, but it is not easy to
ensure QoS stability as client workloads are dynamically changed and the time period to
obtain metrics affects the allocation accuracy. Additionally, per client session QoS can
instead be considered, but it is difficult to predict and limit the number of sessions in
the subvolume.
For this reason, instead of applying dmClock, mClock scheduler can be considered as a
good solution to the noisy neighbor problem. Expected per subvolume QoS IOPS can also be
calculated as follows. (Assuming MDS and OSD requests are almost evenly distributed and
reservation and weight values are omitted for brief exploration.)
[MDS]
- Per MDS Limit IOPS * # of MDSs (Perf MDS Limit IOPS * 1 when ephemeral random pinning
is configured.)
[OSD]
- Per OSD Limit IOPS * # of OSDs
Of course, depending on the workload or server conditions, the above equations may not be
100% satisfied. However, the QoS scheduler can be implemented without the client and
manager modifications.
There are any other comments from a CephFS point of view?
Finally, we are going to implement our prototype that the mClock scheduler is applied to
MDS and then make a pull request to share and discuss them.
Thanks
Yongseok
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
10 Dec
10 Dec
1:56 a.m.
Hi maintainers,
We have implemented this scheduler idea and made a pull request as working in progress
https://github.com/ceph/ceph/pull/38506. I think there are some technical challenges that
we need to discuss. It may not have been fully implemented yet, so if maintainers give us
feedback, we would like to actively improve it together. If there is a chance, I would
like to ask for your interest and review.
Yongseok
1233
days inactive
1309
days old
8 comments
5 participants
participants (5)
-
Eric Ivancich -
Gregory Farnum -
Ilya Dryomov -
Josh Durgin -
Yongseok Oh