On Mon, Sep 28, 2020 at 9:26 PM Josh Durgin <jdurgin@redhat.com> wrote:

Hello Yongseok,

That is a very exciting project! We are also interested in improving QoS. For Pacific, we've been focusing on OSD-side mclock and balancing background work vs clients.

Sridhar is doing extensive testing and fixed a few issues with the op scheduler integration and dmclock library, e.g.:

https://github.com/ceph/ceph/pull/37031 https://github.com/ceph/ceph/pull/37431

It is looking quite promising, we think osd_op_queue = mclock_scheduler will be in good shape for Pacific.

There are a couple outstanding settings changes to make this work:

1) bluestore throttling

If bluestore is not throttled enough, the osd op queue is empty and mclock has no chance to prioritize requests.

On our nvme test hardware bluestore_throttle_bytes and bluestore_throttle_deferred_bytes set to 128K allows us to get near full throughput with the fewest ops in flight at the bluestore level.

2) osd op queue sharding

The mclock algorithm works well with only a single queue (osd_op_num_shards = 1). Further testing determined that keeping the same total amount of parallelism (osd_op_num_threads_per_shard = 16) allows us to get the same performance as the default of 8 shards and 2 threads per shard.

Extending this to the client side like https://github.com/ceph/ceph/pull/20235 would be great. There may be some challenges left there, in terms of accounting for in-flight I/Os. Eric, do you recall if you had found a solution for that?

CephFS is challenging with that model - if multiple clients are accessing the same subvolume, and they are meant to be limited in aggregate, their independent dmClock trackers will not know about in-flight I/O from other clients. Have you considered how to resolve this?

Patrick, Greg, any other comments from a CephFS point of view?

That's basically what I've got. CephFS is tricky because we naively expect the clients to be scaling out as well; most single-mounter scenarios just end up on rbd because it's simpler. We are starting to talk about maybe being able to do QoS in the NFS-Ganesha frontend for NFS users (apparently it recently added some tooling for this), but even then any real deployment will have multiple active servers and I'm not sure if we can "typically" route to a single one to track the global state. I could envision a system where the clients periodically share their activity with the MDS, and the MDS partitions their total QoS limits between any clients working on the same data based on models of their changing "heat", but it would still be really complicated and be prone to a lot of over-limiting or burstiness. :/ -Greg

Josh

On 9/25/20 3:42 AM, Yongseok Oh wrote:

...

Hi Ceph maintainers and developers,

The objective of this is to discuss our work on a dmClock based client QoS management for CephFS.

Our group at LINE maintains Ceph storage clusters such as RGW, RBD, and CephFS to internally support OpenStack and K8S based private cloud environment for various applications and platforms including LINE messenger. We have seen that the RGW and RBD services can provide consistent performance to multiple active users since RGW employes the dmClock QoS scheduler for S3 clients and hypervisors internally utilize I/O throttler for VM block storage clients. Unfortunately, unlike RGW and RBD, CephFS clients can directly issue metadata requests to MDSs and filedata requests OSDs as they want. This situation occasionally (or frequently) happens and the other client performance may be degraded by the noisy neighbor. In the end, consistent performance cannot be guaranteed in our environment. From this observation and motivation, we are now considering the client QoS scheduler using the dmClock library for CephFS.

A few things about how to realize the QoS scheduler.

- Per subvolume QoS management. IOPS resources are only shared among the clients that mount the same root directory. QoS parameters can be easily configured through the extended attributes (similar to quota). Each dmClock scheduler can manage clients' requests using client session information. - MDS QoS management. Client metadata requests like create, lookup, and etc. are managed by dmClock scheduler placed between the dispatcher and the main request handler (e.g., Server::handle_client_request()). We have observed that two active MDSs provide approximately 20KIOPS. As performance capacity is sometimes scarce for lots of clients, QoS management is needed for MDS. - OSD QoS management. We would like to reopen and improve the previous work available at https://github.com/ceph/ceph/pull/20235. - Client QoS management. Each client manages the dmClock tracker to keep track of both rho and delta to be packed to client request messages.

In case of the CLI, QoS parameters are configured using the extended attributes on each subvolume directory. Specifically, separate QoS configurations are considered for both MDSs and OSDs.

setfattr -n ceph.dmclock.mds_reservation -v 200 /volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55 setfattr -n ceph.dmclock.mds_weight -v 500 /volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55 setfattr -n ceph.dmclock.mds_limit -v 1000 /volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55

setfattr -n ceph.dmclock.osd_reservation -v 500 /volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55 setfattr -n ceph.dmclock.osd_weight -v 1000 /volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55 setfattr -n ceph.dmclock.osd_limit -v 2000 /volumes/_nogroup/fdffc126-7961-4bbc-add2-2675b9e35a55

Our QoS work has been kicked off from the previous month. Our first step is to go over the prior work and dmClock algorithm/library. Now we are actively focusing on checking the feasibility of our idea with some modifications to MDS and ceph-fuse. Our development is planned as follows.

- dmClock scheduler will be integrated into MDS and ceph-fuse by December 2020. - dmClock scheduler will be incorporated with OSD by the first half of the next year.

Does the community have any plan to develop per client QoS management? Are there any other issues related to our QoS work? We are looking forward to hearing your valuable comments and feedback at an early stage.

Thanks

Yongseok Oh

Hi Yongseok, apologies for the delayed response. On 10/1/20 7:49 AM, Yongseok Oh wrote:

Hi Josh and Greg,

Let me try to recall major issues related to dmClock QoS scheduler from the previous PRs and your helpful comments.

[MDS] - NFS Ganesha.  Exploiting NFS Ganesha is one of the possible solutions to the client QoS. But, I think introducing a new layer has the potential to cause limitations in terms of overall performance or scalability. It cannot also cover the CephFS native clients.

- Multiple different clients on the same subvolume. In this case, subvolume is probably shared with multiple clients, where each client maintains their own QoS tracker, resulting in inappropriate scheduling. In other words, another layer (or process) must monitor the global QoS state among clients. Now we can simply devise a client group approach that each client group constitutes numerous clients and allocated IOPS are divided equally or based on a policy. For instance, there are four clients in a group running on the same volume and 1000 IOPS are given per group, each client can satisfy a 250 IOPS. Another approach (more complex) is that a client perf monitor classifies clients' workloads and dynamically tunes and reallocates their IOPS based on workloads.

This seems like a good approach.

[OSD] - multiple OP queue shards per OSD. Previously, they have mentioned [1] that it is difficult to provide the right QoS with the less in-flight request as the number of shards increases and requests are distributed to shards. To overcome this issue, they have proposed two solutions. One is that the number of shards is just set to ‘1’. The other is Outstanding I/O, namely OIO, throttler that gathers many requests as much as possible with the insertion of short latency. As clients cannot distinguish between shards in an OSD, they have come up with the shard identifier along with OSD ID [2]. For background operations, normalized rho/delta values are calculated based on their average numbers [3].

Settings shards to 1 is simpler, and has shown the same performance when we keep the total number of threads constant (i.e. 1 shard x 16 threads, instead of today's default of 8 x 2 on ssd). We'll propose changing the defaults to reflect this before Pacific.

[Compatibility] - One of the feature bits needs to be allotted to QoS for compatibility. Since the bits are very limited resources, further discussion and confirmation by maintainers are required [4].

We may be able to piggyback on the SERVER_PACIFIC feature bit to avoid consuming another one. Ilya, any concern about that from the kernel client?

[References] [1] http://bos.itdks.com/f54f1c93811d419398edb8b8c9cb35d0.pdf [2] https://github.com/ceph/ceph/pull/16369 [3] https://github.com/ceph/ceph/pull/18280 [4] https://github.com/ceph/ceph/pull/17450 [5] https://github.com/ceph/ceph/pull/20235

I think rebasing the PR [5] to the master has a high priority to recover the latest state and check the feasibility.

Agreed. Regards, Josh

Hi Josh and maintainers, We have confirmed again that the dmClock algorithm does not address multiple clients running on the same subvolume. In the dmClock, the client tracker monitors how much workload has been performed on each server and acts as a sort of scheduler through sending rho/delta values the servers. Our simple idea was that the total QoS IOPS is divided into multiple clients within a subvolume based on the workload dynamics or evenly manner. For example, assuming that 1000 IOPS is allocated to a subvolume and then the value is shared to periodically multiple clients. If 100 clients simultaneously issue requests on the same volume, each client can consume 10 IOPS by mClock scheduler. Like this, we can consider a client workload metric-based approach, but it is not easy to ensure QoS stability as client workloads are dynamically changed and the time period to obtain metrics affects the allocation accuracy. Additionally, per client session QoS can instead be considered, but it is difficult to predict and limit the number of sessions in the subvolume. For this reason, instead of applying dmClock, mClock scheduler can be considered as a good solution to the noisy neighbor problem. Expected per subvolume QoS IOPS can also be calculated as follows. (Assuming MDS and OSD requests are almost evenly distributed and reservation and weight values are omitted for brief exploration.) [MDS] - Per MDS Limit IOPS * # of MDSs (Perf MDS Limit IOPS * 1 when ephemeral random pinning is configured.) [OSD] - Per OSD Limit IOPS * # of OSDs Of course, depending on the workload or server conditions, the above equations may not be 100% satisfied. However, the QoS scheduler can be implemented without the client and manager modifications. There are any other comments from a CephFS point of view? Finally, we are going to implement our prototype that the mClock scheduler is applied to MDS and then make a pull request to share and discuss them. Thanks Yongseok

Hi maintainers, We have implemented this scheduler idea and made a pull request as working in progress https://github.com/ceph/ceph/pull/38506. I think there are some technical challenges that we need to discuss. It may not have been fully implemented yet, so if maintainers give us feedback, we would like to actively improve it together. If there is a chance, I would like to ask for your interest and review. Yongseok