per client mds throttle
Hi, Are there any plans to implement a per-client throttle on mds client requests? We just had an interesting case where a new cephfs user was hammering an mds from several hosts. In the end we found that their code was doing: while d=getafewbytesofdata(): f=open(file.dat) f.append(d) f.close() By changing their code to: f=open(file.dat) while d=getafewbytesofdata(): f.append(d) f.close() it completely removes their load on the mds (for obvious reasons). In a multi-user environment it's hard to scrutinize every user's application, so we'd prefer to just throttle down the client req rates (and let them suffer from the poor performance). Thoughts? Thanks, Dan
On Tue, 2019-07-02 at 17:24 +0200, Dan van der Ster wrote:
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering an mds from several hosts. In the end we found that their code was doing:
while d=getafewbytesofdata(): f=open(file.dat) f.append(d) f.close()
By changing their code to:
f=open(file.dat) while d=getafewbytesofdata(): f.append(d) f.close()
it completely removes their load on the mds (for obvious reasons).
In a multi-user environment it's hard to scrutinize every user's application, so we'd prefer to just throttle down the client req rates (and let them suffer from the poor performance).
Thoughts?
(cc'ing Xuehan) It sounds like a reasonable thing to do at first glance. There was a patchset recently by Xuehan Xu to add a new io controller policy for cephfs, but that was more focused around OSD ops on behalf of cephfs clients, fwiw, but that's not quite what you're asking about. The challenge with all of these sorts of throttling schemes is how to parcel things out to individual clients. MDS/OSD ops are not a discrete resource, and it's difficult to gauge how much to allocate to each client. I think if we were going to do something along these lines, it'd be good to work out how you'd throttle both MDS and OSD ops to keep a lid on things. That said, this is not a trivial problem to tackle, IMO. Some questions to get you started should you choose to pursue this: - Will you throttle these ops at the MDS or on the clients? Ditto for the OSDs... - How will it work? Will there be a fixed cap of some sort for a given amount of time, or are you more looking to just delay processing ops for a single client when it's "too busy"? - If you're thinking of something more like a cgroup, how will you determine how large a pool of operations you will have, and can parcel out to each client? If you've parceled out 100% of your MDS ops budget, how will you rebalance things when new clients are added or removed from the cluster? - if a client is holding a file lock, then throttling it could delay it releasing locks and that could slow down other (mostly idle) clients that are contending for it. Do we care? How will we deal with that situation if so? - Would you need separate tunables for OSD and MDS ops, or is there some way to tune both under a single knob? -- Jeff Layton <jlayton@redhat.com>
On Wed, Jul 3, 2019 at 12:30 PM Jeff Layton <jlayton@redhat.com> wrote:
On Tue, 2019-07-02 at 17:24 +0200, Dan van der Ster wrote:
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering an mds from several hosts. In the end we found that their code was doing:
while d=getafewbytesofdata(): f=open(file.dat) f.append(d) f.close()
By changing their code to:
f=open(file.dat) while d=getafewbytesofdata(): f.append(d) f.close()
it completely removes their load on the mds (for obvious reasons).
In a multi-user environment it's hard to scrutinize every user's application, so we'd prefer to just throttle down the client req rates (and let them suffer from the poor performance).
Thoughts?
(cc'ing Xuehan)
It sounds like a reasonable thing to do at first glance. There was a patchset recently by Xuehan Xu to add a new io controller policy for cephfs, but that was more focused around OSD ops on behalf of cephfs clients, fwiw, but that's not quite what you're asking about.
The challenge with all of these sorts of throttling schemes is how to parcel things out to individual clients. MDS/OSD ops are not a discrete resource, and it's difficult to gauge how much to allocate to each client.
I think if we were going to do something along these lines, it'd be good to work out how you'd throttle both MDS and OSD ops to keep a lid on things. That said, this is not a trivial problem to tackle, IMO.
Thanks for the reply, Jeff. At the moment I'm only considering adding a simple throttle on the MDS client ops. From the practical standpoint, we have seen clients overloading MDS's already, but haven't suffered from any OSD related load issues. Plus, the distributed QoS stuff should better solve the OSD problem, AFAIU.
Some questions to get you started should you choose to pursue this:
- Will you throttle these ops at the MDS or on the clients? Ditto for the OSDs...
- How will it work? Will there be a fixed cap of some sort for a given amount of time, or are you more looking to just delay processing ops for a single client when it's "too busy"?
My basic idea is to add this machinery early on in handle_client_request: - record each Session's last req timestamp - if now()-last_req_timestamp for a given req/session is less than a configurable delay, inject a delay. (e.g. mds_per_client_request_sleep, defaults to 0, we'd use 0.01 to throttle clients to 100Hz each) That said, I haven't understood exactly how to inject that delay just yet. Is h_c_r async per req, or is it looping with one thread over the queued requests? If it's async per Session or req, then we could just sleep right there in h_c_r. If h_c_r is handled by one thread, we need to be more clever. Is there a standard way to tell a client to retry the req after some delay? Also, this kind of v1 PoC would obviously have the same mds_per_client_request_sleep for all clients. v2 could add a configurable sleep for specific clients. And in addition to the per-client approach, a second idea would be to throttle per mount prefix (which would be useful in cases of multiple clients accessing the same path, e.g. multi-tenant with Manila). A simple way to achieve this would be to use the session's client_metadata.root as a key in a hash of last req time (per mount root), delaying requests as needed (like above). The holy grail path throttle solution would be to allow throttling per subpath, e.g. for a home directory use-case where you have 10000 subdirs in /home/, and we want to throttle any /home/{*}/ to 100Hz. This could be exposed as an xattr on a directory, but for each request we'd have to resolve the path upwards to find a req/s quota (like we do for space quotas) and sleep accordingly.
- If you're thinking of something more like a cgroup, how will you determine how large a pool of operations you will have, and can parcel out to each client? If you've parceled out 100% of your MDS ops budget, how will you rebalance things when new clients are added or removed from the cluster?
I'm not a fan of the cgroup approach because it's nicer if the throttling can be enforced/configured dynamically on the server-side. The simple sleep proposed above is inspired by the various osd sleeps that we've added over the years -- they turn out to be super effective for busy prod clusters. If we realize in prod that the sleep is too aggressive, we can just lower it on the mds's as needed :)
- if a client is holding a file lock, then throttling it could delay it releasing locks and that could slow down other (mostly idle) clients that are contending for it. Do we care? How will we deal with that situation if so?
That would indeed be a concern, but my understanding from checking dispatch_client_request is that these are all client md ops like lookup, create, rm, setxattr; and not the responses to a revoke cap request from the mds to the client. Is that write? Thanks! Dan
- Would you need separate tunables for OSD and MDS ops, or is there some way to tune both under a single knob? -- Jeff Layton <jlayton@redhat.com>
On Wed, 2019-07-03 at 13:44 +0200, Dan van der Ster wrote:
On Wed, Jul 3, 2019 at 12:30 PM Jeff Layton <jlayton@redhat.com> wrote:
On Tue, 2019-07-02 at 17:24 +0200, Dan van der Ster wrote:
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering an mds from several hosts. In the end we found that their code was doing:
while d=getafewbytesofdata(): f=open(file.dat) f.append(d) f.close()
By changing their code to:
f=open(file.dat) while d=getafewbytesofdata(): f.append(d) f.close()
it completely removes their load on the mds (for obvious reasons).
In a multi-user environment it's hard to scrutinize every user's application, so we'd prefer to just throttle down the client req rates (and let them suffer from the poor performance).
Thoughts?
(cc'ing Xuehan)
It sounds like a reasonable thing to do at first glance. There was a patchset recently by Xuehan Xu to add a new io controller policy for cephfs, but that was more focused around OSD ops on behalf of cephfs clients, fwiw, but that's not quite what you're asking about.
The challenge with all of these sorts of throttling schemes is how to parcel things out to individual clients. MDS/OSD ops are not a discrete resource, and it's difficult to gauge how much to allocate to each client.
I think if we were going to do something along these lines, it'd be good to work out how you'd throttle both MDS and OSD ops to keep a lid on things. That said, this is not a trivial problem to tackle, IMO.
Thanks for the reply, Jeff. At the moment I'm only considering adding a simple throttle on the MDS client ops. From the practical standpoint, we have seen clients overloading MDS's already, but haven't suffered from any OSD related load issues. Plus, the distributed QoS stuff should better solve the OSD problem, AFAIU.
Some questions to get you started should you choose to pursue this:
- Will you throttle these ops at the MDS or on the clients? Ditto for the OSDs...
- How will it work? Will there be a fixed cap of some sort for a given amount of time, or are you more looking to just delay processing ops for a single client when it's "too busy"?
My basic idea is to add this machinery early on in handle_client_request: - record each Session's last req timestamp - if now()-last_req_timestamp for a given req/session is less than a configurable delay, inject a delay. (e.g. mds_per_client_request_sleep, defaults to 0, we'd use 0.01 to throttle clients to 100Hz each)
That said, I haven't understood exactly how to inject that delay just yet. Is h_c_r async per req, or is it looping with one thread over the queued requests? If it's async per Session or req, then we could just sleep right there in h_c_r. If h_c_r is handled by one thread, we need to be more clever. Is there a standard way to tell a client to retry the req after some delay?
Also, this kind of v1 PoC would obviously have the same mds_per_client_request_sleep for all clients. v2 could add a configurable sleep for specific clients.
And in addition to the per-client approach, a second idea would be to throttle per mount prefix (which would be useful in cases of multiple clients accessing the same path, e.g. multi-tenant with Manila). A simple way to achieve this would be to use the session's client_metadata.root as a key in a hash of last req time (per mount root), delaying requests as needed (like above).
The holy grail path throttle solution would be to allow throttling per subpath, e.g. for a home directory use-case where you have 10000 subdirs in /home/, and we want to throttle any /home/{*}/ to 100Hz. This could be exposed as an xattr on a directory, but for each request we'd have to resolve the path upwards to find a req/s quota (like we do for space quotas) and sleep accordingly.
Ok, so you're looking at doing something on the MDS. I'll leave the specifics of how that should be implemented to those more familiar with the MDS code. That said, i might be nice to make that somewhat adaptive to allow the client to have short periods of "spiky" MDS activity without delays. That could be altered after the basic, simple method you describe above is implemented however.
- If you're thinking of something more like a cgroup, how will you determine how large a pool of operations you will have, and can parcel out to each client? If you've parceled out 100% of your MDS ops budget, how will you rebalance things when new clients are added or removed from the cluster?
I'm not a fan of the cgroup approach because it's nicer if the throttling can be enforced/configured dynamically on the server-side. The simple sleep proposed above is inspired by the various osd sleeps that we've added over the years -- they turn out to be super effective for busy prod clusters. If we realize in prod that the sleep is too aggressive, we can just lower it on the mds's as needed :)
- if a client is holding a file lock, then throttling it could delay it releasing locks and that could slow down other (mostly idle) clients that are contending for it. Do we care? How will we deal with that situation if so?
That would indeed be a concern, but my understanding from checking dispatch_client_request is that these are all client md ops like lookup, create, rm, setxattr; and not the responses to a revoke cap request from the mds to the client. Is that write?
Sort of. I was mostly thinking of flock()/fcntl() style locks. AFAICT, dispatch_client_request also handles CEPH_MDS_OP_SETFILELOCK too. You'd probably want to avoid delaying any of those calls that are releasing a file lock, but it would probably be ok to delay requests for new locks. But, you bring up a good point; caps and dentry leases are really just a different form of revocable lock and those can be contended as well. I think as an overall design goal, you'd want to focus on delaying requests for "new state" (opens, locks, caps, directory leases), but things like cap or dentry releases should always be processed ASAP. That may be a bit more involved than what you were originally suggesting... -- Jeff Layton <jlayton@redhat.com>
Unfortunately this is VERY complicated. On Wed, Jul 3, 2019 at 4:45 AM Dan van der Ster <dan@vanderster.com> wrote:
On Wed, Jul 3, 2019 at 12:30 PM Jeff Layton <jlayton@redhat.com> wrote:
On Tue, 2019-07-02 at 17:24 +0200, Dan van der Ster wrote:
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering an mds from several hosts. In the end we found that their code was doing:
while d=getafewbytesofdata(): f=open(file.dat) f.append(d) f.close()
By changing their code to:
f=open(file.dat) while d=getafewbytesofdata(): f.append(d) f.close()
it completely removes their load on the mds (for obvious reasons).
In a multi-user environment it's hard to scrutinize every user's application, so we'd prefer to just throttle down the client req rates (and let them suffer from the poor performance).
Thoughts?
(cc'ing Xuehan)
It sounds like a reasonable thing to do at first glance. There was a patchset recently by Xuehan Xu to add a new io controller policy for cephfs, but that was more focused around OSD ops on behalf of cephfs clients, fwiw, but that's not quite what you're asking about.
The challenge with all of these sorts of throttling schemes is how to parcel things out to individual clients. MDS/OSD ops are not a discrete resource, and it's difficult to gauge how much to allocate to each client.
I think if we were going to do something along these lines, it'd be good to work out how you'd throttle both MDS and OSD ops to keep a lid on things. That said, this is not a trivial problem to tackle, IMO.
Thanks for the reply, Jeff. At the moment I'm only considering adding a simple throttle on the MDS client ops. From the practical standpoint, we have seen clients overloading MDS's already, but haven't suffered from any OSD related load issues. Plus, the distributed QoS stuff should better solve the OSD problem, AFAIU.
Some questions to get you started should you choose to pursue this:
- Will you throttle these ops at the MDS or on the clients? Ditto for the OSDs...
- How will it work? Will there be a fixed cap of some sort for a given amount of time, or are you more looking to just delay processing ops for a single client when it's "too busy"?
My basic idea is to add this machinery early on in handle_client_request: - record each Session's last req timestamp - if now()-last_req_timestamp for a given req/session is less than a configurable delay, inject a delay. (e.g. mds_per_client_request_sleep, defaults to 0, we'd use 0.01 to throttle clients to 100Hz each)
That said, I haven't understood exactly how to inject that delay just yet. Is h_c_r async per req, or is it looping with one thread over the queued requests?
There's a single dispatch thread that runs all pending incoming messages. Either you process the message completely and send any necessary replies, or you put the message on some kind of waitlist that gets picked up later (either automatically by the dispatch thread, or when a Timer goes off and retries it).
If it's async per Session or req, then we could just sleep right there in h_c_r. If h_c_r is handled by one thread, we need to be more clever. Is there a standard way to tell a client to retry the req after some delay?
Nope. You'd probably have to create a new ordered queue of delayed messages, and on every loop in _dispatch() check to see if the front timestamp is allowed to be processed yet. That said, this isn't actually nearly sufficient, because...
Also, this kind of v1 PoC would obviously have the same mds_per_client_request_sleep for all clients. v2 could add a configurable sleep for specific clients.
And in addition to the per-client approach, a second idea would be to throttle per mount prefix (which would be useful in cases of multiple clients accessing the same path, e.g. multi-tenant with Manila). A simple way to achieve this would be to use the session's client_metadata.root as a key in a hash of last req time (per mount root), delaying requests as needed (like above).
The holy grail path throttle solution would be to allow throttling per subpath, e.g. for a home directory use-case where you have 10000 subdirs in /home/, and we want to throttle any /home/{*}/ to 100Hz. This could be exposed as an xattr on a directory, but for each request we'd have to resolve the path upwards to find a req/s quota (like we do for space quotas) and sleep accordingly.
- If you're thinking of something more like a cgroup, how will you determine how large a pool of operations you will have, and can parcel out to each client? If you've parceled out 100% of your MDS ops budget, how will you rebalance things when new clients are added or removed from the cluster?
I'm not a fan of the cgroup approach because it's nicer if the throttling can be enforced/configured dynamically on the server-side. The simple sleep proposed above is inspired by the various osd sleeps that we've added over the years -- they turn out to be super effective for busy prod clusters. If we realize in prod that the sleep is too aggressive, we can just lower it on the mds's as needed :)
- if a client is holding a file lock, then throttling it could delay it releasing locks and that could slow down other (mostly idle) clients that are contending for it. Do we care? How will we deal with that situation if so?
That would indeed be a concern, but my understanding from checking dispatch_client_request is that these are all client md ops like lookup, create, rm, setxattr; and not the responses to a revoke cap request from the mds to the client. Is that write?
Unfortunately the load you're seeing from client opens and closes is not one of the reasonably easy-to-handle request types, but an MClientCaps message. Delaying these could generally be VERY BAD, because that's how clients acknowledge file capability changes from the MDS when revoking caps, and do other things that directly feed into the system's performance and correctness. I don't *think* you could get any reordering issues as long as it's just a delay on message delivery from clients, but you'd definitely see things like much worse behavior on any shared systems with Fw access for multiple clients. These are all problems that will eventually need to be solved for QoS reasons, but any quick fixes are going to be very dependent on the use case to be reliable. Given the example you have of a "misbehaving" client, it may be simpler to provide options for de-tuning MDS performance a bit. That open/close flush behavior you and Sage discuss was added specifically to address some performance or cache size issue but could definitely be configurable. -Greg
Thanks!
Dan
- Would you need separate tunables for OSD and MDS ops, or is there some way to tune both under a single knob? -- Jeff Layton <jlayton@redhat.com>
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Tue, 2 Jul 2019, Dan van der Ster wrote:
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering an mds from several hosts. In the end we found that their code was doing:
while d=getafewbytesofdata(): f=open(file.dat) f.append(d) f.close()
By changing their code to:
f=open(file.dat) while d=getafewbytesofdata(): f.append(d) f.close()
it completely removes their load on the mds (for obvious reasons).
This is a tangential point, but: at some point in the past at least the above two code blcoks used to incur the same load on the MDS because the client would only asynchronously release the caps back to the MDS. Did we change that behavior? Maybe we can restore it, but with a very a short delay, so that we cover patterns like the above? sage
In a multi-user environment it's hard to scrutinize every user's application, so we'd prefer to just throttle down the client req rates (and let them suffer from the poor performance).
Thoughts?
Thanks,
Dan _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Wed, Jul 3, 2019 at 2:58 PM Sage Weil <sage@newdream.net> wrote:
On Tue, 2 Jul 2019, Dan van der Ster wrote:
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering an mds from several hosts. In the end we found that their code was doing:
while d=getafewbytesofdata(): f=open(file.dat) f.append(d) f.close()
By changing their code to:
f=open(file.dat) while d=getafewbytesofdata(): f.append(d) f.close()
it completely removes their load on the mds (for obvious reasons).
This is a tangential point, but: at some point in the past at least the above two code blcoks used to incur the same load on the MDS because the client would only asynchronously release the caps back to the MDS. Did we change that behavior? Maybe we can restore it, but with a very a short delay, so that we cover patterns like the above?
With a simple python test, putting the open/close inside the loop [1], I see an update ino req to the mds after each close: 2019-07-03 15:30:08.012349 7f30bfd68700 1 -- 137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 -- client_request(unknown.0:31983 open #0x5000177be10 2019-07-03 15:30:08.012345 caller_uid=0, caller_gid=0{0,}) v4 -- 0x56212b328300 con 0 2019-07-03 15:30:08.016138 7f30bd563700 1 -- 137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 -- client_caps(update ino 0x5000177be10 135309857 seq 4811 tid 61814 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=pAsxXsxFxcwb follows 1 size 3/3 ts 756/0 mtime 2019-07-03 15:30:08.015825) v11 -- 0x562126b5bf80 con 0 2019-07-03 15:30:08.016950 7f30c0569700 1 -- 137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 -- client_caps(update ino 0x5000177be10 135309857 seq 4811 tid 61815 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=pAsxXsxFxcwb follows 1 size 6/3 ts 756/0 mtime 2019-07-03 15:30:08.016637) v11 -- 0x56212728e880 con 0 2019-07-03 15:30:08.017848 7f30bf567700 1 -- 137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 -- client_caps(update ino 0x5000177be10 135309857 seq 4811 tid 61816 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=pAsxXsxFxcwb follows 1 size 9/3 ts 756/0 mtime 2019-07-03 15:30:08.017439) v11 -- 0x56212148b180 con 0 and the flush_ack's from the mds (not exactly corresponding to above): 2019-07-03 15:32:37.831384 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227525 ==== client_caps(grant ino 0x5000177be10 135309857 seq 4812 caps=pAsxLsXsxFsxcrwb dirty=- wanted=pAsxXsxFxwb follows 0 size 1206/0 ts 756/0 mtime 2019-07-03 15:30:08.385521) v11 ==== 299+0+0 (1635817588 0 0) 0x5621252efa80 con 0x56211eefb800 2019-07-03 15:32:37.871171 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227527 ==== client_caps(grant ino 0x5000177be10 135309857 seq 4814 caps=pAsxLsXsxFsxcrwb dirty=- wanted=pAsxXsxFxwb follows 0 size 1206/4194304 ts 756/0 mtime 2019-07-03 15:30:08.385521) v11 ==== 299+0+0 (3626990096 0 0) 0x56212361bb00 con 0x56211eefb800 2019-07-03 15:32:41.018963 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227528 ==== client_caps(flush_ack ino 0x5000177be10 135309857 seq 4814 tid 62216 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=- follows 0 size 0/0 mtime 0.000000) v11 ==== 252+0+0 (197724779 0 0) 0x56212b20e900 con 0x56211eefb800 2019-07-03 15:32:41.019026 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227529 ==== client_caps(flush_ack ino 0x5000177be10 135309857 seq 4814 tid 62217 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=- follows 0 size 0/0 mtime 0.000000) v11 ==== 252+0+0 (197724779 0 0) 0x5621252e9600 con 0x56211eefb800 2019-07-03 15:32:41.019047 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227530 ==== client_caps(flush_ack ino 0x5000177be10 135309857 seq 4814 tid 62218 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=- follows 0 size 0/0 mtime 0.000000) v11 ==== 252+0+0 (197724779 0 0) 0x562123402480 con 0x56211eefb800 With open/close outside the loop, there's very little conversation with the mds... renewing caps and the occasional flush_ack from time to time. This is with 12.2.12 mds, 12.2.12 fuse client. -- Dan [1] while(True): f = open('out.dat', 'a') // append or truncate doesn't make a difference to the client-mds traffic, afaict f.write('aaa') f.close()
sage
In a multi-user environment it's hard to scrutinize every user's application, so we'd prefer to just throttle down the client req rates (and let them suffer from the poor performance).
Thoughts?
Thanks,
Dan _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (4)
-
Dan van der Ster
-
Gregory Farnum
-
Jeff Layton
-
Sage Weil