2 Jul
2019
2 Jul
'19
6:24 p.m.
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering
an mds from several hosts. In the end we found that their code was
doing:
while d=getafewbytesofdata():
f=open(file.dat)
f.append(d)
f.close()
By changing their code to:
f=open(file.dat)
while d=getafewbytesofdata():
f.append(d)
f.close()
it completely removes their load on the mds (for obvious reasons).
In a multi-user environment it's hard to scrutinize every user's
application, so we'd prefer to just throttle down the client req rates
(and let them suffer from the poor performance).
Thoughts?
Thanks,
Dan
3 Jul
3 Jul
1:30 p.m.
On Tue, 2019-07-02 at 17:24 +0200, Dan van der Ster wrote:
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering
an mds from several hosts. In the end we found that their code was
doing:
while d=getafewbytesofdata():
f=open(file.dat)
f.append(d)
f.close()
By changing their code to:
f=open(file.dat)
while d=getafewbytesofdata():
f.append(d)
f.close()
it completely removes their load on the mds (for obvious reasons).
In a multi-user environment it's hard to scrutinize every user's
application, so we'd prefer to just throttle down the client req rates
(and let them suffer from the poor performance).
Thoughts?
(cc'ing Xuehan)
It sounds like a reasonable thing to do at first glance. There was a
patchset recently by Xuehan Xu to add a new io controller policy for
cephfs, but that was more focused around OSD ops on behalf of cephfs
clients, fwiw, but that's not quite what you're asking about.
The challenge with all of these sorts of throttling schemes is how to
parcel things out to individual clients. MDS/OSD ops are not a discrete
resource, and it's difficult to gauge how much to allocate to each
client.
I think if we were going to do something along these lines, it'd be good
to work out how you'd throttle both MDS and OSD ops to keep a lid on
things. That said, this is not a trivial problem to tackle, IMO.
Some questions to get you started should you choose to pursue this:
- Will you throttle these ops at the MDS or on the clients? Ditto for
the OSDs...
- How will it work? Will there be a fixed cap of some sort for a given
amount of time, or are you more looking to just delay processing ops for
a single client when it's "too busy"?
- If you're thinking of something more like a cgroup, how will you
determine how large a pool of operations you will have, and can parcel
out to each client? If you've parceled out 100% of your MDS ops budget,
how will you rebalance things when new clients are added or removed from
the cluster?
- if a client is holding a file lock, then throttling it could delay it
releasing locks and that could slow down other (mostly idle) clients
that are contending for it. Do we care? How will we deal with that
situation if so?
- Would you need separate tunables for OSD and MDS ops, or is there some
way to tune both under a single knob?
--
Jeff Layton <jlayton(a)redhat.com>
2:44 p.m.
On Wed, Jul 3, 2019 at 12:30 PM Jeff Layton <jlayton(a)redhat.com> wrote:
On Tue, 2019-07-02 at 17:24 +0200, Dan van der Ster wrote:
Thanks for the reply, Jeff.
At the moment I'm only considering adding a simple throttle on the MDS
client ops. From the practical standpoint, we have seen clients
overloading MDS's already, but haven't suffered from any OSD related
load issues.
Plus, the distributed QoS stuff should better solve the OSD problem, AFAIU.
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering
an mds from several hosts. In the end we found that their code was
doing:
while d=getafewbytesofdata():
f=open(file.dat)
f.append(d)
f.close()
By changing their code to:
f=open(file.dat)
while d=getafewbytesofdata():
f.append(d)
f.close()
it completely removes their load on the mds (for obvious reasons).
In a multi-user environment it's hard to scrutinize every user's
application, so we'd prefer to just throttle down the client req rates
(and let them suffer from the poor performance).
Thoughts?
(cc'ing Xuehan)
It sounds like a reasonable thing to do at first glance. There was a
patchset recently by Xuehan Xu to add a new io controller policy for
cephfs, but that was more focused around OSD ops on behalf of cephfs
clients, fwiw, but that's not quite what you're asking about.
The challenge with all of these sorts of throttling schemes is how to
parcel things out to individual clients. MDS/OSD ops are not a discrete
resource, and it's difficult to gauge how much to allocate to each
client.
I think if we were going to do something along these lines, it'd be good
to work out how you'd throttle both MDS and OSD ops to keep a lid on
things. That said, this is not a trivial problem to tackle, IMO.
Some questions to get you started should you choose to
pursue this:
- Will you throttle these ops at the MDS or on the clients? Ditto for
the OSDs...
- How will it work? Will there be a fixed cap of some sort for a given
amount of time, or are you more looking to just delay processing ops for
a single client when it's "too busy"?
My basic idea is to add this machinery early on in handle_client_request:
- record each Session's last req timestamp
- if now()-last_req_timestamp for a given req/session is less than a
configurable delay, inject a delay. (e.g.
mds_per_client_request_sleep, defaults to 0, we'd use 0.01 to throttle
clients to 100Hz each)
That said, I haven't understood exactly how to inject that delay just
yet. Is h_c_r async per req, or is it looping with one thread over the
queued requests? If it's async per Session or req, then we could just
sleep right there in h_c_r. If h_c_r is handled by one thread, we need
to be more clever. Is there a standard way to tell a client to retry
the req after some delay?
Also, this kind of v1 PoC would obviously have the same
mds_per_client_request_sleep for all clients. v2 could add a
configurable sleep for specific clients.
And in addition to the per-client approach, a second idea would be to
throttle per mount prefix (which would be useful in cases of multiple
clients accessing the same path, e.g. multi-tenant with Manila).
A simple way to achieve this would be to use the session's
client_metadata.root as a key in a hash of last req time (per mount
root), delaying requests as needed (like above).
The holy grail path throttle solution would be to allow throttling per
subpath, e.g. for a home directory use-case where you have 10000
subdirs in /home/, and we want to throttle any /home/{*}/ to 100Hz.
This could be exposed as an xattr on a directory, but for each request
we'd have to resolve the path upwards to find a req/s quota (like we
do for space quotas) and sleep accordingly.
- If you're thinking of something more like a
cgroup, how will you
determine how large a pool of operations you will have, and can parcel
out to each client? If you've parceled out 100% of your MDS ops budget,
how will you rebalance things when new clients are added or removed from
the cluster?
I'm not a fan of the cgroup approach because it's nicer if the
throttling can be enforced/configured dynamically on the server-side.
The simple sleep proposed above is inspired by the various osd sleeps
that we've added over the years -- they turn out to be super effective
for busy prod clusters.
If we realize in prod that the sleep is too aggressive, we can just
lower it on the mds's as needed :)
- if a client is holding a file lock, then throttling
it could delay it
releasing locks and that could slow down other (mostly idle) clients
that are contending for it. Do we care? How will we deal with that
situation if so?
That would indeed be a concern, but my understanding from checking
dispatch_client_request is that these are all client md ops like
lookup, create, rm, setxattr; and not the responses to a revoke cap
request from the mds to the client.
Is that write?
Thanks!
Dan
> - Would you need separate tunables for OSD and MDS ops, or is there some
> way to tune both under a single knob?
> --
> Jeff Layton <jlayton(a)redhat.com>
>
6:36 p.m.
On Wed, 2019-07-03 at 13:44 +0200, Dan van der Ster wrote:
Sort of. I was mostly thinking of flock()/fcntl() style locks. AFAICT,
dispatch_client_request also handles CEPH_MDS_OP_SETFILELOCK too. You'd
probably want to avoid delaying any of those calls that are releasing a
file lock, but it would probably be ok to delay requests for new locks.
But, you bring up a good point; caps and dentry leases are really just a
different form of revocable lock and those can be contended as well.
I think as an overall design goal, you'd want to focus on delaying
requests for "new state" (opens, locks, caps, directory leases), but
things like cap or dentry releases should always be processed ASAP.
That may be a bit more involved than what you were originally
suggesting...
--
Jeff Layton <jlayton(a)redhat.com>
On Wed, Jul 3, 2019 at 12:30 PM Jeff Layton
<jlayton(a)redhat.com> wrote:
Ok, so you're looking at doing something on the MDS. I'll leave the
specifics of how that should be implemented to those more familiar with
the MDS code.
That said, i might be nice to make that somewhat adaptive to allow the
client to have short periods of "spiky" MDS activity without delays.
That could be altered after the basic, simple method you describe above
is implemented however.
On Tue, 2019-07-02 at 17:24 +0200, Dan van der
Ster wrote:
Thanks for the reply, Jeff.
At the moment I'm only considering adding a simple throttle on the MDS
client ops. From the practical standpoint, we have seen clients
overloading MDS's already, but haven't suffered from any OSD related
load issues.
Plus, the distributed QoS stuff should better solve the OSD problem, AFAIU.
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering
an mds from several hosts. In the end we found that their code was
doing:
while d=getafewbytesofdata():
f=open(file.dat)
f.append(d)
f.close()
By changing their code to:
f=open(file.dat)
while d=getafewbytesofdata():
f.append(d)
f.close()
it completely removes their load on the mds (for obvious reasons).
In a multi-user environment it's hard to scrutinize every user's
application, so we'd prefer to just throttle down the client req rates
(and let them suffer from the poor performance).
Thoughts?
(cc'ing Xuehan)
It sounds like a reasonable thing to do at first glance. There was a
patchset recently by Xuehan Xu to add a new io controller policy for
cephfs, but that was more focused around OSD ops on behalf of cephfs
clients, fwiw, but that's not quite what you're asking about.
The challenge with all of these sorts of throttling schemes is how to
parcel things out to individual clients. MDS/OSD ops are not a discrete
resource, and it's difficult to gauge how much to allocate to each
client.
I think if we were going to do something along these lines, it'd be good
to work out how you'd throttle both MDS and OSD ops to keep a lid on
things. That said, this is not a trivial problem to tackle, IMO.
Some questions to get you started should you
choose to pursue this:
- Will you throttle these ops at the MDS or on the clients? Ditto for
the OSDs...
- How will it work? Will there be a fixed cap of some sort for a given
amount of time, or are you more looking to just delay processing ops for
a single client when it's "too busy"?
My basic idea is to add this machinery early on in handle_client_request:
- record each Session's last req timestamp
- if now()-last_req_timestamp for a given req/session is less than a
configurable delay, inject a delay. (e.g.
mds_per_client_request_sleep, defaults to 0, we'd use 0.01 to throttle
clients to 100Hz each)
That said, I haven't understood exactly how to inject that delay just
yet. Is h_c_r async per req, or is it looping with one thread over the
queued requests? If it's async per Session or req, then we could just
sleep right there in h_c_r. If h_c_r is handled by one thread, we need
to be more clever. Is there a standard way to tell a client to retry
the req after some delay?
Also, this kind of v1 PoC would obviously have the same
mds_per_client_request_sleep for all clients. v2 could add a
configurable sleep for specific clients.
And in addition to the per-client approach, a second idea would be to
throttle per mount prefix (which would be useful in cases of multiple
clients accessing the same path, e.g. multi-tenant with Manila).
A simple way to achieve this would be to use the session's
client_metadata.root as a key in a hash of last req time (per mount
root), delaying requests as needed (like above).
The holy grail path throttle solution would be to allow throttling per
subpath, e.g. for a home directory use-case where you have 10000
subdirs in /home/, and we want to throttle any /home/{*}/ to 100Hz.
This could be exposed as an xattr on a directory, but for each request
we'd have to resolve the path upwards to find a req/s quota (like we
do for space quotas) and sleep accordingly.
- If
you're thinking of something more like a cgroup, how will you
determine how large a pool of operations you will have, and can parcel
out to each client? If you've parceled out 100% of your MDS ops budget,
how will you rebalance things when new clients are added or removed from
the cluster?
I'm not a fan of the cgroup approach because it's nicer if the
throttling can be enforced/configured dynamically on the server-side.
The simple sleep proposed above is inspired by the various osd sleeps
that we've added over the years -- they turn out to be super effective
for busy prod clusters.
If we realize in prod that the sleep is too aggressive, we can just
lower it on the mds's as needed :)
- if a client is holding a file lock, then
throttling it could delay it
releasing locks and that could slow down other (mostly idle) clients
that are contending for it. Do we care? How will we deal with that
situation if so?
That would indeed be a concern, but my understanding from checking
dispatch_client_request is that these are all client md ops like
lookup, create, rm, setxattr; and not the responses to a revoke cap
request from the mds to the client.
Is that write?
8:33 p.m.
Unfortunately this is VERY complicated.
On Wed, Jul 3, 2019 at 4:45 AM Dan van der Ster <dan(a)vanderster.com> wrote:
On Wed, Jul 3, 2019 at 12:30 PM Jeff Layton <jlayton(a)redhat.com> wrote:
There's a single dispatch thread that runs all pending incoming
messages. Either you process the message completely and send any
necessary replies, or you put the message on some kind of waitlist
that gets picked up later (either automatically by the dispatch
thread, or when a Timer goes off and retries it).
On Tue, 2019-07-02 at 17:24 +0200, Dan van der Ster wrote:
Thanks for the reply, Jeff.
At the moment I'm only considering adding a simple throttle on the MDS
client ops. From the practical standpoint, we have seen clients
overloading MDS's already, but haven't suffered from any OSD related
load issues.
Plus, the distributed QoS stuff should better solve the OSD problem, AFAIU.
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering
an mds from several hosts. In the end we found that their code was
doing:
while d=getafewbytesofdata():
f=open(file.dat)
f.append(d)
f.close()
By changing their code to:
f=open(file.dat)
while d=getafewbytesofdata():
f.append(d)
f.close()
it completely removes their load on the mds (for obvious reasons).
In a multi-user environment it's hard to scrutinize every user's
application, so we'd prefer to just throttle down the client req rates
(and let them suffer from the poor performance).
Thoughts?
(cc'ing Xuehan)
It sounds like a reasonable thing to do at first glance. There was a
patchset recently by Xuehan Xu to add a new io controller policy for
cephfs, but that was more focused around OSD ops on behalf of cephfs
clients, fwiw, but that's not quite what you're asking about.
The challenge with all of these sorts of throttling schemes is how to
parcel things out to individual clients. MDS/OSD ops are not a discrete
resource, and it's difficult to gauge how much to allocate to each
client.
I think if we were going to do something along these lines, it'd be good
to work out how you'd throttle both MDS and OSD ops to keep a lid on
things. That said, this is not a trivial problem to tackle, IMO.
Some questions to get you started should you
choose to pursue this:
- Will you throttle these ops at the MDS or on the clients? Ditto for
the OSDs...
- How will it work? Will there be a fixed cap of some sort for a given
amount of time, or are you more looking to just delay processing ops for
a single client when it's "too busy"?
My basic idea is to add this machinery early on in handle_client_request:
- record each Session's last req timestamp
- if now()-last_req_timestamp for a given req/session is less than a
configurable delay, inject a delay. (e.g.
mds_per_client_request_sleep, defaults to 0, we'd use 0.01 to throttle
clients to 100Hz each)
That said, I haven't understood exactly how to inject that delay just
yet. Is h_c_r async per req, or is it looping with one thread over the
queued requests? If it's async per Session or req, then we could
just
sleep right there in h_c_r. If h_c_r is handled by one thread, we need
to be more clever. Is there a standard way to tell a client to retry
the req after some delay?
Nope. You'd probably have to create a new ordered queue of delayed
messages, and on every loop in _dispatch() check to see if the front
timestamp is allowed to be processed yet.
That said, this isn't actually nearly sufficient, because...
Also, this kind of v1 PoC would obviously have the same
mds_per_client_request_sleep for all clients. v2 could add a
configurable sleep for specific clients.
And in addition to the per-client approach, a second idea would be to
throttle per mount prefix (which would be useful in cases of multiple
clients accessing the same path, e.g. multi-tenant with Manila).
A simple way to achieve this would be to use the session's
client_metadata.root as a key in a hash of last req time (per mount
root), delaying requests as needed (like above).
The holy grail path throttle solution would be to allow throttling per
subpath, e.g. for a home directory use-case where you have 10000
subdirs in /home/, and we want to throttle any /home/{*}/ to 100Hz.
This could be exposed as an xattr on a directory, but for each request
we'd have to resolve the path upwards to find a req/s quota (like we
do for space quotas) and sleep accordingly.
Unfortunately the load you're seeing from client opens and closes is
not one of the reasonably easy-to-handle request types, but an
MClientCaps message.
Delaying these could generally be VERY BAD, because that's how clients
acknowledge file capability changes from the MDS when revoking caps,
and do other things that directly feed into the system's performance
and correctness.
I don't *think* you could get any reordering issues as long as it's
just a delay on message delivery from clients, but you'd definitely
see things like much worse behavior on any shared systems with Fw
access for multiple clients.
These are all problems that will eventually need to be solved for QoS
reasons, but any quick fixes are going to be very dependent on the use
case to be reliable.
Given the example you have of a "misbehaving" client, it may be
simpler to provide options for de-tuning MDS performance a bit. That
open/close flush behavior you and Sage discuss was added specifically
to address some performance or cache size issue but could definitely
be configurable.
-Greg
>
> Thanks!
>
> Dan
>
> > - Would you need separate tunables for OSD and MDS ops, or is there some
> > way to tune both under a single knob?
> > --
> > Jeff Layton <jlayton(a)redhat.com>
> >
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
- If you're thinking of something more like a
cgroup, how will you
determine how large a pool of operations you will have, and can parcel
out to each client? If you've parceled out 100% of your MDS ops budget,
how will you rebalance things when new clients are added or removed from
the cluster?
I'm not a fan of the cgroup approach because it's nicer if the
throttling can be enforced/configured dynamically on the server-side.
The simple sleep proposed above is inspired by the various osd sleeps
that we've added over the years -- they turn out to be super effective
for busy prod clusters.
If we realize in prod that the sleep is too aggressive, we can just
lower it on the mds's as needed :)
- if a client is holding a file lock, then
throttling it could delay it
releasing locks and that could slow down other (mostly idle) clients
that are contending for it. Do we care? How will we deal with that
situation if so?
That would indeed be a concern, but my understanding from checking
dispatch_client_request is that these are all client md ops like
lookup, create, rm, setxattr; and not the responses to a revoke cap
request from the mds to the client.
Is that write? 
3:58 p.m.
On Tue, 2 Jul 2019, Dan van der Ster wrote:
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering
an mds from several hosts. In the end we found that their code was
doing:
while d=getafewbytesofdata():
f=open(file.dat)
f.append(d)
f.close()
By changing their code to:
f=open(file.dat)
while d=getafewbytesofdata():
f.append(d)
f.close()
it completely removes their load on the mds (for obvious reasons).
This is a tangential point, but: at some point in the past at least the
above two code blcoks used to incur the same load on the MDS because the
client would only asynchronously release the caps back to the MDS. Did we
change that behavior? Maybe we can restore it, but with a very a short
delay, so that we cover patterns like the above?
sage
>
> In a multi-user environment it's hard to scrutinize every user's
> application, so we'd prefer to just throttle down the client req rates
> (and let them suffer from the poor performance).
>
> Thoughts?
>
> Thanks,
>
> Dan
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
>
>
4:35 p.m.
On Wed, Jul 3, 2019 at 2:58 PM Sage Weil <sage(a)newdream.net> wrote:
On Tue, 2 Jul 2019, Dan van der Ster wrote:
With a simple python test, putting the open/close inside the loop [1],
I see an update ino req to the mds after each close:
2019-07-03 15:30:08.012349 7f30bfd68700 1 --
137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 --
client_request(unknown.0:31983 open #0x5000177be10 2019-07-03
15:30:08.012345 caller_uid=0, caller_gid=0{0,}) v4 -- 0x56212b328300
con 0
2019-07-03 15:30:08.016138 7f30bd563700 1 --
137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 --
client_caps(update ino 0x5000177be10 135309857 seq 4811 tid 61814
caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=pAsxXsxFxcwb follows 1 size 3/3
ts 756/0 mtime 2019-07-03 15:30:08.015825) v11 -- 0x562126b5bf80 con 0
2019-07-03 15:30:08.016950 7f30c0569700 1 --
137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 --
client_caps(update ino 0x5000177be10 135309857 seq 4811 tid 61815
caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=pAsxXsxFxcwb follows 1 size 6/3
ts 756/0 mtime 2019-07-03 15:30:08.016637) v11 -- 0x56212728e880 con 0
2019-07-03 15:30:08.017848 7f30bf567700 1 --
137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 --
client_caps(update ino 0x5000177be10 135309857 seq 4811 tid 61816
caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=pAsxXsxFxcwb follows 1 size 9/3
ts 756/0 mtime 2019-07-03 15:30:08.017439) v11 -- 0x56212148b180 con 0
and the flush_ack's from the mds (not exactly corresponding to above):
2019-07-03 15:32:37.831384 7f30c4d72700 1 --
137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234
227525 ==== client_caps(grant ino 0x5000177be10 135309857 seq 4812
caps=pAsxLsXsxFsxcrwb dirty=- wanted=pAsxXsxFxwb follows 0 size 1206/0
ts 756/0 mtime 2019-07-03 15:30:08.385521) v11 ==== 299+0+0
(1635817588 0 0) 0x5621252efa80 con 0x56211eefb800
2019-07-03 15:32:37.871171 7f30c4d72700 1 --
137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234
227527 ==== client_caps(grant ino 0x5000177be10 135309857 seq 4814
caps=pAsxLsXsxFsxcrwb dirty=- wanted=pAsxXsxFxwb follows 0 size
1206/4194304 ts 756/0 mtime 2019-07-03 15:30:08.385521) v11 ====
299+0+0 (3626990096 0 0) 0x56212361bb00 con 0x56211eefb800
2019-07-03 15:32:41.018963 7f30c4d72700 1 --
137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234
227528 ==== client_caps(flush_ack ino 0x5000177be10 135309857 seq 4814
tid 62216 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=- follows 0 size 0/0
mtime 0.000000) v11 ==== 252+0+0 (197724779 0 0) 0x56212b20e900 con
0x56211eefb800
2019-07-03 15:32:41.019026 7f30c4d72700 1 --
137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234
227529 ==== client_caps(flush_ack ino 0x5000177be10 135309857 seq 4814
tid 62217 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=- follows 0 size 0/0
mtime 0.000000) v11 ==== 252+0+0 (197724779 0 0) 0x5621252e9600 con
0x56211eefb800
2019-07-03 15:32:41.019047 7f30c4d72700 1 --
137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234
227530 ==== client_caps(flush_ack ino 0x5000177be10 135309857 seq 4814
tid 62218 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=- follows 0 size 0/0
mtime 0.000000) v11 ==== 252+0+0 (197724779 0 0) 0x562123402480 con
0x56211eefb800
With open/close outside the loop, there's very little conversation
with the mds... renewing caps and the occasional flush_ack from time
to time.
This is with 12.2.12 mds, 12.2.12 fuse client.
-- Dan
[1]
while(True):
f = open('out.dat', 'a') // append or truncate doesn't make a
difference to the client-mds traffic, afaict
f.write('aaa')
f.close()
> sage
>
>
> >
> > In a multi-user environment it's hard to scrutinize every user's
> > application, so we'd prefer to just throttle down the client req rates
> > (and let them suffer from the poor performance).
> >
> > Thoughts?
> >
> > Thanks,
> >
> > Dan
> > _______________________________________________
> > Dev mailing list -- dev(a)ceph.io
> > To unsubscribe send an email to dev-leave(a)ceph.io
> >
> >
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering
an mds from several hosts. In the end we found that their code was
doing:
while d=getafewbytesofdata():
f=open(file.dat)
f.append(d)
f.close()
By changing their code to:
f=open(file.dat)
while d=getafewbytesofdata():
f.append(d)
f.close()
it completely removes their load on the mds (for obvious reasons).
This is a tangential point, but: at some point in the past at least the
above two code blcoks used to incur the same load on the MDS because the
client would only asynchronously release the caps back to the MDS. Did we
change that behavior? Maybe we can restore it, but with a very a short
delay, so that we cover patterns like the above?
1766
days inactive
1767
days old
6 comments
4 participants
participants (4)
-
Dan van der Ster -
Gregory Farnum -
Jeff Layton -
Sage Weil