On Wed, Jul 3, 2019 at 2:58 PM Sage Weil <sage@newdream.net> wrote:
On Tue, 2 Jul 2019, Dan van der Ster wrote:
Hi,
Are there any plans to implement a per-client throttle on mds client requests?
We just had an interesting case where a new cephfs user was hammering an mds from several hosts. In the end we found that their code was doing:
while d=getafewbytesofdata(): f=open(file.dat) f.append(d) f.close()
By changing their code to:
f=open(file.dat) while d=getafewbytesofdata(): f.append(d) f.close()
it completely removes their load on the mds (for obvious reasons).
This is a tangential point, but: at some point in the past at least the above two code blcoks used to incur the same load on the MDS because the client would only asynchronously release the caps back to the MDS. Did we change that behavior? Maybe we can restore it, but with a very a short delay, so that we cover patterns like the above?
With a simple python test, putting the open/close inside the loop [1], I see an update ino req to the mds after each close: 2019-07-03 15:30:08.012349 7f30bfd68700 1 -- 137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 -- client_request(unknown.0:31983 open #0x5000177be10 2019-07-03 15:30:08.012345 caller_uid=0, caller_gid=0{0,}) v4 -- 0x56212b328300 con 0 2019-07-03 15:30:08.016138 7f30bd563700 1 -- 137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 -- client_caps(update ino 0x5000177be10 135309857 seq 4811 tid 61814 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=pAsxXsxFxcwb follows 1 size 3/3 ts 756/0 mtime 2019-07-03 15:30:08.015825) v11 -- 0x562126b5bf80 con 0 2019-07-03 15:30:08.016950 7f30c0569700 1 -- 137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 -- client_caps(update ino 0x5000177be10 135309857 seq 4811 tid 61815 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=pAsxXsxFxcwb follows 1 size 6/3 ts 756/0 mtime 2019-07-03 15:30:08.016637) v11 -- 0x56212728e880 con 0 2019-07-03 15:30:08.017848 7f30bf567700 1 -- 137.138.157.69:0/165379415 --> 137.138.13.153:6800/2391910234 -- client_caps(update ino 0x5000177be10 135309857 seq 4811 tid 61816 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=pAsxXsxFxcwb follows 1 size 9/3 ts 756/0 mtime 2019-07-03 15:30:08.017439) v11 -- 0x56212148b180 con 0 and the flush_ack's from the mds (not exactly corresponding to above): 2019-07-03 15:32:37.831384 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227525 ==== client_caps(grant ino 0x5000177be10 135309857 seq 4812 caps=pAsxLsXsxFsxcrwb dirty=- wanted=pAsxXsxFxwb follows 0 size 1206/0 ts 756/0 mtime 2019-07-03 15:30:08.385521) v11 ==== 299+0+0 (1635817588 0 0) 0x5621252efa80 con 0x56211eefb800 2019-07-03 15:32:37.871171 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227527 ==== client_caps(grant ino 0x5000177be10 135309857 seq 4814 caps=pAsxLsXsxFsxcrwb dirty=- wanted=pAsxXsxFxwb follows 0 size 1206/4194304 ts 756/0 mtime 2019-07-03 15:30:08.385521) v11 ==== 299+0+0 (3626990096 0 0) 0x56212361bb00 con 0x56211eefb800 2019-07-03 15:32:41.018963 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227528 ==== client_caps(flush_ack ino 0x5000177be10 135309857 seq 4814 tid 62216 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=- follows 0 size 0/0 mtime 0.000000) v11 ==== 252+0+0 (197724779 0 0) 0x56212b20e900 con 0x56211eefb800 2019-07-03 15:32:41.019026 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227529 ==== client_caps(flush_ack ino 0x5000177be10 135309857 seq 4814 tid 62217 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=- follows 0 size 0/0 mtime 0.000000) v11 ==== 252+0+0 (197724779 0 0) 0x5621252e9600 con 0x56211eefb800 2019-07-03 15:32:41.019047 7f30c4d72700 1 -- 137.138.157.69:0/165379415 <== mds.4 137.138.13.153:6800/2391910234 227530 ==== client_caps(flush_ack ino 0x5000177be10 135309857 seq 4814 tid 62218 caps=pAsxLsXsxFsxcrwb dirty=Fw wanted=- follows 0 size 0/0 mtime 0.000000) v11 ==== 252+0+0 (197724779 0 0) 0x562123402480 con 0x56211eefb800 With open/close outside the loop, there's very little conversation with the mds... renewing caps and the occasional flush_ack from time to time. This is with 12.2.12 mds, 12.2.12 fuse client. -- Dan [1] while(True): f = open('out.dat', 'a') // append or truncate doesn't make a difference to the client-mds traffic, afaict f.write('aaa') f.close()
sage
In a multi-user environment it's hard to scrutinize every user's application, so we'd prefer to just throttle down the client req rates (and let them suffer from the poor performance).
Thoughts?
Thanks,
Dan _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io