The rsync job has been copying quite happily for two hours now. The good
news is that the cache size isn't increasing unboundedly with each
request anymore. The bad news is that it still is increasing afterall,
though much slower. I am at 3M inodes now and it started off with 900k,
settling at 1M initially. I had a peak just now of 3.7M, but it went
back down to 3.2M shortly after that.
According to the health status, the client has started failing to
respond to cache pressure, so it's still not working as reliably as I
would like it to. I am also getting this very peculiar message:
MDS cache is too large (7GB/19GB); 52686 inodes in use by clients
I guess the 53k inodes is the number that is actively in use right now
(compared to the 3M for which the client generally holds caps). Is that
so? Cache memory is still well within bounds, however. Perhaps the
message is triggered by the recall settings and just a bit misleading?
On 25/07/2019 09:46, Janek Bevendorff wrote:
>
>> It's possible the MDS is not being aggressive enough with asking the
>> single (?) client to reduce its cache size. There were recent changes
>> [1] to the MDS to improve this. However, the defaults may not be
>> aggressive enough for your client's workload. Can you try:
>>
>> ceph config set mds mds_recall_max_caps 10000
>> ceph config set mds mds_recall_max_decay_rate 1.0
>
> Thank you. I was looking for config directives that do exactly this
> all week. Why are they not documented anywhere outside that blog post?
>
> I added them as you described and the MDS seems to have stabilized and
> stays just under 1M inos now. I will continue to monitor it and see if
> it is working in the long run. Settings like these should be the
> default IMHO. Clients should never be able to crash the server just by
> holding onto their capabilities. If a server decides to drop things
> from its cache, clients must deal with it. Everything else threatens
> the stability of the system (and may even prevent the MDS from ever
> starting again, as we saw).
>
>> Also your other mailings made me think you may still be using the old
>> inode limit for the cache size. Are you using the new
>> mds_cache_memory_limit config option?
>
> No, I am not. I tried it at some point to see if it made things
> better, but just like the memory cache limit, it seemed to have no
> effect whatsoever except for delaying the health warning.
>
>
>> Finally, if this fixes your issue (please let us know!) and you decide
>> to try multiple active MDS, you should definitely use pinning as the
>> parallel create workload will greatly benefit from it.
>
> I will try that, although I directory tree is quite imbalanced.
>
> _______________________________________________
> Ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io