It's possible the MDS is not being aggressive
enough with asking the
single (?) client to reduce its cache size. There were recent changes
 to the MDS to improve this. However, the defaults may not be
aggressive enough for your client's workload. Can you try:
ceph config set mds mds_recall_max_caps 10000
ceph config set mds mds_recall_max_decay_rate 1.0
Thank you. I was looking for config directives that do exactly this all
week. Why are they not documented anywhere outside that blog post?
I added them as you described and the MDS seems to have stabilized and
stays just under 1M inos now. I will continue to monitor it and see if
it is working in the long run. Settings like these should be the default
IMHO. Clients should never be able to crash the server just by holding
onto their capabilities. If a server decides to drop things from its
cache, clients must deal with it. Everything else threatens the
stability of the system (and may even prevent the MDS from ever starting
again, as we saw).
Also your other mailings made me think you may still
be using the old
inode limit for the cache size. Are you using the new
mds_cache_memory_limit config option?
No, I am not. I tried it at some point to see if it made things better,
but just like the memory cache limit, it seemed to have no effect
whatsoever except for delaying the health warning.
Finally, if this fixes your issue (please let us
know!) and you decide
to try multiple active MDS, you should definitely use pinning as the
parallel create workload will greatly benefit from it.
I will try that, although I directory tree is quite imbalanced.