what's the
ceph.com mailing list? I wondered
whether this list is dead but it's the list announced on the official
ceph.com
homepage, isn't it?
There are two mailing lists announced on the website. If you
go to
https://ceph.com/resources/ you will find the
subscribe/unsubscribe/archive links for the (much more active)
ceph.com
MLs. But if you click on "Mailing Lists & IRC page" you will get to a
page where you can subscribe to this list, which is different. Very
confusing.
What did you have the MDS cache size set to at the
time?
< and an inode count between
I actually did not think I'd get a reply here. We are a bit further than
this on the other mailing list. This is the thread:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/036095.html
To sum it up: the ceph client prevents the MDS from freeing its cache,
so inodes keep piling up until either the MDS becomes too slow (fixable
by increasing the beacon grace time) or runs out of memory. The latter
will happen eventually. In the end, my MDSs couldn't even rejoin because
they hit the host's 128GB memory limit and crashed.
This is probably related to using multiple active
metadata servers.
There have been stability issues we're still looking to work out with
the MDS balancer, especially with these batch-create style workloads.
However, if you're willing to use subtree pinning [1] where you
statically assign each directory tree before each rsync job uploads
its data, then that should be safe as the balancer will effectively be
disabled.
The same happens with only one MDS. I tried it with a fresh CephFS and
after two minutes of rsyncing stuff into it, I hit 900k inodes before I
stopped the process.
Alternatively, using one active MDS for the duration
of the batch
upload should work too but may be significantly slower.
The point was to have
multiple for the transfer, because one couldn't
keep up. But now it seems like the problem wasn't really the single-MDS
bottleneck. The problem was rather the unbounded growth of runaway
inodes, which I do not have a solution for.
Same error? Missed heartbeats?
Yeah, same thing.