Hi Stefan, Can you tell if the memory being used is due to the cache not being trimmed fast enough or something else? You might want to try to see if you can track down if the 6.5.1 client isn't releasing CAPS properly or something. Dan Van der Ster might have some insight here as well. Mark On 9/19/23 03:57, Stefan Kooman wrote:
Hi List,
For those of you that are brave enough to run 6.5 CephFS kernel client, we are seeing some interesting things happening. Some of this might be related to this thread [1]. On a couple of shared webhosting platforms we are running CephFS with 6.5.1 kernel. We have disabled "workqueue.cpu_intensive_thresh_us=0" (to prevent CephFS events from seen as cpu intensive). We have seen two MDS OOM situations after that. The MDS allocates ~ 60 GiB of RAM above baseline in ~ 50 seconds. In both OOM situations, a little before the OOM happens, there is a spike of network traffic going out of the MDS to a kernel client (6.5.1). That node gets ~ 700 MiB/s of MDS traffic for also ~ 50 seconds before the MDS process gets killed. Nothing is logged about this. Ceph is HEALTH_OK, no logging by kernel client or MDS whatsoever. The MDS rejoins and is up and active after a couple of minutes. There is no increased load on the MDS or the client that explain this (for as far as we can see).
At this point I don't expect anyone to tell me based on these symptoms what the issue is. But if you encounter similar issues, please update this thread. I'm pretty certain we are hitting a bug (or bugs), as the MDS should not blow itself up like that in any case (but evict the client (that misbehaves?).
Ceph MDS 16.2.11, MDS_MEMORY_TARGET=160GiB.
Gr. Stefan
[1]: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YR5UNKBOKDHP... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/