Hi Patrick, Thanks for pointing this issue, it looks coherent with the timing of RHEL9 kernel client update on our side. We are going to confirm this by using only older clients on CentOS7. Cheers, Adrien Le 22/07/2024 à 16:23, Patrick Donnelly a écrit :
Hi Adrien,
On Mon, Jul 22, 2024 at 5:17 AM Adrien Georget <adrien.georget@cc.in2p3.fr> wrote:
Hi,
For the last 2 months, our MDS is frequently switching to another because of a sudden memory leak. The host has 128G RAM and most of the time the MDS occupies ~20% of memory. And in less than 3 minutes it increases to 100% and crashs with tcmalloc: allocation failed.
We tried to run heap stats / perf dump on the host but we couldn't find any reasons why the memory used by the MDS exploses so quickly. MDS log available here : https://filesender.renater.fr/?s=download&token=c1e60c3c-7f02-4f1e-b23e-f5b25c0cd2a8
Any idea what could lead to this memory leak? Anything we can try to understand what happens or prevent this? We use Pacific 16.2.14. It is probably an instance of this:
https://tracker.ceph.com/issues/66704
Check backports of an MDS fix here: https://tracker.ceph.com/issues/64977
If you can, using an older kernel or wait until the release is available with the backported fix.