Hi, For the last 2 months, our MDS is frequently switching to another because of a sudden memory leak. The host has 128G RAM and most of the time the MDS occupies ~20% of memory. And in less than 3 minutes it increases to 100% and crashs with tcmalloc: allocation failed. We tried to run heap stats / perf dump on the host but we couldn't find any reasons why the memory used by the MDS exploses so quickly. MDS log available here : https://filesender.renater.fr/?s=download&token=c1e60c3c-7f02-4f1e-b23e-f5b25c0cd2a8 Any idea what could lead to this memory leak? Anything we can try to understand what happens or prevent this? We use Pacific 16.2.14. Cheers, Adrien |cluster: id: 86cd8a68-7649-11ed-b2be-5cba2c7fdb30 health: HEALTH_OK services: mon: 5 daemons, quorum cccephadm40,cccephadm42,cccephadm41,cccephadm43,cccephadm44 (age 3d) mgr: cccephadm40.ucqhkr(active, since 105m), standbys: cccephadm41.osnxsd mds: 4/4 daemons up, 1 standby osd: 720 osds: 720 up (since 9d), 720 in (since 7w) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 12 pools, 8801 pgs objects: 1.25G objects, 2.7 PiB usage: 3.5 PiB used, 2.6 PiB / 6.1 PiB avail pgs: 8731 active+clean 59 active+clean+scrubbing+deep 11 active+clean+scrubbing io: client: 2.1 GiB/s rd, 1.2 GiB/s wr, 3.63k op/s rd, 17.47k op/s wr ceph fs status cephfs_astro - 1628 clients ============ RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs_astro.cccephadm41.nbgvxk Reqs: 7205 /s 11.2M 11.1M 708k 2782k 1 active cephfs_astro.cccephadm42.ieexgj Reqs: 812 /s 11.6M 11.5M 554k 1366k 2 active cephfs_astro.cccephadm40.aafhps Reqs: 1 /s 11.4M 11.3M 1346k 462k 3 active cephfs_astro.cccephadm44.jodfcx Reqs: 3 /s 11.4M 11.4M 1094k 29.3k POOL TYPE USED AVAIL cephfs_metadata metadata 484G 13.5T cephfs_default data 0 13.5T cephfs_EC_data data 3463T 1830T cephfs_ssd data 5314G 13.5T STANDBY MDS cephfs_astro.cccephadm43.wetlrp MDS version: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) |
Hi Adrien, On Mon, Jul 22, 2024 at 5:17 AM Adrien Georget <adrien.georget@cc.in2p3.fr> wrote:
Hi,
For the last 2 months, our MDS is frequently switching to another because of a sudden memory leak. The host has 128G RAM and most of the time the MDS occupies ~20% of memory. And in less than 3 minutes it increases to 100% and crashs with tcmalloc: allocation failed.
We tried to run heap stats / perf dump on the host but we couldn't find any reasons why the memory used by the MDS exploses so quickly. MDS log available here : https://filesender.renater.fr/?s=download&token=c1e60c3c-7f02-4f1e-b23e-f5b25c0cd2a8
Any idea what could lead to this memory leak? Anything we can try to understand what happens or prevent this? We use Pacific 16.2.14.
It is probably an instance of this: https://tracker.ceph.com/issues/66704 Check backports of an MDS fix here: https://tracker.ceph.com/issues/64977 If you can, using an older kernel or wait until the release is available with the backported fix. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi Patrick, Thanks for pointing this issue, it looks coherent with the timing of RHEL9 kernel client update on our side. We are going to confirm this by using only older clients on CentOS7. Cheers, Adrien Le 22/07/2024 à 16:23, Patrick Donnelly a écrit :
Hi Adrien,
On Mon, Jul 22, 2024 at 5:17 AM Adrien Georget <adrien.georget@cc.in2p3.fr> wrote:
Hi,
For the last 2 months, our MDS is frequently switching to another because of a sudden memory leak. The host has 128G RAM and most of the time the MDS occupies ~20% of memory. And in less than 3 minutes it increases to 100% and crashs with tcmalloc: allocation failed.
We tried to run heap stats / perf dump on the host but we couldn't find any reasons why the memory used by the MDS exploses so quickly. MDS log available here : https://filesender.renater.fr/?s=download&token=c1e60c3c-7f02-4f1e-b23e-f5b25c0cd2a8
Any idea what could lead to this memory leak? Anything we can try to understand what happens or prevent this? We use Pacific 16.2.14. It is probably an instance of this:
https://tracker.ceph.com/issues/66704
Check backports of an MDS fix here: https://tracker.ceph.com/issues/64977
If you can, using an older kernel or wait until the release is available with the backported fix.
participants (2)
-
Adrien Georget
-
Patrick Donnelly