I hope someone can help us with a MDS caching problem. Ceph version 18.2.4 with cephadm container deployment. Question 1: For me it's not clear how much cache/memory you should allocate for the MDS. Is this based on the number of open files, caps or something else? Question 2/Problem: At the moment we have MDS nodes with 32 GB of memory and a configured cache limit of 20 GB. There are 4 MDS nodes: 2 active and 2 in standby-replay mode (with max_mds set at 2 of course). We pinned top directories to specific ranks, so the balancer isn't used. The memory usage is for the most part increasing, sometimes a little dip with couple hundred MB's freed. After all the memory is consumed, SWAP gets used. This results in a couple of hundred MB's of freed memory, but not much. When eventually the SWAP runs out and the memory is full, the MDS service stops and the cluster logs show: 1. no beacon from mds 2. marking mds up:active laggy 3. replacing mds 4. MDS daemon <daemon> is removed because it is dead or otherwise unavailable For example: we have the top folder app2 and app4 which is pinnend to rank 1. Folder app2 is always accessed by 4 clients (application servers), the same happens with folder app4. Folder app2 is 3 times larger than folder app4 (last time I checked, don't wanna do a du at the moment). After a couple of hours the memory usage of the MDS server stays around 18% (Grafana shows a flatline for 7 hours). At night the 9the client connects and makes first a backup with rsync of the latest snapshot folder of app2 and afterwards the same happens for folder app4 with a pause for 5 minutes. When the backup starts, the memory increases to 70% and stays at 70% after the backup of app2 is completed. 5 minutes later the memory starts increases again with the start of the backup of folder app4. When the backup is done, it's at 78% and stays there for the rest of the day. Why isn't the memory usage decreasing after the rsync is completed? Is there a memory leak with the MDS service? Ps. I have some small log files/Grafana screenshots, not sure how to share. Kind regards, Sake
On Fri, Aug 30, 2024 at 9:22 PM Sake Ceph <ceph@paulusma.eu> wrote:
I hope someone can help us with a MDS caching problem.
Ceph version 18.2.4 with cephadm container deployment.
Question 1: For me it's not clear how much cache/memory you should allocate for the MDS. Is this based on the number of open files, caps or something else?
Question 2/Problem: At the moment we have MDS nodes with 32 GB of memory and a configured cache limit of 20 GB. There are 4 MDS nodes: 2 active and 2 in standby-replay mode (with max_mds set at 2 of course). We pinned top directories to specific ranks, so the balancer isn't used. The memory usage is for the most part increasing, sometimes a little dip with couple hundred MB's freed. After all the memory is consumed, SWAP gets used. This results in a couple of hundred MB's of freed memory, but not much. When eventually the SWAP runs out and the memory is full, the MDS service stops and the cluster logs show: 1. no beacon from mds 2. marking mds up:active laggy 3. replacing mds 4. MDS daemon <daemon> is removed because it is dead or otherwise unavailable
For example: we have the top folder app2 and app4 which is pinnend to rank 1. Folder app2 is always accessed by 4 clients (application servers), the same happens with folder app4. Folder app2 is 3 times larger than folder app4 (last time I checked, don't wanna do a du at the moment). After a couple of hours the memory usage of the MDS server stays around 18% (Grafana shows a flatline for 7 hours). At night the 9the client connects and makes first a backup with rsync of the latest snapshot folder of app2 and afterwards the same happens for folder app4 with a pause for 5 minutes. When the backup starts, the memory increases to 70% and stays at 70% after the backup of app2 is completed. 5 minutes later the memory starts increases again with the start of the backup of folder app4. When the backup is done, it's at 78% and stays there for the rest of the day. Why isn't the memory usage decreasing after the rsync is completed?
Is there a memory leak with the MDS service?
Ps. I have some small log files/Grafana screenshots, not sure how to share.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Sake, There was a known memory leak in MDS standby-replay: https://tracker.ceph.com/issues/48673. However, it is supposedly fixed in version 18.2.4, which you are running. I don't have any non-test cluster on Reef yet, and so can't tell for sure if this is true. Therefore, I guess what you see might be a different issue. Could you please disable standby-replay and retest whether the memory leak still exists? If so, it would be sufficient proof that it is not the same issue. ceph fs set cephfs allow_standby_replay 0 -- Alexander Patrakov
@Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem. @Alexander: the problem is in the active nodes, the standby replay don't have issues anymore. Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. It looks to me the MDS server is caching something forever while it isn't being used.. The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts. Kind regards, Sake
Hello Sake, The combination of two active MDSs and RHEL8 does ring a bell, and I have seen this with Quincy, too. However, what's relevant is the kernel version on the clients. If they run the default 4.18.x kernel from RHEL8, please either upgrade to the mainline kernel or decrease max_mds to 1. If they run a modern kernel, then it is something I do not know about. On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@paulusma.eu> wrote:
@Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem.
@Alexander: the problem is in the active nodes, the standby replay don't have issues anymore.
Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. It looks to me the MDS server is caching something forever while it isn't being used..
The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
I was talking about the hosts where the MDS containers are running on. The clients are all RHEL 9. Kind regards, Sake
Op 31-08-2024 08:34 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Hello Sake,
The combination of two active MDSs and RHEL8 does ring a bell, and I have seen this with Quincy, too. However, what's relevant is the kernel version on the clients. If they run the default 4.18.x kernel from RHEL8, please either upgrade to the mainline kernel or decrease max_mds to 1. If they run a modern kernel, then it is something I do not know about.
On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@paulusma.eu> wrote:
@Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem.
@Alexander: the problem is in the active nodes, the standby replay don't have issues anymore.
Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. It looks to me the MDS server is caching something forever while it isn't being used..
The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
Got it. However, to narrow down the issue, I suggest that you test whether it still exists after the following changes: 1. Reduce max_mds to 1. 2. Do not reduce max_mds to 1, but migrate all clients from a direct CephFS mount to NFS. On Sat, Aug 31, 2024 at 2:55 PM Sake Ceph <ceph@paulusma.eu> wrote:
I was talking about the hosts where the MDS containers are running on. The clients are all RHEL 9.
Kind regards, Sake
Op 31-08-2024 08:34 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Hello Sake,
The combination of two active MDSs and RHEL8 does ring a bell, and I have seen this with Quincy, too. However, what's relevant is the kernel version on the clients. If they run the default 4.18.x kernel from RHEL8, please either upgrade to the mainline kernel or decrease max_mds to 1. If they run a modern kernel, then it is something I do not know about.
On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@paulusma.eu> wrote:
@Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem.
@Alexander: the problem is in the active nodes, the standby replay don't have issues anymore.
Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. It looks to me the MDS server is caching something forever while it isn't being used..
The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
-- Alexander Patrakov
It was worse with 1 MDS, therefor we moved to 2 active MDS with directory pinning (so the balancer won't be an issue/make things extra complicated). The number of caps stay for the most part the same, some ups and downs. I would guess it maybe has something to do with caching the accessed directories or files. Because it increases a lot the first time when using rsync and the second time there isn't really an increase of memory usage, only for a little time when the rsync is running and afterwards it drops again. NFS isn't really an option because it adds another hop for the clients :( Second it happens on our Production environment and I won't be making any changes there for a test. Will try to replicate in our staging environment, but that one has a lot less load on it. Kind regards, Sake
Op 31-08-2024 09:15 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Got it.
However, to narrow down the issue, I suggest that you test whether it still exists after the following changes:
1. Reduce max_mds to 1. 2. Do not reduce max_mds to 1, but migrate all clients from a direct CephFS mount to NFS.
On Sat, Aug 31, 2024 at 2:55 PM Sake Ceph <ceph@paulusma.eu> wrote:
I was talking about the hosts where the MDS containers are running on. The clients are all RHEL 9.
Kind regards, Sake
Op 31-08-2024 08:34 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Hello Sake,
The combination of two active MDSs and RHEL8 does ring a bell, and I have seen this with Quincy, too. However, what's relevant is the kernel version on the clients. If they run the default 4.18.x kernel from RHEL8, please either upgrade to the mainline kernel or decrease max_mds to 1. If they run a modern kernel, then it is something I do not know about.
On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@paulusma.eu> wrote:
@Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem.
@Alexander: the problem is in the active nodes, the standby replay don't have issues anymore.
Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. It looks to me the MDS server is caching something forever while it isn't being used..
The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
-- Alexander Patrakov
Ow it got worse after the upgrade to Reef (was running Quincy). With Quincy the memory usage was also a lot of times around 95% and some swap usage, but never exceeding both to the point of crashing. Kind regards, Sake
Op 31-08-2024 09:15 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Got it.
However, to narrow down the issue, I suggest that you test whether it still exists after the following changes:
1. Reduce max_mds to 1. 2. Do not reduce max_mds to 1, but migrate all clients from a direct CephFS mount to NFS.
On Sat, Aug 31, 2024 at 2:55 PM Sake Ceph <ceph@paulusma.eu> wrote:
I was talking about the hosts where the MDS containers are running on. The clients are all RHEL 9.
Kind regards, Sake
Op 31-08-2024 08:34 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Hello Sake,
The combination of two active MDSs and RHEL8 does ring a bell, and I have seen this with Quincy, too. However, what's relevant is the kernel version on the clients. If they run the default 4.18.x kernel from RHEL8, please either upgrade to the mainline kernel or decrease max_mds to 1. If they run a modern kernel, then it is something I do not know about.
On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@paulusma.eu> wrote:
@Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem.
@Alexander: the problem is in the active nodes, the standby replay don't have issues anymore.
Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. It looks to me the MDS server is caching something forever while it isn't being used..
The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
As a workaround, to reduce the impact of the MDS slowed down by excessive memory consumption, I would suggest installing earlyoom, disabling swap, and configuring earlyoom as follows (usually through /etc/sysconfig/earlyoom, but could be in a different place on your distribution): EARLYOOM_ARGS="-p -r 600 -m 4,4 -s 1,1" On Sat, Aug 31, 2024 at 3:44 PM Sake Ceph <ceph@paulusma.eu> wrote:
Ow it got worse after the upgrade to Reef (was running Quincy). With Quincy the memory usage was also a lot of times around 95% and some swap usage, but never exceeding both to the point of crashing.
Kind regards, Sake
Op 31-08-2024 09:15 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Got it.
However, to narrow down the issue, I suggest that you test whether it still exists after the following changes:
1. Reduce max_mds to 1. 2. Do not reduce max_mds to 1, but migrate all clients from a direct CephFS mount to NFS.
On Sat, Aug 31, 2024 at 2:55 PM Sake Ceph <ceph@paulusma.eu> wrote:
I was talking about the hosts where the MDS containers are running on. The clients are all RHEL 9.
Kind regards, Sake
Op 31-08-2024 08:34 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Hello Sake,
The combination of two active MDSs and RHEL8 does ring a bell, and I have seen this with Quincy, too. However, what's relevant is the kernel version on the clients. If they run the default 4.18.x kernel from RHEL8, please either upgrade to the mainline kernel or decrease max_mds to 1. If they run a modern kernel, then it is something I do not know about.
On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@paulusma.eu> wrote:
@Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem.
@Alexander: the problem is in the active nodes, the standby replay don't have issues anymore.
Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. It looks to me the MDS server is caching something forever while it isn't being used..
The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
Can you tell if the number of objects increases in your cephfs between those bursts? I noticed something similar in a 16.2.15 cluster as well. It's not that heavily used, but it contains home directories and development working directories etc. And when one user checked out a git project, the mds memory usage increased a lot, getting near its configured limit. Before there were around 3,7 Million objects in the cephfs, that user added more than a million more files with his checkout. It wasn't a real issue (yet) because the usage isn't very dynamical and the total number of files is relatively stable. This doesn't really help resolve anything, but if your total number of files grows, I'm not surprised that the mds requires more memory. Zitat von Alexander Patrakov <patrakov@gmail.com>:
As a workaround, to reduce the impact of the MDS slowed down by excessive memory consumption, I would suggest installing earlyoom, disabling swap, and configuring earlyoom as follows (usually through /etc/sysconfig/earlyoom, but could be in a different place on your distribution):
EARLYOOM_ARGS="-p -r 600 -m 4,4 -s 1,1"
On Sat, Aug 31, 2024 at 3:44 PM Sake Ceph <ceph@paulusma.eu> wrote:
Ow it got worse after the upgrade to Reef (was running Quincy). With Quincy the memory usage was also a lot of times around 95% and some swap usage, but never exceeding both to the point of crashing.
Kind regards, Sake
Op 31-08-2024 09:15 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Got it.
However, to narrow down the issue, I suggest that you test whether it still exists after the following changes:
1. Reduce max_mds to 1. 2. Do not reduce max_mds to 1, but migrate all clients from a direct CephFS mount to NFS.
On Sat, Aug 31, 2024 at 2:55 PM Sake Ceph <ceph@paulusma.eu> wrote:
I was talking about the hosts where the MDS containers are
running on. The clients are all RHEL 9.
Kind regards, Sake
Op 31-08-2024 08:34 CEST schreef Alexander Patrakov
<patrakov@gmail.com>:
Hello Sake,
The combination of two active MDSs and RHEL8 does ring a bell, and I have seen this with Quincy, too. However, what's relevant is the kernel version on the clients. If they run the default 4.18.x kernel from RHEL8, please either upgrade to the mainline kernel or decrease max_mds to 1. If they run a modern kernel, then it is something I do not know about.
On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@paulusma.eu> wrote:
@Anthony: it's a small virtualized cluster and indeed SWAP
shouldn't be used, but this doesn't change the problem.
@Alexander: the problem is in the active nodes, the standby
replay don't have issues anymore.
Last night's backup run increased the memory usage to 86%
when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%.
It looks to me the MDS server is caching something forever while it isn't being used..
The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The folders contain a couple of million files, but are really static. We have another folder with a lot of updates and the MDS server for that folder has indeed a continuous increase of memory usage. But I would focus on the app2 and app4 folders, because those have a lot less changes in it. But why keeps the MDS al this information in its memory? If it isn't accessed for more than 20 hours, it should release it in my opinion (even a lot earlier, like after an hour). Kind regards, Sake
Op 02-09-2024 09:33 CEST schreef Eugen Block <eblock@nde.ag>:
Can you tell if the number of objects increases in your cephfs between those bursts? I noticed something similar in a 16.2.15 cluster as well. It's not that heavily used, but it contains home directories and development working directories etc. And when one user checked out a git project, the mds memory usage increased a lot, getting near its configured limit. Before there were around 3,7 Million objects in the cephfs, that user added more than a million more files with his checkout. It wasn't a real issue (yet) because the usage isn't very dynamical and the total number of files is relatively stable. This doesn't really help resolve anything, but if your total number of files grows, I'm not surprised that the mds requires more memory.
Zitat von Alexander Patrakov <patrakov@gmail.com>:
As a workaround, to reduce the impact of the MDS slowed down by excessive memory consumption, I would suggest installing earlyoom, disabling swap, and configuring earlyoom as follows (usually through /etc/sysconfig/earlyoom, but could be in a different place on your distribution):
EARLYOOM_ARGS="-p -r 600 -m 4,4 -s 1,1"
On Sat, Aug 31, 2024 at 3:44 PM Sake Ceph <ceph@paulusma.eu> wrote:
Ow it got worse after the upgrade to Reef (was running Quincy). With Quincy the memory usage was also a lot of times around 95% and some swap usage, but never exceeding both to the point of crashing.
Kind regards, Sake
Op 31-08-2024 09:15 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Got it.
However, to narrow down the issue, I suggest that you test whether it still exists after the following changes:
1. Reduce max_mds to 1. 2. Do not reduce max_mds to 1, but migrate all clients from a direct CephFS mount to NFS.
On Sat, Aug 31, 2024 at 2:55 PM Sake Ceph <ceph@paulusma.eu> wrote:
I was talking about the hosts where the MDS containers are
running on. The clients are all RHEL 9.
Kind regards, Sake
Op 31-08-2024 08:34 CEST schreef Alexander Patrakov
<patrakov@gmail.com>:
Hello Sake,
The combination of two active MDSs and RHEL8 does ring a bell, and I have seen this with Quincy, too. However, what's relevant is the kernel version on the clients. If they run the default 4.18.x kernel from RHEL8, please either upgrade to the mainline kernel or decrease max_mds to 1. If they run a modern kernel, then it is something I do not know about.
On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@paulusma.eu> wrote: > > @Anthony: it's a small virtualized cluster and indeed SWAP
shouldn't be used, but this doesn't change the problem.
> > @Alexander: the problem is in the active nodes, the standby replay don't have issues anymore. > > Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. > It looks to me the MDS server is caching something forever while it isn't being used.. > > The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts. > > Kind regards, > Sake > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
MDS cannot release an inode if a client has cached it (and thus can have newer data than OSDs have). The MDS needs to know at least which client to ask if someone else requests the same file. MDS does ask clients to release caps, but sometimes this doesn't work, and there is no good troubleshooting guide except trying different kernel versions and switching between kernel client / fuse / nfs. On Mon, Sep 2, 2024 at 7:40 PM Sake Ceph <ceph@paulusma.eu> wrote:
The folders contain a couple of million files, but are really static. We have another folder with a lot of updates and the MDS server for that folder has indeed a continuous increase of memory usage. But I would focus on the app2 and app4 folders, because those have a lot less changes in it. But why keeps the MDS al this information in its memory? If it isn't accessed for more than 20 hours, it should release it in my opinion (even a lot earlier, like after an hour).
Kind regards, Sake
Op 02-09-2024 09:33 CEST schreef Eugen Block <eblock@nde.ag>:
Can you tell if the number of objects increases in your cephfs between those bursts? I noticed something similar in a 16.2.15 cluster as well. It's not that heavily used, but it contains home directories and development working directories etc. And when one user checked out a git project, the mds memory usage increased a lot, getting near its configured limit. Before there were around 3,7 Million objects in the cephfs, that user added more than a million more files with his checkout. It wasn't a real issue (yet) because the usage isn't very dynamical and the total number of files is relatively stable. This doesn't really help resolve anything, but if your total number of files grows, I'm not surprised that the mds requires more memory.
Zitat von Alexander Patrakov <patrakov@gmail.com>:
As a workaround, to reduce the impact of the MDS slowed down by excessive memory consumption, I would suggest installing earlyoom, disabling swap, and configuring earlyoom as follows (usually through /etc/sysconfig/earlyoom, but could be in a different place on your distribution):
EARLYOOM_ARGS="-p -r 600 -m 4,4 -s 1,1"
On Sat, Aug 31, 2024 at 3:44 PM Sake Ceph <ceph@paulusma.eu> wrote:
Ow it got worse after the upgrade to Reef (was running Quincy). With Quincy the memory usage was also a lot of times around 95% and some swap usage, but never exceeding both to the point of crashing.
Kind regards, Sake
Op 31-08-2024 09:15 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
Got it.
However, to narrow down the issue, I suggest that you test whether it still exists after the following changes:
1. Reduce max_mds to 1. 2. Do not reduce max_mds to 1, but migrate all clients from a direct CephFS mount to NFS.
On Sat, Aug 31, 2024 at 2:55 PM Sake Ceph <ceph@paulusma.eu> wrote:
I was talking about the hosts where the MDS containers are
running on. The clients are all RHEL 9.
Kind regards, Sake
> Op 31-08-2024 08:34 CEST schreef Alexander Patrakov
<patrakov@gmail.com>:
> > > Hello Sake, > > The combination of two active MDSs and RHEL8 does ring a bell, and I > have seen this with Quincy, too. However, what's relevant is the > kernel version on the clients. If they run the default 4.18.x kernel > from RHEL8, please either upgrade to the mainline kernel or decrease > max_mds to 1. If they run a modern kernel, then it is something I do > not know about. > > On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@paulusma.eu> wrote: > > > > @Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem. > > > > @Alexander: the problem is in the active nodes, the standby replay don't have issues anymore. > > > > Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. > > It looks to me the MDS server is caching something forever while it isn't being used.. > > > > The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts. > > > > Kind regards, > > Sake > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > -- > Alexander Patrakov
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
But the client which is doing the rsync, doesn't hold any caps after the rsync. Cephfs-top shows 0 caps. Even a system reboot of the client doesn't make a change. Kind regards, Sake
Op 03-09-2024 04:01 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
MDS cannot release an inode if a client has cached it (and thus can have newer data than OSDs have). The MDS needs to know at least which client to ask if someone else requests the same file.
MDS does ask clients to release caps, but sometimes this doesn't work, and there is no good troubleshooting guide except trying different kernel versions and switching between kernel client / fuse / nfs.
Ok, thanks for the clarification. This does disprove my theory. On Wed, Sep 4, 2024 at 12:30 AM Sake Ceph <ceph@paulusma.eu> wrote:
But the client which is doing the rsync, doesn't hold any caps after the rsync. Cephfs-top shows 0 caps. Even a system reboot of the client doesn't make a change.
Kind regards, Sake
Op 03-09-2024 04:01 CEST schreef Alexander Patrakov <patrakov@gmail.com>:
MDS cannot release an inode if a client has cached it (and thus can have newer data than OSDs have). The MDS needs to know at least which client to ask if someone else requests the same file.
MDS does ask clients to release caps, but sometimes this doesn't work, and there is no good troubleshooting guide except trying different kernel versions and switching between kernel client / fuse / nfs.
-- Alexander Patrakov
participants (3)
-
Alexander Patrakov
-
Eugen Block
-
Sake Ceph