memory leak in mds?
Hi all, We’re experiencing an issue with CephFS. I think we are facing this issue <https://tracker.ceph.com/issues/64977>. The main symptom is that the MDS starts using a lot of memory within a few minutes and finally it gets killed by OS (Out Of Memory). Sometimes it happens once a week and sometimes 2 times a day. We are running ceph quincy 17.2.7 on both the cluster and clients. I have read through some emails on the mailing list about it, but I didn't find a workaround. Does anyone have any suggestions? Thanks in advance. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es
Hi Dario, A workaround may be to downgrade client's kernel or ceph-fuse version to a lower version than those listed in Enrico's comment #22, I believe. Can't say for sure though since I couldn't verify it myself. Cheers, Frédéric. ________________________________ De : Dario Graña <dgrana@pic.es> Envoyé : vendredi 16 août 2024 16:41 À : ceph-users Objet : [ceph-users] memory leak in mds? Hi all, We’re experiencing an issue with CephFS. I think we are facing this issue <https://tracker.ceph.com/issues/64977>. The main symptom is that the MDS starts using a lot of memory within a few minutes and finally it gets killed by OS (Out Of Memory). Sometimes it happens once a week and sometimes 2 times a day. We are running ceph quincy 17.2.7 on both the cluster and clients. I have read through some emails on the mailing list about it, but I didn't find a workaround. Does anyone have any suggestions? Thanks in advance. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[cc Xiubo] On Fri, Aug 16, 2024 at 8:10 PM Dario Graña <dgrana@pic.es> wrote:
Hi all, We’re experiencing an issue with CephFS. I think we are facing this issue <https://tracker.ceph.com/issues/64977>. The main symptom is that the MDS starts using a lot of memory within a few minutes and finally it gets killed by OS (Out Of Memory). Sometimes it happens once a week and sometimes 2 times a day. We are running ceph quincy 17.2.7 on both the cluster and clients. I have read through some emails on the mailing list about it, but I didn't find a workaround. Does anyone have any suggestions?
It is likely that you are running into the issue described in the mentioned tracker. The change is pending backport to quincy as of now, so the other alternative approach might be downgrading the client.
Thanks in advance.
-- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
Thank you Frédéric and Venky for your answers. I will try to do some tests before changing the production environment. On Mon, Aug 19, 2024 at 8:53 AM Venky Shankar <vshankar@redhat.com> wrote:
[cc Xiubo]
On Fri, Aug 16, 2024 at 8:10 PM Dario Graña <dgrana@pic.es> wrote:
Hi all, We’re experiencing an issue with CephFS. I think we are facing this issue <https://tracker.ceph.com/issues/64977>. The main symptom is that the
MDS
starts using a lot of memory within a few minutes and finally it gets killed by OS (Out Of Memory). Sometimes it happens once a week and sometimes 2 times a day. We are running ceph quincy 17.2.7 on both the cluster and clients. I have read through some emails on the mailing list about it, but I didn't find a workaround. Does anyone have any suggestions?
It is likely that you are running into the issue described in the mentioned tracker. The change is pending backport to quincy as of now, so the other alternative approach might be downgrading the client.
Thanks in advance.
-- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
I was testing to downgrade the *ceph-common* package for a client with alma9, the same OS we use in production. I was trying to install the *ceph-common-17.2.6* package since the troubles began in 17.2.7. But I face a dependency problem nothing provides libthrift-0.14.0.so()(64bit) needed by ceph-common-2:17.2.6-0.el9.x86_64 from Ceph When I try to install with de --nobest flag, dnf suggest to install *16.2.4-5.el9* version. I will test it in a non-production environment, but I want to know if you look safe to use the 16.2.4-5 client version against a 17.2.7 cluster. Thanks in advance. On Mon, Aug 19, 2024 at 10:10 AM Dario Graña <dgrana@pic.es> wrote:
Thank you Frédéric and Venky for your answers. I will try to do some tests before changing the production environment.
On Mon, Aug 19, 2024 at 8:53 AM Venky Shankar <vshankar@redhat.com> wrote:
[cc Xiubo]
On Fri, Aug 16, 2024 at 8:10 PM Dario Graña <dgrana@pic.es> wrote:
Hi all, We’re experiencing an issue with CephFS. I think we are facing this
issue
<https://tracker.ceph.com/issues/64977>. The main symptom is that the MDS starts using a lot of memory within a few minutes and finally it gets killed by OS (Out Of Memory). Sometimes it happens once a week and sometimes 2 times a day. We are running ceph quincy 17.2.7 on both the cluster and clients. I have read through some emails on the mailing list about it, but I didn't find a workaround. Does anyone have any suggestions?
It is likely that you are running into the issue described in the mentioned tracker. The change is pending backport to quincy as of now, so the other alternative approach might be downgrading the client.
Thanks in advance.
-- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
participants (3)
-
Dario Graña
-
Frédéric Nass
-
Venky Shankar