I've seen your other thread. Using 78GB of RAM when the memory limit is set to 64GB is not highly unusual, and doesn't necessarily indicate any problem. It *would* be a problem if the MDS memory grows uncontrollably, however. Otherwise, check those new defaults for caps recall -- they were released around 14.2.19 IIRC. -- Dan On Wed, May 26, 2021 at 12:46 PM Andres Rojas Guerrero <a.rojas(a)csic.es> wrote: > > Thanks for the answer. Yes, during these last weeks I have had memory > consumption problems in the MDS nodes that led, at least it seemed to > me, to performance problems in CephFS. I have been varying, for example: > > mds_cache_memory_limit > mds_min_caps_per_client > mds_health_cache_threshold > mds_max_caps_per_client > mds_cache_reservation > > But without much knowledge and with a trial and error procedure, i.e. > observing how CephFS behaved when changing one of the parameters. > Although I have achieved improvement the procedure does not convince me > at all and that's why I was asking if there was something more reliable ... > > > > > El 26/5/21 a las 12:15, Dan van der Ster escribió: > > Hi, > > > > The mds_cache_memory_limit should be set to something relative to the > > RAM size of the MDS -- maybe 50% is a good rule of thumb, because > > there are a few cases where the RSS can exceed this limit. Your > > experience will help guide what size you need (metadata pool IO > > activity will be really high if the MDS cache is too small) > > > > Otherwise, in recent releases of N/O/P the defaults for those settings > > you mentioned are quite good [1]; I would be surprised if they need > > further tuning for 99% of users. > > Is there any reason you want to start adjusting these params? > > > > Best Regards, > > > > Dan > > > > [1] https://github.com/ceph/ceph/pull/38574 > > > > On Wed, May 26, 2021 at 11:58 AM Andres Rojas Guerrero <a.rojas(a)csic.es> wrote: > >> > >> Hi all, I have observed that the MDS Cache Configuration has 18 parameters: > >> > >> mds_cache_memory_limit > >> mds_cache_reservation > >> mds_health_cache_threshold > >> mds_cache_trim_threshold > >> mds_cache_trim_decay_rate > >> mds_recall_max_caps > >> mds_recall_max_decay_threshold > >> mds_recall_max_decay_rate > >> mds_recall_global_max_decay_threshold > >> mds_recall_warning_threshold > >> mds_recall_warning_decay_rate > >> mds_session_cap_acquisition_throttle > >> mds_session_cap_acquisition_decay_rate > >> mds_session_max_caps_throttle_ratio > >> mds_cap_acquisition_throttle_retry_request_timeout > >> mds_session_cache_liveness_magnitude > >> mds_session_cache_liveness_decay_rate > >> mds_max_caps_per_client > >> > >> > >> > >> I find the Ceph documentation in this section a bit cryptic and I have > >> tried to find some resources that talk about how to tune these > >> parameters, but without success. > >> > >> Does anyone have experience in adjusting these parameters according to > >> the characteristics of the Ceph cluster itself, the hardware and the use > >> of MDS? > >> > >> Regards! > >> -- > >> ******************************************************* > >> Andrés Rojas Guerrero > >> Unidad Sistemas Linux > >> Area Arquitectura Tecnológica > >> Secretaría General Adjunta de Informática > >> Consejo Superior de Investigaciones Científicas (CSIC) > >> Pinar 19 > >> 28006 - Madrid > >> Tel: +34 915680059 -- Ext. 990059 > >> email: a.rojas(a)csic.es > >> ID comunicate.csic.es: @50852720l:matrix.csic.es > >> ******************************************************* > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > ******************************************************* > Andrés Rojas Guerrero > Unidad Sistemas Linux > Area Arquitectura Tecnológica > Secretaría General Adjunta de Informática > Consejo Superior de Investigaciones Científicas (CSIC) > Pinar 19 > 28006 - Madrid > Tel: +34 915680059 -- Ext. 990059 > email: a.rojas(a)csic.es > ID comunicate.csic.es: @50852720l:matrix.csic.es > *******************************************************

FS_DEGRADED indicates that your MDS restarted or stopped responding to health beacons. Are your MDSs going OOM? I see you have two active MDSs. Is your cluster more stable if you use only one single active MDS? -- Dan On Wed, May 26, 2021 at 2:44 PM Andres Rojas Guerrero <a.rojas(a)csic.es> wrote: > > Ok thank's, I will try to update Nautilus. But really I don't > understand the problem, apparently randomly Warnings appear: > > [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) > > cluster [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is > degraded) > > : cluster [DBG] mds.? > [v2:10.100.190.39:6800/2624951349,v1:10.100.190.39:6801/2624951349] > up:rejoin > 2021-05-26 10:55:33.215102 mon.ceph2mon01 (mon.0) 700 : cluster [DBG] > fsmap nxtclfs:2/2 {0=ceph2mon03=up:rejoin,1=ceph2mon01=up:active} 1 > up:standby > > Degrading the filesystem and I have assumed that the problem is due to > the memory consumption of the MDS process, which can reach around 80% or > more of the total memory. > > > > > > El 26/5/21 a las 13:21, Dan van der Ster escribió: > > I've seen your other thread. Using 78GB of RAM when the memory limit > > is set to 64GB is not highly unusual, and doesn't necessarily indicate > > any problem. > > It *would* be a problem if the MDS memory grows uncontrollably, however. > > > > Otherwise, check those new defaults for caps recall -- they were > > released around 14.2.19 IIRC. > > > > -- Dan > > > > On Wed, May 26, 2021 at 12:46 PM Andres Rojas Guerrero <a.rojas(a)csic.es> wrote: > >> > >> Thanks for the answer. Yes, during these last weeks I have had memory > >> consumption problems in the MDS nodes that led, at least it seemed to > >> me, to performance problems in CephFS. I have been varying, for example: > >> > >> mds_cache_memory_limit > >> mds_min_caps_per_client > >> mds_health_cache_threshold > >> mds_max_caps_per_client > >> mds_cache_reservation > >> > >> But without much knowledge and with a trial and error procedure, i.e. > >> observing how CephFS behaved when changing one of the parameters. > >> Although I have achieved improvement the procedure does not convince me > >> at all and that's why I was asking if there was something more reliable ... > >> > >> > >> > >> > >> El 26/5/21 a las 12:15, Dan van der Ster escribió: > >>> Hi, > >>> > >>> The mds_cache_memory_limit should be set to something relative to the > >>> RAM size of the MDS -- maybe 50% is a good rule of thumb, because > >>> there are a few cases where the RSS can exceed this limit. Your > >>> experience will help guide what size you need (metadata pool IO > >>> activity will be really high if the MDS cache is too small) > >>> > >>> Otherwise, in recent releases of N/O/P the defaults for those settings > >>> you mentioned are quite good [1]; I would be surprised if they need > >>> further tuning for 99% of users. > >>> Is there any reason you want to start adjusting these params? > >>> > >>> Best Regards, > >>> > >>> Dan > >>> > >>> [1] https://github.com/ceph/ceph/pull/38574 > >>> > >>> On Wed, May 26, 2021 at 11:58 AM Andres Rojas Guerrero <a.rojas(a)csic.es> wrote: > >>>> > >>>> Hi all, I have observed that the MDS Cache Configuration has 18 parameters: > >>>> > >>>> mds_cache_memory_limit > >>>> mds_cache_reservation > >>>> mds_health_cache_threshold > >>>> mds_cache_trim_threshold > >>>> mds_cache_trim_decay_rate > >>>> mds_recall_max_caps > >>>> mds_recall_max_decay_threshold > >>>> mds_recall_max_decay_rate > >>>> mds_recall_global_max_decay_threshold > >>>> mds_recall_warning_threshold > >>>> mds_recall_warning_decay_rate > >>>> mds_session_cap_acquisition_throttle > >>>> mds_session_cap_acquisition_decay_rate > >>>> mds_session_max_caps_throttle_ratio > >>>> mds_cap_acquisition_throttle_retry_request_timeout > >>>> mds_session_cache_liveness_magnitude > >>>> mds_session_cache_liveness_decay_rate > >>>> mds_max_caps_per_client > >>>> > >>>> > >>>> > >>>> I find the Ceph documentation in this section a bit cryptic and I have > >>>> tried to find some resources that talk about how to tune these > >>>> parameters, but without success. > >>>> > >>>> Does anyone have experience in adjusting these parameters according to > >>>> the characteristics of the Ceph cluster itself, the hardware and the use > >>>> of MDS? > >>>> > >>>> Regards! > >>>> -- > >>>> ******************************************************* > >>>> Andrés Rojas Guerrero > >>>> Unidad Sistemas Linux > >>>> Area Arquitectura Tecnológica > >>>> Secretaría General Adjunta de Informática > >>>> Consejo Superior de Investigaciones Científicas (CSIC) > >>>> Pinar 19 > >>>> 28006 - Madrid > >>>> Tel: +34 915680059 -- Ext. 990059 > >>>> email: a.rojas(a)csic.es > >>>> ID comunicate.csic.es: @50852720l:matrix.csic.es > >>>> ******************************************************* > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> -- > >> ******************************************************* > >> Andrés Rojas Guerrero > >> Unidad Sistemas Linux > >> Area Arquitectura Tecnológica > >> Secretaría General Adjunta de Informática > >> Consejo Superior de Investigaciones Científicas (CSIC) > >> Pinar 19 > >> 28006 - Madrid > >> Tel: +34 915680059 -- Ext. 990059 > >> email: a.rojas(a)csic.es > >> ID comunicate.csic.es: @50852720l:matrix.csic.es > >> ******************************************************* > > -- > ******************************************************* > Andrés Rojas Guerrero > Unidad Sistemas Linux > Area Arquitectura Tecnológica > Secretaría General Adjunta de Informática > Consejo Superior de Investigaciones Científicas (CSIC) > Pinar 19 > 28006 - Madrid > Tel: +34 915680059 -- Ext. 990059 > email: a.rojas(a)csic.es > ID comunicate.csic.es: @50852720l:matrix.csic.es > *******************************************************

On Thu, May 27, 2021 at 9:21 AM Andres Rojas Guerrero <a.rojas(a)csic.es> wrote: 25 isn't many clients, but if they are operating in the same directory it will create a lot of contention between the two MDSs, which might explain some of the issues you observe. I recommend that you reduce back to 1 active mds and observe the system stability and performance. -- dan

I don't think # clients alone is a good measure by which to decide to deploy multiple MDSs -- idle clients create very little load, but just a few badly behaving clients can use all the MDS performance. (If you must hear a number, I can share that we have single MDSs with 2-3000 clients connected.) To detect an overloaded MDS, in most cases the users will notice that metadata ops are becoming slow -- so simple things like ls, mv, or creating files will become slow. The MDS cpu usage will also be very high -- (note that the MDS is not multithreaded, so at most it will saturate somewhere between 100-200%). There are also some op latency metrics in the mds perf dump you can observe -- if simple ops are taking more than a few milliseconds then this is another indication that the MDS load is too high. Next, if you do find that you need multiple MDSs, and if you understand the client workload very well, it is best if you can use subtree pinning to statically pin sub directories to a particular MDS rank. See https://ceph.io/community/new-luminous-cephfs-subtree-pinning/ Without pinning, the MDSs will use a heuristic to move subtrees between themselves -- this doesn't always work very well for all workloads, and can sometimes cause more harm than good. Cheers, Dan On Thu, May 27, 2021 at 9:30 AM Andres Rojas Guerrero <a.rojas(a)csic.es> wrote: > > Oh, very interesting!! I have reduced the number of MDS to one. Only one > question more, out of curiosity, from what number can we consider that > there are many clients? > > > > El 27/5/21 a las 9:24, Dan van der Ster escribió: > > On Thu, May 27, 2021 at 9:21 AM Andres Rojas Guerrero <a.rojas(a)csic.es> wrote: > >> > >> > >> > >> El 26/5/21 a las 16:51, Dan van der Ster escribió: > >>> I see you have two active MDSs. Is your cluster more stable if you use > >>> only one single active MDS? > >> > >> Good question!! I read form Ceph Doc: > >> > >> "You should configure multiple active MDS daemons when your metadata > >> performance is bottlenecked on the single MDS that runs by default." > >> > >> "Workloads that typically benefit from a larger number of active MDS > >> daemons are those with many clients, perhaps working on many separate > >> directories." > >> > >> I have more or less 25 concurrent clients, but working in the same > >> directory, Is that number a lot of clients? > >> > >> And I assumed that two are always better than one. > > > > 25 isn't many clients, but if they are operating in the same directory > > it will create a lot of contention between the two MDSs, which might > > explain some of the issues you observe. > > I recommend that you reduce back to 1 active mds and observe the > > system stability and performance. > > > > -- dan > > > > -- > ******************************************************* > Andrés Rojas Guerrero > Unidad Sistemas Linux > Area Arquitectura Tecnológica > Secretaría General Adjunta de Informática > Consejo Superior de Investigaciones Científicas (CSIC) > Pinar 19 > 28006 - Madrid > Tel: +34 915680059 -- Ext. 990059 > email: a.rojas(a)csic.es > ID comunicate.csic.es: @50852720l:matrix.csic.es > *******************************************************

Hi, after one week with only one a MDS all the errors have vanished and the cluster it's running smoothly! Thank you very much for the help!! El 27/5/21 a las 9:50, Andres Rojas Guerrero escribió: -- ******************************************************* Andrés Rojas Guerrero Unidad Sistemas Linux Area Arquitectura Tecnológica Secretaría General Adjunta de Informática Consejo Superior de Investigaciones Científicas (CSIC) Pinar 19 28006 - Madrid Tel: +34 915680059 -- Ext. 990059 email: a.rojas(a)csic.es ID comunicate.csic.es: @50852720l:matrix.csic.es *******************************************************