2 MDSs behind on trimming on my Ceph Cluster since the upgrade from 18.2.6 (reef) to 19.2.2 (squid)
Dear all, I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2. Here some cluster logs : 8/5/25 09:00 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:50 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:33 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:33 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:30 AM[INF]overall HEALTH_OK 8/5/25 08:22 AM[INF]Cluster is now healthy 8/5/25 08:22 AM[INF]Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:22 AM[WRN]Health check update: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:20 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:03 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:03 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:00 AM[INF]overall HEALTH_OK #ceph fs status cephfs - 50 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818 0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0 POOL TYPE USED AVAIL cephfs_metadata metadata 2630M 2413G cephfs_data data 12.7T 3620G STANDBY MDS cephfs.node3.vdicdn MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) # ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18 }, "mds": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32 } } #ceph orch ps --daemon-type mds NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764 mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18 root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default # ceph mds stat cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ? I saw the folloing issue : <https://tracker.ceph.com/issues/66948> Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ( <https://github.com/ceph/ceph/pull/60838> squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ) maybe related ? Thanks for the help 😊 Best Regards, Edouard Fazenda. <https://www.csti.ch/> Swiss Cloud Provider Edouard Fazenda Technical Support <https://www.csti.ch/> Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40 www.csti.ch
Hi Edouard, ----- Le 8 Mai 25, à 10:15, Edouard FAZENDA <e.fazenda@csti.ch> a écrit :
Dear all,
I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2.
Here some cluster logs :
8/5/25 09:00 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:50 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:40 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326
8/5/25 08:40 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326
8/5/25 08:40 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
8/5/25 08:40 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:33 AM [WRN] Health check update: 2 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:33 AM [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:30 AM [INF] overall HEALTH_OK
8/5/25 08:22 AM [INF] Cluster is now healthy
8/5/25 08:22 AM [INF] Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming)
8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming (525/128)
8/5/25 08:22 AM [WRN] Health check update: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming (525/128)
8/5/25 08:20 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:10 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332
8/5/25 08:10 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332
8/5/25 08:10 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
8/5/25 08:10 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:03 AM [WRN] Health check update: 2 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:03 AM [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:00 AM [INF] overall HEALTH_OK
#ceph fs status
cephfs - 50 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818
0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 2630M 2413G
cephfs_data data 12.7T 3620G
STANDBY MDS
cephfs.node3.vdicdn
MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
# ceph versions
{
"mon": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3
},
"mgr": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2
},
"osd": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18
},
"mds": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3
},
"rgw": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6
},
"overall": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32
}
}
#ceph orch ps --daemon-type mds
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de
mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764
mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18
root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
# ceph mds stat
cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby
Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ?
I saw the folloing issue : [ https://tracker.ceph.com/issues/66948 | Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ] ( [ https://github.com/ceph/ceph/pull/60838 | squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ] ) maybe related ?
There's a fair chance, yes as you said MDS_TRIM alert came after the upgrade, Reef is immune to this bug (as not using major/minor log segment changes) and Squid v19.2.2 does not contain the fix. Wou could try decreasing mds_recall_max_decay_rate to 1 (instead of 1.5) Regards, Frédéric.
Thanks for the help 😊
Best Regards, Edouard Fazenda.
[ https://www.csti.ch/ ] S wiss C loud P rovider
Edouard Fazenda
Technical Support
Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40
[ https://mail.univ-lorraine.fr/www.csti.ch | www.csti.ch ]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
----- Le 9 Mai 25, à 14:10, Frédéric Nass <frederic.nass@univ-lorraine.fr> a écrit :
Hi Edouard,
----- Le 8 Mai 25, à 10:15, Edouard FAZENDA <e.fazenda@csti.ch> a écrit :
Dear all,
I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2.
Here some cluster logs :
8/5/25 09:00 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:50 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:40 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326
8/5/25 08:40 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326
8/5/25 08:40 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
8/5/25 08:40 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:33 AM [WRN] Health check update: 2 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:33 AM [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:30 AM [INF] overall HEALTH_OK
8/5/25 08:22 AM [INF] Cluster is now healthy
8/5/25 08:22 AM [INF] Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming)
8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming (525/128)
8/5/25 08:22 AM [WRN] Health check update: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming (525/128)
8/5/25 08:20 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:10 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332
8/5/25 08:10 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332
8/5/25 08:10 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
8/5/25 08:10 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:03 AM [WRN] Health check update: 2 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:03 AM [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:00 AM [INF] overall HEALTH_OK
#ceph fs status
cephfs - 50 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818
0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 2630M 2413G
cephfs_data data 12.7T 3620G
STANDBY MDS
cephfs.node3.vdicdn
MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
# ceph versions
{
"mon": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3
},
"mgr": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2
},
"osd": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18
},
"mds": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3
},
"rgw": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6
},
"overall": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32
}
}
#ceph orch ps --daemon-type mds
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de
mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764
mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18
root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
# ceph mds stat
cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby
Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ?
I saw the folloing issue : [ https://tracker.ceph.com/issues/66948 | Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ] ( [ https://github.com/ceph/ceph/pull/60838 | squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ] ) maybe related ?
There's a fair chance, yes as you said MDS_TRIM alert came after the upgrade, Reef is immune to this bug (as not using major/minor log segment changes) and Squid v19.2.2 does not contain the fix.
Wou could try decreasing mds_recall_max_decay_rate to 1 (instead of 1.5)
Regards, Frédéric.
Apologies, the email was sent prematurely... What you could try waiting for the fix is to: - increase mds_log_max_segments to 256 (defaults to 128) And eventually: - reduce mds_recall_max_decay_rate to 1 (defaults to 1.5) - reduce mds_recall_max_decay_threshold to 32K (defaults to 128K) - increase mds_recall_global_max_decay_threshold to 256K (defaults to 128K) to allow the MDS to reclaim client caps in a more aggressive manner, though I'm not sure this will prevent MDS_TRIM alert from being triggered. Regards, Frédéric.
Thanks for the help 😊
Best Regards, Edouard Fazenda.
[ https://www.csti.ch/ ] S wiss C loud P rovider
Edouard Fazenda
Technical Support
Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40
[ https://mail.univ-lorraine.fr/www.csti.ch | www.csti.ch ]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Frederic, I have applied the settings you have provided, unfortunately the cluster was back green and yellow afterward with still MDS on trimming again. Thanks for the help Best Regards, <https://www.csti.ch/> Swiss Cloud Provider Edouard Fazenda Technical Support <https://www.csti.ch/> Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40 www.csti.ch From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: vendredi, 9 mai 2025 14:20 To: Edouard FAZENDA <e.fazenda@csti.ch> Cc: ceph-users <ceph-users@ceph.io>; Kelian SAINT-BONNET <k.saintbonnet@csti.ch> Subject: Re: [ceph-users] 2 MDSs behind on trimming on my Ceph Cluster since the upgrade from 18.2.6 (reef) to 19.2.2 (squid) ----- Le 9 Mai 25, à 14:10, Frédéric Nass <frederic.nass@univ-lorraine.fr <mailto:frederic.nass@univ-lorraine.fr> > a écrit : Hi Edouard, ----- Le 8 Mai 25, à 10:15, Edouard FAZENDA <e.fazenda@csti.ch <mailto:e.fazenda@csti.ch> > a écrit : Dear all, I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2. Here some cluster logs : 8/5/25 09:00 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:50 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:33 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:33 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:30 AM[INF]overall HEALTH_OK 8/5/25 08:22 AM[INF]Cluster is now healthy 8/5/25 08:22 AM[INF]Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:22 AM[WRN]Health check update: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:20 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:03 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:03 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:00 AM[INF]overall HEALTH_OK #ceph fs status cephfs - 50 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818 0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0 POOL TYPE USED AVAIL cephfs_metadata metadata 2630M 2413G cephfs_data data 12.7T 3620G STANDBY MDS cephfs.node3.vdicdn MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) # ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18 }, "mds": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32 } } #ceph orch ps --daemon-type mds NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764 mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18 root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default # ceph mds stat cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ? I saw the folloing issue : <https://tracker.ceph.com/issues/66948> Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ( <https://github.com/ceph/ceph/pull/60838> squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ) maybe related ? There's a fair chance, yes as you said MDS_TRIM alert came after the upgrade, Reef is immune to this bug (as not using major/minor log segment changes) and Squid v19.2.2 does not contain the fix. Wou could try decreasing mds_recall_max_decay_rate to 1 (instead of 1.5) Regards, Frédéric. Apologies, the email was sent prematurely... What you could try waiting for the fix is to: - increase mds_log_max_segments to 256 (defaults to 128) And eventually: - reduce mds_recall_max_decay_rate to 1 (defaults to 1.5) - reduce mds_recall_max_decay_threshold to 32K (defaults to 128K) - increase mds_recall_global_max_decay_threshold to 256K (defaults to 128K) to allow the MDS to reclaim client caps in a more aggressive manner, though I'm not sure this will prevent MDS_TRIM alert from being triggered. Regards, Frédéric. Thanks for the help 😊 Best Regards, Edouard Fazenda. <https://www.csti.ch/> Swiss Cloud Provider Edouard Fazenda Technical Support <https://www.csti.ch/> Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40 www.csti.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hi Edouard, Sorry to hear that, although I'm not that surprised. I think you'll have to wait for the fix. Regards, Frédéric. ----- Le 13 Mai 25, à 15:41, Edouard FAZENDA <e.fazenda@csti.ch> a écrit :
Dear Frederic,
I have applied the settings you have provided, unfortunately the cluster was back green and yellow afterward with still MDS on trimming again.
Thanks for the help
Best Regards,
[ https://www.csti.ch/ ] S wiss C loud P rovider
Edouard Fazenda
Technical Support
Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40
[ https://mail.univ-lorraine.fr/www.csti.ch | www.csti.ch ]
From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: vendredi, 9 mai 2025 14:20 To: Edouard FAZENDA <e.fazenda@csti.ch> Cc: ceph-users <ceph-users@ceph.io>; Kelian SAINT-BONNET <k.saintbonnet@csti.ch> Subject: Re: [ceph-users] 2 MDSs behind on trimming on my Ceph Cluster since the upgrade from 18.2.6 (reef) to 19.2.2 (squid)
----- Le 9 Mai 25, à 14:10, Frédéric Nass < [ mailto:frederic.nass@univ-lorraine.fr | frederic.nass@univ-lorraine.fr ] > a écrit :
Hi Edouard,
----- Le 8 Mai 25, à 10:15, Edouard FAZENDA < [ mailto:e.fazenda@csti.ch | e.fazenda@csti.ch ] > a écrit :
Dear all,
I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2.
Here some cluster logs :
8/5/25 09:00 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:50 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:40 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326
8/5/25 08:40 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326
8/5/25 08:40 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
8/5/25 08:40 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:33 AM [WRN] Health check update: 2 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:33 AM [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:30 AM [INF] overall HEALTH_OK
8/5/25 08:22 AM [INF] Cluster is now healthy
8/5/25 08:22 AM [INF] Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming)
8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming (525/128)
8/5/25 08:22 AM [WRN] Health check update: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming (525/128)
8/5/25 08:20 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:10 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332
8/5/25 08:10 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332
8/5/25 08:10 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
8/5/25 08:10 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:03 AM [WRN] Health check update: 2 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:03 AM [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:00 AM [INF] overall HEALTH_OK
#ceph fs status
cephfs - 50 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818
0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 2630M 2413G
cephfs_data data 12.7T 3620G
STANDBY MDS
cephfs.node3.vdicdn
MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
# ceph versions
{
"mon": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3
},
"mgr": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2
},
"osd": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18
},
"mds": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3
},
"rgw": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6
},
"overall": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32
}
}
#ceph orch ps --daemon-type mds
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de
mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764
mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18
root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
# ceph mds stat
cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby
Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ?
I saw the folloing issue : [ https://tracker.ceph.com/issues/66948 | Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ] ( [ https://github.com/ceph/ceph/pull/60838 | squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ] ) maybe related ? There's a fair chance, yes as you said MDS_TRIM alert came after the upgrade, Reef is immune to this bug (as not using major/minor log segment changes) and Squid v19.2.2 does not contain the fix.
Wou could try decreasing mds_recall_max_decay_rate to 1 (instead of 1.5)
Regards,
Frédéric.
Apologies, the email was sent prematurely...
What you could try waiting for the fix is to:
- increase mds_log_max_segments to 256 (defaults to 128)
And eventually:
- reduce mds_recall_max_decay_rate to 1 (defaults to 1.5)
- reduce mds_recall_max_decay_threshold to 32K (defaults to 128K)
- increase mds_recall_global_max_decay_threshold to 256K (defaults to 128K)
to allow the MDS to reclaim client caps in a more aggressive manner, though I'm not sure this will prevent MDS_TRIM alert from being triggered.
Regards,
Frédéric.
Thanks for the help 😊
Best Regards, Edouard Fazenda.
[ https://www.csti.ch/ ] S wiss C loud P rovider
Edouard Fazenda
Technical Support
Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40
[ https://mail.univ-lorraine.fr/www.csti.ch | www.csti.ch ]
_______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ]
Hi Edouard, I had the same problem with on of my setups. I used this article to fix it: https://www.suse.com/support/kb/doc/?id=000019740 A changed these values increasing/decreasing 10%: ceph config set mds mds_cache_trim_threshold xxK (should initially be increased) ceph config set mds mds_cache_trim_decay_rate x.x (should initially be decreased) ceph config set mds mds_cache_memory_limit xxxxxxxxxx (should initially be increased) ceph config set mds mds_recall_max_caps xxxx (should initially be increased) ceph config set mds mds_recall_max_decay_rate x.xx (should initially be decreased) Dominique.
-----Oorspronkelijk bericht----- Van: Frédéric Nass <frederic.nass@univ-lorraine.fr> Verzonden: dinsdag 13 mei 2025 15:46 Aan: Edouard FAZENDA <e.fazenda@csti.ch> CC: ceph-users <ceph-users@ceph.io>; Kelian SAINT-BONNET <k.saintbonnet@csti.ch> Onderwerp: [ceph-users] Re: 2 MDSs behind on trimming on my Ceph Cluster since the upgrade from 18.2.6 (reef) to 19.2.2 (squid)
Hi Edouard,
Sorry to hear that, although I'm not that surprised. I think you'll have to wait for the fix.
Regards, Frédéric.
----- Le 13 Mai 25, à 15:41, Edouard FAZENDA <e.fazenda@csti.ch> a écrit :
Dear Frederic,
I have applied the settings you have provided, unfortunately the cluster was back green and yellow afterward with still MDS on trimming again.
Thanks for the help
Best Regards,
[ https://www.csti.ch/ ] S wiss C loud P rovider
Edouard Fazenda
Technical Support
Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40
[ https://mail.univ-lorraine.fr/www.csti.ch | www.csti.ch ]
From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: vendredi, 9 mai 2025 14:20 To: Edouard FAZENDA <e.fazenda@csti.ch> Cc: ceph-users <ceph-users@ceph.io>; Kelian SAINT-BONNET <k.saintbonnet@csti.ch> Subject: Re: [ceph-users] 2 MDSs behind on trimming on my Ceph Cluster since the upgrade from 18.2.6 (reef) to 19.2.2 (squid)
----- Le 9 Mai 25, à 14:10, Frédéric Nass < [ mailto:frederic.nass@univ-lorraine.fr | frederic.nass@univ-lorraine.fr ] > a écrit :
Hi Edouard,
----- Le 8 Mai 25, à 10:15, Edouard FAZENDA < [ mailto:e.fazenda@csti.ch | e.fazenda@csti.ch ] > a écrit :
Dear all,
I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2.
Here some cluster logs :
8/5/25 09:00 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:50 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:40 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326
8/5/25 08:40 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326
8/5/25 08:40 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
8/5/25 08:40 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:33 AM [WRN] Health check update: 2 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:33 AM [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:30 AM [INF] overall HEALTH_OK
8/5/25 08:22 AM [INF] Cluster is now healthy
8/5/25 08:22 AM [INF] Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming)
8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming (525/128)
8/5/25 08:22 AM [WRN] Health check update: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming (525/128)
8/5/25 08:20 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:10 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332
8/5/25 08:10 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332
8/5/25 08:10 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
8/5/25 08:10 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
8/5/25 08:03 AM [WRN] Health check update: 2 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:03 AM [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)
8/5/25 08:00 AM [INF] overall HEALTH_OK
#ceph fs status
cephfs - 50 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818
0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 2630M 2413G
cephfs_data data 12.7T 3620G
STANDBY MDS
cephfs.node3.vdicdn
MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
# ceph versions
{
"mon": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3
},
"mgr": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2
},
"osd": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18
},
"mds": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3
},
"rgw": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6
},
"overall": {
"ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32
}
}
#ceph orch ps --daemon-type mds
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de
mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764
mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18
root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_me mory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_me mory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_me mory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
mds_cache_memory_limit 4294967296 default
mds_cache_trim_decay_rate 1.000000 default
mds_cache_trim_threshold 262144 default
mds_recall_max_caps 30000 default
mds_recall_max_decay_rate 1.500000 default
# ceph mds stat
cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby
Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ?
I saw the folloing issue : [ https://tracker.ceph.com/issues/66948 | Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ] ( [ https://github.com/ceph/ceph/pull/60838 | squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ] ) maybe related ? There's a fair chance, yes as you said MDS_TRIM alert came after the upgrade, Reef is immune to this bug (as not using major/minor log segment changes) and Squid v19.2.2 does not contain the fix.
Wou could try decreasing mds_recall_max_decay_rate to 1 (instead of 1.5)
Regards,
Frédéric.
Apologies, the email was sent prematurely...
What you could try waiting for the fix is to:
- increase mds_log_max_segments to 256 (defaults to 128)
And eventually:
- reduce mds_recall_max_decay_rate to 1 (defaults to 1.5)
- reduce mds_recall_max_decay_threshold to 32K (defaults to 128K)
- increase mds_recall_global_max_decay_threshold to 256K (defaults to 128K)
to allow the MDS to reclaim client caps in a more aggressive manner, though I'm not sure this will prevent MDS_TRIM alert from being triggered.
Regards,
Frédéric.
Thanks for the help 😊
Best Regards, Edouard Fazenda.
[ https://www.csti.ch/ ] S wiss C loud P rovider
Edouard Fazenda
Technical Support
Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40
[ https://mail.univ-lorraine.fr/www.csti.ch | www.csti.ch ]
_______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ]
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Same situation here after an 18.2.7 to 19.2.2 upgrade - MDS_TRIM warning with constantly increasing count. Does anyone know if it's safe to just ignore the warning until the fix comes out with 19.2.3? I'm worried something is going to break when the count gets large enough.
participants (4)
-
Darrell Enns
-
Dominique Ramaekers
-
Edouard FAZENDA
-
Frédéric Nass