CephFS scrub resulting in MDS_CACHE_OVERSIZED
After starting a recursive scrub on a cephfs with alot of files the MDS cache went oversized. Scrub command: ceph... scrub start / recursive,repair,force I kept an eye on the MDS memory usage - since I was warned that it might go crazy.. and after 2-3 hours, I started getting the warning [WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.generic-mds.<host>.asddje(mds.0): MDS cache is too large (63GB/36GB); 1250394 inodes in use by clients, 28888 stray files I then paused the scrub, resulting in scrub status { "status": "PAUSED (22837086 inodes in the stack)", "scrubs": { "27f0e32a-bc8c-443d-b1f0-534474798ddf": { "path": "/", "tag": "27f0e32a-bc8c-443d-b1f0-534474798ddf", "options": "recursive,repair,force" } } } and expected the cache size to go down again - but it didn't. After +12 hours with no change, I opted to abort the scrub - again expecting that the inodes in the stack would be offloaded from memory. The status after abort command: { "status": "PAUSED (0 inodes in the stack)", "scrubs": {} } But still no changes to the cache size. Since the status after the abort command had "PAUSED" in it, I resumed the scrub, resulting in status: { "status": "no active scrubs running", "scrubs": {} } Still no changes to the cache size. The log from the MDS in standard log level was: debug 2025-06-03T06:48:24.122+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub start {path=/,prefix=scrub start,scrubops=[recursive,repair,force]} (starting...) debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub queued for path: / debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: idle+waiting paths [/] debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: active paths [/] debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between head items and fnode.fragstat! printing dentries debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) get_num_head_items() = 38; fnode.fragstat.nfiles=28 fnode.fragstat.nsubdirs=11 debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between child accounted_rstats and my rstats! debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) total of child dentries: n(v0 rc2025-06-03T06:48:11.042059+0000 b1661845634 127=95+32) debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) my rstats: n(v544237 rc2025-06-03T06:48:11.042059+0000 b1661845650 128=96+32) debug 2025-06-03T06:49:38.689+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:51:49.782+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:55:39.654+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T07:00:56.205+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status .. .. From here it's either - asok_command: scrub status {prefix=scrub status} (starting...) - Updating MDS map to version xxxxxx from mon.3 Until I pause the scrub. Extracts from the perf dump from the MDS: "mds": { .. .. .. "inodes": 23121955, "inodes_top": 3684, "inodes_bottom": 1728, "inodes_pin_tail": 23116543, "inodes_pinned": 23116691, "inodes_expired": 39049803601, "inodes_with_caps": 84593, .. .. } .. .. "mds_mem": { "ino": 23114378, "ino+": 38966647328, "ino-": 38943532950, "dir": 513065, "dir+": 130921896, "dir-": 130408831, "dn": 23121954, "dn+": 39349549680, "dn-": 39326427726, "cap": 87280, "cap+": 6964477825, "cap-": 6964390545, "rss": 79730620, "heap": 223508 }, I've have been reluctant to just fail the MDS, to clear the memory, but when I finally came around to do so I got the error "Error EPERM: MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it" At this moment the number strays reported in the MDS_CACHE_OVERSIZED warning, are now up with a factor 10 (approx. 280000) Which made me pause. This seems like a bug.. But To be honest I don't know quite what to expect, if I just execute with "--yes-i-really-mean-it".. Will the MDS eat huge amount of RAM during replay? (I've seen this before during failover - where MDS ate almost 200GB ram, even though the cache was not oversized.) Any advice on how to proceed? BR. Kasper
Hello Kasper, This is known. Next time, please add at least 1 TB of swap before the scrub, and ignore the warning while the MDS is chewing through all the directories and files. The same huge-swap recommendation applies to the recovery operations. On Wed, Jun 18, 2025 at 3:01 PM Kasper Rasmussen < kasper_steengaard@hotmail.com> wrote:
After starting a recursive scrub on a cephfs with alot of files the MDS cache went oversized.
Scrub command: ceph... scrub start / recursive,repair,force
I kept an eye on the MDS memory usage - since I was warned that it might go crazy.. and after 2-3 hours, I started getting the warning
[WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.generic-mds.<host>.asddje(mds.0): MDS cache is too large (63GB/36GB); 1250394 inodes in use by clients, 28888 stray files
I then paused the scrub, resulting in scrub status
{
"status": "PAUSED (22837086 inodes in the stack)",
"scrubs": {
"27f0e32a-bc8c-443d-b1f0-534474798ddf": {
"path": "/",
"tag": "27f0e32a-bc8c-443d-b1f0-534474798ddf",
"options": "recursive,repair,force"
}
}
}
and expected the cache size to go down again - but it didn't. After +12 hours with no change, I opted to abort the scrub - again expecting that the inodes in the stack would be offloaded from memory.
The status after abort command:
{
"status": "PAUSED (0 inodes in the stack)",
"scrubs": {}
}
But still no changes to the cache size.
Since the status after the abort command had "PAUSED" in it, I resumed the scrub, resulting in status:
{
"status": "no active scrubs running",
"scrubs": {}
}
Still no changes to the cache size.
The log from the MDS in standard log level was:
debug 2025-06-03T06:48:24.122+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub start {path=/,prefix=scrub start,scrubops=[recursive,repair,force]} (starting...) debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub queued for path: / debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: idle+waiting paths [/] debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: active paths [/] debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between head items and fnode.fragstat! printing dentries debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) get_num_head_items() = 38; fnode.fragstat.nfiles=28 fnode.fragstat.nsubdirs=11 debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between child accounted_rstats and my rstats! debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) total of child dentries: n(v0 rc2025-06-03T06:48:11.042059+0000 b1661845634 127=95+32) debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) my rstats: n(v544237 rc2025-06-03T06:48:11.042059+0000 b1661845650 128=96+32) debug 2025-06-03T06:49:38.689+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:51:49.782+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:55:39.654+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T07:00:56.205+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status .. .. From here it's either - asok_command: scrub status {prefix=scrub status} (starting...) - Updating MDS map to version xxxxxx from mon.3 Until I pause the scrub.
Extracts from the perf dump from the MDS:
"mds": { .. .. .. "inodes": 23121955, "inodes_top": 3684, "inodes_bottom": 1728, "inodes_pin_tail": 23116543, "inodes_pinned": 23116691, "inodes_expired": 39049803601, "inodes_with_caps": 84593, .. ..
} .. .. "mds_mem": { "ino": 23114378, "ino+": 38966647328, "ino-": 38943532950, "dir": 513065, "dir+": 130921896, "dir-": 130408831, "dn": 23121954, "dn+": 39349549680, "dn-": 39326427726, "cap": 87280, "cap+": 6964477825, "cap-": 6964390545, "rss": 79730620, "heap": 223508 },
I've have been reluctant to just fail the MDS, to clear the memory, but when I finally came around to do so I got the error
"Error EPERM: MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it"
At this moment the number strays reported in the MDS_CACHE_OVERSIZED warning, are now up with a factor 10 (approx. 280000)
Which made me pause. This seems like a bug.. But To be honest I don't know quite what to expect, if I just execute with "--yes-i-really-mean-it".. Will the MDS eat huge amount of RAM during replay? (I've seen this before during failover - where MDS ate almost 200GB ram, even though the cache was not oversized.) Any advice on how to proceed?
BR. Kasper _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
Hi Alexander Thanks man Forgot to mention ceph version is 18.2.7 Is this described anywhere - bug tracker / docs? Also when you write: "The same huge-swap recommendation applies to the recovery operations." Should I - If I fail over the MDS in current state - expect that it will chew away on huge amounts of RAM, requiring me to add 1TB swap? BR. Kasper ________________________________ From: Alexander Patrakov <patrakov@gmail.com> Sent: Wednesday, June 18, 2025 09:11 To: Kasper Rasmussen <kasper_steengaard@hotmail.com> Cc: ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] CephFS scrub resulting in MDS_CACHE_OVERSIZED Hello Kasper, This is known. Next time, please add at least 1 TB of swap before the scrub, and ignore the warning while the MDS is chewing through all the directories and files. The same huge-swap recommendation applies to the recovery operations. On Wed, Jun 18, 2025 at 3:01 PM Kasper Rasmussen <kasper_steengaard@hotmail.com<mailto:kasper_steengaard@hotmail.com>> wrote: After starting a recursive scrub on a cephfs with alot of files the MDS cache went oversized. Scrub command: ceph... scrub start / recursive,repair,force I kept an eye on the MDS memory usage - since I was warned that it might go crazy.. and after 2-3 hours, I started getting the warning [WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.generic-mds.<host>.asddje(mds.0): MDS cache is too large (63GB/36GB); 1250394 inodes in use by clients, 28888 stray files I then paused the scrub, resulting in scrub status { "status": "PAUSED (22837086 inodes in the stack)", "scrubs": { "27f0e32a-bc8c-443d-b1f0-534474798ddf": { "path": "/", "tag": "27f0e32a-bc8c-443d-b1f0-534474798ddf", "options": "recursive,repair,force" } } } and expected the cache size to go down again - but it didn't. After +12 hours with no change, I opted to abort the scrub - again expecting that the inodes in the stack would be offloaded from memory. The status after abort command: { "status": "PAUSED (0 inodes in the stack)", "scrubs": {} } But still no changes to the cache size. Since the status after the abort command had "PAUSED" in it, I resumed the scrub, resulting in status: { "status": "no active scrubs running", "scrubs": {} } Still no changes to the cache size. The log from the MDS in standard log level was: debug 2025-06-03T06:48:24.122+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub start {path=/,prefix=scrub start,scrubops=[recursive,repair,force]} (starting...) debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub queued for path: / debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: idle+waiting paths [/] debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: active paths [/] debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between head items and fnode.fragstat! printing dentries debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) get_num_head_items() = 38; fnode.fragstat.nfiles=28 fnode.fragstat.nsubdirs=11 debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between child accounted_rstats and my rstats! debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) total of child dentries: n(v0 rc2025-06-03T06:48:11.042059+0000 b1661845634 127=95+32) debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) my rstats: n(v544237 rc2025-06-03T06:48:11.042059+0000 b1661845650 128=96+32) debug 2025-06-03T06:49:38.689+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:51:49.782+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:55:39.654+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T07:00:56.205+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status .. .. From here it's either - asok_command: scrub status {prefix=scrub status} (starting...) - Updating MDS map to version xxxxxx from mon.3 Until I pause the scrub. Extracts from the perf dump from the MDS: "mds": { .. .. .. "inodes": 23121955, "inodes_top": 3684, "inodes_bottom": 1728, "inodes_pin_tail": 23116543, "inodes_pinned": 23116691, "inodes_expired": 39049803601, "inodes_with_caps": 84593, .. .. } .. .. "mds_mem": { "ino": 23114378, "ino+": 38966647328, "ino-": 38943532950, "dir": 513065, "dir+": 130921896, "dir-": 130408831, "dn": 23121954, "dn+": 39349549680, "dn-": 39326427726, "cap": 87280, "cap+": 6964477825, "cap-": 6964390545, "rss": 79730620, "heap": 223508 }, I've have been reluctant to just fail the MDS, to clear the memory, but when I finally came around to do so I got the error "Error EPERM: MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it" At this moment the number strays reported in the MDS_CACHE_OVERSIZED warning, are now up with a factor 10 (approx. 280000) Which made me pause. This seems like a bug.. But To be honest I don't know quite what to expect, if I just execute with "--yes-i-really-mean-it".. Will the MDS eat huge amount of RAM during replay? (I've seen this before during failover - where MDS ate almost 200GB ram, even though the cache was not oversized.) Any advice on how to proceed? BR. Kasper _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> -- Alexander Patrakov
Hi Kasper, I don't know if it will chew up the disproportionate amount of memory now, but it will do so next time. So, I recommend connecting a large-enough SSD now and adding swap when you see the excessive memory consumption. Regarding the ticket, yes, it exists: I found https://tracker.ceph.com/issues/71167 and https://tracker.ceph.com/issues/71136 and https://tracker.ceph.com/issues/52715 but I am sure more such tickets exist, and there is no single root cause. On Wed, Jun 18, 2025 at 3:41 PM Kasper Rasmussen < kasper_steengaard@hotmail.com> wrote:
Hi Alexander
Thanks man
Forgot to mention ceph version is 18.2.7
Is this described anywhere - bug tracker / docs?
Also when you write: "The same huge-swap recommendation applies to the recovery operations."
Should I - If I fail over the MDS in current state - expect that it will chew away on huge amounts of RAM, requiring me to add 1TB swap?
BR. Kasper
------------------------------ *From:* Alexander Patrakov <patrakov@gmail.com> *Sent:* Wednesday, June 18, 2025 09:11 *To:* Kasper Rasmussen <kasper_steengaard@hotmail.com> *Cc:* ceph-users <ceph-users@ceph.io> *Subject:* Re: [ceph-users] CephFS scrub resulting in MDS_CACHE_OVERSIZED
Hello Kasper,
This is known. Next time, please add at least 1 TB of swap before the scrub, and ignore the warning while the MDS is chewing through all the directories and files.
The same huge-swap recommendation applies to the recovery operations.
On Wed, Jun 18, 2025 at 3:01 PM Kasper Rasmussen < kasper_steengaard@hotmail.com> wrote:
After starting a recursive scrub on a cephfs with alot of files the MDS cache went oversized.
Scrub command: ceph... scrub start / recursive,repair,force
I kept an eye on the MDS memory usage - since I was warned that it might go crazy.. and after 2-3 hours, I started getting the warning
[WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.generic-mds.<host>.asddje(mds.0): MDS cache is too large (63GB/36GB); 1250394 inodes in use by clients, 28888 stray files
I then paused the scrub, resulting in scrub status
{
"status": "PAUSED (22837086 inodes in the stack)",
"scrubs": {
"27f0e32a-bc8c-443d-b1f0-534474798ddf": {
"path": "/",
"tag": "27f0e32a-bc8c-443d-b1f0-534474798ddf",
"options": "recursive,repair,force"
}
}
}
and expected the cache size to go down again - but it didn't. After +12 hours with no change, I opted to abort the scrub - again expecting that the inodes in the stack would be offloaded from memory.
The status after abort command:
{
"status": "PAUSED (0 inodes in the stack)",
"scrubs": {}
}
But still no changes to the cache size.
Since the status after the abort command had "PAUSED" in it, I resumed the scrub, resulting in status:
{
"status": "no active scrubs running",
"scrubs": {}
}
Still no changes to the cache size.
The log from the MDS in standard log level was:
debug 2025-06-03T06:48:24.122+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub start {path=/,prefix=scrub start,scrubops=[recursive,repair,force]} (starting...) debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub queued for path: / debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: idle+waiting paths [/] debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: active paths [/] debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between head items and fnode.fragstat! printing dentries debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) get_num_head_items() = 38; fnode.fragstat.nfiles=28 fnode.fragstat.nsubdirs=11 debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between child accounted_rstats and my rstats! debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) total of child dentries: n(v0 rc2025-06-03T06:48:11.042059+0000 b1661845634 127=95+32) debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) my rstats: n(v544237 rc2025-06-03T06:48:11.042059+0000 b1661845650 128=96+32) debug 2025-06-03T06:49:38.689+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:51:49.782+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:55:39.654+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T07:00:56.205+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status .. .. From here it's either - asok_command: scrub status {prefix=scrub status} (starting...) - Updating MDS map to version xxxxxx from mon.3 Until I pause the scrub.
Extracts from the perf dump from the MDS:
"mds": { .. .. .. "inodes": 23121955, "inodes_top": 3684, "inodes_bottom": 1728, "inodes_pin_tail": 23116543, "inodes_pinned": 23116691, "inodes_expired": 39049803601, "inodes_with_caps": 84593, .. ..
} .. .. "mds_mem": { "ino": 23114378, "ino+": 38966647328, "ino-": 38943532950, "dir": 513065, "dir+": 130921896, "dir-": 130408831, "dn": 23121954, "dn+": 39349549680, "dn-": 39326427726, "cap": 87280, "cap+": 6964477825, "cap-": 6964390545, "rss": 79730620, "heap": 223508 },
I've have been reluctant to just fail the MDS, to clear the memory, but when I finally came around to do so I got the error
"Error EPERM: MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it"
At this moment the number strays reported in the MDS_CACHE_OVERSIZED warning, are now up with a factor 10 (approx. 280000)
Which made me pause. This seems like a bug.. But To be honest I don't know quite what to expect, if I just execute with "--yes-i-really-mean-it".. Will the MDS eat huge amount of RAM during replay? (I've seen this before during failover - where MDS ate almost 200GB ram, even though the cache was not oversized.) Any advice on how to proceed?
BR. Kasper _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
-- Alexander Patrakov
participants (2)
-
Alexander Patrakov
-
Kasper Rasmussen