Understanding the cause of (metadata) pool fill-up
Hello, We have a cluster used mostly for CephFS, to which we recently added new disk servers (cephadm managed, running v19.2.6). It's a 45% capacity increase, from 52 to 76 OSD nodes. All new nodes have HDD+external WAL/DB on NVMe drives and some dedicated NVMe drives for MD (w/ colocated WAL/DB). The OSDs were added using the method described here: https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ We also increased the numbe of PGs in the main data (EC) pool (at the same time), before releasing the balancer following https://docs.clyso.com/docs/kb/rados/pg-splitting-with-upmap-remapped/ It went OK for a few days, then we received an alert for the metadata pool being full. Indeed it became full since "something" used all the space on the SSDs that hold the metadata and the "default" pools. These pools are 3-way replicated. The SSDs used for these pools have a colocated WAL/DB. Graphics from our monitoring showing the reduction in MD space available and actual MD usage: https://hidoc.in2p3.fr/s/VsVaULFB9v The reduction of available space (downward slope on the first graph) in the metadata pool started with the rebalance of the data, until zero bytes were left free in the pool. For instance one of the (old) MD OSD: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21829 1.00000 224 GiB 211 GiB 209 GiB 1.4 GiB 1.6 GiB 12 GiB 94.58 1.87 3 up What consummes 200+ GiB on each MD OSD when the total MD pool usage is less than 200 GiB ? Is it the colocated WAL/DB ? There is no "ceph.dir.layout.pool" attribute targeting the "cephfs_ssd" or "cephfs_default" pool (no client has the "p" permission). More details: At some point the smaller MD OSD were really full (> 99% occupation), they all crashed and refuse to start (they are down). Of course the FS is down too. ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 17 ssd 0.21829 1.00000 224 GiB 223 GiB 216 GiB 1.1 GiB 6.4 GiB 424 MiB 99.81 1.97 0 down We managed to add extra OSDs for MD to try to restart, but we are progressing very slowly. "ceph df" (even in detailled mode) does not show what is using that SSD space (this is the state after several days of recovery effort, the FS is not online yet, recovery is still ongoing): [ceph: root@$ADMNODE2 /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 18 PiB 8.7 PiB 9.4 PiB 9.4 PiB 51.93 ssd 682 TiB 555 TiB 127 TiB 127 TiB 18.64 TOTAL 19 PiB 9.3 PiB 9.5 PiB 9.5 PiB 50.75 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 2 8 1.0 GiB 265 3.1 GiB 0 37 TiB cephfs_EC_data 3 16384 7.1 PiB 3.15G 8.9 PiB 63.49 4.1 PiB cephfs_metadata 4 1024 193 GiB 91.25M 578 GiB 0.51 37 TiB cephfs_default 5 128 258 MiB 1.34G 773 MiB 0 37 TiB cephfs_ssd 6 128 0 B 0 0 B 0 37 TiB .rgw.root 7 32 1.9 KiB 4 76 KiB 0 1.7 PiB default.rgw.log 8 32 6.8 KiB 210 760 KiB 0 1.7 PiB default.rgw.control 9 32 0 B 8 0 B 0 1.7 PiB default.rgw.meta 10 32 39 KiB 53 1013 KiB 0 1.7 PiB default.rgw.buckets.index 11 32 3.2 GiB 209 9.6 GiB 0 1.7 PiB default.rgw.buckets.data 12 32 5.9 TiB 12.42M 18 TiB 0.34 1.7 PiB default.rgw.buckets.non-ec 13 32 16 MiB 3.20k 48 MiB 0 1.7 PiB cephfs_nvme 14 4096 2.7 TiB 1.84M 3.3 TiB 1.23 214 TiB [ceph: root@$ADMNODE2 /]# => The issue is: what still consumes 120+ TiB on SSDs ? The sum of "USED" for all pools besides "cephfs_EC_data" is way lower than 127 TiB ("EC_data" and "nvme" are EC pools). Before the extension we had about 24 TiB of SSDs available for the "cephfs_metadata", "cephfs_default" and "cephfs_ssd" pools (the latter is no longer used and was replaced with the "cephfs_nvme" pool). We do not use snapshots at all on this cluster. There are no purge queue to empty on the various MDS. The "full-ratio" settings are at the default values. There are no CephFS (non admin) clients with the ability to change the destination pool or create snapshots, and I think "ceph df" would show if the SSD pools were abused by a "clever user". (the "cephfs_default" usage is due to our recovery effort, it was 0 bytes before) Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
Hi, this line is confusing me:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21829 1.00000 224 GiB 211 GiB 209 GiB 1.4 GiB 1.6 GiB 12 GiB 94.58 1.87 3 up
According to the column "PGS" this OSD only has 3 PGs? Is that really correct? Because in the ceph df output it says the cephfs_metadata pool has 1024 PGs. Or is this the target set after adding more capacity? If that OSD really only has/had 3 PGs I assume that one PG would have almost 70 GB in size, which during backfill would cause issues because backfill requires temporarily more space to backfill (a temp copy of the PGs). Can you show 'ceph osd pool ls detail' and 'ceph -s'? I know someone who would comment that OSDs of 200 GB size is a corner case and not really recommended. IIRC there have been similar reports here on the list. Regards, Eugen Zitat von Loïc Tortay <ceph-users@ceph.io>:
Hello, We have a cluster used mostly for CephFS, to which we recently added new disk servers (cephadm managed, running v19.2.6). It's a 45% capacity increase, from 52 to 76 OSD nodes. All new nodes have HDD+external WAL/DB on NVMe drives and some dedicated NVMe drives for MD (w/ colocated WAL/DB).
The OSDs were added using the method described here: https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ We also increased the numbe of PGs in the main data (EC) pool (at the same time), before releasing the balancer following https://docs.clyso.com/docs/kb/rados/pg-splitting-with-upmap-remapped/
It went OK for a few days, then we received an alert for the metadata pool being full. Indeed it became full since "something" used all the space on the SSDs that hold the metadata and the "default" pools. These pools are 3-way replicated. The SSDs used for these pools have a colocated WAL/DB.
Graphics from our monitoring showing the reduction in MD space available and actual MD usage: https://hidoc.in2p3.fr/s/VsVaULFB9v
The reduction of available space (downward slope on the first graph) in the metadata pool started with the rebalance of the data, until zero bytes were left free in the pool.
For instance one of the (old) MD OSD: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21829 1.00000 224 GiB 211 GiB 209 GiB 1.4 GiB 1.6 GiB 12 GiB 94.58 1.87 3 up
What consummes 200+ GiB on each MD OSD when the total MD pool usage is less than 200 GiB ? Is it the colocated WAL/DB ? There is no "ceph.dir.layout.pool" attribute targeting the "cephfs_ssd" or "cephfs_default" pool (no client has the "p" permission).
More details:
At some point the smaller MD OSD were really full (> 99% occupation), they all crashed and refuse to start (they are down). Of course the FS is down too. ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 17 ssd 0.21829 1.00000 224 GiB 223 GiB 216 GiB 1.1 GiB 6.4 GiB 424 MiB 99.81 1.97 0 down
We managed to add extra OSDs for MD to try to restart, but we are progressing very slowly.
"ceph df" (even in detailled mode) does not show what is using that SSD space (this is the state after several days of recovery effort, the FS is not online yet, recovery is still ongoing): [ceph: root@$ADMNODE2 /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 18 PiB 8.7 PiB 9.4 PiB 9.4 PiB 51.93 ssd 682 TiB 555 TiB 127 TiB 127 TiB 18.64 TOTAL 19 PiB 9.3 PiB 9.5 PiB 9.5 PiB 50.75
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 2 8 1.0 GiB 265 3.1 GiB 0 37 TiB cephfs_EC_data 3 16384 7.1 PiB 3.15G 8.9 PiB 63.49 4.1 PiB cephfs_metadata 4 1024 193 GiB 91.25M 578 GiB 0.51 37 TiB cephfs_default 5 128 258 MiB 1.34G 773 MiB 0 37 TiB cephfs_ssd 6 128 0 B 0 0 B 0 37 TiB .rgw.root 7 32 1.9 KiB 4 76 KiB 0 1.7 PiB default.rgw.log 8 32 6.8 KiB 210 760 KiB 0 1.7 PiB default.rgw.control 9 32 0 B 8 0 B 0 1.7 PiB default.rgw.meta 10 32 39 KiB 53 1013 KiB 0 1.7 PiB default.rgw.buckets.index 11 32 3.2 GiB 209 9.6 GiB 0 1.7 PiB default.rgw.buckets.data 12 32 5.9 TiB 12.42M 18 TiB 0.34 1.7 PiB default.rgw.buckets.non-ec 13 32 16 MiB 3.20k 48 MiB 0 1.7 PiB cephfs_nvme 14 4096 2.7 TiB 1.84M 3.3 TiB 1.23 214 TiB [ceph: root@$ADMNODE2 /]#
=> The issue is: what still consumes 120+ TiB on SSDs ? The sum of "USED" for all pools besides "cephfs_EC_data" is way lower than 127 TiB ("EC_data" and "nvme" are EC pools).
Before the extension we had about 24 TiB of SSDs available for the "cephfs_metadata", "cephfs_default" and "cephfs_ssd" pools (the latter is no longer used and was replaced with the "cephfs_nvme" pool).
We do not use snapshots at all on this cluster. There are no purge queue to empty on the various MDS. The "full-ratio" settings are at the default values.
There are no CephFS (non admin) clients with the ability to change the destination pool or create snapshots, and I think "ceph df" would show if the SSD pools were abused by a "clever user". (the "cephfs_default" usage is due to our recovery effort, it was 0 bytes before)
Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre | _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 05/11/2026 12:45, Eugen Block wrote:
Hi,
this line is confusing me:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21829 1.00000 224 GiB 211 GiB 209 GiB 1.4 GiB 1.6 GiB 12 GiB 94.58 1.87 3 up
According to the column "PGS" this OSD only has 3 PGs? Is that really correct? Because in the ceph df output it says the cephfs_metadata pool has 1024 PGs. Or is this the target set after adding more capacity? If that OSD really only has/had 3 PGs I assume that one PG would have almost 70 GB in size, which during backfill would cause issues because backfill requires temporarily more space to backfill (a temp copy of the PGs).
Thank you for your feedback. The pool now has 1024 PGs but it had 256 at the time, this OSD is now dead/down (see link below).
Can you show 'ceph osd pool ls detail' and 'ceph -s'?
I put the output of these commands there (slightly censored output for "ceph -s"): https://hidoc.in2p3.fr/s/zut5jh10Qc I also included an excerpt of "ceph osd df cephfs_metadata", to show the current number of PGs on the OSDs. Before the "pool full" issue, we had 256 PGs for "cephfs_metadata", we increased to 1024 in an (certainly misguided) attempt to spread the load more evenly. The MD OSD we added are the 1.7 TB ones, the 370 GB ones are on the new nodes. One of the things we do not understand is, as you pointed out, the 200+GiB DATA columns, the value is similar even on the larger OSDs which have more PGs.
I know someone who would comment that OSDs of 200 GB size is a corner case and not really recommended. IIRC there have been similar reports here on the list.
We plan to replace these with 400 GB ones. They're not actually 200/370/400 GB large, they're LVs provisionned with "cephadm" "advanced OSD settings" ("osds_per_device: 4") on larger drives. Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
I had no access to the hidoc link you provided first, so I just tried it again (successfully). This line is similar to the one for OSD.16 (only 2 PGs):
1517 ssd 0.36388 1.00000 373 GiB 237 GiB 231 GiB 514 MiB 6.0 GiB 135 GiB 63.73 2.73 2 up
How many OSDs are part of that pool? This all looks a bit odd to me, but if the PGs were actually so large it would explain the full OSDs. Zitat von Loïc Tortay <tortay@cc.in2p3.fr>:
On 05/11/2026 12:45, Eugen Block wrote:
Hi,
this line is confusing me:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21829 1.00000 224 GiB 211 GiB 209 GiB 1.4 GiB 1.6 GiB 12 GiB 94.58 1.87 3 up
According to the column "PGS" this OSD only has 3 PGs? Is that really correct? Because in the ceph df output it says the cephfs_metadata pool has 1024 PGs. Or is this the target set after adding more capacity? If that OSD really only has/had 3 PGs I assume that one PG would have almost 70 GB in size, which during backfill would cause issues because backfill requires temporarily more space to backfill (a temp copy of the PGs).
Thank you for your feedback.
The pool now has 1024 PGs but it had 256 at the time, this OSD is now dead/down (see link below).
Can you show 'ceph osd pool ls detail' and 'ceph -s'?
I put the output of these commands there (slightly censored output for "ceph -s"): https://hidoc.in2p3.fr/s/zut5jh10Qc
I also included an excerpt of "ceph osd df cephfs_metadata", to show the current number of PGs on the OSDs. Before the "pool full" issue, we had 256 PGs for "cephfs_metadata", we increased to 1024 in an (certainly misguided) attempt to spread the load more evenly. The MD OSD we added are the 1.7 TB ones, the 370 GB ones are on the new nodes.
One of the things we do not understand is, as you pointed out, the 200+GiB DATA columns, the value is similar even on the larger OSDs which have more PGs.
I know someone who would comment that OSDs of 200 GB size is a corner case and not really recommended. IIRC there have been similar reports here on the list.
We plan to replace these with 400 GB ones. They're not actually 200/370/400 GB large, they're LVs provisionned with "cephadm" "advanced OSD settings" ("osds_per_device: 4") on larger drives.
Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
On 05/11/2026 16:29, Eugen Block wrote:
I had no access to the hidoc link you provided first, so I just tried it again (successfully).
This line is similar to the one for OSD.16 (only 2 PGs):
1517 ssd 0.36388 1.00000 373 GiB 237 GiB 231 GiB 514 MiB 6.0 GiB 135 GiB 63.73 2.73 2 up
How many OSDs are part of that pool? This all looks a bit odd to me, but if the PGs were actually so large it would explain the full OSDs.
There are 344 OSDs "up" in that pool (240 "down"): # ceph osd df cephfs_metadata | awk '$NF == "up"' | wc -l 344 # All these "up" OSDs have a "DATA" column value of 230 (324 OSDs) or 231 GiB (20 OSDs). That is consistent with the total line "DATA" value for "ceph osd df cephfs_metadata" on "hidoc": 344*230 GiB ~ 77 TiB. However, on one some "down" OSD, we used "ceph-objectstore-tool" to list the PGs on the OSD and then export these to files. The aggregate size of the files for each OSD if way below 230 GiB, the "cephfs_metadata" pool PGs export files are about 900 MiB each the "cephfs_default" files are about 13 GiB each. Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
participants (2)
-
Eugen Block
-
Loïc Tortay