On 05/11/2026 12:45, Eugen Block wrote:
Hi,
this line is confusing me:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21829 1.00000 224 GiB 211 GiB 209 GiB 1.4 GiB 1.6 GiB 12 GiB 94.58 1.87 3 up
According to the column "PGS" this OSD only has 3 PGs? Is that really correct? Because in the ceph df output it says the cephfs_metadata pool has 1024 PGs. Or is this the target set after adding more capacity? If that OSD really only has/had 3 PGs I assume that one PG would have almost 70 GB in size, which during backfill would cause issues because backfill requires temporarily more space to backfill (a temp copy of the PGs).
Thank you for your feedback. The pool now has 1024 PGs but it had 256 at the time, this OSD is now dead/down (see link below).
Can you show 'ceph osd pool ls detail' and 'ceph -s'?
I put the output of these commands there (slightly censored output for "ceph -s"): https://hidoc.in2p3.fr/s/zut5jh10Qc I also included an excerpt of "ceph osd df cephfs_metadata", to show the current number of PGs on the OSDs. Before the "pool full" issue, we had 256 PGs for "cephfs_metadata", we increased to 1024 in an (certainly misguided) attempt to spread the load more evenly. The MD OSD we added are the 1.7 TB ones, the 370 GB ones are on the new nodes. One of the things we do not understand is, as you pointed out, the 200+GiB DATA columns, the value is similar even on the larger OSDs which have more PGs.
I know someone who would comment that OSDs of 200 GB size is a corner case and not really recommended. IIRC there have been similar reports here on the list.
We plan to replace these with 400 GB ones. They're not actually 200/370/400 GB large, they're LVs provisionned with "cephadm" "advanced OSD settings" ("osds_per_device: 4") on larger drives. Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |