Suspiciously low PG count for CephFS with many small files
I have 2 clusters; both have HDDs and SSDs. Reporting only the HDDs which have their own pools: "rep-cluster": hdd-pool 3-replication, 86 OSDs (16 TiB each), 1024 PGs, 78 %RAW USED, 100 M objects "ec-cluster": hdd-pool erasure k=4 m=2, 58 OSDs (16 TiB each), 256 PGs, 60 %RAW USED, 450 M objects Both are Ceph 18.2.1, Bluestore, and have the autoscaler enabled. As you can see, I have many small objects. My PGs-copies-per-OSD seem far off from the recommendation of 100 PGs per OSD (`mon_target_pg_per_osd`): rep-cluster: 35 PGs/OSD (= 1024*3/86) ec-cluster: 26 PGs/OSD (= 256*6/58) So I'm at least 3x-4x off. Why? Should the autoscaler not have increased the PGs here? `ceph osd pool autoscale-status`: rep-cluster: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK data 349.4T 3.0 1343T 0.7802 1.0 1024 on False ec-cluster: data_ec 347.3T 1.5 876.4T 0.5944 1.0 256 on False I believe that because of this I suffer some drawbacks: * On ec-cluster, a PG contains ~2 TiB and ~2 M objects, causing rebalances to happen in coarse, slow steps. Should I take some steps to force the autoscaler to increase PGs, and if yes, which approach would be best here? Thanks for your tips! Niklas
On Jun 20, 2025, at 8:20 PM, Niklas Hambüchen <mail@nh2.me> wrote:
I have 2 clusters; both have HDDs and SSDs. Reporting only the HDDs which have their own pools:
"rep-cluster": hdd-pool 3-replication, 86 OSDs (16 TiB each), 1024 PGs, 78 %RAW USED, 100 M objects "ec-cluster": hdd-pool erasure k=4 m=2, 58 OSDs (16 TiB each), 256 PGs, 60 %RAW USED, 450 M objects
Both are Ceph 18.2.1, Bluestore, and have the autoscaler enabled. As you can see, I have many small objects.
My PGs-copies-per-OSD seem far off from the recommendation of 100 PGs per OSD (`mon_target_pg_per_osd`):
rep-cluster: 35 PGs/OSD (= 1024*3/86) ec-cluster: 26 PGs/OSD (= 256*6/58)
The nomenclature here can be tricky. As I’ve encountered documentation of what we at least used to call the PG ratio I’ve tried to describe this target as the number of *PG replicas* per OSD, because often enough folks don’t multiply by the replication size / EC K+M when doing the math, which I I see you’ve done. When there are multiple device classes and/or pools, especially with varying data protection strategies, it can get a bit complicated. Please share `ceph osd df` for each cluster, trimmed to include only the column header and a handful of representative OSDs for each device class. And the last two lines with the stddev. And `ceph df` and `ceph balancer status` Check the STDDEV figure at the bottom of `ceph osd df`, though if your SSD OSDs are significantly smaller than the HDDs that can confound the reporting. I have an RFE in to report the standard deviation per-device-class in addition to for the cluster as a whole. Also check the VAR column for OSDs within a device class: # ceph osd df | head ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 217 hdd 18.53969 1.00000 19 TiB 9.8 TiB 9.5 TiB 5 KiB 66 GiB 8.7 TiB 52.92 0.89 115 up 219 hdd 18.53969 1.00000 19 TiB 8.5 TiB 8.2 TiB 1 KiB 71 GiB 10 TiB 46.11 0.77 104 up 221 hdd 18.53969 1.00000 19 TiB 11 TiB 10 TiB 2 KiB 76 GiB 7.9 TiB 57.65 0.97 121 up The VAR(iance) is relative to the average number of PG replicas. Ideally — at least for a given device class — this value will not much more or less than 1.00. In this example the cluster was doubled in size and with the grace of upmap-remapped and the balancer is slowly but surely balancing data, which is why the variances are high.
Reporting only the HDDs which have their own pools
When one has OSDs of varying sizes and/or device classes, the balancer and pg autoscaler can be confounded to varying degrees. Since you have multiple device classes, I imagine you have CRUSH rules that constrain pools to one or the other? "rule_id": 6, "rule_name": "ssd_crush", "type": 1, "steps": [ { "op": "take", "item": -33, "item_name": "default~ssd" Are there any CRUSH rules — especially #0 default replicated rule — that do not specify a device class in this way? If so, are there any pools that select such a rule? If so, changing the default or other rules to specify a device class, or changing pools using them to use a device-class-specific rule, can help.
So I'm at least 3x-4x off. Why? Should the autoscaler not have increased the PGs here?
The autoscaler is a fantastic idea from a usability perspective. It is though imperfect and benefits from kaizen. My understanding is that the autoscaler won’t jump a pg_num value until the new value is (by default) a factor of 3 high or low. I suspect that his enforces a manner of hysteresis, so that small fluctuations in pool usage or OSD count don’t result in annoying flapping back and forth.
I believe that because of this I suffer some drawbacks:
* On ec-cluster, a PG contains ~2 TiB and ~2 M objects, causing rebalances to happen in coarse, slow steps.
That’s one big reason why the current PG ratio target of 100 is suboptimal. The guidance used to be 200, it was retconned to 100 a handful of years ago because reasons. At a time when the largest OSDs were on the order of 8TB. Today one can buy a 122TB SSD, and SKUs double that size are on the horizon. For today I suggest ceph config set global target_size_ratio 250 ceph config set global mon_max_pg_per_osd 1000 The first sets the target back to a sane value; I have a PR pending to change this default. This gives the autoscaler more room to do its thing. The second is a guardrail; it does not itself change calculations, but allows headroom for clusters with varying OSD sizes and/or failure domains of varying weights avoid irksome PG activation failures in certain scenarios. Also, when the cluster contains OSDs of significantly varying weights — regardless of device class — the balancer can be facilitated by setting mgr advanced mgr/balancer/upmap_max_deviation 1 I suspect that the above steps will get you closer to where you want to be.
Should I take some steps to force the autoscaler to increase PGs, and if yes, which approach would be best here?
Thanks for your tips! Niklas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, thanks for your quick reply.
The nomenclature here can be tricky
Yes, I think the Ceph docs should get some minor updates to make the difference between PGs and PG replicas (PG * replicationFactor) even more explicit.
Please share `ceph osd df`
Please see below. * The "rep-cluster" is fully balanced. * The "ec-cluster" got 2 machines added (from 4 to 6) 2 days ago and is thus rebalancing (which is where I noticed the impact of the large PGs and decided to look into it in more detail than before). I've included 2 old machines and 1 new machine. rep-cluster # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS [ a machine with 10 HDDs of 16 TB each, + 2 NVMe SSDs ] 0 ssd 0.16370 1.00000 168 GiB 3.4 GiB 59 MiB 1.6 GiB 1.7 GiB 164 GiB 2.01 0.03 2 up 1 ssd 0.16370 1.00000 168 GiB 3.9 GiB 72 MiB 1.6 GiB 2.3 GiB 164 GiB 2.34 0.03 2 up 2 hdd 14.61089 1.00000 15 TiB 12 TiB 12 TiB 6 KiB 21 GiB 2.6 TiB 82.24 1.05 35 up 3 hdd 14.61089 1.00000 15 TiB 12 TiB 12 TiB 0 B 21 GiB 2.6 TiB 82.27 1.05 35 up ... 11 hdd 14.61089 1.00000 15 TiB 12 TiB 12 TiB 0 B 22 GiB 2.3 TiB 84.54 1.08 36 up [ another such machine ] 12 ssd 0.16370 1.00000 168 GiB 2.7 GiB 48 MiB 818 MiB 1.9 GiB 165 GiB 1.61 0.02 1 up 13 ssd 0.16370 1.00000 168 GiB 4.5 GiB 72 MiB 2.5 GiB 2.0 GiB 163 GiB 2.71 0.03 3 up 14 hdd 14.61089 1.00000 15 TiB 12 TiB 12 TiB 0 B 21 GiB 2.9 TiB 79.93 1.02 34 up 15 hdd 14.61089 1.00000 15 TiB 12 TiB 12 TiB 0 B 21 GiB 2.9 TiB 79.86 1.02 34 up ... 23 hdd 14.61089 1.00000 15 TiB 12 TiB 12 TiB 6 KiB 23 GiB 2.3 TiB 84.54 1.08 36 up TOTAL 1.3 PiB 1.0 PiB 1.0 PiB 40 GiB 1.8 TiB 295 TiB 78.11 MIN/MAX VAR: 0.00/1.08 STDDEV: 32.09 ec-cluster # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS [ a machine with 10 HDDs of 16 TB each, + 2 NVMe SSDs ] 0 ssd 0.16370 1.00000 168 GiB 40 GiB 177 MiB 17 GiB 23 GiB 127 GiB 24.05 0.40 107 up 1 ssd 0.16370 1.00000 168 GiB 30 GiB 173 MiB 7.4 GiB 23 GiB 137 GiB 18.16 0.30 97 up 2 hdd 14.61089 1.00000 15 TiB 12 TiB 12 TiB 8 KiB 53 GiB 2.1 TiB 85.39 1.41 35 up 3 hdd 14.61089 1.00000 15 TiB 10 TiB 10 TiB 1 KiB 47 GiB 4.4 TiB 70.13 1.16 30 up ... 11 hdd 14.61089 1.00000 15 TiB 13 TiB 13 TiB 1 KiB 58 GiB 1.8 TiB 87.88 1.45 38 up [ another such machine ] 12 ssd 0.16370 1.00000 168 GiB 28 GiB 165 MiB 9.4 GiB 19 GiB 139 GiB 16.88 0.28 90 up 13 ssd 0.16370 1.00000 168 GiB 36 GiB 174 MiB 11 GiB 25 GiB 131 GiB 21.71 0.36 103 up 14 hdd 14.61089 1.00000 15 TiB 13 TiB 12 TiB 1 KiB 55 GiB 2.1 TiB 85.75 1.42 37 up 15 hdd 14.61089 1.00000 15 TiB 13 TiB 13 TiB 1 KiB 53 GiB 2.0 TiB 86.63 1.43 39 up ... 23 hdd 14.61089 1.00000 15 TiB 12 TiB 12 TiB 1 KiB 52 GiB 2.3 TiB 84.04 1.39 38 up [ another such machine, new (added to the cluster 2 days ago), that is currently being rebalanced to ] 86 ssd 0.16370 1.00000 168 GiB 32 GiB 161 MiB 5.7 GiB 26 GiB 136 GiB 19.06 0.32 119 up 87 ssd 0.16370 1.00000 168 GiB 38 GiB 599 MiB 12 GiB 25 GiB 130 GiB 22.41 0.37 108 up 88 hdd 14.61089 1.00000 15 TiB 2.0 TiB 1.9 TiB 1 KiB 8.9 GiB 13 TiB 13.47 0.22 5 up 89 hdd 14.61089 1.00000 15 TiB 1.9 TiB 1.8 TiB 1 KiB 7.5 GiB 13 TiB 12.94 0.21 5 up ... 97 hdd 14.61089 1.00000 15 TiB 2.9 TiB 2.9 TiB 1 KiB 12 GiB 12 TiB 19.96 0.33 6 up TOTAL 883 TiB 534 TiB 530 TiB 368 GiB 3.0 TiB 349 TiB 60.51 MIN/MAX VAR: 0.17/1.46 STDDEV: 34.51
STDDEV [..] if your SSD OSDs are significantly smaller than the HDDs that can confound the reporting
Yes, indeed the SSD OSDs are 100x smaller than the HDD OSDs. rep-cluster # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.3 PiB 289 TiB 1.0 PiB 1.0 PiB 78.46 ssd 6.2 TiB 6.1 TiB 81 GiB 81 GiB 1.28 TOTAL 1.3 PiB 296 TiB 1.0 PiB 1.0 PiB 78.11 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 1.1 GiB 222 3.4 GiB 0.06 1.9 TiB data 2 1024 350 TiB 108.95M 1.0 PiB 88.20 47 TiB metadata 3 16 11 GiB 680.79k 12 GiB 0.21 1.9 TiB ec-cluster # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 876 TiB 343 TiB 534 TiB 534 TiB 60.87 ssd 6.6 TiB 5.6 TiB 1.1 TiB 1.1 TiB 16.24 TOTAL 883 TiB 348 TiB 535 TiB 535 TiB 60.54 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 430 MiB 109 1.3 GiB 0.03 1.6 TiB data 2 1024 0 B 219.20M 0 B 0 1.6 TiB data_ec 3 256 360 TiB 236.64M 530 TiB 90.02 40 TiB metadata 4 64 123 GiB 32.86k 369 GiB 7.11 1.6 TiB rep-cluster # ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.018905", "last_optimize_started": "Sat Jun 21 13:45:05 2025", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } ec-cluster # ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000194", "last_optimize_started": "Sat Jun 21 13:45:08 2025", "mode": "upmap", "no_optimization_needed": false, "optimize_result": "Some objects (0.013053) are degraded; try again later", "plans": [] } Potentially useful to know: * The rep-cluster is in HEALTH_OK for a long time. * The ec-cluster suffers from `37 OSD(s) experiencing BlueFS spillover` for a long time (I have not solved that yet; I suspect that Ceph would simply like larger DB/WAL devices on my SSDs for the size / object count I have on the HDDs, but if so that is unfixable for me because I use Hetzner SX134 servers). I do not know if that HEALTH_WARN caused by that spillover will permanently inhibit the balancer. That said, I occasionally use https://github.com/TheJJ/ceph-balancer, which takes into account the actual sizes of objects when balancing. Another question: Why do you inquire about the balancer? Does it affect the autoscaler? So far I thought balancing PGs as a concept comes after the choice/computation of how many PGs to use.
I imagine you have CRUSH rules that constrain pools to one or the other?
Yes.
Are there any CRUSH rules — especially #0 default replicated rule — that do not specify a device class in this way?
No, all all CRUSH rules that are in use do specify a device class: rep-cluster # for POOL in $(ceph osd pool ls); do echo -n "$POOL "; ceph osd pool get "$POOL" crush_rule; done .mgr crush_rule: mgr_replicated_ssd_rule_datacenter data crush_rule: rule_data_datacenter metadata crush_rule: rule_metadata_datacenter mgr_replicated_ssd_rule_datacenter, "type": 1, "steps": { "op": "take", "item": -2, "item_name": "default~ssd" } { "op": "chooseleaf_firstn", "num": 0, "type": "datacenter" } { "op": "emit" } rule_data_datacenter", "type": 1, "steps": { "op": "take", "item": -6, "item_name": "default~hdd" } { "op": "chooseleaf_firstn", "num": 0, "type": "datacenter" } { "op": "emit" } rule_metadata_datacenter", "type": 1, "steps": { "op": "take", "item": -2, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "datacenter" } { "op": "emit" } ec-cluster # for POOL in $(ceph osd pool ls); do echo -n "$POOL "; ceph osd pool get "$POOL" crush_rule; done .mgr crush_rule: mgr_replicated_ssd_rule_datacenter data crush_rule: rule_data_ssd_datacenter data_ec crush_rule: rule_data_ec_datacenter metadata crush_rule: rule_metadata_datacenter mgr_replicated_ssd_rule_datacenter, "type": 1, "steps": { "op": "take", "item": -2, "item_name": "default~ssd" } { "op": "chooseleaf_firstn", "num": 0, "type": "datacenter" } { "op": "emit" } rule_data_ssd_datacenter, "type": 1, "steps": { "op": "take", "item": -2, "item_name": "default~ssd" } { "op": "chooseleaf_firstn", "num": 0, "type": "datacenter" } { "op": "emit" } rule_data_ec_datacenter, "type": 3, "steps": { "op": "set_chooseleaf_tries", "num": 5 } { "op": "set_choose_tries", "num": 100 } { "op": "take", "item": -21, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "datacenter" } { "op": "emit" } rule_metadata_datacenter, "type": 1, "steps": { "op": "take", "item": -2, "item_name": "default~ssd" } { "op": "chooseleaf_firstn", "num": 0, "type": "datacenter" } { "op": "emit" }
My understanding is that the autoscaler won’t jump a pg_num value until the new value is (by default) a factor of 3 high or low
Indeed, but isn't it factor 4x too low already? Is there a way I can see the computations and decisions of the autoscaler? I find it confusing that `ceph osd pool autoscale-status` does not have any column related to OSDs, when `mon_target_pg_per_osd` is a key input to the algorithm that controls the ratio between PGs and OSDs. Following https://ceph.io/en/news/blog/2022/autoscaler_tuning/ section "How do I know what the autoscaler is doing?" # grep 'space, bias' /var/log/ceph/ceph-mgr.backupfs-1.log 2025-06-18T14:40:26.208+0000 7faa03a726c0 0 [pg_autoscaler INFO root] Pool 'benacofs_data_ec' root_id -21 using 0.5698246259869221 of space, bias 1.0, pg target 550.8304717873581 quantized to 512 (current 256) This seems to suggest that 512 PGs should be the target, instead of the current 256, which would bring me within factor 3x ratio. Why doesn't `ceph osd pool autoscale-status` contain any info that suggests that some autoscaling should happen then? There are no other `pg_autoscaler` logs that suggests that it's somehow giving up. Also again here we have "blog-driven documentation"; none of this info from the blog seems to be anywhere in Ceph upstream documentation. It also mentions "ceph progress". In that output, it is annoying that there's no time information at all. The listed events could be recent or years old. It's not even clear what the order is (old to new, or the other way around?). I can use `ceph progress json` but then have to read UNIX timestamps. I filed an issue for it now: https://tracker.ceph.com/issues/71781 I also noticed that the dates in `ceph progress json` look bugged: https://tracker.ceph.com/issues/71782
ceph config set global target_size_ratio 250
I don't fully understand this suggestion. Isn't target_size_ratio "relative to other pools that have target_size_ratio set"? https://docs.ceph.com/en/squid/rados/operations/placement-groups/#specifying... If I set it globally (thus for all pools), isn't the ratio between them still the same?
The first sets the target back to a sane value
How can I check what's currently set? Currently nothing seems set at all: # ceph osd pool get data_ec target_size_ratio Error ENOENT: option 'target_size_ratio' is not set on pool 'data_ec' Similar, how can I check the `threshold` value that one can set with `ceph osd pool set threshold`? (I'll send another email "Why is it still so difficult to just dump all config and where it comes from?" to the list for this.) Also, should I be setting `pg_autoscale_bias` to increase the number of PGs that the autoscaler comes up with, by a fixed factor, to adjust for my small objects? This is suggested by https://docs.redhat.com/en/documentation/red_hat_ceph_storage/4/html/storage...
This property is particularly used for metadata pools which might be small in size but have large number of objects, so scaling them faster is important for better performance.
Separate: I read https://docs.ceph.com/en/squid/rados/operations/balancer/#throttling I think these docs need improvement:
There is a separate setting for how uniform the distribution of PGs must be for the module to consider the cluster adequately balanced. At the time of writing (June 2025), this value defaults to `5`
So "there is a setting, and its default value is 5" ... but what's the name of the setting? Is it `upmap_max_deviation` from 4 paragraphs further down? Thanks, Niklas
Yes, I think the Ceph docs should get some minor updates to make the difference between PGs and PG replicas (PG * replicationFactor) even more explicit.
Please open a tracker and list places you find where this isn’t already made clear.
Can we please have 1 command, that can dump all config (including from config files, monitor central configuration database, all currently running daemons), and nicely point out what's set and overridden where and which value is in effect?
Sounds like an opportunity to enter a tracker issue or a PR.
0 ssd 0.16370 1.00000 168 GiB 3.4 GiB 59 MiB 1.6 GiB 1.7 GiB 164 GiB 2.01 0.03 2 up 1 ssd 0.16370 1.00000 168 GiB 3.9 GiB 72 MiB 1.6 GiB 2.3 GiB 164 GiB 2.34 0.03 2 up
STDDEV [..] if your SSD OSDs are significantly smaller than the HDDs that can confound the reporting
Yes, indeed the SSD OSDs are 100x smaller than the HDD OSDs.
What model are they that they’re that small? Are they enterprise-quality? OSDs that small can present difficulties.
Indeed, but isn't it factor 4x too low already?
Potentially useful to know:
* The rep-cluster is in HEALTH_OK for a long time. * The ec-cluster suffers from `37 OSD(s) experiencing BlueFS spillover` for a long time
How large are those DB+WAL slices? Please share BlueFS stats: https://www.ibm.com/docs/en/storage-ceph/7.1.0?topic=bluefs-viewing-ceph-statistics-ceph-osds
(I have not solved that yet; I suspect that Ceph would simply like larger DB/WAL devices on my SSDs for the size / object count I have on the HDDs, but if so that is unfixable for me because I use Hetzner SX134 servers). I do not know if that HEALTH_WARN caused by that spillover will permanently inhibit the balancer.
I wouldn’t think so, but it may be possible to address them.
That said, I occasionally use https://github.com/TheJJ/ceph-balancer, which takes into account the actual sizes of objects when balancing.
Another question:
Why do you inquire about the balancer? Does it affect the autoscaler?
It can contribute to suboptimal PG ratios on OSDs.
Oddly, not listed in https://docs.ceph.com/en/squid/rados/configuration/ceph-conf/#commands But I think https://docs.ceph.com/en/squid/rados/configuration/ceph-conf/#commands should list it so that from there one can easily see it's legacy.
I look forward to your PR.
My understanding is that the autoscaler won’t jump a pg_num value until the new value is (by default) a factor of 3 high or low
Indeed, but isn't it factor 4x too low already?
One would think.
rep-cluster: 35 PGs/OSD (= 1024*3/86)
35 > 100/3
ec-cluster: 26 PGs/OSD (= 256*6/58) [ another such machine, new (added to the cluster 2 days ago), that is currently being rebalanced to ]
I suspect that once backfill completes you’ll see a ratio > 33
ceph config set global target_size_ratio 250
I don't fully understand this suggestion.
Apologies, I meant mon_target_pg_per_osd = 250
Also, should I be setting `pg_autoscale_bias` to increase the number of PGs that the autoscaler comes up with, by a fixed factor, to adjust for my small objects?
In most cases that should mostly be set for metadata / index pools. Mostly.
This is suggested by https://docs.redhat.com/en/documentation/red_hat_ceph_storage/4/html/storage...
This property is particularly used for metadata pools which might be small in size but have large number of objects, so scaling them faster is important for better performance.
That’s a Nautilus page, so be careful using docs that old. But yes, see above.
Separate: I read https://docs.ceph.com/en/squid/rados/operations/balancer/#throttling I think these docs need improvement:
There is a separate setting for how uniform the distribution of PGs must be for the module to consider the cluster adequately balanced. At the time of writing (June 2025), this value defaults to `5`
So "there is a setting, and its default value is 5" ... but what's the name of the setting? Is it `upmap_max_deviation` from 4 paragraphs further down?
Yes.
Can we please have 1 command, that can dump all config
Sounds like an opportunity to enter a tracker issue or a PR.
Done: https://tracker.ceph.com/issues/71783
What model are they that they’re that small? Are they enterprise-quality? OSDs that small can present difficulties. They are ~1 TiB enterprise SSDs: Samsung PM983, 960GB (MZQLB960HAJR-00007) Our cluster currently has no need for SSD storage (only metadata which is only 7 %USED). Most of the SSD's space is used for the DB/WAL for the HDD SSDs.
For a machine with 10x 16TiB HDD + 2x 960GiB SSD: * Each SSD carries DB/WAL for 5 HDDs, so 5 * 60 GiB each. * 170 GB SSD for SSD OSDs. * The rest of the SSD for the OS disk.
How large are those DB+WAL slices? Please share BlueFS stats:
`ceph health detail` example (osd.2 is one such 16 TiB HDD): osd.2 spilled over 5.2 GiB metadata from 'db' device (47 GiB used of 60 GiB) to slow device ceph daemon osd.2 bluefs stats 1 : device size 0xee5afe000 : using 0xbdc800000(47 GiB) 2 : device size 0xe8d7ed00000 : using 0xc81f80fb000(13 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 14 MiB 0 B 0 B 0 B 12 MiB 1 WAL 0 B 915 MiB 0 B 0 B 0 B 502 MiB 35 DB 0 B 9.7 GiB 0 B 0 B 0 B 6.4 GiB 104 SLOW 0 B 37 GiB 5.2 GiB 0 B 0 B 37 GiB 607 TOTAL 0 B 47 GiB 5.2 GiB 0 B 0 B 0 B 747 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 1.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 11 GiB 0 B 0 B 0 B 6.7 GiB SLOW 0 B 39 GiB 8.4 GiB 0 B 0 B 42 GiB TOTAL 0 B 50 GiB 8.4 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB
Oddly, not listed> I look forward to your PR. Done: https://github.com/ceph/ceph/pull/64074
I suspect that once backfill completes you’ll see a ratio > 33 Shouldn't that have been the case already before I added the new machines, given that the PG count didn't change?
Apologies, I meant mon_target_pg_per_osd = 250 That makes sense, thanks!
`upmap_max_deviation` Done: https://github.com/ceph/ceph/pull/64075
Hi Anthony and others, I have now increased the number of PGs on my cluster, but the results are a bit surprising: I increased the settings by 4x and obtained a PG increase by 8x. Wondering if you have insights why that might be. Details: Defaults: mon_target_pg_per_osd 100 mon_max_pg_per_osd 250 `ceph osd pool autoscale-status` before: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 435.8M 3.0 6788G 0.0002 1.0 1 on False data 0 3.0 6788G 0.0000 1.0 1024 on False data_ec 345.1T 1.5 876.4T 0.5906 1.0 256 on False metadata 123.8G 3.0 6788G 0.0548 4.0 64 on False I increased the `*_pg_per_osd` settings by 4x by running: ceph config set global mon_target_pg_per_osd 400 ceph config set global mon_max_pg_per_osd 1000 `ceph osd pool autoscale-status` after: # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 435.8M 3.0 6788G 0.0002 1.0 1 on False data 0 3.0 6788G 0.0000 1.0 1024 on False data_ec 345.1T 1.5 876.4T 0.5907 1.0 256 2048 on False metadata 123.8G 3.0 6788G 0.0548 4.0 64 512 on False `ceph osd pool autoscale-status` after a few minutes more: # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 435.8M 3.0 6788G 0.0002 1.0 1 on False data 0 3.0 6788G 0.0000 1.0 1024 on False data_ec 343.9T 1.5 876.4T 0.5886 1.0 2048 on False metadata 127.5G 3.0 6788G 0.0564 4.0 512 on False So the `PG_NUM` for `data_ec` increased **8x**, from `256` to `2048`. This is surprising to me: * The Ceph autoscaler should increase `PG_NUM` by factors of 2x. It chose to not do that in the "before" state. * When I increased `*_pg_per_osd` settings by 4x, I consequently expected that `PG_NUM` would increase by <= 4x. But it increased 8x. Side remark: While `PG_NUM` is already 2048, the actual number of PGs in `ceph status` is still increasing (currently `1830 pgs`, and some being added every couple minutes). I believe this part is as expected. Thanks! Niklas
Hi Anthony and others,
I have now increased the number of PGs on my cluster, but the results are a bit surprising: I increased the settings by 4x and obtained a PG increase by 8x.
Wondering if you have insights why that might be.
Your prior values were extremely low, which no doubt contributes.
`ceph osd pool autoscale-status` before:
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 435.8M 3.0 6788G 0.0002 1.0 1 on False data 0 3.0 6788G 0.0000 1.0 1024 on False data_ec 345.1T 1.5 876.4T 0.5906 1.0 256 on False metadata 123.8G 3.0 6788G 0.0548 4.0 64 on False
Is this cluster serving RGW? RBD? CephFS? Those pool names are unusual.
I increased the `*_pg_per_osd` settings by 4x by running:
ceph config set global mon_target_pg_per_osd 400
I suggested 250.
ceph config set global mon_max_pg_per_osd 1000
Remember that mon_max_pg_per_osd is a failsafe, it does not affect the autoscaler’s determinations.
`ceph osd pool autoscale-status` after:
# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 435.8M 3.0 6788G 0.0002 1.0 1 on False data 0 3.0 6788G 0.0000 1.0 1024 on False data_ec 345.1T 1.5 876.4T 0.5907 1.0 256 2048 on False metadata 123.8G 3.0 6788G 0.0548 4.0 64 512 on False
`ceph osd pool autoscale-status` after a few minutes more:
# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 435.8M 3.0 6788G 0.0002 1.0 1 on False data 0 3.0 6788G 0.0000 1.0 1024 on False data_ec 343.9T 1.5 876.4T 0.5886 1.0 2048 on False metadata 127.5G 3.0 6788G 0.0564 4.0 512 on False
` ceph osd pool ls detail` will show you a bit more detail - the pg_num vs pgp_num values for each pool and given your names, the application association for each.
This is surprising to me:
* The Ceph autoscaler should increase `PG_NUM` by factors of 2x. It chose to not do that in the "before" state.
It was constrained tightly, and in order to avoid flapping it only takes action when the value of pg_num it sets will increase by at least 3x.
* When I increased `*_pg_per_osd` settings by 4x, I consequently expected that `PG_NUM` would increase by <= 4x. But it increased 8x.
You started off way low, and set the target very high, so this is not entirely surprising.
Side remark: While `PG_NUM` is already 2048, the actual number of PGs in `ceph status` is still increasing (currently `1830 pgs`, and some being added every couple minutes). I believe this part is as expected.
yes, pg_num vs pgp_num. PGs are split or merged incrementally to limit the impact on the cluster.
Thanks! Niklas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is this cluster serving RGW? RBD? CephFS? Those pool names are unusual.
Just CephFS. I named the pools this way, following https://docs.ceph.com/en/reef/cephfs/createfs/#creating-a-file-system
I suggested 250.
Yes, but it is actually great to have even less objects per PG, because then I'd arrive at 250k objects/PG (instead of my 2M from before), which should make the recovery time of an individual PG more reasonable. So I think it's great that I get 8x more PGs. But I'd like to understand _why_ it's happening as it did, because I expected that a 4x increase of `*_pg_per_osd` should only be able to achieve a 4x PG increase. I'm wondering if I had hit an autoscaler bug before (that my PGs for data_ec should really have been at 512 instead of 256), which would be good to report if so. I think your explanation with `NEW PG_NUM` having to be 3x larger for the autoscaler to take action (https://docs.ceph.com/en/latest/rados/operations/placement-groups/#viewing-p...) makes sense:
It was constrained tightly, and in order to avoid flapping it only takes action when the value of pg_num it sets will increase by at least 3x.
E.g. if before I was at 256, and `NEW PG_NUM` was at at 500 (< 768 = 3*256), then it would not take action; if my increase in settings by 4x would result in `NEW PG_NUM` being 500 * 4 = 2000, it makes sense that it then sets it to 2048. So I think that explains it sufficiently, thanks!
Remember that mon_max_pg_per_osd is a failsafe, it does not affect the autoscaler’s determinations.
Yes, that makes sense.
` ceph osd pool ls detail` will show you a bit more detail - the pg_num vs pgp_num values for each pool and given your names, the application association for each.
For reference, here's my output: pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21504 flags hashpspool stripe_width 0 pg_num_min 1 application mgr,mgr_devicehealth read_balance_score 15.79 pool 2 'data' replicated size 3 min_size 2 crush_rule 7 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 21506 lfor 0/1905/12064 flags hashpspool stripe_width 0 pg_num_min 1024 application cephfs read_balance_score 2.67 pool 3 'data_ec' erasure profile ec_profile size 6 min_size 5 crush_rule 8 object_hash rjenkins pg_num 505 pgp_num 377 pg_num_target 2048 pgp_num_target 2048 autoscale_mode on last_change 24867 lfor 0/2200/24867 flags hashpspool,ec_overwrites stripe_width 16384 application cephfs pool 4 'metadata' replicated size 3 min_size 2 crush_rule 9 object_hash rjenkins pg_num 306 pgp_num 178 pg_num_target 512 pgp_num_target 512 autoscale_mode on last_change 24869 lfor 0/17183/24869 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.93
On Jun 30, 2025, at 9:53 AM, Niklas Hambüchen <mail@nh2.me> wrote:
Is this cluster serving RGW? RBD? CephFS? Those pool names are unusual.
Just CephFS. I named the pools this way, following https://docs.ceph.com/en/reef/cephfs/createfs/#creating-a-file-system
That I think shows names like cephfs_metadata and cephfs_data. At such a time that you mix in RBD and RGW you may find less-descriptive names confusing. ymmv.
I suggested 250.
Yes, but it is actually great to have even less objects per PG, because then I'd arrive at 250k objects/PG (instead of my 2M from before), which should make the recovery time of an individual PG more reasonable.
Recovery AIUI proceeds at object granularity, so there’s less of a benefit there than you might think. More PGs also means more peering and memory use. Note also that this target is a maximum, depending on where the calculated pg_num values for pools land the effective PG ratio as reported by the PGS column in `ceph osd df` will usually be lower. Sometimes there can be too much of a good thing. I still suggest a more modest value to start.
So I think it's great that I get 8x more PGs.
But I'd like to understand _why_ it's happening as it did, because I expected that a 4x increase of `*_pg_per_osd` should only be able to achieve a 4x PG increase.
Remember that the autoscaler is juggling the parameters of multiple pools, and with both EC and replicated pools in the mix the calculations become nuanced. So the default max PG per OSD target of 100 might constrain each pool to a difference extent, and as you increase it, each pool’s calculated value may increase independently. It’s not a strict multiplier.
I'm wondering if I had hit an autoscaler bug before (that my PGs for data_ec should really have been at 512 instead of 256), which would be good to report if so.
I don’t think there’s a bug as such, but rather that the default target of 100 is suboptimal in the BlueStore era, which leads to the autoscaler’s hands sometimes being tied with respect to what it really should be doing.
I think your explanation with `NEW PG_NUM` having to be 3x larger for the autoscaler to take action (https://docs.ceph.com/en/latest/rados/operations/placement-groups/#viewing-p...) makes sense:
It was constrained tightly, and in order to avoid flapping it only takes action when the value of pg_num it sets will increase by at least 3x.
E.g. if before I was at 256, and `NEW PG_NUM` was at at 500 (< 768 = 3*256), then it would not take action; if my increase in settings by 4x would result in `NEW PG_NUM` being 500 * 4 = 2000, it makes sense that it then sets it to 2048.
pg_num values should always be a power of 2. Other values are possible but lead to certain suboptimal dynamics including the potential for decreased balancer efficiency.
So I think that explains it sufficiently, thanks!
Glad to help.
Remember that mon_max_pg_per_osd is a failsafe, it does not affect the autoscaler’s determinations.
Yes, that makes sense.
` ceph osd pool ls detail` will show you a bit more detail - the pg_num vs pgp_num values for each pool and given your names, the application association for each.
For reference, here's my output:
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21504 flags hashpspool stripe_width 0 pg_num_min 1 application mgr,mgr_devicehealth read_balance_score 15.79
This will always only have one PG.
pool 2 'data' replicated size 3 min_size 2 crush_rule 7 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 21506 lfor 0/1905/12064 flags hashpspool stripe_width 0 pg_num_min 1024 application cephfs read_balance_score 2.67
pool 3 'data_ec' erasure profile ec_profile size 6 min_size 5 crush_rule 8 object_hash rjenkins pg_num 505 pgp_num 377 pg_num_target 2048 pgp_num_target 2048 autoscale_mode on last_change 24867 lfor 0/2200/24867 flags hashpspool,ec_overwrites stripe_width 16384 application cephfs
This pool is in the process of being scaled — or mon_max_pg_per_osd is not properly increased. Run ceph config dump | grep mon_max_pg_per_osd See if you have this set at `global` scope. It’s possible that there are different values set for `global` and `osd` scopes, which would lead to the global setting not taking effect.
pool 4 'metadata' replicated size 3 min_size 2 crush_rule 9 object_hash rjenkins pg_num 306 pgp_num 178 pg_num_target 512 pgp_num_target 512 autoscale_mode on last_change 24869 lfor 0/17183/24869 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.93
Same here.
Hi Niklas, Most admins who haven't taken the time to tame this somewhat wild animal, like myself, will tell you to disable it and choose for yourself the amount of PGs that you think is right to allocate to each pool to get as many OSDs as possible to participate by distributing the budget of 150-200 PGs/ HDD OSD among all the pools in the best way possible. But you could try setting the autoscaler to "warn" mode and modifying the target_size_ratio and/or setting the bulk flag on the "large" pools (those containing the most data or the most objects) to see what it will do and then decide whether to apply its recommendations if they seem relevant to you. These two blog posts [1] [2] should shed some light on the subject. Regards, Frédéric. [1] https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/ [2] https://ceph.io/en/news/blog/2022/autoscaler_tuning/ ________________________________ De : Niklas Hambüchen <mail@nh2.me> Envoyé : samedi 21 juin 2025 02:22 À : ceph-users@ceph.io Objet : [ceph-users] Suspiciously low PG count for CephFS with many small files I have 2 clusters; both have HDDs and SSDs. Reporting only the HDDs which have their own pools: "rep-cluster": hdd-pool 3-replication, 86 OSDs (16 TiB each), 1024 PGs, 78 %RAW USED, 100 M objects "ec-cluster": hdd-pool erasure k=4 m=2, 58 OSDs (16 TiB each), 256 PGs, 60 %RAW USED, 450 M objects Both are Ceph 18.2.1, Bluestore, and have the autoscaler enabled. As you can see, I have many small objects. My PGs-copies-per-OSD seem far off from the recommendation of 100 PGs per OSD (`mon_target_pg_per_osd`): rep-cluster: 35 PGs/OSD (= 1024*3/86) ec-cluster: 26 PGs/OSD (= 256*6/58) So I'm at least 3x-4x off. Why? Should the autoscaler not have increased the PGs here? `ceph osd pool autoscale-status`: rep-cluster: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK data 349.4T 3.0 1343T 0.7802 1.0 1024 on False ec-cluster: data_ec 347.3T 1.5 876.4T 0.5944 1.0 256 on False I believe that because of this I suffer some drawbacks: * On ec-cluster, a PG contains ~2 TiB and ~2 M objects, causing rebalances to happen in coarse, slow steps. Should I take some steps to force the autoscaler to increase PGs, and if yes, which approach would be best here? Thanks for your tips! Niklas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Anthony D'Atri
-
Frédéric Nass
-
Niklas Hambüchen