Yes, I think the Ceph docs should get some minor updates to make the difference between PGs and PG replicas (PG * replicationFactor) even more explicit.
Please open a tracker and list places you find where this isn’t already made clear.
Can we please have 1 command, that can dump all config (including from config files, monitor central configuration database, all currently running daemons), and nicely point out what's set and overridden where and which value is in effect?
Sounds like an opportunity to enter a tracker issue or a PR.
0 ssd 0.16370 1.00000 168 GiB 3.4 GiB 59 MiB 1.6 GiB 1.7 GiB 164 GiB 2.01 0.03 2 up 1 ssd 0.16370 1.00000 168 GiB 3.9 GiB 72 MiB 1.6 GiB 2.3 GiB 164 GiB 2.34 0.03 2 up
STDDEV [..] if your SSD OSDs are significantly smaller than the HDDs that can confound the reporting
Yes, indeed the SSD OSDs are 100x smaller than the HDD OSDs.
What model are they that they’re that small? Are they enterprise-quality? OSDs that small can present difficulties.
Indeed, but isn't it factor 4x too low already?
Potentially useful to know:
* The rep-cluster is in HEALTH_OK for a long time. * The ec-cluster suffers from `37 OSD(s) experiencing BlueFS spillover` for a long time
How large are those DB+WAL slices? Please share BlueFS stats: https://www.ibm.com/docs/en/storage-ceph/7.1.0?topic=bluefs-viewing-ceph-statistics-ceph-osds
(I have not solved that yet; I suspect that Ceph would simply like larger DB/WAL devices on my SSDs for the size / object count I have on the HDDs, but if so that is unfixable for me because I use Hetzner SX134 servers). I do not know if that HEALTH_WARN caused by that spillover will permanently inhibit the balancer.
I wouldn’t think so, but it may be possible to address them.
That said, I occasionally use https://github.com/TheJJ/ceph-balancer, which takes into account the actual sizes of objects when balancing.
Another question:
Why do you inquire about the balancer? Does it affect the autoscaler?
It can contribute to suboptimal PG ratios on OSDs.
Oddly, not listed in https://docs.ceph.com/en/squid/rados/configuration/ceph-conf/#commands But I think https://docs.ceph.com/en/squid/rados/configuration/ceph-conf/#commands should list it so that from there one can easily see it's legacy.
I look forward to your PR.
My understanding is that the autoscaler won’t jump a pg_num value until the new value is (by default) a factor of 3 high or low
Indeed, but isn't it factor 4x too low already?
One would think.
rep-cluster: 35 PGs/OSD (= 1024*3/86)
35 > 100/3
ec-cluster: 26 PGs/OSD (= 256*6/58) [ another such machine, new (added to the cluster 2 days ago), that is currently being rebalanced to ]
I suspect that once backfill completes you’ll see a ratio > 33
ceph config set global target_size_ratio 250
I don't fully understand this suggestion.
Apologies, I meant mon_target_pg_per_osd = 250
Also, should I be setting `pg_autoscale_bias` to increase the number of PGs that the autoscaler comes up with, by a fixed factor, to adjust for my small objects?
In most cases that should mostly be set for metadata / index pools. Mostly.
This is suggested by https://docs.redhat.com/en/documentation/red_hat_ceph_storage/4/html/storage...
This property is particularly used for metadata pools which might be small in size but have large number of objects, so scaling them faster is important for better performance.
That’s a Nautilus page, so be careful using docs that old. But yes, see above.
Separate: I read https://docs.ceph.com/en/squid/rados/operations/balancer/#throttling I think these docs need improvement:
There is a separate setting for how uniform the distribution of PGs must be for the module to consider the cluster adequately balanced. At the time of writing (June 2025), this value defaults to `5`
So "there is a setting, and its default value is 5" ... but what's the name of the setting? Is it `upmap_max_deviation` from 4 paragraphs further down?
Yes.