Re: An issue during expanding an cluster - one osd getting close to nearfull
Thank you Eugen and Anthony for your comments. Expanding a cluster with one host with the same amount of disks as in other nodes worked perfectly fine. We also (unwantingly) made a test with expanding with one disk(osd) and we have crossed the near-full state. Also thank you for sharing some more details, it is really informative to get those insights into more corner case scenarios. I wish that Ceph documentation would have a few more guidelines, something similar to what Clyso has put together. Regards, Robert On Wed, 1 Jul 2026 at 15:07, Eugen Block via ceph-users <ceph-users@ceph.io> wrote:
Hi,
I would try to avoid adding a host with only few OSDs if other hosts are equipped differently. Although I haven't checked in a while I don't expect the behavior to have changed in recent versions.
So here's my theory: If you add a new host with only one or two OSDs, Ceph can assign many more PGs than average to the first up OSD in order to comply with failure domain (probably host). Same goes for the last OSD during shut down. We had a related case four years ago when we tried to rebuild OSD nodes in a customer cluster. Since the customer cluster was still on default values for mon_max_pg_per_osd and osd_max_pg_per_osd_hard_ratio, the (too many) assigned PGs led to inactive PGs:
maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750
When more OSDs were created on that host (ceph-volume does that sequentially), the issue resolved eventually, except for some inactive PGs, failing the MGR resolved those as well. Since you have around 360 PGs per OSD in average, I assume you increased mon_max_pg_per_osd (default 250), maybe also osd_max_pg_per_osd_hard_ratio (default 3), you most likely dodged the "inactive PG" bullet.
So my recommendation would be to rather add the hosts with all OSDs at the same time but with crush weight 0. That way you have time to inspect the outcome (for example with dedicated DB/WAL devices you can check if they were created according to the specs). Then reweight the OSDs to the correct value, some users do that gradually to prevent an overload.
Note the recent discussion about osd heartbeats when adding multiple hosts at once:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHVPS247M4E4...
Regards, Eugen
Zitat von Robert Lukan via ceph-users <ceph-users@ceph.io>:
Hi everyone,
We are currently in the middle of migrating our infrastructure to a Proxmox/Ceph stack and ran into a concerning edge case while expanding the cluster. I am hoping to get some advice on the safest way to handle this without risking a nearfull OSD.
*The Topology & Situation:*
*Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs.
*Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and osd.41).
*Cluster Average:* ~360 PGs per OSD, ~60% utilization.
*Ceph Version:* Squid 19.2.3
The last time we were expanding the cluster, we had issues when expanding it with many OSDs at the same time(it was a memory problem on a host), so we decided to be more careful and add 2 disks at the time.
During the backfill phase, we noticed osd.40 catching significantly more PGs than osd.41. We paused the migration by setting the nobackfill flag to prevent a potential nearfull situation.
*The Current State (ceph osd df):*
osd.40: 423 PGs, 72.56% used,
osd.41: 307 PGs, 51.87% used,
We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we have paused the backfill.
If we were to unset nobackfill, the incoming data for those 218 PGs would risk pushing osd.40 past the 85% nearfull threshold. However, our balancer refused to generate any optimization plans. It reports: "optimize_result": "Unable to find further optimization... or distribution is already perfect"
We "understand" why this is happening: the balancer is evaluating only when backfilling is done. However, it is completely ignoring the transient acting state, which was heavily overloading osd.40
*Our Temporary Workaround:* To prevent the potential issue, we have temporarily set the reweight of osd.40 to 0 (effectively draining it out) so we can safely lift the nobackfill flag and let the cluster settle without hitting capacity limits.
*My Questions for the Community:*
What is the standard best practice for handling this "small bucket" transient imbalance when adding a small number of OSDs to a new host? Or is this just a bad idea in general ?
Is there a way to force the built-in upmap balancer to respect the transient/acting capacity of an OSD so it doesn't allow a drive to go towards 85% during a migration? I understand we were not there, but being 12 % away was "close" enough.
One of the ideas was to reweight the offending OSD, to remove a few PGs, so it would not be so close to nearfull situation. But this approach is tedious and requires "baby-sitting".
I guess this might be a good candidate for upmap remap script, but we were not "brave" enough to run in it a production environment the first time.
Thanks,
Robert _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, thanks for the confirmation, glad that it was helpful. I still have the upmap-remapped procedure on my to-do list, I just played with it in a lab cluster a while ago but not yet on a real production cluster. Regards Eugen Zitat von Robert Lukan <robert@lukan.org>:
Thank you Eugen and Anthony for your comments. Expanding a cluster with one host with the same amount of disks as in other nodes worked perfectly fine. We also (unwantingly) made a test with expanding with one disk(osd) and we have crossed the near-full state. Also thank you for sharing some more details, it is really informative to get those insights into more corner case scenarios.
I wish that Ceph documentation would have a few more guidelines, something similar to what Clyso has put together.
Regards, Robert
On Wed, 1 Jul 2026 at 15:07, Eugen Block via ceph-users <ceph-users@ceph.io> wrote:
Hi,
I would try to avoid adding a host with only few OSDs if other hosts are equipped differently. Although I haven't checked in a while I don't expect the behavior to have changed in recent versions.
So here's my theory: If you add a new host with only one or two OSDs, Ceph can assign many more PGs than average to the first up OSD in order to comply with failure domain (probably host). Same goes for the last OSD during shut down. We had a related case four years ago when we tried to rebuild OSD nodes in a customer cluster. Since the customer cluster was still on default values for mon_max_pg_per_osd and osd_max_pg_per_osd_hard_ratio, the (too many) assigned PGs led to inactive PGs:
maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750
When more OSDs were created on that host (ceph-volume does that sequentially), the issue resolved eventually, except for some inactive PGs, failing the MGR resolved those as well. Since you have around 360 PGs per OSD in average, I assume you increased mon_max_pg_per_osd (default 250), maybe also osd_max_pg_per_osd_hard_ratio (default 3), you most likely dodged the "inactive PG" bullet.
So my recommendation would be to rather add the hosts with all OSDs at the same time but with crush weight 0. That way you have time to inspect the outcome (for example with dedicated DB/WAL devices you can check if they were created according to the specs). Then reweight the OSDs to the correct value, some users do that gradually to prevent an overload.
Note the recent discussion about osd heartbeats when adding multiple hosts at once:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHVPS247M4E4...
Regards, Eugen
Zitat von Robert Lukan via ceph-users <ceph-users@ceph.io>:
Hi everyone,
We are currently in the middle of migrating our infrastructure to a Proxmox/Ceph stack and ran into a concerning edge case while expanding the cluster. I am hoping to get some advice on the safest way to handle this without risking a nearfull OSD.
*The Topology & Situation:*
*Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs.
*Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and osd.41).
*Cluster Average:* ~360 PGs per OSD, ~60% utilization.
*Ceph Version:* Squid 19.2.3
The last time we were expanding the cluster, we had issues when expanding it with many OSDs at the same time(it was a memory problem on a host), so we decided to be more careful and add 2 disks at the time.
During the backfill phase, we noticed osd.40 catching significantly more PGs than osd.41. We paused the migration by setting the nobackfill flag to prevent a potential nearfull situation.
*The Current State (ceph osd df):*
osd.40: 423 PGs, 72.56% used,
osd.41: 307 PGs, 51.87% used,
We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we have paused the backfill.
If we were to unset nobackfill, the incoming data for those 218 PGs would risk pushing osd.40 past the 85% nearfull threshold. However, our balancer refused to generate any optimization plans. It reports: "optimize_result": "Unable to find further optimization... or distribution is already perfect"
We "understand" why this is happening: the balancer is evaluating only when backfilling is done. However, it is completely ignoring the transient acting state, which was heavily overloading osd.40
*Our Temporary Workaround:* To prevent the potential issue, we have temporarily set the reweight of osd.40 to 0 (effectively draining it out) so we can safely lift the nobackfill flag and let the cluster settle without hitting capacity limits.
*My Questions for the Community:*
What is the standard best practice for handling this "small bucket" transient imbalance when adding a small number of OSDs to a new host? Or is this just a bad idea in general ?
Is there a way to force the built-in upmap balancer to respect the transient/acting capacity of an OSD so it doesn't allow a drive to go towards 85% during a migration? I understand we were not there, but being 12 % away was "close" enough.
One of the ideas was to reweight the offending OSD, to remove a few PGs, so it would not be so close to nearfull situation. But this approach is tedious and requires "baby-sitting".
I guess this might be a good candidate for upmap remap script, but we were not "brave" enough to run in it a production environment the first time.
Thanks,
Robert _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Robert Lukan