Thanks Anthony changing the scheduler will require to restart all OSDs , right using "ceph orch restart osd " Is this done in a staggering manner or I need to "stagger " them ? Steven On Fri, 11 Jul 2025 at 12:14, Anthony D'Atri <aad@dreamsnake.net> wrote:
What you describe sounds like expected behavior. It’s a feature!
Since … Nautilus I think, you or the autoscaler sets pg_num and the cluster gradually steps up pgp_num until it matches.
Increasing pg_num means splitting PGs, which in turn perturbs the inputs to the CRUSH hash function, so data moves: backfill.
Moving data on HDDs isn’t fast, especially with EC. These are all random, fragmented writes, so model 70 MB/s to a given drive.
As expected, the backfilling started ...and it never ended ...even now after more than 1 week I still have about 29 pgs backfilling and 13 backfilling_wait
Back pre-Nautilus this would have been a thundering herd of backfill. You don’t know how good we have it now ;)
What worries me is that the number of backfilling PGs varies very little over time e.g 28 and 12 ALTHOUGH there is constant "recovery" traffic between 250 and 350MiB
The number of PGs backfillING at any given time is a function of multiple things, including the value of osd_max_backfills. EC means each write ties up 6 drives, so there’s a bit more gridlock compared to replicated pools.
The "recovery" seems to be doing something ( but number of objects remain the same )
The number of objects, or the number of *misplaced/remapped* objects?
Is it showing *keys* per second? RGW stores a lot of omap data in RocksDB.
Since the recovery should run over the cluster network and the amount of data in the pool is not huge, I am not sure why it takes so many days - it seems stuck actually
Have you reverted to the wpq scheduler?
osd_op_queue = wpq osd_mclock_override_recovery_settings
You can also increase the value of osd_max_backfills
The only strange thing I noticed is a discrepancy between the number of PG and PGP that the pool currently has ...and what autoscale-status says
It’s in the process of doing what you asked.
Any help / suggestions would be very appreciated
What I have tried so for : increase recovery speed ( by changing mclock profile to "high_recovery_ops" and overriding various parameters) (recovery_max_active, recovery_max_active_hdd ... etc)
If the default mclock scheduler is enabled, that has issues for some deployments. There are code improvements in the works, but for now I suggest reverting to wpq.
redeploying some of the OSDs that were "UP_PRIMARY but part of the backfill_wait PGs
Redeploying OSDs isn’t often called for, and can chum the waters. It also adds a lot of backfill/recovery to what you already have going on.
If you want a gentle goose when things seem stuck, you can try
ceph osd down XXX
for the lead OSD of a given PG, one at a time
or ceph pg repeer xx.yyyy