On Fri, Apr 3, 2026 at 4:06 PM Anthony D'Atri via ceph-users < ceph-users@ceph.io> wrote:
* Are you looking to change the profile to get more usable capacity? * Are you looking to change the profile to get more durability / sustain more failures? Not necessarily, we would rather have a responsive rgw for example during failures, so when there is backfill/recovery going on that rgw does not stop responding.
The wider profile may actually make that worse.
Since you mention RGW and only mention HDDs, does that mean that your index pool is on HDDs? There's a good chance that was a factor. HDDs have very limited IOPS and all of the RGW pools that aren't bucket data really should be on SSDs. I speculate that client ops, backfill, and RGW ops were DoSing each other.
Forgot to add, index pools are on NVMEs. Some old OSD nodes have 2 NVMEe inside, newer ones still do not have, so we will have to consider adding NVMEs also.
An expedient way to add those would be something like
https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcTRl12SIsjlOJTId1V6Zk...
A PCI-e AIC that accepts 2x NVMe M.2 drives, assuming that your motherboard can do PCIe bifurcation. You could populate each with like 2x 1.9T Micron 7450 and place the RGW pools on those. For this purpose having more OSDs is more beneficial than having larger OSDs. You could easily see the size of your existing pools to help gauge what you need.
Thx.
It happened a few times. e.g. slow/faulty disk from on which reweight was just set
You aren't using "ceph osd reweight" or "ceph osd reweight-by-utilization" are you? Those are legacies and should be avoided. You want the REWEIGHT value on each OSD to be 1.0000 and to use the manager balancer.
ceph config set mgr mgr/balancer/upmap_max_deviation 1
Actually I am doing `ceph osd crush reweight X ...` when using it. But not for faulty OSDs. On faulty OSDs, I set the OSD to out, replace it, activate it. I am using following procedure: 1. systemctl stop ceph-osd@{id} 2. ceph osd down osd.{id} 3. ceph osd out osd.{id} 4. pocakas rebalance 5. ceph osd destroy {id} --yes-i-really-mean-it --- 6. replace disk... 7. ceph-volume lvm prepare --osd-id {id} --data /dev/sdX 8. ceph-volume lvm list 9. ceph-volume lvm activate {id} {osd fsid} Does that mean you're only using RGW? No RBD, no CephFS? This informs the
process by which you would migrate data between pools. If you don't have capacity concerns, maybe migration isn't worth the trouble.
Only RGW for now. We did test CephFS a little bit, but RGW suffices our needs currently. Rok