On 7/7/25 18:34, mhnx wrote:
Hello!
Few years ago I build a "dc-a:12 + dc-b:12 = 24" node ceph cluster with Nautilus v14.2.16 A year ago the cluster upgraded to Octopus and it was running fine. Recently I added 4+4=8 new nodes with identical hardware and SSD drives. When I created OSD's with Octopus, The cluster usage increased from %50 to %78!!
What does a "ceph osd df tree" gives you?
The weird problem is, the new OSD's become nearfull and hold more size even if they have the same or less amount of PG's.
I had to reweight new OSD's to 0.9 to make them equal size usage.. I increased the PG count 8192 to 16384 and ran balancer, it became worse and I have %84 usage now!
Remember that Ceph is limited by the fullest OSD in the cluster. Do you have old clients? If not, try to get rid of reweight and start using upmap. It is way more efficient in getting a cluster well balanced. I would recommend using this balance script: https://github.com/TheJJ/ceph-balancer Maybe first reset all the reweigths (first do: ceph osd set nobackfill). Then run this script: https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-remap... And after that run the ceph-balancer script. That should help tremendously if the cluster is imbalanced.
I guess OSD or PG code changed between nautilus <-> octopus and it generates this problem.
What version of Octopus are you running?
Can anyone help me with experience or knowledge about this? What should I do?
My solution idea: I'm thinking of destroy and re-create old OSD's as a solution but I need to re-create 144x3.8TB Sas SSD OSD's and it means 4-5 days of maintenance.
Also I have 2 osd per drive because it was recommended at Nautilus times. How about this? Should I keep the config or should I use 1 osd per 3.8TB SAS SSD ? What is the recommendation for Octopus and Quincy?
I would recommend upgrading to newer, supported versions, maybe go to Pacific and then Reef. Modern versions of Ceph do not gain from deploying multiple OSDs per drive. What Ceph services are you running (MDS, RGW, RBD)? Gr. Stefan