Thanks, and thanks for the log file OTR which simply showed:
2021-01-29 23:17:32.567 7f6155cae700 4 mgr[balancer] prepared 0/10 changes
This indeed means that balancer believes those pools are all balanced
according to the config (which you have set to the defaults).
Could you please also share the output of `ceph osd df tree` so we can
see the distribution and OSD weights?
You might need simply to decrease the upmap_max_deviation from the
default of 5. On our clusters we do:
ceph config set mgr mgr/balancer/upmap_max_deviation 1
Cheers, Dan
On Fri, Jan 29, 2021 at 11:25 PM Francois Legrand <fleg(a)lpnhe.in2p3.fr> wrote:
>
> Hi Dan,
>
> Here is the output of ceph balancer status :
>
> /ceph balancer status//
> //{//
> // "last_optimize_duration": "0:00:00.074965", //
> // "plans": [], //
> // "mode": "upmap", //
> // "active": true, //
> // "optimize_result": "Unable to find further optimization, or
> pool(s) pg_num is decreasing, or distribution is already perfect", //
> // "last_optimize_started": "Fri Jan 29 23:13:31 2021"//
> //}/
>
>
> F.
>
> Le 29/01/2021 à 10:57, Dan van der Ster a écrit :
> > Hi Francois,
> >
> > What is the output of `ceph balancer status` ?
> > Also, can you increase the debug_mgr to 4/5 then share the log file of
> > the active mgr?
> >
> > Best,
> >
> > Dan
> >
> > On Fri, Jan 29, 2021 at 10:54 AM Francois Legrand <fleg(a)lpnhe.in2p3.fr>
wrote:
> >> Thanks for your suggestion. I will have a look !
> >>
> >> But I am a bit surprised that the "official" balancer seems so
unefficient !
> >>
> >> F.
> >>
> >> Le 28/01/2021 à 12:00, Jonas Jelten a écrit :
> >>> Hi!
> >>>
> >>> We also suffer heavily from this so I wrote a custom balancer which
yields much better results:
> >>>
https://github.com/TheJJ/ceph-balancer
> >>>
> >>> After you run it, it echoes the PG movements it suggests. You can then
just run those commands the cluster will balance more.
> >>> It's kinda work in progress, so I'm glad about your feedback.
> >>>
> >>> Maybe it helps you :)
> >>>
> >>> -- Jonas
> >>>
> >>> On 27/01/2021 17.15, Francois Legrand wrote:
> >>>> Hi all,
> >>>> I have a cluster with 116 disks (24 new disks of 16TB added in
december and the rest of 8TB) running nautilus 14.2.16.
> >>>> I moved (8 month ago) from crush_compat to upmap balancing.
> >>>> But the cluster seems not well balanced, with a number of pgs on the
8TB disks varying from 26 to 52 ! And an occupation from 35 to 69%.
> >>>> The recent 16 TB disks are more homogeneous with 48 to 61 pgs and
space between 30 and 43%.
> >>>> Last week, I realized that some osd were maybe not using upmap
because I did a ceph osd crush weight-set ls and got (compat) as result.
> >>>> Thus I ran a ceph osd crush weight-set rm-compat which triggered
some rebalancing. Now there is no more recovery for 2 days, but the cluster is still
unbalanced.
> >>>> As far as I understand, upmap is supposed to reach an equal number
of pgs on all the disks (I guess weighted by their capacity).
> >>>> Thus I would expect more or less 30 pgs on the 8TB disks and 60 on
the 16TB and around 50% usage on all. Which is not the case (by far).
> >>>> The problem is that it impact the free available space in the pools
(264Ti while there is more than 578Ti free in the cluster) because free space seems to be
based on space available before the first osd will be full !
> >>>> Is it normal ? Did I missed something ? What could I do ?
> >>>>
> >>>> F.
> >>>> _______________________________________________
> >>>> ceph-users mailing list -- ceph-users(a)ceph.io
> >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io
> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io