On 8/31/19 3:42 AM, Zoltan Arnold Nagy wrote:
Originally our osd tree looked like this:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2073.15186 root default -14 176.63100 rack s01-rack -19 176.63100 host s01 <snip osds> -15 171.29900 rack s02-rack -20 171.29900 host s02 <snip osds>
etc. You get the idea. It was a legacy thing as we've been upgrading this cluster since probably firefly, and started with way less hardware.
The crush rule was set up like this originally:
step take default step chooseleaf firstn 0 type rack
which we have modified to
step take default step chooseleaf firstn 0 type host
taking advantage of chooseleaf's behavior (eg searching in depth instead of just a single level).
Now we thought we could get rid of the rack buckets simply by moving the host buckets to the root using "ceph osd crush move s01 root=default", however this resulted in a bunch of data movement.
Swapping the IDs manually in the crushmap seems to work (verified via crushtool's --compare), eg. changing the ID of s01 to s01-rack's and vice versa, including all shadow trees.
Looking around I saw that there is a swap-bucket command but that does not swap the IDs just bucket contents, so would result in data movement.
Other than manually editing the crushmap is there a better way to achieve this? Is this way the most optimal?
If you are on Luminous+ version - upmap. k