Adding datacenter level to CRUSH tree causes rebalancing
Hi Ceph users, I have a Ceph 16.2.7 cluster that so far has been replicated over the `host` failure domain. All `hosts` have been chosen to be in different `datacenter`s, so that was sufficient. Now I wish to add more hosts, including some in already-used data centers, so I'm planning to use CRUSH's `datacenter` failure domain instead. My problem is that when I add the `datacenter`s into the CRUSH tree, Ceph decides that it should now rebalance the entire cluster. This seems unnecessary, and wrong. Before, `ceph osd tree` (some OSDs omitted for legibility): ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000 After assigning of `datacenter` crush buckets: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -18 146.43625 datacenter FSN-DC16 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -17 146.43625 datacenter FSN-DC18 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000 -16 146.43625 datacenter FSN-DC4 -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000 This shows that the tree is essentially unchanged, it just "gained a level". In `ceph status` I now get: pgs: 1167541260/1595506041 objects misplaced (73.177%) If I remove the `datacenter` level again, then the misplacement disappears. On a minimal testing cluster, this misplacement issue did not appear. Why does Ceph think that these objects are misplaced when I add the datacenter level? Is there a more correct way to do this? Thanks!
Hi Niklas, I am not sure why you are surprised. In a large cluster, you should expect some rebalancing on every crush map or crush map rule change. Ceph doesn't just enforce the failure domain, it also whants to have a "perfect" pseudo-random distribution across the clusters based on the crush map hierarchy and the rules you provided. Rebalancing in itself is not a problem, apart from adding some load to your cluster. As you are running Pacific, you need to tune properly your osd_max_backfills limit so that the rebalancing is fast enough and doesn't stress too much your cluster. Since Quincy there is a new scheduler that doesn't rely on these settings but tries to achieve fairness at the OSD level between the different type of loads and my experience with some large rebalancing on a cluster with 200 OSDs it that the result is pretty good. If your test cluster is small enough, it may be that the current placement is not really sensitive to the change of the failure domain as there is not enough different placement options for the placement algorithm to change something. Take it as an excpetion rather than the normal behaviour. Cheers, Michel Le 15/07/2023 à 20:02, Niklas Hambüchen a écrit :
Hi Ceph users,
I have a Ceph 16.2.7 cluster that so far has been replicated over the `host` failure domain. All `hosts` have been chosen to be in different `datacenter`s, so that was sufficient.
Now I wish to add more hosts, including some in already-used data centers, so I'm planning to use CRUSH's `datacenter` failure domain instead.
My problem is that when I add the `datacenter`s into the CRUSH tree, Ceph decides that it should now rebalance the entire cluster. This seems unnecessary, and wrong.
Before, `ceph osd tree` (some OSDs omitted for legibility):
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000
After assigning of `datacenter` crush buckets:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -18 146.43625 datacenter FSN-DC16 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -17 146.43625 datacenter FSN-DC18 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000 -16 146.43625 datacenter FSN-DC4 -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000
This shows that the tree is essentially unchanged, it just "gained a level".
In `ceph status` I now get:
pgs: 1167541260/1595506041 objects misplaced (73.177%)
If I remove the `datacenter` level again, then the misplacement disappears.
On a minimal testing cluster, this misplacement issue did not appear.
Why does Ceph think that these objects are misplaced when I add the datacenter level? Is there a more correct way to do this?
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Based on my understanding of CRUSH it basically works down the hierarchy and then randomly (but deterministically for a given CRUSH map) picks buckets (based on the specific selection rule) on that level for the object and then it does this recursively until it ends up at the leaf nodes. Given that you introduced a whole hierarchy level just below the top, objects will now be distributed differently since the pseudo-random hash-based selection strategy may now for example put an object that used to be in node-4 under FSN-DC16 instead So basically when you fiddle with the hierarchy you can generally expect lots of data movement everywhere downstream of your change. On Sun, 16 Jul 2023 at 06:03, Niklas Hambüchen <mail@nh2.me> wrote:
Hi Ceph users,
I have a Ceph 16.2.7 cluster that so far has been replicated over the `host` failure domain. All `hosts` have been chosen to be in different `datacenter`s, so that was sufficient.
Now I wish to add more hosts, including some in already-used data centers, so I'm planning to use CRUSH's `datacenter` failure domain instead.
My problem is that when I add the `datacenter`s into the CRUSH tree, Ceph decides that it should now rebalance the entire cluster. This seems unnecessary, and wrong.
Before, `ceph osd tree` (some OSDs omitted for legibility):
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000
After assigning of `datacenter` crush buckets:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -18 146.43625 datacenter FSN-DC16 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -17 146.43625 datacenter FSN-DC18 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000 -16 146.43625 datacenter FSN-DC4 -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000
This shows that the tree is essentially unchanged, it just "gained a level".
In `ceph status` I now get:
pgs: 1167541260/1595506041 objects misplaced (73.177%)
If I remove the `datacenter` level again, then the misplacement disappears.
On a minimal testing cluster, this misplacement issue did not appear.
Why does Ceph think that these objects are misplaced when I add the datacenter level? Is there a more correct way to do this?
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you both Michel and Christian. Looks like I will have to do the rebalancing eventually. From past experience with Ceph 16 the rebalance will likely take at least a month with my 500 M objects. It seems like a good idea to upgrade to Ceph 17 first as Michel suggests. Unless: I was hoping that Ceph might have a way to reduce the rebalancing, given that all constraints about failure domains are already fulfilled. In particular, I was wondering whether I could play with the names of the "datacenter"s, to bring them in the same (alphabetical?) order as the hosts were so far. I suspect that this is what avoided the reshuffling on my my mini test cluster. I think it would be in alignment with Table 1 from the CRUSH paper: https://ceph.com/assets/pdfs/weil-crush-sc06.pdf E.g. perhaps take(root) select(1, row) select(3, cabinet) emit yields the same result as take(root) select(3, row) select(1, cabinet) emit ? Niklas
Hi Niklas, As I said, ceph placement is based on more than fulfilling the failure domain constraint. This is a core feature in ceph design. There is no reason for a rebalancing on a cluster with a few hundreds OSDs to last a month. Just before 17 you have to adjust the max backfills parameter whose default is 1, a very conservative value. Using 2 should already reduce to rebalancing to a few days. But my experience shows that if it an option, upgrading to quincy first may be a better option due to to the autotuning of the number of backfills based on the real load of the cluster. If your cluster is using cephadm, upgrading to quincy is very straightforward and should be complete I. A couple of hours for the cluster size I mentioned. Cheers, Michel Sent from my mobile Le 20 juillet 2023 20:15:54 Niklas Hambüchen <mail@nh2.me> a écrit :
Thank you both Michel and Christian.
Looks like I will have to do the rebalancing eventually. From past experience with Ceph 16 the rebalance will likely take at least a month with my 500 M objects.
It seems like a good idea to upgrade to Ceph 17 first as Michel suggests.
Unless:
I was hoping that Ceph might have a way to reduce the rebalancing, given that all constraints about failure domains are already fulfilled.
In particular, I was wondering whether I could play with the names of the "datacenter"s, to bring them in the same (alphabetical?) order as the hosts were so far. I suspect that this is what avoided the reshuffling on my my mini test cluster. I think it would be in alignment with Table 1 from the CRUSH paper: https://ceph.com/assets/pdfs/weil-crush-sc06.pdf
E.g. perhaps
take(root) select(1, row) select(3, cabinet) emit
yields the same result as
take(root) select(3, row) select(1, cabinet) emit
?
Niklas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I can believe the month timeframe for a cluster with multiple large spinners behind each HBA. I’ve witnessed such personally.
On Jul 20, 2023, at 4:16 PM, Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:
Hi Niklas,
As I said, ceph placement is based on more than fulfilling the failure domain constraint. This is a core feature in ceph design. There is no reason for a rebalancing on a cluster with a few hundreds OSDs to last a month. Just before 17 you have to adjust the max backfills parameter whose default is 1, a very conservative value. Using 2 should already reduce to rebalancing to a few days. But my experience shows that if it an option, upgrading to quincy first may be a better option due to to the autotuning of the number of backfills based on the real load of the cluster.
If your cluster is using cephadm, upgrading to quincy is very straightforward and should be complete I. A couple of hours for the cluster size I mentioned.
Cheers,
Michel Sent from my mobile Le 20 juillet 2023 20:15:54 Niklas Hambüchen <mail@nh2.me> a écrit :
Thank you both Michel and Christian.
Looks like I will have to do the rebalancing eventually. From past experience with Ceph 16 the rebalance will likely take at least a month with my 500 M objects.
It seems like a good idea to upgrade to Ceph 17 first as Michel suggests.
Unless:
I was hoping that Ceph might have a way to reduce the rebalancing, given that all constraints about failure domains are already fulfilled.
In particular, I was wondering whether I could play with the names of the "datacenter"s, to bring them in the same (alphabetical?) order as the hosts were so far. I suspect that this is what avoided the reshuffling on my my mini test cluster. I think it would be in alignment with Table 1 from the CRUSH paper: https://ceph.com/assets/pdfs/weil-crush-sc06.pdf
E.g. perhaps
take(root) select(1, row) select(3, cabinet) emit
yields the same result as
take(root) select(3, row) select(1, cabinet) emit
?
Niklas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I can believe the month timeframe for a cluster with multiple large spinners behind each HBA. I’ve witnessed such personally.
I do have the numbers for this: My original post showed "1167541260/1595506041 objects misplaced (73.177%)". During my last recovery with Ceph 16.2.7, the recovery speed was: * 150 objects/s with osd_max_backfills = 1 * 350 objects/s with osd_max_backfills = 6 Computing: 1167541260 / 350 / 3600 / 24 = 38 days. The hardware is 3 machines * (19 HDDs * 16 TB), with 10 Gbit/s networking. The reason it is slow is that there are many small files == objects, and Ceph does a seek for each object during scrubbing and recovery (see https://tracker.ceph.com/issues/59584).
participants (4)
-
Anthony D'Atri
-
Christian Wuerdig
-
Michel Jouvin
-
Niklas Hambüchen