Current best practice for migrating from one EC profile to another?
Hi, As we expand our cluster (adding nodes), we'd like to take advantage of better EC profiles enabled by higher server/rack counts. I understand, as Ceph currently exists (15.2.4), there is no way to live-migrate from one EC profile to another on an existing pool, for example, from 4+2 to 17+3 when going from 7 nodes to 21. Is this correct? How are people accomplishing migrations such as this one (or 4+2 to 9+3, for example) with minimal disruption to services that utilize RBDs sitting on top of these pools? I found: https://ceph.io/geen-categorie/ceph-pool-migration/ , which requires effectively shutting down access during migration (which is doable, but not ideal) - and from what I've read - has some potential downsides (specifically referencing the cppool method). The pools are all EC for data, and an rbd pool exists for metadata. Thank you!
Den tis 28 juli 2020 kl 18:50 skrev David Orman <ormandj@corenode.com>:
Hi,
As we expand our cluster (adding nodes), we'd like to take advantage of better EC profiles enabled by higher server/rack counts. I understand, as Ceph currently exists (15.2.4), there is no way to live-migrate from one EC profile to another on an existing pool, for example, from 4+2 to 17+3 when going from 7 nodes to 21. Is this correct?
..depends on how brave you are, but mostly it's true. Mostly people stick to the configured EC size(s) on the pools even if you buy more nodes, the EC profile doesn't have to be close to the number of nodes, it can well be 4+2 even with 21 nodes, it will just choose 6 random nodes out of the 21 for each PG and spread out evenly in your new larger cluster. Also, 17+3 means 20 nodes are involved in every reasonably sized IO, and that may not be optimal, both in terms of CPU usage, and of course that if your previous speed was ok when 6 nodes were needed to reassemble a piece of data, you can now do 3 of those 4+2-IOs in parallel if your network allows. Also, with 17+3 on 21 nodes, then a two node failure means it is continually degraded* until a host comes back up, but with k+m being < 15 would mean that the cluster can repair itself onto those ~15 nodes and still be in full swing even if up to 6 nodes would be lost, however improbable such a failure would be. *) So the +3 normally means three hosts could fail, but having only 19 hosts on a 17+3 would mean it can never reach active+clean for any PG, since there aren't 20 hosts to use. You wouldn't lose data, but all PGs would be degraded/undersized meaning you run the whole cluster at 17+2 at best. -- May the most significant bit of your life be positive.
I'm facing the same issue. My cluster will have an expansion and I wanna modify the ec profile too. What I can think of is to create a new profile and a new pool, and then migrate the data from the old pool to the new one. Finally rename the pools as I can use the new pool just like nothing happened. And of course, the access will be shut down while migrating. I'm looking for a more efficient way, which has no need to stop the clients, to deal with this. Janne Johansson <icepic.dz@gmail.com> 于2020年7月29日周三 上午1:43写道：
Den tis 28 juli 2020 kl 18:50 skrev David Orman <ormandj@corenode.com>:
Hi,
As we expand our cluster (adding nodes), we'd like to take advantage of better EC profiles enabled by higher server/rack counts. I understand, as Ceph currently exists (15.2.4), there is no way to live-migrate from one EC profile to another on an existing pool, for example, from 4+2 to 17+3 when going from 7 nodes to 21. Is this correct?
..depends on how brave you are, but mostly it's true.
Mostly people stick to the configured EC size(s) on the pools even if you buy more nodes, the EC profile doesn't have to be close to the number of nodes, it can well be 4+2 even with 21 nodes, it will just choose 6 random nodes out of the 21 for each PG and spread out evenly in your new larger cluster.
Also, 17+3 means 20 nodes are involved in every reasonably sized IO, and that may not be optimal, both in terms of CPU usage, and of course that if your previous speed was ok when 6 nodes were needed to reassemble a piece of data, you can now do 3 of those 4+2-IOs in parallel if your network allows.
Also, with 17+3 on 21 nodes, then a two node failure means it is continually degraded* until a host comes back up, but with k+m being < 15 would mean that the cluster can repair itself onto those ~15 nodes and still be in full swing even if up to 6 nodes would be lost, however improbable such a failure would be.
*) So the +3 normally means three hosts could fail, but having only 19 hosts on a 17+3 would mean it can never reach active+clean for any PG, since there aren't 20 hosts to use. You wouldn't lose data, but all PGs would be degraded/undersized meaning you run the whole cluster at 17+2 at best.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
David Orman
-
Janne Johansson
-
Zhenshi Zhou