moving EC pool from HDD to SSD without downtime
I need to move a 6+2 EC pool from HDDs to SSDs while storage must remain accessible. All SSDs and HDDs are within the same failure domains. The crush rule in question is rule sr-rbd-data-one { id 5 type erasure min_size 3 max_size 8 step set_chooseleaf_tries 50 step set_choose_tries 1000 step take ServerRoom class hdd step chooseleaf indep 0 type host step emit } and I would be inclined just to change the entry "step take ServerRoom class hdd" to "step take ServerRoom class ssd" and wait for the dust to settle. However, this will almost certainly lead to all PGs being undersized and inaccessible as all objects are in the wrong place. I noticed that this is not an issue with PGs created by replicated rules as they can contain more OSDs than the replication factor while objects are moved. The same does not apply to EC rules. I suspect this is due to the setting "max_size 8", which does not allow for more than 6+2=8 OSDs being a member of a PG. What is the correct way to do what I need to do? Can I just set "max_size 16" and go? Will this work with EC rules? If not, what are my options? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
On Mon, Sep 30, 2019 at 7:42 PM Frank Schilder <frans@dtu.dk> wrote:
and I would be inclined just to change the entry "step take ServerRoom class hdd" to "step take ServerRoom class ssd" and wait for the dust to settle.
yes
However, this will almost certainly lead to all PGs being undersized and inaccessible as all objects are in the wrong place.
no
I noticed that this is not an issue with PGs created by replicated rules as they can contain more OSDs than the replication factor while objects are moved. The same does not apply to EC rules. I suspect this is due to the setting "max_size 8", which does not allow for more than 6+2=8 OSDs being a member of a PG.
no
What is the correct way to do what I need to do? Can I just set "max_size 16" and go? Will this work with EC rules? If not, what are my options?
just change it, it won't be pretty and might have an impact on performance but it will stay available
Thanks!
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Poul! For reference to everyone finding this thread, this procedure works indeed as intended: ceph osd getcrushmap -o crush.map crushtool -d crush.map -o crush.txt # edit crush rule: "step take ServerRoom class hdd" --> "step take ServerRoom class ssd" crushtool -o crush-new.map -c crush.txt ceph osd set norebalance ceph osd set nobackfill ceph osd setcrushmap -i crush-new.map # wait for peering to finish, you will see 100% objects misplaced but all PGs active+... ceph osd unset norebalance ceph osd unset nobackfill Ceph will now happily move objects while storage is fully redundant and r/w accessible. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
participants (2)
-
Frank Schilder
-
Paul Emmerich