Once you have your additional 5 nodes you can adjust your crushrule to have failure domain = host and ceph will rebalance the data automatically for you. This will involve quite a bit of data movement (at least 50% of your data will need to be migrated) so can take some time. Also the official recommendation is to run min_size = K+2 on an EC pool which in your case would make your cluster unavailable for any maintenance event that takes out a whole node. With 14 TB drives, drive failure recovery will take significant amount of time (especially the backfill of the new drive which can easily take a couple of weeks) Keep in mind that recovery generally increases the pressure on cluster resources significantly so if you design your cluster for the happy day case only, you will suffer greatly when trouble hits (and this is a question of "when" and not "if"). I know plenty people who would not run a cluster with at least one node to spare on their failure domain (i.e. for EC run at least K + M + 1 nodes or for replication N run at least N + 1 nodes). Generally people's time and mental health are worth more (or should be) than the few $ (comparatively) you'd save in HW. On Fri, 26 Mar 2021 at 08:17, by morphin <morphinwithyou@gmail.com> wrote:
Thank you for the answers.
But I don't have problem with setting 8+2. The problem is the expansion.
I need to move the 5 node with data in it and add 5 node later because they're in different city. The goal I'm trying to reach is 8+2 (host crush rule) So I want to cut the data 10 pieces and put them in 5 node. After adding the other 5 node I want to move 5 piece of the data to new nodes and have 8+2 in the end.
Also the data is S3 and I shouldn't break RGW multisite to continue later... If I can not continue then the data will be garbage anyway. If I use internet for sync the data, it will take 1 month or more. Its easier to take 5 node in "B" and bring to "A" and create cluster, sync the data and move back to B :) Because of that I need to create 5 node cluster first with "8+2 EC", sync the data, move the 5 node to B datacenter, add the other 5 nodes later and rebalance all the data to reach 8+2 (host) But I really don't know it will work. I'm using Replication this is the first time dealing with EC setup.
BTW: Every node has 20x14TB Sas and 4x900GB SSD for RGW index. ( SSD's are Replication 3)
Dan van der Ster <dan@vanderster.com>, 25 Mar 2021 Per, 22:03 tarihinde şunu yazdı:
Here's a crush ruleset for 8+2 that will choose 2 osds per host:
rule cephfs_data_82 { id 4 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 5 type host step choose indep 2 type osd step emit }
This is kind of useful because if you set min_size to 8, you could even
lose an entire host and stay online.
Cheers, dan
On Thu, Mar 25, 2021, 7:02 PM by morphin <morphinwithyou@gmail.com>
wrote:
Hello.
I have 5 node Cluster in A datacenter. Also I have same 5 node in B
datacenter.
They're gonna be 10 node 8+2 EC cluster for backup but I need to add the 5 node later. I have to sync my S3 data with multisite on the 5 node cluster in A datacenter and move them to the B and add the other 5 node to the same cluster.
The question is: Can I create 8+2 ec pool on 5 node cluster and add the 5 node later? How can I rebalance the data after that? Or is there any better solution in my case? what should I do? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io