Hi Janne, I use the default profile(2+1) and set failure-domain=host, is my best practice? Janne Johansson <icepic.dz@gmail.com> 于2020年6月26日周五 下午4:59写道：

Den fre 26 juni 2020 kl 10:32 skrev Zhenshi Zhou <deaderzzs@gmail.com>:

...

Hi all,

I'm going to deploy a cluster with erasure code pool for cold storage. There are 3 servers for me to set up the cluster, 12 OSDs on each server. Does that mean the data is secure while 1/3 OSDs of the cluster is down, or only 2 of the OSDs is down , if I set the ec profile with k=4 and m=2.

By default, crush will want to place each part (of 6 in your case for EC 4+2) on a host of its own, to maximize data safety. Since you can't do that with 3 hosts, you must make sure no more than 2 pieces end up on a single host ever, so you can't just move from failure-domain=host to domain=osd, since that would place all 6 pieces on the same host but different OSDs which would be bad.

You need to make the crush rule pick two different OSDs per host, but not more. One way could be to make a tree where hosts has half of its OSDs in one branch and the other half in another (lets call it subhost in this example), then you get 3*2 subhosts, and you make crush pick placement from subhosts and it will always put two pieces per OSD host, never on the same OSD and it will allow one host to be down for a while.

I would like to add that data is not very secure when you have no redundancy at all left. Machines will crash, they will require maintenance, patches, bios updates and things like that, and having NO redundancy while you have planned or unplanned downtime will be placing the data at huge risk, _any_ surprise in this situation would immediately lead to data loss.

Also, if one box dies, the cluster can't run and can't recover until you have a new host back in, so you are already running at the edge of data safety in your normal case. Even if this will "work", ceph as being a cluster really should have N+1 hosts or more if your data split (replication factor or EC k+m) is equal to N.

-- May the most significant bit of your life be positive.

On 26/06/2020 8:08 pm, Zhenshi Zhou wrote:

Hi Lindsay,

I have only 3 hosts, and is there any method to set a EC pool cluster in a better way

There's failure domain by OSD, which Janne knows far better than I :) -- Lindsay

I'm running EC 8+2 with 'failure domain OSD' on a 3 node cluster with 24 OSDs. Until one has 10s of nodes it pretty much has to be failure domain OSD. The documentation lists certain other important settings which it took time to find. Most important are recommendations to have a small replicated pool in certain places for metadata. -Dave On Fri, Jun 26, 2020, 7:32 AM Zhenshi Zhou <deaderzzs@gmail.com> wrote:

I will give it a try, thanks:)

Lindsay Mathieson <lindsay.mathieson@gmail.com> 于2020年6月26日周五 下午7:07写道：

...

On 26/06/2020 8:08 pm, Zhenshi Zhou wrote:

...

Hi Lindsay,

I have only 3 hosts, and is there any method to set a EC pool cluster in a better way

There's failure domain by OSD, which Janne knows far better than I :)

-- Lindsay

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I have only 3 hosts at present, and I tend to use EC pools because I don't have much budgets. The cluster is used for cold backup and it doesn't need high qos as well. <DHilsbos@performair.com> 于2020年6月26日周五 下午11:40写道：

As others have pointed out; setting the failure domain to OSD is dangerous because then all 6 chunks for an object can end up on the same host. 6 hosts really seems like the minimum to mess with EC pools.

Adding a bucket type between host and osd seems like a good idea here, if you absolutely must use EC pools.

Perhaps something that corresponds to the HBAs / disk controllers?

Thank you,

Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com

-----Original Message----- From: Lindsay Mathieson [mailto:lindsay.mathieson@gmail.com] Sent: Friday, June 26, 2020 4:08 AM To: Zhenshi Zhou Cc: ceph-users Subject: [ceph-users] Re: fault tolerant about erasure code pool

On 26/06/2020 8:08 pm, Zhenshi Zhou wrote:

...

Hi Lindsay,

I have only 3 hosts, and is there any method to set a EC pool cluster in a better way

There's failure domain by OSD, which Janne knows far better than I :)

-- Lindsay _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I think in your case I would just go with failure domain OSD until you have enough servers to change that. If you want to have a bit better uptime, with 3 servers you could consider EC 4+2 and use a "simple" technique to have at most 2 shards per physical host. We had the same issue, needed availability under maintenance but couldn't afford the necessary server count. Physical host and logical host, that is, crush host bucket, are two independent things. It is possible to create additional host buckets and define in ceph.conf which OSD shows up in such buckets with entries like [osd.0] crush location = "host=ceph-21" Host ceph-21 does not yet exist as a physical host in our cluster. You will also need to set min_size=k(=4) until you get additional hosts. What I found most useful is per host to use the hostname of the physical host and an intended hostname of a future host. Once you get the new host, physically move the OSDs belonging to this host bucket to the future host, remove the ceph.conf entries and you are done + there will be no rebalancing either because the crush map itself does not change. This will require very careful bookkeeping though. Also, a drawback is that you need the OSD ID, which implies that OSDs will show up first in the bucket of the physical host and need to be moved manually to the bucket of the fake host. This is additional admin workload that requires careful attention to details as OSDs wrongly configured will move themselves to the wrong bucket on restart. You also need to develop a procedure to find the physical disks associated with host buckets. My recommendation would be to push hard for money for extra servers. The added work load and the increased chance of accidents easily costs in salary what you need in hardware. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Zhenshi Zhou <deaderzzs@gmail.com> Sent: 28 June 2020 06:58:56 To: ceph-users Subject: [ceph-users] Re: fault tolerant about erasure code pool I'm going to try the way Janne said, but what confused me is the expansion of the cluster. Should I change the profile if I add hosts which don't have the same OSDs as these 3 hosts? Zhenshi Zhou <deaderzzs@gmail.com> 于2020年6月28日周日 下午12:53写道：

I have only 3 hosts at present, and I tend to use EC pools because I don't have much budgets. The cluster is used for cold backup and it doesn't need high qos as well.

<DHilsbos@performair.com> 于2020年6月26日周五 下午11:40写道：

...

As others have pointed out; setting the failure domain to OSD is dangerous because then all 6 chunks for an object can end up on the same host. 6 hosts really seems like the minimum to mess with EC pools.

Adding a bucket type between host and osd seems like a good idea here, if you absolutely must use EC pools.

Perhaps something that corresponds to the HBAs / disk controllers?

Thank you,

Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com

-----Original Message----- From: Lindsay Mathieson [mailto:lindsay.mathieson@gmail.com] Sent: Friday, June 26, 2020 4:08 AM To: Zhenshi Zhou Cc: ceph-users Subject: [ceph-users] Re: fault tolerant about erasure code pool

On 26/06/2020 8:08 pm, Zhenshi Zhou wrote:

...

Hi Lindsay,

I have only 3 hosts, and is there any method to set a EC pool cluster in a better way

There's failure domain by OSD, which Janne knows far better than I :)

-- Lindsay _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io