Hi All, Looking for some help/explanation around erasure code pools, etc. I set up a 3-node Ceph (Quincy) cluster with each box holding 7 OSDs (HDDs) and each box running Monitor, Manager, and iSCSI Gateway. For the record the cluster runs beautifully, without resource issues, etc. I created an Erasure Code Profile, etc: ~~~ ceph osd erasure-code-profile set my_ec_profile plugin=jerasure k=4 m=2 crush-failure-domain=osd ceph osd crush rule create-erasure my_ec_rule my_ec_profile ceph osd crush rule create-replicated my_replicated_rule default host ~~~ My Crush Map is: ~~~ # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root # buckets host ceph_1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.34769 item osd.1 weight 5.45799 item osd.2 weight 5.45799 item osd.3 weight 5.45799 item osd.4 weight 5.45799 item osd.5 weight 5.45799 item osd.6 weight 5.45799 } host ceph_2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.7 weight 5.34769 item osd.8 weight 5.45799 item osd.9 weight 5.45799 item osd.10 weight 5.45799 item osd.11 weight 5.45799 item osd.12 weight 5.45799 item osd.13 weight 5.45799 } host ceph_3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.14 weight 5.34769 item osd.15 weight 5.45799 item osd.16 weight 5.45799 item osd.17 weight 5.45799 item osd.18 weight 5.45799 item osd.19 weight 5.45799 item osd.20 weight 5.45799 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 114.28693 alg straw2 hash 0 # rjenkins1 item ceph_1 weight 38.09564 item ceph_2 weight 38.09564 item ceph_3 weight 38.09564 } # rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_replicated_rule { id 1 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_ec_rule { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit } # end crush map ~~~ Finally I create a pool: ~~~ ceph osd pool create my_pool 32 32 erasure my_ec_profile my_ec_rule ceph osd pool application enable my_meta_pool rbd rbd pool init my_meta_pool rbd pool init my_pool rbd create --size 16T my_pool/my_disk_1 --data-pool my_pool --image-feature journaling ~~~ So all this is to have some VMs (oVirt VMs, for the record) with automatic fall-over in the case of a Ceph Node loss - ie I was trying to "replicate" a 3-Disk RAID 5 array across the Ceph Nodes, so that I could loose a Node and still have a working set of VMs. However, I took one of the Ceph Nodes down (gracefully) for some maintenance the other day and I lost *all* the VMs (ie oVirt complained that there was no active pool). As soon as I brought the down node back up everything was good again. So my question is: What did I do wrong with my config? Sound I, for example, change the EC Profile to `k=2, m=1`, but how is that practically different from `k=4, m=2` - yes, the later spreads the pool over more disks, but it should still only put 2 disks on each node, shouldn't it? Thanks in advance Cheers Dulux-Oz
First problem here is you are using crush-failure-domain=osd when you should use crush-failure-domain=host. With three hosts, you should use k=2, m=1; this is not recommended in production environment. On Mon, Dec 4, 2023, 23:26 duluxoz <duluxoz@gmail.com> wrote:
Hi All,
Looking for some help/explanation around erasure code pools, etc.
I set up a 3-node Ceph (Quincy) cluster with each box holding 7 OSDs (HDDs) and each box running Monitor, Manager, and iSCSI Gateway. For the record the cluster runs beautifully, without resource issues, etc.
I created an Erasure Code Profile, etc:
~~~ ceph osd erasure-code-profile set my_ec_profile plugin=jerasure k=4 m=2 crush-failure-domain=osd ceph osd crush rule create-erasure my_ec_rule my_ec_profile ceph osd crush rule create-replicated my_replicated_rule default host ~~~
My Crush Map is:
~~~ # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54
# devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd
# types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root
# buckets host ceph_1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.34769 item osd.1 weight 5.45799 item osd.2 weight 5.45799 item osd.3 weight 5.45799 item osd.4 weight 5.45799 item osd.5 weight 5.45799 item osd.6 weight 5.45799 } host ceph_2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.7 weight 5.34769 item osd.8 weight 5.45799 item osd.9 weight 5.45799 item osd.10 weight 5.45799 item osd.11 weight 5.45799 item osd.12 weight 5.45799 item osd.13 weight 5.45799 } host ceph_3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.14 weight 5.34769 item osd.15 weight 5.45799 item osd.16 weight 5.45799 item osd.17 weight 5.45799 item osd.18 weight 5.45799 item osd.19 weight 5.45799 item osd.20 weight 5.45799 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 114.28693 alg straw2 hash 0 # rjenkins1 item ceph_1 weight 38.09564 item ceph_2 weight 38.09564 item ceph_3 weight 38.09564 }
# rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_replicated_rule { id 1 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_ec_rule { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit }
# end crush map ~~~
Finally I create a pool:
~~~ ceph osd pool create my_pool 32 32 erasure my_ec_profile my_ec_rule ceph osd pool application enable my_meta_pool rbd rbd pool init my_meta_pool rbd pool init my_pool rbd create --size 16T my_pool/my_disk_1 --data-pool my_pool --image-feature journaling ~~~
So all this is to have some VMs (oVirt VMs, for the record) with automatic fall-over in the case of a Ceph Node loss - ie I was trying to "replicate" a 3-Disk RAID 5 array across the Ceph Nodes, so that I could loose a Node and still have a working set of VMs.
However, I took one of the Ceph Nodes down (gracefully) for some maintenance the other day and I lost *all* the VMs (ie oVirt complained that there was no active pool). As soon as I brought the down node back up everything was good again.
So my question is: What did I do wrong with my config?
Sound I, for example, change the EC Profile to `k=2, m=1`, but how is that practically different from `k=4, m=2` - yes, the later spreads the pool over more disks, but it should still only put 2 disks on each node, shouldn't it?
Thanks in advance
Cheers
Dulux-Oz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks David, I knew I had something wrong :-) Just for my own edification: Why is k=2, m=1 not recommended for production? Considered to "fragile", or something else? Cheers Dulux-Oz On 05/12/2023 19:53, David Rivera wrote:
First problem here is you are using crush-failure-domain=osd when you should use crush-failure-domain=host. With three hosts, you should use k=2, m=1; this is not recommended in production environment.
On Mon, Dec 4, 2023, 23:26 duluxoz <duluxoz@gmail.com> wrote:
Hi All,
Looking for some help/explanation around erasure code pools, etc.
I set up a 3-node Ceph (Quincy) cluster with each box holding 7 OSDs (HDDs) and each box running Monitor, Manager, and iSCSI Gateway. For the record the cluster runs beautifully, without resource issues, etc.
I created an Erasure Code Profile, etc:
~~~ ceph osd erasure-code-profile set my_ec_profile plugin=jerasure k=4 m=2 crush-failure-domain=osd ceph osd crush rule create-erasure my_ec_rule my_ec_profile ceph osd crush rule create-replicated my_replicated_rule default host ~~~
My Crush Map is:
~~~ # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54
# devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd
# types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root
# buckets host ceph_1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.34769 item osd.1 weight 5.45799 item osd.2 weight 5.45799 item osd.3 weight 5.45799 item osd.4 weight 5.45799 item osd.5 weight 5.45799 item osd.6 weight 5.45799 } host ceph_2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.7 weight 5.34769 item osd.8 weight 5.45799 item osd.9 weight 5.45799 item osd.10 weight 5.45799 item osd.11 weight 5.45799 item osd.12 weight 5.45799 item osd.13 weight 5.45799 } host ceph_3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.14 weight 5.34769 item osd.15 weight 5.45799 item osd.16 weight 5.45799 item osd.17 weight 5.45799 item osd.18 weight 5.45799 item osd.19 weight 5.45799 item osd.20 weight 5.45799 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 114.28693 alg straw2 hash 0 # rjenkins1 item ceph_1 weight 38.09564 item ceph_2 weight 38.09564 item ceph_3 weight 38.09564 }
# rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_replicated_rule { id 1 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_ec_rule { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit }
# end crush map ~~~
Finally I create a pool:
~~~ ceph osd pool create my_pool 32 32 erasure my_ec_profile my_ec_rule ceph osd pool application enable my_meta_pool rbd rbd pool init my_meta_pool rbd pool init my_pool rbd create --size 16T my_pool/my_disk_1 --data-pool my_pool --image-feature journaling ~~~
So all this is to have some VMs (oVirt VMs, for the record) with automatic fall-over in the case of a Ceph Node loss - ie I was trying to "replicate" a 3-Disk RAID 5 array across the Ceph Nodes, so that I could loose a Node and still have a working set of VMs.
However, I took one of the Ceph Nodes down (gracefully) for some maintenance the other day and I lost *all* the VMs (ie oVirt complained that there was no active pool). As soon as I brought the down node back up everything was good again.
So my question is: What did I do wrong with my config?
Sound I, for example, change the EC Profile to `k=2, m=1`, but how is that practically different from `k=4, m=2` - yes, the later spreads the pool over more disks, but it should still only put 2 disks on each node, shouldn't it?
Thanks in advance
Cheers
Dulux-Oz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 12/5/23 10:01, duluxoz wrote:
Thanks David, I knew I had something wrong :-)
Just for my own edification: Why is k=2, m=1 not recommended for production? Considered to "fragile", or something else?
It is the same as a replicated pool with size=2. Only one host can go down. After that you risk to lose data. Erasure coding is possible with a cluster size of 10 nodes or more. With smaller clusters you have to go with replicated pools. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi Robert, Le 05/12/2023 à 10:05, Robert Sander a écrit :
On 12/5/23 10:01, duluxoz wrote:
Thanks David, I knew I had something wrong :-)
Just for my own edification: Why is k=2, m=1 not recommended for production? Considered to "fragile", or something else?
It is the same as a replicated pool with size=2. Only one host can go down. After that you risk to lose data.
Erasure coding is possible with a cluster size of 10 nodes or more. With smaller clusters you have to go with replicated pools.
Could you explain why 10 nodes are required for EC ? On my side, I'm working on building my first (small) Ceph cluster using E.C. and I was thinking about 5 nodes and k=4 m=2. With a failure domain on host and several osd by nodes, in my mind this setup may run degraded with 3 nodes using 2 distincts osd by node and the ultimate possibility to loose an additional node without loosing data. Of course with sufficient free storage available. Am I totally wrong in my first ceph approach ? Patrick
On Tue, Dec 5, 2023 at 5:16 AM Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> wrote:
On my side, I'm working on building my first (small) Ceph cluster using E.C. and I was thinking about 5 nodes and k=4 m=2. With a failure domain on host and several osd by nodes, in my mind this setup may run degraded with 3 nodes using 2 distincts osd by node and the ultimate possibility to loose an additional node without loosing data. Of course with sufficient free storage available.
When the failure domain is the host, then k/m are referring to hosts, so to have k=4,m=2 you'd need an absolute minimum of 6 hosts, but that would be pushing it. To start to actually use all that space they'd need to be equally balanced, and that configuration wouldn't allow for any kind of recovery if you lost a host, until you add a new one. You typically want more hosts than k+m. With only 3 hosts size=3, min=2 is probably the best way to go, as inefficient as it is. Keep in mind that distributed filesystems in general, and especially ceph, tend to work best when you have a very large number of hosts. If you want host-level redundancy and EC you need a fair number of them. I'm just running a home cluster so I get by with less than 10, but I would not seriously consider my current config for any kind of serious use, and I'm not using k=4,m=2. Rich
Hi, To return to my comparison with SANs, on a SAN you have spare disks to repair a failed disk. On Ceph, you therefore need at least one more host (k+m+1). If we take into consideration the formalities/delivery times of a new server, k+m+2 is not luxury (Depending on the growth of your volume). ________________________________________________________ Cordialement, *David CASIER* ________________________________________________________ Le mar. 5 déc. 2023 à 11:17, Patrick Begou < Patrick.Begou@univ-grenoble-alpes.fr> a écrit :
Hi Robert,
Le 05/12/2023 à 10:05, Robert Sander a écrit :
On 12/5/23 10:01, duluxoz wrote:
Thanks David, I knew I had something wrong :-)
Just for my own edification: Why is k=2, m=1 not recommended for production? Considered to "fragile", or something else?
It is the same as a replicated pool with size=2. Only one host can go down. After that you risk to lose data.
Erasure coding is possible with a cluster size of 10 nodes or more. With smaller clusters you have to go with replicated pools.
Could you explain why 10 nodes are required for EC ?
On my side, I'm working on building my first (small) Ceph cluster using E.C. and I was thinking about 5 nodes and k=4 m=2. With a failure domain on host and several osd by nodes, in my mind this setup may run degraded with 3 nodes using 2 distincts osd by node and the ultimate possibility to loose an additional node without loosing data. Of course with sufficient free storage available.
Am I totally wrong in my first ceph approach ?
Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ok, so I've misunderstood the meaning of failure domain. If there is no way to request using 2 osd/node and node as failure domain, with 5 nodes k=3+m=1 is not secure enough and I will have to use k=2+m=2, so like a raid1 setup. A little bit better than replication in the point of view of global storage capacity. Patrick Le 05/12/2023 à 12:19, David C. a écrit :
Hi,
To return to my comparison with SANs, on a SAN you have spare disks to repair a failed disk.
On Ceph, you therefore need at least one more host (k+m+1).
If we take into consideration the formalities/delivery times of a new server, k+m+2 is not luxury (Depending on the growth of your volume).
________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le mar. 5 déc. 2023 à 11:17, Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> a écrit :
Hi Robert,
Le 05/12/2023 à 10:05, Robert Sander a écrit : > On 12/5/23 10:01, duluxoz wrote: >> Thanks David, I knew I had something wrong :-) >> >> Just for my own edification: Why is k=2, m=1 not recommended for >> production? Considered to "fragile", or something else? > > It is the same as a replicated pool with size=2. Only one host can go > down. After that you risk to lose data. > > Erasure coding is possible with a cluster size of 10 nodes or more. > With smaller clusters you have to go with replicated pools. > Could you explain why 10 nodes are required for EC ?
On my side, I'm working on building my first (small) Ceph cluster using E.C. and I was thinking about 5 nodes and k=4 m=2. With a failure domain on host and several osd by nodes, in my mind this setup may run degraded with 3 nodes using 2 distincts osd by node and the ultimate possibility to loose an additional node without loosing data. Of course with sufficient free storage available.
Am I totally wrong in my first ceph approach ?
Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Patrick, If your hardware is new and you are confident in the support of your hardware and can consider future expansion, you can possibly start with a k=3 and m=2. It is true that we generally prefer to divide (k) the data by an exponent 2, but k=3 does the job Be careful, it is difficult/painful to change profiles later (need data migration). ________________________________________________________ Cordialement, *David CASIER* ________________________________________________________ Le mar. 5 déc. 2023 à 12:35, Patrick Begou < Patrick.Begou@univ-grenoble-alpes.fr> a écrit :
Ok, so I've misunderstood the meaning of failure domain. If there is no way to request using 2 osd/node and node as failure domain, with 5 nodes k=3+m=1 is not secure enough and I will have to use k=2+m=2, so like a raid1 setup. A little bit better than replication in the point of view of global storage capacity.
Patrick
Le 05/12/2023 à 12:19, David C. a écrit :
Hi,
To return to my comparison with SANs, on a SAN you have spare disks to repair a failed disk.
On Ceph, you therefore need at least one more host (k+m+1).
If we take into consideration the formalities/delivery times of a new server, k+m+2 is not luxury (Depending on the growth of your volume).
________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le mar. 5 déc. 2023 à 11:17, Patrick Begou < Patrick.Begou@univ-grenoble-alpes.fr> a écrit :
Hi Robert,
Le 05/12/2023 à 10:05, Robert Sander a écrit :
On 12/5/23 10:01, duluxoz wrote:
Thanks David, I knew I had something wrong :-)
Just for my own edification: Why is k=2, m=1 not recommended for production? Considered to "fragile", or something else?
It is the same as a replicated pool with size=2. Only one host can go down. After that you risk to lose data.
Erasure coding is possible with a cluster size of 10 nodes or more. With smaller clusters you have to go with replicated pools.
Could you explain why 10 nodes are required for EC ?
On my side, I'm working on building my first (small) Ceph cluster using E.C. and I was thinking about 5 nodes and k=4 m=2. With a failure domain on host and several osd by nodes, in my mind this setup may run degraded with 3 nodes using 2 distincts osd by node and the ultimate possibility to loose an additional node without loosing data. Of course with sufficient free storage available.
Am I totally wrong in my first ceph approach ?
Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Dec 5, 2023 at 6:35 AM Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> wrote:
Ok, so I've misunderstood the meaning of failure domain. If there is no way to request using 2 osd/node and node as failure domain, with 5 nodes k=3+m=1 is not secure enough and I will have to use k=2+m=2, so like a raid1 setup. A little bit better than replication in the point of view of global storage capacity.
I'm not sure what you mean by requesting 2osd/node. If the failure domain is set to the host, then by default k/m refer to hosts, and the PGs will be spread across all OSDs on all hosts, but with any particular PG only being present on one OSD on each host. You can get fancy with device classes and crush rules and such and be more specific with how they're allocated, but that would be the typical behavior. Since k/m refer to hosts, then k+m must be less than or equal to the number of hosts or you'll have a degraded pool because there won't be enough hosts to allocate them all. It won't ever stack them across multiple OSDs on the same host with that configuration. k=2,m=2 with min=3 would require at least 4 hosts (k+m), and would allow you to operate degraded with a single host down, and the PGs would become inactive but would still be recoverable with two hosts down. While strictly speaking only 4 hosts are required, you'd do better to have more than that since then the cluster can immediately recover from a loss, assuming you have sufficient space. As you say it is no more space-efficient than RAID1 or size=2, and it suffers write amplification for modifications, but it does allow recovery after the loss of up to two hosts, and you can operate degraded with one host down which allows for somewhat high availability. -- Rich
You can structure your crush map so that you get multiple EC chunks per host in a way that you can still survive a host outage outage even though you have fewer hosts than k+1 For example if you run an EC=4+2 profile on 3 hosts you can structure your crushmap so that you have 2 chunks per host. This means even if one host is down you are still guaranteed to have 4 chunks available. If you then set min_size = 4 you can still operate your cluster in that situation - albeit risky since any additional failure in that time will lead to data loss. However in a highly constrained setup it might be a trade-off that's worth it for you. There have been examples of this on this mailing list in the past. On Wed, 6 Dec 2023 at 12:11, Rich Freeman <r-ceph@rich0.org> wrote:
On Tue, Dec 5, 2023 at 6:35 AM Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> wrote:
Ok, so I've misunderstood the meaning of failure domain. If there is no way to request using 2 osd/node and node as failure domain, with 5 nodes k=3+m=1 is not secure enough and I will have to use k=2+m=2, so like a raid1 setup. A little bit better than replication in the point of view of global storage capacity.
I'm not sure what you mean by requesting 2osd/node. If the failure domain is set to the host, then by default k/m refer to hosts, and the PGs will be spread across all OSDs on all hosts, but with any particular PG only being present on one OSD on each host. You can get fancy with device classes and crush rules and such and be more specific with how they're allocated, but that would be the typical behavior.
Since k/m refer to hosts, then k+m must be less than or equal to the number of hosts or you'll have a degraded pool because there won't be enough hosts to allocate them all. It won't ever stack them across multiple OSDs on the same host with that configuration.
k=2,m=2 with min=3 would require at least 4 hosts (k+m), and would allow you to operate degraded with a single host down, and the PGs would become inactive but would still be recoverable with two hosts down. While strictly speaking only 4 hosts are required, you'd do better to have more than that since then the cluster can immediately recover from a loss, assuming you have sufficient space. As you say it is no more space-efficient than RAID1 or size=2, and it suffers write amplification for modifications, but it does allow recovery after the loss of up to two hosts, and you can operate degraded with one host down which allows for somewhat high availability.
-- Rich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 06/12/2023 à 00:11, Rich Freeman a écrit :
On Tue, Dec 5, 2023 at 6:35 AM Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> wrote:
Ok, so I've misunderstood the meaning of failure domain. If there is no way to request using 2 osd/node and node as failure domain, with 5 nodes k=3+m=1 is not secure enough and I will have to use k=2+m=2, so like a raid1 setup. A little bit better than replication in the point of view of global storage capacity.
I'm not sure what you mean by requesting 2osd/node. If the failure domain is set to the host, then by default k/m refer to hosts, and the PGs will be spread across all OSDs on all hosts, but with any particular PG only being present on one OSD on each host. You can get fancy with device classes and crush rules and such and be more specific with how they're allocated, but that would be the typical behavior.
Since k/m refer to hosts, then k+m must be less than or equal to the number of hosts or you'll have a degraded pool because there won't be enough hosts to allocate them all. It won't ever stack them across multiple OSDs on the same host with that configuration.
k=2,m=2 with min=3 would require at least 4 hosts (k+m), and would allow you to operate degraded with a single host down, and the PGs would become inactive but would still be recoverable with two hosts down. While strictly speaking only 4 hosts are required, you'd do better to have more than that since then the cluster can immediately recover from a loss, assuming you have sufficient space. As you say it is no more space-efficient than RAID1 or size=2, and it suffers write amplification for modifications, but it does allow recovery after the loss of up to two hosts, and you can operate degraded with one host down which allows for somewhat high availability.
Hi Rich, My understood was that k and m were for EC chunks not hosts. 🙁 Of course if k and m are hosts the best choice would be k=2 and m=2. When Christian wrote: /For example if you run an EC=4+2 profile on 3 hosts you can structure your crushmap so that you have 2 chunks per host. This means even if one host is down you are still guaranteed to have 4 chunks available./ This is that I had thought before (and using 5 nodes instead of 3 as the Christian's example). But it does not match what you explain if k and m are nodes. I'm a little bit confused with crushmap settings. Patrick
Hi Patrick, Yes K and M are chunks, but the default crush map is a chunk per host, which is probably the best way to do it, but I'm no expert. I'm not sure why you would want to do a crush map with 2 chunks per host and min size 4 as it' s just asking for trouble at some point, in my opinion. Anyway, take a look at this post if your interested in doing 2 chunks per host it will give you an idea of crushmap setup, https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NB3M22GNAC7V... . Regards, Curt On Wed, Dec 6, 2023 at 6:26 PM Patrick Begou < Patrick.Begou@univ-grenoble-alpes.fr> wrote:
Le 06/12/2023 à 00:11, Rich Freeman a écrit :
On Tue, Dec 5, 2023 at 6:35 AM Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> wrote:
Ok, so I've misunderstood the meaning of failure domain. If there is no way to request using 2 osd/node and node as failure domain, with 5 nodes k=3+m=1 is not secure enough and I will have to use k=2+m=2, so like a raid1 setup. A little bit better than replication in the point of view of global storage capacity.
I'm not sure what you mean by requesting 2osd/node. If the failure domain is set to the host, then by default k/m refer to hosts, and the PGs will be spread across all OSDs on all hosts, but with any particular PG only being present on one OSD on each host. You can get fancy with device classes and crush rules and such and be more specific with how they're allocated, but that would be the typical behavior.
Since k/m refer to hosts, then k+m must be less than or equal to the number of hosts or you'll have a degraded pool because there won't be enough hosts to allocate them all. It won't ever stack them across multiple OSDs on the same host with that configuration.
k=2,m=2 with min=3 would require at least 4 hosts (k+m), and would allow you to operate degraded with a single host down, and the PGs would become inactive but would still be recoverable with two hosts down. While strictly speaking only 4 hosts are required, you'd do better to have more than that since then the cluster can immediately recover from a loss, assuming you have sufficient space. As you say it is no more space-efficient than RAID1 or size=2, and it suffers write amplification for modifications, but it does allow recovery after the loss of up to two hosts, and you can operate degraded with one host down which allows for somewhat high availability.
Hi Rich,
My understood was that k and m were for EC chunks not hosts. 🙁 Of course if k and m are hosts the best choice would be k=2 and m=2.
When Christian wrote: /For example if you run an EC=4+2 profile on 3 hosts you can structure your crushmap so that you have 2 chunks per host. This means even if one host is down you are still guaranteed to have 4 chunks available./
This is that I had thought before (and using 5 nodes instead of 3 as the Christian's example). But it does not match what you explain if k and m are nodes.
I'm a little bit confused with crushmap settings.
Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, the post linked in the previous message is a good source for different approaches. To provide some first-hand experience, I was operating a pool with a 6+2 EC profile on 4 hosts for a while (until we got more hosts) and the "subdivide a physical host into 2 crush-buckets" approach is actually working best (I basically tried all the approaches described in the linked post and they all had pitfalls). Procedure is more or less: - add second (logical) host bucket for each physical host by suffixing the host name with "-B" (ceph osd crush add-bucket <name> <type> <location>) - move half the OSDs per host to this new host bucket (ceph osd crush move osd.ID host=HOSTNAME-B) - make this location persist reboot of the OSDs (ceph config set osd.ID crush_location host=HOSTNAME-B") This will allow you to move OSDs back easily when you get more hosts and can afford the recommended 1 shard per host. It will also show which and where OSDs are moved to with a simple "ceph config dump | grep crush_location". Bets of all, you don't have to fiddle around with crush maps and hope they do what you want. Just use failure domain host and you are good. No more than 2 host buckets per physical host means no more than 2 shards per physical host with default placement rules. I was operating this set-up with min_size=6 and feeling bad about it due to the reduced maintainability (risk of data loss during maintenance). Its not great really, but sometimes there is no way around it. I was happy when I got the extra hosts. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Curt <lightspd@gmail.com> Sent: Wednesday, December 6, 2023 3:56 PM To: Patrick Begou Cc: ceph-users@ceph.io Subject: [ceph-users] Re: EC Profiles & DR Hi Patrick, Yes K and M are chunks, but the default crush map is a chunk per host, which is probably the best way to do it, but I'm no expert. I'm not sure why you would want to do a crush map with 2 chunks per host and min size 4 as it' s just asking for trouble at some point, in my opinion. Anyway, take a look at this post if your interested in doing 2 chunks per host it will give you an idea of crushmap setup, https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NB3M22GNAC7V... . Regards, Curt
Le 06/12/2023 à 16:21, Frank Schilder a écrit :
Hi,
the post linked in the previous message is a good source for different approaches.
To provide some first-hand experience, I was operating a pool with a 6+2 EC profile on 4 hosts for a while (until we got more hosts) and the "subdivide a physical host into 2 crush-buckets" approach is actually working best (I basically tried all the approaches described in the linked post and they all had pitfalls).
Procedure is more or less:
- add second (logical) host bucket for each physical host by suffixing the host name with "-B" (ceph osd crush add-bucket <name> <type> <location>) - move half the OSDs per host to this new host bucket (ceph osd crush move osd.ID host=HOSTNAME-B) - make this location persist reboot of the OSDs (ceph config set osd.ID crush_location host=HOSTNAME-B")
This will allow you to move OSDs back easily when you get more hosts and can afford the recommended 1 shard per host. It will also show which and where OSDs are moved to with a simple "ceph config dump | grep crush_location". Bets of all, you don't have to fiddle around with crush maps and hope they do what you want. Just use failure domain host and you are good. No more than 2 host buckets per physical host means no more than 2 shards per physical host with default placement rules.
I was operating this set-up with min_size=6 and feeling bad about it due to the reduced maintainability (risk of data loss during maintenance). Its not great really, but sometimes there is no way around it. I was happy when I got the extra hosts.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Curt<lightspd@gmail.com> Sent: Wednesday, December 6, 2023 3:56 PM To: Patrick Begou Cc:ceph-users@ceph.io Subject: [ceph-users] Re: EC Profiles & DR
Hi Patrick,
Yes K and M are chunks, but the default crush map is a chunk per host, which is probably the best way to do it, but I'm no expert. I'm not sure why you would want to do a crush map with 2 chunks per host and min size 4 as it' s just asking for trouble at some point, in my opinion. Anyway, take a look at this post if your interested in doing 2 chunks per host it will give you an idea of crushmap setup, https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NB3M22GNAC7V... .
Regards, Curt
Thanks all for this details that clarify many things for me. Rich, yes I'm starting with 5 nodes and 4 HDD/node to set up the first Ceph cluster in the laboratory and my goal is to increase this cluster (may be up to 10 nodes) and to add storage in the nodes (until 12 OSD per node). It is a starting point for capacitif storage connected to my two clusters (400 cores + 256 cores). Thanks Franck for these details, as a newbie Iwould never have thought to this strategy. In my mind, this is the best way for starting the first setup and moving to a more standard configuration later. I've all the template now, just have to dive deeper in the details to build it. Patrick
On Wed, Dec 6, 2023 at 9:25 AM Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> wrote:
My understood was that k and m were for EC chunks not hosts. 🙁 Of course if k and m are hosts the best choice would be k=2 and m=2.
A few others have already replied - as they said if the failure domain is set to host then it will put only one chunk per host for each PG. You can get fancy to alter this behavior (something that hadn't originally occurred to me), but you'll need to use care to ensure that your host redundancy is what you want it to be. The main reason to do that I would think would either be as an interim configuration, or if you want to have a different level of disk redundancy than host redundancy. There might be some use cases I'm not thinking of, but it is definitely atypical. In any case, if you start putting multiple chunks for a PG on a single host, then you'll have to use care to ensure that you can achieve whatever goals you have for host failures and high availability. You'll lose more replicas when a single host goes down. If anything, I think the more common pattern I've seen is to have even more distribution of chunks than the host level, such as distributing them by racks/switches/PDUs/datacenters/etc. Personally if I had 5 nodes and they were balanced, I'd probably just run k=2, m=2, or even just size=3, and set the failure domain to host. I avoid manually editing crush maps but my needs are fairly straightforward - I'm just using hdd/ssd device classes and every pool is on one or the other with a host failure domain. I'm also using rook which abstracts it a bit further, but it isn't doing anything too fancy with the actual pools besides mapping them to k8s entities and running the various daemons. -- Rich
Usually EC requires at least k+1 to be up and active for the pool to be working. Setting the min value to k risks dataloss. ________________________________ From: duluxoz <duluxoz@gmail.com> Sent: 05 December 2023 09:01 To: rivera.david87@gmail.com <rivera.david87@gmail.com>; matthew@peregrineit.net <matthew@peregrineit.net> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: EC Profiles & DR CAUTION: This email originates from outside THG Thanks David, I knew I had something wrong :-) Just for my own edification: Why is k=2, m=1 not recommended for production? Considered to "fragile", or something else? Cheers Dulux-Oz On 05/12/2023 19:53, David Rivera wrote:
First problem here is you are using crush-failure-domain=osd when you should use crush-failure-domain=host. With three hosts, you should use k=2, m=1; this is not recommended in production environment.
On Mon, Dec 4, 2023, 23:26 duluxoz <duluxoz@gmail.com> wrote:
Hi All,
Looking for some help/explanation around erasure code pools, etc.
I set up a 3-node Ceph (Quincy) cluster with each box holding 7 OSDs (HDDs) and each box running Monitor, Manager, and iSCSI Gateway. For the record the cluster runs beautifully, without resource issues, etc.
I created an Erasure Code Profile, etc:
~~~ ceph osd erasure-code-profile set my_ec_profile plugin=jerasure k=4 m=2 crush-failure-domain=osd ceph osd crush rule create-erasure my_ec_rule my_ec_profile ceph osd crush rule create-replicated my_replicated_rule default host ~~~
My Crush Map is:
~~~ # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54
# devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd
# types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root
# buckets host ceph_1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.34769 item osd.1 weight 5.45799 item osd.2 weight 5.45799 item osd.3 weight 5.45799 item osd.4 weight 5.45799 item osd.5 weight 5.45799 item osd.6 weight 5.45799 } host ceph_2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.7 weight 5.34769 item osd.8 weight 5.45799 item osd.9 weight 5.45799 item osd.10 weight 5.45799 item osd.11 weight 5.45799 item osd.12 weight 5.45799 item osd.13 weight 5.45799 } host ceph_3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.14 weight 5.34769 item osd.15 weight 5.45799 item osd.16 weight 5.45799 item osd.17 weight 5.45799 item osd.18 weight 5.45799 item osd.19 weight 5.45799 item osd.20 weight 5.45799 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 114.28693 alg straw2 hash 0 # rjenkins1 item ceph_1 weight 38.09564 item ceph_2 weight 38.09564 item ceph_3 weight 38.09564 }
# rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_replicated_rule { id 1 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_ec_rule { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit }
# end crush map ~~~
Finally I create a pool:
~~~ ceph osd pool create my_pool 32 32 erasure my_ec_profile my_ec_rule ceph osd pool application enable my_meta_pool rbd rbd pool init my_meta_pool rbd pool init my_pool rbd create --size 16T my_pool/my_disk_1 --data-pool my_pool --image-feature journaling ~~~
So all this is to have some VMs (oVirt VMs, for the record) with automatic fall-over in the case of a Ceph Node loss - ie I was trying to "replicate" a 3-Disk RAID 5 array across the Ceph Nodes, so that I could loose a Node and still have a working set of VMs.
However, I took one of the Ceph Nodes down (gracefully) for some maintenance the other day and I lost *all* the VMs (ie oVirt complained that there was no active pool). As soon as I brought the down node back up everything was good again.
So my question is: What did I do wrong with my config?
Sound I, for example, change the EC Profile to `k=2, m=1`, but how is that practically different from `k=4, m=2` - yes, the later spreads the pool over more disks, but it should still only put 2 disks on each node, shouldn't it?
Thanks in advance
Cheers
Dulux-Oz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Matthew, To make a simplistic comparison, it is generally not recommended to raid 5 with large disks (>1 TB) due to the probability (low but not zero) of losing another disk during the rebuild. So imagine losing a host full of disks. Additionally, min_size=1 means you can no longer maintain your cluster (update, etc.), it's dangerous. Unless you can afford to lose/rebuild your cluster, you should never have a min_size <2 ________________________________________________________ Cordialement, *David CASIER* ________________________________________________________ Le mar. 5 déc. 2023 à 10:03, duluxoz <duluxoz@gmail.com> a écrit :
Thanks David, I knew I had something wrong :-)
Just for my own edification: Why is k=2, m=1 not recommended for production? Considered to "fragile", or something else?
Cheers
Dulux-Oz
On 05/12/2023 19:53, David Rivera wrote:
First problem here is you are using crush-failure-domain=osd when you should use crush-failure-domain=host. With three hosts, you should use k=2, m=1; this is not recommended in production environment.
On Mon, Dec 4, 2023, 23:26 duluxoz <duluxoz@gmail.com> wrote:
Hi All,
Looking for some help/explanation around erasure code pools, etc.
I set up a 3-node Ceph (Quincy) cluster with each box holding 7 OSDs (HDDs) and each box running Monitor, Manager, and iSCSI Gateway. For the record the cluster runs beautifully, without resource issues, etc.
I created an Erasure Code Profile, etc:
~~~ ceph osd erasure-code-profile set my_ec_profile plugin=jerasure k=4 m=2 crush-failure-domain=osd ceph osd crush rule create-erasure my_ec_rule my_ec_profile ceph osd crush rule create-replicated my_replicated_rule default host ~~~
My Crush Map is:
~~~ # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54
# devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd
# types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root
# buckets host ceph_1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.34769 item osd.1 weight 5.45799 item osd.2 weight 5.45799 item osd.3 weight 5.45799 item osd.4 weight 5.45799 item osd.5 weight 5.45799 item osd.6 weight 5.45799 } host ceph_2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.7 weight 5.34769 item osd.8 weight 5.45799 item osd.9 weight 5.45799 item osd.10 weight 5.45799 item osd.11 weight 5.45799 item osd.12 weight 5.45799 item osd.13 weight 5.45799 } host ceph_3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.14 weight 5.34769 item osd.15 weight 5.45799 item osd.16 weight 5.45799 item osd.17 weight 5.45799 item osd.18 weight 5.45799 item osd.19 weight 5.45799 item osd.20 weight 5.45799 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 114.28693 alg straw2 hash 0 # rjenkins1 item ceph_1 weight 38.09564 item ceph_2 weight 38.09564 item ceph_3 weight 38.09564 }
# rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_replicated_rule { id 1 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_ec_rule { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit }
# end crush map ~~~
Finally I create a pool:
~~~ ceph osd pool create my_pool 32 32 erasure my_ec_profile my_ec_rule ceph osd pool application enable my_meta_pool rbd rbd pool init my_meta_pool rbd pool init my_pool rbd create --size 16T my_pool/my_disk_1 --data-pool my_pool --image-feature journaling ~~~
So all this is to have some VMs (oVirt VMs, for the record) with automatic fall-over in the case of a Ceph Node loss - ie I was trying to "replicate" a 3-Disk RAID 5 array across the Ceph Nodes, so that I could loose a Node and still have a working set of VMs.
However, I took one of the Ceph Nodes down (gracefully) for some maintenance the other day and I lost *all* the VMs (ie oVirt complained that there was no active pool). As soon as I brought the down node back up everything was good again.
So my question is: What did I do wrong with my config?
Sound I, for example, change the EC Profile to `k=2, m=1`, but how is that practically different from `k=4, m=2` - yes, the later spreads the pool over more disks, but it should still only put 2 disks on each node, shouldn't it?
Thanks in advance
Cheers
Dulux-Oz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
And the second issue is with k4 m2 you'll have min_size = 5 which means if one host is down your PGs become inactive, which is what you most likely experienced. Zitat von David Rivera <rivera.david87@gmail.com>:
First problem here is you are using crush-failure-domain=osd when you should use crush-failure-domain=host. With three hosts, you should use k=2, m=1; this is not recommended in production environment.
On Mon, Dec 4, 2023, 23:26 duluxoz <duluxoz@gmail.com> wrote:
Hi All,
Looking for some help/explanation around erasure code pools, etc.
I set up a 3-node Ceph (Quincy) cluster with each box holding 7 OSDs (HDDs) and each box running Monitor, Manager, and iSCSI Gateway. For the record the cluster runs beautifully, without resource issues, etc.
I created an Erasure Code Profile, etc:
~~~ ceph osd erasure-code-profile set my_ec_profile plugin=jerasure k=4 m=2 crush-failure-domain=osd ceph osd crush rule create-erasure my_ec_rule my_ec_profile ceph osd crush rule create-replicated my_replicated_rule default host ~~~
My Crush Map is:
~~~ # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54
# devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd
# types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root
# buckets host ceph_1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.34769 item osd.1 weight 5.45799 item osd.2 weight 5.45799 item osd.3 weight 5.45799 item osd.4 weight 5.45799 item osd.5 weight 5.45799 item osd.6 weight 5.45799 } host ceph_2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.7 weight 5.34769 item osd.8 weight 5.45799 item osd.9 weight 5.45799 item osd.10 weight 5.45799 item osd.11 weight 5.45799 item osd.12 weight 5.45799 item osd.13 weight 5.45799 } host ceph_3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.14 weight 5.34769 item osd.15 weight 5.45799 item osd.16 weight 5.45799 item osd.17 weight 5.45799 item osd.18 weight 5.45799 item osd.19 weight 5.45799 item osd.20 weight 5.45799 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 114.28693 alg straw2 hash 0 # rjenkins1 item ceph_1 weight 38.09564 item ceph_2 weight 38.09564 item ceph_3 weight 38.09564 }
# rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_replicated_rule { id 1 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_ec_rule { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit }
# end crush map ~~~
Finally I create a pool:
~~~ ceph osd pool create my_pool 32 32 erasure my_ec_profile my_ec_rule ceph osd pool application enable my_meta_pool rbd rbd pool init my_meta_pool rbd pool init my_pool rbd create --size 16T my_pool/my_disk_1 --data-pool my_pool --image-feature journaling ~~~
So all this is to have some VMs (oVirt VMs, for the record) with automatic fall-over in the case of a Ceph Node loss - ie I was trying to "replicate" a 3-Disk RAID 5 array across the Ceph Nodes, so that I could loose a Node and still have a working set of VMs.
However, I took one of the Ceph Nodes down (gracefully) for some maintenance the other day and I lost *all* the VMs (ie oVirt complained that there was no active pool). As soon as I brought the down node back up everything was good again.
So my question is: What did I do wrong with my config?
Sound I, for example, change the EC Profile to `k=2, m=1`, but how is that practically different from `k=4, m=2` - yes, the later spreads the pool over more disks, but it should still only put 2 disks on each node, shouldn't it?
Thanks in advance
Cheers
Dulux-Oz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (11)
-
Christian Wuerdig
-
Curt
-
Danny Webb
-
David C.
-
David Rivera
-
duluxoz
-
Eugen Block
-
Frank Schilder
-
Patrick Begou
-
Rich Freeman
-
Robert Sander