Load on drives of different sizes in ceph
Hi cephers, I'm looking for some advice on what to do about drives of different sizes in the same cluster. We have so far kept the drive sizes consistent on our main ceph cluster (using 8TB drives). We're getting some new hardware with larger, 12TB drives next, and I'm pondering on how best to configure them. If I just simply add them, they will have 1.5x the data (which is less of a problem), but will also get 1.5x the iops - so I presume it will slow the whole cluster down as a result (these drives will be busy, and the rest will not be as much). I'm wondering how people generally handle this. I'm more concerned about these larger drives being busier than the rest - so I'd like to be able to put for example 1/3 drive of less accessed data on them in addition to the usual data - to use the extra capacity but not increase the load on them. Is there an easy way to accomplish this? One possibility is to run two OSDs on the drive (in two crush hierarchies), which isn't ideal. Can I just run one OSD somehow and put it into two crush roots, or something similar? Andras
Hi Andras, El 31/3/20 a las 16:42, Andras Pataki escribió:
I'm looking for some advice on what to do about drives of different sizes in the same cluster.
We have so far kept the drive sizes consistent on our main ceph cluster (using 8TB drives). We're getting some new hardware with larger, 12TB drives next, and I'm pondering on how best to configure them. If I just simply add them, they will have 1.5x the data (which is less of a problem), but will also get 1.5x the iops - so I presume it will slow the whole cluster down as a result (these drives will be busy, and the rest will not be as much). I'm wondering how people generally handle this.
I'm more concerned about these larger drives being busier than the rest - so I'd like to be able to put for example 1/3 drive of less accessed data on them in addition to the usual data - to use the extra capacity but not increase the load on them. Is there an easy way to accomplish this? One possibility is to run two OSDs on the drive (in two crush hierarchies), which isn't ideal. Can I just run one OSD somehow and put it into two crush roots, or something similar? You should adjust the weight of the new 12TB disk OSDs, to match the weight of the current 8TB OSDs.
That will make the new disks the same as the old disks to Ceph :-) But you'll lost the extra 4TB space, until you remove the 8TB disks from the cluster... Cheers Eneko -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es
You can adjust the Primary Affinity down on the larger drives so they’ll get less read load. In one test I’ve seen this result in a 10-15% increase in read throughout but it depends on your situation. Optimal settings would require calculations that make my head hurt, maybe someone has a tool but I haven’t seen it. You might want to ensure that the drives are spread evenly across your (unspecified) failure domains so that the extra capacity isn’t wasted, again depending on your topology.
On Mar 31, 2020, at 8:49 AM, Eneko Lacunza <elacunza@binovo.es> wrote:
Hi Andras,
El 31/3/20 a las 16:42, Andras Pataki escribió: I'm looking for some advice on what to do about drives of different sizes in the same cluster.
We have so far kept the drive sizes consistent on our main ceph cluster (using 8TB drives). We're getting some new hardware with larger, 12TB drives next, and I'm pondering on how best to configure them. If I just simply add them, they will have 1.5x the data (which is less of a problem), but will also get 1.5x the iops - so I presume it will slow the whole cluster down as a result (these drives will be busy, and the rest will not be as much). I'm wondering how people generally handle this.
I'm more concerned about these larger drives being busier than the rest - so I'd like to be able to put for example 1/3 drive of less accessed data on them in addition to the usual data - to use the extra capacity but not increase the load on them. Is there an easy way to accomplish this? One possibility is to run two OSDs on the drive (in two crush hierarchies), which isn't ideal. Can I just run one OSD somehow and put it into two crush roots, or something similar? You should adjust the weight of the new 12TB disk OSDs, to match the weight of the current 8TB OSDs.
That will make the new disks the same as the old disks to Ceph :-) But you'll lost the extra 4TB space, until you remove the 8TB disks from the cluster...
Cheers Eneko
-- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is not there a way to Deal with this kind of setup when playing with the "weight" of an OSD? I dont mean the "crush weight". I am in a Situation where i had to Think about to add a Server with 24 x 2TB disks - my other osd nodes has 12 x 4TB. Which is in Sum 48TB per node in both situations. Am 31. März 2020 18:10:24 MESZ schrieb Anthony D'Atri <aad@dreamsnake.net>:
You can adjust the Primary Affinity down on the larger drives so they’ll get less read load. In one test I’ve seen this result in a 10-15% increase in read throughout but it depends on your situation.
Optimal settings would require calculations that make my head hurt, maybe someone has a tool but I haven’t seen it.
You might want to ensure that the drives are spread evenly across your (unspecified) failure domains so that the extra capacity isn’t wasted, again depending on your topology.
On Mar 31, 2020, at 8:49 AM, Eneko Lacunza <elacunza@binovo.es> wrote:
Hi Andras,
El 31/3/20 a las 16:42, Andras Pataki escribió: I'm looking for some advice on what to do about drives of different sizes in the same cluster.
We have so far kept the drive sizes consistent on our main ceph cluster (using 8TB drives). We're getting some new hardware with larger, 12TB drives next, and I'm pondering on how best to configure them. If I just simply add them, they will have 1.5x the data (which is less of a problem), but will also get 1.5x the iops - so I presume it will slow the whole cluster down as a result (these drives will be busy, and the rest will not be as much). I'm wondering how people generally handle this.
I'm more concerned about these larger drives being busier than the rest - so I'd like to be able to put for example 1/3 drive of less accessed data on them in addition to the usual data - to use the extra capacity but not increase the load on them. Is there an easy way to accomplish this? One possibility is to run two OSDs on the drive (in two crush hierarchies), which isn't ideal. Can I just run one OSD somehow and put it into two crush roots, or something similar? You should adjust the weight of the new 12TB disk OSDs, to match the weight of the current 8TB OSDs.
That will make the new disks the same as the old disks to Ceph :-) But you'll lost the extra 4TB space, until you remove the 8TB disks from the cluster...
Cheers Eneko
-- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sure - you can play with the weights or crush weights to make sure that all drives fill evenly to their respective capacities. The consequence of doing this is that about twice as much data will be on the drives with twice the size obviously. But - a perhaps less glaring consequence is that the double size drives will also, by default, get twice the I/O requests (iops). With spinning drives, in general, the number of iops a drive can do is fairly independent of its size (not completely, but it is certainly not proportional). So when adding larger drives this way, the larger drives will determine the overall performance of the cluster - and the smaller drives will be less I/O loaded. There are a few possible choices that I've gathered, none of them ideal: 1. Live with it. Eventually when there are lots of larger drives, they will dominate the performance. 2. Use only part of the drive. This keeps performance - but wastes space. 3. As suggested by Anthony below - read affinity can solve the read iops balance - but it doesn't do much for writes. 4. Create two OSDs on the larger drive. This would allow part of the larger drive to host data that is accessed less frequently (from a different pool) to equalize the iops. Unfortunately this is less than ideal since the two OSDs don't coordinate to schedule I/O for a single spinning drive. Would it be possible to have a single OSD per drive, and still host let's say 2/3 one pool and 1/3 another? This would require the OSD to be in two different crush hierarchies with different weights - I don't know if that is possible. 5. Put the larger drives into a different ceph pool. This way I/O can be controlled by distributing data into pools. Our problem is that there isn't a way to do this transparently with cephfs (there isn't a 100% correct way of moving files between pools). We already do data moves across pools with some safeguards, but without help from the MDS this is not possible to do 100% correctly (it would require holding MDS caps while the data is moved - which isn't possible via the POSIX API). Am I missing any choices/points? Andras On 4/6/20 5:20 PM, ceph@elchaka.de wrote:
Is not there a way to Deal with this kind of setup when playing with the "weight" of an OSD? I dont mean the "crush weight".
I am in a Situation where i had to Think about to add a Server with 24 x 2TB disks - my other osd nodes has 12 x 4TB. Which is in Sum 48TB per node in both situations.
Am 31. März 2020 18:10:24 MESZ schrieb Anthony D'Atri <aad@dreamsnake.net>:
You can adjust the Primary Affinity down on the larger drives so they’ll get less read load. In one test I’ve seen this result in a 10-15% increase in read throughout but it depends on your situation.
Optimal settings would require calculations that make my head hurt, maybe someone has a tool but I haven’t seen it.
You might want to ensure that the drives are spread evenly across your (unspecified) failure domains so that the extra capacity isn’t wasted, again depending on your topology.
On Mar 31, 2020, at 8:49 AM, Eneko Lacunza <elacunza@binovo.es> wrote: Hi Andras,
El 31/3/20 a las 16:42, Andras Pataki escribió: I'm looking for some advice on what to do about drives of different sizes in the same cluster. We have so far kept the drive sizes consistent on our main ceph cluster (using 8TB drives). We're getting some new hardware with larger, 12TB drives next, and I'm pondering on how best to configure them. If I just simply add them, they will have 1.5x the data (which is less of a problem), but will also get 1.5x the iops - so I presume it will slow the whole cluster down as a result (these drives will be busy, and the rest will not be as much). I'm wondering how people generally handle this. I'm more concerned about these larger drives being busier than the rest - so I'd like to be able to put for example 1/3 drive of less accessed data on them in addition to the usual data - to use the extra capacity but not increase the load on them. Is there an easy way to accomplish this? One possibility is to run two OSDs on the drive (in two crush hierarchies), which isn't ideal. Can I just run one OSD somehow and put it into two crush roots, or something similar? You should adjust the weight of the new 12TB disk OSDs, to match the weight of the current 8TB OSDs. That will make the new disks the same as the old disks to Ceph :-) But you'll lost the extra 4TB space, until you remove the 8TB disks from the cluster... Cheers Eneko
-- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Andras Pataki
-
Anthony D'Atri
-
ceph＠elchaka.de
-
Eneko Lacunza