How big an OSD disk could be? - ceph-users - lists.ceph.io

List overview All Threads
Download

How big an OSD disk could be?

Re: How big an OSD disk could be?

Re: Some confusion around PG, OSD...

huxiaoyu＠horebdata.cn

12 Mar 2021 12 Mar '21

8:10 p.m.

Dear cephers, Just wonder how big an OSD disk could be? Currently the biggest HDD has a capacity of 18TB or 20TB. It is suitable for an OSD still? Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB or 100TB a single OSD? What could be the potential limit? best regards, samuel huxiaoyu(a)horebdata.cn

Reply

Show replies by date

Marc

12 Mar 12 Mar

8:16 p.m.

I assume the limits are those that linux imposes. iops are the limits. One 20TB has 100 iops and 4x5TB have 400 iops. 400 iops serves more clients that 100 iops. You decide what you need/want to have.

-----Original Message----- From: huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> Sent: 12 March 2021 18:10 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] How big an OSD disk could be? Dear cephers, Just wonder how big an OSD disk could be? Currently the biggest HDD has a capacity of 18TB or 20TB. It is suitable for an OSD still? Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB or 100TB a single OSD? What could be the potential limit? best regards, samuel huxiaoyu(a)horebdata.cn _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Reply

huxiaoyu＠horebdata.cn

8:30 p.m.

Good point. IOPS is indeed an limitation. Thanks Marc for the tip Any other aspects on the limits of bigger capacity hard disk drives? huxiaoyu(a)horebdata.cn From: Marc Date: 2021-03-12 18:16 To: huxiaoyu(a)horebdata.cn; ceph-users Subject: RE: [ceph-users] How big an OSD disk could be? I assume the limits are those that linux imposes. iops are the limits. One 20TB has 100 iops and 4x5TB have 400 iops. 400 iops serves more clients that 100 iops. You decide what you need/want to have.

-----Original Message----- From: huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> Sent: 12 March 2021 18:10 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] How big an OSD disk could be? Dear cephers, Just wonder how big an OSD disk could be? Currently the biggest HDD has a capacity of 18TB or 20TB. It is suitable for an OSD still? Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB or 100TB a single OSD? What could be the potential limit? best regards, samuel huxiaoyu(a)horebdata.cn _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Reply

Robert Sander

8:34 p.m.

Am 12.03.21 um 18:30 schrieb huxiaoyu(a)horebdata.cn:

Any other aspects on the limits of bigger capacity hard disk drives?

Recovery will take longer increasing the risk of another failure in the same time. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

Reply

Martin Verges

8:59 p.m.

A good mix of size and performance is the Seagate 2X14 MACH.2 Dual Actor 14TB HDD. This drive reports as 2x 7TB individual block devices and you install a OSD on each. https://croit.io/blog/benchmarking-seagate-exos2x14-mach-2-hdds We have a bunch of them in a permanent test cluster if someone want to do some specific Ceph workload benchmarks, feel free to drop me a mail. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Fr., 12. März 2021 um 18:35 Uhr schrieb Robert Sander <r.sander(a)heinlein-support.de>de>: > > Am 12.03.21 um 18:30 schrieb huxiaoyu(a)horebdata.cn: > > > Any other aspects on the limits of bigger capacity hard disk drives? > > Recovery will take longer increasing the risk of another failure in the > same time. > > Regards > -- > Robert Sander > Heinlein Support GmbH > Schwedter Str. 8/9b, 10119 Berlin > > http://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 93818 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Reply

Marc

13 Mar 13 Mar

2:55 p.m.

A good mix of size and performance is the Seagate 2X14 MACH.2 Dual Actor 14TB HDD. This drive reports as 2x 7TB individual block devices and you install a OSD on each.

My first thought was, wow quite nice this dual exposes itself as two drives. I was always under the impression that it was just one single drive with just more iops. But come to think of it, with such a solution you do create a failure domain that ceph does not know about. With the default host failure domain the only problem is that you have an increased risk of 2 drives failing at once. Is it possible to configure this drive whether it exposes itself as two drives or just one bigger-faster one?

Reply

Janne Johansson

3:07 p.m.

Den lör 13 mars 2021 kl 12:56 skrev Marc <Marc(a)f1-outsourcing.eu>eu>:

A good mix of size and performance is the Seagate 2X14 MACH.2 Dual Actor 14TB HDD. This drive reports as 2x 7TB individual block devices and you install a OSD on each.

My first thought was, wow quite nice this dual exposes itself as two drives. I was always under the impression that it was just one single drive with just more iops. But come to think of it, with such a solution you do create a failure domain that ceph does not know about. With the default host failure domain the only problem is that you have an increased risk of 2 drives failing at once. Is it possible to configure this drive whether it exposes itself as two drives or just one bigger-faster one?

Well, if you run with failure-domain=host, then if it says "I have 8 14TB drives and one failed" or "I have 16 7TB drives and two failed" isn't going to matter much in terms of recovery, is it? It would mostly matter for failure-domain=OSD, otherwise it seems about equal. -- May the most significant bit of your life be positive.

Reply

Marc

3:11 p.m.

Well, if you run with failure-domain=host, then if it says "I have 8 14TB drives and one failed" or "I have 16 7TB drives and two failed" isn't going to matter much in terms of recovery, is it? It would mostly matter for failure-domain=OSD, otherwise it seems about equal.

Yes, but especially in small clusters, people are changing the failure domain to osd to be able to use EC (like I have ;))

Reply

Martin Verges

4:13 p.m.

If you have a small cluster, without host redundancy, you are still able to configure this in Ceph to be handled correctly by adding a drive failure domain between host and OSD level. So yes you need to change more then just failure-domain=OSD, as this would be a problem. However it is absolutely the same as to having multiple OSDs per NVMe as some people do it. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 13. März 2021 um 13:11 Uhr schrieb Marc <Marc(a)f1-outsourcing.eu>eu>: > > > Well, if you run with failure-domain=host, then if it says "I have 8 > > 14TB drives and one failed" or "I have 16 7TB drives and two failed" > > isn't going to matter much in terms of recovery, is it? > > It would mostly matter for failure-domain=OSD, otherwise it seems about > > equal. > > Yes, but especially in small clusters, people are changing the failure domain to osd to be able to use EC (like I have ;))

Reply

Martin Verges

4:10 p.m.

failure-domain=host

yes (or rack/room/datacenter/..), for regular clusters it's therefore absolute no problem as you correctly assumed. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 13. März 2021 um 13:08 Uhr schrieb Janne Johansson <icepic.dz(a)gmail.com>om>: > > Den lör 13 mars 2021 kl 12:56 skrev Marc <Marc(a)f1-outsourcing.eu>eu>: > > > A good mix of size and performance is the Seagate 2X14 MACH.2 Dual > > > Actor 14TB HDD. > > > This drive reports as 2x 7TB individual block devices and you install > > > a OSD on each. > > > > My first thought was, wow quite nice this dual exposes itself as two drives. I was always under the impression that it was just one single drive with just more iops. > > But come to think of it, with such a solution you do create a failure domain that ceph does not know about. With the default host failure domain the only problem is that you have an increased risk of 2 drives failing at once. > > Is it possible to configure this drive whether it exposes itself as two drives or just one bigger-faster one? > > Well, if you run with failure-domain=host, then if it says "I have 8 > 14TB drives and one failed" or "I have 16 7TB drives and two failed" > isn't going to matter much in terms of recovery, is it? > It would mostly matter for failure-domain=OSD, otherwise it seems about equal. > > -- > May the most significant bit of your life be positive.

Reply

Anthony D'Atri

12 Mar 12 Mar

9:06 p.m.

I assume the limits are those that linux imposes. iops are the limits. One 20TB has 100 iops and 4x5TB have 400 iops. 400 iops serves more clients that 100 iops. You decide what you need/want to have.

Any other aspects on the limits of bigger capacity hard disk drives?

Recovery will take longer increasing the risk of another failure in the same time.

Indeed — often overlooked, and it DOES happen. This is especially perilous if one is flirting with diaster with 2R pools. HDD capacity has grown far more rapidly than HDD interfaces have become faster. More vs fewer spindles helps, but it’s also true that every drive bay has a certain cost: a fraction of the chassis, switch ports, power, DC RUs. There are ultradense chassis for sure, which make sense in certain situations, but not others. If your cluster overall isn’t huge, say you have 6 ultradense chassis. I’ve seen ones that require special power to be run to the racks, jut into the aisles, are really bottlenecked by the NICs and RAM capacity, etc. And racks that at most can be half full because of weight or more often amps, those wasted RUs have a cost. Then when one node is down, that’s 1/6 of the total, which could be ugly if the mon_osd_down_out_subtree_limit doesn’t forestall backups — recovery of a large fraction of the cluster will require a large fraction of free space on the other nodes. Recovery/backfill to/from an ultradense node may well overwhelm the NIC/HBA too. This is one reason that we’ll see flash, eg. QLC, increasingly viable as an HDD replacement. One can get a bunch of TB into an RU using ruler drives without sacrificing half the rack, and recovery (and thus exposure to data loss / unavailability) is faster than with HDDs. — aad

Reply

Dan van der Ster

13 Mar 13 Mar

4:28 p.m.

On Fri, Mar 12, 2021 at 6:35 PM Robert Sander <r.sander(a)heinlein-support.de> wrote:

Am 12.03.21 um 18:30 schrieb huxiaoyu(a)horebdata.cn:

Any other aspects on the limits of bigger capacity hard disk drives?

Recovery will take longer increasing the risk of another failure in the same time.

Another limitation is that OSDs should store 100 PGs each regardless of their size, so those PGs will each need to store many more objects and therefore recovery, scrubbing, removal, listing, etc... will all take longer and longer. So perhaps we'll need to change the OSD to allow for 500 or 1000 PGs per OSD eventually, (meaning also that PGs per cluster needs to scale up too!) Cheers, Dan > Regards > -- > Robert Sander > Heinlein Support GmbH > Schwedter Str. 8/9b, 10119 Berlin > > http://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 93818 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Reply

Martin Verges

7:04 p.m.

So perhaps we'll need to change the OSD to allow for 500 or 1000 PGs

We had a support case last year where we where forced to set the OSD limit to >4000 for a few days, and had more then 4k active PGs on that single OSD. You can do that, however it is quite uncommon. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx On Sat, 13 Mar 2021 at 14:29, Dan van der Ster <dan(a)vanderster.com> wrote: > > On Fri, Mar 12, 2021 at 6:35 PM Robert Sander > <r.sander(a)heinlein-support.de> wrote: > > > > Am 12.03.21 um 18:30 schrieb huxiaoyu(a)horebdata.cn: > > > > > Any other aspects on the limits of bigger capacity hard disk drives? > > > > Recovery will take longer increasing the risk of another failure in the > > same time. > > > > Another limitation is that OSDs should store 100 PGs each regardless > of their size, so those PGs will each need to store many more objects > and therefore recovery, scrubbing, removal, listing, etc... will all > take longer and longer. >

So perhaps we'll need to change the OSD to allow for 500 or 1000 PGs

> per OSD eventually, (meaning also that PGs per cluster needs to scale > up too!) > > Cheers, Dan > > > Regards > > -- > > Robert Sander > > Heinlein Support GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > http://www.heinlein-support.de > > > > Tel: 030 / 405051-43 > > Fax: 030 / 405051-19 > > > > Zwangsangaben lt. §35a GmbHG: > > HRB 93818 B / Amtsgericht Berlin-Charlottenburg, > > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Reply

huxiaoyu＠horebdata.cn

14 Mar 14 Mar

10:04 a.m.

Indeed，scaling up PGs in an OSD may be needed for larger HDDs. Increasing the number of PGs by 5 fold or 10 fold would have adverse impact of OSD peering. What is the practical limits on the number of PGs per OSD with default setting, OR should we tuning some Ceph default setting for accommodating 1000 or more PGs per OSD? best regards, samuel huxiaoyu(a)horebdata.cn From: Dan van der Ster Date: 2021-03-13 14:28 To: Robert Sander CC: Ceph Users Subject: [ceph-users] Re: How big an OSD disk could be? On Fri, Mar 12, 2021 at 6:35 PM Robert Sander <r.sander(a)heinlein-support.de> wrote:

Am 12.03.21 um 18:30 schrieb huxiaoyu(a)horebdata.cn:

Any other aspects on the limits of bigger capacity hard disk drives?

Recovery will take longer increasing the risk of another failure in the same time.

Another limitation is that OSDs should store 100 PGs each regardless of their size, so those PGs will each need to store many more objects and therefore recovery, scrubbing, removal, listing, etc... will all take longer and longer. So perhaps we'll need to change the OSD to allow for 500 or 1000 PGs per OSD eventually, (meaning also that PGs per cluster needs to scale up too!) Cheers, Dan

Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Reply

1145

days inactive

1147

days old

ceph-users@ceph.io

Manage subscription

13 comments

7 participants

tags (0)

participants (7)

Anthony D'Atri
Dan van der Ster
huxiaoyu＠horebdata.cn
Janne Johansson
Marc
Martin Verges
Robert Sander