Hi all, newbie question: The documentation seems to suggest that with ceph-volume, one OSD is created for each HDD (cf. 4-HDD-example in https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/) This seems odd: what if a server has a finite number of disks? I was going to try cephfs on ~10 servers with 70 HDD each. That would make each system having to deal with 70 OSDs, on 70 LVs? Really no aggregation of the disks? Regards, Thomas -- -------------------------------------------------------------------- Thomas Roth Department: IT GSI Helmholtzzentrum für Schwerionenforschung GmbH www.gsi.de
On 22.06.21 11:55, Thomas Roth wrote:
That would make each system having to deal with 70 OSDs, on 70 LVs?
Yes. And 70 is a rather unusual number of HDDs in a Ceph node. Normally you have something like 20 to 24 block devices in a single node. Each OSD needs CPU and RAM. You could theoretically RAID0 multiple disks and then put an OSD on top of that but this would create very large OSDs which are not good for recovering data. Recovering such a "beast" just would take too long. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, On 22.06.21 11:55, Thomas Roth wrote:
Hi all,
newbie question:
The documentation seems to suggest that with ceph-volume, one OSD is created for each HDD (cf. 4-HDD-example in https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/)
This seems odd: what if a server has a finite number of disks? I was going to try cephfs on ~10 servers with 70 HDD each. That would make each system having to deal with 70 OSDs, on 70 LVs?
Really no aggregation of the disks?
The recommendation is one OSD per HDD. And most deployment tools (ceph-ansible / cephadm) follow this guide. The main reason is keeping the single failure unit small. Using raid with several HDDs might result in 10-20 TB of data to be recovered if a single disk fails. The same goes for large storage boxes; just imagine a complete unrecoverable failure of a box with 70x 10TB disks. More data to recover results in longer recovery time, which increases the probability for a second or (usually fatal) third failure. If you plan to use other setups you either need to adopt the deployment tools (should be possible with ceph-ansible, difficult in case of cephadm), or deploy the OSDs manually. We use manual deployment and linux software raid with raid level 0 and two/three disks in larger storage boxes, also due to the memory /CPU core requirements. Regards, Burkhard
Hi, just an addition: currentl CEPH releases also include disk monitoring (e.g. SMART and other health related features). These do not work with raid devices. You will need external monitoring for your OSD disks. Regards, Burkhard
That is the idea, what is wrong with this concept? If you aggregate disks, you still aggregate 70 disks, and you still be having 70 disks. Everything you do that ceph can't be aware of creates a potential misinterpretation of the reality and make ceph act in a way it should not.
-----Original Message----- Sent: Tuesday, 22 June 2021 11:55 To: ceph-users@ceph.io Subject: [ceph-users] HDD <-> OSDs
Hi all,
newbie question:
The documentation seems to suggest that with ceph-volume, one OSD is created for each HDD (cf. 4-HDD-example in https://docs.ceph.com/en/latest/rados/configuration/bluestore-config- ref/)
This seems odd: what if a server has a finite number of disks? I was going to try cephfs on ~10 servers with 70 HDD each. That would make each system having to deal with 70 OSDs, on 70 LVs?
Really no aggregation of the disks?
Regards, Thomas -- -------------------------------------------------------------------- Thomas Roth Department: IT
GSI Helmholtzzentrum für Schwerionenforschung GmbH www.gsi.de _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you all for the clarification! I just did not grasp the concept before, probably because I am used to those systems that form a layer on top of the local file system. If ceph does it all, down to the magnetic platter, all the better. Cheers Thomas On 6/22/21 12:15 PM, Marc wrote:
That is the idea, what is wrong with this concept? If you aggregate disks, you still aggregate 70 disks, and you still be having 70 disks. Everything you do that ceph can't be aware of creates a potential misinterpretation of the reality and make ceph act in a way it should not.
-----Original Message----- Sent: Tuesday, 22 June 2021 11:55 To: ceph-users@ceph.io Subject: [ceph-users] HDD <-> OSDs
Hi all,
newbie question:
The documentation seems to suggest that with ceph-volume, one OSD is created for each HDD (cf. 4-HDD-example in https://docs.ceph.com/en/latest/rados/configuration/bluestore-config- ref/)
This seems odd: what if a server has a finite number of disks? I was going to try cephfs on ~10 servers with 70 HDD each. That would make each system having to deal with 70 OSDs, on 70 LVs?
Really no aggregation of the disks?
Regards, Thomas -- -------------------------------------------------------------------- Thomas Roth Department: IT
GSI Helmholtzzentrum für Schwerionenforschung GmbH www.gsi.de _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
hi thomas, just a quick note. If you have a few large osds, ceph will have problems distributing the data based on the number of placement groups and the number of objects per placement group, ... I recommend reading the concept of placement groups. ___________________________________ Clyso GmbH - Ceph Foundation Member support@clyso.com https://www.clyso.com Am 22.06.2021 um 12:56 schrieb Thomas Roth:
Thank you all for the clarification!
I just did not grasp the concept before, probably because I am used to those systems that form a layer on top of the local file system. If ceph does it all, down to the magnetic platter, all the better.
Cheers Thomas
On 6/22/21 12:15 PM, Marc wrote:
That is the idea, what is wrong with this concept? If you aggregate disks, you still aggregate 70 disks, and you still be having 70 disks. Everything you do that ceph can't be aware of creates a potential misinterpretation of the reality and make ceph act in a way it should not.
-----Original Message----- Sent: Tuesday, 22 June 2021 11:55 To: ceph-users@ceph.io Subject: [ceph-users] HDD <-> OSDs
Hi all,
newbie question:
The documentation seems to suggest that with ceph-volume, one OSD is created for each HDD (cf. 4-HDD-example in https://docs.ceph.com/en/latest/rados/configuration/bluestore-config- ref/)
This seems odd: what if a server has a finite number of disks? I was going to try cephfs on ~10 servers with 70 HDD each. That would make each system having to deal with 70 OSDs, on 70 LVs?
Really no aggregation of the disks?
Regards, Thomas -- -------------------------------------------------------------------- Thomas Roth Department: IT
GSI Helmholtzzentrum für Schwerionenforschung GmbH www.gsi.de _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den tis 22 juni 2021 kl 11:55 skrev Thomas Roth <t.roth@gsi.de>:
Hi all, newbie question: The documentation seems to suggest that with ceph-volume, one OSD is created for each HDD (cf. 4-HDD-example in https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/)
This seems odd: what if a server has a finite number of disks? I was going to try cephfs on ~10 servers with 70 HDD each. That would make each system having to deal with 70 OSDs, on 70 LVs? Really no aggregation of the disks?
There is nothing inherently wrong with having 70 OSDs if you have 70 drives. As others have said, it brings benefits to allow single faults to be kept do as small a domain as possible and if you read up on ZFS guides, I am sure they will say the same thing, hand over each separate raw device over to zfs, and let zfs place the data in the optimal way for each raw device, instead of globbing together the lot and pretend it is one large disks when it actually isn't. With one huge 70-piece raid0 you might end up having a lot of writes and reads hit a small number of disks at a time which probably will be worse for your overall performance. -- May the most significant bit of your life be positive.
participants (7)
-
Burkhard Linke
-
Clyso GmbH - Ceph Foundation Member
-
Janne Johansson
-
Konstantin Shalygin
-
Marc
-
Robert Sander
-
Thomas Roth