22 Jun
2021
22 Jun
'21
2:25 p.m.
Hi all,
newbie question:
The documentation seems to suggest that with ceph-volume, one OSD is created for each HDD
(cf. 4-HDD-example in
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/)
This seems odd: what if a server has a finite number of disks? I was going to try cephfs
on ~10 servers with 70 HDD each. That would make each system
having to deal with 70 OSDs, on 70 LVs?
Really no aggregation of the disks?
Regards,
Thomas
--
--------------------------------------------------------------------
Thomas Roth
Department: IT
GSI Helmholtzzentrum für Schwerionenforschung GmbH
www.gsi.de
22 Jun
22 Jun
2:42 p.m.
On 22.06.21 11:55, Thomas Roth wrote:
That would make each system
having to deal with 70 OSDs, on 70 LVs?
Yes. And 70 is a rather unusual number of HDDs in a Ceph node.
Normally you have something like 20 to 24 block devices in a single
node. Each OSD needs CPU and RAM.
You could theoretically RAID0 multiple disks and then put an OSD on top
of that but this would create very large OSDs which are not good for
recovering data. Recovering such a "beast" just would take too long.
Regards
--
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 93818 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
2:42 p.m.
Hi,
On 22.06.21 11:55, Thomas Roth wrote:
Hi all,
newbie question:
The documentation seems to suggest that with ceph-volume, one OSD is
created for each HDD (cf. 4-HDD-example in
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/)
This seems odd: what if a server has a finite number of disks? I was
going to try cephfs on ~10 servers with 70 HDD each. That would make
each system having to deal with 70 OSDs, on 70 LVs?
Really no aggregation of the disks?
The recommendation is one OSD per HDD. And most deployment tools
(ceph-ansible / cephadm) follow this guide. The main reason is keeping
the single failure unit small. Using raid with several HDDs might result
in 10-20 TB of data to be recovered if a single disk fails. The same
goes for large storage boxes; just imagine a complete unrecoverable
failure of a box with 70x 10TB disks. More data to recover results in
longer recovery time, which increases the probability for a second or
(usually fatal) third failure.
If you plan to use other setups you either need to adopt the deployment
tools (should be possible with ceph-ansible, difficult in case of
cephadm), or deploy the OSDs manually.
We use manual deployment and linux software raid with raid level 0 and
two/three disks in larger storage boxes, also due to the memory /CPU
core requirements.
Regards,
Burkhard
3:05 p.m.
Hi,
just an addition:
currentl CEPH releases also include disk monitoring (e.g. SMART and
other health related features). These do not work with raid devices. You
will need external monitoring for your OSD disks.
Regards,
Burkhard
2:45 p.m.
That is the idea, what is wrong with this concept? If you aggregate disks, you still
aggregate 70 disks, and you still be having 70 disks.
Everything you do that ceph can't be aware of creates a potential misinterpretation of
the reality and make ceph act in a way it should not.
-----Original Message-----
Sent: Tuesday, 22 June 2021 11:55
To: ceph-users(a)ceph.io
Subject: [ceph-users] HDD <-> OSDs
Hi all,
newbie question:
The documentation seems to suggest that with ceph-volume, one OSD is
created for each HDD (cf. 4-HDD-example in
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-
ref/)
This seems odd: what if a server has a finite number of disks? I was
going to try cephfs on ~10 servers with 70 HDD each. That would make
each system
having to deal with 70 OSDs, on 70 LVs?
Really no aggregation of the disks?
Regards,
Thomas
--
--------------------------------------------------------------------
Thomas Roth
Department: IT
GSI Helmholtzzentrum für Schwerionenforschung GmbH
www.gsi.de
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:26 p.m.
Thank you all for the clarification!
I just did not grasp the concept before, probably because I am used to those systems that
form a layer on top of the local file system. If ceph does
it all, down to the magnetic platter, all the better.
Cheers
Thomas
On 6/22/21 12:15 PM, Marc wrote:
>
> That is the idea, what is wrong with this concept? If you aggregate disks, you still
aggregate 70 disks, and you still be having 70 disks.
> Everything you do that ceph can't be aware of creates a potential
misinterpretation of the reality and make ceph act in a way it should not.
>
>
>
>> -----Original Message-----
>> Sent: Tuesday, 22 June 2021 11:55
>> To: ceph-users(a)ceph.io
>> Subject: [ceph-users] HDD <-> OSDs
>>
>> Hi all,
>>
>> newbie question:
>>
>> The documentation seems to suggest that with ceph-volume, one OSD is
>> created for each HDD (cf. 4-HDD-example in
>> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-
>> ref/)
>>
>> This seems odd: what if a server has a finite number of disks? I was
>> going to try cephfs on ~10 servers with 70 HDD each. That would make
>> each system
>> having to deal with 70 OSDs, on 70 LVs?
>>
>> Really no aggregation of the disks?
>>
>>
>> Regards,
>> Thomas
>> --
>> --------------------------------------------------------------------
>> Thomas Roth
>> Department: IT
>>
>> GSI Helmholtzzentrum für Schwerionenforschung GmbH
>> www.gsi.de
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
3:55 p.m.
hi thomas, just a quick note. If you have a few large osds, ceph will
have problems distributing the data based on the number of placement
groups and the number of objects per placement group, ...
I recommend reading the concept of placement groups.
___________________________________
Clyso GmbH - Ceph Foundation Member
support(a)clyso.com
https://www.clyso.com
Am 22.06.2021 um 12:56 schrieb Thomas Roth:
> Thank you all for the clarification!
>
> I just did not grasp the concept before, probably because I am used to
> those systems that form a layer on top of the local file system. If
> ceph does it all, down to the magnetic platter, all the better.
>
> Cheers
> Thomas
>
> On 6/22/21 12:15 PM, Marc wrote:
>>
>> That is the idea, what is wrong with this concept? If you aggregate
>> disks, you still aggregate 70 disks, and you still be having 70 disks.
>> Everything you do that ceph can't be aware of creates a potential
>> misinterpretation of the reality and make ceph act in a way it should
>> not.
>>
>>
>>
>>> -----Original Message-----
>>> Sent: Tuesday, 22 June 2021 11:55
>>> To: ceph-users(a)ceph.io
>>> Subject: [ceph-users] HDD <-> OSDs
>>>
>>> Hi all,
>>>
>>> newbie question:
>>>
>>> The documentation seems to suggest that with ceph-volume, one OSD is
>>> created for each HDD (cf. 4-HDD-example in
>>> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-
>>> ref/)
>>>
>>> This seems odd: what if a server has a finite number of disks? I was
>>> going to try cephfs on ~10 servers with 70 HDD each. That would make
>>> each system
>>> having to deal with 70 OSDs, on 70 LVs?
>>>
>>> Really no aggregation of the disks?
>>>
>>>
>>> Regards,
>>> Thomas
>>> --
>>> --------------------------------------------------------------------
>>> Thomas Roth
>>> Department: IT
>>>
>>> GSI Helmholtzzentrum für Schwerionenforschung GmbH
>>> www.gsi.de
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:44 p.m.
Den tis 22 juni 2021 kl 11:55 skrev Thomas Roth <t.roth(a)gsi.de>de>:
Hi all,
newbie question:
The documentation seems to suggest that with ceph-volume, one OSD is created for each HDD
(cf. 4-HDD-example in
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/)
This seems odd: what if a server has a finite number of disks? I was going to try cephfs
on ~10 servers with 70 HDD each. That would make each system
having to deal with 70 OSDs, on 70 LVs?
Really no aggregation of the disks?
There is nothing inherently wrong with having 70 OSDs if you have 70
drives. As others have said, it brings benefits to allow single faults
to be kept do as small a domain as possible and if you read up on ZFS
guides, I am sure they will say the same thing, hand over each
separate raw device over to zfs, and let zfs place the data in the
optimal way for each raw device, instead of globbing together the lot
and pretend it is one large disks when it actually isn't.
With one huge 70-piece raid0 you might end up having a lot of writes
and reads hit a small number of disks at a time which probably will be
worse for your overall performance.
--
May the most significant bit of your life be positive.
1057
days inactive
1057
days old
8 comments
7 participants
participants (7)
-
Burkhard Linke -
Clyso GmbH - Ceph Foundation Member -
Janne Johansson -
Konstantin Shalygin -
Marc -
Robert Sander -
Thomas Roth