Re: How big an OSD disk could be?
Den fre 12 mars 2021 kl 18:10 skrev huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn>:
Dear cephers, Just wonder how big an OSD disk could be? Currently the biggest HDD has a capacity of 18TB or 20TB. It is suitable for an OSD still? Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB or 100TB a single OSD? What could be the potential limit?
I think your patience will be the limit. If you stick 8x or 10x 20TB drives in a box, it will take a very long time to fill them to say, 50%. That would be some 80-100TB to move over the network and a lot of IO spread over only 8-10 drives before it settles, which would be a long time on spinning drives. Unless the objects are huge, 100TB of files/objects would mean many many millions of writes just to fill the OSD and its list of contents. For spinners, it would mean many many millions of head movements just to write it all out, hopefully never encountering a random bad block. After you have filled that up, if such a host crashes or needs maintenance, another 80-100TB will need recreating from the other huge drives. I don't think there are specific limitations on the size itself, but as the single drive becomes larger and larger, just adding a new host or a drive will mean the cluster is rebalancing for days or weeks if not more. At some point you would end up having the cluster almost never in HEALTH_OK state because of normal replacements, expansions and other surprises, which in turn could cause secondary problems with mon DBs and things like that. -- May the most significant bit of your life be positive.
Thanks a lot for the insightful comments. huxiaoyu@horebdata.cn From: Janne Johansson Date: 2021-03-13 11:36 To: huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] How big an OSD disk could be? Den fre 12 mars 2021 kl 18:10 skrev huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn>:
Dear cephers, Just wonder how big an OSD disk could be? Currently the biggest HDD has a capacity of 18TB or 20TB. It is suitable for an OSD still? Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB or 100TB a single OSD? What could be the potential limit?
I think your patience will be the limit. If you stick 8x or 10x 20TB drives in a box, it will take a very long time to fill them to say, 50%. That would be some 80-100TB to move over the network and a lot of IO spread over only 8-10 drives before it settles, which would be a long time on spinning drives. Unless the objects are huge, 100TB of files/objects would mean many many millions of writes just to fill the OSD and its list of contents. For spinners, it would mean many many millions of head movements just to write it all out, hopefully never encountering a random bad block. After you have filled that up, if such a host crashes or needs maintenance, another 80-100TB will need recreating from the other huge drives. I don't think there are specific limitations on the size itself, but as the single drive becomes larger and larger, just adding a new host or a drive will mean the cluster is rebalancing for days or weeks if not more. At some point you would end up having the cluster almost never in HEALTH_OK state because of normal replacements, expansions and other surprises, which in turn could cause secondary problems with mon DBs and things like that. -- May the most significant bit of your life be positive.
After you have filled that up, if such a host crashes or needs maintenance, another 80-100TB will need recreating from the other huge drives.
A judicious setting of mon_osd_down_out_subtree_limit can help mitigate the thundering herd FWIW.
I don't think there are specific limitations on the size itself, but as the single drive becomes larger and larger, just adding a new host or a drive will mean the cluster is rebalancing for days or weeks if not more.
Especially if EC is used. A corollary here is that IOPS/TB decrease as HDDs grow larger. We see some incremental tweaks, but in the end the interface speed hasn’t grown in some time. Seek and rotational latency are helped somewhat by increasing areal density, though capacity growth is also acheived by making the platters increasingly thinner and more numerous: recent drives pack as many as 9 in there (perhaps fewer for SMR models). I’ve seen scale deployments cap HDD size at, say, 8TB because the IOPS/TB beyond was increasingly untenable, depending of course on the use-case.
At some point you would end up having the cluster almost never in HEALTH_OK state because of normal replacements, expansions and other surprises
With recent releases backfill doesn’t trigger HEALTH_WARN, though, right?
which in turn could cause secondary problems with mon DBs and things like that.
Your point is well made, though — Dan @ CERN observed several years ago that with a sufficently large cluster one has to come to terms with backfill going on all the time. The idea here is that mon DB compaction tends to block if there is any degradation — with at least some releases, that means even if you have HEALTH_OK, but some OSDs are down/out.
participants (3)
-
Anthony D'Atri
-
huxiaoyu＠horebdata.cn
-
Janne Johansson