Well heck, you’re good to go unless these are converged with compute. 168 threads / 16 OSDs = ~10 threads per OSD, with some left over for OS, observability, etc. You’re more than good. Suggest using BIOS settings and TuneD to disable deep C-states, verify with `power top`. Increased cooling, performance thermal profile. Disable IOMMU in GRUB defaults, tune the TCP stack, somaxconn, nf_conntrack, etc. 768GB is more than ample. For 16 OSDs I would nominally spec 192GB. I might almost split the 30TB SSDs into 2x OSDs each to gain even more parallelism.

On Jan 24, 2025, at 10:34 AM, Bruno Gomes Pessanha <bruno.pessanha@gmail.com> wrote:

ram: 768GB cpu: AMD EPYC 9634 84-Core

On Fri, 24 Jan 2025 at 15:48, Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>> wrote:

...

It’s difficult to fully answer your question with the information provided. Notably, your networking setup and the RAM / CPU SKUs are important inputs.

Assuming that the hosts have or would have sufficient CPU and RAM for the additional OSDs there wouldn’t necessarily be a downside, though you might wish to use a gradual balancing strategy.

The new drives are double the size of the old, so unless you take steps they will get double the PGs and thus double the workload of the existing drives. But since you aren’t subject to the SATA bottleneck, unless your hosts are PCI Gen 3 and your networking insufficient, I suspect that you’ll be fine.

You could use a custom device class and CRUSH rule to segregate the larger/faster drives into their own pool(s), but if you’re adding capacity for existing use-cases, I’d probably just go for it and celebrate the awesome hardware.

...

On Jan 24, 2025, at 9:35 AM, Bruno Gomes Pessanha <bruno.pessanha@gmail.com <mailto:bruno.pessanha@gmail.com>> wrote:

I have a Ceph Reef cluster with 10 hosts with 16 nvme slots but only half occupied with 15TB (2400 KIOPS) drives. 80 drives in total. I want to add another 80 to fully populate the slots. The question: What would be the downside if I expand the cluster with 80 x 30TB (3300 KIOPS) drives?

Thank you!

Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>

-- Bruno Gomes Pessanha

I reckon that balancing is by far the biggest issue you are likely to have because most Ceph releases (I do not know about Reef) have difficulty balancing across drives of different sizes even with configuration changes.

There were some bugs around Firefly-Hammer with failure domains having very different aggregate weights, but with some forethought I’m not aware of recent situations where CRUSH fails in this scenario. The pg autoscaler and balancer module may have difficulty with complex CRUSH topology, but I doubt the OP has such, and if so the JJ Balancer is reputed to work well. Now, it is *ideal* to have and OSD monoculture, but Ceph does pretty well with homogenity. Say one is doing 3x replication and has 3 failure domains, to keep it simple we’ll say 3 hosts. If those *host* CRUSH buckets have aggregate weights like 100TB, 100TB, and 200TB, then the usable raw capacity will be 100TB †, because CRUSH has to place one copy of data on each host, and once the smaller two hosts are full, game over. Now say the larger and smaller drives and thus OSDs are spread more or less evenly across the hosts, so that all three have ~133TB aggregate weights. All raw capacity can be used. This is one reason why it is advantageous when feasible to have at least one more failure domain than demanded by replication policy, so that Ceph can do the right thing. Say we have 4 hosts now, 100TB, 100TB, 100TB, and 150TB, Ceph will be able to use most or all of the raw capacity. With however 100TB, 100TB, 100TB, and 1000TB, that massive variance in failure domain weight probably would prevent all of the heaviest failure domain from being usable. Note that the OP describes 10 hosts, only half populated with 15TB OSDs today. With 10 hosts, the failure domain for CRUSH rules is most likely *host*, so adding 8x 30TB OSDs to each results in all failure domains being equal in weight. Ceph will be able to use all of the raw capacity. The larger OSDs (and thus their drives) will naturally receive approximately double the number of PGs compared to the smaller OSDs. Thus those drives will be ~ twice as busy. With NVMe that probably isn’t an issue, especially if the hosts are PCI Gen 4 or later, and adequate RAM is available. The 30TB SSDs almost certainly are Gen 4 or later.

* Assign different CRUSH weights. This configuration change is "supposed" to work.

Short-stroking? Sure it’ll work, but you’d waste 1.2PB of raw capacity, so that isn’t a great solution.

* Assign the 30TB drives to a different class and use them for new "pools".

Very possible, but probably not necessary, unless say one of the drive models is TLC and the other is QLC, in which case one may wish to segregate the workloads with pools.

* Split each 30TB drive into two OSDs. Not a good idea for HDDs of course but these are low latency SSDs.

You could do that. If the number of OSDs and hosts were very low this might have a certain appeal - I’ve done that myself. In the OP’s case, I think that wouldn’t accomplish much other than using more RAM.

The main other problem with large capacity OSDs is the size of PGs, which can become very large with the default targets numbers of PGs, and a previous commenter mentioned that.

There are enough failure domains here that this wouldn’t be a showstopper, especially if pg_nums and/or the autoscaler’s target are raised to like 200-400.

I think that the current configuration style where one sets the number of PGs rather than the size of PGs leads people astray.

Ceph places PGs based on CRUSH weight, so as long as pg_num for a given pool is a power of two, and there are a halfway decent number of OSDs — which in this case is true — the above strategies would seem roughly equivalent.

In general my impression is that current Ceph defaults and its very design (a single level of grouping: PGs) were meant to be used with OSDs at most 1TB in size and larger OSDs are anyhow not a good idea,

I don’t follow, I know of no intrinsic issue with larger OSDs. Were one to mix, say, 122TB OSDs and 1TB OSDs, or even 30TB OSDs and 1TB OSDs the imbalance could be detrimental to performance and one would need to pay close attention to the aforementioned overdose guardrails.

but of course there are many people who know better, and good luck to them. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

† modulo base 2 vs 10, backfill/full ratios, etc.