Thank you for your explanations and references. I will check them all. In the meantime it turned out that the disks for Ceph will come from SAN

Be prepared for the additional latency and amplified network traffic.

Probably in this case the per OSD CPU cores can be lowered to 2 CPU/OSD. But I will try to find some references for this usecase.

It’s hard to give guidance for such things beyond “it depends”, but it does depend on the CPUs and media in use.

...

I think it depends on the type of cluster you're trying to build.

Indeed. If you’re storing archival data, you might not care that much about throughput or latency. Object storage a bit more, block you usually do care about latency.

...

If made of HDD/SSD OSDs, then 2 cores per OSD is probably still valid.

Back in the Dumpling era the rule of thumb was 1 vcore/thread per OSD. Mind you the performance of a current-gen vcore then vs today are worlds apart, but also then most people were stuck with HDDs. And Ceph itself has gained substantially in terms of performance.

...

I believe the 5-6 cores per OSDs recommendation you mentioned relates to all flash (NVMe) clusters where CPUs and especially memory bandwidth can't always keep up with all flash storage and small IO workloads.

Indeed, Mark and Dan’s journey to 1TB/s presentation is a great resource. Beyond a few cores, you often are into a long tail of diminishing returns. It depends on what you’re solving for.

...

"Ceph can easily utilize five or six cores on real clusters and up to about fourteen cores on single OSDs in isolation" [1]

I think I’m to thank — or blame as you see fit — for that sentence.

...

I'd say 'real clusters' here stands for all flash NVMe clusters with multiples OSDs on multiple hosts with x3 replication

It was indeed intended to mean a production-class cluster with replication / EC, vs a standalone OSD not doing any subops. While I maintain that HDDs are usually a false economy, I was not intending to limit a “real” cluster to non-rotational media. Probably “production-class” would have been a better term.

...

while 'a single OSDs in isolation' refers to a single NVMe OSD, maybe on a single host cluster with no replication at all.

Yes. My understanding of some of Mark’s perf work is that it was against a single OSD with no sub-ops, to test the OSD code without confounding variables. In a production-class cluster, network latency and congestion may easily be dominant factors.

...

But this may need further confirmation. At the end of day, if your intention is to build an all flash cluster then I think you should multiply 5-6 cores by the number of NVMe OSDs and chose CPU accordingly (with the highest frequency you can get for the bucks).

My sense is that as a first-order model, with modern CPUs for NVMe OSDs, perhaps start with 4 vcores / threads per, and more is gravy. If running mons/mgrs, etc. on the same nodes, plan for a few cores for those. After say 5-6 you get into diminishing return territory. Watch out for C-states, and on EPYCs, IOMMU. Basically all the things in Mark and Dan’s excellent presentation.

...

You might want to check Mark's talks [2][3] and studies [4][5][6] about all flash Ceph clusters. It explains it all and suggests some modern hardware for all flash storage, if that's what you're building.

Cheers, Frédéric.

[1]https://github.com/ceph/ceph/pull/44466#discussion_r779650295 [2]https://youtu.be/S2rPA7qlSYY [3]https://youtu.be/pGwwlaCXfzo [4]https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/ [5]https://docs.clyso.com/blog/ceph-a-journey-to-1tibps [6]https://docs.clyso.com/blog/clyso-enterprise-storage-all-flash-ceph-deployme...

----- Le 18 Sep 24, à 10:15, Laszlo Budailaszlo@componentsoft.eu a écrit :

...

Hello everyone,

I'm trying to understand the CPU requirements for recent versions of CEPH. Reading the documentation (https://docs.ceph.com/en/latest/start/hardware-recommendations/) I cannot get any conclusion about how to plan CPUs for ceph. There is the following statement:

"With earlier releases of Ceph, we would make hardware recommendations based on the number of cores per OSD, but this cores-per-osd metric is no longer as useful a metric as the number of cycles per IOP and the number of IOPS per OSD. For example, with NVMe OSD drives, Ceph can easily utilize five or six cores on real clusters and up to about fourteen cores on single OSDs in isolation. So cores per OSD are no longer as pressing a concern as they were. When selecting hardware, select for IOPS per core."

How should I understand this? On one side it's saying "with NVMe OSD drives, Ceph can easily utilize five or six cores on real clusters" and then continues: "and up to about fourteen cores on single OSDs in isolation." so should I cont 5/6 cores per NVMe OSD or 14cores?

And after all that there's the next sentence that says: " So cores per OSD are no longer as pressing a concern as they were.", But if a single OSD can consume 5/6 cpu cures (that could go up to 14) then I would assume that cores per OSD is still an important concern.

Can anyone explain these CPU requirements? Or point me to some other documents that describes in more details the resources required for Ceph?

Thank you, Laszlo _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io