Hello Nathan,
I'm going to deploy a new cluster soon based on 6.4TB NVME PCI-E Cards, I will have only 1 NVME card per node and 38 nodes.
The use case is to offer cephfs volumes for a k8s platform, I plan to use an EC-POOL 8+3 for the cephfs_data pool.
Do you have recommendations for the setup or mistakes to avoid? I use ceph-ansible to deploy all myclusters.
In order to get optimal performance out of NVMe, you will want very fast cores, and you will probably have to split each NVMe card into 2-4 OSD partitions in order to throw enough cores at it.
That's a good idea ! If I have enough time, I'll try to do some benchmark with 2 and 4 OSD partitions.
I’ve been trying unsuccessfully to convince some folks of the need for fast cores, there’s the idea that the effect would be slight. Do you have any numbers? I’ve also read a claim that each BlueStore will use 3-4 cores, They’re listening to me though about splitting the card into multiple OSDs.
Bluestore will use about 4 cores, but in my experience, the maximum utilization I've seen has been something like: 100%, 100%, 50%, 50%
So those first 2 cores are the bottleneck for pure OSD IOPS. This sort of pattern isn't uncommon in multithreaded programs. This was on HDD OSDs with DB/WAL on NVMe, as well as some small metadata OSDs on pure NVMe. SSD OSDs default to 2 threads per shard, and HDD to 1, but we had to set HDD to 2 as well when we enabled NVMe WAL/DB. Otherwise the OSDs ran out of CPU and failed to heartbeat when under load. I believe that if we had 50% faster cores, we might not have needed to do this.
On SSDs/NVMe you can compensate for slower cores with more OSDs, but of course only for parallel operations. Anything that is serial+synchronous, not so much. I would expect something like 4 OSDs per NVMe, 4 cores per OSD. That's already 16 cores per node just for OSDs.
Our bottleneck in practice is the Ceph MDS, which seems to use exactly 2 cores and has no setting to change this. As far as I can tell, if we had 50% faster cores just for the MDS, I would expect roughly +50% performance in terms of metadata ops/second. Each filesystem has it's own rank-0 MDS, so this load will be split across daemons. The MDS can also use a ton of RAM (32GB) if the clients have a working set of 1 million+ files. Multi-mds exists to further split the load, but is quite new and I would not trust it. CephFS in general is likely where you will have the most issues, as it both new and complex compared to a simple object store. Having an MDS in standby-replay mode keeps it's RAM cache synced with the active, so you get far faster failover ( O(seconds) rather than O(minutes) with a few million file caps) but you use the same RAM again.
So, IMHO, you will want at least: CPU: 16 cores per 1-card NVMe OSD node. 2 cores per filesystem (maybe 1 if you don't expect a lot of simultaneous load?)
RAM: The Bluestore default is 4GB per OSD, so 16GB per node. ~32GB of RAM per active and standby-replay MDS if you expect file counts in the millions, so 64GB per filesystem.
The context is 3 Intel Server 1U for MONs/MDSs/MGRs services + K8s daemons CPU : 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (24c/48t) Memory : 64GB Disk OS : 2x Intel SSD DC S3520 240GB 38 Dell C4140 1U for OSD nodes : CPU : 2 x Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz (28c/56t) Memory : 384GB GPU : 4 Nvidia V100 32GB NVLink Disk OS : M.2 240G NVME : Dell 6.4TB NVME PCI-E Drive (Samsung PM1725b), only 1 slot available Each server is used in a k8s cluster to give access to GPUs and CPUs for X-learning labs. Ceph have to share the CPU and memory with the compute K8s cluster.
128GB of RAM per node ought to do, if you have less than 14 filesystems?
I plan to have only 1 filesystem. Thanks to all those useful information. Best regards, -- Yoann Moulin EPFL IC-IT