>>> I'm going to deploy a new cluster soon based on 6.4TB NVME PCI-E Cards, I will have only 1 NVME card per node and 38 nodes. >>> >>> The use case is to offer cephfs volumes for a k8s platform, I plan to use an EC-POOL 8+3 for the cephfs_data pool. >>> >>> Do you have recommendations for the setup or mistakes to avoid? I use ceph-ansible to deploy all myclusters. >> >> In order to get optimal performance out of NVMe, you will want very >> fast cores, and you will probably have to split each NVMe card into >> 2-4 OSD partitions in order to throw enough cores at it.

Bluestore will use about 4 cores, but in my experience, the maximum utilization I've seen has been something like: 100%, 100%, 50%, 50% So those first 2 cores are the bottleneck for pure OSD IOPS. This sort of pattern isn't uncommon in multithreaded programs. This was on HDD OSDs with DB/WAL on NVMe, as well as some small metadata OSDs on pure NVMe. SSD OSDs default to 2 threads per shard, and HDD to 1, but we had to set HDD to 2 as well when we enabled NVMe WAL/DB. Otherwise the OSDs ran out of CPU and failed to heartbeat when under load. I believe that if we had 50% faster cores, we might not have needed to do this. On SSDs/NVMe you can compensate for slower cores with more OSDs, but of course only for parallel operations. Anything that is serial+synchronous, not so much. I would expect something like 4 OSDs per NVMe, 4 cores per OSD. That's already 16 cores per node just for OSDs. Our bottleneck in practice is the Ceph MDS, which seems to use exactly 2 cores and has no setting to change this. As far as I can tell, if we had 50% faster cores just for the MDS, I would expect roughly +50% performance in terms of metadata ops/second. Each filesystem has it's own rank-0 MDS, so this load will be split across daemons. The MDS can also use a ton of RAM (32GB) if the clients have a working set of 1 million+ files. Multi-mds exists to further split the load, but is quite new and I would not trust it. CephFS in general is likely where you will have the most issues, as it both new and complex compared to a simple object store. Having an MDS in standby-replay mode keeps it's RAM cache synced with the active, so you get far faster failover ( O(seconds) rather than O(minutes) with a few million file caps) but you use the same RAM again. So, IMHO, you will want at least: CPU: 16 cores per 1-card NVMe OSD node. 2 cores per filesystem (maybe 1 if you don't expect a lot of simultaneous load?) RAM: The Bluestore default is 4GB per OSD, so 16GB per node. ~32GB of RAM per active and standby-replay MDS if you expect file counts in the millions, so 64GB per filesystem.

128GB of RAM per node ought to do, if you have less than 14 filesystems?