I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes.
Ensure the firmware is up to date on each!
My initial plan was to split the NVMes into 5x 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 HDDs creating hybrid devices). My hope was that for performance, the one NVMe data partition could then be used for 4x replicated SSD-only metadata pools.
What SSD and HDD SKUs exactly please?
The hybrid devices would be for 4+2 EC cephfs.
Depending on your workload, you might find that the WAL+DB offload doesn't get you much.
Previously I had tested this successfully on Squid using a series of manual device adds, but on Tentacle it is seeing pre-created LVM partitions as in use by the file system where Squid did not (ceph orch device ls ceph01).
My understanding is a manual device add approach, while possible, removes the benefits of orch (creating "unmanaged" devices) and should be avoided. But would it even be possible for orch to be fed a yaml with the NVMes operating as both a data and db device?
If you specify the paths explicitly, that should work.
The obvious answer is to have a third NVMe per node operating as data for the 4x replicated metadata pool, but I am not sure if my budget permits that.
What's your file workload like? You might consider making the OSD partition larger and the WAL+DB offload partitions smaller.