Thank you for taking the time to reply! The drive models are, per node: - 2x 2 TB NVMe Samsung MZ1L21T9HCLS - 4x 20 TB HDD Western Digital WUH722020BLE6L4 - 4x 20 TB HDD Toshiba MG10ACA20TE I had manually split the NVMes in 5x 300 GB partitions (1 for metadata pools, 4 for 4x HDD db hybrid pairings). Adding the devices using orch gave an error, but after my initial email I double-checked and they were added, but without their db pairs: # Adding NVMe/HDD hybrid OSD gets device not found error despite it showing up normally in LVM or lsblk sudo ceph orch daemon add osd ceph01:data_devices=/dev/mapper/ceph--hdd--01-block--02,db_devices=/dev/mapper/ceph--nvme--01-db--02,encrypted=true Error: Device /dev/mapper/ceph--hdd--01-block--02 is not found on host ceph01 # OSD gets added despite the error, but does not have a DB pairing (also obvious when the OSD = 18.2 TB not 18.5 TB) sudo ceph osd tree ... "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "0", "bluefs_single_shared_device": "1", ... This actually does work as expected if the OSDs are created by feeding orch a yaml instead of using it to manually create OSDs so this is a minor issue I worked around. "Depending on your workload, you might find that the WAL+DB offload doesn't get you much." What would you suggest with the NVMes/HDDs I have? I've also considered just splitting the NVMes into 4 partitions for DB+WAL HDD pairings (no NVME partition for an SSD-only metadata pool). Something like the below yaml would be simplest, but skipping the SSD-only metadata pool might be a mistake. The workload is not performance sensitive, the cluster is replacing a normal Samba file share server. Capacity is more of a concern so I was targeting EC 4+2. service_type: osd service_id: hdd-with-nvme-db placement: host_pattern: '*' spec: encrypted: true data_devices: rotational: true db_devices: rotational: false db_slots: 4 On Thu, Mar 19, 2026 at 9:53 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes.
Ensure the firmware is up to date on each!
My initial plan was to split the NVMes into 5x 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 HDDs creating hybrid devices). My hope was that for performance, the one NVMe data partition could then be used for 4x replicated SSD-only metadata pools.
What SSD and HDD SKUs exactly please?
The hybrid devices would be for 4+2 EC cephfs.
Depending on your workload, you might find that the WAL+DB offload doesn't get you much.
Previously I had tested this successfully on Squid using a series of
manual
device adds, but on Tentacle it is seeing pre-created LVM partitions as in use by the file system where Squid did not (ceph orch device ls ceph01).
My understanding is a manual device add approach, while possible, removes the benefits of orch (creating "unmanaged" devices) and should be avoided. But would it even be possible for orch to be fed a yaml with the NVMes operating as both a data and db device?
If you specify the paths explicitly, that should work.
The obvious answer is to have a third NVMe per node operating as data for the 4x replicated metadata pool, but I am not sure if my budget permits that.
What's your file workload like? You might consider making the OSD partition larger and the WAL+DB offload partitions smaller.
-- Ryan Sleeth