Re: Recommended approach for hybrid devices
I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes.
Ensure the firmware is up to date on each!
My initial plan was to split the NVMes into 5x 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 HDDs creating hybrid devices). My hope was that for performance, the one NVMe data partition could then be used for 4x replicated SSD-only metadata pools.
What SSD and HDD SKUs exactly please?
The hybrid devices would be for 4+2 EC cephfs.
Depending on your workload, you might find that the WAL+DB offload doesn't get you much.
Previously I had tested this successfully on Squid using a series of manual device adds, but on Tentacle it is seeing pre-created LVM partitions as in use by the file system where Squid did not (ceph orch device ls ceph01).
My understanding is a manual device add approach, while possible, removes the benefits of orch (creating "unmanaged" devices) and should be avoided. But would it even be possible for orch to be fed a yaml with the NVMes operating as both a data and db device?
If you specify the paths explicitly, that should work.
The obvious answer is to have a third NVMe per node operating as data for the 4x replicated metadata pool, but I am not sure if my budget permits that.
What's your file workload like? You might consider making the OSD partition larger and the WAL+DB offload partitions smaller.
Thank you for taking the time to reply! The drive models are, per node: - 2x 2 TB NVMe Samsung MZ1L21T9HCLS - 4x 20 TB HDD Western Digital WUH722020BLE6L4 - 4x 20 TB HDD Toshiba MG10ACA20TE I had manually split the NVMes in 5x 300 GB partitions (1 for metadata pools, 4 for 4x HDD db hybrid pairings). Adding the devices using orch gave an error, but after my initial email I double-checked and they were added, but without their db pairs: # Adding NVMe/HDD hybrid OSD gets device not found error despite it showing up normally in LVM or lsblk sudo ceph orch daemon add osd ceph01:data_devices=/dev/mapper/ceph--hdd--01-block--02,db_devices=/dev/mapper/ceph--nvme--01-db--02,encrypted=true Error: Device /dev/mapper/ceph--hdd--01-block--02 is not found on host ceph01 # OSD gets added despite the error, but does not have a DB pairing (also obvious when the OSD = 18.2 TB not 18.5 TB) sudo ceph osd tree ... "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "0", "bluefs_single_shared_device": "1", ... This actually does work as expected if the OSDs are created by feeding orch a yaml instead of using it to manually create OSDs so this is a minor issue I worked around. "Depending on your workload, you might find that the WAL+DB offload doesn't get you much." What would you suggest with the NVMes/HDDs I have? I've also considered just splitting the NVMes into 4 partitions for DB+WAL HDD pairings (no NVME partition for an SSD-only metadata pool). Something like the below yaml would be simplest, but skipping the SSD-only metadata pool might be a mistake. The workload is not performance sensitive, the cluster is replacing a normal Samba file share server. Capacity is more of a concern so I was targeting EC 4+2. service_type: osd service_id: hdd-with-nvme-db placement: host_pattern: '*' spec: encrypted: true data_devices: rotational: true db_devices: rotational: false db_slots: 4 On Thu, Mar 19, 2026 at 9:53 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes.
Ensure the firmware is up to date on each!
My initial plan was to split the NVMes into 5x 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 HDDs creating hybrid devices). My hope was that for performance, the one NVMe data partition could then be used for 4x replicated SSD-only metadata pools.
What SSD and HDD SKUs exactly please?
The hybrid devices would be for 4+2 EC cephfs.
Depending on your workload, you might find that the WAL+DB offload doesn't get you much.
Previously I had tested this successfully on Squid using a series of
manual
device adds, but on Tentacle it is seeing pre-created LVM partitions as in use by the file system where Squid did not (ceph orch device ls ceph01).
My understanding is a manual device add approach, while possible, removes the benefits of orch (creating "unmanaged" devices) and should be avoided. But would it even be possible for orch to be fed a yaml with the NVMes operating as both a data and db device?
If you specify the paths explicitly, that should work.
The obvious answer is to have a third NVMe per node operating as data for the 4x replicated metadata pool, but I am not sure if my budget permits that.
What's your file workload like? You might consider making the OSD partition larger and the WAL+DB offload partitions smaller.
-- Ryan Sleeth
Another way to phrase my last question is: if a cluster is setup with hybrid OSDs (HDD + SSD/NVMe partition) and all CephFS data/metadata pools are put on these OSDs is Ceph 'smart enough' to put the metadata workloads on the SSD portions of the hybrid OSDs (assuming they are large enough, for me it would be ~400 GB partition per 20 TB HDD)? On Thu, Mar 19, 2026 at 11:15 PM Ryan Sleeth <crsleeth@gmail.com> wrote:
Thank you for taking the time to reply! The drive models are, per node:
- 2x 2 TB NVMe Samsung MZ1L21T9HCLS - 4x 20 TB HDD Western Digital WUH722020BLE6L4 - 4x 20 TB HDD Toshiba MG10ACA20TE
I had manually split the NVMes in 5x 300 GB partitions (1 for metadata pools, 4 for 4x HDD db hybrid pairings). Adding the devices using orch gave an error, but after my initial email I double-checked and they were added, but without their db pairs:
# Adding NVMe/HDD hybrid OSD gets device not found error despite it showing up normally in LVM or lsblk sudo ceph orch daemon add osd ceph01:data_devices=/dev/mapper/ceph--hdd--01-block--02,db_devices=/dev/mapper/ceph--nvme--01-db--02,encrypted=true Error: Device /dev/mapper/ceph--hdd--01-block--02 is not found on host ceph01 # OSD gets added despite the error, but does not have a DB pairing (also obvious when the OSD = 18.2 TB not 18.5 TB) sudo ceph osd tree ... "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "0", "bluefs_single_shared_device": "1", ...
This actually does work as expected if the OSDs are created by feeding orch a yaml instead of using it to manually create OSDs so this is a minor issue I worked around.
"Depending on your workload, you might find that the WAL+DB offload doesn't get you much." What would you suggest with the NVMes/HDDs I have? I've also considered just splitting the NVMes into 4 partitions for DB+WAL HDD pairings (no NVME partition for an SSD-only metadata pool). Something like the below yaml would be simplest, but skipping the SSD-only metadata pool might be a mistake. The workload is not performance sensitive, the cluster is replacing a normal Samba file share server. Capacity is more of a concern so I was targeting EC 4+2.
service_type: osd service_id: hdd-with-nvme-db placement: host_pattern: '*' spec: encrypted: true data_devices: rotational: true db_devices: rotational: false db_slots: 4
On Thu, Mar 19, 2026 at 9:53 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes.
Ensure the firmware is up to date on each!
My initial plan was to split the NVMes into 5x 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 HDDs creating hybrid devices). My hope was that for performance, the one NVMe data partition could then be used for 4x replicated SSD-only metadata pools.
What SSD and HDD SKUs exactly please?
The hybrid devices would be for 4+2 EC cephfs.
Depending on your workload, you might find that the WAL+DB offload doesn't get you much.
Previously I had tested this successfully on Squid using a series of
manual
device adds, but on Tentacle it is seeing pre-created LVM partitions as in use by the file system where Squid did not (ceph orch device ls ceph01).
My understanding is a manual device add approach, while possible, removes the benefits of orch (creating "unmanaged" devices) and should be avoided. But would it even be possible for orch to be fed a yaml with the NVMes operating as both a data and db device?
If you specify the paths explicitly, that should work.
The obvious answer is to have a third NVMe per node operating as data for the 4x replicated metadata pool, but I am not sure if my budget permits that.
What's your file workload like? You might consider making the OSD partition larger and the WAL+DB offload partitions smaller.
-- Ryan Sleeth
-- Ryan Sleeth
participants (2)
-
Anthony D'Atri
-
Ryan Sleeth