We have had success using this spec format, not sure if this is helpful but we specify the block_db_size and use ceph-volume and LVM to achieve this. service_type: osd service_id: noncolocated_hdd service_name: osd.noncolocated_hdd spec: block_db_size: 250G crush_device_class: hdd data_devices: rotational: 1 size: 5.46T db_devices: rotational: 0 size: 3.49T filter_logic: AND objectstore: bluestore -- Regards, Nikhil Mitra Cisco Confidential From: Mikael Öhman <micketeer@gmail.com> Date: Tuesday, November 4, 2025 at 10:31 AM To: Robert Sander <r.sander@heinlein-support.de> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: HDD replacements with shared NVME lvm for DB - what am I missing here Thanks Robert, I will keep an eye on the issue I never actually had any success with this even back when running ceph 18 either (actually, i think I saw the same with 17). It's very possible that I was simply doing something wrong back then. E.g. I just noticed that ceph orch device ls also won't find any of my NVMEs at all if they happen to be picked up by multipath. I.e. these won't even show up in ceph orch device ls at all (lsblk): nvme0n1 259:0 0 745.2G 0 disk └─mpathar 253:82 0 745.2G 0 mpath ├─ceph--1b309b1e--a4a6--4861--b16c--7c06ecde1a3d-osd--db--82b8fd01--ede3--4c37--a145--f2bea1cf8f28 253:88 0 53.2G 0 lvm ... but after i blacklist the nvmes from multipathd, they starts showing up in device ls (but still won't be used by orch due to the current issues): nvme0n1 259:0 0 745.2G 0 disk ├─ceph--419bd62d--8cd0--4ff1--9b29--f87c9c6aae8b-osd--db--a9967578--3127--4506--a8b8--182621b4f179 253:29 0 53.2G 0 lvm Maybe that was what prevented the nvme from being picked up before 🤔 On Tue, Nov 4, 2025 at 2:38 PM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per NVME). It's all using ceph orch, containerized setup, using LVMs, so it's
Am 04.11.25 um 2:01 PM schrieb Mikael Öhman: probably
the most conventional HDD based setup one can do.
I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0
But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup:
Ceph 19 has this bug:
https://tracker.ceph.com/issues/72696
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io