Hi Nikhil (and also Torkil who wrote a similar suggestion to me directly) Thanks for the suggestions I tried out adding the filter_logic and sizes to see if i could nudge it to the correct setup, e.g. --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: block_db_size: 50000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 50G:700G filter_logic: AND Just adding "filter_logic: AND" changed nothing; it still wants to only use the DATA device. If i add the size and block_db_size, then it still continues to ignore the available space on the NVME, and then just ignores the specification and does this madness instead: (mpathau is the slow HDD): ################ OSDSPEC PREVIEWS ################ +---------+----------+-------------+---------------------+---------------------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+----------+-------------+---------------------+---------------------+-----+ |osd |osd_spec |mimer-osd02 |/dev/mapper/mpathau |/dev/mapper/mpathau |- | +---------+----------+-------------+---------------------+---------------------+-----+ Yikes! That's terrible.. Maybe the issue that Robert linked is blocking the NVME from being used (I am on ceph 19 here and the symptoms fit exactly what Robert described, if i wipe the entire NVME it probably works (it did last i time i tried)), but it's also quite frustrating that it so happily ignores the specification in all these scenarios. Specification is treated more like a recommendation On Tue, Nov 4, 2025 at 5:51 PM Nikhil Mitra (nikmitra) <nikmitra@cisco.com> wrote:
We have had success using this spec format, not sure if this is helpful but we specify the block_db_size and use ceph-volume and LVM to achieve this.
service_type: osd service_id: noncolocated_hdd service_name: osd.noncolocated_hdd spec: block_db_size: 250G crush_device_class: hdd data_devices: rotational: 1 size: 5.46T db_devices: rotational: 0 size: 3.49T filter_logic: AND objectstore: bluestore
--
Regards,
Nikhil Mitra
* Cisco Confidential From: *Mikael Öhman <micketeer@gmail.com> *Date: *Tuesday, November 4, 2025 at 10:31 AM *To: *Robert Sander <r.sander@heinlein-support.de> *Cc: *ceph-users@ceph.io <ceph-users@ceph.io> *Subject: *[ceph-users] Re: HDD replacements with shared NVME lvm for DB - what am I missing here
Thanks Robert, I will keep an eye on the issue I never actually had any success with this even back when running ceph 18 either (actually, i think I saw the same with 17). It's very possible that I was simply doing something wrong back then. E.g. I just noticed that ceph orch device ls also won't find any of my NVMEs at all if they happen to be picked up by multipath. I.e. these won't even show up in ceph orch device ls at all (lsblk): nvme0n1
259:0 0 745.2G 0 disk └─mpathar
253:82 0 745.2G 0 mpath
├─ceph--1b309b1e--a4a6--4861--b16c--7c06ecde1a3d-osd--db--82b8fd01--ede3--4c37--a145--f2bea1cf8f28 253:88 0 53.2G 0 lvm ... but after i blacklist the nvmes from multipathd, they starts showing up in device ls (but still won't be used by orch due to the current issues): nvme0n1
259:0 0 745.2G 0 disk
├─ceph--419bd62d--8cd0--4ff1--9b29--f87c9c6aae8b-osd--db--a9967578--3127--4506--a8b8--182621b4f179 253:29 0 53.2G 0 lvm
Maybe that was what prevented the nvme from being picked up before 🤔
On Tue, Nov 4, 2025 at 2:38 PM Robert Sander <r.sander@heinlein-support.de
wrote:
Hi,
My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs
NVME). It's all using ceph orch, containerized setup, using LVMs, so it's
Am 04.11.25 um 2:01 PM schrieb Mikael Öhman: per probably
the most conventional HDD based setup one can do.
I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0
But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup:
Ceph 19 has this bug:
https://tracker.ceph.com/issues/72696
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io