HDD replacements with shared NVME lvm for DB - what am I missing here
My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per NVME). It's all using ceph orch, containerized setup, using LVMs, so it's probably the most conventional HDD based setup one can do. I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0 But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup: ceph orch apply -i osd_spec_osd02.yml --dry-run ... ################ OSDSPEC PREVIEWS ################ +---------+----------+-------------+---------------------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+----------+-------------+---------------------+----+-----+ |osd |osd_spec |mimer-osd02 |/dev/mapper/mpathau |- |- | +---------+----------+-------------+---------------------+----+-----+ which is the worst outcome, so I have to set all specs to be unmanaged, since otherwise they do the wrong thing automatically. So, most of https://docs.ceph.com/en/squid/cephadm/services/osd/ can just be ignored, since it won't work. Instead I'm stuck with this long complicated procedure, ensuring all specs are unmanaged, recreate the db lvm manually making up a UUID for it so that it matches the rest of the osds, then manually enter a shell, copy over the client.bootstrap-osd keyring, run this huge ceph-volume lvm prepare --bluestore --no-systemd --osd-id 12345 --data /dev/mapper/mpathxx --block.db /dev/ceph-2e401d48-931b-4529-88c0-d36424560xxx/osd-db-61264cc4-0f40-458b-a4d1-08709b919xxx manually adding the daemon, starting it. I don't love this procedure and it's hard to explain all the things that can go wrong to new staff. I really miss the basic step by step of how to perform the most common operation; replacing a hdd with a ssd db partition. I have tried things with --replace and --no-destroy (a flag that isn't documented at all), but I can't understand from the documentation what it wants from me when I have separate DB partitions. -------- A bit of a tangent, butI also don't understand "REJECT REASONS" from ceph orch device ls Insufficient space (<10 extents) on vgs, LVM detected Why even bother looking at extents on VGs if the presence of LVM by itself already rejects the drive. In my case, it does detect that nvme0n1 has sufficient space on the vgs: mimer-osd02 /dev/mapper/mpathau hdd 12.7T Yes 10m ago mimer-osd02 /dev/nvme0n1 ssd KCM61VUL800G_7170A009TM38 745G No 10m ago Has a FileSystem, LVM detected mimer-osd02 /dev/nvme1n1 ssd KCM61VUL800G_7170A004TM38 745G No 10m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected mimer-osd02 /dev/nvme2n1 ssd KCM61VUL800G_7170A007TM38 745G No 10m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected but, "LVM detected" already rejects it. Checking the code (src/ceph-volume/ceph_volume/util_device.py): self.available_lvm, self.rejected_reasons_lvm = self._check_lvm_reject_reasons() self.available_raw, self.rejected_reasons_raw = self._check_raw_reject_reasons() self.available = self.available_lvm and self.available_raw self.rejected_reasons = list(set(self.rejected_reasons_lvm + self.rejected_reasons_raw)) So apparently, the device must be available both as RAW (i.e. have no vgs) and also have enough space on its LVMs (i.e. have enough vgs). This code doesn't make any sense at all. In addition, is also checks disk.blkdid for any TYPE field @property def has_fs(self) -> bool: self.load_blkid_api() return 'TYPE' in self.blkid_api and for a device in lvm it would have something like {'UUID': '3OWtic-3RCI-G3g5-7M6D-pKpb-eMJe-X0zgif', 'TYPE': 'LVM2_member'} so again it doesn't make any sense to ever have _check_lvm_reject_reasons, since it must never be a LVM2_member anyway. Best regards, Mikael
Hi, Am 04.11.25 um 2:01 PM schrieb Mikael Öhman:
My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per NVME). It's all using ceph orch, containerized setup, using LVMs, so it's probably the most conventional HDD based setup one can do.
I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0
But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup:
Ceph 19 has this bug: https://tracker.ceph.com/issues/72696 Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Thanks Robert, I will keep an eye on the issue I never actually had any success with this even back when running ceph 18 either (actually, i think I saw the same with 17). It's very possible that I was simply doing something wrong back then. E.g. I just noticed that ceph orch device ls also won't find any of my NVMEs at all if they happen to be picked up by multipath. I.e. these won't even show up in ceph orch device ls at all (lsblk): nvme0n1 259:0 0 745.2G 0 disk └─mpathar 253:82 0 745.2G 0 mpath ├─ceph--1b309b1e--a4a6--4861--b16c--7c06ecde1a3d-osd--db--82b8fd01--ede3--4c37--a145--f2bea1cf8f28 253:88 0 53.2G 0 lvm ... but after i blacklist the nvmes from multipathd, they starts showing up in device ls (but still won't be used by orch due to the current issues): nvme0n1 259:0 0 745.2G 0 disk ├─ceph--419bd62d--8cd0--4ff1--9b29--f87c9c6aae8b-osd--db--a9967578--3127--4506--a8b8--182621b4f179 253:29 0 53.2G 0 lvm Maybe that was what prevented the nvme from being picked up before 🤔 On Tue, Nov 4, 2025 at 2:38 PM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per NVME). It's all using ceph orch, containerized setup, using LVMs, so it's
Am 04.11.25 um 2:01 PM schrieb Mikael Öhman: probably
the most conventional HDD based setup one can do.
I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0
But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup:
Ceph 19 has this bug:
https://tracker.ceph.com/issues/72696
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We have had success using this spec format, not sure if this is helpful but we specify the block_db_size and use ceph-volume and LVM to achieve this. service_type: osd service_id: noncolocated_hdd service_name: osd.noncolocated_hdd spec: block_db_size: 250G crush_device_class: hdd data_devices: rotational: 1 size: 5.46T db_devices: rotational: 0 size: 3.49T filter_logic: AND objectstore: bluestore -- Regards, Nikhil Mitra Cisco Confidential From: Mikael Öhman <micketeer@gmail.com> Date: Tuesday, November 4, 2025 at 10:31 AM To: Robert Sander <r.sander@heinlein-support.de> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: HDD replacements with shared NVME lvm for DB - what am I missing here Thanks Robert, I will keep an eye on the issue I never actually had any success with this even back when running ceph 18 either (actually, i think I saw the same with 17). It's very possible that I was simply doing something wrong back then. E.g. I just noticed that ceph orch device ls also won't find any of my NVMEs at all if they happen to be picked up by multipath. I.e. these won't even show up in ceph orch device ls at all (lsblk): nvme0n1 259:0 0 745.2G 0 disk └─mpathar 253:82 0 745.2G 0 mpath ├─ceph--1b309b1e--a4a6--4861--b16c--7c06ecde1a3d-osd--db--82b8fd01--ede3--4c37--a145--f2bea1cf8f28 253:88 0 53.2G 0 lvm ... but after i blacklist the nvmes from multipathd, they starts showing up in device ls (but still won't be used by orch due to the current issues): nvme0n1 259:0 0 745.2G 0 disk ├─ceph--419bd62d--8cd0--4ff1--9b29--f87c9c6aae8b-osd--db--a9967578--3127--4506--a8b8--182621b4f179 253:29 0 53.2G 0 lvm Maybe that was what prevented the nvme from being picked up before 🤔 On Tue, Nov 4, 2025 at 2:38 PM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per NVME). It's all using ceph orch, containerized setup, using LVMs, so it's
Am 04.11.25 um 2:01 PM schrieb Mikael Öhman: probably
the most conventional HDD based setup one can do.
I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0
But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup:
Ceph 19 has this bug:
https://tracker.ceph.com/issues/72696
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Nikhil (and also Torkil who wrote a similar suggestion to me directly) Thanks for the suggestions I tried out adding the filter_logic and sizes to see if i could nudge it to the correct setup, e.g. --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: block_db_size: 50000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 50G:700G filter_logic: AND Just adding "filter_logic: AND" changed nothing; it still wants to only use the DATA device. If i add the size and block_db_size, then it still continues to ignore the available space on the NVME, and then just ignores the specification and does this madness instead: (mpathau is the slow HDD): ################ OSDSPEC PREVIEWS ################ +---------+----------+-------------+---------------------+---------------------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+----------+-------------+---------------------+---------------------+-----+ |osd |osd_spec |mimer-osd02 |/dev/mapper/mpathau |/dev/mapper/mpathau |- | +---------+----------+-------------+---------------------+---------------------+-----+ Yikes! That's terrible.. Maybe the issue that Robert linked is blocking the NVME from being used (I am on ceph 19 here and the symptoms fit exactly what Robert described, if i wipe the entire NVME it probably works (it did last i time i tried)), but it's also quite frustrating that it so happily ignores the specification in all these scenarios. Specification is treated more like a recommendation On Tue, Nov 4, 2025 at 5:51 PM Nikhil Mitra (nikmitra) <nikmitra@cisco.com> wrote:
We have had success using this spec format, not sure if this is helpful but we specify the block_db_size and use ceph-volume and LVM to achieve this.
service_type: osd service_id: noncolocated_hdd service_name: osd.noncolocated_hdd spec: block_db_size: 250G crush_device_class: hdd data_devices: rotational: 1 size: 5.46T db_devices: rotational: 0 size: 3.49T filter_logic: AND objectstore: bluestore
--
Regards,
Nikhil Mitra
* Cisco Confidential From: *Mikael Öhman <micketeer@gmail.com> *Date: *Tuesday, November 4, 2025 at 10:31 AM *To: *Robert Sander <r.sander@heinlein-support.de> *Cc: *ceph-users@ceph.io <ceph-users@ceph.io> *Subject: *[ceph-users] Re: HDD replacements with shared NVME lvm for DB - what am I missing here
Thanks Robert, I will keep an eye on the issue I never actually had any success with this even back when running ceph 18 either (actually, i think I saw the same with 17). It's very possible that I was simply doing something wrong back then. E.g. I just noticed that ceph orch device ls also won't find any of my NVMEs at all if they happen to be picked up by multipath. I.e. these won't even show up in ceph orch device ls at all (lsblk): nvme0n1
259:0 0 745.2G 0 disk └─mpathar
253:82 0 745.2G 0 mpath
├─ceph--1b309b1e--a4a6--4861--b16c--7c06ecde1a3d-osd--db--82b8fd01--ede3--4c37--a145--f2bea1cf8f28 253:88 0 53.2G 0 lvm ... but after i blacklist the nvmes from multipathd, they starts showing up in device ls (but still won't be used by orch due to the current issues): nvme0n1
259:0 0 745.2G 0 disk
├─ceph--419bd62d--8cd0--4ff1--9b29--f87c9c6aae8b-osd--db--a9967578--3127--4506--a8b8--182621b4f179 253:29 0 53.2G 0 lvm
Maybe that was what prevented the nvme from being picked up before 🤔
On Tue, Nov 4, 2025 at 2:38 PM Robert Sander <r.sander@heinlein-support.de
wrote:
Hi,
My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs
NVME). It's all using ceph orch, containerized setup, using LVMs, so it's
Am 04.11.25 um 2:01 PM schrieb Mikael Öhman: per probably
the most conventional HDD based setup one can do.
I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0
But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup:
Ceph 19 has this bug:
https://tracker.ceph.com/issues/72696
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is the NVMe (DB) drive also have other data on it ? Or is it an empty drive dedicated for the DB ? What does ceph orch device ls <hostname> output give you for that device ? -- Regards, Nikhil Mitra Cisco Confidential From: Mikael Öhman <micketeer@gmail.com> Date: Tuesday, November 4, 2025 at 11:57 AM To: Nikhil Mitra (nikmitra) <nikmitra@cisco.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: HDD replacements with shared NVME lvm for DB - what am I missing here Hi Nikhil (and also Torkil who wrote a similar suggestion to me directly) Thanks for the suggestions I tried out adding the filter_logic and sizes to see if i could nudge it to the correct setup, e.g. --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: block_db_size: 50000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 50G:700G filter_logic: AND Just adding "filter_logic: AND" changed nothing; it still wants to only use the DATA device. If i add the size and block_db_size, then it still continues to ignore the available space on the NVME, and then just ignores the specification and does this madness instead: (mpathau is the slow HDD): ################ OSDSPEC PREVIEWS ################ +---------+----------+-------------+---------------------+---------------------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+----------+-------------+---------------------+---------------------+-----+ |osd |osd_spec |mimer-osd02 |/dev/mapper/mpathau |/dev/mapper/mpathau |- | +---------+----------+-------------+---------------------+---------------------+-----+ Yikes! That's terrible.. Maybe the issue that Robert linked is blocking the NVME from being used (I am on ceph 19 here and the symptoms fit exactly what Robert described, if i wipe the entire NVME it probably works (it did last i time i tried)), but it's also quite frustrating that it so happily ignores the specification in all these scenarios. Specification is treated more like a recommendation On Tue, Nov 4, 2025 at 5:51 PM Nikhil Mitra (nikmitra) <nikmitra@cisco.com<mailto:nikmitra@cisco.com>> wrote: We have had success using this spec format, not sure if this is helpful but we specify the block_db_size and use ceph-volume and LVM to achieve this. service_type: osd service_id: noncolocated_hdd service_name: osd.noncolocated_hdd spec: block_db_size: 250G crush_device_class: hdd data_devices: rotational: 1 size: 5.46T db_devices: rotational: 0 size: 3.49T filter_logic: AND objectstore: bluestore -- Regards, Nikhil Mitra Cisco Confidential From: Mikael Öhman <micketeer@gmail.com<mailto:micketeer@gmail.com>> Date: Tuesday, November 4, 2025 at 10:31 AM To: Robert Sander <r.sander@heinlein-support.de<mailto:r.sander@heinlein-support.de>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> <ceph-users@ceph.io<mailto:ceph-users@ceph.io>> Subject: [ceph-users] Re: HDD replacements with shared NVME lvm for DB - what am I missing here Thanks Robert, I will keep an eye on the issue I never actually had any success with this even back when running ceph 18 either (actually, i think I saw the same with 17). It's very possible that I was simply doing something wrong back then. E.g. I just noticed that ceph orch device ls also won't find any of my NVMEs at all if they happen to be picked up by multipath. I.e. these won't even show up in ceph orch device ls at all (lsblk): nvme0n1 259:0 0 745.2G 0 disk └─mpathar 253:82 0 745.2G 0 mpath ├─ceph--1b309b1e--a4a6--4861--b16c--7c06ecde1a3d-osd--db--82b8fd01--ede3--4c37--a145--f2bea1cf8f28 253:88 0 53.2G 0 lvm ... but after i blacklist the nvmes from multipathd, they starts showing up in device ls (but still won't be used by orch due to the current issues): nvme0n1 259:0 0 745.2G 0 disk ├─ceph--419bd62d--8cd0--4ff1--9b29--f87c9c6aae8b-osd--db--a9967578--3127--4506--a8b8--182621b4f179 253:29 0 53.2G 0 lvm Maybe that was what prevented the nvme from being picked up before 🤔 On Tue, Nov 4, 2025 at 2:38 PM Robert Sander <r.sander@heinlein-support.de<mailto:r.sander@heinlein-support.de>> wrote:
Hi,
My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per NVME). It's all using ceph orch, containerized setup, using LVMs, so it's
Am 04.11.25 um 2:01 PM schrieb Mikael Öhman: probably
the most conventional HDD based setup one can do.
I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0
But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup:
Ceph 19 has this bug:
https://tracker.ceph.com/issues/72696
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
The NVME has nothing else other than the 14 LVM partitions (all created by ceph orch when it was originally deployed). So it's not empty now; it has the other 13 lvms on it for the other working OSDs, but it has ~50 GB left which won't ever get reused (maybe due to the bug that Roberto mentioned). I included the example of ceph orch device ls in the first mail, the entire second section, and it reports the expected "LVM detected" and "Has a FileSystem" (which is due to it being a "LVM2_Member"), as expected given the code, but I think the code doesn't make any sense.
The NVME has nothing else other than the 14 LVM partitions (all created by ceph orch when it was originally deployed). So it's not empty now; it has the other 13 lvms on it for the other working OSDs, but it has ~50 GB left which won't ever get reused
Not strictly true. All SSDs have some degree of overprovisioning: physical NAND capacity that is not exposed to the host, but is used for sparing and other housekeeping. The difference between read-intensive and mixed-use SKUs is often simply that a greater percentage is thus reserved. So you can think of that 50GB as getting you longer endurance.
participants (4)
-
Anthony D'Atri
-
Mikael Öhman
-
Nikhil Mitra (nikmitra)
-
Robert Sander