Hey Anthony, The NVMEs are Micron 7500 Pro with 7.6TB space (MTFDKCC7T6TGP-1BK1DABYYR). We haven't exactly aimed for multipath disks. Lets say that they were ordered by „accident“ ;) Nevertheless, I was able to pinpoint the issue. It also has something to do with multipath-disks. The issue is described here https://tracker.ceph.com/issues/63862 TL;DR: The inventory command for ceph-volume was not able to list mpath-disks when they were already used as osd. They got falsely filtered out because they got detected as mapper devices with LVs on top. I added a patch-file for this https://tracker.ceph.com/attachments/8006 It appears that this only affects Ceph Reef and older. The code for ceph-volume/util/disk.py has been changed since then. --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

Am 11.03.2026 um 15:43 schrieb Anthony D'Atri via ceph-users <ceph-users@ceph.io>:

...

- 36 data disks, registered as mpath-devices - 2 NVMEs, which act as block.db for all 36 spinning disks.

That’s an unusually high ratio. Which model exactly are the NVMe SSDs? And why are you using multipathing with the HDDs?

...

I hacked a workaround, which removes the last conditional check (occupied_slots < fast_slots_per_device) and hard-coded the expected size of each block.db in my deployment file:

Honestly that’s what I often do, especially when applying the orch service to systems with OSDs that were previously deployed under a different scheme. Hybrid OSDs can be messy.

...

service_type: osd service_id: delta2024_osd service_name: osd.delta2024_osd placement: label: delta2024 spec: block_db_size: 397G data_devices: rotational: 1 size: '15T:' db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore

This gives the expected results. In my opinion, cephadm sends a wrong „ceph-volume lvm batch“ command to the osd-node. It should always include all of the disks, since running it is promised to be idempotent. With the full list of disks, ceph-volume should be able to calculate correct slots for block.db.

Did I find a bug here or is this expected behavior?

Good question. Bitte enter a ticket at tracker.ceph.com <http://tracker.ceph.com/>.

...

--------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907

Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io