Hey friends, Our production system is still running ceph with manual Debian packages. Since I’d like to migrate to a deployment based on cephadm, I experimented the migration and usage of day 1 ops on our development environment. Both systems run with v18.2.7. The migration of the dev-system went well. The only headache I have is with replacing OSDs (simulating the replacement of a faulty disk). The osd-node has the following setup: - 36 data disks, registered as mpath-devices - 2 NVMEs, which act as block.db for all 36 spinning disks. Therefore each NVME holds 18 LVs. For the deployment I used a very simple file: ``` service_type: osd service_id: delta2024_osd placement: label: delta2024 spec: data_devices: size: '15T:' rotational: 1 db_devices: size: '6T:' rotational: 0 objectstore: bluestore encrypted: true ``` This deploys all osds as expected. On cephadm-logs, I see the „ceph-volume lvm batch“ command, including all 36 disks as arguments. Now I want to simulate the replacement of a failing disk: - Set the deployment to unmanaged via „ceph orch set-unmanaged …“ - Zap one disk with „ceph orch osd rm 56 —zap --replace“ - „vgs“ show, that one of the NVMEs has space again. - „ceph orch device ls —wide —refresh“ shows the zapped disk as available again - Set the deployment to managed again „ceph orch set-managed …“ - I expect, that the OSD gets re-deployed again, but nothing happens. This time, cephadm sends a shorter command to my osd-node: „ceph-volume lvm batch --auto /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 --no-systemd —yes“ This exits with the message, that none of the passed fast-devices are available. After digging in the code for ceph-volume, I may found an issue: Since only one disk is included in the batch call (/dev/mapper/mpathz), ceph-volume tries to calculate a whole NVME disk for its block.db. This, of course, doesn’t work. It has no clue on how to properly slice a LV for the new block.db on an existing NVME drive given the passed arguments. The check that fails is this snippet (ceph_volume/devices/lvm/batch.py): while abs_size <= free_size and len(ret) < new_osds and occupied_slots < fast_slots_per_device: free_size -= abs_size.b occupied_slots += 1 ret.append((dev.path, relative_size, abs_size, requested_slots)) When using an empty node, occupied_slots = 0, and fast_slots_per_device = 18. Therefore the deployment works. When only replacing one OSD, occupied_slots = 17 (because one LV got zapped) and fast_slots_per_device = 1. Therefore the last conditional check is false, the loop is never entered and no OSD is created as result. I hacked a workaround, which removes the last conditional check (occupied_slots < fast_slots_per_device) and hard-coded the expected size of each block.db in my deployment file: ``` service_type: osd service_id: delta2024_osd service_name: osd.delta2024_osd placement: label: delta2024 spec: block_db_size: 397G data_devices: rotational: 1 size: '15T:' db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore ``` This gives the expected results. In my opinion, cephadm sends a wrong „ceph-volume lvm batch“ command to the osd-node. It should always include all of the disks, since running it is promised to be idempotent. With the full list of disks, ceph-volume should be able to calculate correct slots for block.db. Did I find a bug here or is this expected behavior? --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens