Unable to replace OSDs deployed with ceph-volume lvm batch
Hi, we had a failing hard disk, and I replace it and want to create a new OSD on it now. But ceph-volume fails under these circumstances. In the original setup, the OSDs were created with ceph-volume lvm batch using a bunch of drives and a NVMe device for bluestore db. The batch mode uses a volume group on the NVMe device instead of partitions.I have removed the former db logical volume, the lvm setup for the former hard disk and all other remainders. Creating a new OSD with any combination of devices now fails: --data /dev/sda --block.db <nvme device> --data /dev/sda --block.db <nvme volume group> --data /dev/sda --block.db <lv created manually in nvme volume group> # ceph-volume lvm create --bluestore --data /dev/sda --block.db /dev/ceph-block-dbs-ea684aa8-544e-4c4a-8664-6cb50b3116b8/osd-block-db-a8f1489a-d97b-479e-b9a7-30fc9fa99cb5 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 55cfb9f8-aa30-4f8b-8b95-a43d3f97fe5b Running command: /sbin/vgcreate -s 1G --force --yes ceph-1a2ddc14-780b-45e4-a036-e928862e6ccb /dev/sda stdout: Physical volume "/dev/sda" successfully created. stdout: Volume group "ceph-1a2ddc14-780b-45e4-a036-e928862e6ccb" successfully created Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-55cfb9f8-aa30-4f8b-8b95-a43d3f97fe5b ceph-1a2ddc14-780b-45e4-a036-e928862e6ccb stdout: Logical volume "osd-block-55cfb9f8-aa30-4f8b-8b95-a43d3f97fe5b" created. --> blkid could not detect a PARTUUID for device: /dev/ceph-block-dbs-ea684aa8-544e-4c4a-8664-6cb50b3116b8/osd-block-db-a8f1489a-d97b-479e-b9a7-30fc9fa99cb5 --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.136 --yes-i-really-mean-it stderr: purged osd.136 --> RuntimeError: unable to use device In all cases ceph-volume is not able to detect a partition uuid for the db device (which is correct, since the device is a logical volume....). Running 'ceph-volume lvm batch' again results in a OSD without using the NVMe device as db. So what is the recommended way to manually create an OSD with a certain hard disk and an existing logical volume as db partition? I would like to avoid to zap all other OSDs using the NVMe device and recreate them in a single run with 'ceph-volume lvm batch ...'. Regards, Burkhard
Hi, On 09.09.19 17:39, Burkhard Linke wrote:
# ceph-volume lvm create --bluestore --data /dev/sda --block.db /dev/ceph-block-dbs-ea684aa8-544e-4c4a-8664-6cb50b3116b8/osd-block-db-a8f1489a-d97b-479e-b9a7-30fc9fa99cb5
When using an LV on a VG omit the leading /dev/ in the command line argument. Otherwise ceph-volume thinks it is a "real" device. ceph-volume lvm create --bluestore --data /dev/sda --block.db ceph-block-dbs-ea684aa8-544e-4c4a-8664-6cb50b3116b8/osd-block-db-a8f1489a-d97b-479e-b9a7-30fc9fa99cb5 should work. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, On 9/9/19 5:55 PM, Robert Sander wrote:
Hi,
On 09.09.19 17:39, Burkhard Linke wrote:
# ceph-volume lvm create --bluestore --data /dev/sda --block.db /dev/ceph-block-dbs-ea684aa8-544e-4c4a-8664-6cb50b3116b8/osd-block-db-a8f1489a-d97b-479e-b9a7-30fc9fa99cb5 When using an LV on a VG omit the leading /dev/ in the command line argument. Otherwise ceph-volume thinks it is a "real" device.
ceph-volume lvm create --bluestore --data /dev/sda --block.db ceph-block-dbs-ea684aa8-544e-4c4a-8664-6cb50b3116b8/osd-block-db-a8f1489a-d97b-479e-b9a7-30fc9fa99cb5
should work.
Thanks, that did the trick. Regards, Burkhard
participants (2)
-
Burkhard Linke
-
Robert Sander