Responding partially to my own query, I have decided on the following structure, in order to have encrypted OSDs/Bluestore journals and not wait for proper ceph-volume support. 1) SSD(s), fully encrypted, acting as PV(s) for VG(s) to store LVs for the Block DBs. My current setup is 1 SSD for 4 Block DBs. The names of the VGs or the LVs can be chosen freely. My VG name is "bluestore" 2) Each of my SATA disks gets encrypted fully by me, then I build a PV/VG/LV for all the space. 3) For all these encrypted disks, I create a keyfile and auto-open them on reboot. 4) I create the OSDs with a string such as: ceph-volume lvm prepare --bluestore --block.db bluestore/ceph04bluf --data ceph04vgf/ceph04lvf (as mentioned, the VG/LV names are yours to choose) This way proper LVM tags are placed on all block devices. If for some reason the OSDs are not properly activated (as in my case) or the /var/lib/ceph/osd/ceph-<id> directories not populated, use the following script: #!/bin/bash # the assumption is that all of my OSD block VGs are in the form of /dev/`hostname`vg<id> #you can change this string to whatever you like for mylv in $( ls /dev/`hostname`vg*/* ); do # check whether we have ceph tags, as I might have created an LV but not yet provisioned it lvs -o lv_tags $mylv | grep ceph >/dev/null || continue # I have decided to eval the whole lvm tags to variables prepended by my, for future use eval `lvs -o lv_tags $mylv |tail -n1|sed -e 's/^[ ]*//g;s/,/\n/g'|cut -d '.' -f2-|sed -e 's/^/my/g'` ceph-volume lvm activate --bluestore $myosd_id $myosd_fsid done On Fri, Dec 4, 2020 at 5:37 PM Panayiotis Gotsis <panos.gotsis@gmail.com> wrote:
Hello,
I have made some tests with creating OSDs and I have found out that there are big issues with the ceph-volume functionality.
1) If using dmcrypt and separate data and db block devices, ceph-volume creates cryprodevs/PVs/VGs/LVs for both devices. This might seem as normal, until one considers the possibility that a single SSD will support multiple OSDs and the whole concept of being able to resize bluestore DB partitions gets very complicated like this; I can have a full SSD encrypted, create a PV/VG on top and provide the LVs myself without an extra layer of encryption.
2) If I try to circumvent this limitation by asking ceph-volume to use already encrypted devices for both the data and the bluestore db, the OSDs are not auto scanned at startup, ceph-volume simple scan complaints that they are not real devices and nothing starts.
Am I missing something?