Hi Matt, Marc, I'm using Ceph Otopus with cephadm as the orchestration tool. I've tried adding OSDs with ceph orch daemon add ... but it's pretty limited. For one, you can't create dmcrypt OSD with it nor having a separate db device. I found that the most reliable way to create OSD with cephadm orchestration tool is via the spec file (i.e. ceph orch apply osd -i osd.spec ). For example, you can ask it to find all the HDD disk by certain model, size... on a particular host(s) and make them into osds. Here is a simple spec file: service_type: osd service_id: furry-osd placement: host_pattern: 'furry' data_devices: size: '5900G:6000G' encrypted: true You can find more info here: https://docs.ceph.com/en/latest/cephadm/drivegroups/ However, this method only works with full disks, not partitions nor LVs. You can use 'ceph orch device ls <host> --refresh' to list all avail disks a particular host and why certain disks are't avail. My understanding with the ceph-volume lvm is that it uses LV labels exclusively to find the block/db/wal devices using LVM. During startup, it will use lvm to find the OSD blocks, setup dmcrypt volume (if required), create proper links, and execute ceph-osd command. The existing links in /var/lib/ceph/ceph-osd/ would be overrided by the info from the LV tags. You can use lvs -o lv_tags on an LV to see all the labels created for an OSD. Hope it helps. --Tri Hoang October 1, 2020 10:23 AM, "Matt Larson" &lt;larsonmattr(a)gmail.com&gt; wrote: > Hi Marc, > > Did you have any success with `ceph-volume` for activating your OSD? > > I am having a similar problem where the command `ceph-bluestore-tool` > fails to be able to read a label for a previously created OSD on an > LVM partition. I had previously been using the OSD without issues, but > after a reboot it fails to load. > > 1. I had initially created my OSD using Ceph Octopus 15.x with `ceph > orch daemon add osd <my hostname>:boot/cephfs_meta` that was able to > create an OSD on the LVM partition and bring up an OSD. > 2. After a reboot, the OSD fails to come up, with error from > `ceph-bluestore-tool` happening inside the container specifically > being unable to read the label of the device. > 3. When I query the symlinked device /dev/boot/cephfs_meta -> > /dev/dm3, with `dmsetup info /dev/dm-3`, I can see the state is active > and that it has a UUID, etc. > 4. I installed `ceph-osd` CentOS package providing the > ceph-bluestore-tool, and tried to manually test and `sudo > ceph-bluestore-tool show-label --dev /dev/dm-3` fails to read the > label. When I try with other OSD's that were created for entire disks > this command is able to read the label and print out information. > > I am considering submitting a ticket to the ceph issue tracker, as I > am unable to figure out why the ceph-bluestore-tool cannot read the > labels and it seems either the OSD was initially created incorrectly > or there is a bug in ceph-bluestore-tool. > > One possibility is that I did not have the LVM2 package installed on > this host prior to the `ceph orch daemon add ..` command and this > caused a particular issue with the LVM partition OSD. > > -Matt > > On Sat, Sep 19, 2020 at 9:11 AM Marc Roos &lt;M.Roos(a)f1-outsourcing.eu&gt; wrote: > >> [@]# ceph-volume lvm activate 36 82b94115-4dfb-4ed0-8801-def59a432b0a >> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-36 >> Running command: /usr/bin/ceph-authtool >> /var/lib/ceph/osd/ceph-36/lockbox.keyring --create-keyring --name >> client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --add-key >> AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ== >> stdout: creating /var/lib/ceph/osd/ceph-36/lockbox.keyring >> added entity client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a >> auth(key=AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ==) >> Running command: /usr/bin/chown -R ceph:ceph >> /var/lib/ceph/osd/ceph-36/lockbox.keyring >> Running command: /usr/bin/ceph --cluster ceph --name >> client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --keyring >> /var/lib/ceph/osd/ceph-36/lockbox.keyring config-key get >> dm-crypt/osd/82b94115-4dfb-4ed0-8801-def59a432b0a/luks >> Running command: /usr/sbin/cryptsetup --key-file - --allow-discards >> luksOpen >> /dev/ceph-9263e83b-7660-4f5b-843a-2111e882a17e/osd-block-82b94115-4dfb-4 >> ed0-8801-def59a432b0a I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb >> stderr: Device I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb already exists. >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36 >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph >> prime-osd-dir --dev /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb >> --path /var/lib/ceph/osd/ceph-36 --no-mon-config >> stderr: failed to read label for >> /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb: (2) No such file or >> directory >> --> RuntimeError: command returned non-zero exit status: 1 >> >> dmsetup ls lists this???? >> >> Where is an option to set the weight? As far as I can see you can only >> set this after peering started? >> >> How can I mount this tmpfs manually to inspect this? Maybe put in the >> manual[1]? >> >> [1] >> https://docs.ceph.com/en/latest/ceph-volume/lvm/activate >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Matt Larson, PhD > Madison, WI 53705 U.S.A. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

If such 'simple' tool as ceph-volume is not properly working, how can I trust cephadm to be good? Maybe ceph development should rethink trying to pump out quickly new releases, and take a bit more time for testing. I am already sticking to the oldest supported version just because of this. -----Original Message----- Cc: ceph-users Subject: Re: [ceph-users] Re: ceph-volume quite buggy compared to ceph-disk Hi Matt, Marc, I'm using Ceph Otopus with cephadm as the orchestration tool. I've tried adding OSDs with ceph orch daemon add ... but it's pretty limited. For one, you can't create dmcrypt OSD with it nor having a separate db device. I found that the most reliable way to create OSD with cephadm orchestration tool is via the spec file (i.e. ceph orch apply osd -i osd.spec ). For example, you can ask it to find all the HDD disk by certain model, size... on a particular host(s) and make them into osds. Here is a simple spec file: service_type: osd service_id: furry-osd placement: host_pattern: 'furry' data_devices: size: '5900G:6000G' encrypted: true You can find more info here: https://docs.ceph.com/en/latest/cephadm/drivegroups/ However, this method only works with full disks, not partitions nor LVs. You can use 'ceph orch device ls <host> --refresh' to list all avail disks a particular host and why certain disks are't avail. My understanding with the ceph-volume lvm is that it uses LV labels exclusively to find the block/db/wal devices using LVM. During startup, it will use lvm to find the OSD blocks, setup dmcrypt volume (if required), create proper links, and execute ceph-osd command. The existing links in /var/lib/ceph/ceph-osd/ would be overrided by the info from the LV tags. You can use lvs -o lv_tags on an LV to see all the labels created for an OSD. Hope it helps. --Tri Hoang