Re: ceph-volume quite buggy compared to ceph-disk - ceph-users

1 Oct 2020

Hi Marc,

 Did you have any success with `ceph-volume` for activating your OSD?

 I am having a similar problem where the command `ceph-bluestore-tool`
fails to be able to read a label for a previously created OSD on an
LVM partition. I had previously been using the OSD without issues, but
after a reboot it fails to load.

 1. I had initially created my OSD using Ceph Octopus 15.x with `ceph
orch daemon add osd <my hostname>:boot/cephfs_meta` that was able to
create an OSD on the LVM partition and bring up an OSD.
 2. After a reboot, the OSD fails to come up, with error from
`ceph-bluestore-tool` happening inside the container specifically
being unable to read the label of the device.
 3. When I query the symlinked device /dev/boot/cephfs_meta ->
/dev/dm3, with `dmsetup info /dev/dm-3`, I can see the state is active
and that it has a UUID, etc.
 4. I installed `ceph-osd` CentOS package providing the
ceph-bluestore-tool, and tried to manually test and `sudo
ceph-bluestore-tool show-label --dev /dev/dm-3` fails to read the
label. When I try with other OSD's that were created for entire disks
this command is able to read the label and print out information.

 I am considering submitting a ticket to the ceph issue tracker, as I
am unable to figure out why the ceph-bluestore-tool cannot read the
labels and it seems either the OSD was initially created incorrectly
or there is a bug in ceph-bluestore-tool.

 One possibility is that I did not have the LVM2 package installed on
this host prior to the `ceph orch daemon add ..` command and this
caused a particular issue with the LVM partition OSD.

 -Matt

On Sat, Sep 19, 2020 at 9:11 AM Marc Roos &lt;M.Roos(a)f1-outsourcing.eu&gt; wrote:
...

 [@]# ceph-volume lvm activate 36 82b94115-4dfb-4ed0-8801-def59a432b0a
 Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-36
 Running command: /usr/bin/ceph-authtool
 /var/lib/ceph/osd/ceph-36/lockbox.keyring --create-keyring --name
 client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --add-key
 AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ==
  stdout: creating /var/lib/ceph/osd/ceph-36/lockbox.keyring
 added entity client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a
 auth(key=AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ==)
 Running command: /usr/bin/chown -R ceph:ceph
 /var/lib/ceph/osd/ceph-36/lockbox.keyring
 Running command: /usr/bin/ceph --cluster ceph --name
 client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --keyring
 /var/lib/ceph/osd/ceph-36/lockbox.keyring config-key get
 dm-crypt/osd/82b94115-4dfb-4ed0-8801-def59a432b0a/luks
 Running command: /usr/sbin/cryptsetup --key-file - --allow-discards
 luksOpen
 /dev/ceph-9263e83b-7660-4f5b-843a-2111e882a17e/osd-block-82b94115-4dfb-4
 ed0-8801-def59a432b0a I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb
  stderr: Device I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb already exists.
 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36
 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
 prime-osd-dir --dev /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb
 --path /var/lib/ceph/osd/ceph-36 --no-mon-config
  stderr: failed to read label for
 /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb: (2) No such file or
 directory
 -->  RuntimeError: command returned non-zero exit status: 1

 dmsetup ls lists this????

 Where is an option to set the weight? As far as I can see you can only
 set this after peering started?

 How can I mount this tmpfs manually to inspect this? Maybe put in the
 manual[1]?

 [1]
 https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/

 _______________________________________________
 ceph-users mailing list -- ceph-users(a)ceph.io
 To unsubscribe send an email to ceph-users-leave(a)ceph.io 

-- 
Matt Larson, PhD
Madison, WI  53705 U.S.A.