Re: ceph-volume quite buggy compared to ceph-disk
Hi Marc, Did you have any success with `ceph-volume` for activating your OSD? I am having a similar problem where the command `ceph-bluestore-tool` fails to be able to read a label for a previously created OSD on an LVM partition. I had previously been using the OSD without issues, but after a reboot it fails to load. 1. I had initially created my OSD using Ceph Octopus 15.x with `ceph orch daemon add osd <my hostname>:boot/cephfs_meta` that was able to create an OSD on the LVM partition and bring up an OSD. 2. After a reboot, the OSD fails to come up, with error from `ceph-bluestore-tool` happening inside the container specifically being unable to read the label of the device. 3. When I query the symlinked device /dev/boot/cephfs_meta -> /dev/dm3, with `dmsetup info /dev/dm-3`, I can see the state is active and that it has a UUID, etc. 4. I installed `ceph-osd` CentOS package providing the ceph-bluestore-tool, and tried to manually test and `sudo ceph-bluestore-tool show-label --dev /dev/dm-3` fails to read the label. When I try with other OSD's that were created for entire disks this command is able to read the label and print out information. I am considering submitting a ticket to the ceph issue tracker, as I am unable to figure out why the ceph-bluestore-tool cannot read the labels and it seems either the OSD was initially created incorrectly or there is a bug in ceph-bluestore-tool. One possibility is that I did not have the LVM2 package installed on this host prior to the `ceph orch daemon add ..` command and this caused a particular issue with the LVM partition OSD. -Matt On Sat, Sep 19, 2020 at 9:11 AM Marc Roos <M.Roos@f1-outsourcing.eu> wrote:
[@]# ceph-volume lvm activate 36 82b94115-4dfb-4ed0-8801-def59a432b0a Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-36 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-36/lockbox.keyring --create-keyring --name client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --add-key AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ== stdout: creating /var/lib/ceph/osd/ceph-36/lockbox.keyring added entity client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a auth(key=AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ==) Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/lockbox.keyring Running command: /usr/bin/ceph --cluster ceph --name client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --keyring /var/lib/ceph/osd/ceph-36/lockbox.keyring config-key get dm-crypt/osd/82b94115-4dfb-4ed0-8801-def59a432b0a/luks Running command: /usr/sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-9263e83b-7660-4f5b-843a-2111e882a17e/osd-block-82b94115-4dfb-4 ed0-8801-def59a432b0a I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb stderr: Device I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb already exists. Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb --path /var/lib/ceph/osd/ceph-36 --no-mon-config stderr: failed to read label for /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb: (2) No such file or directory --> RuntimeError: command returned non-zero exit status: 1
dmsetup ls lists this????
Where is an option to set the weight? As far as I can see you can only set this after peering started?
How can I mount this tmpfs manually to inspect this? Maybe put in the manual[1]?
[1] https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Matt Larson, PhD Madison, WI 53705 U.S.A.
Hi Matt, Marc, I'm using Ceph Otopus with cephadm as the orchestration tool. I've tried adding OSDs with ceph orch daemon add ... but it's pretty limited. For one, you can't create dmcrypt OSD with it nor having a separate db device. I found that the most reliable way to create OSD with cephadm orchestration tool is via the spec file (i.e. ceph orch apply osd -i osd.spec ). For example, you can ask it to find all the HDD disk by certain model, size... on a particular host(s) and make them into osds. Here is a simple spec file: service_type: osd service_id: furry-osd placement: host_pattern: 'furry' data_devices: size: '5900G:6000G' encrypted: true You can find more info here: https://docs.ceph.com/en/latest/cephadm/drivegroups/ However, this method only works with full disks, not partitions nor LVs. You can use 'ceph orch device ls <host> --refresh' to list all avail disks a particular host and why certain disks are't avail. My understanding with the ceph-volume lvm is that it uses LV labels exclusively to find the block/db/wal devices using LVM. During startup, it will use lvm to find the OSD blocks, setup dmcrypt volume (if required), create proper links, and execute ceph-osd command. The existing links in /var/lib/ceph/ceph-osd/ would be overrided by the info from the LV tags. You can use lvs -o lv_tags on an LV to see all the labels created for an OSD. Hope it helps. --Tri Hoang October 1, 2020 10:23 AM, "Matt Larson" <larsonmattr@gmail.com> wrote:
Hi Marc,
Did you have any success with `ceph-volume` for activating your OSD?
I am having a similar problem where the command `ceph-bluestore-tool` fails to be able to read a label for a previously created OSD on an LVM partition. I had previously been using the OSD without issues, but after a reboot it fails to load.
1. I had initially created my OSD using Ceph Octopus 15.x with `ceph orch daemon add osd <my hostname>:boot/cephfs_meta` that was able to create an OSD on the LVM partition and bring up an OSD. 2. After a reboot, the OSD fails to come up, with error from `ceph-bluestore-tool` happening inside the container specifically being unable to read the label of the device. 3. When I query the symlinked device /dev/boot/cephfs_meta -> /dev/dm3, with `dmsetup info /dev/dm-3`, I can see the state is active and that it has a UUID, etc. 4. I installed `ceph-osd` CentOS package providing the ceph-bluestore-tool, and tried to manually test and `sudo ceph-bluestore-tool show-label --dev /dev/dm-3` fails to read the label. When I try with other OSD's that were created for entire disks this command is able to read the label and print out information.
I am considering submitting a ticket to the ceph issue tracker, as I am unable to figure out why the ceph-bluestore-tool cannot read the labels and it seems either the OSD was initially created incorrectly or there is a bug in ceph-bluestore-tool.
One possibility is that I did not have the LVM2 package installed on this host prior to the `ceph orch daemon add ..` command and this caused a particular issue with the LVM partition OSD.
-Matt
On Sat, Sep 19, 2020 at 9:11 AM Marc Roos <M.Roos@f1-outsourcing.eu> wrote:
[@]# ceph-volume lvm activate 36 82b94115-4dfb-4ed0-8801-def59a432b0a Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-36 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-36/lockbox.keyring --create-keyring --name client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --add-key AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ== stdout: creating /var/lib/ceph/osd/ceph-36/lockbox.keyring added entity client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a auth(key=AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ==) Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/lockbox.keyring Running command: /usr/bin/ceph --cluster ceph --name client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --keyring /var/lib/ceph/osd/ceph-36/lockbox.keyring config-key get dm-crypt/osd/82b94115-4dfb-4ed0-8801-def59a432b0a/luks Running command: /usr/sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-9263e83b-7660-4f5b-843a-2111e882a17e/osd-block-82b94115-4dfb-4 ed0-8801-def59a432b0a I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb stderr: Device I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb already exists. Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb --path /var/lib/ceph/osd/ceph-36 --no-mon-config stderr: failed to read label for /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb: (2) No such file or directory --> RuntimeError: command returned non-zero exit status: 1
dmsetup ls lists this????
Where is an option to set the weight? As far as I can see you can only set this after peering started?
How can I mount this tmpfs manually to inspect this? Maybe put in the manual[1]?
[1] https://docs.ceph.com/en/latest/ceph-volume/lvm/activate
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Matt Larson, PhD Madison, WI 53705 U.S.A. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If such 'simple' tool as ceph-volume is not properly working, how can I trust cephadm to be good? Maybe ceph development should rethink trying to pump out quickly new releases, and take a bit more time for testing. I am already sticking to the oldest supported version just because of this. -----Original Message----- Cc: ceph-users Subject: Re: [ceph-users] Re: ceph-volume quite buggy compared to ceph-disk Hi Matt, Marc, I'm using Ceph Otopus with cephadm as the orchestration tool. I've tried adding OSDs with ceph orch daemon add ... but it's pretty limited. For one, you can't create dmcrypt OSD with it nor having a separate db device. I found that the most reliable way to create OSD with cephadm orchestration tool is via the spec file (i.e. ceph orch apply osd -i osd.spec ). For example, you can ask it to find all the HDD disk by certain model, size... on a particular host(s) and make them into osds. Here is a simple spec file: service_type: osd service_id: furry-osd placement: host_pattern: 'furry' data_devices: size: '5900G:6000G' encrypted: true You can find more info here: https://docs.ceph.com/en/latest/cephadm/drivegroups/ However, this method only works with full disks, not partitions nor LVs. You can use 'ceph orch device ls <host> --refresh' to list all avail disks a particular host and why certain disks are't avail. My understanding with the ceph-volume lvm is that it uses LV labels exclusively to find the block/db/wal devices using LVM. During startup, it will use lvm to find the OSD blocks, setup dmcrypt volume (if required), create proper links, and execute ceph-osd command. The existing links in /var/lib/ceph/ceph-osd/ would be overrided by the info from the LV tags. You can use lvs -o lv_tags on an LV to see all the labels created for an OSD. Hope it helps. --Tri Hoang
Did you have any success with `ceph-volume` for activating your OSD?
No, I have tried with ceph-volume prepare and ceph-volume activate, but got errors also. The only way for me to currently create an osd without hasle is: ceph-volume lvm zap --destroy /dev/sdf && ceph-volume lvm create --data /dev/sdf --dmcrypt and really like this, with the && But first read this[1] before you use this.
I am having a similar problem where the command `ceph-bluestore-tool` fails to be able to read a label for a previously created OSD on an LVM partition. I had previously been using the OSD without issues, but after a reboot it fails to load.
ceph-volume is creating the systemd links I think, so if you go even more low level you have to create these yourself. Check if your ceph-volumes are existing and are mounted. I have them like this. [@ ~]# find /etc/ -iname "*ceph-volume*" /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-38-7cfec20d- e963-4908-b4fc-0020f050a0d3.service ...
1. I had initially created my OSD using Ceph Octopus 15.x with `ceph orch daemon add osd <my hostname>:boot/cephfs_meta` that was able to create an OSD on the LVM partition and bring up an OSD. 2. After a reboot, the OSD fails to come up, with error from `ceph-bluestore-tool` happening inside the container specifically being unable to read the label of the device.
check these systemd entries and/or logs I guess ;)
3. When I query the symlinked device /dev/boot/cephfs_meta -> /dev/dm3, with `dmsetup info /dev/dm-3`, I can see the state is active and that it has a UUID, etc.
I have these for a running osd, you have to check indeed if the tags are existing on the vg. I think this is the process of an lvm dmcrypt osd[2] [@ ~]# dmsetup ls --tree ... H1U5hz-j51i-HLeY-wzHx-Rqzv-buSc-xVIGnB (253:7) └─ceph--bb97bd0e--9edb--4d29--b6e8--0876359edc3c-osd--block--23bf0dfe --6678--4633--b7ac--f4133da785be (253:6 [@ ~]# lvs LV VG Attr LSize Pool Origin Data% ... osd-block-b232f7a5-8409-4992-ad4d-b5bbb8ffa2e1 ceph-e459af98-4013-4860-84b8-2eb80fbd2f57 -wi-ao---- <7.28t
4. I installed `ceph-osd` CentOS package providing the ceph-bluestore-tool, and tried to manually test and `sudo ceph-bluestore-tool show-label --dev /dev/dm-3` fails to read the label. When I try with other OSD's that were created for entire disks this command is able to read the label and print out information.
I am considering submitting a ticket to the ceph issue tracker, as I am unable to figure out why the ceph-bluestore-tool cannot read the labels and it seems either the OSD was initially created incorrectly or there is a bug in ceph-bluestore-tool.
I cannot advice on this because I have not a clear understanding on what the procedure is to create and osd on this level.
One possibility is that I did not have the LVM2 package installed on this host prior to the `ceph orch daemon add ..` command and this caused a particular issue with the LVM partition OSD.
I am using centos7, which has the lvm2 by default. I am also having problems. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg06624.html [2] https://www.mail-archive.com/ceph-users@ceph.io/msg06405.html
participants (3)
-
Marc Roos
-
Matt Larson
-
tri＠postix.net