2 Oct
2020
2 Oct
'20
5:03 a.m.
Good evening,
since 2018 we have been using a custom script to create disks /
partitions, because at the time both ceph-disk and ceph-volume exhibited
bugs that made them unreliable for us.
We recently re-tested ceph-volume and while it seems generally speaking
[0] to work, using LVM seems to introduce an additional layer that is
not needed from our side.
The script we created is little less than 100 lines long and works by
specifying the device and its device class:
./ceph-osd-create-start /dev/sdd ssd
Everything else is determined by the script. As the script is very
simple and work independently of any init system, we wanted to discuss
whether it would make sense for anyone else to re-integrate something
like it back into ceph upstream?
We are aware that ceph-disk has been deprecated, however
ceph-volume raw had some issues our setup [2].
The script itself can be found on [1]. At the end it has some ungleich
specifics, but we'd be very open to remove that.
Best regards,
Nico
[0] https://tracker.ceph.com/issues/47724
[1]
https://code.ungleich.ch/ungleich-public/ungleich-tools/-/blob/master/ceph-…
[2]
[00:19:45] server8.place6:/var/lib/ceph/osd# ceph-volume raw prepare --bluestore --data
/dev/sdn
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
d0fa7074-6cdf-4947-ac1e-e73dd0ec1fe8
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-3
--> Executable selinuxenabled not in PATH:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Running command: /bin/chown -R ceph:ceph /dev/sdn
Running command: /bin/ln -s /dev/sdn /var/lib/ceph/osd/ceph-3/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-3/activate.monmap
stderr: 2020-09-28 00:20:48.276 7f8c90f27700 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory
2020-09-28 00:20:48.276 7f8c90f27700 -1 AuthRegistry(0x7f8c8c081d08) no keyring found at
/etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
disabling cephx
stderr: got monmap epoch 12
Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-3/keyring --create-keyring
--name osd.3 --add-key AQA/EHFfINr/HhAAVFAP9NF2LLGFrJEGXbbMSw==
stdout: creating /var/lib/ceph/osd/ceph-3/keyring
added entity osd.3 auth(key=AQA/EHFfINr/HhAAVFAP9NF2LLGFrJEGXbbMSw==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3
--monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osd-data
/var/lib/ceph/osd/ceph-3/ --osd-uuid d0fa7074-6cdf-4947-ac1e-e73dd0ec1fe8 --setuser ceph
--setgroup ceph
stderr: 2020-09-28 00:20:48.760 7f6e34ebfd80 -1 bluestore(/var/lib/ceph/osd/ceph-3/)
_read_fsid unparsable uuid
stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 bdev(0x561a9a0f6700
/var/lib/ceph/osd/ceph-3//block) _lock flock failed on /var/lib/ceph/osd/ceph-3//block
stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 bdev(0x561a9a0f6700
/var/lib/ceph/osd/ceph-3//block) open failed to lock /var/lib/ceph/osd/ceph-3//block: (11)
Resource temporarily unavailable
stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 OSD::mkfs: couldn't mount
ObjectStore: error (11) Resource temporarily unavailable
stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 ** ERROR: error creating empty object
store in /var/lib/ceph/osd/ceph-3/: (11) Resource temporarily unavailable
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.3 --yes-i-really-mean-it
stderr: 2020-09-28 00:20:56.072 7f3e446a9700 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory
2020-09-28 00:20:56.072 7f3e446a9700 -1 AuthRegistry(0x7f3e3c081d08) no keyring found at
/etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
disabling cephx
stderr: purged osd.3
--> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph
--osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap
--keyfile - --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid
d0fa7074-6cdf-4947-ac1e-e73dd0ec1fe8 --setuser ceph --setgroup ceph
[00:20:56] server8.place6:/var/lib/ceph/osd#
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch