28 May
2020
28 May
'20
12:41 a.m.
Hi, trying to migrate a second ceph cluster to Cephadm. All the host successfully migrated
from "legacy" except one of the OSD hosts (cephadm kept duplicating osd ids e.g.
two "osd.5", still not sure why). To make things easier, we re-provisioned the
node (reinstalled from netinstall, applied the same SaltStack traits as the other nodes,
wiped the disks) and tried to use cephadm to setup the OSD's.
So, orch correctly starts the provisioning processes (a docker container running
ceph-volume is created). But the provisioning never completes (docker exec):
# ps axu
root 1 0.1 0.2 99272 22488 ? Ss 15:26 0:01
/usr/libexec/platform-python -s /usr/sbin/ceph-volume lvm batch --no-auto /dev/sdb
/dev/sdc --dmcrypt --yes --no-systemd
root 807 0.9 0.5 154560 44120 ? S<L 15:26 0:06 /usr/sbin/cryptsetup
--key-file - --allow-discards luksOpen
/dev/ceph-851cae40-3270-45ea-b788-be6e05465e92/osd-data-e3157b54-f6b9-4ec9-ab12-e289f52c00a4
Afr6Ct-ok4h-pBEy-GfFF-xxYl-EKwi-cHhjZc
# cat /var/log/ceph/ceph-volume.log
Running command: /usr/sbin/cryptsetup --batch-mode --key-file - luksFormat
/dev/ceph-851cae40-3270-45ea-b788-be6e05465e92/osd-data-e3157b54-f6b9-4ec9-ab12-e289f52c00a4
Running command: /usr/sbin/cryptsetup --key-file - --allow-discards luksOpen
/dev/ceph-851cae40-3270-45ea-b788-be6e05465e92/osd-data-e3157b54-f6b9-4ec9-ab12-e289f52c00a4
Afr6Ct-ok4h-pBEy-GfFF-xxYl-EKwi-cHhjZc
# docker ps
2956dec0450d ceph/ceph:v15 "/usr/sbin/ceph-volu…" 14
minutes ago Up 14 minutes condescending_nightingale
# cat osd_spec_default.yaml
service_type: osd
service_id: osd_spec_default
placement:
host_pattern: '*'
data_devices:
all: true
encrypted: true
It looks like cephadm hangs on luksOpen.
Is this expected (encryption is mentioned to be supported, outside of no documentation)?
28 May
28 May
3:31 a.m.
I noticed the luks volumes were open, even though luksOpen hung. I killed cryptsetup (once
per disk) and ceph-volume continued and eventually created the osd's for the host
(yes, this node will be slated for another reinstall when cephadm is stabilized).
Is there a way to remove an osd service spec with the current tooling? The drives are
immediately locked when the node is added to orch.
2 Jun
2 Jun
10:34 p.m.
Just following up. Is hanging, during the provisioning of an encrypted OSD, expected
behavior with the current tooling?
4 Jun
4 Jun
6:28 p.m.
encrypted OSDS should land in the next octopus release:
https://tracker.ceph.com/issues/44625
Am 27.05.20 um 20:31 schrieb m(a)silvenga.com:
I noticed the luks volumes were open, even though
luksOpen hung. I killed cryptsetup (once per disk) and ceph-volume continued and
eventually created the osd's for the host (yes, this node will be slated for another
reinstall when cephadm is stabilized).
Is there a way to remove an osd service spec with the current tooling? The drives are
immediately locked when the node is added to orch.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer
6 Jun
6 Jun
9:31 a.m.
Cool. Cool cool cool.
Looks like the issue I was experiencing was resolved by
https://github.com/ceph/ceph/pull/34745. Didn't know encrypted OSD's weren't
supported at all. v15.2.0 did seem to handle them fine, looks like 15.2.1 and 15.2.2 have
some regression there. Under 15.2.1 OSD's are now failing to come up on reboot without
resorting to killing cryptsetup.
Thanks for the link, I was looking in the completely wrong place!
1422
days inactive
1432
days old
4 comments
2 participants
participants (2)
-
m＠silvenga.com -
Sebastian Wagner