On Fri, 20 Jan 2023 at 13:12, seccentral <seccentral@protonmail.com> wrote:
Hello, Thank you for the valuable info, and especially for the slack link (it's not listed on the community page) The ceph-volume command was issued in the following manner : login to my 1st vps from which I performed the boostrap with cephadm exec
sudo cephadm shell
which gets me root shell inside the container and then ceph-volume [...] etc.
you shouldn't run ceph-volume commands like that in order to create OSDs from a cephadm shell. That will result in OSDs not being managed by cephadm as they should be. you probably want to run `ceph orch daemon add osd <host>:data_devices=/dev/sdb,db_devices=/dev/mapper/ssd0-ssd0_0,method=raw` or use a service spec like the following: ``` --- service_type: osd service_id: mix_raw_lvm placement: hosts: - node123 data_devices: paths: - /dev/sdb db_devices: paths: - /dev/mapper/ssd0-ssd0_0 method: raw ```
----- I nuked my environment to recreate the issues and paste them here so my new vg/lv names are different
root@dev0:/# ceph-volume raw prepare --bluestore --data /dev/sdb --block.db /dev/mapper/ssd0-ssd0_0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0011a2a8-084b-4d79-ab8f-2503dfc2c804 stderr: 2023-01-20T11:50:23.495+0000 7fdeebd02700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2023-01-20T11:50:23.495+0000 7fdeebd02700 -1 AuthRegistry(0x7fdee4060d70) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2023-01-20T11:50:23.495+0000 7fdeebd02700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2023-01-20T11:50:23.495+0000 7fdeebd02700 -1 AuthRegistry(0x7fdee4064440) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2023-01-20T11:50:23.499+0000 7fdeebd02700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2023-01-20T11:50:23.499+0000 7fdeebd02700 -1 AuthRegistry(0x7fdeebd00ea0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2023-01-20T11:50:23.503+0000 7fdee929d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2023-01-20T11:50:23.503+0000 7fdeea29f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2023-01-20T11:50:23.503+0000 7fdee9a9e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2023-01-20T11:50:23.503+0000 7fdeebd02700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication stderr: [errno 13] RADOS permission denied (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id
After manually ln -s /etc/ceph/ceph.keyring /var/lib/ceph/bootstrap-osd/ i got the the credentials from ceph auth ls and added them to the keyring file respecting it's syntax
I wouldn't do that, I think if you have the admin keyring you can do something like `ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring`, but again, you don't need to do that if you create your OSD with cephadm as I described above. [client.bootstrap-osd]
key = AQA5vcdj/pClABAAt9hDro+HC73wrZysJSHyAg== caps mon = "allow profile bootstrap-osd"
Then it worked:
root@dev0:/# ceph-volume raw prepare --bluestore --data /dev/sdb --block.db /dev/mapper/ssd0-ssd0_0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4d47af7e-cf8c-451a-8773-894854e3ce8a Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-3 Running command: /usr/bin/chown -R ceph:ceph /dev/sdb Running command: /usr/bin/ln -s /dev/sdb /var/lib/ceph/osd/ceph-3/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-3/activate.monmap stderr: got monmap epoch 3 --> Creating keyring file for osd.3 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/ Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ssd0-ssd0_0 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --bluestore-block-db-path /dev/mapper/ssd0-ssd0_0 --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid 4d47af7e-cf8c-451a-8773-894854e3ce8a --setuser ceph --setgroup ceph stderr: 2023-01-20T11:50:57.723+0000 7f0ee2e4b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid --> ceph-volume raw clear prepare successful for: /dev/sdb
So it creates osd entries in /var/lib/ceph/osd
I'm surprised it worked, which Ceph release are you deploying? I will have to check a few details.
As for the mgr logs, I'm trying to figure out how to get them: I listed all pods using podman ps on the host then podman logs <container_id> of the node that has mgr in the name. It's a lot to parse but I found something relevant:
2023-01-20T12:02:14.680+0000 7f0839777700 -1 log_channel(cephadm) log [ERR] : Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/51d65c78-9713-11ed-b841-c7c07153e51c/mon.dev0/config Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE= quay.io/ceph/ceph@sha256:0560b16bec6e84345f29fb6693cd2430884e6efff16a95d5bdd0bb06d7661c45 -e NODE_NAME=dev0.ddhosted.ro -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/51d65c78-9713-11ed-b841-c7c07153e51c:/var/run/ceph:z -v /var/log/ceph/51d65c78-9713-11ed-b841-c7c07153e51c:/var/log/ceph:z -v /var/lib/ceph/51d65c78-9713-11ed-b841-c7c07153e51c/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpmo5i2jom:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph0tezmty:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:0560b16bec6e84345f29fb6693cd2430884e6efff16a95d5bdd0bb06d7661c45 lvm batch --no-auto /dev/sdb /dev/sdc --yes --no-systemd
they are sent in journald, so you should be able to see mgr logs with `journalctl -l -u <daemon-name>` ---
However, that being said, this time around after issuing the ceph-volume command that 'provisions' the osds, two podman containers spawned, one for each osd so it seems to have worked. I'm a bit confused but will be researching more into this . I may have messed up my dev env really bad initially, so maybe that's why it didnt previously work.
20d41af95386 quay.io/ceph/ceph@sha256:0560b16bec6e84345f29fb6693cd2430884e6efff16a95d5bdd0bb06d7661c45 -n osd.1 -f --set... 16 minutes ago Up 16 minutes ago ceph-51d65c78-9713-11ed-b841-c7c07153e51c-osd-1 5f836b90b5b2 quay.io/ceph/ceph@sha256:0560b16bec6e84345f29fb6693cd2430884e6efff16a95d5bdd0bb06d7661c45 -n osd.2 -f --set... 16 minutes ago Up 16 minutes ago ceph-51d65c78-9713-11ed-b841-c7c07153e51c-osd-2
they must have been created by cephadm.
Regards, -- *Guillaume Abrioux*Senior Software Engineer