[ceph-users] bugs ceph-volume scripting

I have been creating lvm osd's with: ceph-volume lvm zap --destroy /dev/sdf && ceph-volume lvm create --data /dev/sdf --dmcrypt Because this procedure failed: ceph-volume lvm zap --destroy /dev/sdf (waiting on slow human typing) ceph-volume lvm create --data /dev/sdf --dmcrypt However when I was looking at the /var/lib/ceph/osd I expected these lvm mounts to list as world writable[1] So I decided to compare: [@osd]# cp -a ceph-33 ceph-33.bak2 [@osd]# service ceph-osd@33 stop [@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 start [BUG1] With the zap && create, I seem to be running the osd without the tmpfs mounted?! So that means that if I reboot this node and the content is different from the tmpfs. I have a serious problem. So I am trying to unmount ceph-33 [BUG2] Wtf ceph-osd@33 is running! Something started (ceph-volume?) the osd, without even having a chance to inspect the difference of these folders. If ceph-volume 'by design' starts the osd, then design is bad, nobody is expecting this behaviour and I have no idea what can go wrong if this startup data in tmpfs ceph-33 mount is different from the lvm create files on the os disk. Stopping again osd-33 [@osd]# service ceph-osd@33 stop Trying to unmount again ceph-33 [@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 stop [BUG3] service ceph-volume just does not unmount tmpfs. I have to unmount with umount /var/lib/ceph/osd/ceph-33 Inspecting differences of both, [@osd]# ls -l ceph-33.bak2 ceph-33.new ceph-33.new: total 28 lrwxrwxrwx 1 ceph ceph 50 Oct 1 10:06 block -> /dev/mapper/1K8AX3-D3Gv-VKdY-0wTW-qjgd-txAu-JbNJHo -rw------- 1 ceph ceph 37 Oct 1 10:06 ceph_fsid -rw------- 1 ceph ceph 37 Oct 1 10:06 fsid -rw------- 1 ceph ceph 56 Oct 1 10:06 keyring -rw------- 1 ceph ceph 106 Oct 1 10:06 lockbox.keyring -rw------- 1 ceph ceph 6 Oct 1 10:06 ready -rw------- 1 ceph ceph 10 Oct 1 10:06 type -rw------- 1 ceph ceph 3 Oct 1 10:06 whoami ceph-33.bak2: total 56 -rw-r----- 1 ceph ceph 373 Sep 30 21:23 activate.monmap lrwxrwxrwx 1 ceph ceph 50 Sep 30 21:23 block -> /dev/mapper/1K8AX3-D3Gv-VKdY-0wTW-qjgd-txAu-JbNJHo -rw------- 1 ceph ceph 2 Sep 30 21:23 bluefs -rw------- 1 ceph ceph 37 Sep 30 21:23 ceph_fsid -rw-r----- 1 ceph ceph 37 Sep 30 21:23 fsid -rw------- 1 ceph ceph 56 Sep 30 21:23 keyring -rw------- 1 ceph ceph 8 Sep 30 21:23 kv_backend -rw------- 1 ceph ceph 106 Sep 30 21:23 lockbox.keyring -rw------- 1 ceph ceph 21 Sep 30 21:23 magic -rw------- 1 ceph ceph 4 Sep 30 21:23 mkfs_done -rw------- 1 ceph ceph 41 Sep 30 21:23 osd_key -rw------- 1 ceph ceph 6 Sep 30 21:23 ready -rw------- 1 ceph ceph 3 Sep 30 21:23 require_osd_release -rw------- 1 ceph ceph 10 Sep 30 21:23 type -rw------- 1 ceph ceph 3 Sep 30 21:23 whoami The contents of the files in new (tmpfs) luckily are the same as in bak2 (ceph-volume create?). However as you can see I miss quite a few files in the tmpfs So I am giving it a try and start this osd.33 with the tmpfs mounted. [@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 start^C [@osd]# ps -ef | grep ceph-osd | grep 33 [@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 start Redirecting to /bin/systemctl start ceph-volume(a)lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670.service And indeed, again ceph-osd is started. [@osd]# ps -ef | grep ceph-osd | grep 33 ceph 1651105 1 48 11:29 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 33 --setuser ceph --setgroup ceph [QUESTION1] Should I just copy these files like kv_backend, mkfs_done to the tmpfs mount? I seem to have these files on other ceph-volume created osd's. [QUESTION2] Is there a reasonable explanation for running into such issues, before I start thinking this is shitty scripting? [1] https://tracker.ceph.com/issues/47549