[ceph-users] Re: [Octopus] OSD won’t work with Docker

I have a situation were OSDs won’t work as Docker containers with Octopus on an Ubuntu 2020.04 host. The cephadm adopt —style legacy —name osd.8 command works as expected, and sets up the /var/lib/ceph/ directory as expected: root@balin:~# ll /var/lib/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667/osd.8/ total 64 drwx------ 2 167 167 4096 Jul 3 08:39 ./ drwx------ 8 167 167 4096 Jul 3 08:30 ../ lrwxrwxrwx 1 167 167 93 Jul 3 08:39 block -> /dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-block-5b229640-14e5-4e7f-9993-9368d172c30c -rw------- 1 167 167 37 Jul 3 08:39 ceph_fsid -rw------- 1 167 167 283 Jul 3 08:14 config -rw------- 1 167 167 37 Jul 3 08:39 fsid -rw------- 1 167 167 55 Jul 3 08:39 keyring -rw------- 1 167 167 6 Jul 3 08:39 ready -rw------- 1 167 167 3 May 14 21:41 require_osd_release -rw------- 1 167 167 10 Jul 3 08:39 type -rw------- 1 167 167 38 Jul 3 08:14 unit.configured -rw------- 1 167 167 48 May 14 21:41 unit.created -rw------- 1 167 167 28 Jul 3 08:14 unit.image -rw------- 1 167 167 825 Jul 3 08:14 unit.poststop -rw------- 1 167 167 2367 Jul 3 08:14 unit.run -rw------- 1 167 167 2 Jul 3 08:39 whoami However, the problem shows up when the container starts. Jul 03 08:39:40 balin systemd[1]: Started Ceph osd.8 for c3d06c94-bb66-4f84-bf78-470a2364b667. Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-blo> Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/ln -snf /dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-block-5b229640-14e5-4e7f-9993-9368d172c30c /var/li> Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-8/block Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 Jul 03 08:39:41 balin bash[5412]: --> ceph-volume lvm activate successful for osd ID: 8 Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40  0 set uid:gid to 167:167 (ceph:ceph) Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40  0 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable), pro> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40  0 pidfile_write: ignore empty --pid-file Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40  1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40  1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40  1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open size 12000134430720 (0xae9f> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40  1 bluestore(/var/lib/ceph/osd/ceph-8) _set_cache_sizes cache_size 1073741824 meta 0.4 > Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40  1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40  1 bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40  1 bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) open size 12000134430720 (0xae9f> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-8/block size 11 TiB Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40  1 bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) close Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.888+0000 7f52b1e70f40  1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) close Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.141+0000 7f52b1e70f40  0 starting osd.8 osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40  0 load: jerasure load: lrc load: isa Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40  1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40  1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 -1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open open got: (13) Permission d> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40  0 osd.8:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40  0 osd.8:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40  0 osd.8:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40  0 osd.8:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40  0 osd.8:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/c> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40  1 bluestore(/var/lib/ceph/osd/ceph-8) _mount path /var/lib/ceph/osd/ceph-8 Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/c> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40  1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40  1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open open got: (13) Permission d> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 osd.8 0 OSD:init: unable to mount object store Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1  ** ERROR: osd init failed: (13) Permission denied Jul 03 08:39:42 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Main process exited, code=exited, status=1/FAILURE Jul 03 08:39:42 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Failed with result 'exit-code'. Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Scheduled restart job, restart counter is at 6. Jul 03 08:39:53 balin systemd[1]: Stopped Ceph osd.8 for c3d06c94-bb66-4f84-bf78-470a2364b667. Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Start request repeated too quickly. Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Failed with result 'exit-code'. Jul 03 08:39:53 balin systemd[1]: Failed to start Ceph osd.8 for c3d06c94-bb66-4f84-bf78-470a2364b667. There’s a permission denied issue with the block, and I can’t find what might be causing that. If I set up an OSD fresh, it works fine with the docker container until I reboot, and then I get the same permission denied error. If I use ceph-volume lvm activate a standard ceph systemd service is created, and the OSD comes online without any problem, though it’s not visible to ceph orch.