3 Jul
2020
3 Jul
'20
5:02 p.m.
I have a situation were OSDs won’t work as Docker containers with Octopus on an Ubuntu
2020.04 host.
The cephadm adopt —style legacy —name osd.8 command works as expected, and sets up
the /var/lib/ceph/ directory as expected:
root@balin:~# ll /var/lib/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667/osd.8/
total 64
drwx------ 2 167 167 4096 Jul 3 08:39 ./
drwx------ 8 167 167 4096 Jul 3 08:30 ../
lrwxrwxrwx 1 167 167 93 Jul 3 08:39 block ->
/dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-block-5b229640-14e5-4e7f-9993-9368d172c30c
-rw------- 1 167 167 37 Jul 3 08:39 ceph_fsid
-rw------- 1 167 167 283 Jul 3 08:14 config
-rw------- 1 167 167 37 Jul 3 08:39 fsid
-rw------- 1 167 167 55 Jul 3 08:39 keyring
-rw------- 1 167 167 6 Jul 3 08:39 ready
-rw------- 1 167 167 3 May 14 21:41 require_osd_release
-rw------- 1 167 167 10 Jul 3 08:39 type
-rw------- 1 167 167 38 Jul 3 08:14 unit.configured
-rw------- 1 167 167 48 May 14 21:41 unit.created
-rw------- 1 167 167 28 Jul 3 08:14 unit.image
-rw------- 1 167 167 825 Jul 3 08:14 unit.poststop
-rw------- 1 167 167 2367 Jul 3 08:14 unit.run
-rw------- 1 167 167 2 Jul 3 08:39 whoami
However, the problem shows up when the container starts.
Jul 03 08:39:40 balin systemd[1]: Started Ceph osd.8 for
c3d06c94-bb66-4f84-bf78-470a2364b667.
Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-8
Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/ceph-bluestore-tool
--cluster=ceph prime-osd-dir --dev
/dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-blo>
Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/ln -snf
/dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-block-5b229640-14e5-4e7f-9993-9368d172c30c
/var/li>
Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-8/block
Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-8
Jul 03 08:39:41 balin bash[5412]: --> ceph-volume lvm activate successful for osd ID:
8
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40 0 set
uid:gid to 167:167 (ceph:ceph)
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40 0 ceph
version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable), pro>
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40 0
pidfile_write: ignore empty --pid-file
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev
create path /var/lib/ceph/osd/ceph-8/block type kernel
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1
bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph>
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1
bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open size 12000134430720 (0xae9f>
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1
bluestore(/var/lib/ceph/osd/ceph-8) _set_cache_sizes cache_size 1073741824 meta 0.4 >
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev
create path /var/lib/ceph/osd/ceph-8/block type kernel
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1
bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph>
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1
bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) open size 12000134430720 (0xae9f>
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1
bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-8/block size 11 TiB
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1
bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) close
Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.888+0000 7f52b1e70f40 1
bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) close
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.141+0000 7f52b1e70f40 0
starting osd.8 osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 0 load:
jerasure load: lrc load: isa
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 1 bdev
create path /var/lib/ceph/osd/ceph-8/block type kernel
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 1
bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 -1
bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open open got: (13) Permission d>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0
osd.8:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0
osd.8:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0
osd.8:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0
osd.8:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0
osd.8:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1
bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/c>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 1
bluestore(/var/lib/ceph/osd/ceph-8) _mount path /var/lib/ceph/osd/ceph-8
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1
bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/c>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 1 bdev
create path /var/lib/ceph/osd/ceph-8/block type kernel
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 1
bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1
bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open open got: (13) Permission d>
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 osd.8
0 OSD:init: unable to mount object store
Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 **
ERROR: osd init failed: (13) Permission denied
Jul 03 08:39:42 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service:
Main process exited, code=exited, status=1/FAILURE
Jul 03 08:39:42 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service:
Failed with result 'exit-code'.
Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service:
Scheduled restart job, restart counter is at 6.
Jul 03 08:39:53 balin systemd[1]: Stopped Ceph osd.8 for
c3d06c94-bb66-4f84-bf78-470a2364b667.
Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service:
Start request repeated too quickly.
Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service:
Failed with result 'exit-code'.
Jul 03 08:39:53 balin systemd[1]: Failed to start Ceph osd.8 for
c3d06c94-bb66-4f84-bf78-470a2364b667.
There’s a permission denied issue with the block, and I can’t find what might be causing
that. If I set up an OSD fresh, it works fine with the docker container until I reboot,
and then I get the same permission denied error. If I use ceph-volume lvm activate a
standard ceph systemd service is created, and the OSD comes online without any problem,
though it’s not visible to ceph orch.