I didn't get any reply on this issue, so I tried some steps: - I removed Apparmor (Ubuntu right...) - I restarted the server - osd's unmanaged: #ceph orch set-unmanaged osd.all-available-devices (because when I want to create lvm's this service was bothering me) - Created lvm on a disk Then on creating the osd I get this: ceph orch daemon add osd hvs005:/dev/vgsdc/hvs005_sdc Created no osd(s) on host hvs005; already created? I removed config keys of the first available osd number also the auth... No luck... Can somewone give me some pointers on how to continue creating osd? Note: My setup is a simple setup deployed with cephadm and docker...
-----Oorspronkelijk bericht----- Van: Dominique Ramaekers <dominique.ramaekers@cometal.be> Verzonden: maandag 20 januari 2025 11:59 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] osd won't restart
Hi,
Strange thing just happened (ceph v19.2.0). I added two disks to a host. Kernel recognized nicely the two disks and they appeared as available devices in ceph.
After 15 minutes osd's were not created, so I looked at the logs: /usr/bin/docker: stderr --> Creating keyring file for osd.36 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/keyring /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/ /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 36 --monmap /var/lib/ceph/osd/ceph- 36/activate.monmap --keyfile - --osdspec-affinity all-available-devices --osd- data /var/lib/ceph/osd/ceph-36/ --osd-uuid 41675779-943d-4dca-baa3- 3a4f6ace004a --setuser ceph --setgroup ceph /usr/bin/docker: stderr stderr: 2025-01-20T08:21:01.979+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator &) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/docker: stderr stderr: 2025-01-20T08:21:01.979+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator &) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/docker: stderr stderr: 2025-01-20T08:21:01.980+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator &) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/docker: stderr stderr: 2025-01-20T08:21:01.980+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36/) _read_fsid unparsable uuid /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.075+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.506+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.506+0000 79e9fea34640 -1 bdev(0x566a21f14a80 /var/lib/ceph/osd/ceph-36//block) open open got: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.506+0000 79e9fea34640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.506+0000 79e9fea34640 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-36/: (13) Permission denied /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes
As it sad "Permission denied" and I have already osd's running, I thought it was an issue that docker may be updated but not restarted. So I did 'systemcrl restart docker.service'. Now none of the managed osd's are going back online!!! 'systemctl start ceph-dd4b0610-b4d2-11ec-bb58- d1b32ae31585@osd.18.service<mailto:ceph-dd4b0610-b4d2-11ec-bb58- d1b32ae31585@osd.18.service>' fails with not much explanation...
Only unmanaged osd have no issue...
I didn't payed much attention to the log entry '_read_fsid unparsable uuid'... So I think there is more going on. Permission denied is logical if the path is wrong... see dubbel slash in '_read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block' Is this a bug like https://github.com/rook/rook/issues/10219 ?
Can I get around this without recreating these osd as unmanaged?
Thanks in advance.
Dominique.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io