Hi, I try to add OSDs to my new Cluster (Ubuntu 24.04 + podman), Four devices are listed as available: root@ceph-1:~# ceph-volume inventory Device Path Size Device nodes rotates available Model name /dev/nvme0n1 1.82 TB nvme0n1 False True KINGSTON SNV2S2000G /dev/sda 1.82 TB sda True True WDC WD20EFZX-68A /dev/sdb 16.37 TB sdb True True TOSHIBA MG09ACA1 /dev/sdc 931.51 GB sdc True True WDC WD10EFRX-68J but none of them can be added: root@ceph-1:~# ceph orch daemon add osd ceph-1:/dev/sda Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1862, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 184, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 120, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 109, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1374, in _daemon_add_osd raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 241, in raise_if_exception raise e RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/mon.ceph-1/config Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in <module> File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5567, in main File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 409, in _infer_config File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 437, in _infer_image File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 3288, in command_ceph_volume File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 918, in get_container_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/daemons/ceph.py", line 422, in get_ceph_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 760, in selinux_enabled File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 743, in kernel_security File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2) I found some hints with apparmor and MongoDB, but that does not help in this case. Any help appreciated. Kind regards Thomas
1. Make sure you have enough RAM on ceph-1 and the "ls -h /" indicates that the system disk is less than 70% full (managed services eat a LOt of disk space!) 2. Check your selunix audit log to make sure nothing's being blocked. 3. Check your /var/lib/ceph and /var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10 for existing OSD directories. Cross-reference this to a "ceph osd tree" inventory of ceph-1. You can't define something if it's defined, albeit defectively. 4. Don't forget to zap /dev/sda with the volume-zap utility to clear out any annoying pre-existing infrastructure! Regards, Tim On 11/6/24 05:53, tpDev Tester wrote:
Hi,
I try to add OSDs to my new Cluster (Ubuntu 24.04 + podman), Four devices are listed as available:
root@ceph-1:~# ceph-volume inventory
Device Path Size Device nodes rotates available Model name /dev/nvme0n1 1.82 TB nvme0n1 False True KINGSTON SNV2S2000G /dev/sda 1.82 TB sda True True WDC WD20EFZX-68A /dev/sdb 16.37 TB sdb True True TOSHIBA MG09ACA1 /dev/sdc 931.51 GB sdc True True WDC WD10EFRX-68J
but none of them can be added:
root@ceph-1:~# ceph orch daemon add osd ceph-1:/dev/sda Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1862, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 184, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 120, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 109, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1374, in _daemon_add_osd raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 241, in raise_if_exception raise e RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/mon.ceph-1/config Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in <module> File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5567, in main File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 409, in _infer_config File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 437, in _infer_image File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 3288, in command_ceph_volume File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 918, in get_container_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/daemons/ceph.py", line 422, in get_ceph_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 760, in selinux_enabled File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 743, in kernel_security File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2)
I found some hints with apparmor and MongoDB, but that does not help in this case.
Any help appreciated.
Kind regards
Thomas
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 06.11.2024 um 17:58 schrieb Tim Holloway:
2. Check your selunix audit log to make sure nothing's being blocked.
I disabled apparmor completely and ceph orch daemon add osd ceph-x:/dev/sdy worked. Perhaps it is worth investigating more closely which Apparmor rule is blocking the ‘add osd’. Thanks for your help so far. Kind regards Thomas
I see you mentioned apparmor and MongoDB, so I guess there's a chance you found https://tracker.ceph.com/issues/66389 already (your traceback also looks the same). Other than making sure that relevant apparmor file it's parsing doesn't contain settings with spaces or trying to manually apply the fix from https://github.com/ceph/ceph/pull/57955/files#diff-5acc9785d0e913430134b549d... (it's in the squid branch at this point, but hasn't made it into any actual releases) I don't remember any workarounds being found. On Wed, Nov 6, 2024 at 5:54 AM tpDev Tester <tpdev.tester@gmail.com> wrote:
Hi,
I try to add OSDs to my new Cluster (Ubuntu 24.04 + podman), Four devices are listed as available:
root@ceph-1:~# ceph-volume inventory
Device Path Size Device nodes rotates available Model name /dev/nvme0n1 1.82 TB nvme0n1 False True KINGSTON SNV2S2000G /dev/sda 1.82 TB sda True True WDC WD20EFZX-68A /dev/sdb 16.37 TB sdb True True TOSHIBA MG09ACA1 /dev/sdc 931.51 GB sdc True True WDC WD10EFRX-68J
but none of them can be added:
root@ceph-1:~# ceph orch daemon add osd ceph-1:/dev/sda Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1862, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 184, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 120, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 109, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1374, in _daemon_add_osd raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 241, in raise_if_exception raise e RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/mon.ceph-1/config Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py",
line 5579, in <module> File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py",
line 5567, in main File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py",
line 409, in _infer_config File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py",
line 324, in _infer_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py",
line 437, in _infer_image File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py",
line 311, in _validate_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py",
line 3288, in command_ceph_volume File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py",
line 918, in get_container_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/daemons/ceph.py",
line 422, in get_ceph_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py",
line 760, in selinux_enabled File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py",
line 743, in kernel_security File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py",
line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2)
I found some hints with apparmor and MongoDB, but that does not help in this case.
Any help appreciated.
Kind regards
Thomas
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Adam King
-
Tim Holloway
-
tpDev Tester