Ceph 15.2.3 on Ubuntu 20.04 with odroid xu4 / python thread Problem
Hi all, i try to run ceph client tools on an odroid xu4 (armhf) with Ubuntu 20.04 on python 3.8.5. Unfortunately there is the following error on each "ceph" command (even in ceph --help) Traceback (most recent call last): File "/usr/bin/ceph", line 1275, in <module> retval = main() File "/usr/bin/ceph", line 981, in main cluster_handle = run_in_thread(rados.Rados, File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1342, in run_in_thread raise Exception("timed out") Exception: timed out With this Server I access an existing Ceph-Cluster with the same hardware. I checked the code part, there is just a thread start and a join (waiting for finish a RadosThread). Maybe this is a python error in combination with armhf architecture? Maybe someone can help. Thanks and greetings Dominik
Hi, I have a similar if not the same issue. I run armbian buster on my odroid hc2 which is the same as a xu4 and I get the following error, trying to build a cluster with ceph-ansible: ASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] *************************************************** Wednesday 06 January 2021 21:46:44 +0000 (0:00:00.073) 0:02:01.697 ***** fatal: [192.168.2.123]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:02.979200' end: '2021-01-06 22:46:48.049074' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.069874' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> fatal: [odroidxu4]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:03.510973' end: '2021-01-06 22:46:48.514102' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.003129' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4e292c82-bb4d-4581-aead-46ff635fda69 stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: In function 'void md_config_t::set_val_default(ConfigValues&, const ConfigTracker&, const string&, const string&)' thread b0e3a460 time 2021-01-06 22:46:48.357354 stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: 259: FAILED ceph_assert(o) stderr: ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xeb) [0xb18b26a4] stderr: 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0xd) [0xb18b2802] stderr: 3: (md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x69) [0xb195be0a] stderr: 4: (md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool)+0x15d31) [0xb1972ac6] stderr: 5: (CephContext::CephContext(unsigned int, code_environment_t, int)+0x10ef) [0xb193d090] stderr: 6: (common_preinit(CephInitParameters const&, code_environment_t, int)+0x7d) [0xb1956af6] stderr: 7: (()+0x2046a) [0xb639546a] stderr: 8: (rados_create2()+0x55) [0xb639589e] Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> Trying to run the failed command on one of the odroid nodes: root@odroidxu4:~# /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd Traceback (most recent call last): File "/usr/bin/ceph", line 1273, in <module> retval = main() File "/usr/bin/ceph", line 982, in main conffile=conffile) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread raise Exception("timed out") Exception: timed out Any clues? Best Regards, Oliver
On Thursday, January 7, 2021, Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi,
I have a similar if not the same issue. I run armbian buster on my odroid hc2 which is the same as a xu4 and I get the following error, trying to build a cluster with ceph-ansible:
We have a fix for a similar issue recently. See https://github.com/ceph/ceph/pull/38665. Could you give it a shot? I will backport it to LTS branches if it helps.
ASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] *************************************************** Wednesday 06 January 2021 21:46:44 +0000 (0:00:00.073) 0:02:01.697 ***** fatal: [192.168.2.123]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:02.979200' end: '2021-01-06 22:46:48.049074' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.069874' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> fatal: [odroidxu4]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:03.510973' end: '2021-01-06 22:46:48.514102' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.003129' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4e292c82-bb4d-4581-aead-46ff635fda69 stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: In function 'void md_config_t::set_val_default(ConfigValues&, const ConfigTracker&, const string&, const string&)' thread b0e3a460 time 2021-01-06 22:46:48.357354 stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: 259: FAILED ceph_assert(o) stderr: ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xeb) [0xb18b26a4] stderr: 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0xd) [0xb18b2802] stderr: 3: (md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x69) [0xb195be0a] stderr: 4: (md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool)+0x15d31) [0xb1972ac6] stderr: 5: (CephContext::CephContext(unsigned int, code_environment_t, int)+0x10ef) [0xb193d090] stderr: 6: (common_preinit(CephInitParameters const&, code_environment_t, int)+0x7d) [0xb1956af6] stderr: 7: (()+0x2046a) [0xb639546a] stderr: 8: (rados_create2()+0x55) [0xb639589e] Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
Trying to run the failed command on one of the odroid nodes:
root@odroidxu4:~# /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd Traceback (most recent call last): File "/usr/bin/ceph", line 1273, in <module> retval = main() File "/usr/bin/ceph", line 982, in main conffile=conffile) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread raise Exception("timed out") Exception: timed out
Any clues?
Best Regards,
Oliver _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Regards Kefu Chai
Hi, thanks for the quick reply. I will test it. Do I have to recompile ceph in order to test it? Am 07.01.2021 um 02:13 schrieb kefu chai:
On Thursday, January 7, 2021, Oliver Weinmann <oliver.weinmann@me.com <mailto:oliver.weinmann@me.com>> wrote:
Hi,
I have a similar if not the same issue. I run armbian buster on my odroid hc2 which is the same as a xu4 and I get the following error, trying to build a cluster with ceph-ansible:
We have a fix for a similar issue recently. See https://github.com/ceph/ceph/pull/38665 <https://github.com/ceph/ceph/pull/38665>. Could you give it a shot? I will backport it to LTS branches if it helps.
ASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] *************************************************** Wednesday 06 January 2021 21:46:44 +0000 (0:00:00.073) 0:02:01.697 ***** fatal: [192.168.2.123]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:02.979200' end: '2021-01-06 22:46:48.049074' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.069874' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> fatal: [odroidxu4]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:03.510973' end: '2021-01-06 22:46:48.514102' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.003129' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4e292c82-bb4d-4581-aead-46ff635fda69 stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: In function 'void md_config_t::set_val_default(ConfigValues&, const ConfigTracker&, const string&, const string&)' thread b0e3a460 time 2021-01-06 22:46:48.357354 stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: 259: FAILED ceph_assert(o) stderr: ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xeb) [0xb18b26a4] stderr: 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0xd) [0xb18b2802] stderr: 3: (md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x69) [0xb195be0a] stderr: 4: (md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool)+0x15d31) [0xb1972ac6] stderr: 5: (CephContext::CephContext(unsigned int, code_environment_t, int)+0x10ef) [0xb193d090] stderr: 6: (common_preinit(CephInitParameters const&, code_environment_t, int)+0x7d) [0xb1956af6] stderr: 7: (()+0x2046a) [0xb639546a] stderr: 8: (rados_create2()+0x55) [0xb639589e] Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
Trying to run the failed command on one of the odroid nodes:
root@odroidxu4:~# /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd Traceback (most recent call last): File "/usr/bin/ceph", line 1273, in <module> retval = main() File "/usr/bin/ceph", line 982, in main conffile=conffile) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread raise Exception("timed out") Exception: timed out
Any clues?
Best Regards,
Oliver _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Regards Kefu Chai
Oliver Weinmann <oliver.weinmann@me.com>于2021年1月7日 周四16:32写道：
Hi,
thanks for the quick reply. I will test it. Do I have to recompile ceph in order to test it?
No, you just need to apply the change of ceph_argparse.py.
Am 07.01.2021 um 02:13 schrieb kefu chai:
On Thursday, January 7, 2021, Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi,
I have a similar if not the same issue. I run armbian buster on my odroid hc2 which is the same as a xu4 and I get the following error, trying to build a cluster with ceph-ansible:
We have a fix for a similar issue recently. See https://github.com/ceph/ceph/pull/38665. Could you give it a shot? I will backport it to LTS branches if it helps.
ASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] *************************************************** Wednesday 06 January 2021 21:46:44 +0000 (0:00:00.073) 0:02:01.697 ***** fatal: [192.168.2.123]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:02.979200' end: '2021-01-06 22:46:48.049074' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.069874' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> fatal: [odroidxu4]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:03.510973' end: '2021-01-06 22:46:48.514102' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.003129' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4e292c82-bb4d-4581-aead-46ff635fda69 stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: In function 'void md_config_t::set_val_default(ConfigValues&, const ConfigTracker&, const string&, const string&)' thread b0e3a460 time 2021-01-06 22:46:48.357354 stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: 259: FAILED ceph_assert(o) stderr: ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xeb) [0xb18b26a4] stderr: 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0xd) [0xb18b2802] stderr: 3: (md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x69) [0xb195be0a] stderr: 4: (md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool)+0x15d31) [0xb1972ac6] stderr: 5: (CephContext::CephContext(unsigned int, code_environment_t, int)+0x10ef) [0xb193d090] stderr: 6: (common_preinit(CephInitParameters const&, code_environment_t, int)+0x7d) [0xb1956af6] stderr: 7: (()+0x2046a) [0xb639546a] stderr: 8: (rados_create2()+0x55) [0xb639589e] Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
Trying to run the failed command on one of the odroid nodes:
root@odroidxu4:~# /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd Traceback (most recent call last): File "/usr/bin/ceph", line 1273, in <module> retval = main() File "/usr/bin/ceph", line 982, in main conffile=conffile) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread raise Exception("timed out") Exception: timed out
Any clues?
Best Regards,
Oliver _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Regards Kefu Chai
-- Regards Kefu Chai
Ok, I replaced the whole file ceph_argparse.py with the patched one from github. Instead of an throwing an error it now seems to be stuck forever. Or am I to impatient? I'm running debian buster so this is not the latest ceph release octopus, but nautilus: root@odroidxu4:~# dpkg -l ceph Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-=================-============-=================================== ii ceph 14.2.15-3~bpo10+1 armhf distributed storage and file system Am 07.01.2021 um 13:07 schrieb kefu chai:
Oliver Weinmann <oliver.weinmann@me.com <mailto:oliver.weinmann@me.com>>于2021年1月7日 周四16:32写道：
Hi,
thanks for the quick reply. I will test it. Do I have to recompile ceph in order to test it?
No, you just need to apply the change of ceph_argparse.py.
Am 07.01.2021 um 02:13 schrieb kefu chai:
On Thursday, January 7, 2021, Oliver Weinmann <oliver.weinmann@me.com <mailto:oliver.weinmann@me.com>> wrote:
Hi,
I have a similar if not the same issue. I run armbian buster on my odroid hc2 which is the same as a xu4 and I get the following error, trying to build a cluster with ceph-ansible:
We have a fix for a similar issue recently. See https://github.com/ceph/ceph/pull/38665 <https://github.com/ceph/ceph/pull/38665>. Could you give it a shot? I will backport it to LTS branches if it helps.
ASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] *************************************************** Wednesday 06 January 2021 21:46:44 +0000 (0:00:00.073) 0:02:01.697 ***** fatal: [192.168.2.123]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:02.979200' end: '2021-01-06 22:46:48.049074' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.069874' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> fatal: [odroidxu4]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:03.510973' end: '2021-01-06 22:46:48.514102' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.003129' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4e292c82-bb4d-4581-aead-46ff635fda69 stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: In function 'void md_config_t::set_val_default(ConfigValues&, const ConfigTracker&, const string&, const string&)' thread b0e3a460 time 2021-01-06 22:46:48.357354 stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: 259: FAILED ceph_assert(o) stderr: ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xeb) [0xb18b26a4] stderr: 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0xd) [0xb18b2802] stderr: 3: (md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x69) [0xb195be0a] stderr: 4: (md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool)+0x15d31) [0xb1972ac6] stderr: 5: (CephContext::CephContext(unsigned int, code_environment_t, int)+0x10ef) [0xb193d090] stderr: 6: (common_preinit(CephInitParameters const&, code_environment_t, int)+0x7d) [0xb1956af6] stderr: 7: (()+0x2046a) [0xb639546a] stderr: 8: (rados_create2()+0x55) [0xb639589e] Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
Trying to run the failed command on one of the odroid nodes:
root@odroidxu4:~# /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd Traceback (most recent call last): File "/usr/bin/ceph", line 1273, in <module> retval = main() File "/usr/bin/ceph", line 982, in main conffile=conffile) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread raise Exception("timed out") Exception: timed out
Any clues?
Best Regards,
Oliver _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Regards Kefu Chai
-- Regards Kefu Chai
Oliver Weinmann <oliver.weinmann@me.com>于2021年1月8日 周五04:30写道：
Ok, I replaced the whole file ceph_argparse.py with the patched one from github. Instead of an throwing an error it now seems to be stuck forever. Or am I to impatient? I'm running
I don’t think so. In a healthy cluster, the command should complete in no more than 1 second. I just checked the revision history of ceph_argparse.py, there are a bunch of changes since the release of nautilus. My guess is that the version in master might include some bits not compatible with nautilus? So, I’d suggest only cherry-pick the change in that PR, and try again.
debian buster so this is not the latest ceph release octopus, but nautilus:
root@odroidxu4:~# dpkg -l ceph Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description
+++-==============-=================-============-=================================== ii ceph 14.2.15-3~bpo10+1 armhf distributed storage and file system Am 07.01.2021 um 13:07 schrieb kefu chai:
Oliver Weinmann <oliver.weinmann@me.com>于2021年1月7日 周四16:32写道：
Hi,
thanks for the quick reply. I will test it. Do I have to recompile ceph in order to test it?
No, you just need to apply the change of ceph_argparse.py.
Am 07.01.2021 um 02:13 schrieb kefu chai:
On Thursday, January 7, 2021, Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi,
I have a similar if not the same issue. I run armbian buster on my odroid hc2 which is the same as a xu4 and I get the following error, trying to build a cluster with ceph-ansible:
We have a fix for a similar issue recently. See https://github.com/ceph/ceph/pull/38665. Could you give it a shot? I will backport it to LTS branches if it helps.
ASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] *************************************************** Wednesday 06 January 2021 21:46:44 +0000 (0:00:00.073) 0:02:01.697 ***** fatal: [192.168.2.123]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:02.979200' end: '2021-01-06 22:46:48.049074' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.069874' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> fatal: [odroidxu4]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:03.510973' end: '2021-01-06 22:46:48.514102' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.003129' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4e292c82-bb4d-4581-aead-46ff635fda69 stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: In function 'void md_config_t::set_val_default(ConfigValues&, const ConfigTracker&, const string&, const string&)' thread b0e3a460 time 2021-01-06 22:46:48.357354 stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: 259: FAILED ceph_assert(o) stderr: ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xeb) [0xb18b26a4] stderr: 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0xd) [0xb18b2802] stderr: 3: (md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x69) [0xb195be0a] stderr: 4: (md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool)+0x15d31) [0xb1972ac6] stderr: 5: (CephContext::CephContext(unsigned int, code_environment_t, int)+0x10ef) [0xb193d090] stderr: 6: (common_preinit(CephInitParameters const&, code_environment_t, int)+0x7d) [0xb1956af6] stderr: 7: (()+0x2046a) [0xb639546a] stderr: 8: (rados_create2()+0x55) [0xb639589e] Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
Trying to run the failed command on one of the odroid nodes:
root@odroidxu4:~# /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd Traceback (most recent call last): File "/usr/bin/ceph", line 1273, in <module> retval = main() File "/usr/bin/ceph", line 982, in main conffile=conffile) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread raise Exception("timed out") Exception: timed out
Any clues?
Best Regards,
Oliver _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Regards Kefu Chai
-- Regards Kefu Chai
-- Regards Kefu Chai
Hi again, it took me some time but I figured out that on ubuntu focal there is a more recent version of ceph (15.2.7) available. So I gave it a try and replaced the ceph_argparse.py file but it still stuck running the command: [2021-01-11 23:44:06,340][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c3a567be-08e0-4802-8b08-07d6891de485 any more clues? Am 08.01.2021 um 10:03 schrieb kefu chai:
Oliver Weinmann <oliver.weinmann@me.com>于2021年1月8日 周五04:30写道：
Ok, I replaced the whole file ceph_argparse.py with the patched one from github. Instead of an throwing an error it now seems to be stuck forever. Or am I to impatient? I'm running
I don’t think so. In a healthy cluster, the command should complete in no more than 1 second. I just checked the revision history of ceph_argparse.py, there are a bunch of changes since the release of nautilus. My guess is that the version in master might include some bits not compatible with nautilus? So, I’d suggest only cherry-pick the change in that PR, and try again.
debian buster so this is not the latest ceph release octopus, but nautilus:
root@odroidxu4:~# dpkg -l ceph Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description
+++-==============-=================-============-=================================== ii ceph 14.2.15-3~bpo10+1 armhf distributed storage and file system Am 07.01.2021 um 13:07 schrieb kefu chai:
Oliver Weinmann <oliver.weinmann@me.com>于2021年1月7日 周四16:32写道：
Hi,
thanks for the quick reply. I will test it. Do I have to recompile ceph in order to test it?
No, you just need to apply the change of ceph_argparse.py.
Am 07.01.2021 um 02:13 schrieb kefu chai:
On Thursday, January 7, 2021, Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi,
I have a similar if not the same issue. I run armbian buster on my odroid hc2 which is the same as a xu4 and I get the following error, trying to build a cluster with ceph-ansible:
We have a fix for a similar issue recently. See https://github.com/ceph/ceph/pull/38665. Could you give it a shot? I will backport it to LTS branches if it helps.
ASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] *************************************************** Wednesday 06 January 2021 21:46:44 +0000 (0:00:00.073) 0:02:01.697 ***** fatal: [192.168.2.123]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:02.979200' end: '2021-01-06 22:46:48.049074' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.069874' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> fatal: [odroidxu4]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:03.510973' end: '2021-01-06 22:46:48.514102' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.003129' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4e292c82-bb4d-4581-aead-46ff635fda69 stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: In function 'void md_config_t::set_val_default(ConfigValues&, const ConfigTracker&, const string&, const string&)' thread b0e3a460 time 2021-01-06 22:46:48.357354 stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: 259: FAILED ceph_assert(o) stderr: ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xeb) [0xb18b26a4] stderr: 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0xd) [0xb18b2802] stderr: 3: (md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x69) [0xb195be0a] stderr: 4: (md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool)+0x15d31) [0xb1972ac6] stderr: 5: (CephContext::CephContext(unsigned int, code_environment_t, int)+0x10ef) [0xb193d090] stderr: 6: (common_preinit(CephInitParameters const&, code_environment_t, int)+0x7d) [0xb1956af6] stderr: 7: (()+0x2046a) [0xb639546a] stderr: 8: (rados_create2()+0x55) [0xb639589e] Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
Trying to run the failed command on one of the odroid nodes:
root@odroidxu4:~# /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd Traceback (most recent call last): File "/usr/bin/ceph", line 1273, in <module> retval = main() File "/usr/bin/ceph", line 982, in main conffile=conffile) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread raise Exception("timed out") Exception: timed out
Any clues?
Best Regards,
Oliver _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Regards Kefu Chai
-- Regards Kefu Chai
-- Regards Kefu Chai
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Coud it be a problem that I'm running a mix of x86 and arm? I use vagrant / virtualbox for the mons. Currently I only have two odroid hc-2 devices available. Am 11.01.2021 um 23:48 schrieb Oliver Weinmann:
Hi again,
it took me some time but I figured out that on ubuntu focal there is a more recent version of ceph (15.2.7) available. So I gave it a try and replaced the ceph_argparse.py file but it still stuck running the command:
[2021-01-11 23:44:06,340][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c3a567be-08e0-4802-8b08-07d6891de485
any more clues?
Am 08.01.2021 um 10:03 schrieb kefu chai:
Oliver Weinmann <oliver.weinmann@me.com>于2021年1月8日 周五04:30写道：
Ok, I replaced the whole file ceph_argparse.py with the patched one from github. Instead of an throwing an error it now seems to be stuck forever. Or am I to impatient? I'm running
I don’t think so. In a healthy cluster, the command should complete in no more than 1 second. I just checked the revision history of ceph_argparse.py, there are a bunch of changes since the release of nautilus. My guess is that the version in master might include some bits not compatible with nautilus? So, I’d suggest only cherry-pick the change in that PR, and try again.
debian buster so this is not the latest ceph release octopus, but nautilus:
root@odroidxu4:~# dpkg -l ceph Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description
+++-==============-=================-============-===================================
ii ceph 14.2.15-3~bpo10+1 armhf distributed storage and file system Am 07.01.2021 um 13:07 schrieb kefu chai:
Oliver Weinmann <oliver.weinmann@me.com>于2021年1月7日 周四16:32写道：
Hi,
thanks for the quick reply. I will test it. Do I have to recompile ceph in order to test it?
No, you just need to apply the change of ceph_argparse.py.
Am 07.01.2021 um 02:13 schrieb kefu chai:
On Thursday, January 7, 2021, Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi,
I have a similar if not the same issue. I run armbian buster on my odroid hc2 which is the same as a xu4 and I get the following error, trying to build a cluster with ceph-ansible:
We have a fix for a similar issue recently. See https://github.com/ceph/ceph/pull/38665. Could you give it a shot? I will backport it to LTS branches if it helps.
ASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] *************************************************** Wednesday 06 January 2021 21:46:44 +0000 (0:00:00.073) 0:02:01.697 ***** fatal: [192.168.2.123]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:02.979200' end: '2021-01-06 22:46:48.049074' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.069874' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> fatal: [odroidxu4]: FAILED! => changed=true cmd: - ceph-volume - --cluster - ceph - lvm - batch - --bluestore - --yes - /dev/sda delta: '0:00:03.510973' end: '2021-01-06 22:46:48.514102' msg: non-zero return code rc: 1 start: '2021-01-06 22:46:45.003129' stderr: |- --> DEPRECATION NOTICE --> You are using the legacy automatic disk sorting behavior --> The Pacific release will change the default to --no-auto --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4e292c82-bb4d-4581-aead-46ff635fda69 stderr: Traceback (most recent call last): stderr: File "/usr/bin/ceph", line 1273, in <module> stderr: retval = main() stderr: File "/usr/bin/ceph", line 982, in main stderr: conffile=conffile) stderr: File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread stderr: raise Exception("timed out") stderr: Exception: timed out stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: In function 'void md_config_t::set_val_default(ConfigValues&, const ConfigTracker&, const string&, const string&)' thread b0e3a460 time 2021-01-06 22:46:48.357354 stderr: /build/ceph-Ti7FjJ/ceph-14.2.15/src/common/config.cc: 259: FAILED ceph_assert(o) stderr: ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xeb) [0xb18b26a4] stderr: 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0xd) [0xb18b2802] stderr: 3: (md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x69) [0xb195be0a] stderr: 4: (md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool)+0x15d31) [0xb1972ac6] stderr: 5: (CephContext::CephContext(unsigned int, code_environment_t, int)+0x10ef) [0xb193d090] stderr: 6: (common_preinit(CephInitParameters const&, code_environment_t, int)+0x7d) [0xb1956af6] stderr: 7: (()+0x2046a) [0xb639546a] stderr: 8: (rados_create2()+0x55) [0xb639589e] Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 415, in main self._execute(plan) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute c.create(argparse.Namespace(**args)) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 173, in create_id raise RuntimeError('Unable to create a new OSD id') RuntimeError: Unable to create a new OSD id stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
Trying to run the failed command on one of the odroid nodes:
root@odroidxu4:~# /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8854fc6d-d637-40a9-a1b1-b8e2eeee0afd Traceback (most recent call last): File "/usr/bin/ceph", line 1273, in <module> retval = main() File "/usr/bin/ceph", line 982, in main conffile=conffile) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1320, in run_in_thread raise Exception("timed out") Exception: timed out
Any clues?
Best Regards,
Oliver _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Regards Kefu Chai
-- Regards Kefu Chai
-- Regards Kefu Chai
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Dominik H
-
kefu chai
-
Oliver Weinmann