(Don't drop the list in your responses) There's a lot going on in the cephadm.log, it's not clear which of these are relevant to the issue. This one looks bad though: ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not appear to be an IPv4 or IPv6 address Did you build this cluster with 19.2.1 or did you upgrade? Do you have the output from 'ceph orch host ls' lying around (I know, currently it's not working)? Are you using IPv6? I haven't seen this issue yet. Zitat von Moritz Baumann <mo@mo.homeip.net>:
Am 09.03.2025 um 15:22 schrieb Moritz Baumann:
I dont want to sound stupid, but how do I get the log?
Am 09.03.2025 um 11:11 schrieb Eugen Block:
ceph versions
ceph01[22]:~# cephadm shell -- ceph log last cephadm Inferring fsid 4cfb1138-d2a5-11ef-87eb-1c34da408e52 Inferring config /var/lib/ceph/4cfb1138-d2a5-11ef-87eb-1c34da408e52/mon.ceph01/config Not using image 'f2efb0401a30ec7eda97b6da76b314bd081fcb910cc5dcd826bc7c72c9dfdd7d' as it's not in list of non-dangling images with ceph=True label 2025-02-25T08:42:54.590667+0000 mgr.ceph01.ombogc (mgr.180813) 1706 : cephadm [INF] maintenance mode request for ceph05.inf.ethz.ch has SET the noout group 2025-02-25T08:43:38.380838+0000 mgr.ceph01.ombogc (mgr.180813) 1729 : cephadm [ERR] Detected invalid grafana certificate on host ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', 'no start line')] 2025-02-25T08:44:40.343420+0000 mgr.ceph01.ombogc (mgr.180813) 1762 : cephadm [INF] Adjusting osd_memory_target on ceph04.inf.ethz.ch to 33423M 2025-02-25T08:44:40.860933+0000 mgr.ceph01.ombogc (mgr.180813) 1764 : cephadm [ERR] Detected invalid grafana certificate on host ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', 'no start line')] 2025-02-25T08:45:41.993564+0000 mgr.ceph01.ombogc (mgr.180813) 1797 : cephadm [ERR] Detected invalid grafana certificate on host ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', 'no start line')] 2025-02-25T08:46:42.758717+0000 mgr.ceph01.ombogc (mgr.180813) 1830 : cephadm [ERR] Detected invalid grafana certificate on host ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', 'no start line')] 2025-02-25T08:47:44.190946+0000 mgr.ceph01.ombogc (mgr.180813) 1864 : cephadm [ERR] Unable to reach remote host ceph05.inf.ethz.ch. SSH connection closed Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 245, in _execute_command r = await conn.run(str(rcmd), input=stdin) File "/lib/python3.9/site-packages/asyncssh/connection.py", line 4250, in run process = await self.create_process(*args, **kwargs) # type: ignore File "/lib/python3.9/site-packages/asyncssh/connection.py", line 4128, in create_process chan, process = await self.create_session( File "/lib/python3.9/site-packages/asyncssh/connection.py", line 4018, in create_session chan = SSHClientChannel(self, self._loop, encoding, errors, File "/lib/python3.9/site-packages/asyncssh/channel.py", line 1085, in __init__ super().__init__(conn, loop, encoding, errors, window, max_pktsize) File "/lib/python3.9/site-packages/asyncssh/channel.py", line 140, in __init__ self._recv_chan: Optional[int] = conn.add_channel(self) File "/lib/python3.9/site-packages/asyncssh/connection.py", line 1307, in add_channel raise ChannelOpenError(OPEN_CONNECT_FAILED, asyncssh.misc.ChannelOpenError: SSH connection closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/cephadm/module.py", line 2224, in exit_host_maintenance outs, errs, _code = self.wait_async( File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/serve.py", line 1669, in _run_cephadm python = await self.mgr.ssh._check_execute_command(host, cmd, addr=addr) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 300, in _check_execute_command out, err, code = await self._execute_command(host, cmd, stdin, addr, log_command) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 253, in _execute_command raise HostConnectionError(f'Unable to reach remote host {host}. {str(e)}', host, address) cephadm.ssh.HostConnectionError: Unable to reach remote host ceph05.inf.ethz.ch. SSH connection closed 2025-02-25T08:47:44.762770+0000 mgr.ceph01.ombogc (mgr.180813) 1865 : cephadm [ERR] Detected invalid grafana certificate on host ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', 'no start line')] 2025-02-25T08:47:55.631018+0000 mgr.ceph01.ombogc (mgr.180813) 1874 : cephadm [INF] exit maintenance request has UNSET for the noout group on host ceph05.inf.ethz.ch 2025-02-25T08:48:47.623644+0000 mgr.ceph01.ombogc (mgr.180813) 1901 : cephadm [INF] Adjusting osd_memory_target on ceph05.inf.ethz.ch to 32911M 2025-02-25T08:48:47.827026+0000 mgr.ceph01.ombogc (mgr.180813) 1903 : cephadm [ERR] Detected invalid grafana certificate on host ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', 'no start line')] 2025-02-25T08:49:21.108932+0000 mgr.ceph01.ombogc (mgr.180813) 1925 : cephadm [INF] Schedule redeploy daemon grafana.ceph05 2025-02-25T08:49:21.292879+0000 mgr.ceph01.ombogc (mgr.180813) 1926 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2025-02-25T08:49:21.328348+0000 mgr.ceph01.ombogc (mgr.180813) 1929 : cephadm [INF] Deploying daemon grafana.ceph05 on ceph05.inf.ethz.ch 2025-02-25T09:07:00.824841+0000 mgr.ceph01.ombogc (mgr.180813) 2461 : cephadm [INF] Detected new or changed devices on ceph05.inf.ethz.ch 2025-02-25T09:07:01.376245+0000 mgr.ceph01.ombogc (mgr.180813) 2462 : cephadm [INF] Detected new or changed devices on ceph04.inf.ethz.ch 2025-02-28T06:18:29.439683+0000 mgr.ceph01.ombogc (mgr.180813) 137418 : cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not appear to be an IPv4 or IPv6 address') Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1281, in communicate req.parse_request() File "/lib/python3.9/site-packages/cheroot/server.py", line 714, in parse_request success = self.read_request_line() File "/lib/python3.9/site-packages/cheroot/server.py", line 815, in read_request_line scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri) File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit _check_bracketed_host(bracketed_host) File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4 File "/lib64/python3.9/ipaddress.py", line 53, in ip_address raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address') ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not appear to be an IPv4 or IPv6 address
2025-02-28T06:18:29.441452+0000 mgr.ceph01.ombogc (mgr.180813) 137419 : cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not appear to be an IPv4 or IPv6 address') Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1281, in communicate req.parse_request() File "/lib/python3.9/site-packages/cheroot/server.py", line 714, in parse_request success = self.read_request_line() File "/lib/python3.9/site-packages/cheroot/server.py", line 815, in read_request_line scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri) File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit _check_bracketed_host(bracketed_host) File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4 File "/lib64/python3.9/ipaddress.py", line 53, in ip_address raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address') ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not appear to be an IPv4 or IPv6 address
2025-03-07T06:17:28.282574+0000 mgr.ceph01.ombogc (mgr.180813) 447472 : cephadm [ERR] [07/Mar/2025:06:17:28] ENGINE ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not appear to be an IPv4 or IPv6 address') Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1281, in communicate req.parse_request() File "/lib/python3.9/site-packages/cheroot/server.py", line 714, in parse_request success = self.read_request_line() File "/lib/python3.9/site-packages/cheroot/server.py", line 815, in read_request_line scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri) File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit _check_bracketed_host(bracketed_host) File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4 File "/lib64/python3.9/ipaddress.py", line 53, in ip_address raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address') ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not appear to be an IPv4 or IPv6 address
2025-03-09T07:36:00.522250+0000 mgr.ceph01.ombogc (mgr.180813) 536135 : cephadm [ERR] ALERT: Cannot stop active Mgr daemon, Please switch active Mgrs with 'ceph mgr fail ceph01.ombogc' Note: Warnings can be bypassed with the --force flag Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/cephadm/module.py", line 2165, in enter_host_maintenance raise OrchestratorError( orchestrator._interface.OrchestratorError: ALERT: Cannot stop active Mgr daemon, Please switch active Mgrs with 'ceph mgr fail ceph01.ombogc' Note: Warnings can be bypassed with the --force flag