29 Oct
2019
29 Oct
'19
9:14 p.m.
Hi,
in my unhealthy cluster I cannot run several ceph osd command because
they hang, e.g.
ceph osd df
ceph osd pg dump
Also, ceph balancer status hangs.
How can I fix this issue?
THX
29 Oct
29 Oct
9:52 p.m.
On Oct 29, 2019, at 9:44 AM, Thomas Schneider <74cmonty(a)gmail.com> wrote:
in my unhealthy cluster I cannot run several ceph osd
command because
they hang, e.g.
ceph osd df
ceph osd pg dump
Also, ceph balancer status hangs.
How can I fix this issue?
Check the status of your ceph-mgr processes (restart them if needed and check the logs for
more details). Those are responsible for handling those commands in recent releases.
Bryan
10:08 p.m.
Thanks.
2 of 4 MGR nodes are sick.
I have stopped MGR services on both nodes.
When I start the service again on node A, I get this in its log:
root@ld5508:~# tail -f /var/log/ceph/ceph-mgr.ld5508.log
2019-10-29 17:32:02.024 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.028 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.032 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.040 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.044 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.048 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.052 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.060 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.064 7fe209fe8700 -1 received signal: Terminated
from /sbin/init (PID: 1) UID: 0
2019-10-29 17:32:02.064 7fe209fe8700 -1 mgr handle_signal *** Got signal
Terminated ***
2019-10-29 17:37:54.319 7f0e26fc1dc0 0 set uid:gid to 64045:64045
(ceph:ceph)
2019-10-29 17:37:54.319 7f0e26fc1dc0 0 ceph version 14.2.4
(65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process
ceph-mgr, pid 250399
2019-10-29 17:37:54.319 7f0e26fc1dc0 0 pidfile_write: ignore empty
--pid-file
2019-10-29 17:37:54.331 7f0e26fc1dc0 1 mgr[py] Loading python module
'ansible'
2019-10-29 17:37:54.503 7f0e26fc1dc0 1 mgr[py] Loading python module
'balancer'
2019-10-29 17:37:54.531 7f0e26fc1dc0 1 mgr[py] Loading python module
'crash'
2019-10-29 17:37:54.551 7f0e26fc1dc0 1 mgr[py] Loading python module
'dashboard'
2019-10-29 17:37:54.915 7f0e26fc1dc0 1 mgr[py] Loading python module
'deepsea'
2019-10-29 17:37:55.071 7f0e26fc1dc0 1 mgr[py] Loading python module
'devicehealth'
2019-10-29 17:37:55.103 7f0e26fc1dc0 1 mgr[py] Loading python module
'influx'
2019-10-29 17:37:55.127 7f0e26fc1dc0 1 mgr[py] Loading python module
'insights'
2019-10-29 17:37:55.207 7f0e26fc1dc0 1 mgr[py] Loading python module
'iostat'
2019-10-29 17:37:55.227 7f0e26fc1dc0 1 mgr[py] Loading python module
'localpool'
2019-10-29 17:37:55.247 7f0e26fc1dc0 1 mgr[py] Loading python module
'orchestrator_cli'
2019-10-29 17:37:55.295 7f0e26fc1dc0 1 mgr[py] Loading python module
'pg_autoscaler'
2019-10-29 17:37:55.347 7f0e26fc1dc0 1 mgr[py] Loading python module
'progress'
2019-10-29 17:37:55.387 7f0e26fc1dc0 1 mgr[py] Loading python module
'prometheus'
2019-10-29 17:37:55.599 7f0e26fc1dc0 1 mgr[py] Loading python module
'rbd_support'
2019-10-29 17:37:55.647 7f0e26fc1dc0 1 mgr[py] Loading python module
'restful'
2019-10-29 17:37:55.959 7f0e26fc1dc0 1 mgr[py] Loading python module
'selftest'
2019-10-29 17:37:55.983 7f0e26fc1dc0 1 mgr[py] Loading python module
'status'
2019-10-29 17:37:56.015 7f0e26fc1dc0 1 mgr[py] Loading python module
'telegraf'
2019-10-29 17:37:56.051 7f0e26fc1dc0 1 mgr[py] Loading python module
'telemetry'
2019-10-29 17:37:56.331 7f0e26fc1dc0 1 mgr[py] Loading python module
'test_orchestrator'
2019-10-29 17:37:56.399 7f0e26fc1dc0 1 mgr[py] Loading python module
'volumes'
2019-10-29 17:37:56.459 7f0e26fc1dc0 1 mgr[py] Loading python module
'zabbix'
2019-10-29 17:37:56.503 7f0e21cdd700 1 mgr load Constructed class from
module: dashboard
2019-10-29 17:37:56.503 7f0e214dc700 0 ms_deliver_dispatch: unhandled
message 0x56346f978400 mon_map magic: 0 v1 from mon.0 v2:10.97.206.93:3300/0
2019-10-29 17:37:56.507 7f0e214dc700 0 client.0 ms_handle_reset on
v2:10.97.206.93:6912/22258
2019-10-29 17:37:56.743 7f0e16363700 0 mgr[dashboard]
[29/Oct/2019:17:37:56] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
File
"/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line
2021, in start
self.tick()
File
"/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line
2090, in tick
s, ssl_env = self.ssl_adapter.wrap(s)
File
"/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py",
line 67, in wrap
server_side=True)
File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket
_context=self)
File "/usr/lib/python2.7/ssl.py", line 599, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
error: [Errno 0] Error
^C
This looks like a severe issue.
Am 29.10.2019 um 17:22 schrieb Bryan Stillwell:
On Oct 29, 2019, at 9:44 AM, Thomas Schneider
<74cmonty(a)gmail.com> wrote:
in my unhealthy cluster I cannot run several ceph
osd command because
they hang, e.g.
ceph osd df
ceph osd pg dump
Also, ceph balancer status hangs.
How can I fix this issue?
Check the status of your ceph-mgr processes (restart
them if needed and check the logs for more details). Those are responsible for handling
those commands in recent releases.
Bryan
10:14 p.m.
I would look into a potential network problem. Check for errors on both the server side
and on the switch side.
Otherwise I'm not really sure what's going on. Someone else will have to jump
into the conversation.
Bryan
On Oct 29, 2019, at 10:38 AM, Thomas Schneider <74cmonty(a)gmail.com> wrote:
Notice: This email is from an external sender.
Thanks.
2 of 4 MGR nodes are sick.
I have stopped MGR services on both nodes.
When I start the service again on node A, I get this in its log:
root@ld5508:~# tail -f /var/log/ceph/ceph-mgr.ld5508.log
2019-10-29 17:32:02.024 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.028 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.032 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.040 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.044 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.048 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.052 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.060 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
connect got BADAUTHORIZER
2019-10-29 17:32:02.064 7fe209fe8700 -1 received signal: Terminated
from /sbin/init (PID: 1) UID: 0
2019-10-29 17:32:02.064 7fe209fe8700 -1 mgr handle_signal *** Got signal
Terminated ***
2019-10-29 17:37:54.319 7f0e26fc1dc0 0 set uid:gid to 64045:64045
(ceph:ceph)
2019-10-29 17:37:54.319 7f0e26fc1dc0 0 ceph version 14.2.4
(65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process
ceph-mgr, pid 250399
2019-10-29 17:37:54.319 7f0e26fc1dc0 0 pidfile_write: ignore empty
--pid-file
2019-10-29 17:37:54.331 7f0e26fc1dc0 1 mgr[py] Loading python module
'ansible'
2019-10-29 17:37:54.503 7f0e26fc1dc0 1 mgr[py] Loading python module
'balancer'
2019-10-29 17:37:54.531 7f0e26fc1dc0 1 mgr[py] Loading python module
'crash'
2019-10-29 17:37:54.551 7f0e26fc1dc0 1 mgr[py] Loading python module
'dashboard'
2019-10-29 17:37:54.915 7f0e26fc1dc0 1 mgr[py] Loading python module
'deepsea'
2019-10-29 17:37:55.071 7f0e26fc1dc0 1 mgr[py] Loading python module
'devicehealth'
2019-10-29 17:37:55.103 7f0e26fc1dc0 1 mgr[py] Loading python module
'influx'
2019-10-29 17:37:55.127 7f0e26fc1dc0 1 mgr[py] Loading python module
'insights'
2019-10-29 17:37:55.207 7f0e26fc1dc0 1 mgr[py] Loading python module
'iostat'
2019-10-29 17:37:55.227 7f0e26fc1dc0 1 mgr[py] Loading python module
'localpool'
2019-10-29 17:37:55.247 7f0e26fc1dc0 1 mgr[py] Loading python module
'orchestrator_cli'
2019-10-29 17:37:55.295 7f0e26fc1dc0 1 mgr[py] Loading python module
'pg_autoscaler'
2019-10-29 17:37:55.347 7f0e26fc1dc0 1 mgr[py] Loading python module
'progress'
2019-10-29 17:37:55.387 7f0e26fc1dc0 1 mgr[py] Loading python module
'prometheus'
2019-10-29 17:37:55.599 7f0e26fc1dc0 1 mgr[py] Loading python module
'rbd_support'
2019-10-29 17:37:55.647 7f0e26fc1dc0 1 mgr[py] Loading python module
'restful'
2019-10-29 17:37:55.959 7f0e26fc1dc0 1 mgr[py] Loading python module
'selftest'
2019-10-29 17:37:55.983 7f0e26fc1dc0 1 mgr[py] Loading python module
'status'
2019-10-29 17:37:56.015 7f0e26fc1dc0 1 mgr[py] Loading python module
'telegraf'
2019-10-29 17:37:56.051 7f0e26fc1dc0 1 mgr[py] Loading python module
'telemetry'
2019-10-29 17:37:56.331 7f0e26fc1dc0 1 mgr[py] Loading python module
'test_orchestrator'
2019-10-29 17:37:56.399 7f0e26fc1dc0 1 mgr[py] Loading python module
'volumes'
2019-10-29 17:37:56.459 7f0e26fc1dc0 1 mgr[py] Loading python module
'zabbix'
2019-10-29 17:37:56.503 7f0e21cdd700 1 mgr load Constructed class from
module: dashboard
2019-10-29 17:37:56.503 7f0e214dc700 0 ms_deliver_dispatch: unhandled
message 0x56346f978400 mon_map magic: 0 v1 from mon.0 v2:10.97.206.93:3300/0
2019-10-29 17:37:56.507 7f0e214dc700 0 client.0 ms_handle_reset on
v2:10.97.206.93:6912/22258
2019-10-29 17:37:56.743 7f0e16363700 0 mgr[dashboard]
[29/Oct/2019:17:37:56] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
File
"/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line
2021, in start
self.tick()
File
"/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line
2090, in tick
s, ssl_env = self.ssl_adapter.wrap(s)
File
"/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py",
line 67, in wrap
server_side=True)
File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket
_context=self)
File "/usr/lib/python2.7/ssl.py", line 599, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
error: [Errno 0] Error
^C
This looks like a severe issue.
Am 29.10.2019 um 17:22 schrieb Bryan Stillwell:
On Oct 29, 2019, at 9:44 AM, Thomas Schneider
<74cmonty(a)gmail.com> wrote:
in my unhealthy cluster I cannot run several ceph
osd command because
they hang, e.g.
ceph osd df
ceph osd pg dump
Also, ceph balancer status hangs.
How can I fix this issue?
Check the status of your ceph-mgr processes (restart
them if needed and check the logs for more details). Those are responsible for handling
those commands in recent releases.
Bryan
10:16 p.m.
I have checked the network already.
There's no indication for a problem with the network, means there are no
dropped packages and loadtest with iperf shows good performance.
Am 29.10.2019 um 17:44 schrieb Bryan Stillwell:
I would look into a potential network problem. Check
for errors on both the server side and on the switch side.
Otherwise I'm not really sure what's going on. Someone else will have to jump
into the conversation.
Bryan
On Oct 29, 2019, at 10:38 AM, Thomas Schneider <74cmonty(a)gmail.com> wrote:
> Notice: This email is from an external sender.
>
>
>
> Thanks.
>
> 2 of 4 MGR nodes are sick.
> I have stopped MGR services on both nodes.
>
> When I start the service again on node A, I get this in its log:
> root@ld5508:~# tail -f /var/log/ceph/ceph-mgr.ld5508.log
> 2019-10-29 17:32:02.024 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
> v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
> s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
> connect got BADAUTHORIZER
> 2019-10-29 17:32:02.028 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
> v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
> s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
> connect got BADAUTHORIZER
> 2019-10-29 17:32:02.032 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
> v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
> s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
> connect got BADAUTHORIZER
> 2019-10-29 17:32:02.040 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
> v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
> s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
> connect got BADAUTHORIZER
> 2019-10-29 17:32:02.044 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
> v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
> s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
> connect got BADAUTHORIZER
> 2019-10-29 17:32:02.048 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
> v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
> s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
> connect got BADAUTHORIZER
> 2019-10-29 17:32:02.052 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
> v1:10.97.206.96:7055/17961 conn(0x564582ad5180 0x5645991ca800 :-1
> s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
> connect got BADAUTHORIZER
> 2019-10-29 17:32:02.060 7fe20e881700 0 --1- 10.97.206.96:0/201758478 >>
> v1:10.97.206.96:7055/17961 conn(0x5645977b5180 0x564582b2e800 :-1
> s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2
> connect got BADAUTHORIZER
> 2019-10-29 17:32:02.064 7fe209fe8700 -1 received signal: Terminated
> from /sbin/init (PID: 1) UID: 0
> 2019-10-29 17:32:02.064 7fe209fe8700 -1 mgr handle_signal *** Got signal
> Terminated ***
> 2019-10-29 17:37:54.319 7f0e26fc1dc0 0 set uid:gid to 64045:64045
> (ceph:ceph)
> 2019-10-29 17:37:54.319 7f0e26fc1dc0 0 ceph version 14.2.4
> (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process
> ceph-mgr, pid 250399
> 2019-10-29 17:37:54.319 7f0e26fc1dc0 0 pidfile_write: ignore empty
> --pid-file
> 2019-10-29 17:37:54.331 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'ansible'
> 2019-10-29 17:37:54.503 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'balancer'
> 2019-10-29 17:37:54.531 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'crash'
> 2019-10-29 17:37:54.551 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'dashboard'
> 2019-10-29 17:37:54.915 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'deepsea'
> 2019-10-29 17:37:55.071 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'devicehealth'
> 2019-10-29 17:37:55.103 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'influx'
> 2019-10-29 17:37:55.127 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'insights'
> 2019-10-29 17:37:55.207 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'iostat'
> 2019-10-29 17:37:55.227 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'localpool'
> 2019-10-29 17:37:55.247 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'orchestrator_cli'
> 2019-10-29 17:37:55.295 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'pg_autoscaler'
> 2019-10-29 17:37:55.347 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'progress'
> 2019-10-29 17:37:55.387 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'prometheus'
> 2019-10-29 17:37:55.599 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'rbd_support'
> 2019-10-29 17:37:55.647 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'restful'
> 2019-10-29 17:37:55.959 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'selftest'
> 2019-10-29 17:37:55.983 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'status'
> 2019-10-29 17:37:56.015 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'telegraf'
> 2019-10-29 17:37:56.051 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'telemetry'
> 2019-10-29 17:37:56.331 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'test_orchestrator'
> 2019-10-29 17:37:56.399 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'volumes'
> 2019-10-29 17:37:56.459 7f0e26fc1dc0 1 mgr[py] Loading python module
> 'zabbix'
> 2019-10-29 17:37:56.503 7f0e21cdd700 1 mgr load Constructed class from
> module: dashboard
> 2019-10-29 17:37:56.503 7f0e214dc700 0 ms_deliver_dispatch: unhandled
> message 0x56346f978400 mon_map magic: 0 v1 from mon.0 v2:10.97.206.93:3300/0
> 2019-10-29 17:37:56.507 7f0e214dc700 0 client.0 ms_handle_reset on
> v2:10.97.206.93:6912/22258
> 2019-10-29 17:37:56.743 7f0e16363700 0 mgr[dashboard]
> [29/Oct/2019:17:37:56] ENGINE Error in HTTPServer.tick
> Traceback (most recent call last):
> File
> "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line
> 2021, in start
> self.tick()
> File
> "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line
> 2090, in tick
> s, ssl_env = self.ssl_adapter.wrap(s)
> File
> "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py",
> line 67, in wrap
> server_side=True)
> File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket
> _context=self)
> File "/usr/lib/python2.7/ssl.py", line 599, in __init__
> self.do_handshake()
> File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake
> self._sslobj.do_handshake()
> error: [Errno 0] Error
>
> ^C
>
> This looks like a severe issue.
>
>
> Am 29.10.2019 um 17:22 schrieb Bryan Stillwell:
>> On Oct 29, 2019, at 9:44 AM, Thomas Schneider <74cmonty(a)gmail.com> wrote:
>>> in my unhealthy cluster I cannot run several ceph osd command because
>>> they hang, e.g.
>>> ceph osd df
>>> ceph osd pg dump
>>>
>>> Also, ceph balancer status hangs.
>>>
>>> How can I fix this issue?
>> Check the status of your ceph-mgr processes (restart them if needed and check the
logs for more details). Those are responsible for handling those commands in recent
releases.
>>
>> Bryan
30 Oct
30 Oct
7:16 a.m.
Hi,
May be Mon service have problem, pls check your mon service
Br,
----------------------------------------------
Dương Tuấn Dũng
Email: dungdt.aicgroup(a)gmail.com
ĐT: 0986153686
On Tue, Oct 29, 2019 at 10:45 PM Thomas Schneider <74cmonty(a)gmail.com>
wrote:
Hi,
in my unhealthy cluster I cannot run several ceph osd command because
they hang, e.g.
ceph osd df
ceph osd pg dump
Also, ceph balancer status hangs.
How can I fix this issue?
THX
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1672
days inactive
1673
days old
5 comments
3 participants
participants (3)
-
Bryan Stillwell -
Thomas Schneider -
tuan dung