20 Oct
2020
20 Oct
'20
5:39 p.m.
Hello
I'm facing an issue with ceph. I cannot run any ceph command. It literally hangs. I
need to hit CTRL-C to get this:
^CCluster connection interrupted or timed out
This is on Ubuntu 16.04. Also, I use Graphana with Prometheus to get information from the
cluster, but now there is no data to graph. Any clue?
BQ_BEGIN
cephadm version
BQ_END
BQ_BEGIN
INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version 15.2.4
(7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
BQ_END
cephadm ls
[
{
"style": "cephadm:v1",
"name": "mon.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mon.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"afbe6ef76198bf05ec972e832077849d4a4438bd56f2e177aeb9b11146577baf",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T19:03:16.759730",
"created": "2020-09-04T23:30:30.250336",
"deployed": "2020-09-04T23:48:20.956277",
"configured": "2020-09-04T23:48:22.100283"
},
{
"style": "cephadm:v1",
"name": "mgr.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mgr.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"1737b2cf46310025c0ae853c3b48400320fb35b0443f6ab3ef3d6cbb10f460d8",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T20:43:38.329529",
"created": "2020-09-04T23:30:31.110341",
"deployed": "2020-09-04T23:47:41.604057",
"configured": "2020-09-05T00:00:21.064246"
}
]
Thank you in advance.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
20 Oct
20 Oct
6:32 p.m.
Your mon container seems up and running, have you tried restarting it?
You just have one mon, is that correct? Do you see anything in the logs?
cephadm logs --name mon.osswrkprbe001
How long do you wait until you hit CTRL-C? There's a
connection-timeout option for ceph commands, maybe try a higher timeout?
ceph --connect-timeout 60 status
Is the node hosting the mon showing any issues in dmesg, df -h, syslog, etc.?
Regards,
Eugen
Zitat von Emanuel Alejandro Castelli <ecastelli(a)telecentro.net.ar>ar>:
Hello
I'm facing an issue with ceph. I cannot run any ceph command. It
literally hangs. I need to hit CTRL-C to get this:
^CCluster connection interrupted or timed out
This is on Ubuntu 16.04. Also, I use Graphana with Prometheus to get
information from the cluster, but now there is no data to graph. Any
clue?
BQ_BEGIN
cephadm version
BQ_END
BQ_BEGIN
INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version
15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
BQ_END
cephadm ls
[
{
"style": "cephadm:v1",
"name": "mon.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mon.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"afbe6ef76198bf05ec972e832077849d4a4438bd56f2e177aeb9b11146577baf",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T19:03:16.759730",
"created": "2020-09-04T23:30:30.250336",
"deployed": "2020-09-04T23:48:20.956277",
"configured": "2020-09-04T23:48:22.100283"
},
{
"style": "cephadm:v1",
"name": "mgr.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mgr.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"1737b2cf46310025c0ae853c3b48400320fb35b0443f6ab3ef3d6cbb10f460d8",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T20:43:38.329529",
"created": "2020-09-04T23:30:31.110341",
"deployed": "2020-09-04T23:47:41.604057",
"configured": "2020-09-05T00:00:21.064246"
}
]
Thank you in advance.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 |
ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:54 p.m.
I have 3 MON, I don't know why it's showing only one.
root@osswrkprbe001:~# ceph --connect-timeout 60 status
Cluster connection interrupted or timed out
cephadm logs --name mon.osswrkprbe001 --> Is there any way to go to a specific date?
Because it stars from Oct 4. I want to check from Oct 16 and ahead. I suspect that
something happened that day.
Also, I don't know how to troubleshoot this. I did the same (./cephadm logs --name
mon.osswrkprbe002) in the second MON but it starts the logs from Sep 30. I would need to
check Oct 16 also.
I would appreciate if you can help me with the troubleshooting.
Thank you.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Eugen Block" <eblock(a)nde.ag>
To: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:02:35 AM
Subject: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
Your mon container seems up and running, have you tried restarting it?
You just have one mon, is that correct? Do you see anything in the logs?
cephadm logs --name mon.osswrkprbe001
How long do you wait until you hit CTRL-C? There's a
connection-timeout option for ceph commands, maybe try a higher timeout?
ceph --connect-timeout 60 status
Is the node hosting the mon showing any issues in dmesg, df -h, syslog, etc.?
Regards,
Eugen
Zitat von Emanuel Alejandro Castelli <ecastelli(a)telecentro.net.ar>ar>:
Hello
I'm facing an issue with ceph. I cannot run any ceph command. It
literally hangs. I need to hit CTRL-C to get this:
^CCluster connection interrupted or timed out
This is on Ubuntu 16.04. Also, I use Graphana with Prometheus to get
information from the cluster, but now there is no data to graph. Any
clue?
BQ_BEGIN
cephadm version
BQ_END
BQ_BEGIN
INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version
15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
BQ_END
cephadm ls
[
{
"style": "cephadm:v1",
"name": "mon.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mon.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"afbe6ef76198bf05ec972e832077849d4a4438bd56f2e177aeb9b11146577baf",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T19:03:16.759730",
"created": "2020-09-04T23:30:30.250336",
"deployed": "2020-09-04T23:48:20.956277",
"configured": "2020-09-04T23:48:22.100283"
},
{
"style": "cephadm:v1",
"name": "mgr.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mgr.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"1737b2cf46310025c0ae853c3b48400320fb35b0443f6ab3ef3d6cbb10f460d8",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T20:43:38.329529",
"created": "2020-09-04T23:30:31.110341",
"deployed": "2020-09-04T23:47:41.604057",
"configured": "2020-09-05T00:00:21.064246"
}
]
Thank you in advance.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 |
ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:57 p.m.
From MON1, dmesg I get this:
[3348025.306195] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348033.241973] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348048.089325] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348049.209243] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348050.201209] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348052.185167] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348056.280992] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348064.216703] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348078.808431] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348079.192418] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348080.220345] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348082.232299] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348086.232103] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348094.167722] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348110.411216] libceph: mon0 192.168.14.150:6789 socket closed (con state OPEN)
[3348140.245900] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348141.173884] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348142.229859] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348144.213777] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348148.437674] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348157.397327] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348170.965496] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348172.213118] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348173.205087] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348175.188934] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348179.412719] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348187.348441] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348201.683707] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348202.195745] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348203.187654] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348205.175585] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348209.363409] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348217.299298] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
But....from MON2 I get this:
[5242753.074620] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242761.266727] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242779.959468] libceph: mon0 192.168.14.150:6789 socket closed (con state OPEN)
[5242806.834049] libceph: mon1 192.168.14.151:6789 socket error on write
[5242808.049952] libceph: mon1 192.168.14.151:6789 socket error on write
[5242809.041947] libceph: mon1 192.168.14.151:6789 socket error on write
[5242811.057917] libceph: mon1 192.168.14.151:6789 socket error on write
[5242815.285867] libceph: mon1 192.168.14.151:6789 socket error on write
[5242824.241921] libceph: mon1 192.168.14.151:6789 socket error on write
[5242837.554174] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242838.034339] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242839.026139] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242841.010177] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242845.234101] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242853.169905] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242870.102324] libceph: mon0 192.168.14.150:6789 socket closed (con state OPEN)
[5242901.041812] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242902.033763] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242903.026350] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242905.009497] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242909.233740] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242917.169724] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242931.761103] libceph: mon1 192.168.14.151:6789 socket error on write
[5242932.049095] libceph: mon1 192.168.14.151:6789 socket error on write
[5242933.041234] libceph: mon1 192.168.14.151:6789 socket error on write
[5242935.057066] libceph: mon1 192.168.14.151:6789 socket error on write
[5242939.185290] libceph: mon1 192.168.14.151:6789 socket error on write
[5242947.120965] libceph: mon1 192.168.14.151:6789 socket error on write
[5242962.481572] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242963.025508] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242964.017160] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242966.033128] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Emanuel Alejandro Castelli" <ecastelli(a)telecentro.net.ar>
To: "Eugen Block" <eblock(a)nde.ag>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:24:31 AM
Subject: Re: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
I have 3 MON, I don't know why it's showing only one.
root@osswrkprbe001:~# ceph --connect-timeout 60 status
Cluster connection interrupted or timed out
cephadm logs --name mon.osswrkprbe001 --> Is there any way to go to a specific date?
Because it stars from Oct 4. I want to check from Oct 16 and ahead. I suspect that
something happened that day.
Also, I don't know how to troubleshoot this. I did the same (./cephadm logs --name
mon.osswrkprbe002) in the second MON but it starts the logs from Sep 30. I would need to
check Oct 16 also.
I would appreciate if you can help me with the troubleshooting.
Thank you.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Eugen Block" <eblock(a)nde.ag>
To: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:02:35 AM
Subject: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
Your mon container seems up and running, have you tried restarting it?
You just have one mon, is that correct? Do you see anything in the logs?
cephadm logs --name mon.osswrkprbe001
How long do you wait until you hit CTRL-C? There's a
connection-timeout option for ceph commands, maybe try a higher timeout?
ceph --connect-timeout 60 status
Is the node hosting the mon showing any issues in dmesg, df -h, syslog, etc.?
Regards,
Eugen
Zitat von Emanuel Alejandro Castelli <ecastelli(a)telecentro.net.ar>ar>:
Hello
I'm facing an issue with ceph. I cannot run any ceph command. It
literally hangs. I need to hit CTRL-C to get this:
^CCluster connection interrupted or timed out
This is on Ubuntu 16.04. Also, I use Graphana with Prometheus to get
information from the cluster, but now there is no data to graph. Any
clue?
BQ_BEGIN
cephadm version
BQ_END
BQ_BEGIN
INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version
15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
BQ_END
cephadm ls
[
{
"style": "cephadm:v1",
"name": "mon.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mon.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"afbe6ef76198bf05ec972e832077849d4a4438bd56f2e177aeb9b11146577baf",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T19:03:16.759730",
"created": "2020-09-04T23:30:30.250336",
"deployed": "2020-09-04T23:48:20.956277",
"configured": "2020-09-04T23:48:22.100283"
},
{
"style": "cephadm:v1",
"name": "mgr.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mgr.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"1737b2cf46310025c0ae853c3b48400320fb35b0443f6ab3ef3d6cbb10f460d8",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T20:43:38.329529",
"created": "2020-09-04T23:30:31.110341",
"deployed": "2020-09-04T23:47:41.604057",
"configured": "2020-09-05T00:00:21.064246"
}
]
Thank you in advance.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 |
ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:59 p.m.
And the same for MON3
[5243018.443159] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243033.801504] libceph: mon2 192.168.14.152:6789 socket error on write
[5243034.473450] libceph: mon2 192.168.14.152:6789 socket error on write
[5243035.497397] libceph: mon2 192.168.14.152:6789 socket error on write
[5243037.481225] libceph: mon2 192.168.14.152:6789 socket error on write
[5243041.480864] libceph: mon2 192.168.14.152:6789 socket error on write
[5243049.672236] libceph: mon2 192.168.14.152:6789 socket error on write
[5243064.519492] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243065.479388] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243066.471478] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243068.455281] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243072.454806] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243080.646202] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243095.236682] libceph: mon2 192.168.14.152:6789 socket error on write
[5243096.484606] libceph: mon2 192.168.14.152:6789 socket error on write
[5243097.476518] libceph: mon2 192.168.14.152:6789 socket error on write
[5243099.492380] libceph: mon2 192.168.14.152:6789 socket error on write
[5243103.684014] libceph: mon2 192.168.14.152:6789 socket error on write
[5243111.619439] libceph: mon2 192.168.14.152:6789 socket error on write
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Emanuel Alejandro Castelli" <ecastelli(a)telecentro.net.ar>
To: "Eugen Block" <eblock(a)nde.ag>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:27:15 AM
Subject: Re: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
From MON1, dmesg I get this:
[3348025.306195] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348033.241973] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348048.089325] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348049.209243] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348050.201209] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348052.185167] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348056.280992] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348064.216703] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348078.808431] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348079.192418] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348080.220345] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348082.232299] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348086.232103] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348094.167722] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348110.411216] libceph: mon0 192.168.14.150:6789 socket closed (con state OPEN)
[3348140.245900] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348141.173884] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348142.229859] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348144.213777] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348148.437674] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348157.397327] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348170.965496] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348172.213118] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348173.205087] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348175.188934] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348179.412719] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348187.348441] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348201.683707] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348202.195745] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348203.187654] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348205.175585] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348209.363409] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348217.299298] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
But....from MON2 I get this:
[5242753.074620] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242761.266727] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242779.959468] libceph: mon0 192.168.14.150:6789 socket closed (con state OPEN)
[5242806.834049] libceph: mon1 192.168.14.151:6789 socket error on write
[5242808.049952] libceph: mon1 192.168.14.151:6789 socket error on write
[5242809.041947] libceph: mon1 192.168.14.151:6789 socket error on write
[5242811.057917] libceph: mon1 192.168.14.151:6789 socket error on write
[5242815.285867] libceph: mon1 192.168.14.151:6789 socket error on write
[5242824.241921] libceph: mon1 192.168.14.151:6789 socket error on write
[5242837.554174] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242838.034339] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242839.026139] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242841.010177] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242845.234101] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242853.169905] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242870.102324] libceph: mon0 192.168.14.150:6789 socket closed (con state OPEN)
[5242901.041812] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242902.033763] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242903.026350] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242905.009497] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242909.233740] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242917.169724] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242931.761103] libceph: mon1 192.168.14.151:6789 socket error on write
[5242932.049095] libceph: mon1 192.168.14.151:6789 socket error on write
[5242933.041234] libceph: mon1 192.168.14.151:6789 socket error on write
[5242935.057066] libceph: mon1 192.168.14.151:6789 socket error on write
[5242939.185290] libceph: mon1 192.168.14.151:6789 socket error on write
[5242947.120965] libceph: mon1 192.168.14.151:6789 socket error on write
[5242962.481572] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242963.025508] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242964.017160] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242966.033128] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Emanuel Alejandro Castelli" <ecastelli(a)telecentro.net.ar>
To: "Eugen Block" <eblock(a)nde.ag>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:24:31 AM
Subject: Re: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
I have 3 MON, I don't know why it's showing only one.
root@osswrkprbe001:~# ceph --connect-timeout 60 status
Cluster connection interrupted or timed out
cephadm logs --name mon.osswrkprbe001 --> Is there any way to go to a specific date?
Because it stars from Oct 4. I want to check from Oct 16 and ahead. I suspect that
something happened that day.
Also, I don't know how to troubleshoot this. I did the same (./cephadm logs --name
mon.osswrkprbe002) in the second MON but it starts the logs from Sep 30. I would need to
check Oct 16 also.
I would appreciate if you can help me with the troubleshooting.
Thank you.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Eugen Block" <eblock(a)nde.ag>
To: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:02:35 AM
Subject: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
Your mon container seems up and running, have you tried restarting it?
You just have one mon, is that correct? Do you see anything in the logs?
cephadm logs --name mon.osswrkprbe001
How long do you wait until you hit CTRL-C? There's a
connection-timeout option for ceph commands, maybe try a higher timeout?
ceph --connect-timeout 60 status
Is the node hosting the mon showing any issues in dmesg, df -h, syslog, etc.?
Regards,
Eugen
Zitat von Emanuel Alejandro Castelli <ecastelli(a)telecentro.net.ar>ar>:
Hello
I'm facing an issue with ceph. I cannot run any ceph command. It
literally hangs. I need to hit CTRL-C to get this:
^CCluster connection interrupted or timed out
This is on Ubuntu 16.04. Also, I use Graphana with Prometheus to get
information from the cluster, but now there is no data to graph. Any
clue?
BQ_BEGIN
cephadm version
BQ_END
BQ_BEGIN
INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version
15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
BQ_END
cephadm ls
[
{
"style": "cephadm:v1",
"name": "mon.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mon.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"afbe6ef76198bf05ec972e832077849d4a4438bd56f2e177aeb9b11146577baf",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T19:03:16.759730",
"created": "2020-09-04T23:30:30.250336",
"deployed": "2020-09-04T23:48:20.956277",
"configured": "2020-09-04T23:48:22.100283"
},
{
"style": "cephadm:v1",
"name": "mgr.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mgr.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"1737b2cf46310025c0ae853c3b48400320fb35b0443f6ab3ef3d6cbb10f460d8",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T20:43:38.329529",
"created": "2020-09-04T23:30:31.110341",
"deployed": "2020-09-04T23:47:41.604057",
"configured": "2020-09-05T00:00:21.064246"
}
]
Thank you in advance.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 |
ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:34 p.m.
Hello Eugen,
Rebooted the other two MONs fixed the problem
root@osswrkprbe001:~# ceph status
cluster:
id: 56820176-ae5b-4e58-84a2-442b2fc03e6d
health: HEALTH_OK
services:
mon: 3 daemons, quorum osswrkprbe001,osswrkprbe002,osswrkprbe003 (age 3m)
mgr: osswrkprbe002(active, since 6m), standbys: osswrkprbe001, osswrkprbe003
osd: 3 osds: 3 up, 3 in
data:
pools: 6 pools, 162 pgs
objects: 16.27k objects, 58 GiB
usage: 173 GiB used, 2.0 TiB / 2.2 TiB avail
pgs: 162 active+clean
io:
client: 341 B/s rd, 473 KiB/s wr, 0 op/s rd, 43 op/s wr
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Emanuel Alejandro Castelli" <ecastelli(a)telecentro.net.ar>
To: "Eugen Block" <eblock(a)nde.ag>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:29:05 AM
Subject: Re: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
And the same for MON3
[5243018.443159] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243033.801504] libceph: mon2 192.168.14.152:6789 socket error on write
[5243034.473450] libceph: mon2 192.168.14.152:6789 socket error on write
[5243035.497397] libceph: mon2 192.168.14.152:6789 socket error on write
[5243037.481225] libceph: mon2 192.168.14.152:6789 socket error on write
[5243041.480864] libceph: mon2 192.168.14.152:6789 socket error on write
[5243049.672236] libceph: mon2 192.168.14.152:6789 socket error on write
[5243064.519492] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243065.479388] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243066.471478] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243068.455281] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243072.454806] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243080.646202] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[5243095.236682] libceph: mon2 192.168.14.152:6789 socket error on write
[5243096.484606] libceph: mon2 192.168.14.152:6789 socket error on write
[5243097.476518] libceph: mon2 192.168.14.152:6789 socket error on write
[5243099.492380] libceph: mon2 192.168.14.152:6789 socket error on write
[5243103.684014] libceph: mon2 192.168.14.152:6789 socket error on write
[5243111.619439] libceph: mon2 192.168.14.152:6789 socket error on write
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Emanuel Alejandro Castelli" <ecastelli(a)telecentro.net.ar>
To: "Eugen Block" <eblock(a)nde.ag>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:27:15 AM
Subject: Re: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
From MON1, dmesg I get this:
[3348025.306195] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348033.241973] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348048.089325] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348049.209243] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348050.201209] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348052.185167] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348056.280992] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348064.216703] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348078.808431] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348079.192418] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348080.220345] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348082.232299] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348086.232103] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348094.167722] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348110.411216] libceph: mon0 192.168.14.150:6789 socket closed (con state OPEN)
[3348140.245900] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348141.173884] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348142.229859] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348144.213777] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348148.437674] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348157.397327] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348170.965496] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348172.213118] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348173.205087] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348175.188934] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348179.412719] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348187.348441] libceph: mon1 192.168.14.151:6789 socket closed (con state CONNECTING)
[3348201.683707] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348202.195745] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348203.187654] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348205.175585] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348209.363409] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[3348217.299298] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
But....from MON2 I get this:
[5242753.074620] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242761.266727] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242779.959468] libceph: mon0 192.168.14.150:6789 socket closed (con state OPEN)
[5242806.834049] libceph: mon1 192.168.14.151:6789 socket error on write
[5242808.049952] libceph: mon1 192.168.14.151:6789 socket error on write
[5242809.041947] libceph: mon1 192.168.14.151:6789 socket error on write
[5242811.057917] libceph: mon1 192.168.14.151:6789 socket error on write
[5242815.285867] libceph: mon1 192.168.14.151:6789 socket error on write
[5242824.241921] libceph: mon1 192.168.14.151:6789 socket error on write
[5242837.554174] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242838.034339] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242839.026139] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242841.010177] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242845.234101] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242853.169905] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242870.102324] libceph: mon0 192.168.14.150:6789 socket closed (con state OPEN)
[5242901.041812] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242902.033763] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242903.026350] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242905.009497] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242909.233740] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242917.169724] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242931.761103] libceph: mon1 192.168.14.151:6789 socket error on write
[5242932.049095] libceph: mon1 192.168.14.151:6789 socket error on write
[5242933.041234] libceph: mon1 192.168.14.151:6789 socket error on write
[5242935.057066] libceph: mon1 192.168.14.151:6789 socket error on write
[5242939.185290] libceph: mon1 192.168.14.151:6789 socket error on write
[5242947.120965] libceph: mon1 192.168.14.151:6789 socket error on write
[5242962.481572] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242963.025508] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242964.017160] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
[5242966.033128] libceph: mon2 192.168.14.152:6789 socket closed (con state CONNECTING)
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Emanuel Alejandro Castelli" <ecastelli(a)telecentro.net.ar>
To: "Eugen Block" <eblock(a)nde.ag>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:24:31 AM
Subject: Re: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
I have 3 MON, I don't know why it's showing only one.
root@osswrkprbe001:~# ceph --connect-timeout 60 status
Cluster connection interrupted or timed out
cephadm logs --name mon.osswrkprbe001 --> Is there any way to go to a specific date?
Because it stars from Oct 4. I want to check from Oct 16 and ahead. I suspect that
something happened that day.
Also, I don't know how to troubleshoot this. I did the same (./cephadm logs --name
mon.osswrkprbe002) in the second MON but it starts the logs from Sep 30. I would need to
check Oct 16 also.
I would appreciate if you can help me with the troubleshooting.
Thank you.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Eugen Block" <eblock(a)nde.ag>
To: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:02:35 AM
Subject: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
Your mon container seems up and running, have you tried restarting it?
You just have one mon, is that correct? Do you see anything in the logs?
cephadm logs --name mon.osswrkprbe001
How long do you wait until you hit CTRL-C? There's a
connection-timeout option for ceph commands, maybe try a higher timeout?
ceph --connect-timeout 60 status
Is the node hosting the mon showing any issues in dmesg, df -h, syslog, etc.?
Regards,
Eugen
Zitat von Emanuel Alejandro Castelli <ecastelli(a)telecentro.net.ar>ar>:
Hello
I'm facing an issue with ceph. I cannot run any ceph command. It
literally hangs. I need to hit CTRL-C to get this:
^CCluster connection interrupted or timed out
This is on Ubuntu 16.04. Also, I use Graphana with Prometheus to get
information from the cluster, but now there is no data to graph. Any
clue?
BQ_BEGIN
cephadm version
BQ_END
BQ_BEGIN
INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version
15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
BQ_END
cephadm ls
[
{
"style": "cephadm:v1",
"name": "mon.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mon.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"afbe6ef76198bf05ec972e832077849d4a4438bd56f2e177aeb9b11146577baf",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T19:03:16.759730",
"created": "2020-09-04T23:30:30.250336",
"deployed": "2020-09-04T23:48:20.956277",
"configured": "2020-09-04T23:48:22.100283"
},
{
"style": "cephadm:v1",
"name": "mgr.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mgr.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"1737b2cf46310025c0ae853c3b48400320fb35b0443f6ab3ef3d6cbb10f460d8",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T20:43:38.329529",
"created": "2020-09-04T23:30:31.110341",
"deployed": "2020-09-04T23:47:41.604057",
"configured": "2020-09-05T00:00:21.064246"
}
]
Thank you in advance.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 |
ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:05 p.m.
Your 'cephadm ls' output was only from one node, I assumed you just
bootstrapped the first node.
The 'cephadm logs' command should provide pager-output so you can
scroll or search for a specific date.
I'm not sure what caused this but "error on write" is bad. As I
already wrote check the filesystems on your nodes, dmesg etc. It seems
as if two of your MONs are down which would make your cluster
unavailable (no quorum). Is mon3 up and running? Bringing back one of
the other two MONs would bring the cluster back up.
Zitat von Emanuel Alejandro Castelli <ecastelli(a)telecentro.net.ar>ar>:
From MON1, dmesg I get this:
[3348025.306195] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348033.241973] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348048.089325] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348049.209243] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348050.201209] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348052.185167] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348056.280992] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348064.216703] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348078.808431] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348079.192418] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348080.220345] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348082.232299] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348086.232103] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348094.167722] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348110.411216] libceph: mon0 192.168.14.150:6789 socket closed
(con state OPEN)
[3348140.245900] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348141.173884] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348142.229859] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348144.213777] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348148.437674] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348157.397327] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348170.965496] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348172.213118] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348173.205087] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348175.188934] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348179.412719] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348187.348441] libceph: mon1 192.168.14.151:6789 socket closed
(con state CONNECTING)
[3348201.683707] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348202.195745] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348203.187654] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348205.175585] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348209.363409] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[3348217.299298] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
But....from MON2 I get this:
[5242753.074620] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242761.266727] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242779.959468] libceph: mon0 192.168.14.150:6789 socket closed
(con state OPEN)
[5242806.834049] libceph: mon1 192.168.14.151:6789 socket error on write
[5242808.049952] libceph: mon1 192.168.14.151:6789 socket error on write
[5242809.041947] libceph: mon1 192.168.14.151:6789 socket error on write
[5242811.057917] libceph: mon1 192.168.14.151:6789 socket error on write
[5242815.285867] libceph: mon1 192.168.14.151:6789 socket error on write
[5242824.241921] libceph: mon1 192.168.14.151:6789 socket error on write
[5242837.554174] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242838.034339] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242839.026139] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242841.010177] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242845.234101] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242853.169905] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242870.102324] libceph: mon0 192.168.14.150:6789 socket closed
(con state OPEN)
[5242901.041812] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242902.033763] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242903.026350] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242905.009497] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242909.233740] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242917.169724] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242931.761103] libceph: mon1 192.168.14.151:6789 socket error on write
[5242932.049095] libceph: mon1 192.168.14.151:6789 socket error on write
[5242933.041234] libceph: mon1 192.168.14.151:6789 socket error on write
[5242935.057066] libceph: mon1 192.168.14.151:6789 socket error on write
[5242939.185290] libceph: mon1 192.168.14.151:6789 socket error on write
[5242947.120965] libceph: mon1 192.168.14.151:6789 socket error on write
[5242962.481572] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242963.025508] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242964.017160] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
[5242966.033128] libceph: mon2 192.168.14.152:6789 socket closed
(con state CONNECTING)
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 |
ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Emanuel Alejandro Castelli" <ecastelli(a)telecentro.net.ar>
To: "Eugen Block" <eblock(a)nde.ag>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:24:31 AM
Subject: Re: [ceph-users] Re: Problems with ceph command - Octupus -
Ubuntu 16.04
I have 3 MON, I don't know why it's showing only one.
root@osswrkprbe001:~# ceph --connect-timeout 60 status
Cluster connection interrupted or timed out
cephadm logs --name mon.osswrkprbe001 --> Is there any way to go to
a specific date? Because it stars from Oct 4. I want to check from
Oct 16 and ahead. I suspect that something happened that day.
Also, I don't know how to troubleshoot this. I did the same
(./cephadm logs --name mon.osswrkprbe002) in the second MON but it
starts the logs from Sep 30. I would need to check Oct 16 also.
I would appreciate if you can help me with the troubleshooting.
Thank you.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 |
ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
----- Original Message -----
From: "Eugen Block" <eblock(a)nde.ag>
To: "ceph-users" <ceph-users(a)ceph.io>
Sent: Tuesday, October 20, 2020 10:02:35 AM
Subject: [ceph-users] Re: Problems with ceph command - Octupus - Ubuntu 16.04
Your mon container seems up and running, have you tried restarting it?
You just have one mon, is that correct? Do you see anything in the logs?
cephadm logs --name mon.osswrkprbe001
How long do you wait until you hit CTRL-C? There's a
connection-timeout option for ceph commands, maybe try a higher timeout?
ceph --connect-timeout 60 status
Is the node hosting the mon showing any issues in dmesg, df -h, syslog, etc.?
Regards,
Eugen
Zitat von Emanuel Alejandro Castelli <ecastelli(a)telecentro.net.ar>ar>:
Hello
I'm facing an issue with ceph. I cannot run any ceph command. It
literally hangs. I need to hit CTRL-C to get this:
^CCluster connection interrupted or timed out
This is on Ubuntu 16.04. Also, I use Graphana with Prometheus to get
information from the cluster, but now there is no data to graph. Any
clue?
BQ_BEGIN
cephadm version
BQ_END
BQ_BEGIN
INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version
15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
BQ_END
cephadm ls
[
{
"style": "cephadm:v1",
"name": "mon.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mon.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"afbe6ef76198bf05ec972e832077849d4a4438bd56f2e177aeb9b11146577baf",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T19:03:16.759730",
"created": "2020-09-04T23:30:30.250336",
"deployed": "2020-09-04T23:48:20.956277",
"configured": "2020-09-04T23:48:22.100283"
},
{
"style": "cephadm:v1",
"name": "mgr.osswrkprbe001",
"fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d",
"systemd_unit":
"ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mgr.osswrkprbe001"e001",
"enabled": true,
"state": "running",
"container_id":
"1737b2cf46310025c0ae853c3b48400320fb35b0443f6ab3ef3d6cbb10f460d8",
"container_image_name": "docker.io/ceph/ceph:v15.2.1",
"container_image_id":
"bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2",
"version": "15.2.1",
"started": "2020-10-19T20:43:38.329529",
"created": "2020-09-04T23:30:31.110341",
"deployed": "2020-09-04T23:47:41.604057",
"configured": "2020-09-05T00:00:21.064246"
}
]
Thank you in advance.
Saludos,
EMANUEL CASTELLI
Arquitecto de Información - Gerencia OSS
C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 |
ecastelli(a)telecentro.net.ar
Lavardén 157 1er piso. CABA (C1437FBC)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1303
days inactive
1303
days old
6 comments
2 participants
participants (2)
-
Emanuel Alejandro Castelli -
Eugen Block