6 May
2021
6 May
'21
6:33 p.m.
Hello,
I have a small 6 nodes Octopus 15.2.11 cluster installed on bare metal with cephadm and I
added a second OSD to one of my 3 OSD nodes. I started then copying data to my ceph fs
mounted with kernel mount but then both OSDs on that specific nodes crashed.
To this topic I have the following questions:
1) How can I find out why the two OSD crashed? because everything is in podman containers
I don't know where are the logs to find out the reason why this happened. From the OS
itself everything looks ok, there was no out of memory error.
2) I would assume the two OSD container would restart on their own but this is not the
case it looks like. How can I restart manually these 2 OSD containers on that node? I
believe this should be a "cephadm orch" command?
The health of the cluster right now is:
CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)
PG_DEGRADED: Degraded data redundancy: 132518/397554 objects degraded (33.333%), 65
pgs degraded, 65 pgs undersized
Thank your for your hints.
Best regards,
Mabi
6 May
6 May
6:47 p.m.
On 05/06 14:03, mabi wrote:
Hello,
I have a small 6 nodes Octopus 15.2.11 cluster installed on bare metal with cephadm and I
added a second OSD to one of my 3 OSD nodes. I started then copying data to my ceph fs
mounted with kernel mount but then both OSDs on that specific nodes crashed.
To this topic I have the following questions:
1) How can I find out why the two OSD crashed? because everything is in podman containers
I don't know where are the logs to find out the reason why this happened. From the OS
itself everything looks ok, there was no out of memory error.
There should be some logs under /var/log/ceph/<cluster_fsid>/osd.<osd_id>/ on
the host/hosts that were running the osds.
I found myself sometimes though disabling the '--rm' flag for the pod in the
'unit.run' script under
/va/lib/ceph/<ceph_fsid>/osd.<id>/unit.run to make podman persist the
container and be able to do a 'podman logs' on it.
Though that's probably sensible only when debugging.
2) I would assume the two OSD container would restart on their own but this is not the
case it looks like. How can I restart manually these 2 OSD containers on that node? I
believe this should be a "cephadm orch" command?
I think 'ceph orch daemon redeploy' might do it? What is the output of 'ceph
orch ls' and 'ceph orch ps'?
The health of the cluster right now is:
CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)
PG_DEGRADED: Degraded data redundancy: 132518/397554 objects degraded (33.333%), 65
pgs degraded, 65 pgs undersized
Thank your for your hints.
Best regards,
Mabi
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
8:58 p.m.
Thank you very much for the hint regarding the log files, I wasn't aware that it still
saves the logs on the host although everything is running in containers nowadays.
So there was nothing in the log files but I could find out that finally the host (a
RasPi4) could not cope with 2 SSD external USB disks connected to it. Probably due to not
enough power, so the disks disappeared and the OSD went away with them. After a restart of
the host the disks where back as well as the OSD containers. So I have now remove that
second OSD and will keep only one single OSD per server.
For reference here is the relevant part of the kernel log I saw:
[Thu May 6 15:24:34 2021] blk_update_request: I/O error, dev sda, sector 40063143 op
0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[Thu May 6 15:24:34 2021] usb 1-1-port4: over-current change #1
and of course it did that for both sda and sdb.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, May 6, 2021 4:17 PM, David Caro <dcaro(a)wikimedia.org> wrote:
On 05/06 14:03, mabi wrote:
Hello,
I have a small 6 nodes Octopus 15.2.11 cluster installed on bare metal with cephadm and I
added a second OSD to one of my 3 OSD nodes. I started then copying data to my ceph fs
mounted with kernel mount but then both OSDs on that specific nodes crashed.
To this topic I have the following questions:
1. How can I find out why the two OSD crashed? because everything is in podman
containers I don't know where are the logs to find out the reason why this happened.
From the OS itself everything looks ok, there was no out of memory error.
There should be some logs under /var/log/ceph/<cluster_fsid>/osd.<osd_id>/ on
the host/hosts that were running the osds.
I found myself sometimes though disabling the '--rm' flag for the pod in the
'unit.run' script under
/va/lib/ceph/<ceph_fsid>/osd.<id>/unit.run to make podman persist the
container and be able to do a 'podman logs' on it.
Though that's probably sensible only when debugging.
2. I would assume the two OSD container would
restart on their own but this is not the case it looks like. How can I restart manually
these 2 OSD containers on that node? I believe this should be a "cephadm orch"
command?
I think 'ceph orch daemon redeploy' might do it? What is the output of 'ceph
orch ls' and 'ceph orch ps'?
The health of the cluster right now is:
CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)
PG_DEGRADED: Degraded data redundancy: 132518/397554 objects degraded (33.333%), 65
pgs degraded, 65 pgs undersized
Thank your for your hints.
Best regards,
Mabi
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
David Caro
SRE - Cloud Services
Wikimedia Foundation https://wikimediafoundation.org/
PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1097
days inactive
1097
days old
2 comments
2 participants
participants (2)
-
David Caro -
mabi