1 Apr
2020
1 Apr
'20
2:10 p.m.
So, this is following on from a discussion in the #ceph IRC channel, where we seem to have
reached the limit of what we can do.
I have a ~15 node, 311 OSD cluster. (20 OSDs per node).
The cluster is Nautilus - the 3 MONs + the first 8 OSD hosts were installed as Mimic and
upgraded to Nautilus with ceph-ansible ; the remaining OSD hosts were added directly with
Nautilus as they were only added in a few weeks ago.
Yesterday, suddenly, about half of the OSDs (~140) were marked Down, and a number of slow
operations were detected.
Initially, examining the logs (and with a bit of help from IRC), I noticed that the
ansible roles used to build the newer OSDs had configured chrony incorrectly, and their
clocks were drifting.
(There were BADAUTHORIZER errors in OSD logs, too.)
I fixed the chrony configuration... and we (including people in IRC) expected everything
to just... stabilise.
Things have not stabilised, which leads me to suspect that there are other issues at
play.
After noticing a number of issues with mgrs deadlocking in Nautilus - eg
https://tracker.ceph.com/issues/17170 https://tracker.ceph.com/issues/43048 - I tried
stopping all mgrs and mons, and then slowly bringing them up.
This has not helped.
Interestingly, the OSDs with slow ops (some of which are marked down) report ops_in_flight
which are "wait for new map", whilst the lead mon believes those same ops are
timed out.
(I can of course, telnet to every OSD, even the down ones, from other OSDs, including ones
which report issues talking to them on the same port; and from the lead mon.)
I am wondering if this is an example of: https://tracker.ceph.com/issues/44184 as we did
create a new pool shortly after adding the new OSD host nodes... but it isn't clear
from that ticket [or the discussion on this list] how to fix this, other than removing the
pool - which I can't do, as we need this pool to exist, and the pool is replaces needs
to be decomissioned.
Can anyone advise what I should do next? At present, obviously, the cluster is unusable.
1 Apr
1 Apr
2:34 p.m.
(I note that some of the down OSDs still report issues with secret dissemination:
2020-04-01 14:32:11.265 7f9d9a7be700 0 auth: could not find secret_id=5010
2020-04-01 14:32:11.265 7f9d9a7be700 0 cephx: verify_authorizer could not get service
secret for service osd secret_id=5010
2020-04-01 14:32:11.265 7f9d9a7be700 0 --1- [v2:(this host's
ip):6884/1181561,v1:(this host's ip):6902/1181561] >> v1:(another OSD host
ip):6819/11112555 conn(0x55f47c670800 0x55f464728000 :6902
s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2: got bad
authorizer, auth_reply_len=0
)
2 Apr
2 Apr
10:37 a.m.
Update a day later:
the cluster is *very slowly* recovering, it looks like: we're now at 113 OSDs down
(improved from 140 OSDs down when everything broke) - but it took a day before anything
changed here, and it looks like we're recovering at a rate of about 1 -2 OSDs per
hour...
So I'm not just talking to myself here, it would be great if anyone could offer any
suggestions about: what might have happened, or how I can stop this happening again?
It's not sustainable to have a Ceph cluster take several days to recover from half its
OSDs just "deciding to all go down".
2:32 p.m.
So, the recovery stalled a few more OSDs in, but looking at the disks with OSDs marked
down, I noticed that, despite systemctl reporting that the OSD processes were all *up*,
several of them had not written to their logs since they rotated.
Suspecting that these OSDs were stalled, I've started logging into each OSD host and
doing:
ls -lh /var/log/ceph/*.log
checking for logs with a size of 0,
and then
systemctl restart ceph-osd@xxx
for all xxx with zero sized logs.
(I've checked each of these first with
systemctl status ceph-osd xxx
and they all report that the process is up...)
This seems to be helping recovery dramatically...
but if I look in the logs for each of the "frozen" OSDs before I restart them
[obviously, in the rotated log], there's no sign of why the crash actually happens -
there's a lot of complaining about how they can't talk to other OSDs as in
previous emails in this thread, and then suddenly, nothing.
It would be lovely if anyone could comment on thoughts about what's happening here.
7 Apr
7 Apr
10:42 p.m.
Hello,
I am just answearing to let you know that there are people around you and Seeing your
messages.
Unfortunality i can not help much, but i did have a Strange issue like you where few osds
went Down or stay up but i couldnt reach the node via ssh.
The case: we have done updates on our Switch (LACP) and this leads to weird pictures for
the osds.
That said, i guess there could be a network issue in your case.
I am sorry to not being a big Help.
Hope your Cluster is in health State now!
- Mehmet
Am 2. April 2020 15:32:38 MESZ schrieb aoanla(a)gmail.com:
So, the recovery stalled a few more OSDs in, but
looking at the disks
with OSDs marked down, I noticed that, despite systemctl reporting that
the OSD processes were all *up*, several of them had not written to
their logs since they rotated.
Suspecting that these OSDs were stalled, I've started logging into each
OSD host and doing:
ls -lh /var/log/ceph/*.log
checking for logs with a size of 0,
and then
systemctl restart ceph-osd@xxx
for all xxx with zero sized logs.
(I've checked each of these first with
systemctl status ceph-osd xxx
and they all report that the process is up...)
This seems to be helping recovery dramatically...
but if I look in the logs for each of the "frozen" OSDs before I
restart them [obviously, in the rotated log], there's no sign of why
the crash actually happens - there's a lot of complaining about how
they can't talk to other OSDs as in previous emails in this thread, and
then suddenly, nothing.
It would be lovely if anyone could comment on thoughts about what's
happening here.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8 Apr
8 Apr
2:23 a.m.
don't know whether you have same problem as us.
We saw several times the osd down after reboot, we fixed by
304 ls -al /dev/dm-*
305 chown ceph. /dev/dm-*
306 for i in `ceph osd tree | grep down | awk '{print $4}' | awk -F'.'
'{print $2}'`; do systemctl reset-failed ceph-osd(a)$i.service && systemctl
start ceph-osd(a)$i.service; done
307 ls -al /dev/dm-*
308 ceph osd tree | grep down
> -----Original Message-----
> From: ceph(a)elchaka.de <ceph(a)elchaka.de>
> Sent: Wednesday, April 8, 2020 5:43 AM
> To: ceph-users(a)ceph.io
> Subject: [ceph-users] Re: Multiple OSDs down, and won't come up (possibly
> related to other Nautilus issues)
>
> Hello,
>
> I am just answearing to let you know that there are people around you and
> Seeing your messages.
>
> Unfortunality i can not help much, but i did have a Strange issue like you where
> few osds went Down or stay up but i couldnt reach the node via ssh.
>
> The case: we have done updates on our Switch (LACP) and this leads to weird
> pictures for the osds.
>
> That said, i guess there could be a network issue in your case.
>
> I am sorry to not being a big Help.
> Hope your Cluster is in health State now!
>
> - Mehmet
>
> Am 2. April 2020 15:32:38 MESZ schrieb aoanla(a)gmail.com:
> >So, the recovery stalled a few more OSDs in, but looking at the disks
> >with OSDs marked down, I noticed that, despite systemctl reporting that
> >the OSD processes were all *up*, several of them had not written to
> >their logs since they rotated.
> >
> >Suspecting that these OSDs were stalled, I've started logging into each
> >OSD host and doing:
> >
> >ls -lh /var/log/ceph/*.log
> >
> >checking for logs with a size of 0,
> >
> >and then
> >
> >systemctl restart ceph-osd@xxx
> >
> >for all xxx with zero sized logs.
> >(I've checked each of these first with systemctl status ceph-osd xxx
> >and they all report that the process is up...)
> >
> >This seems to be helping recovery dramatically...
> >
> >but if I look in the logs for each of the "frozen" OSDs before I
> >restart them [obviously, in the rotated log], there's no sign of why
> >the crash actually happens - there's a lot of complaining about how
> >they can't talk to other OSDs as in previous emails in this thread, and
> >then suddenly, nothing.
> >
> >It would be lovely if anyone could comment on thoughts about what's
> >happening here.
> >_______________________________________________
> >ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> >email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
> ceph-users-leave(a)ceph.io
1501
days inactive
1508
days old
5 comments
3 participants
participants (3)
-
aoanla＠gmail.com -
ceph＠elchaka.de -
Sang, Oliver