Multiple OSDs down, and won't come up (possibly related to other Nautilus issues)
So, this is following on from a discussion in the #ceph IRC channel, where we seem to have reached the limit of what we can do. I have a ~15 node, 311 OSD cluster. (20 OSDs per node). The cluster is Nautilus - the 3 MONs + the first 8 OSD hosts were installed as Mimic and upgraded to Nautilus with ceph-ansible ; the remaining OSD hosts were added directly with Nautilus as they were only added in a few weeks ago. Yesterday, suddenly, about half of the OSDs (~140) were marked Down, and a number of slow operations were detected. Initially, examining the logs (and with a bit of help from IRC), I noticed that the ansible roles used to build the newer OSDs had configured chrony incorrectly, and their clocks were drifting. (There were BADAUTHORIZER errors in OSD logs, too.) I fixed the chrony configuration... and we (including people in IRC) expected everything to just... stabilise. Things have not stabilised, which leads me to suspect that there are other issues at play. After noticing a number of issues with mgrs deadlocking in Nautilus - eg https://tracker.ceph.com/issues/17170 https://tracker.ceph.com/issues/43048 - I tried stopping all mgrs and mons, and then slowly bringing them up. This has not helped. Interestingly, the OSDs with slow ops (some of which are marked down) report ops_in_flight which are "wait for new map", whilst the lead mon believes those same ops are timed out. (I can of course, telnet to every OSD, even the down ones, from other OSDs, including ones which report issues talking to them on the same port; and from the lead mon.) I am wondering if this is an example of: https://tracker.ceph.com/issues/44184 as we did create a new pool shortly after adding the new OSD host nodes... but it isn't clear from that ticket [or the discussion on this list] how to fix this, other than removing the pool - which I can't do, as we need this pool to exist, and the pool is replaces needs to be decomissioned. Can anyone advise what I should do next? At present, obviously, the cluster is unusable.
(I note that some of the down OSDs still report issues with secret dissemination: 2020-04-01 14:32:11.265 7f9d9a7be700 0 auth: could not find secret_id=5010 2020-04-01 14:32:11.265 7f9d9a7be700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=5010 2020-04-01 14:32:11.265 7f9d9a7be700 0 --1- [v2:(this host's ip):6884/1181561,v1:(this host's ip):6902/1181561] >> v1:(another OSD host ip):6819/11112555 conn(0x55f47c670800 0x55f464728000 :6902 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2: got bad authorizer, auth_reply_len=0 )
Update a day later: the cluster is *very slowly* recovering, it looks like: we're now at 113 OSDs down (improved from 140 OSDs down when everything broke) - but it took a day before anything changed here, and it looks like we're recovering at a rate of about 1 -2 OSDs per hour... So I'm not just talking to myself here, it would be great if anyone could offer any suggestions about: what might have happened, or how I can stop this happening again? It's not sustainable to have a Ceph cluster take several days to recover from half its OSDs just "deciding to all go down".
So, the recovery stalled a few more OSDs in, but looking at the disks with OSDs marked down, I noticed that, despite systemctl reporting that the OSD processes were all *up*, several of them had not written to their logs since they rotated. Suspecting that these OSDs were stalled, I've started logging into each OSD host and doing: ls -lh /var/log/ceph/*.log checking for logs with a size of 0, and then systemctl restart ceph-osd@xxx for all xxx with zero sized logs. (I've checked each of these first with systemctl status ceph-osd xxx and they all report that the process is up...) This seems to be helping recovery dramatically... but if I look in the logs for each of the "frozen" OSDs before I restart them [obviously, in the rotated log], there's no sign of why the crash actually happens - there's a lot of complaining about how they can't talk to other OSDs as in previous emails in this thread, and then suddenly, nothing. It would be lovely if anyone could comment on thoughts about what's happening here.
Hello, I am just answearing to let you know that there are people around you and Seeing your messages. Unfortunality i can not help much, but i did have a Strange issue like you where few osds went Down or stay up but i couldnt reach the node via ssh. The case: we have done updates on our Switch (LACP) and this leads to weird pictures for the osds. That said, i guess there could be a network issue in your case. I am sorry to not being a big Help. Hope your Cluster is in health State now! - Mehmet Am 2. April 2020 15:32:38 MESZ schrieb aoanla@gmail.com:
So, the recovery stalled a few more OSDs in, but looking at the disks with OSDs marked down, I noticed that, despite systemctl reporting that the OSD processes were all *up*, several of them had not written to their logs since they rotated.
Suspecting that these OSDs were stalled, I've started logging into each OSD host and doing:
ls -lh /var/log/ceph/*.log
checking for logs with a size of 0,
and then
systemctl restart ceph-osd@xxx
for all xxx with zero sized logs. (I've checked each of these first with systemctl status ceph-osd xxx and they all report that the process is up...)
This seems to be helping recovery dramatically...
but if I look in the logs for each of the "frozen" OSDs before I restart them [obviously, in the rotated log], there's no sign of why the crash actually happens - there's a lot of complaining about how they can't talk to other OSDs as in previous emails in this thread, and then suddenly, nothing.
It would be lovely if anyone could comment on thoughts about what's happening here. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
don't know whether you have same problem as us. We saw several times the osd down after reboot, we fixed by 304 ls -al /dev/dm-* 305 chown ceph. /dev/dm-* 306 for i in `ceph osd tree | grep down | awk '{print $4}' | awk -F'.' '{print $2}'`; do systemctl reset-failed ceph-osd@$i.service && systemctl start ceph-osd@$i.service; done 307 ls -al /dev/dm-* 308 ceph osd tree | grep down
-----Original Message----- From: ceph@elchaka.de <ceph@elchaka.de> Sent: Wednesday, April 8, 2020 5:43 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Multiple OSDs down, and won't come up (possibly related to other Nautilus issues)
Hello,
I am just answearing to let you know that there are people around you and Seeing your messages.
Unfortunality i can not help much, but i did have a Strange issue like you where few osds went Down or stay up but i couldnt reach the node via ssh.
The case: we have done updates on our Switch (LACP) and this leads to weird pictures for the osds.
That said, i guess there could be a network issue in your case.
I am sorry to not being a big Help. Hope your Cluster is in health State now!
- Mehmet
Am 2. April 2020 15:32:38 MESZ schrieb aoanla@gmail.com:
So, the recovery stalled a few more OSDs in, but looking at the disks with OSDs marked down, I noticed that, despite systemctl reporting that the OSD processes were all *up*, several of them had not written to their logs since they rotated.
Suspecting that these OSDs were stalled, I've started logging into each OSD host and doing:
ls -lh /var/log/ceph/*.log
checking for logs with a size of 0,
and then
systemctl restart ceph-osd@xxx
for all xxx with zero sized logs. (I've checked each of these first with systemctl status ceph-osd xxx and they all report that the process is up...)
This seems to be helping recovery dramatically...
but if I look in the logs for each of the "frozen" OSDs before I restart them [obviously, in the rotated log], there's no sign of why the crash actually happens - there's a lot of complaining about how they can't talk to other OSDs as in previous emails in this thread, and then suddenly, nothing.
It would be lovely if anyone could comment on thoughts about what's happening here. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
aoanla＠gmail.com
-
ceph＠elchaka.de
-
Sang, Oliver