This is the second time this happened in a couple of weeks. The MDS locks up and the stand-by can't take over so the Montiors black list them. I try to unblack list them, but they still say this in the logs mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance) Looking at a pg dump, it looks like the epoch is passed that. $ ceph pg map 3.756 osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115] Last time, it seemed to just recover after about an hour all by it's self. Any way to speed this up? Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Thu, Mar 12, 2020 at 1:41 PM Robert LeBlanc <robert@leblancnet.us> wrote:
This is the second time this happened in a couple of weeks. The MDS locks up and the stand-by can't take over so the Montiors black list them. I try to unblack list them, but they still say this in the logs
mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance)
Looking at a pg dump, it looks like the epoch is passed that.
$ ceph pg map 3.756 osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115]
Last time, it seemed to just recover after about an hour all by it's self. Any way to speed this up?
try restart the standby mds
Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I tried both several times. I looks like it just had to read through the entire journal. I wish there was more progress notification about journal reading progress in debug less than 10 because 10 is way too noisy. That could give us an idea of how much longer there is left to go. It seems that the MDS got way too behind on segments ~14,000 from some naughty clients and caused the journal to explode and the MDS to eventually just not respond to the monitors. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Mar 12, 2020 at 12:48 AM Yan, Zheng <ukernel@gmail.com> wrote:
On Thu, Mar 12, 2020 at 1:41 PM Robert LeBlanc <robert@leblancnet.us> wrote:
This is the second time this happened in a couple of weeks. The MDS locks up and the stand-by can't take over so the Montiors black list them. I
try
to unblack list them, but they still say this in the logs
mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance)
Looking at a pg dump, it looks like the epoch is passed that.
$ ceph pg map 3.756 osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115]
Last time, it seemed to just recover after about an hour all by it's self. Any way to speed this up?
try restart the standby mds
Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Mar 11, 2020 at 10:41 PM Robert LeBlanc <robert@leblancnet.us> wrote:
This is the second time this happened in a couple of weeks. The MDS locks up and the stand-by can't take over so the Montiors black list them. I try to unblack list them, but they still say this in the logs
mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance)
Do not *ever* unblacklist an MDS. Restart the daemon.
Looking at a pg dump, it looks like the epoch is passed that.
$ ceph pg map 3.756 osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115]
Last time, it seemed to just recover after about an hour all by it's self. Any way to speed this up?
We need more cluster information, error messages, client versions/types, etc. to help. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Wed, May 6, 2020 at 2:45 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Wed, Mar 11, 2020 at 10:41 PM Robert LeBlanc <robert@leblancnet.us> wrote:
This is the second time this happened in a couple of weeks. The MDS locks up and the stand-by can't take over so the Montiors black list them. I
try
to unblack list them, but they still say this in the logs
mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance)
Do not *ever* unblacklist an MDS. Restart the daemon.
These messages showed up after the MDS was restarted right after the boot messages. What was actually happening was the MDS was busy reading the journal which took about an hour to do. I'm not sure why in this situation the journal got so big, but it was not due to the blacklist, just a coincidence that it happened to be the same amount of time as the blacklist timeout. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
participants (3)
-
Patrick Donnelly
-
Robert LeBlanc
-
Yan, Zheng