12 Mar
2020
12 Mar
'20
2:40 a.m.
This is the second time this happened in a couple of weeks. The MDS locks
up and the stand-by can't take over so the Montiors black list them. I try
to unblack list them, but they still say this in the logs
mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance)
Looking at a pg dump, it looks like the epoch is passed that.
$ ceph pg map 3.756
osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115]
Last time, it seemed to just recover after about an hour all by it's self.
Any way to speed this up?
Thank you,
Robert LeBlanc
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
12 Mar
12 Mar
4:48 a.m.
On Thu, Mar 12, 2020 at 1:41 PM Robert LeBlanc <robert(a)leblancnet.us> wrote:
This is the second time this happened in a couple of weeks. The MDS locks
up and the stand-by can't take over so the Montiors black list them. I try
to unblack list them, but they still say this in the logs
mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance)
Looking at a pg dump, it looks like the epoch is passed that.
$ ceph pg map 3.756
osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115]
Last time, it seemed to just recover after about an hour all by it's self.
Any way to speed this up?
try restart the standby mds
> Thank you,
> Robert LeBlanc
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:58 p.m.
I tried both several times. I looks like it just had to read through the
entire journal. I wish there was more progress notification about journal
reading progress in debug less than 10 because 10 is way too noisy. That
could give us an idea of how much longer there is left to go. It seems that
the MDS got way too behind on segments ~14,000 from some naughty clients
and caused the journal to explode and the MDS to eventually just not
respond to the monitors.
Thank you,
Robert LeBlanc
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Thu, Mar 12, 2020 at 12:48 AM Yan, Zheng <ukernel(a)gmail.com> wrote:
On Thu, Mar 12, 2020 at 1:41 PM Robert LeBlanc
<robert(a)leblancnet.us>
wrote:
This is the second time this happened in a couple of weeks. The MDS locks
up and the stand-by can't take over so the Montiors black list them. I
try
to unblack list them, but they still say this in
the logs
mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance)
Looking at a pg dump, it looks like the epoch is passed that.
$ ceph pg map 3.756
osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115]
Last time, it seemed to just recover after about an hour all by it's
self.
Any way to speed this up?
try restart the standby mds
Thank you,
Robert LeBlanc
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6 May
6 May
6:45 p.m.
On Wed, Mar 11, 2020 at 10:41 PM Robert LeBlanc <robert(a)leblancnet.us> wrote:
This is the second time this happened in a couple of weeks. The MDS locks
up and the stand-by can't take over so the Montiors black list them. I try
to unblack list them, but they still say this in the logs
mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance)
Do not *ever* unblacklist an MDS. Restart the daemon.
Looking at a pg dump, it looks like the epoch is
passed that.
$ ceph pg map 3.756
osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115]
Last time, it seemed to just recover after about an hour all by it's self.
Any way to speed this up?
We need more cluster information, error messages, client
versions/types, etc. to help.
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
7 May
7 May
1:36 p.m.
On Wed, May 6, 2020 at 2:45 PM Patrick Donnelly <pdonnell(a)redhat.com> wrote:
On Wed, Mar 11, 2020 at 10:41 PM Robert LeBlanc
<robert(a)leblancnet.us>
wrote:
These messages showed up after the MDS was restarted right after the boot
messages.
What was actually happening was the MDS was busy reading the journal which
took about an hour to do. I'm not sure why in this situation the journal
got so big, but it was not due to the blacklist, just a coincidence that it
happened to be the same amount of time as the blacklist timeout.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
This is the second time this happened in a couple of weeks. The MDS locks
up and the stand-by can't take over so the Montiors black list them. I
try
to unblack list them, but they still say this in
the logs
mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance)
Do not *ever* unblacklist an MDS. Restart the daemon.
1461
days inactive
1517
days old
4 comments
3 participants
participants (3)
-
Patrick Donnelly -
Robert LeBlanc -
Yan, Zheng