On 2/19/20 8:49 AM, Sean Matheny wrote:
Thanks,
If the OSDs have a newer epoch of the OSDMap than
the MON it won't work.
How can I verify this? (i.e the epoch of the monitor vs the epoch of the
osd(s))
Check the status of the OSDs:
$ ceph daemon osd.X status
This should tell the newest map it has.
Then check on the mons:
$ ceph osd dump|head -n 10
Or using ceph-monstore-tool to see what the latest map is the MON has.
Wido
> Cheers,
> Sean
>
>
>> On 19/02/2020, at 7:25 PM, Wido den Hollander <wido(a)42on.com
>> <mailto:wido@42on.com>> wrote:
>>
>>
>>
>> On 2/19/20 5:45 AM, Sean Matheny wrote:
>>> I wanted to add a specific question to the previous post, in the
>>> hopes it’s easier to answer.
>>>
>>> We have a Luminous monitor restored from the OSDs using
>>> ceph-object-tool, which seems like the best chance of any success. We
>>> followed this rough process:
>>>
>>>
https://tracker.ceph.com/issues/24419
>>>
>>> The monitor has come up (as a single monitor cluster), but it’s
>>> reporting wildly inaccurate info, such as the number of osds that are
>>> down (157 but all 223 are down), and hosts (1, but all 14 are down).
>>>
>>
>> Have you verified that the MON's database has the same epoch of the
>> OSDMap (or newer) as all the other OSDs?
>>
>> If the OSDs have a newer epoch of the OSDMap than the MON it won't work.
>>
>>> The OSD Daemons are still off, but I’m not sure if starting them back
>>> up with this monitor will make things worse. The fact that this mon
>>> daemon can’t even see how many OSDs are correctly down makes me think
>>> that nothing good will come from turning the OSDs back on.
>>>
>>> Do I run risk of further corruption (i.e. on the Ceph side, not
>>> client data as the cluster is paused) if I proceed and turn on the
>>> osd daemons? Or is it worth a shot?
>>>
>>> Are these Ceph health metrics commonly inaccurate until it can talk
>>> to the daemons?
>>
>> The PG stats will be inaccurate indeed and the number of OSDs can vary
>> as long as they aren't able to peer with each other and the MONs.
>>
>>>
>>> (Also other commands like `ceph osd tree` agree with the below `ceph
>>> -s` so far)
>>>
>>> Many thanks for any wisdom… I just don’t want to make things
>>> (unnecessarily) much worse.
>>>
>>> Cheers,
>>> Sean
>>>
>>>
>>> root@ntr-mon01:/var/log/ceph# ceph -s
>>> cluster:
>>> id: ababdd7f-1040-431b-962c-c45bea5424aa
>>> health: HEALTH_WARN
>>> pauserd,pausewr,noout,norecover,noscrub,nodeep-scrub
>>> flag(s) set
>>> 157 osds down
>>> 1 host (15 osds) down
>>> Reduced data availability: 12225 pgs inactive, 885 pgs
>>> down, 673 pgs peering
>>> Degraded data redundancy: 14829054/35961087 objects
>>> degraded (41.236%), 2869 pgs degraded, 2995 pgs undersized services:
>>> mon: 1 daemons, quorum ntr-mon01
>>> mgr: ntr-mon01(active)
>>> osd: 223 osds: 66 up, 223 in
>>> flags pauserd,pausewr,noout,norecover,noscrub,nodeep-scrub data:
>>> pools: 14 pools, 15220 pgs
>>> objects: 10.58M objects, 40.1TiB
>>> usage: 43.0TiB used, 121TiB / 164TiB avail
>>> pgs: 70.085% pgs unknown
>>> 10.237% pgs not active
>>> 14829054/35961087 objects degraded (41.236%)
>>> 10667 unknown
>>> 2869 active+undersized+degraded
>>> 885 down
>>> 673 peering
>>> 126 active+undersized
>>>
>>>
>>> On 19/02/2020, at 10:18 AM, Sean Matheny <s.matheny(a)auckland.ac.nz
>>>
<mailto:s.matheny@auckland.ac.nz><mailto:s.matheny@auckland.ac.nz>>
>>> wrote:
>>>
>>> Hi folks,
>>>
>>> Our entire cluster is down at the moment.
>>>
>>> We started upgrading from 12.2.13 to 14.2.7 with the monitors. The
>>> first monitor we upgraded crashed. We reverted to luminous on this
>>> one and tried another, and it was fine. We upgraded the rest, and
>>> they all worked.
>>>
>>> Then we upgraded the first one again, and after it became the leader,
>>> it died. Then the second one became the leader, and it died. Then the
>>> third became the leader, and it died, leaving mon 4 and 5 unable to
>>> form a quorum.
>>>
>>> We tried creating a single monitor cluster by editing the monmap of
>>> mon05, and it died in the same way, just without the paxos
>>> negotiation first.
>>>
>>> We have tried to revert to a luminous (12.2.12) monitor backup taken
>>> a few hours before the crash. The mon daemon will start, but is
>>> flooded with blocked requests and unknown pgs after a while. For
>>> better or worse we removed the “noout” flag and 144 of 232 OSDs are
>>> now showing as down:
>>>
>>> cluster:
>>> id: ababdd7f-1040-431b-962c-c45bea5424aa
>>> health: HEALTH_ERR
>>> noout,nobackfill,norecover flag(s) set
>>> 101 osds down
>>> 9 hosts (143 osds) down
>>> 1 auth entities have invalid capabilities
>>> Long heartbeat ping times on back interface seen, longest
>>> is 15424.178 msec
>>> Long heartbeat ping times on front interface seen, longest
>>> is 14763.145 msec
>>> Reduced data availability: 521 pgs inactive, 48 pgs stale
>>> 274 slow requests are blocked > 32 sec
>>> 88 stuck requests are blocked > 4096 sec
>>> 1303 slow ops, oldest one blocked for 174 sec,
>>> mon.ntr-mon01 has slow ops
>>> too many PGs per OSD (299 > max 250) services:
>>> mon: 1 daemons, quorum ntr-mon01 (age 3m)
>>> mgr: ntr-mon01(active, since 30m)
>>> mds: cephfs:1 {0=akld2e18u42=up:active(laggy or crashed)}
>>> osd: 223 osds: 66 up, 167 in
>>> flags noout,nobackfill,norecover
>>> rgw: 2 daemons active (ntr-rgw01, ntr-rgw02) data:
>>> pools: 14 pools, 15220 pgs
>>> objects: 35.26M objects, 134 TiB
>>> usage: 379 TiB used, 1014 TiB / 1.4 PiB avail
>>> pgs: 3.423% pgs unknown
>>> 14651 active+clean
>>> 521 unknown
>>> 48 stale+active+clean io:
>>> client: 20 KiB/s rd, 439 KiB/s wr, 7 op/s rd, 54 op/s wr
>>>
>>> These luminous OSD daemons are not down, but are all in fact running.
>>> They just have no comms with the monitor:
>>>
>>> 2020-02-19 10:12:33.565680 7ff222e24700 1 osd.0 pg_epoch: 305104
>>> pg[100.37as3( v 129516'2 (0'0,129516'2]
local-lis/les=297268/297269
>>> n=0 ec=129502/129502 lis/c 297268/297268 les/c/f 297269/297358/0
>>> 297268/297268/161526) [41,192,216,0,160,117]p41(0) r=3 lpr=305101
>>> crt=129516'2 lcod 0'0 unknown NOTIFY mbc={}] state<Start>:
>>> transitioning to Stray
>>> 2020-02-19 10:12:33.565861 7ff222e24700 1 osd.0 pg_epoch: 305104
>>> pg[4.53c( v 305046'1933429 (304777'1931907,305046'1933429]
>>> local-lis/les=298009/298010 n=7350 ec=768/768 lis/c 298009/298009
>>> les/c/f 298010/298010/0 297268/298009/298009) [0,61,103] r=0
>>> lpr=305101 crt=305046'1933429 lcod 0'0 mlcod 0'0 unknown mbc={}]
>>> state<Start>: transitioning to Primary
>>> 2020-02-19 10:12:33.566742 7ff222e24700 1 osd.0 pg_epoch: 305104
>>> pg[100.des4( v 129516'1 (0'0,129516'1]
local-lis/les=292010/292011
>>> n=1 ec=129502/129502 lis/c 292010/292010 les/c/f 292011/292417/0
>>> 292010/292010/280955) [149,62,209,187,0,98]p149(0) r=4 lpr=305072
>>> crt=129516'1 lcod 0'0 unknown NOTIFY mbc={}] state<Start>:
>>> transitioning to Stray
>>> 2020-02-19 10:12:33.566896 7ff23ccd9e00 0 osd.0 305104 done with
>>> init, starting boot process
>>> 2020-02-19 10:12:33.566956 7ff23ccd9e00 1 osd.0 305104 start_boot
>>>
>>> One oddity in our deployment is that there was a test mds instance,
>>> and it is running mimic. I shut it down, as the monitor trace has an
>>> MDS call in it, but the nautilus monitors still die the same way.
>>>
>>> "mds": {
>>> "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0)
>>> mimic (stable)": 1
>>> },
>>>
>>>
>>> ...
>>> -11> 2020-02-18 09:50:00.800 7fd164a1a700 5
>>> mon.ntr-mon02(a)1(leader).paxos(paxos recovering c 85448935..85449502)
>>> is_readable = 0 - now=2020-02-18 09:50:00.804429
>>> lease_expire=0.000000 has v0 lc 85449502
>>> -10> 2020-02-18 09:50:00.800 7fd164a1a700 5
>>> mon.ntr-mon02(a)1(leader).paxos(paxos recovering c 85448935..85449502)
>>> is_readable = 0 - now=2020-02-18 09:50:00.804446
>>> lease_expire=0.000000 has v0 lc 85449502
>>> -9> 2020-02-18 09:50:00.800 7fd164a1a700 5
>>> mon.ntr-mon02(a)1(leader).paxos(paxos recovering c 85448935..85449502)
>>> is_readable = 0 - now=2020-02-18 09:50:00.804460
>>> lease_expire=0.000000 has v0 lc 85449502
>>> -8> 2020-02-18 09:50:00.800 7fd164a1a700 4 set_mon_vals no
>>> callback set
>>> -7> 2020-02-18 09:50:00.800 7fd164a1a700 4 mgrc handle_mgr_map Got
>>> map version 2301191
>>> -6> 2020-02-18 09:50:00.804 7fd164a1a700 4 mgrc handle_mgr_map
>>> Active mgr is now v1:10.31.88.17:6801/2924412
>>> -5> 2020-02-18 09:50:00.804 7fd164a1a700 0 log_channel(cluster)
>>> log [DBG] : monmap e25: 5 mons at
>>>
{ntr-mon01=v1:10.31.88.14:6789/0,ntr-mon02=v1:10.31.88.15:6789/0,ntr-mon03=v1:10.31.88.16:6789/0,ntr-mon04=v1:10.31.88.17:6789/0,ntr-mon05=v1:10.31.88.18:6789/0}
>>> -4> 2020-02-18 09:50:00.804 7fd164a1a700 10 log_client _send_to_mon
>>> log to self
>>> -3> 2020-02-18 09:50:00.804 7fd164a1a700 10 log_client log_queue
>>> is 3 last_log 3 sent 2 num 3 unsent 1 sending 1
>>> -2> 2020-02-18 09:50:00.804 7fd164a1a700 10 log_client will send
>>> 2020-02-18 09:50:00.806845 mon.ntr-mon02 (mon.1) 3 : cluster [DBG]
>>> monmap e25: 5 mons at
>>>
{ntr-mon01=v1:10.31.88.14:6789/0,ntr-mon02=v1:10.31.88.15:6789/0,ntr-mon03=v1:10.31.88.16:6789/0,ntr-mon04=v1:10.31.88.17:6789/0,ntr-mon05=v1:10.31.88.18:6789/0}
>>> -1> 2020-02-18 09:50:00.804 7fd164a1a700 5
>>> mon.ntr-mon02(a)1(leader).paxos(paxos active c 85448935..85449502)
>>> is_readable = 1 - now=2020-02-18 09:50:00.806920
>>> lease_expire=2020-02-18 09:50:05.804479 has v0 lc 85449502
>>> 0> 2020-02-18 09:50:00.812 7fd164a1a700 -1 *** Caught signal
>>> (Aborted) **
>>> in thread 7fd164a1a700 thread_name:ms_dispatch
>>>
>>> ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8)
>>> nautilus (stable)
>>> 1: (()+0x11390) [0x7fd171e98390]
>>> 2: (gsignal()+0x38) [0x7fd1715e5428]
>>> 3: (abort()+0x16a) [0x7fd1715e702a]
>>> 4: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7fd173673bf5]
>>> 5: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7fd173667bd6]
>>> 6: (()+0x8b6c21) [0x7fd173667c21]
>>> 7: (()+0x8c2e34) [0x7fd173673e34]
>>> 8: (std::__throw_out_of_range(char const*)+0x3f) [0x7fd17367f55f]
>>> 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x79ae00]
>>> 10: (MDSMonitor::tick()+0xc9) [0x79c669]
>>> 11: (MDSMonitor::on_active()+0x28) [0x785e88]
>>> 12: (PaxosService::_active()+0xdd) [0x6d4b2d]
>>> 13: (Context::complete(int)+0x9) [0x600789]
>>> 14: (void finish_contexts<std::__cxx11::list<Context*,
>>> std::allocator<Context*> > >(CephContext*,
>>> std::__cxx11::list<Context*, std::allocator<Context*> >&,
int)+0xa8)
>>> [0x6299a8]
>>> 15: (Paxos::finish_round()+0x76) [0x6cb276]
>>> 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xbff)
>>> [0x6cc47f]
>>> 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x24b)
>>> [0x6ccf2b]
>>> 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x15c5)
>>> [0x5fa6f5]
>>> 19: (Monitor::_ms_dispatch(Message*)+0x4d2) [0x5fad42]
>>> 20: (Monitor::ms_dispatch(Message*)+0x26) [0x62b046]
>>> 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message>
>>> const&)+0x26) [0x6270b6]
>>> 22: (DispatchQueue::entry()+0x1219) [0x7fd1732b7e59]
>>> 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fd17336836d]
>>> 24: (()+0x76ba) [0x7fd171e8e6ba]
>>> 25: (clone()+0x6d) [0x7fd1716b741d]
>>> ...
>>>
>>> Ceph versions output
>>>
>>> {
>>> "mon": {
>>> "ceph version 12.2.13
>>> (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1,
>>> "ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8)
>>> nautilus (stable)": 4
>>> },
>>> "mgr": {
>>> "ceph version 12.2.12
>>> (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 1,
>>> "ceph version 12.2.13
>>> (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1,
>>> "ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8)
>>> nautilus (stable)": 2
>>> },
>>> "osd": {
>>> "ceph version 12.2.11
>>> (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 175,
>>> "ceph version 12.2.12
>>> (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 32,
>>> "ceph version 12.2.13
>>> (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 16
>>> },
>>> "mds": {
>>> "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0)
>>> mimic (stable)": 1
>>> },
>>> "rgw": {
>>> "ceph version 12.2.12
>>> (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 2
>>> },
>>> "overall": {
>>> "ceph version 12.2.11
>>> (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 175,
>>> "ceph version 12.2.12
>>> (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 35,
>>> "ceph version 12.2.13
>>> (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 18,
>>> "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0)
>>> mimic (stable)": 1,
>>> "ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8)
>>> nautilus (stable)": 6
>>> }
>>> }
>>>
>>> We’ve filed a bug report with the actions of the actual cascading
>>> crash described above (when we upgraded mon01 and it became the leader):
>>>
https://tracker.ceph.com/issues/44185 (parts here copied from that
>>> report)
>>>
>>> Right now we’re not sure what the best path to some sort of recovery
>>> would be. All OSD Daemons are still on Luminous, so AFAICT, we could
>>> build the monitor db from the OSDs with
>>>
https://github.com/ceph/ceph/blob/luminous/doc/rados/troubleshooting/troubl…
>>> which describes using this script:
>>>
>>>
>>> #!/bin/bash
>>> hosts="ntr-sto01 ntr-sto02"
>>> ms=/tmp/mon-store/
>>> mkdir $ms
>>> # collect the cluster map from OSDs
>>> for host in $hosts; do
>>> echo $host
>>> rsync -avz $ms root@$host:$ms
>>> rm -rf $ms
>>> ssh root@$host <<EOF
>>> for osd in /var/lib/ceph/osd/ceph-*; do
>>> ceph-objectstore-tool --data-path \$osd --op update-mon-db
>>> --mon-store-path $ms
>>> done
>>> EOF
>>> rsync -avz root@$host:$ms $ms
>>> done
>>>
>>> If this is our best idea to try, should we try the mon store from the
>>> above script on a luminous or nautilus mon daemon? Any other ideas to
>>> try at this dark hour? : \
>>>
>>> Cheers,
>>> Sean
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> <mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io>
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>
<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> <mailto:ceph-users-leave@ceph.io>
>