We found the vmcore and backtrace and created
https://tracker.ceph.com/issues/49210
Cheers, Dan
On Mon, Feb 8, 2021 at 10:58 AM Dan van der Ster <dan(a)vanderster.com> wrote:
>
> Hi,
>
> Sorry to ping this old thread, but we have a few kernel client nodes
> stuck like this after an outage on their network.
> MDS's are running v14.2.11 and the client has kernel
> 3.10.0-1127.19.1.el7.x86_64.
>
> This is the first time at our lab that clients didn't reconnect after
> a network issue (but this might be the first large client network
> outage after we upgraded from luminous to nautilus).
>
> It looks identical to Florian's issue:
>
> Feb 08 10:07:23 hpc-qcd027.cern.ch kernel: libceph: mds0
> 10.32.5.17:6821 socket closed (con state NEGOTIATING)
> Feb 08 10:07:51 hpc-qcd027.cern.ch kernel: ceph: get_quota_realm: ino
> (10004fe5035.fffffffffffffffe) null i_snap_realm
>
> The full kernel log | grep ceph is at
https://termbin.com/zdwc
>
> As of now, this client's mountpoint is "stuck" and it does not have a
> session open on mds.0, but has sessions on mds.1 and mds.2 (see below
> [1]).
>
> I evicted this client from all mds's but the client didn't manage to
reconnect:
>
> Feb 08 10:20:01 hpc-qcd027.cern.ch kernel: libceph: mds1
> 188.185.88.47:6801 socket closed (con state OPEN)
> Feb 08 10:20:01 hpc-qcd027.cern.ch kernel: libceph: mds2
> 188.185.88.90:6801 socket closed (con state OPEN)
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: libceph: mds1
> 188.185.88.47:6801 connection reset
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: libceph: reset on mds1
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds1 closed our session
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds1 reconnect start
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: libceph: mds2
> 188.185.88.90:6801 connection reset
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: libceph: reset on mds2
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds2 closed our session
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds2 reconnect start
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds1 reconnect denied
> Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds2 reconnect denied
> Feb 08 10:20:21 hpc-qcd027.cern.ch kernel: ceph: get_quota_realm: ino
> (10004fe5035.fffffffffffffffe) null i_snap_realm
> Feb 08 10:20:51 hpc-qcd027.cern.ch kernel: ceph: get_quota_realm: ino
> (10004fe5035.fffffffffffffffe) null i_snap_realm
>
> Here are some logs from mds.0:
>
> # egrep '10.32.3.150|137564444'
> /var/log/ceph/ceph-mds.cephflax-mds-ca21a8a1c6.log
> 2021-02-08 09:16:46.875 7f9b22faa700 0 log_channel(cluster) log [WRN]
> : evicting unresponsive client hpc-qcd027.cern.ch:hpc (137564444),
> after 304.536 seconds
> 2021-02-08 09:17:28.326 7f9b28a6c700 0 --1-
> [v2:188.184.96.191:6800/1781566860,v1:188.184.96.191:6801/1781566860]
> >> v1:10.32.3.150:0/3998218413 conn(0x562ac988b800 0x562e1a6e8800
> :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=0).handle_connect_message_2 accept we reset (peer sent cseq 1),
> sending RESETSESSION
> 2021-02-08 09:17:28.628 7f9b28a6c700 0 --1-
> [v2:188.184.96.191:6800/1781566860,v1:188.184.96.191:6801/1781566860]
> >> v1:10.32.3.150:0/3998218413 conn(0x5629ce1c6800 0x56274776f000
> :6801 s=OPENED pgs=26571 cs=1 l=0).fault server, going to standby
> 2021-02-08 09:49:56.318 7f9b28a6c700 0 --1-
> [v2:188.184.96.191:6800/1781566860,v1:188.184.96.191:6801/1781566860]
> >> v1:10.32.3.150:0/3998218413 conn(0x5629f5eaa800 0x5627460ab800
> :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=0).handle_connect_message_2 accept peer reset, then tried to connect
> to us, replacing
>
> compared with mds.1 where the reconnect succeeded:
>
> # egrep '10.32.3.150|137564444'
> /var/log/ceph/ceph-mds.cephflax-mds-370212ad58.log
> 2021-02-08 09:16:42.970 7fc98299b700 0 log_channel(cluster) log [WRN]
> : evicting unresponsive client hpc-qcd027.cern.ch:hpc (137564444),
> after 300.629 seconds
> 2021-02-08 09:17:28.327 7fc987c2f700 0 --1-
> [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >>
> v1:10.32.3.150:0/3998218413 conn(0x55a652fea400 0x55a7220ba800 :6801
> s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=0).handle_connect_message_2 accept we reset (peer sent cseq 1),
> sending RESETSESSION
> 2021-02-08 09:17:28.414 7fc987c2f700 0 --1-
> [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >>
> v1:10.32.3.150:0/3998218413 conn(0x55a791481000 0x55a6c7a2d800 :6801
> s=OPENED pgs=26573 cs=1 l=0).fault server, going to standby
> 2021-02-08 10:05:12.810 7fc988430700 0 --1-
> [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >>
> v1:10.32.3.150:0/3998218413 conn(0x55a75789d400 0x55a78695b000 :6801
> s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=0).handle_connect_message_2 accept peer reset, then tried to connect
> to us, replacing
> 2021-02-08 10:05:13.374 7fc97e192700 2 mds.1.server New client
> session:
addr="v1:10.32.3.150:0/3998218413",elapsed=0.057278,throttled=0.000007,status="ACCEPTED",root="/hpcqcd"
> 2021-02-08 10:20:01.666 7fc98499f700 1 mds.1.242924 Evicting client
> session 137564444 (v1
> 10.32.3.150:0/3998218413)
> 2021-02-08 10:20:01.666 7fc98499f700 0 log_channel(cluster) log [INF]
> : Evicting client session 137564444 (v1:10.32.3.150:0/3998218413)
> 2021-02-08 10:20:02.343 7fc988430700 0 --1-
> [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >>
> v1:10.32.3.150:0/3998218413 conn(0x55a7a5d81c00 0x55a7823d1000 :6801
> s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=0).handle_connect_message_2 accept we reset (peer sent cseq 2),
> sending RESETSESSION
> 2021-02-08 10:20:02.345 7fc988430700 0 --1-
> [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >>
> v1:10.32.3.150:0/3998218413 conn(0x55a75bb6a000 0x55a744f69800 :6801
> s=OPENED pgs=26588 cs=1 l=0).fault server, going to standby
>
>
> We have this in the mds config:
> mds session blacklist on evict = false
> mds session blacklist on timeout = false
>
> The clients are working fine after they are rebooted.
>
> Given the age of this thread -- maybe is this a known issue and
> already solved in newer kernels?
>
> Note that during this incident a few clients also crashed and rebooted
> -- we are still trying to get the kernel backtrace for those cases, to
> see if it matches
https://tracker.ceph.com/issues/40862.
>
> Thanks!
>
> Dan
>
> [1] session ls:
> mds.cephflax-mds-ca21a8a1c6: []
> mds.cephflax-mds-370212ad58: [
> {
> "id": 137564444,
> "entity": {
> "name": {
> "type": "client",
> "num": 137564444
> },
> "addr": {
> "type": "v1",
> "addr": "10.32.3.150:0",
> "nonce": 3998218413
> }
> },
> "state": "open",
> "num_leases": 0,
> "num_caps": 0,
> "request_load_avg": 0,
> "uptime": 844.92158032299994,
> "requests_in_flight": 0,
> "completed_requests": 0,
> "reconnecting": false,
> "recall_caps": {
> "value": 0,
> "halflife": 60
> },
> "release_caps": {
> "value": 0,
> "halflife": 60
> },
> "recall_caps_throttle": {
> "value": 0,
> "halflife": 2.5
> },
> "recall_caps_throttle2o": {
> "value": 0,
> "halflife": 0.5
> },
> "session_cache_liveness": {
> "value": 0,
> "halflife": 300
> },
> "inst": "client.137564444 v1:10.32.3.150:0/3998218413",
> "completed_requests": [],
> "prealloc_inos": [],
> "used_inos": [],
> "client_metadata": {
> "features": "0x00000000000000ff",
> "entity_id": "hpc",
> "hostname": "hpc-qcd027.cern.ch",
> "kernel_version": "3.10.0-1127.19.1.el7.x86_64",
> "root": "/hpcqcd"
> }
> }
> ]
> mds.cephflax-mds-adccf51169: [
> {
> "id": 137564444,
> "entity": {
> "name": {
> "type": "client",
> "num": 137564444
> },
> "addr": {
> "type": "v1",
> "addr": "10.32.3.150:0",
> "nonce": 3998218413
> }
> },
> "state": "open",
> "num_leases": 0,
> "num_caps": 0,
> "request_load_avg": 0,
> "uptime": 1761.964491447,
> "requests_in_flight": 0,
> "completed_requests": 0,
> "reconnecting": false,
> "recall_caps": {
> "value": 0,
> "halflife": 60
> },
> "release_caps": {
> "value": 0,
> "halflife": 60
> },
> "recall_caps_throttle": {
> "value": 0,
> "halflife": 2.5
> },
> "recall_caps_throttle2o": {
> "value": 0,
> "halflife": 0.5
> },
> "session_cache_liveness": {
> "value": 0,
> "halflife": 300
> },
> "inst": "client.137564444 v1:10.32.3.150:0/3998218413",
> "completed_requests": [],
> "prealloc_inos": [],
> "used_inos": [],
> "client_metadata": {
> "features": "0x00000000000000ff",
> "entity_id": "hpc",
> "hostname": "hpc-qcd027.cern.ch",
> "kernel_version": "3.10.0-1127.19.1.el7.x86_64",
> "root": "/hpcqcd"
> }
> }
> ]
>
>
> On Mon, Oct 14, 2019 at 12:03 PM Florian Pritz
> <florian.pritz(a)rise-world.com> wrote:
> >
> > On Wed, Oct 02, 2019 at 10:24:41PM +0800, "Yan, Zheng"
<ukernel(a)gmail.com> wrote:
> > > Can you reproduce this. If you can, run 'ceph daemon mds.x session
ls'
> > > before restart mds.
> >
> > I just managed to run into this issue again. 'ceph daemon mds.x session
> > ls' doesn't work because apparently our setup doesn't have the
admin
> > socket in the expected place. I've therefore used 'ceph tell mds.0
> > session ls' which I think should be the same expect for how the daemon
> > is contacted.
> >
> > When the issue happens and 2 clients are hanging, 'ceph tell mds.0
> > session ls' shows only 9 clients instead of 11. The hanging clients are
> > missing from the list. Once they are rebooted they show up in the
> > output.
> >
> > On a potentially interesting note: The clients that were hanging this
> > time are the same ones as last time. They aren't set up any differently
> > from the others as far as I can tell though.
> >
> > Florian
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io