14 Jun
2021
14 Jun
'21
6:07 p.m.
Hello everyone!
We had a switch outage and the ceph kernel mount did not work anymore.
This is the fstab entry:
10.99.10.1:/somefolder /cephfs ceph _netdev,nofail,name=cephcluster,secret=IsSecret
0 0
I reproduced it with disabling the vlan on the switch on which the ceph is reachable,
which gives a icmp-not-reachable.
I did this for five minutes, after that, "ls /cephfs" just gives a
"permission denied"
in dmesg i can see this:
[ 1412.994921] libceph: mon1 10.99.10.4:6789 session lost, hunting for new mon
[ 1413.009325] libceph: mon0 10.99.10.1:6789 session established
[ 1452.998646] libceph: mon2 10.99.15.3:6789 session lost, hunting for new mon
[ 1452.998679] libceph: mon0 10.99.10.1:6789 session lost, hunting for new mon
[ 1461.989549] libceph: mon4 10.99.15.5:6789 socket closed (con state CONNECTING)
---
[ 1787.045148] libceph: mon3 10.99.15.4:6789 socket closed (con state CONNECTING)
[ 1787.062587] libceph: mon0 10.99.10.1:6789 session established
[ 1787.086103] libceph: mon4 10.99.15.5:6789 session established
[ 1814.028761] libceph: mds0 10.99.10.4:6801 socket closed (con state OPEN)
[ 1815.029811] libceph: mds0 10.99.10.4:6801 connection reset
[ 1815.029829] libceph: reset on mds0
[ 1815.029831] ceph: mds0 closed our session
[ 1815.029833] ceph: mds0 reconnect start
[ 1815.052219] ceph: mds0 reconnect denied
[ 1815.052229] ceph: dropping dirty Fw state for ffff9d9085da1340 1099512175611
[ 1815.052231] ceph: dropping dirty+flushing Fw state for ffff9d9085da1340 1099512175611
[ 1815.273008] libceph: mds0 10.99.10.4:6801 socket closed (con state NEGOTIATING)
[ 1816.033241] ceph: mds0 rejected session
[ 1829.018643] ceph: mds0 hung
[ 1880.088504] ceph: mds0 came back
[ 1880.088662] ceph: mds0 caps renewed
[ 1880.094018] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null
i_snap_realm
[ 1881.100367] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null
i_snap_realm
[ 2046.768969] conntrack: generic helper won't handle protocol 47. Please consider
loading the specific helper module.
[ 2061.731126] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null
i_snap_realm
Is this a bug to report or wrong configuration?
Did someone else had this before?
To solve the problem, a simple remount does the trick.
Thanks in advance
Simon
14 Jun
14 Jun
6:20 p.m.
Hi,
CephFS clients are blacklisted if they do not react to heartbeat
packets. The MDS will deny the reconnect:
[ 1815.029831] ceph: mds0 closed our session
[ 1815.029833] ceph: mds0 reconnect start
[ 1815.052219] ceph: mds0 reconnect denied
[ 1815.052229] ceph: dropping dirty Fw state for ffff9d9085da1340 1099512175611
[ 1815.052231] ceph: dropping dirty+flushing Fw state for ffff9d9085da1340
1099512175611
[ 1815.273008] libceph: mds0 10.99.10.4:6801 socket closed (con state NEGOTIATING)
[ 1816.033241] ceph: mds0 rejected session
[ 1829.018643] ceph: mds0 hung
[ 1880.088504] ceph: mds0 came back
[ 1880.088662] ceph: mds0 caps renewed
[ 1880.094018] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null
i_snap_realm
[ 1881.100367] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null
i_snap_realm
[ 2046.768969] conntrack: generic helper won't handle protocol 47. Please consider
loading the specific helper module.
[ 2061.731126] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null
i_snap_realm
This will render the mount useless until a complete remount is
happening. You can verify this by printing the osd block list after the
mount point is not usable anymore using the 'ceph osd blocklist ls' command.
The intention of this behavior is the handling of rogue / faulty
clients. If your client currently hold the caps for important
directories and the machine has a hardware error (and won't come back
soon), the access to the directories would be blocked. Other clients
won't be able to access them until the broken machine comes back.
Network outage is another example.
You can configure the mds session timeout that triggers blacklisting.
But keep in mind that simply using a longer timeout may lead to other
problems in case of real errors.
Regards,
Burkhard
7:53 p.m.
Hi Simon,
If you have a recent enough kernel, you can try the "recover_session"
mount option [1]. Read the doc and be aware of what will happen if the
client try to recover after being blacklisted.
[1]: https://docs.ceph.com/en/latest/man/8/mount.ceph/#basic
Weiwen Hu
在 2021/6/14 下午11:07, Simon Sutter 写道:
Hello everyone!
We had a switch outage and the ceph kernel mount did not work anymore.
This is the fstab entry:
10.99.10.1:/somefolder /cephfs ceph
_netdev,nofail,name=cephcluster,secret=IsSecret 0 0
I reproduced it with disabling the vlan on the switch on which the ceph is reachable,
which gives a icmp-not-reachable.
I did this for five minutes, after that, "ls /cephfs" just gives a
"permission denied"
in dmesg i can see this:
[ 1412.994921] libceph: mon1 10.99.10.4:6789 session lost, hunting for new mon
[ 1413.009325] libceph: mon0 10.99.10.1:6789 session established
[ 1452.998646] libceph: mon2 10.99.15.3:6789 session lost, hunting for new mon
[ 1452.998679] libceph: mon0 10.99.10.1:6789 session lost, hunting for new mon
[ 1461.989549] libceph: mon4 10.99.15.5:6789 socket closed (con state CONNECTING)
---
[ 1787.045148] libceph: mon3 10.99.15.4:6789 socket closed (con state CONNECTING)
[ 1787.062587] libceph: mon0 10.99.10.1:6789 session established
[ 1787.086103] libceph: mon4 10.99.15.5:6789 session established
[ 1814.028761] libceph: mds0 10.99.10.4:6801 socket closed (con state OPEN)
[ 1815.029811] libceph: mds0 10.99.10.4:6801 connection reset
[ 1815.029829] libceph: reset on mds0
[ 1815.029831] ceph: mds0 closed our session
[ 1815.029833] ceph: mds0 reconnect start
[ 1815.052219] ceph: mds0 reconnect denied
[ 1815.052229] ceph: dropping dirty Fw state for ffff9d9085da1340 1099512175611
[ 1815.052231] ceph: dropping dirty+flushing Fw state for ffff9d9085da1340
1099512175611
[ 1815.273008] libceph: mds0 10.99.10.4:6801 socket closed (con state NEGOTIATING)
[ 1816.033241] ceph: mds0 rejected session
[ 1829.018643] ceph: mds0 hung
[ 1880.088504] ceph: mds0 came back
[ 1880.088662] ceph: mds0 caps renewed
[ 1880.094018] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null
i_snap_realm
[ 1881.100367] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null
i_snap_realm
[ 2046.768969] conntrack: generic helper won't handle protocol 47. Please consider
loading the specific helper module.
[ 2061.731126] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null
i_snap_realm
Is this a bug to report or wrong configuration?
Did someone else had this before?
To solve the problem, a simple remount does the trick.
Thanks in advance
Simon
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1071
days inactive
1071
days old
2 comments
3 participants
participants (3)
-
Burkhard Linke -
Simon Sutter -
胡玮文