On Fri, Sep 6, 2019 at 7:33 AM Vang Le-Quy <vle(a)its.aau.dk> wrote:
Hi Paul
Thank you for suggestion.
Firstly, a correction “I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1”
is incorrect. I meant that version 12.2.8-0ubuntu0.18.04.1 on node N01
works, . 12.2.12-0ubuntu0.18.04.2 is having problem.
After upgrading to version 12.2.12-0ubuntu0.18.04.2, it doesn’t work
anymore. No changes on client keys file or on the Ceph cluster. Therefore
I deduce that it is a problem with client (ceph-common) software.
Looks like a CephX caps issue for your client. Please refer to the
documentation and verify you are using the correct caps [1].
~ Vang
*From: *Paul Emmerich <paul.emmerich(a)croit.io>
*Date: *Friday, 6 September 2019 at 13.13
*To: *Vang Le-Quy <vle(a)its.aau.dk>
*Cc: *"ceph-users(a)ceph.io" <ceph-users(a)ceph.io>
*Subject: *Re: [ceph-users] Ceph client failed to mount RBD device after
reboot
failure to break locks is usually due to incorrect permissions on the key,
check if the client key has "profile rbd" mon permissions
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at
https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 1:06 PM Vang Le-Quy <vle(a)its.aau.dk> wrote:
I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have
too ceph client n01 and n03 with following software versions.
# n01 ceph-common:
Installed: 12.2.8-0ubuntu0.18.04.1
Candidate: 12.2.12-0ubuntu0.18.04.2
# n03 ceph-common:
Installed: 12.2.11-0ubuntu0.18.04.2
Candidate: 12.2.12-0ubuntu0.18.04.2
# n01 uname -a
Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux
# n03 uname -a
Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
Problem:
N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is
something on client side, I upgrade n01 to have the same package version to
n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD
partition. Syslog shows messages like the following:
Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners
detected
Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems
dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of
client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire
lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners
detected
Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems
dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of
client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire
lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners
detected
Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems
dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of
client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire
lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners
detected
[image: signature_1482564062]
Vang Quy Le
Special Consultant in Data Science and Infrastructure
T: (+45) 9940 7710 | Email: vle(a)its.aau.dk <vle(a)its.aau.dk>
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
[1]
https://docs.ceph.com/docs/luminous/rbd/rados-rbd-cmds/#create-a-block-devi…
--
Jason