6 Sep
2019
6 Sep
'19
3:34 p.m.
I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client
n01 and n03 with following software versions.
# n01 ceph-common:
Installed: 12.2.8-0ubuntu0.18.04.1
Candidate: 12.2.12-0ubuntu0.18.04.2
# n03 ceph-common:
Installed: 12.2.11-0ubuntu0.18.04.2
Candidate: 12.2.12-0ubuntu0.18.04.2
# n01 uname -a
Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64
x86_64 GNU/Linux
# n03 uname -a
Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
Problem:
N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on
client side, I upgrade n01 to have the same package version to n03 including ceph-common.
And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the
following:
Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners detected
[signature_1482564062]
Vang Quy Le
Special Consultant in Data Science and Infrastructure
T: (+45) 9940 7710 | Email: vle@its.aau.dk<mailto:vle@its.aau.dk>
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |
6 Sep
6 Sep
3:43 p.m.
failure to break locks is usually due to incorrect permissions on the key,
check if the client key has "profile rbd" mon permissions
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 1:06 PM Vang Le-Quy <vle(a)its.aau.dk> wrote:
I am suspecting that there is a bug in
12.2.8-0ubuntu0.18.04.1. We have
too ceph client n01 and n03 with following software versions.
# n01 ceph-common:
Installed: 12.2.8-0ubuntu0.18.04.1
Candidate: 12.2.12-0ubuntu0.18.04.2
# n03 ceph-common:
Installed: 12.2.11-0ubuntu0.18.04.2
Candidate: 12.2.12-0ubuntu0.18.04.2
# n01 uname -a
Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux
# n03 uname -a
Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
Problem:
N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is
something on client side, I upgrade n01 to have the same package version to
n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD
partition. Syslog shows messages like the following:
Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners
detected
Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems
dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of
client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire
lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners
detected
Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems
dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of
client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire
lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners
detected
Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems
dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of
client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire
lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners
detected
[image: signature_1482564062]
Vang Quy Le
Special Consultant in Data Science and Infrastructure
T: (+45) 9940 7710 | Email: vle(a)its.aau.dk <vle(a)its.aau.dk>
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:01 p.m.
Hi Paul
Thank you for suggestion.
Firstly, a correction “I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1” is
incorrect. I meant that version 12.2.8-0ubuntu0.18.04.1 on node N01 works, .
12.2.12-0ubuntu0.18.04.2 is having problem.
After upgrading to version 12.2.12-0ubuntu0.18.04.2, it doesn’t work anymore. No changes
on client keys file or on the Ceph cluster. Therefore I deduce that it is a problem with
client (ceph-common) software.
~ Vang
From: Paul Emmerich <paul.emmerich(a)croit.io>
Date: Friday, 6 September 2019 at 13.13
To: Vang Le-Quy <vle(a)its.aau.dk>
Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io>
Subject: Re: [ceph-users] Ceph client failed to mount RBD device after reboot
failure to break locks is usually due to incorrect permissions on the key, check if the
client key has "profile rbd" mon permissions
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io<http://www.croit.io>
Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 1:06 PM Vang Le-Quy
<vle@its.aau.dk<mailto:vle@its.aau.dk>> wrote:
I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client
n01 and n03 with following software versions.
# n01 ceph-common:
Installed: 12.2.8-0ubuntu0.18.04.1
Candidate: 12.2.12-0ubuntu0.18.04.2
# n03 ceph-common:
Installed: 12.2.11-0ubuntu0.18.04.2
Candidate: 12.2.12-0ubuntu0.18.04.2
# n01 uname -a
Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64
x86_64 GNU/Linux
# n03 uname -a
Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
Problem:
N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on
client side, I upgrade n01 to have the same package version to n03 including ceph-common.
And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the
following:
Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners detected
[signature_1482564062]
Vang Quy Le
Special Consultant in Data Science and Infrastructure
T: (+45) 9940 7710 | Email: vle@its.aau.dk<mailto:vle@its.aau.dk>
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
4:36 p.m.
On Fri, Sep 6, 2019 at 7:33 AM Vang Le-Quy <vle(a)its.aau.dk> wrote:
Hi Paul
Thank you for suggestion.
Firstly, a correction “I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1”
is incorrect. I meant that version 12.2.8-0ubuntu0.18.04.1 on node N01
works, . 12.2.12-0ubuntu0.18.04.2 is having problem.
After upgrading to version 12.2.12-0ubuntu0.18.04.2, it doesn’t work
anymore. No changes on client keys file or on the Ceph cluster. Therefore
I deduce that it is a problem with client (ceph-common) software.
Looks like a CephX caps issue for your client. Please refer to the
documentation and verify you are using the correct caps [1].
~ Vang
*From: *Paul Emmerich <paul.emmerich(a)croit.io>
*Date: *Friday, 6 September 2019 at 13.13
*To: *Vang Le-Quy <vle(a)its.aau.dk>
*Cc: *"ceph-users(a)ceph.io" <ceph-users(a)ceph.io>
*Subject: *Re: [ceph-users] Ceph client failed to mount RBD device after
reboot
failure to break locks is usually due to incorrect permissions on the key,
check if the client key has "profile rbd" mon permissions
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 1:06 PM Vang Le-Quy <vle(a)its.aau.dk> wrote:
I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have
too ceph client n01 and n03 with following software versions.
# n01 ceph-common:
Installed: 12.2.8-0ubuntu0.18.04.1
Candidate: 12.2.12-0ubuntu0.18.04.2
# n03 ceph-common:
Installed: 12.2.11-0ubuntu0.18.04.2
Candidate: 12.2.12-0ubuntu0.18.04.2
# n01 uname -a
Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux
# n03 uname -a
Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
Problem:
N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is
something on client side, I upgrade n01 to have the same package version to
n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD
partition. Syslog shows messages like the following:
Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners
detected
Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems
dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of
client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire
lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners
detected
Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems
dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of
client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire
lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners
detected
Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems
dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of
client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire
lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners
detected
[image: signature_1482564062]
Vang Quy Le
Special Consultant in Data Science and Infrastructure
T: (+45) 9940 7710 | Email: vle(a)its.aau.dk <vle(a)its.aau.dk>
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
[1]
https://docs.ceph.com/docs/luminous/rbd/rados-rbd-cmds/#create-a-block-devi…
--
Jason
4:53 p.m.
Hi Jason
My client key content looks like the following:
#####[
[client.n01]
key = AQA301KcfFVtKxAAyJxY9uTvNh3gn7eH3iHhKw==
#####
So Paul and you may be right, the key lacks suitable caps. Can I just manually add caps to
the key file or it has to be regenerated from the CEPH cluster?
From: Jason Dillaman <jdillama(a)redhat.com>
Reply-To: "dillaman(a)redhat.com" <dillaman(a)redhat.com>
Date: Friday, 6 September 2019 at 14.06
To: Vang Le-Quy <vle(a)its.aau.dk>
Cc: Paul Emmerich <paul.emmerich(a)croit.io>io>, "ceph-users(a)ceph.io"
<ceph-users(a)ceph.io>
Subject: Re: [ceph-users] Re: Ceph client failed to mount RBD device after reboot
On Fri, Sep 6, 2019 at 7:33 AM Vang Le-Quy
<vle@its.aau.dk<mailto:vle@its.aau.dk>> wrote:
Hi Paul
Thank you for suggestion.
Firstly, a correction “I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1” is
incorrect. I meant that version 12.2.8-0ubuntu0.18.04.1 on node N01 works, .
12.2.12-0ubuntu0.18.04.2 is having problem.
After upgrading to version 12.2.12-0ubuntu0.18.04.2, it doesn’t work anymore. No changes
on client keys file or on the Ceph cluster. Therefore I deduce that it is a problem with
client (ceph-common) software.
Looks like a CephX caps issue for your client. Please refer to the documentation and
verify you are using the correct caps [1].
~ Vang
From: Paul Emmerich <paul.emmerich@croit.io<mailto:paul.emmerich@croit.io>>
Date: Friday, 6 September 2019 at 13.13
To: Vang Le-Quy <vle@its.aau.dk<mailto:vle@its.aau.dk>>
Cc: "ceph-users@ceph.io<mailto:ceph-users@ceph.io>"
<ceph-users@ceph.io<mailto:ceph-users@ceph.io>>
Subject: Re: [ceph-users] Ceph client failed to mount RBD device after reboot
failure to break locks is usually due to incorrect permissions on the key, check if the
client key has "profile rbd" mon permissions
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io<http://www.croit.io>
Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 1:06 PM Vang Le-Quy
<vle@its.aau.dk<mailto:vle@its.aau.dk>> wrote:
I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client
n01 and n03 with following software versions.
# n01 ceph-common:
Installed: 12.2.8-0ubuntu0.18.04.1
Candidate: 12.2.12-0ubuntu0.18.04.2
# n03 ceph-common:
Installed: 12.2.11-0ubuntu0.18.04.2
Candidate: 12.2.12-0ubuntu0.18.04.2
# n01 uname -a
Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64
x86_64 GNU/Linux
# n03 uname -a
Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
Problem:
N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on
client side, I upgrade n01 to have the same package version to n03 including ceph-common.
And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the
following:
Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners detected
[signature_1482564062]
Vang Quy Le
Special Consultant in Data Science and Infrastructure
T: (+45) 9940 7710 | Email: vle@its.aau.dk<mailto:vle@its.aau.dk>
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
[1] https://docs.ceph.com/docs/luminous/rbd/rados-rbd-cmds/#create-a-block-devi…
--
Jason
7 Sep
7 Sep
1:12 a.m.
Hi
ceph auth caps client.n01 mon ‘profile rbd’ osd ‘profile rbd’
should do the trick.
JC
On Sep 6, 2019, at 05:23, Vang Le-Quy
<vle(a)its.aau.dk> wrote:
Hi Jason
My client key content looks like the following:
#####[
[client.n01]
key = AQA301KcfFVtKxAAyJxY9uTvNh3gn7eH3iHhKw==
#####
So Paul and you may be right, the key lacks suitable caps. Can I just manually add caps
to the key file or it has to be regenerated from the CEPH cluster?
From: Jason Dillaman <jdillama(a)redhat.com <mailto:jdillama@redhat.com>>
Reply-To: "dillaman(a)redhat.com <mailto:dillaman@redhat.com>"
<dillaman(a)redhat.com <mailto:dillaman@redhat.com>>
Date: Friday, 6 September 2019 at 14.06
To: Vang Le-Quy <vle(a)its.aau.dk <mailto:vle@its.aau.dk>>
Cc: Paul Emmerich <paul.emmerich(a)croit.io <mailto:paul.emmerich@croit.io>>,
"ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>" <ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>>
Subject: Re: [ceph-users] Re: Ceph client failed to mount RBD device after reboot
On Fri, Sep 6, 2019 at 7:33 AM Vang Le-Quy <vle(a)its.aau.dk
<mailto:vle@its.aau.dk>> wrote:
Hi Paul
Thank you for suggestion.
Firstly, a correction “I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1” is
incorrect. I meant that version 12.2.8-0ubuntu0.18.04.1 on node N01 works, .
12.2.12-0ubuntu0.18.04.2 is having problem.
After upgrading to version 12.2.12-0ubuntu0.18.04.2, it doesn’t work anymore. No changes
on client keys file or on the Ceph cluster. Therefore I deduce that it is a problem with
client (ceph-common) software.
Looks like a CephX caps issue for your client. Please refer to the documentation and
verify you are using the correct caps [1].
~ Vang
From: Paul Emmerich <paul.emmerich(a)croit.io <mailto:paul.emmerich@croit.io>>
Date: Friday, 6 September 2019 at 13.13
To: Vang Le-Quy <vle(a)its.aau.dk <mailto:vle@its.aau.dk>>
Cc: "ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>"
<ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>>
Subject: Re: [ceph-users] Ceph client failed to mount RBD device after reboot
failure to break locks is usually due to incorrect permissions on the key, check if the
client key has "profile rbd" mon permissions
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
<https://croit.io/>
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io <http://www.croit.io/>
Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 1:06 PM Vang Le-Quy <vle(a)its.aau.dk
<mailto:vle@its.aau.dk>> wrote:
I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client
n01 and n03 with following software versions.
# n01 ceph-common:
Installed: 12.2.8-0ubuntu0.18.04.1
Candidate: 12.2.12-0ubuntu0.18.04.2
# n03 ceph-common:
Installed: 12.2.11-0ubuntu0.18.04.2
Candidate: 12.2.12-0ubuntu0.18.04.2
# n01 uname -a
Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64
x86_64 GNU/Linux
# n03 uname -a
Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
Problem:
N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on
client side, I upgrade n01 to have the same package version to n03 including ceph-common.
And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the
following:
Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems dead, breaking
lock
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of client2508785 failed:
-13
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners detected
<image001.png>
Vang Quy Le
Special Consultant in Data Science and Infrastructure
T: (+45) 9940 7710 | Email: vle(a)its.aau.dk <mailto:vle@its.aau.dk>
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
[1] https://docs.ceph.com/docs/luminous/rbd/rados-rbd-cmds/#create-a-block-devi…
<https://docs.ceph.com/docs/luminous/rbd/rados-rbd-cmds/#create-a-block-device-user>
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
1704
days inactive
1704
days old
5 comments
4 participants
participants (4)
-
Jason Dillaman -
JC Lopez -
Paul Emmerich -
Vang Le-Quy