I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client n01 and n03 with following software versions.

# n01 ceph-common:

  Installed: 12.2.8-0ubuntu0.18.04.1

  Candidate: 12.2.12-0ubuntu0.18.04.2

# n03 ceph-common:

  Installed: 12.2.11-0ubuntu0.18.04.2

  Candidate: 12.2.12-0ubuntu0.18.04.2

# n01 uname -a

Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# n03 uname -a

Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Problem:

N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on client side, I upgrade n01 to have the same package version to n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the following:

Sep  6 12:52:33 n01 kernel: [  875.160275] rbd: rbd0: no lock owners detected

Sep  6 12:52:33 n01 kernel: [  875.161069] rbd: rbd0: client2508785 seems dead, breaking lock

Sep  6 12:52:33 n01 kernel: [  875.161723] rbd: rbd0: blacklist of client2508785 failed: -13

Sep  6 12:52:33 n01 kernel: [  875.161724] rbd: rbd0: failed to acquire lock: -13

Sep  6 12:52:33 n01 kernel: [  875.161956] rbd: rbd0: no lock owners detected

Sep  6 12:52:33 n01 kernel: [  875.162755] rbd: rbd0: client2508785 seems dead, breaking lock

Sep  6 12:52:33 n01 kernel: [  875.163415] rbd: rbd0: blacklist of client2508785 failed: -13

Sep  6 12:52:33 n01 kernel: [  875.163415] rbd: rbd0: failed to acquire lock: -13

Sep  6 12:52:33 n01 kernel: [  875.163645] rbd: rbd0: no lock owners detected

Sep  6 12:52:33 n01 kernel: [  875.164431] rbd: rbd0: client2508785 seems dead, breaking lock

Sep  6 12:52:33 n01 kernel: [  875.165082] rbd: rbd0: blacklist of client2508785 failed: -13

Sep  6 12:52:33 n01 kernel: [  875.165082] rbd: rbd0: failed to acquire lock: -13

Sep  6 12:52:33 n01 kernel: [  875.165355] rbd: rbd0: no lock owners detected

Vang Quy Le
Special Consultant in Data Science and Infrastructure 

T: (+45) 9940 7710 | Email: vle@its.aau.dk
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |