Fwd: question on rbd locks
Hello All, Is there a way to specify that a lock (shared or exclusive) on an rbd volume be released if the client machine becomes unreachable or irresponsive? In one of our clusters, we use rbd locks on volumes to make sure provide a kind of shared or exclusive access - to make sure there are no writers when someone is reading and there are no readers when someone is writing. However, we often run into issues when one of the machines gets into kernel panic or something and the whole pipeline gets stalled. So is there a way to tell ceph to release the lock if the client becomes unavailable? Thanks, Shridhar
On Tue, Apr 7, 2020 at 6:49 PM Void Star Nill <void.star.nill@gmail.com> wrote:
So is there a way to tell ceph to release the lock if the client becomes unavailable?
That's the task of the new client trying to take the lock, it needs to kick out the old client and blacklist the connection to ensure consistency. A common problem is using incorrect permissions on the Ceph keyring. Check that you are using "profile rbd" for mon permissions Paul
Thanks, Shridhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Apr 7, 2020 at 6:49 PM Void Star Nill <void.star.nill@gmail.com> wrote:
Hello All,
Is there a way to specify that a lock (shared or exclusive) on an rbd volume be released if the client machine becomes unreachable or irresponsive?
In one of our clusters, we use rbd locks on volumes to make sure provide a kind of shared or exclusive access - to make sure there are no writers when someone is reading and there are no readers when someone is writing.
However, we often run into issues when one of the machines gets into kernel panic or something and the whole pipeline gets stalled.
What kind of kernel panics are you running into? Do you have any panic messages or stack traces captured? Thanks, Ilya
I experienced an issue where lock didn't get cleared automatically on RBDs. When a kvm hosts crashed the locks never cleared. It was a permission issue on cephx. Maybe test with a admin user? Maybe post what permissions you have for that user with `ceph auth list`? Glen -----Original Message----- From: Ilya Dryomov <idryomov@gmail.com> Sent: Wednesday, 8 April 2020 5:48 PM To: Void Star Nill <void.star.nill@gmail.com> Cc: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: Fwd: question on rbd locks On Tue, Apr 7, 2020 at 6:49 PM Void Star Nill <void.star.nill@gmail.com> wrote:
Hello All,
Is there a way to specify that a lock (shared or exclusive) on an rbd volume be released if the client machine becomes unreachable or irresponsive?
In one of our clusters, we use rbd locks on volumes to make sure provide a kind of shared or exclusive access - to make sure there are no writers when someone is reading and there are no readers when someone is writing.
However, we often run into issues when one of the machines gets into kernel panic or something and the whole pipeline gets stalled.
What kind of kernel panics are you running into? Do you have any panic messages or stack traces captured? Thanks, Ilya _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.
Thanks Ilya, Paul. I dont have the panic traces and probably they are not related to rbd. I was merely describing our use case. On our setup that we manage, we have a software layer similar to Kubernetes CSI that orchestrates the volume map/unmap on behalf of the users. We are currently using volume locks as a way to protect the volumes from inadvertent concurrent write mounts that could lead to FS corruption as most of the volumes run with ext3/4. So in our orchestration, we take a shared on volumes that are read-only mounts, thus we can allow concurrent multiple read-only mounts, and we take exclusive lock for read-write mounts so that we can reject other RO/RW mounts while the first RW mount is in use. All this orchestration happens in a distributed manner across all our compute nodes - so it is not easy to determine when we should kick out the dead connections and claim the lock. We need to intervene manually and resolve such issues as of now. So I am looking for a way to do this deterministically. Thanks, Shridhar On Wed, 8 Apr 2020 at 02:48, Ilya Dryomov <idryomov@gmail.com> wrote:
On Tue, Apr 7, 2020 at 6:49 PM Void Star Nill <void.star.nill@gmail.com> wrote:
Hello All,
Is there a way to specify that a lock (shared or exclusive) on an rbd volume be released if the client machine becomes unreachable or irresponsive?
In one of our clusters, we use rbd locks on volumes to make sure provide
a
kind of shared or exclusive access - to make sure there are no writers when someone is reading and there are no readers when someone is writing.
However, we often run into issues when one of the machines gets into kernel panic or something and the whole pipeline gets stalled.
What kind of kernel panics are you running into? Do you have any panic messages or stack traces captured?
Thanks,
Ilya
Paul, Ilya, others, Any inputs on this? Thanks, Shridhar On Thu, 9 Apr 2020 at 12:30, Void Star Nill <void.star.nill@gmail.com> wrote:
Thanks Ilya, Paul.
I dont have the panic traces and probably they are not related to rbd. I was merely describing our use case.
On our setup that we manage, we have a software layer similar to Kubernetes CSI that orchestrates the volume map/unmap on behalf of the users. We are currently using volume locks as a way to protect the volumes from inadvertent concurrent write mounts that could lead to FS corruption as most of the volumes run with ext3/4.
So in our orchestration, we take a shared on volumes that are read-only mounts, thus we can allow concurrent multiple read-only mounts, and we take exclusive lock for read-write mounts so that we can reject other RO/RW mounts while the first RW mount is in use.
All this orchestration happens in a distributed manner across all our compute nodes - so it is not easy to determine when we should kick out the dead connections and claim the lock. We need to intervene manually and resolve such issues as of now. So I am looking for a way to do this deterministically.
Thanks, Shridhar
On Wed, 8 Apr 2020 at 02:48, Ilya Dryomov <idryomov@gmail.com> wrote:
On Tue, Apr 7, 2020 at 6:49 PM Void Star Nill <void.star.nill@gmail.com> wrote:
Hello All,
Is there a way to specify that a lock (shared or exclusive) on an rbd volume be released if the client machine becomes unreachable or irresponsive?
In one of our clusters, we use rbd locks on volumes to make sure
provide a
kind of shared or exclusive access - to make sure there are no writers when someone is reading and there are no readers when someone is writing.
However, we often run into issues when one of the machines gets into kernel panic or something and the whole pipeline gets stalled.
What kind of kernel panics are you running into? Do you have any panic messages or stack traces captured?
Thanks,
Ilya
As Paul said, a lock is typically broken by a new client trying to grab it. As part of that the existing lock holder needs to be blacklisted, unless you fence using some type of STONITH. The question of whether the existing lock holder is dead can't be answered in isolation. For example, the exclusive-lock feature (automatic cooperative lock transitions to ensure that only a single client is writing to the image at any given time) uses watches. If the existing lock holder has a watch, it is considered alive and the lock is requested cooperatively. Otherwise, it is considered dead and the lock is broken. This is implemented with care to avoid various corner cases related to watches and blacklisting: the client will not grab the lock without having a watch established, the client will update the lock cookie if the watch is lost and reestablished, the client will not use pre-blacklist osdmaps for any post-blacklist I/O, etc. Since you are grabbing locks manually in the orchestration layer, it is up to the orchestration layer to decide when (and how) to break them. rbd can't make that decision for you -- consider a case where the device is alive and ready to serve I/O, but the workload is stuck for some other reason. Thanks, Ilya On Sun, Apr 12, 2020 at 8:42 PM Void Star Nill <void.star.nill@gmail.com> wrote:
Paul, Ilya, others,
Any inputs on this?
Thanks, Shridhar
On Thu, 9 Apr 2020 at 12:30, Void Star Nill <void.star.nill@gmail.com> wrote:
Thanks Ilya, Paul.
I dont have the panic traces and probably they are not related to rbd. I was merely describing our use case.
On our setup that we manage, we have a software layer similar to Kubernetes CSI that orchestrates the volume map/unmap on behalf of the users. We are currently using volume locks as a way to protect the volumes from inadvertent concurrent write mounts that could lead to FS corruption as most of the volumes run with ext3/4.
So in our orchestration, we take a shared on volumes that are read-only mounts, thus we can allow concurrent multiple read-only mounts, and we take exclusive lock for read-write mounts so that we can reject other RO/RW mounts while the first RW mount is in use.
All this orchestration happens in a distributed manner across all our compute nodes - so it is not easy to determine when we should kick out the dead connections and claim the lock. We need to intervene manually and resolve such issues as of now. So I am looking for a way to do this deterministically.
Thanks, Shridhar
On Wed, 8 Apr 2020 at 02:48, Ilya Dryomov <idryomov@gmail.com> wrote:
On Tue, Apr 7, 2020 at 6:49 PM Void Star Nill <void.star.nill@gmail.com> wrote:
Hello All,
Is there a way to specify that a lock (shared or exclusive) on an rbd volume be released if the client machine becomes unreachable or irresponsive?
In one of our clusters, we use rbd locks on volumes to make sure provide a kind of shared or exclusive access - to make sure there are no writers when someone is reading and there are no readers when someone is writing.
However, we often run into issues when one of the machines gets into kernel panic or something and the whole pipeline gets stalled.
What kind of kernel panics are you running into? Do you have any panic messages or stack traces captured?
Thanks,
Ilya
That makes sense. Thanks Ilya. On Mon, Apr 13, 2020 at 4:10 AM Ilya Dryomov <idryomov@gmail.com> wrote:
As Paul said, a lock is typically broken by a new client trying to grab it. As part of that the existing lock holder needs to be blacklisted, unless you fence using some type of STONITH.
The question of whether the existing lock holder is dead can't be answered in isolation. For example, the exclusive-lock feature (automatic cooperative lock transitions to ensure that only a single client is writing to the image at any given time) uses watches. If the existing lock holder has a watch, it is considered alive and the lock is requested cooperatively. Otherwise, it is considered dead and the lock is broken. This is implemented with care to avoid various corner cases related to watches and blacklisting: the client will not grab the lock without having a watch established, the client will update the lock cookie if the watch is lost and reestablished, the client will not use pre-blacklist osdmaps for any post-blacklist I/O, etc.
Since you are grabbing locks manually in the orchestration layer, it is up to the orchestration layer to decide when (and how) to break them. rbd can't make that decision for you -- consider a case where the device is alive and ready to serve I/O, but the workload is stuck for some other reason.
Thanks,
Ilya
On Sun, Apr 12, 2020 at 8:42 PM Void Star Nill <void.star.nill@gmail.com> wrote:
Paul, Ilya, others,
Any inputs on this?
Thanks, Shridhar
On Thu, 9 Apr 2020 at 12:30, Void Star Nill <void.star.nill@gmail.com>
Thanks Ilya, Paul.
I dont have the panic traces and probably they are not related to rbd.
I was merely describing our use case.
On our setup that we manage, we have a software layer similar to
Kubernetes CSI that orchestrates the volume map/unmap on behalf of the users. We are currently using volume locks as a way to protect the volumes from inadvertent concurrent write mounts that could lead to FS corruption as most of the volumes run with ext3/4.
So in our orchestration, we take a shared on volumes that are read-only
mounts, thus we can allow concurrent multiple read-only mounts, and we take exclusive lock for read-write mounts so that we can reject other RO/RW mounts while the first RW mount is in use.
All this orchestration happens in a distributed manner across all our
compute nodes - so it is not easy to determine when we should kick out the dead connections and claim the lock. We need to intervene manually and resolve such issues as of now. So I am looking for a way to do this deterministically.
Thanks, Shridhar
On Wed, 8 Apr 2020 at 02:48, Ilya Dryomov <idryomov@gmail.com> wrote:
On Tue, Apr 7, 2020 at 6:49 PM Void Star Nill <
void.star.nill@gmail.com> wrote:
Hello All,
Is there a way to specify that a lock (shared or exclusive) on an rbd volume be released if the client machine becomes unreachable or irresponsive?
In one of our clusters, we use rbd locks on volumes to make sure
wrote: provide a
kind of shared or exclusive access - to make sure there are no writers when someone is reading and there are no readers when someone is writing.
However, we often run into issues when one of the machines gets into kernel panic or something and the whole pipeline gets stalled.
What kind of kernel panics are you running into? Do you have any panic messages or stack traces captured?
Thanks,
Ilya
participants (4)
-
Glen Baars
-
Ilya Dryomov
-
Paul Emmerich
-
Void Star Nill