26 Jan
2023
26 Jan
'23
6:44 p.m.
Hi all,
we are observing a problem on a libvirt virtualisation cluster that
might come from ceph rbd clients. Something went wrong during execution
of a live-migration operation and as a result we have two instances of
the same VM running on 2 different hosts, the source- and the
destination host. What we observe now is the the exclusive lock of the
RBD disk image moves between these two clients periodically (every few
minutes the owner flips).
Hi Frank,
If you are talking about RBD exclusive lock feature ("exclusive-lock"
under "features" in "rbd info" output) then this is expected. This
feature provides automatic cooperative lock transitions between clients
to ensure that only a single client is writing to the image at any
given time. It's there to protect internal per-image data structures
such as the object map, the journal or the client-side PWL (persistent
write log) cache from concurrent modifications in case the image is
opened by two or more clients. The name is confusing but it's NOT
about preventing other clients from opening and writing to the image.
Rather it's about serializing those writes.
27 Jan
27 Jan
11:19 a.m.
New subject: Ceph rbd clients surrender exclusive lock in critical situation
Hi Mark,
thanks a lot! This seems to address the issue we observe, at least to a large degree.
I believe we had 2 VMs running after a failed live-migration as well and in this case it
doesn't seem like it will help. Maybe its possible to add a bit of logic for this case
as well (similar to fencing). My experience was that the write lock moves to the target VM
and then there is a reasonable time interval before it is handed back. This might be a
sufficient window of opportunity to kill hard a VM that should not run before it acquires
the write log again.
Thanks for that link! A script template like that could actually be added to the ceph
documentation under rbd locks. It seems to be a really important and useful use case for
image locking.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Marc <Marc(a)f1-outsourcing.eu>
Sent: 26 January 2023 18:44:41
To: Frank Schilder; 'ceph-users(a)ceph.io'
Subject: RE: [ceph-users] Re: Ceph rbd clients surrender exclusive lock in critical
situation
I can remember asking this also quite some time ago. Maybe this is helpful
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.wogri…
Hi all,
we are observing a problem on a libvirt virtualisation cluster that
might come
from ceph rbd clients. Something went wrong during execution
of a live-migration operation and as a result we have two instances of
the same VM running on 2 different hosts, the source- and the
destination host. What we observe now is the the exclusive lock of the
RBD disk image moves between these two clients periodically (every few
minutes the owner flips).
Hi Frank,
If you are talking about RBD exclusive lock feature ("exclusive-lock"
under "features" in "rbd info" output) then this is expected. This
feature provides automatic cooperative lock transitions between clients
to ensure that only a single client is writing to the image at any
given time. It's there to protect internal per-image data structures
such as the object map, the journal or the client-side PWL (persistent
write log) cache from concurrent modifications in case the image is
opened by two or more clients. The name is confusing but it's NOT
about preventing other clients from opening and writing to the image.
Rather it's about serializing those writes.
3:15 p.m.
New subject: Ceph rbd clients surrender exclusive lock in critical situation
On Fri, Jan 27, 2023 at 11:21 AM Frank Schilder <frans(a)dtu.dk> wrote:
Hi Mark,
thanks a lot! This seems to address the issue we observe, at least to a large degree.
I believe we had 2 VMs running after a failed live-migration as well and in this case it
doesn't seem like it will help. Maybe its possible to add a bit of logic for this case
as well (similar to fencing). My experience was that the write lock moves to the target VM
and then there is a reasonable time interval before it is handed back. This might be a
sufficient window of opportunity to kill hard a VM that should not run before it acquires
the write log again.
Thanks for that link! A script template like that could actually be added to the ceph
documentation under rbd locks. It seems to be a really important and useful use case for
image locking.
Hi Frank,
The script at [1] looks a bit suspicious to me because it uses shared
locking (--shared option) and checks whether the image is locked by
grepping "rbd lock list" output. There is a bunch of VM states
("migrate", "prepare", etc) and a couple of different lock IDs are
employed ("migrate", "startup", "libvirt") so I could be
wrong -- such
nasty state transitions may just not be possible in libvirt -- but
considered purely in isolation the following
function lock {
rbd=$1
locktype=$2
...
rbd lock add $rbd $locktype --shared libvirt
}
if is_locked $rbd libvirt
then
...
exit 257
fi
lock $rbd libvirt
< presumably VM is allowed to start >
could easily allow to start two VMs on the same $rbd image if invoked
in parallel on two different nodes.
For now, I have just updated the documentation at [2] to highlight and
warn about the automatic lock transitions behavior.
[1] https://www.wogri.at/scripts/ceph-libvirt-locking/
[2] https://docs.ceph.com/en/quincy/rbd/rbd-exclusive-locks/
Thanks,
Ilya
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Marc <Marc(a)f1-outsourcing.eu>
> Sent: 26 January 2023 18:44:41
> To: Frank Schilder; 'ceph-users(a)ceph.io'
> Subject: RE: [ceph-users] Re: Ceph rbd clients surrender exclusive lock in critical
situation
>
> > >
> > > Hi all,
> > >
> > > we are observing a problem on a libvirt virtualisation cluster that
> > might come from ceph rbd clients. Something went wrong during execution
> > of a live-migration operation and as a result we have two instances of
> > the same VM running on 2 different hosts, the source- and the
> > destination host. What we observe now is the the exclusive lock of the
> > RBD disk image moves between these two clients periodically (every few
> > minutes the owner flips).
> >
> > Hi Frank,
> >
> > If you are talking about RBD exclusive lock feature ("exclusive-lock"
> > under "features" in "rbd info" output) then this is
expected. This
> > feature provides automatic cooperative lock transitions between clients
> > to ensure that only a single client is writing to the image at any
> > given time. It's there to protect internal per-image data structures
> > such as the object map, the journal or the client-side PWL (persistent
> > write log) cache from concurrent modifications in case the image is
> > opened by two or more clients. The name is confusing but it's NOT
> > about preventing other clients from opening and writing to the image.
> > Rather it's about serializing those writes.
> >
>
>
> I can remember asking this also quite some time ago. Maybe this is helpful
>
>
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.wogri…
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:09 p.m.
New subject: Ceph rbd clients surrender exclusive lock in critical situation
Hi Ilya,
yes, it has race conditions. However, it seems to address the specific case that is
causing us headaches.
About possible improvements. I tried to understand the documentation about rbd image
locks, but probably failed. I don't understand what the difference between an
exclusive lock and a shared lock is. As far as I understood, neither of these locks have
any influence on client IO, nor do they affect the exclusive write-lock transition.
Does omitting --shared make the rbd lock command an atomic operation, meaning that lock
operations become serialised and two clients cannot add the same lock in a race (only one
client will succeed in a race)? Or does it mean that only one client gets write access? If
it does have influence on client IO, how would the hand-over during live migration have to
happen?
Thanks for adding a note to the documentation and any further help!
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
behaviour.
4:31 p.m.
New subject: Ceph rbd clients surrender exclusive lock in critical situation
On Fri, Jan 27, 2023 at 4:09 PM Frank Schilder <frans(a)dtu.dk> wrote:
Hi Ilya,
yes, it has race conditions. However, it seems to address the specific case that is
causing us headaches.
About possible improvements. I tried to understand the documentation about rbd image
locks, but probably failed. I don't understand what the difference between an
exclusive lock and a shared lock is. As far as I understood, neither of these locks have
any influence on client IO, nor do they affect the exclusive write-lock transition.
An exclusive lock can be held by only one client at a time. A shared
lock can be held by multiple clients as long as they all pass the same
lock tag ("--shared <lock tag>" option on the CLI).
Does omitting --shared make the rbd lock command an atomic operation, meaning that lock
operations become serialised and two clients cannot add the same lock in a race (only one
client will succeed in a race)?
Yes.
Or does it mean that only one client gets write
access?
No -- "rbd lock" locks are advisory. They are there for the
orchestration layer to take advantage of. Any client that hasn't been
prevented from doing so by the orchestration layer is going to be able
to write to the image, whether it's holding such a lock or not.
This is similar in concept to Unix file locking ("man 2 flock").
If it does have influence on client IO, how would the
hand-over during live migration have to happen?
Being advisory, "rbd lock" locks don't have a direct influence on
client I/O. Only the lock provided by the exclusive-lock image feature
does.
Thanks,
Ilya
4:43 p.m.
New subject: Ceph rbd clients surrender exclusive lock in critical situation
Hi Ilya,
thanks for that information. It sounds like one can use exclusive locks in the hook script
to resolve race conditions. I will have a look. Not sure if it will help reducing the
number of states (shared log tags) though.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Ilya Dryomov <idryomov(a)gmail.com>
Sent: 27 January 2023 16:31:26
To: Frank Schilder
Cc: Marc; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: Ceph rbd clients surrender exclusive lock in critical
situation
On Fri, Jan 27, 2023 at 4:09 PM Frank Schilder <frans(a)dtu.dk> wrote:
Hi Ilya,
yes, it has race conditions. However, it seems to address the specific case that is
causing us headaches.
About possible improvements. I tried to understand the documentation about rbd image
locks, but probably failed. I don't understand what the difference between an
exclusive lock and a shared lock is. As far as I understood, neither of these locks have
any influence on client IO, nor do they affect the exclusive write-lock transition.
An exclusive lock can be held by only one client at a time. A shared
lock can be held by multiple clients as long as they all pass the same
lock tag ("--shared <lock tag>" option on the CLI).
Does omitting --shared make the rbd lock command an atomic operation, meaning that lock
operations become serialised and two clients cannot add the same lock in a race (only one
client will succeed in a race)?
Yes.
Or does it mean that only one client gets write
access?
No -- "rbd lock" locks are advisory. They are there for the
orchestration layer to take advantage of. Any client that hasn't been
prevented from doing so by the orchestration layer is going to be able
to write to the image, whether it's holding such a lock or not.
This is similar in concept to Unix file locking ("man 2 flock").
If it does have influence on client IO, how would the
hand-over during live migration have to happen?
Being advisory, "rbd lock" locks don't have a direct influence on
client I/O. Only the lock provided by the exclusive-lock image feature
does.
Thanks,
Ilya
28 Jan
28 Jan
1:13 p.m.
New subject: Ceph rbd clients surrender exclusive lock in critical situation
Everybody has apparatuses at home and some of the time they breakdown. I prescribe you to
https://serviceservotech.com/appliance-repair/ utilize a help that can rapidly and
proficiently fix them so you don't need to purchase another one. I have been to this
help oftentimes previously and they have consistently given me quality assistance.
463
days inactive
465
days old
6 comments
4 participants
participants (4)
-
Frank Schilder -
Ilya Dryomov -
Marc -
Mored1948＠jourrapide.com