Dear Eugen, many thanks for confirming! Then I think the only explanation left is that this was fixed in one of the Ceph releases in between. And the good news for sure is that it seems we have done "the right thing" as the commands are the same :-). A quick release-note-grepping does not yield anything rightaway, but that probably means we should upgrade in steps and retry enabling this from time to time. Upgrades will take us quite a while, as we are running with packages (several reasons, e.g. config management with Puppet and also hardware too old to be compatible with the latest container microarch requirements). Thanks a lot and cheers from Bonn, Oliver Am 28.10.26 um 15:48 schrieb Eugen Block:
Doesn't make a difference for me if there's an exclusive lock or not. Now I have one (enabled it in qemu.conf):
site1:~ # rbd lock ls pool1/cirros98 There is 1 exclusive lock on this image. Locker ID Address client.3228166 auto 140275846584448 192.168.124.16:0/2668393047
# rbd mirror image enable pool1/cirros98 journal Mirroring enabled
# rbd mirror image disable pool1/cirros98 Mirroring disabled
# rbd mirror image enable pool1/cirros98 snapshot Mirroring enabled
The lock is still present throughout these commands (watching it in a different console tab).
just wondering whether disabling snapshot mirroring also purges the primary mirror-snapshot right away
In my case it does remove the snapshots immediately.
Zitat von Eugen Block <eblock@nde.ag>:
Hm, I didn't look at lock ls before, just at the watchers. Currently, the primary site doesn't have a lock:
kvm host (and the primary ceph cluster, both clusters are single-host clusters):
site1:~ # rbd lock ls pool1/cirros98 site1:~ #
site2:~ # rbd lock ls pool1/cirros98 There is 1 exclusive lock on this image. Locker ID Address client.924272 auto 94778371914496 192.168.124.71:0/1924281534
I'll see if starting the vm with a lock makes a difference here.
Zitat von Oliver Freyermuth <freyermuth@physik.uni-bonn.de>:
Dear Eugen,
Am 28.10.26 um 14:59 schrieb Eugen Block:
I just installed kvm on the machine on the primary cluster where mirroring is enabled already. I started a vm from an existing rbd image, snapshot mirroring was already enabled automatically.
thanks a lot!
I disabled snapshot mirroring on that image, waited till it's gone one site 2. Then I enabled journal-based mirroring, waited a bit, disabled again and re-enabled snapshot mirroring, successfully.
This is indeed different from what we see... of course, in our case, snapshot mirroring was never active, as we already failed there. As the setup failed on our end when trying to create the first mirror snapshot while the VM is running, just wondering whether disabling snapshot mirroring also purges the primary mirror-snapshot right away — otherwise, this could hide the issue?
The image now has two watchers:
# rbd status pool1/cirros98 Watchers: watcher=192.168.124.71:0/3373425616 client.3224269 cookie=94778371912192 watcher=192.168.124.16:0/549636915 client.3228069 cookie=139998686980192
The second watcher is the kvm host, the other one is the mirroring-daemon. So I can't reproduce what you're experiencing.
That's intriguing, and likely means we should progress to a newer Ceph release (we should anyways)...
In case the setup is still up, it would also be interesting if you see an exclusive lock from the KVM host, here an example from our Octopus system:
# rbd lock ls rbd/bafgw.physik.uni-bonn.de-disk2 There is 1 exclusive lock on this image. Locker ID Address client.163639847 auto 140355336359952 10.160.12.8:0/3331485578
Docs also mention that even this exclusive lock should be handed over (and back) automatically: https://docs.ceph.com/en/pacific/rbd/rbd-exclusive-locks/ But it seems that this is not happening for us...
The image info on the remote site is this:
# rbd status pool1/cirros98 Watchers: watcher=192.168.124.71:0/1924281534 client.924272 cookie=9477837191449
Both clusters are running on 19.2.5 on openSUSE Leap 16.1 (but I don't think the OS matters).
On our end, it is RockyLinux 8, but I also do not think OS matters. KVM hosts are also using RockyLinux 8.
Cheers and thanks, Oliver
Zitat von Oliver Freyermuth <freyermuth@physik.uni-bonn.de>:
Hi Eugen,
many thanks in advance!
For reference, we have the feature "exclusive-lock" enabled on the volumes, i.e. our images have: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling (and journaling is gone of course when disabling journal mirroring).
However, it seems that journal-based mirroring can be toggled even without the RBD cli / librbd / rbd-mirroring-daemon getting an exclusive lock on the image first (that's always held by the actual running VM), while with snapshot-based mirroring, it just seems to give up already before enabling mirroring / creating the first mirror snapshot unless it can get an exclusive lock, which would require to shut down the VM. But hopefully there is a better way :-).
Cheers and thanks, Oliver
Am 28.10.26 um 14:09 schrieb Eugen Block:
Hi,
according to the docs [0] the exclusive-lock is also required for journal-based mirroring.
It's been a year or two since I last dealt with rbd-mirroring, one of our customers switched from mirroring between two clusters to a stretch cluster, so mirroring became obsolete.
But as far as I remember they never used journal-based mirroring and then switched, I know that they used snapshot-based mirroring until the switch to a stretched cluster. Let me check my notes, or maybe I still have my test clusters lying around...
[0] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/#enable-image-journaling-f...
Zitat von Oliver Freyermuth <ceph-users@ceph.io>:
> Dear Cephalopodians, > > long time no write — please bear with the painful question about an Octopus cluster (upgrade planned) for which we'd like to switch from RBD Mirroring via journaling to snapshot-based mirroring (both clusters involved are "latest" Octopus). > > The RBD volumes are used via libvirt/QEMU(KVM), using librbd. > > We have first switched the pool mode over to "image" mode: > rbd mirror pool enable rbd image > (we were using pool before) to be able to use the snapshot mode for individual images. > > Then, after disabling mirroring completely for an image: > rbd mirror image disable rbd/myvm-disk1 > which works fine and disabled the journaling and journal-based mirroring, we try to switch to mirror snapshots: > rbd --debug-rbd=6 mirror image enable rbd/myvm-disk1 snapshot > > Sadly, this reveals the messages appended in full below, the critical part (only visible with debug level >=5) seems to be: > handle_acquire_lock: unable to acquire exclusive lock > > Naturally, the running VMs are holding an exclusive lock on the images, and we are also doing regular "normal" snapshots daily. > > Are we correct to interpret this as follows? > * Journal-mirroring can be toggled any time, no exclusive lock required. > * Enabling snapshot-based mirroring requires an exclusive lock(?), i.e. shutting down the affected VM? Or is there a better way (breaking the lock seems unsafe)? > > Is this expected behaviour, or a bug (and / or is this different in later Ceph versions), or are we missing a step / some configuration, or just misinterpreting the output? > It would of course be great operationally if we could get away without a VM shutdown to toggle this :-). > > Furthermore, if somebody has experience with mirror snapshots in general, any experience is welcome. > For reference, we are right now using journal-based mirroring and in parallel performing daily (regular) snapshots with fstrim/fsfreeze before each snapshot (via the qemu guest agent) and backing these up with Benji, so we have two backup paths in place (mirroring plus Benji to a separate system). We plan to also perform (manual/scripted) mirror snapshots in parallel to regular snapshots to profit from the advantages of the trim+freeze+snapshot approach :-). > > Cheers, > Oliver > > ------------- > > 2026-10-28T13:15:22.838+0100 7fafca678500 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 ImageRequestWQ: ictx=0x55720aba5e20 > 2026-10-28T13:15:22.849+0100 7fafa8ff9700 5 librbd::ImageCtx: 0x55720aba5e20: disabling zero-copy writes > 2026-10-28T13:15:22.852+0100 7fafa8ff9700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 block_writes: 0x55720aba5e20, num=1 > 2026-10-28T13:15:22.852+0100 7faf9bfff700 5 librbd::cache::WriteAroundObjectDispatch: 0x7faf94006020 init: > 2026-10-28T13:15:22.852+0100 7faf9bfff700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 register_object_dispatch: object_dispatch_layer=1 > 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 SimpleSchedulerObjectDispatch: ictx=0x55720aba5e20 > 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 init: > 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 register_object_dispatch: object_dispatch_layer=4 > 2026-10-28T13:15:22.857+0100 7fafa8ff9700 5 librbd::Operations: 0x55720ab986c0 snap_create: snap_name=.mirror.primary.919fa522-7d50-49c6-8b9d-e7926ade404a.ef881760-f1cf-4fcf-8fe5-cf491359bf4a > 2026-10-28T13:15:22.859+0100 7fafa8ff9700 5 librbd::managed_lock::AcquireRequest: 0x7faf94007a40 handle_break_lock: lock owner is still alive > 2026-10-28T13:15:22.859+0100 7faf9bfff700 5 librbd::ManagedLock: 0x7faf8c0150b8 handle_acquire_lock: unable to acquire exclusive lock > 2026-10-28T13:15:22.860+0100 7fafa8ff9700 5 librbd::Watcher: 0x7faf8c003030 notifications_blocked: blocked=0 > 2026-10-28T13:15:25.126+0100 7faf9bfff700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7faf8c01a600 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument > 2026-10-28T13:15:25.126+0100 7faf9bfff700 -1 librbd::mirror::EnableRequest: 0x55720aa7aa30 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument > 2026-10-28T13:15:25.126+0100 7fafca678500 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument > 2026-10-28T13:15:25.126+0100 7fafca678500 5 librbd::Watcher: 0x7faf8c003030 block_notifies: blocked_count=1 > 2026-10-28T13:15:25.126+0100 7faf9bfff700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 shut_down: shut_down: in_flight=0 > 2026-10-28T13:15:25.126+0100 7faf9bfff700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 unblock_writes: 0x55720aba5e20, num=0 > 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 shut_down: > 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::ObjectDispatch: 0x55720abe98a0 shut_down: > 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 shut_down: > 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::cache::WriteAroundObjectDispatch: 0x7faf94006020 shut_down: > > > -- > Oliver Freyermuth > Universität Bonn > Physikalisches Institut, Raum 1.055 > Käthe-Kümmel-Straße 1 > 53115 Bonn > -- > Tel.: +49 228 73 2367 > Fax: +49 228 73 7869 > -- > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --
-- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --
-- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --