Dear Cephalopodians, long time no write — please bear with the painful question about an Octopus cluster (upgrade planned) for which we'd like to switch from RBD Mirroring via journaling to snapshot-based mirroring (both clusters involved are "latest" Octopus). The RBD volumes are used via libvirt/QEMU(KVM), using librbd. We have first switched the pool mode over to "image" mode: rbd mirror pool enable rbd image (we were using pool before) to be able to use the snapshot mode for individual images. Then, after disabling mirroring completely for an image: rbd mirror image disable rbd/myvm-disk1 which works fine and disabled the journaling and journal-based mirroring, we try to switch to mirror snapshots: rbd --debug-rbd=6 mirror image enable rbd/myvm-disk1 snapshot Sadly, this reveals the messages appended in full below, the critical part (only visible with debug level >=5) seems to be: handle_acquire_lock: unable to acquire exclusive lock Naturally, the running VMs are holding an exclusive lock on the images, and we are also doing regular "normal" snapshots daily. Are we correct to interpret this as follows? * Journal-mirroring can be toggled any time, no exclusive lock required. * Enabling snapshot-based mirroring requires an exclusive lock(?), i.e. shutting down the affected VM? Or is there a better way (breaking the lock seems unsafe)? Is this expected behaviour, or a bug (and / or is this different in later Ceph versions), or are we missing a step / some configuration, or just misinterpreting the output? It would of course be great operationally if we could get away without a VM shutdown to toggle this :-). Furthermore, if somebody has experience with mirror snapshots in general, any experience is welcome. For reference, we are right now using journal-based mirroring and in parallel performing daily (regular) snapshots with fstrim/fsfreeze before each snapshot (via the qemu guest agent) and backing these up with Benji, so we have two backup paths in place (mirroring plus Benji to a separate system). We plan to also perform (manual/scripted) mirror snapshots in parallel to regular snapshots to profit from the advantages of the trim+freeze+snapshot approach :-). Cheers, Oliver ------------- 2026-10-28T13:15:22.838+0100 7fafca678500 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 ImageRequestWQ: ictx=0x55720aba5e20 2026-10-28T13:15:22.849+0100 7fafa8ff9700 5 librbd::ImageCtx: 0x55720aba5e20: disabling zero-copy writes 2026-10-28T13:15:22.852+0100 7fafa8ff9700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 block_writes: 0x55720aba5e20, num=1 2026-10-28T13:15:22.852+0100 7faf9bfff700 5 librbd::cache::WriteAroundObjectDispatch: 0x7faf94006020 init: 2026-10-28T13:15:22.852+0100 7faf9bfff700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 register_object_dispatch: object_dispatch_layer=1 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 SimpleSchedulerObjectDispatch: ictx=0x55720aba5e20 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 init: 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 register_object_dispatch: object_dispatch_layer=4 2026-10-28T13:15:22.857+0100 7fafa8ff9700 5 librbd::Operations: 0x55720ab986c0 snap_create: snap_name=.mirror.primary.919fa522-7d50-49c6-8b9d-e7926ade404a.ef881760-f1cf-4fcf-8fe5-cf491359bf4a 2026-10-28T13:15:22.859+0100 7fafa8ff9700 5 librbd::managed_lock::AcquireRequest: 0x7faf94007a40 handle_break_lock: lock owner is still alive 2026-10-28T13:15:22.859+0100 7faf9bfff700 5 librbd::ManagedLock: 0x7faf8c0150b8 handle_acquire_lock: unable to acquire exclusive lock 2026-10-28T13:15:22.860+0100 7fafa8ff9700 5 librbd::Watcher: 0x7faf8c003030 notifications_blocked: blocked=0 2026-10-28T13:15:25.126+0100 7faf9bfff700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7faf8c01a600 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2026-10-28T13:15:25.126+0100 7faf9bfff700 -1 librbd::mirror::EnableRequest: 0x55720aa7aa30 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2026-10-28T13:15:25.126+0100 7fafca678500 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument 2026-10-28T13:15:25.126+0100 7fafca678500 5 librbd::Watcher: 0x7faf8c003030 block_notifies: blocked_count=1 2026-10-28T13:15:25.126+0100 7faf9bfff700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 shut_down: shut_down: in_flight=0 2026-10-28T13:15:25.126+0100 7faf9bfff700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 unblock_writes: 0x55720aba5e20, num=0 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 shut_down: 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::ObjectDispatch: 0x55720abe98a0 shut_down: 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 shut_down: 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::cache::WriteAroundObjectDispatch: 0x7faf94006020 shut_down: -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --