Hi everyone, apologies in advance; this will be long. It's also been through a bunch of edits and rewrites, so I don't know how well I'm expressing myself at this stage — please holler if anything is unclear and I'll be happy to try to clarify. I am currently in the process of investigating the behavior of OpenStack Nova instances when being snapshotted and suspended, in conjunction with qemu-guest-agent (qemu-ga). I realize that RBD-backed Nova/libvirt instances are expected to behave differently from file-backed ones, but I think I might have reason to believe that the RBD-backed ones are indeed behaving incorrectly, and I'd like to verify that. So first up, for comparison, let's recap how a Nova/libvirt/KVM instance behaves when it is *not* backed by RBD (such as, it's using a qcow2 file that is on a Nova compute node in /var/lib/nova/instances), is booted from an image with the hw_qemu_guest_agent=yes meta property set, and runs qemu-guest-agent within the guest: - User issues "nova suspend" or "openstack server suspend". - If nova-compute on the compute node decides that the instance has qemu-guest-agent running (which is the case if it's qemu or kvm, and its image has hw_qemu_guest_agent=yes), it sends a guest-sync command over the guest agent VirtIO serial port. This command registers in the qemu-ga log file in the guest. - nova-compute on the compute node sends a libvirt managed-save command. - Nova reports the instance as suspended. - User issues "nova resume" or "openstack server resume". - nova-compute on the compute node sends a libvirt start command. - Again, if nova-compute on the compute node knows that the instance has qemu-guest-agent running, it sends another command over the serial port, namely guest-set-time. This, too, registers in the guest's qemu-ga log. - Nova reports the instance as active (running normally) again. Now, when I instead use a Nova environment that is fully RBD-backed, I see exactly the same behavior as described above. So I know that in principle, nova-compute/qemu-ga communication works in both an RBD-backed and a non-RBD-backed environment. However, things appear to get very different when it comes to snapshots. Again, starting with a file-backed environment: - User issues "nova image-create" or "openstack server image create". - If nova-compute on the compute node decides that the instance can be quiesced (which is the case if it's qemu or kvm, and its image has hw_qemu_guest_agent=yes), then it sends a "guest-fsfreeze-freeze" command over the guest agent VirtIO serial port. - The guest agent inside the guest loops over all mounted filesystems, and issues the FIFREEZE ioctl (which maps to the kernel freeze_super() function). This can be seen in the qemu-ga log file in the guest, and it is also verifiable by using ftrace on the qemu-ga PID and checking for the freeze_super() function call. - nova-compute then takes a live snapshot of the instance. - Once complete, the guest gets a "guest-fsfreeze-thaw" command, and again I can see this in the qemu-ga log, and with ftrace. And now with RBD: - User issues "nova image-create" or "openstack server image create". - The guest-fsfreeze-freeze agent command never happens. Now I can see the info message from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… in my nova-compute log, which confirms that we're attempting a live snapshot. I also do *not* see the warning from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac…, so it looks like the direct_snapshot() call from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… succeeds. This is defined in https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… and it uses RBD functionality only. Importantly, it never interacts with qemu-ga, so it appears to not worry at all about freezing the filesystem. (Which does seem to contradict https://docs.ceph.com/docs/master/rbd/rbd-openstack/?highlight=uuid#image-p…, by the way, so that may be a documentation bug.) Now here's another interesting part. Were the direct snapshot to fail, if I read https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… and https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… correctly, the fallback behavior would be as follows: The domain would next be "suspended" (note, again this is Nova suspend, which maps to libvirt managed-save per https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac…), then snapshotted using a libvirt call and resumed again post-snapshot. In which case there would be a guest-sync call on suspend. And it's this part that has me a bit worried. If an RBD backed instance, on a successful snapshot, never freezes its filesystem *and* never does any kind of sync, either, doesn't that mean that such an instance can't be made to produce consistent snapshots? (Particularly in the case of write-back caching, which is recommended and normally safe for RBD/virtio devices.) Or is there some magic within the Qemu RBD storage driver that I am unaware of, that makes any such contortions unnecessary?

Thanks in advance for your insights! Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

On Aug 21, 2019, at 11:41 AM, Florian Haas <florian(a)citynetwork.eu> wrote: Hi Jason! Thanks for the quick reply. On 21/08/2019 16:51, Jason Dillaman wrote:> https://opendev.org/openstack/nova/commit/824c3706a3ea691781f4fcc4453881517… OK, wow... that was 4 years ago, does that mean that quiesce/freeze/thaw for RBD-backed Nova instances has probably been non-functional throughout that time? Looking at the reno for that commit I had an idea for a workaround: So, suppose that deployers running Nova with ephemeral disks on RBD prefer snapshot consistency over this shortcut. Until Nova fixes the direct_snapshot() call, I figured that such deployers could tweak the caps for the Nova CephX identity such that that user were no longer allowed to write to the Glance pool.

> On Aug 21, 2019, at 11:41 AM, Florian Haas <florian(a)citynetwork.eu> wrote: > > Hi Jason! Thanks for the quick reply. > > On 21/08/2019 16:51, Jason Dillaman wrote:> >> It just looks like this was an oversight from the OpenStack developers >> when Nova RBD "direct" ephemeral image snapshot support was added [1]. >> I would open a bug ticket against Nova for the issue.> >> [1] > https://opendev.org/openstack/nova/commit/824c3706a3ea691781f4fcc4453881517… > > OK, wow... that was 4 years ago, does that mean that quiesce/freeze/thaw > for RBD-backed Nova instances has probably been non-functional > throughout that time?

Yes, that would be my recommendation. > > Under those circumstances, the snapshot creation (in the ephemeral pool) > would work, but then the clone() call in this line should throw > nova.exception.Forbidden from an rbd.PermissionError: > > https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… > > Which should then trigger this except block: > > https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… > > ... and Nova/libvirt should go back to the (arguably more correct) fallback. > > Would you agree with that assessment, or am I missing something? (Just > trying to make sure that I don't give the Nova folks the wrong facts.) > > Thanks again! > > Cheers, > Florian

It just looks like this was an oversight from the OpenStack developers when Nova RBD "direct" ephemeral image snapshot support was added [1]. I would open a bug ticket against Nova for the issue.