RBD, OpenStack Nova, libvirt, qemu-guest-agent, and FIFREEZE: is this working as intended?
Hi everyone, apologies in advance; this will be long. It's also been through a bunch of edits and rewrites, so I don't know how well I'm expressing myself at this stage — please holler if anything is unclear and I'll be happy to try to clarify. I am currently in the process of investigating the behavior of OpenStack Nova instances when being snapshotted and suspended, in conjunction with qemu-guest-agent (qemu-ga). I realize that RBD-backed Nova/libvirt instances are expected to behave differently from file-backed ones, but I think I might have reason to believe that the RBD-backed ones are indeed behaving incorrectly, and I'd like to verify that. So first up, for comparison, let's recap how a Nova/libvirt/KVM instance behaves when it is *not* backed by RBD (such as, it's using a qcow2 file that is on a Nova compute node in /var/lib/nova/instances), is booted from an image with the hw_qemu_guest_agent=yes meta property set, and runs qemu-guest-agent within the guest: - User issues "nova suspend" or "openstack server suspend". - If nova-compute on the compute node decides that the instance has qemu-guest-agent running (which is the case if it's qemu or kvm, and its image has hw_qemu_guest_agent=yes), it sends a guest-sync command over the guest agent VirtIO serial port. This command registers in the qemu-ga log file in the guest. - nova-compute on the compute node sends a libvirt managed-save command. - Nova reports the instance as suspended. - User issues "nova resume" or "openstack server resume". - nova-compute on the compute node sends a libvirt start command. - Again, if nova-compute on the compute node knows that the instance has qemu-guest-agent running, it sends another command over the serial port, namely guest-set-time. This, too, registers in the guest's qemu-ga log. - Nova reports the instance as active (running normally) again. Now, when I instead use a Nova environment that is fully RBD-backed, I see exactly the same behavior as described above. So I know that in principle, nova-compute/qemu-ga communication works in both an RBD-backed and a non-RBD-backed environment. However, things appear to get very different when it comes to snapshots. Again, starting with a file-backed environment: - User issues "nova image-create" or "openstack server image create". - If nova-compute on the compute node decides that the instance can be quiesced (which is the case if it's qemu or kvm, and its image has hw_qemu_guest_agent=yes), then it sends a "guest-fsfreeze-freeze" command over the guest agent VirtIO serial port. - The guest agent inside the guest loops over all mounted filesystems, and issues the FIFREEZE ioctl (which maps to the kernel freeze_super() function). This can be seen in the qemu-ga log file in the guest, and it is also verifiable by using ftrace on the qemu-ga PID and checking for the freeze_super() function call. - nova-compute then takes a live snapshot of the instance. - Once complete, the guest gets a "guest-fsfreeze-thaw" command, and again I can see this in the qemu-ga log, and with ftrace. And now with RBD: - User issues "nova image-create" or "openstack server image create". - The guest-fsfreeze-freeze agent command never happens. Now I can see the info message from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... in my nova-compute log, which confirms that we're attempting a live snapshot. I also do *not* see the warning from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9..., so it looks like the direct_snapshot() call from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... succeeds. This is defined in https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... and it uses RBD functionality only. Importantly, it never interacts with qemu-ga, so it appears to not worry at all about freezing the filesystem. (Which does seem to contradict https://docs.ceph.com/docs/master/rbd/rbd-openstack/?highlight=uuid#image-pr..., by the way, so that may be a documentation bug.) Now here's another interesting part. Were the direct snapshot to fail, if I read https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... and https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... correctly, the fallback behavior would be as follows: The domain would next be "suspended" (note, again this is Nova suspend, which maps to libvirt managed-save per https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9...), then snapshotted using a libvirt call and resumed again post-snapshot. In which case there would be a guest-sync call on suspend. And it's this part that has me a bit worried. If an RBD backed instance, on a successful snapshot, never freezes its filesystem *and* never does any kind of sync, either, doesn't that mean that such an instance can't be made to produce consistent snapshots? (Particularly in the case of write-back caching, which is recommended and normally safe for RBD/virtio devices.) Or is there some magic within the Qemu RBD storage driver that I am unaware of, that makes any such contortions unnecessary? Thanks in advance for your insights! Cheers, Florian
On Wed, Aug 21, 2019 at 9:34 AM Florian Haas <florian@citynetwork.eu> wrote:
Hi everyone,
apologies in advance; this will be long. It's also been through a bunch of edits and rewrites, so I don't know how well I'm expressing myself at this stage — please holler if anything is unclear and I'll be happy to try to clarify.
I am currently in the process of investigating the behavior of OpenStack Nova instances when being snapshotted and suspended, in conjunction with qemu-guest-agent (qemu-ga). I realize that RBD-backed Nova/libvirt instances are expected to behave differently from file-backed ones, but I think I might have reason to believe that the RBD-backed ones are indeed behaving incorrectly, and I'd like to verify that.
So first up, for comparison, let's recap how a Nova/libvirt/KVM instance behaves when it is *not* backed by RBD (such as, it's using a qcow2 file that is on a Nova compute node in /var/lib/nova/instances), is booted from an image with the hw_qemu_guest_agent=yes meta property set, and runs qemu-guest-agent within the guest:
- User issues "nova suspend" or "openstack server suspend".
- If nova-compute on the compute node decides that the instance has qemu-guest-agent running (which is the case if it's qemu or kvm, and its image has hw_qemu_guest_agent=yes), it sends a guest-sync command over the guest agent VirtIO serial port. This command registers in the qemu-ga log file in the guest.
- nova-compute on the compute node sends a libvirt managed-save command.
- Nova reports the instance as suspended.
- User issues "nova resume" or "openstack server resume".
- nova-compute on the compute node sends a libvirt start command.
- Again, if nova-compute on the compute node knows that the instance has qemu-guest-agent running, it sends another command over the serial port, namely guest-set-time. This, too, registers in the guest's qemu-ga log.
- Nova reports the instance as active (running normally) again.
Now, when I instead use a Nova environment that is fully RBD-backed, I see exactly the same behavior as described above. So I know that in principle, nova-compute/qemu-ga communication works in both an RBD-backed and a non-RBD-backed environment.
However, things appear to get very different when it comes to snapshots.
Again, starting with a file-backed environment:
- User issues "nova image-create" or "openstack server image create".
- If nova-compute on the compute node decides that the instance can be quiesced (which is the case if it's qemu or kvm, and its image has hw_qemu_guest_agent=yes), then it sends a "guest-fsfreeze-freeze" command over the guest agent VirtIO serial port.
- The guest agent inside the guest loops over all mounted filesystems, and issues the FIFREEZE ioctl (which maps to the kernel freeze_super() function). This can be seen in the qemu-ga log file in the guest, and it is also verifiable by using ftrace on the qemu-ga PID and checking for the freeze_super() function call.
- nova-compute then takes a live snapshot of the instance.
- Once complete, the guest gets a "guest-fsfreeze-thaw" command, and again I can see this in the qemu-ga log, and with ftrace.
And now with RBD:
- User issues "nova image-create" or "openstack server image create".
- The guest-fsfreeze-freeze agent command never happens.
Now I can see the info message from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... in my nova-compute log, which confirms that we're attempting a live snapshot.
I also do *not* see the warning from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9..., so it looks like the direct_snapshot() call from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... succeeds. This is defined in https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... and it uses RBD functionality only. Importantly, it never interacts with qemu-ga, so it appears to not worry at all about freezing the filesystem.
(Which does seem to contradict https://docs.ceph.com/docs/master/rbd/rbd-openstack/?highlight=uuid#image-pr..., by the way, so that may be a documentation bug.)
Now here's another interesting part. Were the direct snapshot to fail, if I read https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... and https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... correctly, the fallback behavior would be as follows: The domain would next be "suspended" (note, again this is Nova suspend, which maps to libvirt managed-save per https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9...), then snapshotted using a libvirt call and resumed again post-snapshot. In which case there would be a guest-sync call on suspend.
And it's this part that has me a bit worried. If an RBD backed instance, on a successful snapshot, never freezes its filesystem *and* never does any kind of sync, either, doesn't that mean that such an instance can't be made to produce consistent snapshots? (Particularly in the case of write-back caching, which is recommended and normally safe for RBD/virtio devices.) Or is there some magic within the Qemu RBD storage driver that I am unaware of, that makes any such contortions unnecessary?
It just looks like this was an oversight from the OpenStack developers when Nova RBD "direct" ephemeral image snapshot support was added [1]. I would open a bug ticket against Nova for the issue.
Thanks in advance for your insights!
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://opendev.org/openstack/nova/commit/824c3706a3ea691781f4fcc4453881517a... -- Jason
Hi Jason! Thanks for the quick reply. On 21/08/2019 16:51, Jason Dillaman wrote:>
It just looks like this was an oversight from the OpenStack developers when Nova RBD "direct" ephemeral image snapshot support was added [1]. I would open a bug ticket against Nova for the issue.> [1] https://opendev.org/openstack/nova/commit/824c3706a3ea691781f4fcc4453881517a...
OK, wow... that was 4 years ago, does that mean that quiesce/freeze/thaw for RBD-backed Nova instances has probably been non-functional throughout that time? Looking at the reno for that commit I had an idea for a workaround:
features: - When RBD is used for ephemeral disks and image storage, make snapshot use Ceph directly, and update Glance with the new location. In case of failure, it will gracefully fallback to the "generic" snapshot method. This requires changing the typical permissions for the Nova Ceph user (if using authx) to allow writing to the pool where vm images are stored, and it also requires configuring Glance to provide a v2 endpoint with direct_url support enabled (there are security implications to doing this). See http://docs.ceph.com/docs/master/rbd/rbd-openstack/ for more information on configuring OpenStack with RBD.
So, suppose that deployers running Nova with ephemeral disks on RBD prefer snapshot consistency over this shortcut. Until Nova fixes the direct_snapshot() call, I figured that such deployers could tweak the caps for the Nova CephX identity such that that user were no longer allowed to write to the Glance pool. Under those circumstances, the snapshot creation (in the ephemeral pool) would work, but then the clone() call in this line should throw nova.exception.Forbidden from an rbd.PermissionError: https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... Which should then trigger this except block: https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... ... and Nova/libvirt should go back to the (arguably more correct) fallback. Would you agree with that assessment, or am I missing something? (Just trying to make sure that I don't give the Nova folks the wrong facts.) Thanks again! Cheers, Florian
On Aug 21, 2019, at 11:41 AM, Florian Haas <florian@citynetwork.eu> wrote:
Hi Jason! Thanks for the quick reply.
On 21/08/2019 16:51, Jason Dillaman wrote:>
It just looks like this was an oversight from the OpenStack developers when Nova RBD "direct" ephemeral image snapshot support was added [1]. I would open a bug ticket against Nova for the issue.> [1] https://opendev.org/openstack/nova/commit/824c3706a3ea691781f4fcc4453881517a...
OK, wow... that was 4 years ago, does that mean that quiesce/freeze/thaw for RBD-backed Nova instances has probably been non-functional throughout that time?
Looking at the reno for that commit I had an idea for a workaround:
features: - When RBD is used for ephemeral disks and image storage, make snapshot use Ceph directly, and update Glance with the new location. In case of failure, it will gracefully fallback to the "generic" snapshot method. This requires changing the typical permissions for the Nova Ceph user (if using authx) to allow writing to the pool where vm images are stored, and it also requires configuring Glance to provide a v2 endpoint with direct_url support enabled (there are security implications to doing this). See http://docs.ceph.com/docs/master/rbd/rbd-openstack/ for more information on configuring OpenStack with RBD.
So, suppose that deployers running Nova with ephemeral disks on RBD prefer snapshot consistency over this shortcut. Until Nova fixes the direct_snapshot() call, I figured that such deployers could tweak the caps for the Nova CephX identity such that that user were no longer allowed to write to the Glance pool.
Yes, that would be my recommendation.
Under those circumstances, the snapshot creation (in the ephemeral pool) would work, but then the clone() call in this line should throw nova.exception.Forbidden from an rbd.PermissionError:
https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9...
Which should then trigger this except block:
https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9...
... and Nova/libvirt should go back to the (arguably more correct) fallback.
Would you agree with that assessment, or am I missing something? (Just trying to make sure that I don't give the Nova folks the wrong facts.)
Thanks again!
Cheers, Florian
On Wed, Aug 21, 2019 at 11:53 AM Jason Dillaman <jdillama@redhat.com> wrote:
On Aug 21, 2019, at 11:41 AM, Florian Haas <florian@citynetwork.eu> wrote:
Hi Jason! Thanks for the quick reply.
On 21/08/2019 16:51, Jason Dillaman wrote:>
It just looks like this was an oversight from the OpenStack developers when Nova RBD "direct" ephemeral image snapshot support was added [1]. I would open a bug ticket against Nova for the issue.> [1] https://opendev.org/openstack/nova/commit/824c3706a3ea691781f4fcc4453881517a...
OK, wow... that was 4 years ago, does that mean that quiesce/freeze/thaw for RBD-backed Nova instances has probably been non-functional throughout that time?
Just to clarify, in the initial implementation, only cold snapshots were supported for RBD [1] so there was no need to quiesce the disk. The issue was introduced by [2] when that restriction for RBD images was removed about a year later.
Looking at the reno for that commit I had an idea for a workaround:
features: - When RBD is used for ephemeral disks and image storage, make snapshot use Ceph directly, and update Glance with the new location. In case of failure, it will gracefully fallback to the "generic" snapshot method. This requires changing the typical permissions for the Nova Ceph user (if using authx) to allow writing to the pool where vm images are stored, and it also requires configuring Glance to provide a v2 endpoint with direct_url support enabled (there are security implications to doing this). See http://docs.ceph.com/docs/master/rbd/rbd-openstack/ for more information on configuring OpenStack with RBD.
So, suppose that deployers running Nova with ephemeral disks on RBD prefer snapshot consistency over this shortcut. Until Nova fixes the direct_snapshot() call, I figured that such deployers could tweak the caps for the Nova CephX identity such that that user were no longer allowed to write to the Glance pool.
Yes, that would be my recommendation.
Under those circumstances, the snapshot creation (in the ephemeral pool) would work, but then the clone() call in this line should throw nova.exception.Forbidden from an rbd.PermissionError:
https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9...
Which should then trigger this except block:
https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9...
... and Nova/libvirt should go back to the (arguably more correct) fallback.
Would you agree with that assessment, or am I missing something? (Just trying to make sure that I don't give the Nova folks the wrong facts.)
Thanks again!
Cheers, Florian
[1] https://opendev.org/openstack/nova/src/commit/824c3706a3ea691781f4fcc4453881... [2] https://opendev.org/openstack/nova/src/commit/231832354932e26f0d76af1cf1711e... -- Jason
Florian; Forgive my lack of knowledge of OpenStack, and your environment / use case. Why would you need / want to snapshot an ephemeral disk? Isn't the point of ephemeral storage to not be persistent? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Florian Haas [mailto:florian@citynetwork.eu] Sent: Wednesday, August 21, 2019 8:41 AM To: dillaman@redhat.com Cc: ceph-users Subject: [ceph-users] Re: RBD, OpenStack Nova, libvirt, qemu-guest-agent, and FIFREEZE: is this working as intended? Hi Jason! Thanks for the quick reply. On 21/08/2019 16:51, Jason Dillaman wrote:>
It just looks like this was an oversight from the OpenStack developers when Nova RBD "direct" ephemeral image snapshot support was added [1]. I would open a bug ticket against Nova for the issue.> [1] https://opendev.org/openstack/nova/commit/824c3706a3ea691781f4fcc4453881517a...
OK, wow... that was 4 years ago, does that mean that quiesce/freeze/thaw for RBD-backed Nova instances has probably been non-functional throughout that time? Looking at the reno for that commit I had an idea for a workaround:
features: - When RBD is used for ephemeral disks and image storage, make snapshot use Ceph directly, and update Glance with the new location. In case of failure, it will gracefully fallback to the "generic" snapshot method. This requires changing the typical permissions for the Nova Ceph user (if using authx) to allow writing to the pool where vm images are stored, and it also requires configuring Glance to provide a v2 endpoint with direct_url support enabled (there are security implications to doing this). See http://docs.ceph.com/docs/master/rbd/rbd-openstack/ for more information on configuring OpenStack with RBD.
So, suppose that deployers running Nova with ephemeral disks on RBD prefer snapshot consistency over this shortcut. Until Nova fixes the direct_snapshot() call, I figured that such deployers could tweak the caps for the Nova CephX identity such that that user were no longer allowed to write to the Glance pool. Under those circumstances, the snapshot creation (in the ephemeral pool) would work, but then the clone() call in this line should throw nova.exception.Forbidden from an rbd.PermissionError: https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... Which should then trigger this except block: https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac9... ... and Nova/libvirt should go back to the (arguably more correct) fallback. Would you agree with that assessment, or am I missing something? (Just trying to make sure that I don't give the Nova folks the wrong facts.) Thanks again! Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 21/08/2019 18:05, DHilsbos@performair.com wrote:
Florian;
Forgive my lack of knowledge of OpenStack, and your environment / use case.
Why would you need / want to snapshot an ephemeral disk? Isn't the point of ephemeral storage to not be persistent?
Fair point, but please consider that if you use an ephemeral VM as a template for other VMs (a common motivation for snapshotting), you might not care about the consistency of the VMs themselves, but you probably do care about the consistency of the template. But, for that use-case you could argue that you should just shut down the VM and take a clean snapshot then. However, in OpenStack Nova you may also use boot-from-volume, meaning you're running a VM that is expected to be *wholly* persistent, rather than ephemeral, and in that case the consistency of a snapshot taken while the instance is running is rather important. So just to be sure I took your cue and retested to see whether the same issue also applied to an instance using boot-from-volume. And lo and behold, the problem does not apply — if I configure an instance to boot from a volume, I get fsfreeze just as intended. (I have yet to dig up the code path for this.) So, evidently the situation can be summarized as: - Ephemeral boot, *without* RBD, with or without attached volumes: freeze/thaw if hw_qemu_guest_agent=yes, resulting in consistent snapshots. - Ephemeral boot *from* RBD, also with or without attached volumes: no freeze/thaw, resulting in potentially inconsistent snapshots even with hw_qemu_guest_agent=yes. - Boot-from-volume from RBD: freeze/thaw if hw_qemu_guest_agent=yes, resulting in consistent snapshots. Bit odd, that. :) But at least there's another available workaround: if you need to ensure snapshot consistency, use boot-from-volume. Thanks for the nudge in that direction, Dominic! Cheers, Florian
Just following up here to report back and close the loop: On 21/08/2019 16:51, Jason Dillaman wrote:
It just looks like this was an oversight from the OpenStack developers when Nova RBD "direct" ephemeral image snapshot support was added [1]. I would open a bug ticket against Nova for the issue.
Done: https://bugs.launchpad.net/nova/+bug/1841160 Thanks again for your help! Cheers, Florian
participants (3)
-
DHilsbos＠performair.com
-
Florian Haas
-
Jason Dillaman