Rotating cephx client keys in openstack VMs
Hello, I'm researching a way to rotate cephx client keys to aes256k 100% online for openstack VMs (qemu) with ceph rbd volumes attached (librbd). The problem ----------- Unfortunately I don't see any reasonable way to do it. Obviously I might be missing something (or a lot of something) Usually libvirt configures VM disk with following auth: <disk ...> <auth username='cephxuser'> <secret type='ceph' uuid='id-of-libvirt-secret'/> </auth> </disk> The cephx key is stored in a libvirt secret, it is read when VM starts and then stored in RAM for a VM lifetime (librbd AFAIK). If we rotate cephxuser key the living VMs will eventually stop working - when either auth_mon_ticket_ttl is exhausted or some connection to mon needs to be re-established librbd will try to authenticate with old key and fail. Potential solutions ------------------- A - rotate key, and distribute it as the same libvirt secret: - running VMs will eventually stop working - hard reboot or live migration shall fix them - stopped (SHUTOFF) VMs will be OK when started - new VMs and new attachments will be OK B - use new user and new key - new libvirt secret - running VMs will be OK, as long as we keep old user and key, live migration will probably fix the VM and use new user and new key - stopped (SHUTOFF) VMs will be OK when started, but will fail when old user/key is removed. username and libvirt secret uuid is kept in nova database and nothing will update it - new VMs and new attachments will be OK C - use the same user and have 2 keys for it - old and new. Eventually remove old one - running VMs will be OK, they will switch to new key during live migration - stopped (SHUTOFF) VMs will be OK, libvirt secret points to the new key so when VM is started it will use it. User is the same so no DB update is required - new VMs and new attachments will be OK A - not acceptable, breaks running VMs B - very difficult, requires a lot of VM migration, requires a lot of work to fix SHELVED/SHUTOFF VMs (no tooling for that AFAIK?) C - looks the best but not possible, ceph doesn't allow multiple keys per single user Summary ------- If my understanding is correct the effort to rotate the keys for openstack is tremendous and requires nova/cinder DB modification to fix SHUTOFF vms. Probably SHELVED vms will have the same problem. If you reached this point, thanks for reading :) Any ideas or comments? Bartosz Rabiega
Hi Bartosz, thanks a lot for your thoughts on how to rotate the rbd client key. Even though I'm using Ganeti and not openstack, some general problems remain the same. On Fri, 18 Sep 2026, Bartosz Rabiega wrote:
If we rotate cephxuser key the living VMs will eventually stop working - when either auth_mon_ticket_ttl is exhausted or some connection to mon needs to be re-established librbd will try to authenticate with old key and fail.
That is a fundamental observation. While I'm not able to approve this technically it sounds very reasonable to me. I got the impression that others seem to neglect this?
Potential solutions ------------------- ... B - use new user and new key - new libvirt secret - running VMs will be OK, as long as we keep old user and key, live migration will probably fix the VM and use new user and new key - stopped (SHUTOFF) VMs will be OK when started, but will fail when old user/key is removed. username and libvirt secret uuid is kept in nova database and nothing will update it - new VMs and new attachments will be OK
That option is my favourite with the libvirt-less Ganeti. It seems you are unsure if live migration will switch to the new user? For Ganeti I can tell, that it will not (old user kept in runtime config). Stopped VMs don't have anything persistent in it's config WRT ceph auth in Ganeti. They will pick up the new user/key on startup. So everything I need to do, is to stop and start all VMs :-). Luckily in my environment this seems possible over a long period (~1 year). Other than that, I don't see other options yet. Thanks, Sascha.
participants (2)
-
Bartosz Rabiega
-
Sascha Lucas