Hello, I'm researching a way to rotate cephx client keys to aes256k 100% online for openstack VMs (qemu) with ceph rbd volumes attached (librbd). The problem ----------- Unfortunately I don't see any reasonable way to do it. Obviously I might be missing something (or a lot of something) Usually libvirt configures VM disk with following auth: <disk ...> <auth username='cephxuser'> <secret type='ceph' uuid='id-of-libvirt-secret'/> </auth> </disk> The cephx key is stored in a libvirt secret, it is read when VM starts and then stored in RAM for a VM lifetime (librbd AFAIK). If we rotate cephxuser key the living VMs will eventually stop working - when either auth_mon_ticket_ttl is exhausted or some connection to mon needs to be re-established librbd will try to authenticate with old key and fail. Potential solutions ------------------- A - rotate key, and distribute it as the same libvirt secret: - running VMs will eventually stop working - hard reboot or live migration shall fix them - stopped (SHUTOFF) VMs will be OK when started - new VMs and new attachments will be OK B - use new user and new key - new libvirt secret - running VMs will be OK, as long as we keep old user and key, live migration will probably fix the VM and use new user and new key - stopped (SHUTOFF) VMs will be OK when started, but will fail when old user/key is removed. username and libvirt secret uuid is kept in nova database and nothing will update it - new VMs and new attachments will be OK C - use the same user and have 2 keys for it - old and new. Eventually remove old one - running VMs will be OK, they will switch to new key during live migration - stopped (SHUTOFF) VMs will be OK, libvirt secret points to the new key so when VM is started it will use it. User is the same so no DB update is required - new VMs and new attachments will be OK A - not acceptable, breaks running VMs B - very difficult, requires a lot of VM migration, requires a lot of work to fix SHELVED/SHUTOFF VMs (no tooling for that AFAIK?) C - looks the best but not possible, ceph doesn't allow multiple keys per single user Summary ------- If my understanding is correct the effort to rotate the keys for openstack is tremendous and requires nova/cinder DB modification to fix SHUTOFF vms. Probably SHELVED vms will have the same problem. If you reached this point, thanks for reading :) Any ideas or comments? Bartosz Rabiega