Hi Nico, Nico Schottelius wrote:

Robert Sander <ceph-users@ceph.io> writes:

...

Hi,

Am 21.08.26 um 4:10 PM schrieb Nico Schottelius:

...

a) All VPS/VMs need to be restarted To be able to rotate keys, aside from the almost impossible flow to actually doing the rotation (more to that below), this will affect every consumer of VMs. The usual setup is that all hypervisors are using the same key.

VM live migration should be able to do the trick here. When the new compute node already has the new key and the librbd version to support it.

That's a very good point. However this might also need work (or hacks) in the hypervisor sphere, to support using different keys in the 2 qemu processes. By default in opennebula, the ceph key is saved in a libvirtd secret. So in the best case, the new process will read it from the local secret store.

In the not so great case, opennebula (and other hypervisors) might infer the key from the previously deployed VM, instead of reading it from the local secret store. (*)

(*) I'm not sure about the state of other hypervisors, but in the opennebula case keys are actually embededd in the XML VM definition (sic!) that is saved in the database (sic!!) and copied on creation of the VM (!!!) and will never every be changed.

I think the problem is not the secret/key itself, Ceph username is the problem, as it seems to be global in OpenNebula. The problem you assume does not exist in OpenNebula - it correctly refers to the libvirt secret UUID, not the Ceph user key itself. You can put another key under the same libvirt secret UUID if you want: # virsh dumpxml one-... [...] <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <auth username='libvirt'> <secret type='ceph' uuid='9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf'/> </auth> <source protocol='rbd' name='one/one-1234-123456-0' index='2'> <host name='mon1' port='6789'/> <host name='mon2' port='6789'/> <host name='mon3' port='6789'/> </source> Create a new Ceph user with # ceph auth get-or-create client.libvirt2026 \ mon 'profile rbd' \ osd 'profile rbd pool=one' \ mgr 'profile rbd pool=one' \ -o libvirt2026.keyring Import the key under the same UUID on an idle/disabled node: # cat client.libvirt2026.secret.xml <secret ephemeral='no' private='no'> <uuid>9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf</uuid> <usage type='ceph'> <name>client.libvirt2026 secret</name> </usage> </secret> # virsh secret-undefine 9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf # virsh secret-define client.libvirt2026.secret.xml # KEY=`ceph auth get-key client.libvirt2026` # virsh secret-set-value --secret 9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf --base64 $KEY But it does not work, as I cannot persuade OpenNebula to use a different Ceph user instead of "libvirt" on that node (as opposed to the whole cluster, which would probably break running VMs). In the running VM template there are the following embedded values for a Ceph disk: "DISK": { [...] "CEPH_HOST": "mon1 mon2 mon3", "CEPH_SECRET": "9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf", "CEPH_TRASH": "yes", "CEPH_USER": "libvirt", [...] FWIW, I also created this post on OpenNebula forum: https://forum.opennebula.io/t/ceph-key-rollover/14547 -Yenya

A likely working hack is to modify the key inside the XML inside the database and then trigger live migration with a small time window of uncertainty that if the VM crashes on the original host before the live migration is triggered, it will likely not restart.

-- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl