On Fri, Nov 29, 2024, 5:33 AM Adrien Georget <adrien.georget@cc.in2p3.fr> wrote:
Hello,
We are using Ceph as a storage backend for Openstack (Cinder, Nova, Glance, Manila) and we are replacing old hardware hosting Ceph monitors (MON,MGR,MDS) to new ones. I have already added the new ones in production, monitors successfully joined the quorum and new MGR/MDS are standby.
For the monitors, I'm sure that the monmap is already up to date and Openstack clients are already aware of the change and it should not be a problem when I will next shut down the old monitors. The ceph.conf will be updated in all Openstack controllers to replace "mon host" with the new ones before shutting old mons down.
But I have some doubts with the resilience of Openstack Manila service because the IP addresses of the monitors look hardcoded in the export location of the manila share : The manila show command returns for example :
| export_locations | | | | path = 134.158.208.140:6789,134.158.208.141:6789,134.158.208.142:6789:/volumes/EC_manila/_nogroup/7a6c05d9-2fea-43b1-a6d4-06eec1e384f2
| | | share_instance_id = 7a6c05d9-2fea-43b1-a6d4-06eec1e384f2 |
Has anyone already done this kind of migration in the past and can confirm my doubts? Is there any process to update shares?
Cheers, Adrien _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I can't speak for Manila, but for Cinder/Glance/Nova this is a bit of a headache. Unfortunately, the mon IPs get hard coded there as well, both in the database and in the libvirt XML. Go to any nova-compute node with a Ceph-backed Cinder volume (or Nova image, including config drives) attached to it and run `virsh dumpxml <UUID>` and you'll see it. Unfortunately, changing all of the mon IPs will result in the situation where you can neither live-migrate your VMs nor will you be able to start/hard reboot VMs until volumes are detached and attached with the new monitor IPs. The only way we found around this with zero downtime was to rebuild _some_ of the ceph-mons with new IPs, and then leverage some custom patches (which I can share) that rewrite the libvirt and database info during a live-migration (so, in essence, we had to live-migrate each VM once in order to pull this off) with the new set of intended mon IPs (not the ones currently in ceph.conf). If you don't require live-migration or don't use it, you can probably get away with just doing some database updates (carefully!). The VMs do observe monmap changes at runtime like any other RADOS client - it's only when you try to perform control plane actions against them that it becomes a problem, because it'll use the mon IPs in the database (which are old) and not from ceph.conf in that case. Thanks, Tyler