upgrading from Nautilus on CentOS7 to Octopus on Ubuntu 20.04.2
Hi, I’ve been running a staging Ceph environment on CentOS 7/Nautilus for quite a while now. Because of many good reasons that you can probably guess, I am currently trying to move this staging environment to Octopus on Ubuntu 20.04.2. Since I’m trying to keep the data, but don’t mind downtime at all, my plan was to reinstall one server at a time, removing them from the cluster and adding them back to the cluster, since Octopus and Nautilus should still be compatible with each other. I started with one monitor and now I’m stuck in a weird state. Essentially : -Monitors see each other and are establishing connections to each other. -Monitor clocks are synchronized -Monmap was injected in the reinstalled monitor -New monitor is recognized in Ceph -s but stuck out of the quorum Is there something that could prevent a new monitor from establishing quorum if the monmap is the same, the clock is synchronized and it can contact other monitors on the network? Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
On 6/28/21 10:46 PM, Jean-Philippe Méthot wrote:
Hi,
I’ve been running a staging Ceph environment on CentOS 7/Nautilus for quite a while now. Because of many good reasons that you can probably guess, I am currently trying to move this staging environment to Octopus on Ubuntu 20.04.2.
Since I’m trying to keep the data, but don’t mind downtime at all, my plan was to reinstall one server at a time, removing them from the cluster and adding them back to the cluster, since Octopus and Nautilus should still be compatible with each other. I started with one monitor and now I’m stuck in a weird state. Essentially :
-Monitors see each other and are establishing connections to each other. -Monitor clocks are synchronized -Monmap was injected in the reinstalled monitor -New monitor is recognized in Ceph -s but stuck out of the quorum
Is there something that could prevent a new monitor from establishing quorum if the monmap is the same, the clock is synchronized and it can contact other monitors on the network?
Have you followed the manual deployment method for bootstrapping monitors [1]? And the troubleshooting mon guide [2]? Have you removed the monitor from Ceph prior to the re-install? And removed its kerying? I have done several "rescue drills" to bootstrap monitors by hand. The order of all commands is very important, as well as making sure that permissions are set correctly (ceph.ceph). I have not tried to upgrade this way (I think I would have copied over the old mon store and have ceph upgrade it if necessary). You might still want to try that (stop a nautilus mon, copy over data, start octopus mon). Gr. Stefan [1]: https://docs.ceph.com/en/latest/install/manual-deployment/ [2]: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/
I had followed the steps in the documentation, but it’s likely that something went wrong during those steps. I copied over the mon store instead, as you recommended, and it fixed my problem. Thanks for your help, now I can finally progress with this. Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
Le 29 juin 2021 à 04:07, Stefan Kooman <stefan@bit.nl> a écrit :
On 6/28/21 10:46 PM, Jean-Philippe Méthot wrote:
Hi, I’ve been running a staging Ceph environment on CentOS 7/Nautilus for quite a while now. Because of many good reasons that you can probably guess, I am currently trying to move this staging environment to Octopus on Ubuntu 20.04.2. Since I’m trying to keep the data, but don’t mind downtime at all, my plan was to reinstall one server at a time, removing them from the cluster and adding them back to the cluster, since Octopus and Nautilus should still be compatible with each other. I started with one monitor and now I’m stuck in a weird state. Essentially : -Monitors see each other and are establishing connections to each other. -Monitor clocks are synchronized -Monmap was injected in the reinstalled monitor -New monitor is recognized in Ceph -s but stuck out of the quorum Is there something that could prevent a new monitor from establishing quorum if the monmap is the same, the clock is synchronized and it can contact other monitors on the network?
Have you followed the manual deployment method for bootstrapping monitors [1]? And the troubleshooting mon guide [2]?
Have you removed the monitor from Ceph prior to the re-install? And removed its kerying?
I have done several "rescue drills" to bootstrap monitors by hand. The order of all commands is very important, as well as making sure that permissions are set correctly (ceph.ceph). I have not tried to upgrade this way (I think I would have copied over the old mon store and have ceph upgrade it if necessary). You might still want to try that (stop a nautilus mon, copy over data, start octopus mon).
Gr. Stefan
[1]: https://docs.ceph.com/en/latest/install/manual-deployment/ [2]: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/
participants (2)
-
Jean-Philippe Méthot
-
Stefan Kooman