Experimental upgrade of a Cephadm-managed Squid cluster to Ubuntu Noble (walk-through and RFC)
Hi everyone, as part of keeping our Ceph course updated, we recently went through the *experimental* process of upgrading a Cephadm-managed cluster from Ubuntu Jammy to Noble. Note that at this point there are no community-built Ceph packages that are available for Noble, though there *are* packages available directly from the Ubuntu mirrors. We figured that this shouldn't make too big of a difference, considering the Ceph deployment is all containerised anyway. We did eventually manage to successfully upgrade the cluster, with minimal disruption. That said, DO NOT consider the rest of this message to be an instruction for how to do this in production. Rather, it's meant to highlight some quirks that — I think — still need ironing out. Again: for the time being, DO NOT RUN THIS IN PRODUCTION, please and thank you. :) What's included here is a walkthrough of the process, and I would ask for feedback on this walkthrough. Specifically, if someone notices an error or omission in this process, please point them out. Each one of these steps needs to be repeated for every node in the cluster. It assumes a working Cephadm-managed cluster on Ubuntu Jammy with Podman, running Ceph Squid. All commands should be run as root. == 0. If the host you're about to upgrade is a MON, ensure that your orchestrator placement policy allows for the mon service to move to a different host. (This is to ensure that throughout your OS upgrade, you always have 3 MONs available.) 1. Disable orchestrator scheduling for the affected node: "ceph orch host label add <host> _no_schedule". 2. Wait for services (except OSDs) to be redeployed. 3. Check with "ceph orch host ok-to-stop <host>" to see whether the host reports "presumed safe to stop". (May require failing a mgr instance.) 4. Set the noout flag for that host: "ceph osd set-group noout <host>". 5. Check with "ceph health detail" to see if it says "host <host> has flags noout". 6. Shell into the host. 7. Make sure the host is fully updated on Jammy: "apt update && apt upgrade -y". Reboot the host if necessary. 8. Stop all remaining Ceph services (at this point, this should only be OSDs) with "systemctl stop ceph.target". 9. If your host has the cephadm package installed from a Ceph community repo, manually create the "/var/lib/cephadm" directory. This is required by the Ubuntu cephadm package, which during the OS upgrade will be cross-upgraded from a different package source. Failure to create this directory will cause this package upgrade to fail. I presume that this issue will go away whenever the Ceph community repos start including packages for Noble. 10. Run "do-release-upgrade" confirming all steps *except* the automated reboot at the end (opt out of that). 11. Remove the MongoDB_Compass profile from the /etc/apparmor.d directory. (If left in place, it will cause CEPHADM_REFRESH_FAILED errors.) 12. Remove any legacy ceph-osd packages, if installed on the host: "apt remove ceph-osd". (Failure to do so will keep your OSD containers from starting.) 13. Reboot the host. 14. Re-enable orchestrator scheduling with "ceph orch host label rm <host> _no_schedule". 15. Unset the noout flag for the host: "ceph osd unset-group noout <host>". 16. Check overall cluster health with "ceph -s" and "ceph health detail" before proceeding to the next node. == Once carried out for each Cephadm-managed Ubuntu node, you should have a reasonable expection of having a working Ceph/Noble test cluster. Again: do not run this in production, however, feedback on the above steps is much appreciated. Please let us know your thoughts. Cheers, Florian
Hi Florian, On 17.12.24 20:10, Florian Haas wrote:
1. Disable orchestrator scheduling for the affected node: "ceph orch host label add <host> _no_schedule".
14. Re-enable orchestrator scheduling with "ceph orch host label rm <host> _no_schedule".
Wouldn't it be easier to run "ceph orch host maintenance enter HOST" before and "ceph orch host maintenance exit HOST" after the upgrade? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
On 18/12/2024 15:37, Robert Sander wrote:
Hi Florian,
On 17.12.24 20:10, Florian Haas wrote:
1. Disable orchestrator scheduling for the affected node: "ceph orch host label add <host> _no_schedule".
14. Re-enable orchestrator scheduling with "ceph orch host label rm <host> _no_schedule".
Wouldn't it be easier to run "ceph orch host maintenance enter HOST" before and "ceph orch host maintenance exit HOST" after the upgrade?
I'm not a big fan of maintenance mode, to be honest. To illustrate why, assume you've got 3 Mons in your cluster. Now, on one of your physical hosts that runs a Mon, you enter maintenance mode. This will just shut down the Mon. Now you proceed with the system upgrade, which will vary in length. During that time you're running on two Mons. Now, something unexpected happens on another node that runs another Mon. Boom, your cluster is now offline, and you need to scramble to fix things. If conversely you set _no_schedule and you still have other hosts to migrate your Mon to (per your placement policy), then you'll run on 3 Mons throughout. And I prefer to have a policy that works on all nodes in a cluster. That's why I'd rather not have separate procedures for Mons and non-Mons. Hence the steps as described, which should work on all node types, Mon or not. Also, while maintenance does stop and disable the systemd ceph.target, meaning the services won't come up even if the host is rebooted, "systemctl status ceph.target" will still return "active" and "enabled" which may break assumptions by monitoring systems, orchestration frameworks, etc. That's why I currently prefer working with the _no_schedule label, over working with maintenance mode. Am I making sense? Cheers, Florian
Hi Florian, Am 12/18/24 um 16:18 schrieb Florian Haas:
To illustrate why, assume you've got 3 Mons in your cluster.
Now, on one of your physical hosts that runs a Mon, you enter maintenance mode. This will just shut down the Mon. Now you proceed with the system upgrade, which will vary in length. During that time you're running on two Mons.
Now, something unexpected happens on another node that runs another Mon. Boom, your cluster is now offline, and you need to scramble to fix things.
Yes, this is a risk. But shouldn't you run on 5 MONs today? At least this seems to be the recommended number from the detail service spec.
If conversely you set _no_schedule and you still have other hosts to migrate your Mon to (per your placement policy), then you'll run on 3 Mons throughout.
Then maybe the maintenance mode should also set this label.
Also, while maintenance does stop and disable the systemd ceph.target, meaning the services won't come up even if the host is rebooted, "systemctl status ceph.target" will still return "active" and "enabled" which may break assumptions by monitoring systems, orchestration frameworks, etc.
Your step 8 also just stops the ceph.target. Where is the difference? Kindest Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
On 20/12/2024 09:16, Robert Sander wrote:
Hi Florian,
Am 12/18/24 um 16:18 schrieb Florian Haas:
To illustrate why, assume you've got 3 Mons in your cluster.
Now, on one of your physical hosts that runs a Mon, you enter maintenance mode. This will just shut down the Mon. Now you proceed with the system upgrade, which will vary in length. During that time you're running on two Mons.
Now, something unexpected happens on another node that runs another Mon. Boom, your cluster is now offline, and you need to scramble to fix things.
Yes, this is a risk. But shouldn't you run on 5 MONs today? At least this seems to be the recommended number from the detail service spec.
The consensus heard at Cephalocon was still "for the vast majority of clusters you'll be fine with 3 MONs," so it is safe to assume that that is what most Ceph clusters out there are configured with today.
If conversely you set _no_schedule and you still have other hosts to migrate your Mon to (per your placement policy), then you'll run on 3 Mons throughout.
Then maybe the maintenance mode should also set this label.
I think the semantics of maintenance mode are explicitly designed to be "shut down services on one node and *don't* move them around", which totally has merit in certain scenarios. Rolling OS upgrade on a cluster just isn't one of them, in my humble opinion.
Also, while maintenance does stop and disable the systemd ceph.target, meaning the services won't come up even if the host is rebooted, "systemctl status ceph.target" will still return "active" and "enabled" which may break assumptions by monitoring systems, orchestration frameworks, etc.
Your step 8 also just stops the ceph.target. Where is the difference?
If I run "systemctl stop ceph.target" on a node in the cluster where I've been testing this, "podman ps" is empty and "systemctl status ceph.target" returns "inactive (dead)". Likewise, "systemctl is-active ceph.target" returns "inactive" and exits 3. If conversely I put a node in maintenance mode, "systemctl is-active ceph.target" returns "active" and exits 0, while "podman ps" likewise shows no running containers. I do not know *why* that is the case and I'm afraid I don't an opportunity to dig into that now, but that's the current behaviour as I see it. Cheers, Florian
Just to comment on the ceph.target. Technically in a containerized ceph a node can host daemons from *many ceph clusters* (each with its own ceph_fsid). The ceph.target is a global unit and it's the root for all the clusters running in the node. There's another target which is specific to each cluster (ceph-<fsid>.target). From my testing env where I created two clusters and I forced maintenance mode for the first one only: [root@ceph-node-2 ~]# systemctl list-dependencies ceph.target ceph.target ○ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8.target ○ │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@ceph-exporter.ceph-node-2.service ○ │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@crash.ceph-node-2.service × │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@mgr.ceph-node-2.ptlcoi.service ○ │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@mon.ceph-node-2.service × │ └─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@node-exporter.ceph-node-2.service ● └─ceph-a3cf42a0-becc-11ef-9470-52540012a496.target ● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@ceph-exporter.ceph-node-2.service ● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@crash.ceph-node-2.service ● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@mgr.ceph-node-2.bodyuz.service ● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@mon.ceph-node-2.service ● └─ceph-a3cf42a0-becc-11ef-9470-52540012a496@node-exporter.ceph-node-2.service *Global target:* [root@ceph-node-2 ~]# systemctl is-active ceph.target active *First cluster:*
systemctl is-active ceph-789c5638-bec0-11ef-9350-5254002ff0d8.target inactive
*Second cluster:*
systemctl is-active ceph-a3cf42a0-becc-11ef-9470-52540012a496.target active
Best, Redouane. On Fri, Dec 20, 2024 at 11:14 AM Florian Haas <florian.haas@cleura.com> wrote:
Hi Florian,
Am 12/18/24 um 16:18 schrieb Florian Haas:
To illustrate why, assume you've got 3 Mons in your cluster.
Now, on one of your physical hosts that runs a Mon, you enter maintenance mode. This will just shut down the Mon. Now you proceed with the system upgrade, which will vary in length. During that time you're running on two Mons.
Now, something unexpected happens on another node that runs another Mon. Boom, your cluster is now offline, and you need to scramble to fix
On 20/12/2024 09:16, Robert Sander wrote: things.
Yes, this is a risk. But shouldn't you run on 5 MONs today? At least this seems to be the recommended number from the detail service spec.
The consensus heard at Cephalocon was still "for the vast majority of clusters you'll be fine with 3 MONs," so it is safe to assume that that is what most Ceph clusters out there are configured with today.
If conversely you set _no_schedule and you still have other hosts to migrate your Mon to (per your placement policy), then you'll run on 3 Mons throughout.
Then maybe the maintenance mode should also set this label.
I think the semantics of maintenance mode are explicitly designed to be "shut down services on one node and *don't* move them around", which totally has merit in certain scenarios. Rolling OS upgrade on a cluster just isn't one of them, in my humble opinion.
Also, while maintenance does stop and disable the systemd ceph.target, meaning the services won't come up even if the host is rebooted, "systemctl status ceph.target" will still return "active" and "enabled" which may break assumptions by monitoring systems, orchestration frameworks, etc.
Your step 8 also just stops the ceph.target. Where is the difference?
If I run "systemctl stop ceph.target" on a node in the cluster where I've been testing this, "podman ps" is empty and "systemctl status ceph.target" returns "inactive (dead)". Likewise, "systemctl is-active ceph.target" returns "inactive" and exits 3.
If conversely I put a node in maintenance mode, "systemctl is-active ceph.target" returns "active" and exits 0, while "podman ps" likewise shows no running containers.
I do not know *why* that is the case and I'm afraid I don't an opportunity to dig into that now, but that's the current behaviour as I see it.
Cheers, Florian
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 02/01/2025 16:37, Redouane Kachach wrote:
Just to comment on the ceph.target. Technically in a containerized ceph a node can host daemons from *many ceph clusters* (each with its own ceph_fsid).
The ceph.target is a global unit and it's the root for all the clusters running in the node. There's another target which is specific to each cluster (ceph-<fsid>.target). From my testing env where I created two clusters and I forced maintenance mode for the first one only:
[root@ceph-node-2 ~]# systemctl list-dependencies ceph.target ceph.target ○ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8.target ○ │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@ceph-exporter.ceph-node-2.service ○ │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@crash.ceph-node-2.service × │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@mgr.ceph-node-2.ptlcoi.service ○ │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@mon.ceph-node-2.service × │ └─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@node-exporter.ceph-node-2.service ● └─ceph-a3cf42a0-becc-11ef-9470-52540012a496.target ● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@ceph-exporter.ceph-node-2.service ● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@crash.ceph-node-2.service ● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@mgr.ceph-node-2.bodyuz.service ● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@mon.ceph-node-2.service ● └─ceph-a3cf42a0-becc-11ef-9470-52540012a496@node-exporter.ceph-node-2.service
*Global target:* [root@ceph-node-2 ~]# systemctl is-active ceph.target active
*First cluster:*
systemctl is-active ceph-789c5638-bec0-11ef-9350-5254002ff0d8.target inactive
*Second cluster:*
systemctl is-active ceph-a3cf42a0-becc-11ef-9470-52540012a496.target active
Right, so in my view that's one more reason *not* to use maintenance mode in a distro upgrade, since stopping ceph.target ensures that all Ceph-related services are stopped on a node, even in the — somewhat uncommon — case of that node running services related to multiple clusters. Wouldn't you agree? Cheers, Florian
Probably this current behaviour (of not disabling the whole ceph-target) when entering the maintenance node is not correct as the whole node is affected (and any cluster(s) running on the same). I'll raise this in the next cephadm weekly and see what the team thinks. On Thu, Jan 2, 2025 at 5:22 PM Florian Haas <florian.haas@cleura.com> wrote:
On 02/01/2025 16:37, Redouane Kachach wrote:
Just to comment on the ceph.target. Technically in a containerized ceph a node can host daemons from *many ceph clusters* (each with its own ceph_fsid).
The ceph.target is a global unit and it's the root for all the clusters running in the node. There's another target which is specific to each cluster (ceph-<fsid>.target). From my testing env where I created two clusters and I forced maintenance mode for the first one only:
[root@ceph-node-2 ~]# systemctl list-dependencies ceph.target ceph.target ○ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8.target ○ │
├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@ceph-exporter.ceph-node-2.service
○ │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@crash.ceph-node-2.service × │
├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@mgr.ceph-node-2.ptlcoi.service
○ │ ├─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@mon.ceph-node-2.service × │
└─ceph-789c5638-bec0-11ef-9350-5254002ff0d8@node-exporter.ceph-node-2.service
● └─ceph-a3cf42a0-becc-11ef-9470-52540012a496.target ●
├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@ceph-exporter.ceph-node-2.service
● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@crash.ceph-node-2.service ●
├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@mgr.ceph-node-2.bodyuz.service
● ├─ceph-a3cf42a0-becc-11ef-9470-52540012a496@mon.ceph-node-2.service ●
└─ceph-a3cf42a0-becc-11ef-9470-52540012a496@node-exporter.ceph-node-2.service
*Global target:* [root@ceph-node-2 ~]# systemctl is-active ceph.target active
*First cluster:*
systemctl is-active ceph-789c5638-bec0-11ef-9350-5254002ff0d8.target inactive
*Second cluster:*
systemctl is-active ceph-a3cf42a0-becc-11ef-9470-52540012a496.target active
Right, so in my view that's one more reason *not* to use maintenance mode in a distro upgrade, since stopping ceph.target ensures that all Ceph-related services are stopped on a node, even in the — somewhat uncommon — case of that node running services related to multiple clusters. Wouldn't you agree?
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Florian Haas
-
Redouane Kachach
-
Robert Sander