On 13-11-2023 22:35, jarulsam@uwyo.edu wrote:
Hi everyone,
I have a storage cluster running RHCS v4 (old, I know) and am looking to upgrade it soon. I would also like to migrate from RHCS to the open source version of Ceph at some point, as our support contract with RedHat for Ceph is likely going to not be renewed going forward.
I was wondering if anyone has any advice as to how to upgrade our cluster with minimal production impact. I have the following server configuration:
We just finished doing pretty much this, albeit starting on RHEL 8. I don't think we had downtime at any point.
+ 3x monitors + 3x metadata servers + 2x RadosGWs with 2x servers running HAproxy and keepalived for HA RadosGWs. + 19x OSDs - 110TB HDD and 1TB NVMe each. (Total ~2.1PB raw)
Currently, I have RHCS v4 installed baremetal on RHEL 7. I see that newer versions of Ceph require containerized deployments so I am thinking it is best to first migrate to a containerized installation then try and upgrade everything else.
My first inclination is to do the upgrade like this:
1. Move existing installation to containerized, maintain all the same versions and OS installations.
2. Pull one monitor, fresh reinstall RHEL 9, reinstall RHCS v4, readd to cluster. Repeat for all the monitors.
3. Pull one MDS, do the same as step 2 but for MDS.
4. Pull one RadosGW, do the same as step 2 but for RadosGW.
5. Pull one OSD, rebalance, fresh reinstall RHEL 9, reinstall RHCS v4, readd to cluster, rebalance. Repeat for all OSDs.
6. Upgrade RHCS to OSS Ceph Octopus -> Pacific -> Quincy -> Reef.
Does such a plan seem reasonable? Are there any major pitfalls of an approach like this? Ideally, I would just rebuild an entire new cluster on Ceph Reef, however there are obvious budgetary issues with such a plan.
My biggest concerns are with moving to a containerized installation, then migrating from RHCS to OSS Ceph.
We engaged Red Hat support for the upgrade path and this is the top level steps we used: " 1. Upgrade OS to 8.4 # Min os version required is 8.2EUS/8.4 [A] (Step1/step2 position can be interchanged) 2. Switch from BM to container with switch-from-non-containerized-to-containerized-ceph-daemons.yml playbook. 3. Upgrade RHCS 4 to RHCS 5 with rolling upgrade playbook using ceph-ansible. 4. Once upgraded, handover the ceph-ansible control to cephadm (new / moving forward tool for RHCS 5) with help of cephadm-adopt playbook. " Then: Reinstalled all hosts with RHEL 9 one at a time, re-balancing in between and moving non-OSD services around with service specs (we got rid of our dedicated MONs/MDSs servers along the way). Finally going upstream from RHCS 5 (17.2.6): " $ ceph config dump | grep registry global basic container_image registry.redhat.io/rhceph/rhceph-5-rhel8@sha256:a193b0de114d19d2efd8750046b5d25da07e2c570e3c4eb4bd93e6de4b90a25a * mgr advanced mgr/cephadm/container_image_base registry.redhat.io/rhceph/rhceph-5-rhel8:latest $ ceph config rm global container_image $ ceph config rm mgr mgr/cephadm/container_image_base $ ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 " Mvh. Torkil
Any advice or feedback is much appreciated.
Best,
Josh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark