I have both a small test cluster and a larger production cluster. They are (were, for the test cluster) running Rocky Linux 8.9. They are both updated originally from Pacific, currently at reef 18.2.2.These are all rpm installs. It has come time to consider upgrades to Rocky 9.3. As there is no officially supported upgrade procedure from Rocky 8 to Rocky 9, I wanted to document my procedure for the test cluster(it worked!) before I move on to my production cluster, and verify that what I have done makes sense and ask a few questions. As such, this involves a reinstall of the OS. For my test cluster, my mon nodes also have all the other services: ceph-mon, ceph-mds, ceph-radosgw and ceph-mgr. ALL of my OSDs are LVMs, and they each have wal devices, also as LVMs. My aim was to preserve these across the reinstall of the OS, and I did. To be clear,all the OSDs are not on the same physical disk as the OS and the mon/mgr/mds/radosgw, and most are grouped on their own node. First, I set noout, norecover, nobackfill # ceph osd set noout# ceph osd set norecover# ceph osd set nobackfill On one of the mon nodes, I did the following, to save off the monmap, osdmap, and crushmap in case something went wildly wrong # systemctl stop ceph-mon.target# ceph-mon -i <short servername> --extract-monmap /tmp/monmap.bin# systemcl start ceph-mon.target# ceph osd getcrushmap -o /tmp/crush.bin# ceph osd getmap -o /tmp/osd.bin Then on EVERY node, both mons and osd nodes, I tarred up /etc/ceph and /var/lib/ceph # tar cvf /tmp/varlibceph.tar /etc/ceph /var/lib/ceph I then saved off each tar file on each node to a location not being reinstalled, as such I had a tarfile per osd node and also one for each mon node. I also saved off the monmap, crushmap and osdmaps created above tothe same location. I then went sequentially through the mon/mds/radosgw/mgr servers, one at a time. As I am using kickstart/cobbler, I told kickstartto ignore all disks EXCEPT the one that the original 8.9 OS was installed on. For this I had to use the following in the kickstart file. ignoredisk --only-use=disk/by-path/pci-id-of-root-disk I did this because, at least on my hardware, Rocky 9 reorders drive letters sda, sdb, sdc, etc based on recognition order on *every* boot, which mightoverwrite an OSD LVM if I wasn't careful enough. Note that while I group some commands that follow below, I only did the mon nodes sequentially, whille I grouped together osd nodes and did those groups allat the same time based on my failure domain for my crushmap. After the Rocky 9 reinstall and a reboot: I then installed the appropriate el9 18.2.2 ceph packages $ dnf install ceph-mon (mon nodes)# dnf install ceph-mds (mds nodes) # dnf install ceph-radosgw (radosgw nodes)# dnf install ceph-mgr (mgr nodes)# dnf install ceph-osd (osd nodes) For the various nodes, once the reinstall was done to Rocky 9, I re-enabled the firewall # firewall-cmd --add-service ceph-mon --permanent ( for mon nodes)# firewall-cmd --add-service ceph --permanent (for osd nodes)# firewall-cmd --add-service https --permanent (for radosgw servers I am running them on port 443, with certs)# systemctl restart firewalld and then of course restarted firewalld I then restored /etc/ceph and /var/lib/ceph for each node from the individual tar file backups per node. For the non-OSD nodes, I re-enabled services: # systemctl enable ceph-mon@short-name-of-server.service (mon nodes)# systemctl enable ceph-mds@short-name-of-server.service (mds nodes)# systemctl enable ceph-radosgw@rgw.short-name-of-server.service (radosgw servers)# systemctl enable ceph-mgr@short-name-of-server.service (mgr nodes) For the OSD nodes, "ceph-volume lvm list" shows this for each OSD, even after reinstallingto Rocky 9 (which is good evidence that I installed on the proper disk and did not overwritean OSD: # ceph-volume lvm list block device /dev/ceph-c5b97619-4184-4637-9b82-2575004dba41/osd-block-abb210f3-52cf-4d4b-8745-126dc57287da block uuid lmxbj0-zgcB-5MwI-wpsf-9yut-3VlG-p9DoFL cephx lockbox secret cluster fsid 0f3b6c81-3814-4dc5-a19e-f7307543e56c cluster name ceph crush device class encrypted 0 osd fsid abb210f3-52cf-4d4b-8745-126dc57287da osd id 7 osdspec affinity type block vdo 0 devices /dev/sdj I had to enable units of the form: # systemctl enable ceph-lvm-$osd-id-$osd-fsid.service For the above osd lvm, this would be: # systemctl enable ceph-volume@lvm-7-abb210f3-52cf-4d4b-8745-126dc57287da.service So If I had 10 OSD LVMs on a given node, I have to enable 10 services of the form above. After reboot of either a mon or osd node (depending on which server I am doing), all comes up just fine. My questions: 1. Did I do anything wrong?2. Did I miss something?3. is there an official way to do this that is different from my way?4. Will anything I did (or did not) do bite me when I later do an update say to 18.2.6 or 19? 4. Is there an 'official' way to back up things like /etc/ceph and /var/lib/ceph that would be '*better**? Of course, after I update all the OSDs, I take off noout, norecover, nobackfill # ceph osd unset nobackfill# ceph osd unset norecover# ceph osd unset noout Any thoughts would be appreciated. Again I am updating from 18.2.2 on Rocky 8 to the same release (but different rpms), 18.2.2 on Rocky 9. -Chris