[ceph-users] Re: Rolling upgrade model to new OS

Hi Drew, I performed the upgrade from Nautilus (bare-metal deployment) -> Octopus (podman containerization) and RHEL-7 -> RHEL-8. Everything was done in-place. My sequence was: ceph osd noout/norebalance shutdown/disable running services perform full OS upgrade install necessary software like podman start ceph monitor in podman ensure it is joining the rest of cluster ensure the monitor is stable deploy osds with the same ids I had previously ensure everything is stable deploy additional daemons (if needed) check cluster's overall health state if everything is fine, proceed to the next node There wasn't any major issue related to the upgrade, however I hit some minors. E.g. Sometimes for some reason monitors didn't join the cluster. or osds started flapping, but all problems I saw - were fixable in the observable period of time. My personal opinion - if you can use docker, then use docker and test the ceph images before you deploy it into production. I used to use our staging environment. Very likely, you should make several changes in systemd unit files and/or in the docker/podman command. Don't forget to mount all directories ceph is using, like /var/lib/ceph, etc. To troubleshoot the first start, I started the container without -d, to monitor the container's output in real time, then when I had all mistakes fixed, I restarted it using the systemd unit. Unit files you can grab on ceph-ansible github. Good luck! Vladimir On Fri, Jun 4, 2021 at 8:55 AM Drew Weaver <drew.weaver(a)thenap.com> wrote: