On Sat, Oct 14, 2023 at 5:14 PM Dave Hall <kdhall@binghamton.edu> wrote:
Hello.
It's been a while. For the past couple years I've had a cluster running Nautilus on Debian 10 using the Debian Ceph packages, and deployed with Ceph-Ansible. It's not a huge cluster - 10 OSD nodes with 80 x 12TB HDD OSDs, plus 3 management nodes, and about 40% full at the moment - but it is a critical resource for one of our researchers.
Back then I had some misgivings about non-Debian packages and also about containerized Ceph. I don't know if my feelings about these things have changed that much, but it's time to upgrade, and, with the advent of cephadm it looks like it's just better to stay mainstream.
I know it's probably the last thing you want to hear while looking down the barrel of a gun for a research cluster running an outdated version of Ceph -- but Debian 10 is *old*. You may be at the end of the road on Nautilus until you upgrade. You can see this in that e.g., download.ceph.com provides buster builds for Nautilus, but not Pacific: https://download.ceph.com/debian-nautilus/dists/ https://download.ceph.com/debian-pacific/dists/ Containerization would let you stay on Debian 10 by running Debian 11/12 containers on top of it. That being said, I'm sure you know that security updates ended 06/22 for Debian 10... not to mention the Debian 10 kernel is getting long in the tooth, and containizeration won't help you there. If you do proceed on with containerization, I would recommend containerizing your existing Nautilus cluster before trying to upgrade. Trying to do both container-ize an existing cluster while upgrading it is really just asking for trouble. Ceph upgrades without anything else moving around can be hard enough as it is!
So I'm looking for advice on how to get from where I'm at to at least Pacific or Quincy.
Be wary that you should not upgrade across more than 2 releases at a time per official documentation. If you're on Nautilus, you're looking at Pacific tops unless you want to gamble a bit: https://docs.ceph.com/en/latest/releases/quincy/#upgrading-from-pre-octopus-...
I've read a little in the last couple days. I've seen various opinions on (not) skipping releases and on when to switch to cephadm. I'm also concerned about cleaning up those old Debian packages - will there be a point where I can 'apt-get purge' them without harming the cluster.
One particular thing: The upgrade instructions in various places on docs.ceph.com say something like
Upgrade monitors by installing the new packages and restarting the monitor daemons.
So the diddly here is that you have to upgrade Ceph components in a precise order - all mons first, then all mgrs, etc. I imagine the disclaimer you're reading is oriented towards the fact that lots of folks run both mons and mgrs on the same host. If you apt-get dist-upgrade on one "mon/mgr" host, you would effectively have a mgr of release N+1 running prior to all mons being upgraded to N+1. The *best-case* scenario for this is that the mgr will not start anymore until the rest of the mons are upgraded.
To me this is kind of vague. Perhaps there is a different concept fo 'packages' within the cephadm environment. I could really use some clarification on this.
I'd also consider decommissioning a few nodes, setting up a new cluster on fresh Debian installs. and migrating the data and remaining nodes. This would be a long and painful process - decommission a node, move it, move some data, decommission another node - and I don't know what effect it would have on external references to our object store.
I think you may be overlooking something here -- there's no need to move data to do that. You just set noout, rebuild a host, and then run `ceph-volume lvm activate --all` after you're back up on Debian 11. That command will scan for LVMs for OSDs and just prop everything back up for you -- systemd units and all. There will be minimal recovery activity to restore degraded objects that had writes to them while the host was rebuilding, but it should be fairly minimal.
Please advise.
I am more than willing to extend my expertise to my alma mater here -- let me know if there's any way I can help. I have a heap of experience in upgrading Ubuntu/Ceph clusters with no downtime.
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io