Hi All after a long while being in health_err state (due to an unfound object, which we eventually decided to "forget"), we are now planning to upgrade our cluster which is running Pacific (at least on the mons/mdss/osds, the gateways are by accident running quincy already). The installation is via packages from ceph.com, unless it's quincy from ubuntu. ceph versions: "mon": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "mgr": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "osd": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 252, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12 }, "mds": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 2 }, "rgw": {"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 }, "overall": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 260, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 } The OS on the mons and mdss are still ubuntu 18.04, the osds are a mix of ubuntu 18 and ubuntu 20. The gateways are ubuntu 22.04, which is why these are already on quincy. The plan is to move to quincy and eventually cephadm/containered ceph, since that is apparently "the way to go", though I have my doubts. The steps we think are the right order are: - reinstall the mons with ubuntu 22.04 + quincy - reinstall the osds (same) - reinstall the mdss (same) Once this is up and running, we want to investigate and migrate to cephadm orchestration. Alternative appear to be: move to orchestration first and then upgrade ceph to quincy (possibly skipping the ubuntu upgrade?) Another alternative could be to upgrade to quincy on ubuntu 18.04 using packages, but I haven't investigated the availability of quincy packages for ubuntu 18.04 (which is out of free (LTS) support by canonical) Cheers /Simon