[ceph-users] Upgrade failed, now ceph orch broken

Hi All, I have a cluster that that I tried upgrading from 15.2.4 to 15.2.5 using the command ‘ceph orch upgrade start —ceph-version 15.2.5’. After upgrading two of the three mgrs, the third mgr failed and the upgrade stopped. I was able to get the third mgr upgraded by changing the systemd file and manually removing the podman image. I have upgraded ceph-common outside the containers on all the mgr/mon nodes. The cluster repots HEALTH_OK. However, any ‘ceph orch’ command I issue now always returns: Error ENOENT: Module not found If I run ‘ceph orch ps —verbose’, I get: parsed_args: Namespace(admin_socket=None, block=False, cephconf=None, client_id=None, client_name=None, cluster=None, cluster_timeout=None, completion=False, help=False, input_file=None, output_file=None, output_format=None, period=1, setgroup=None, setuser=None, status=False, verbose=True, version=False, watch=False, watch_channel=None, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['orch', 'ps'] cmd000: pg stat ... better match: 2.5 > 1.5: orch ps [<hostname>] [<service_name>] [<daemon_type>] [<daemon_id>] [plain|json|json-pretty|yaml] [--refresh] bestcmds_sorted: [{'flags': 8, 'help': 'List daemons known to orchestrator', 'module': 'mgr', 'perm': 'r', 'sig': [argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=orch), argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=ps), argdesc(<class 'ceph_argparse.CephString'>, req=False, name=hostname, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=False, name=service_name, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=False, name=daemon_type, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=False, name=daemon_id, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephChoices'>, req=False, name=format, n=1, numseen=0, strings=plain|json|json-pretty|yaml), argdesc(<class 'ceph_argparse.CephBool'>, req=False, name=refresh, n=1, numseen=0)]}] Submitting command: {'prefix': 'orch ps', 'target': ('mon-mgr', '')} submit ['{"prefix": "orch ps", "target": ["mon-mgr", ""]}'] to mon-mgr Error ENOENT: Module not found Any help would be appreciated. Thanks, -TJ Ragan