Recent upgrade from 18.2 to 19.2, upgrade went fine. Since the upgrade and a manager fail over, I can no longer run orchestrator commands. The only error I can find on an active manager daemon is the following, or it is the only one that stands out. 2025-01-06T18:48:41.698+0000 7fcf42b99640 -1 mgr load Failed to construct class in 'cephadm' 2025-01-06T18:48:41.698+0000 7fcf42b99640 -1 mgr load Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 667, in __init__ self.cert_key_store.load() File "/usr/share/ceph/mgr/cephadm/inventory.py", line 2073, in load self.known_certs[entity] = json.loads(v) File "/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2025-01-06T18:48:41.702+0000 7fcf42b99640 -1 mgr operator() Failed to run module in active mode ('cephadm') For any thing to really work in the dashboard I must have the balancer off. While the balance is off I can make changes in dashboard to the orchestrator, and it doesn't give me any trouble. When trying different commands from a ceph node say "ceph cephadm config-check status" it returns "Error ENOTSUP: Module 'cephadm' is not enabled/loaded (required by command 'cephadm config-check status'): use `ceph mgr module enable cephadm` to enable it". Running "ceph mgr module enable cephadm" returns "module 'cephadm' is already enabled". I really don't know where to look or what to try to resolve this. Any "ceph orch" command results in "Error ENOENT: Module not found" I don't know that my issue is related to https://tracker.ceph.com/issues/68657, but maybe it is. I have tried the following. Manually adding a new mgr daemon on different node, it starts runs the dashboard fine, but things are still not functional. Failed the mgr several times. Disabled/Enabled balancer. Disabled/Enabled mgr modules. Disabled/Enabled dashboard. All physical nodes are running Debian 12. Frank Frampton Senior Network Services Administrator Salt Lake City School District Desk: (801) 578-8223 Follow the district: Facebook<https://www.facebook.com/slcschools> | Instagram<https://instagram.com/slcschools> | Twitter<https://twitter.com/slcschools> | www.slcschools.org<http://www.slcschools.org/> Excellence and Equity: every student, every classroom, every day Scanned By Microsoft EOP