Hello I'm trying to update my ceph cluster from the latest release of quincy 17.2.8 to reef 18.2.4. After starting the upgrade with 'ceph orch upgrade start --ceph-version 18.2.4' it stopped having upgraded 3/4 of my MGR daemons and everything else stayed on quincy. Any ceph command is not working saying orchestrator module is not found and Error ENOENT: Module not found. I have already tried looking at the osd issue hinted at https://tracker.ceph.com/issues/67329 but this happened on a test cluster without any OSDs on it as well. The current status given in the upgrade is the following ceph config-key get mgr/cephadm/upgrade_state {"target_name": "quay.io/ceph/ceph:v18.2.4", "progress_id": "a9e4d538-4c98-4307-8170-a82ca8ff300c", "target_id": "2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a", "target_digests": ["quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906", "quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77"], "target_version": "18.2.4", "fs_original_max_mds": null, "fs_original_allow_standby_replay": null, "error": "UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host hostname-redacted at addr (IPv4-address)", "paused": true, "daemon_types": null, "hosts": null, "services": null, "total_count": null, "remaining_count": null} How can I make the orchestrator work again to either rollback or upgrade to a newer version? the goal is to update this all the way to squid. Thank you
Please see https://docs.ceph.com/en/quincy/cephadm/upgrade/#potential-problems
On Feb 8, 2025, at 6:08 PM, cristian.tavarez@hostpapa.com wrote:
ENOENT: Module not found.
It comes from this backport (https://github.com/ceph/ceph/pull/56713) that made it to 17.2.8 but was not in 18.2.4, there is an extra argument that you need to remove from the mgr/cephadm/client_keyring to be able to proceed with the upgrade. This is an example below on how to hack it. # ceph config-key get "mgr/cephadm/client_keyrings" {"client.admin": {"entity": "client.admin", "placement": {"label": "_admin"}, "mode": 384, "uid": 0, "gid": 0, "include_ceph_conf": true}} # ceph config-key set "mgr/cephadm/client_keyrings" '{"client.admin": {"entity": "client.admin", "placement": {"label": "_admin"}, "mode": 384, "uid": 0, "gid": 0}}' set mgr/cephadm/client_keyrings # ceph config-key get "mgr/cephadm/client_keyrings" {"client.admin": {"entity": "client.admin", "placement": {"label": "_admin"}, "mode": 384, "uid": 0, "gid": 0}} https://tracker.ceph.com/issues/67660#note-4 Besides that the only other way to get out of this situation is to keep failing the mgr ` ceph mgr fail ` until you get the mgr that is 17.2.8 back as active to get the `ceph orch` commands back and stop the upgrade ` ceph orch upgrade stop`, but there is a short window to do this until it becomes unresponsive again. You find this error by looking at the journalctl of the 17.2.8 mgr.
Hello, This solution solved the issue, I also manually stopped some mgr processes to increase the chances the active one becomes 17.2.8. After that it stayed as active. Thank you
participants (3)
-
Anthony D'Atri
-
cristian.tavarez＠hostpapa.com
-
Joshua Blanch