Cephadm - Error ENOENT: Module not found
Hello, After a successful upgrade of a Ceph cluster from 16.2.7 to 16.2.11, I needed to downgrade it back to 16.2.7 as I found an issue with the new version. I expected that running the downgrade with:`ceph orch upgrade start --ceph-version 16.2.7` should have worked fine. However, it blocked right after the downgrade of the first MGR daemon. In fact, the downgraded daemon is not able to use the cephadm module anymore. Any `ceph orch` command fails with the following error: ``` $ ceph orch ps Error ENOENT: Module not found ``` And the downgrade process is therefore blocked. These are the logs of the MGR when issuing the command: ``` Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.557+0000 7f828fe8c700 0 log_channel(audit) log [DBG] : from='client.3136173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.list_daemons(*(None, None), **{'daemon_id': None, 'host': None, 'refresh': False}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.get_feature_set(*(), **{}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 mgr.server reply reply (2) No such file or directory Module not found ``` Other interesting MGR logs are: ``` 2023-03-28T11:05:59.519+0000 7fcd16314700 4 mgr get_store get_store key: mgr/cephadm/upgrade_state 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Failed to construct class in 'cephadm' 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Traceback (most recent call last): e "/usr/share/ceph/mgr/cephadm/module.py", line 450, in __init__ elf.upgrade = CephadmUpgrade(self) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 111, in __init__ elf.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 92, in from_json eturn cls(**c) rror: __init__() got an unexpected keyword argument 'daemon_types' 2023-03-28T11:05:59.521+0000 7fcd16314700 -1 mgr operator() Failed to run module in active mode ('cephadm') ``` Which seem to relate to the new feature of staggered upgrades. Please note that before, everything was working fine with version 16.2.7. I am currently stuck in this situation with only one MGR daemon on version 16.2.11 which is the only one still working fine: ``` [root@astano01 ~]# ceph orch ps | grep mgr mgr.astano02.mzmewn astano02 *:8443,9283 running (5d) 43s ago 2y 455M - 16.2.11 7a63bce27215 e2d7806acf16 mgr.astano03.qtzccn astano03 *:8443,9283 running (3m) 22s ago 95m 383M - 16.2.7 463ec4b1fdc0 cc0d88864fa1 ``` Does anyone already faced this issue or knows how can I make the 16.2.7 MGR load the cephadm module correctly? Thanks in advance for any help!
for the specific issue with that traceback, you can probably resolve that by removing the stored upgrade state. We put it at `mgr/cephadm/upgrade_state` I believe (can check "ceph config-key ls" and look for something related to upgrade state if that doesn't work) so running "ceph config-key rm mgr/cephadm/upgrade_state" should remove the old one. Then I'd say manually downgrade the mgr daemons to avoid this happening again (process is roughly the same as https://docs.ceph.com/en/quincy/cephadm/upgrade/#upgrading-to-a-version-that...) and at that point you should be able to try using an upgrade command again. On Thu, Mar 30, 2023 at 11:07 AM <elia.oggian@cscs.ch> wrote:
Hello, After a successful upgrade of a Ceph cluster from 16.2.7 to 16.2.11, I needed to downgrade it back to 16.2.7 as I found an issue with the new version.
I expected that running the downgrade with:`ceph orch upgrade start --ceph-version 16.2.7` should have worked fine. However, it blocked right after the downgrade of the first MGR daemon. In fact, the downgraded daemon is not able to use the cephadm module anymore. Any `ceph orch` command fails with the following error:
``` $ ceph orch ps Error ENOENT: Module not found ``` And the downgrade process is therefore blocked.
These are the logs of the MGR when issuing the command:
``` Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.557+0000 7f828fe8c700 0 log_channel(audit) log [DBG] : from='client.3136173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.list_daemons(*(None, None), **{'daemon_id': None, 'host': None, 'refresh': False}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.get_feature_set(*(), **{}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 mgr.server reply reply (2) No such file or directory Module not found ```
Other interesting MGR logs are: ``` 2023-03-28T11:05:59.519+0000 7fcd16314700 4 mgr get_store get_store key: mgr/cephadm/upgrade_state 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Failed to construct class in 'cephadm' 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Traceback (most recent call last): e "/usr/share/ceph/mgr/cephadm/module.py", line 450, in __init__ elf.upgrade = CephadmUpgrade(self) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 111, in __init__ elf.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 92, in from_json eturn cls(**c) rror: __init__() got an unexpected keyword argument 'daemon_types'
2023-03-28T11:05:59.521+0000 7fcd16314700 -1 mgr operator() Failed to run module in active mode ('cephadm') ``` Which seem to relate to the new feature of staggered upgrades.
Please note that before, everything was working fine with version 16.2.7.
I am currently stuck in this situation with only one MGR daemon on version 16.2.11 which is the only one still working fine:
``` [root@astano01 ~]# ceph orch ps | grep mgr mgr.astano02.mzmewn astano02 *:8443,9283 running (5d) 43s ago 2y 455M - 16.2.11 7a63bce27215 e2d7806acf16 mgr.astano03.qtzccn astano03 *:8443,9283 running (3m) 22s ago 95m 383M - 16.2.7 463ec4b1fdc0 cc0d88864fa1 ```
Does anyone already faced this issue or knows how can I make the 16.2.7 MGR load the cephadm module correctly?
Thanks in advance for any help! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Adam, sorry for the very late reply. I also found out that the "mgr/cephadm/upgrade_state" config key was the issue. I actually just modified the config key and removed the unknown fields. This made "ceph orch" commands work again. Great. However, the downgrade process was quickly stuck on another step, with a similar message: ``` 2023-03-28T16:05:40.820967+0200 mgr.naret-monitor02.ciqvgv [WRN] unable to load cached state for naret-monitor01: DaemonDescription: __init__() got an unexpected keyword argument 'cpu_percentage' 2023-03-28T16:05:40.821262+0200 mgr.naret-monitor02.ciqvgv [WRN] unable to load cached state for naret-monitor02: DaemonDescription: __init__() got an unexpected keyword argument 'cpu_percentage' 2023-03-28T16:05:40.821499+0200 mgr.naret-monitor02.ciqvgv [WRN] unable to load cached state for naret-monitor03: DaemonDescription: __init__() got an unexpected keyword argument 'cpu_percentage' 2023-03-28T16:05:40.822704+0200 mgr.naret-monitor02.ciqvgv [WRN] unable to load cached state for naret-osd01: DaemonDescription: __init__() got an unexpected keyword argument 'cpu_percentage' 2023-03-28T16:05:40.823935+0200 mgr.naret-monitor02.ciqvgv [WRN] unable to load cached state for naret-osd02: DaemonDescription: __init__() got an unexpected keyword argument 'cpu_percentage' 2023-03-28T16:05:40.825008+0200 mgr.naret-monitor02.ciqvgv [WRN] unable to load cached state for naret-osd03: DaemonDescription: __init__() got an unexpected keyword argument 'cpu_percentage' ``` I did not find a solution to this, and as the downgrade process was again stuck, we finally decided to move forward and try a major upgrade to Quincy, version 17.2.5. The upgrade succeeded and everything is working fine now. It seems that the downgrade is not supported and, unfortunately, testing the upgrades in a test environment is not sufficient. We tested the upgrade on a small test cluster, and everything went fine, but when we did the upgrade on a much bigger production cluster this issue showed up: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/53NNVLLKLMYV... and we were therefore blocked. Luckily we found a solution in the end :) Thanks anyway for your help! Best Regards, Elia Oggian
participants (2)
-
Adam King
-
elia.oggian＠cscs.ch