upgrading from 15.2.17 to 16.2.11 - Health ERROR
hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Cluster was in health ok before starting.
that looks like it was expecting a json structure somewhere and got a blank string. Is there anything in the logs (ceph log last 100 info cephadm)? If not, might be worth trying a couple mgr failovers (I'm assuming only one got upgraded, so first failover would go back to the 15.2.17 one and then a second failover would go back to the 16.2.11 one) and then rechecking the logs. I'd expect this to generate a traceback there and it's hard to say what happened without that. On Tue, Mar 7, 2023 at 11:42 AM <xadhoom76@gmail.com> wrote:
hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0)
Cluster was in health ok before starting. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I cannnot find anything interesting in the cephadm.log now the error is HEALTH_ERR Module 'cephadm' has failed: 'cephadm' Idea how to fix it ?
I find out with ceph orch ps cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2 srvcephprod04 stopped 4m ago - <unknown> <unknown> <unknown> <unknown> cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2 srvcephprod06 stopped 4m ago - <unknown> <unknown> <unknown> <unknown> cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2 srvcephprod07 stopped 4m ago - <unknown> <unknown> <unknown> <unknown> And cannot remove.
looking at ceph orch upgrade check I find out }, "cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2": { "current_id": null, "current_name": null, "current_version": null }, Could this lead to the issue?
The things in "ceph orch ps" output are gathered by checking the contents of the /var/lib/ceph/<cluster-fsid>/ directory on the host. Those "cephadm.<hash>" files get deployed normally though, and aren't usually reported in "ceph orch ps" as it should only report things that are directories rather than files. You could maybe try going and removing them anyway to see what happens (cephadm should just deploy another one though). Would be interested anyway in what the contents of /var/lib/ceph/<cluster-fsid>/ are on that srvcephprod07 node and also what "cephadm ls" spits out on that node (you would have to put a copy of the cephadm tool on the host to run that). As for the logs, the "cephadm.log" on the host is only the log of what the cephadm tool has done on that host, not what the cephadm mgr module is running. Could maybe try "ceph mgr fail; ceph -W cephadm" and let it sit for a bit to see if you get a traceback printout that way. On Fri, Mar 10, 2023 at 10:41 AM <xadhoom76@gmail.com> wrote:
looking at ceph orch upgrade check I find out },
"cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2": { "current_id": null, "current_name": null, "current_version": null },
Could this lead to the issue? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
which version of cephadm you are using? ___________________________________ Clyso GmbH - Ceph Foundation Member Am 10.03.23 um 11:17 schrieb xadhoom76@gmail.com:
looking at ceph orch upgrade check I find out }, "cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2": { "current_id": null, "current_name": null, "current_version": null },
Could this lead to the issue? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
cephadm is 16.2.11, because the error comes from the upgrade from 15 to 16. Il giorno lun 13 mar 2023 alle ore 18:27 Clyso GmbH - Ceph Foundation Member <joachim.kraftmayer@clyso.com> ha scritto:
which version of cephadm you are using?
___________________________________ Clyso GmbH - Ceph Foundation Member
Am 10.03.23 um 11:17 schrieb xadhoom76@gmail.com:
looking at ceph orch upgrade check I find out },
"cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2": {
"current_id": null, "current_name": null, "current_version": null },
Could this lead to the issue? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Main error now is [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) If we disable cephadm, the health becomes ok. So is there a way to change the cephadm's version ? Mar 27 09:38:22 srvcephprod04 bash[11273]: debug 2023-03-27T07:38:22.281+0000 7f69c0f2a700 -1 Traceback (most recent call last): Mar 27 09:38:22 srvcephprod04 bash[11273]: File "/usr/share/ceph/mgr/cephadm/module.py", line 539, in serve Mar 27 09:38:22 srvcephprod04 bash[11273]: serve.serve() Mar 27 09:38:22 srvcephprod04 bash[11273]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 76, in serve Mar 27 09:38:22 srvcephprod04 bash[11273]: self.convert_tags_to_repo_digest() Mar 27 09:38:22 srvcephprod04 bash[11273]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 957, in convert_tags_to_repo_digest Mar 27 09:38:22 srvcephprod04 bash[11273]: image_info = self._get_container_image_info(container_image_ref) Mar 27 09:38:22 srvcephprod04 bash[11273]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1366, in _get_container_image_info Mar 27 09:38:22 srvcephprod04 bash[11273]: json.loads(str(self.mgr.get_store('registry_credentials')))) Mar 27 09:38:22 srvcephprod04 bash[11273]: File "/lib64/python3.6/json/__init__.py", line 354, in loads Mar 27 09:38:22 srvcephprod04 bash[11273]: return _default_decoder.decode(s) Mar 27 09:38:22 srvcephprod04 bash[11273]: File "/lib64/python3.6/json/decoder.py", line 339, in decode Mar 27 09:38:22 srvcephprod04 bash[11273]: obj, end = self.raw_decode(s, idx=_w(s, 0).end()) Mar 27 09:38:22 srvcephprod04 bash[11273]: File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode Mar 27 09:38:22 srvcephprod04 bash[11273]: raise JSONDecodeError("Expecting value", s, err.value) from None Mar 27 09:38:22 srvcephprod04 bash[11273]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Mar 27 09:38:22 srvcephprod04 bash[11273]: debug 2023-03-27T07:38:22.321+0000 7f69b6d16700 0 [prometheus INFO root] server_addr: 0.0.0.0 server_port: 9283
I've met this issue when try to upgrade octopus 15.2.17 to 16.2.13 last night. Upgrade process failed at mgr module phase after the new MGR version become to active state. I tried to enable debug `ceph config set mgr mgr/cephadm/log_to_cluster_level debug ` and I saw the message like @xadhoom76 about config_key "registry_credentials" I guessed the root cause because of this line `json.loads(str(self.mgr.get_store('registry_credentials'))))` and the key_store had a wrong value. Then I got the empty value when run this command "ceph config-key dump | grep 'registry_credentials'" and the same for "ceph config-key get mgr/cephadm/registry_credentials" By check `cephadm` source i see the value should be a json format like that ceph config-key set mgr/cephadm/registry_credentials '{"url": "registry.local:5000", "username": "user-deployer", "password": "xxxyyyyzzzzzz"}' After set this key and `ceph mgr fail` to reload , my cluster issue was gone .
participants (5)
-
Adam King
-
Alessandro Bolgia
-
Clyso GmbH - Ceph Foundation Member
-
letonphat1988＠gmail.com
-
xadhoom76＠gmail.com