7 Mar
2023
7 Mar
'23
10:26 a.m.
hi , starting upgrade from 15.2.17 i got this error
Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0)
Cluster was in health ok before starting.
7 Mar
7 Mar
12:51 p.m.
that looks like it was expecting a json structure somewhere and got a blank
string. Is there anything in the logs (ceph log last 100 info cephadm)? If
not, might be worth trying a couple mgr failovers (I'm assuming only one
got upgraded, so first failover would go back to the 15.2.17 one and then a
second failover would go back to the 16.2.11 one) and then rechecking the
logs. I'd expect this to generate a traceback there and it's hard to say
what happened without that.
On Tue, Mar 7, 2023 at 11:42 AM <xadhoom76(a)gmail.com> wrote:
hi , starting upgrade from 15.2.17 i got this error
Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0)
Cluster was in health ok before starting.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10 Mar
10 Mar
5:50 a.m.
I cannnot find anything interesting in the cephadm.log
now the error is
HEALTH_ERR
Module 'cephadm' has failed: 'cephadm'
Idea how to fix it ?
5:58 a.m.
I find out with ceph orch ps
cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2 srvcephprod04
stopped 4m ago - <unknown> <unknown>
<unknown>
<unknown>
cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2 srvcephprod06
stopped 4m ago - <unknown> <unknown>
<unknown>
<unknown>
cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2 srvcephprod07
stopped 4m ago - <unknown> <unknown>
<unknown>
<unknown>
And cannot remove.
6:17 a.m.
looking at ceph orch upgrade check
I find out
},
"cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2": {
"current_id": null,
"current_name": null,
"current_version": null
},
Could this lead to the issue?
2:13 p.m.
The things in "ceph orch ps" output are gathered by checking the contents
of the /var/lib/ceph/<cluster-fsid>/ directory on the host. Those
"cephadm.<hash>" files get deployed normally though, and aren't
usually
reported in "ceph orch ps" as it should only report things that are
directories rather than files. You could maybe try going and removing them
anyway to see what happens (cephadm should just deploy another one though).
Would be interested anyway in what the contents of
/var/lib/ceph/<cluster-fsid>/ are on that srvcephprod07 node and also what
"cephadm ls" spits out on that node (you would have to put a copy of the
cephadm tool on the host to run that).
As for the logs, the "cephadm.log" on the host is only the log of what the
cephadm tool has done on that host, not what the cephadm mgr module is
running. Could maybe try "ceph mgr fail; ceph -W cephadm" and let it sit
for a bit to see if you get a traceback printout that way.
On Fri, Mar 10, 2023 at 10:41 AM <xadhoom76(a)gmail.com> wrote:
looking at ceph orch upgrade check
I find out
},
"cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2":
{
"current_id": null,
"current_name": null,
"current_version": null
},
Could this lead to the issue?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
13 Mar
13 Mar
2:27 p.m.
which version of cephadm you are using?
___________________________________
Clyso GmbH - Ceph Foundation Member
Am 10.03.23 um 11:17 schrieb xadhoom76(a)gmail.com:
> looking at ceph orch upgrade check
> I find out
> },
>
"cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2": {
> "current_id": null,
> "current_name": null,
> "current_version": null
> },
>
>
> Could this lead to the issue?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
14 Mar
14 Mar
4:32 a.m.
cephadm is 16.2.11, because the error comes from the upgrade from 15 to 16.
Il giorno lun 13 mar 2023 alle ore 18:27 Clyso GmbH - Ceph Foundation
Member <joachim.kraftmayer(a)clyso.com> ha scritto:
which version of cephadm you are using?
___________________________________
Clyso GmbH - Ceph Foundation Member
Am 10.03.23 um 11:17 schrieb xadhoom76(a)gmail.com:
looking at ceph orch upgrade check
I find out
},
"cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2":
{
"current_id": null,
"current_name": null,
"current_version": null
},
Could this lead to the issue?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
29 Mar
29 Mar
5 a.m.
Main error now is
[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Expecting value: line 1
column 1 (char 0)
Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0)
If we disable cephadm, the health becomes ok.
So is there a way to change the cephadm's version ?
Mar 27 09:38:22 srvcephprod04 bash[11273]: debug 2023-03-27T07:38:22.281+0000 7f69c0f2a700
-1 Traceback (most recent call last):
Mar 27 09:38:22 srvcephprod04 bash[11273]: File
"/usr/share/ceph/mgr/cephadm/module.py", line 539, in serve
Mar 27 09:38:22 srvcephprod04 bash[11273]: serve.serve()
Mar 27 09:38:22 srvcephprod04 bash[11273]: File
"/usr/share/ceph/mgr/cephadm/serve.py", line 76, in serve
Mar 27 09:38:22 srvcephprod04 bash[11273]: self.convert_tags_to_repo_digest()
Mar 27 09:38:22 srvcephprod04 bash[11273]: File
"/usr/share/ceph/mgr/cephadm/serve.py", line 957, in
convert_tags_to_repo_digest
Mar 27 09:38:22 srvcephprod04 bash[11273]: image_info =
self._get_container_image_info(container_image_ref)
Mar 27 09:38:22 srvcephprod04 bash[11273]: File
"/usr/share/ceph/mgr/cephadm/serve.py", line 1366, in _get_container_image_info
Mar 27 09:38:22 srvcephprod04 bash[11273]:
json.loads(str(self.mgr.get_store('registry_credentials'))))
Mar 27 09:38:22 srvcephprod04 bash[11273]: File
"/lib64/python3.6/json/__init__.py", line 354, in loads
Mar 27 09:38:22 srvcephprod04 bash[11273]: return _default_decoder.decode(s)
Mar 27 09:38:22 srvcephprod04 bash[11273]: File
"/lib64/python3.6/json/decoder.py", line 339, in decode
Mar 27 09:38:22 srvcephprod04 bash[11273]: obj, end = self.raw_decode(s, idx=_w(s,
0).end())
Mar 27 09:38:22 srvcephprod04 bash[11273]: File
"/lib64/python3.6/json/decoder.py", line 357, in raw_decode
Mar 27 09:38:22 srvcephprod04 bash[11273]: raise JSONDecodeError("Expecting
value", s, err.value) from None
Mar 27 09:38:22 srvcephprod04 bash[11273]: json.decoder.JSONDecodeError: Expecting value:
line 1 column 1 (char 0)
Mar 27 09:38:22 srvcephprod04 bash[11273]: debug 2023-03-27T07:38:22.321+0000 7f69b6d16700
0 [prometheus INFO root] server_addr: 0.0.0.0 server_port: 9283
4 Jul
4 Jul
12:44 a.m.
I've met this issue when try to upgrade octopus 15.2.17 to 16.2.13 last night. Upgrade
process failed at mgr module phase after the new MGR version become to active state. I
tried to enable debug `ceph config set mgr mgr/cephadm/log_to_cluster_level debug
` and I saw the message like @xadhoom76 about config_key "registry_credentials"
I guessed the root cause because of this line
`json.loads(str(self.mgr.get_store('registry_credentials'))))` and the key_store
had a wrong value. Then I got the empty value when run this command "ceph config-key
dump | grep 'registry_credentials'" and the same for "ceph config-key
get mgr/cephadm/registry_credentials"
By check `cephadm` source i see the value should be a json format like that
ceph config-key set mgr/cephadm/registry_credentials '{"url":
"registry.local:5000", "username": "user-deployer",
"password": "xxxyyyyzzzzzz"}'
After set this key and `ceph mgr fail` to reload , my cluster issue was gone .
303
days inactive
422
days old
9 comments
5 participants
participants (5)
-
Adam King -
Alessandro Bolgia -
Clyso GmbH - Ceph Foundation Member -
letonphat1988＠gmail.com -
xadhoom76＠gmail.com