I appreciate your kind words. 😎🙂 Frederics link has the correct answer, remove the respective field from the json. Zitat von Malte Stroem <malte.stroem@gmail.com>:
You're so cool, Eugen. Somehow you seem to find out everything.
Yes, this seems to be the issue and I suspected a bug there.
Looking here:
https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/services/osd.p...
The diff is included in the code.
What can I do now? Get the latest cephadm and put it on the node?
What about the cephadm under /var/lib/ceph/fsid?
I am not sure how to continue.
I would download the latest cephadm and put it under /usr/sbin.
Then disable the module with
ceph mgr module disable cephadm
and enable it
ceph mgr module enable cephadm
Best, Malte
On 17.10.24 19:20, Eugen Block wrote:
Oh why didn’t you mention earlier that you removed OSDs? 😄 it sounds like this one:
https://tracker.ceph.com/issues/67329
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hello Redouane,
thank you. Interesting.
ceph config-key dump
shows about 42000 lines.
What can I search for? Something with OSDs.
But there are thousands of entries.
And if I find something, how can I fix that?
I think there are entries of the OSDs from the broken node we removed.
Best, Malte
On 17.10.24 17:46, Redouane Kachach wrote:
So basically it's failing here:
 self.to_remove_osds.load_from_store()
This function is responsible of loading Specs from the mon-store. The information is stored in json format and it seems the stored json for the OSD(s) is not valid for some reason. You can see what's stored in the mon-store by running:
ceph config-key dump
Don't share the information publicly here especially if it's a production cluster as it may have sensitive information about your cluster.
Best, Redo.
On Thu, Oct 17, 2024 at 5:04 PM Malte Stroem <malte.stroem@gmail.com> wrote:
Thanks Eugen & Redouane,
of course I tried enabling and disabling the cephadm module for the MGRs.
Running ceph mgr module enable cephadm produces this output in the MGR log:
-1 mgr load Failed to construct class in 'cephadm'  -1 mgr load Traceback (most recent call last):   File "/usr/share/ceph/mgr/cephadm/module.py", line 619, in __init__     self.to_remove_osds.load_from_store()   File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 922, in load_from_store     for osd in json.loads(v):   File "/lib64/python3.9/json/__init__.py", line 346, in loads     return _default_decoder.decode(s)   File "/lib64/python3.9/json/decoder.py", line 337, in decode     obj, end = self.raw_decode(s, idx=_w(s, 0).end())   File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode     raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
 -1 mgr operator() Failed to run module in active mode ('cephadm')
This comes from inside the MGR container because it's Python3.9. On the hosts it'S Python3.11.
I think of redeploying an MGR.
Can I stop the existing MGRs?
Redeploying with ceph orch does not work of course, but I think this will work:
https://docs.ceph.com/en/latest/cephadm/troubleshooting/#manually- deploying-a-manager-daemon
because cephadm standalone is working. Crazy as it sounds.
What do you think?
Best, Malte
On 17.10.24 12:49, Eugen Block wrote:
Hi,
if you just execute cephadm commands, those are issued locally on the hosts, they won't confirm an orchestrator issue immediately. What does the active MGR log? It could show a stack trace or error messages which could point to a root cause.
> What about the cephadm files under /var/lib/ceph/fsid? Can I replace > the latest?
Those are the cephadm versions the orchestrator actually uses, it will just download them again from your registry (or upstream). Can you share:
ceph -s ceph versions MGR logs (active MGR)
Thanks, Eugen
Zitat von Malte Stroem <malte.stroem@gmail.com>:
> Hello, > > I am still struggling here and do not know the root cause of > this issue. > > Searching the list I found lots of people who had the same or a > similar problem the last years. > > However there is no solution four our cluster. > > Disabling and enabling the cephadm module does not work. There are no > error messages. When we run "ceph orch..." we get the error message: > > Error ENOENT: No orchestrator configured (try `ceph orch set backend`) > > But every single cephadm command works! > > cephadm ls for example. > > Stopping and restarting the MGRs did not help. Removing the .asok > files did not help. > > I think of stopping both MGRs and trying to deploy a new MGR like this: > > https://docs.ceph.com/en/latest/cephadm/troubleshooting/#manually- > deploying-a-manager-daemon > > How could I find the root cause? Is the cephadm somehow broken? > > What about the cephadm files under /var/lib/ceph/fsid? Can I replace > the latest? > > Best, > Malte > > On 16.10.24 14:54, Malte Stroem wrote: >> Hi Laimis, >> >> that did not work. Still ceph orch does not work. >> >> Best, >> Malte >> >> On 16.10.24 14:12, Malte Stroem wrote: >>> Thank you, Laimis. >>> >>> And you got the same error message? That's strange. >>> >>> In the mean time I try to check for clients connected. No Kubernetes >>> and CephFS, but RGWs. >>> >>> Best, >>> Malte >>> >>> On 16.10.24 14:01, Laimis Juzeliūnas wrote: >>>> Hi Malte, >>>> >>>> We have faced this recently when upgrading to Squid from >>>> latest Reef. >>>> As a temporary workaround we disabled the balancer with ‘ceph >>>> balancer off’ and restarted mgr daemons. >>>> We are suspecting older clients (from Kubernetes RBD mounts as well >>>> as CephFS mounts) on servers with incompatible client versions but >>>> are yet to dig through it. >>>> >>>> Best, >>>> Laimis J. >>>> >>>>> On 16 Oct 2024, at 14:57, Malte Stroem <malte.stroem@gmail.com> >>>>> wrote: >>>>> >>>>> Error ENOENT: No orchestrator configured (try `ceph orch set >>>>> backend`) >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io