Hi Malte, so I would only suggest to bring up a new MGR, issue a failover to that MGR and see if you get the orchestrator to work again. It should suffice to change the container_image in the unit.run file (/var/lib/ceph/{FSID}/mgr.{MGR}/unit.run): CONTAINER_IMAGE={NEWER IMAGE} So stop one MGR, change the container image, start it and make sure it takes over as the active MGR.
But I would like to know if I could replace the cephadm on one running node, stop the MGR and deploy a new MGR on that node with this:
https://docs.ceph.com/en/latest/cephadm/troubleshooting/#manually-deploying-...
cephadm --image <container-image> deploy --fsid <fsid> --name mgr.hostname.smfvfd --config-json config-json.json
This approach probably works as well, but I haven't tried that yet.
And I still do not know what places cephadm... under /var/lib/ceph/fsid.
Does that happen when I enable the orchestrator in the MGR?
And can I replace that cephadm by hand?
The orchestrator would automatically download the respective cephadm image into that directory if you changed the container_image config value(s). But I wouldn't do that because you could break your cluster. If for some reason a MON, OSD or some other Ceph daemon would need to be redeployed, you would basically upgrade it. That's why I would suggest to only start one single MGR daemon with a newer version to see how it goes. In case you get the orchestrator to work again, I would "downgrade" it again and see what happens next. Zitat von Eugen Block <eblock@nde.ag>:
I’m on a mobile phone right now, I can’t go into much detail right now. But I don’t think it’s necessary to rebuild an entire node, just a mgr. otherwise you risk cluster integrity if you redeploy a mon as well with a newer image. I’ll respond later in more detail.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Well, thank you, Eugen. That is what I planned to do.
Rebuild the broken node and start a MON and a MGR there with the latest images. Then I will stop the other MGRs and have a look if it's working.
But I would like to know if I could replace the cephadm on one running node, stop the MGR and deploy a new MGR on that node with this:
https://docs.ceph.com/en/latest/cephadm/troubleshooting/#manually-deploying-...
cephadm --image <container-image> deploy --fsid <fsid> --name mgr.hostname.smfvfd --config-json config-json.json
And I still do not know what places cephadm... under /var/lib/ceph/fsid.
Does that happen when I enable the orchestrator in the MGR?
And can I replace that cephadm by hand?
Best, Malte
On 18.10.24 12:11, Eugen Block wrote:
Okay, then I misinterpreted your former statement:
I think there are entries of the OSDs from the broken node we removed.
So the stack trace in the log points to the osd_remove_queue, but I don't understand why it's empty. Is there still some OSD removal going on or something? Did you paste your current cluster status already? You could probably try starting a Squid mgr daemon by replacing the container image in the unit.run file and see how that goes.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hello Eugen,
thanks a lot. However:
ceph config-key get mgr/cephadm/osd_remove_queue
is empty!
Damn.
So should I get a new cephadm with the diff included?
Best, Malte
On 17.10.24 23:48, Eugen Block wrote:
Save the current output to a file:
ceph config-key get mgr/cephadm/osd_remove_queue > remove_queue.json
Then remove the original_weight key from the json and set the modified key again with: ceph config-key set … Then fail the mgr.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hello Frederic, Hello Eugen,
yes, but I am not sure how to do it.
The links says:
> the config-key responsible was mgr/cephadm/osd_remove_queue > > This is what it looked like before. After removing the > original_weight field and setting the variable again, the > cephadm module loads and orch works.
So now: Do I remove the value of mgr/cephadm/osd_remove_queue?
Or:
What is meant by:
"After removing the original_weight field and setting the variable again, the cephadm module loads and orch works."
I can enter a MGR's container and open the file:
/usr/share/ceph/mgr/cephadm/services/osd.py
But what is meant by "removing the original_weight field and setting the variable again" and what JSON do you mean, Eugen?
> osd_obj = OSD.from_json(osd, rm_util=self.rm_util)
Code looks like this:
def load_from_store(self) -> None: with self.lock: for k, v in self.mgr.get_store_prefix('osd_remove_queue').items(): for osd in json.loads(v): logger.debug(f"Loading osd ->{osd} from store") osd_obj = OSD.from_json(osd, rm_util=self.rm_util) if osd_obj is not None: self.osds.add(osd_obj)
I am a bit lost here.
Best, Malte
On 17.10.24 21:50, Eugen Block wrote: > I appreciate your kind words. 😎🙂 > Frederics link has the correct answer, remove the respective > field from the json. > > Zitat von Malte Stroem <malte.stroem@gmail.com>: > >> You're so cool, Eugen. Somehow you seem to find out everything. >> >> Yes, this seems to be the issue and I suspected a bug there. >> >> Looking here: >> >> https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/ >> services/osd.py >> >> The diff is included in the code. >> >> What can I do now? Get the latest cephadm and put it on the node? >> >> What about the cephadm under /var/lib/ceph/fsid? >> >> I am not sure how to continue. >> >> I would download the latest cephadm and put it under /usr/sbin. >> >> Then disable the module with >> >> ceph mgr module disable cephadm >> >> and enable it >> >> ceph mgr module enable cephadm >> >> Best, >> Malte >> >> On 17.10.24 19:20, Eugen Block wrote: >>> Oh why didn’t you mention earlier that you removed OSDs? 😄 >>> it sounds like this one: >>> >>> https://tracker.ceph.com/issues/67329 >>> >>> Zitat von Malte Stroem <malte.stroem@gmail.com>: >>> >>>> Hello Redouane, >>>> >>>> thank you. Interesting. >>>> >>>> ceph config-key dump >>>> >>>> shows about 42000 lines. >>>> >>>> What can I search for? Something with OSDs. >>>> >>>> But there are thousands of entries. >>>> >>>> And if I find something, how can I fix that? >>>> >>>> I think there are entries of the OSDs from the broken node >>>> we removed. >>>> >>>> Best, >>>> Malte >>>> >>>> On 17.10.24 17:46, Redouane Kachach wrote: >>>>> So basically it's failing here: >>>>> >>>>>> self.to_remove_osds.load_from_store() >>>>> >>>>> This function is responsible of loading Specs from the >>>>> mon- store. The >>>>> information is stored in json format and it seems the >>>>> stored json for the OSD(s) is not valid for some reason. >>>>> You can see what's >>>>> stored in the mon-store by running: >>>>> >>>>>> ceph config-key dump >>>>> >>>>> Don't share the information publicly here especially if it's a >>>>> production cluster as it may have sensitive information >>>>> about your cluster. >>>>> >>>>> Best, >>>>> Redo. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Oct 17, 2024 at 5:04 PM Malte Stroem >>>>> <malte.stroem@gmail.com> wrote: >>>>> >>>>>> Thanks Eugen & Redouane, >>>>>> >>>>>> of course I tried enabling and disabling the cephadm >>>>>> module for the MGRs. >>>>>> >>>>>> Running ceph mgr module enable cephadm produces this >>>>>> output in the MGR log: >>>>>> >>>>>> -1 mgr load Failed to construct class in 'cephadm' >>>>>> -1 mgr load Traceback (most recent call last): >>>>>> File "/usr/share/ceph/mgr/cephadm/module.py", line >>>>>> 619, in __init__ >>>>>> self.to_remove_osds.load_from_store() >>>>>> File "/usr/share/ceph/mgr/cephadm/services/osd.py", >>>>>> line 922, in >>>>>> load_from_store >>>>>> for osd in json.loads(v): >>>>>> File "/lib64/python3.9/json/__init__.py", line 346, in loads >>>>>> return _default_decoder.decode(s) >>>>>> File "/lib64/python3.9/json/decoder.py", line 337, in decode >>>>>> obj, end = self.raw_decode(s, idx=_w(s, 0).end()) >>>>>> File "/lib64/python3.9/json/decoder.py", line 355, in >>>>>> raw_decode >>>>>> raise JSONDecodeError("Expecting value", s, >>>>>> err.value) from None >>>>>> json.decoder.JSONDecodeError: Expecting value: line 1 >>>>>> column 1 (char 0) >>>>>> >>>>>> >>>>>> -1 mgr operator() Failed to run module in active mode >>>>>> ('cephadm') >>>>>> >>>>>> This comes from inside the MGR container because it's >>>>>> Python3.9. On the >>>>>> hosts it'S Python3.11. >>>>>> >>>>>> I think of redeploying an MGR. >>>>>> >>>>>> Can I stop the existing MGRs? >>>>>> >>>>>> Redeploying with ceph orch does not work of course, but I >>>>>> think this >>>>>> will work: >>>>>> >>>>>> >>>>>> https://docs.ceph.com/en/latest/cephadm/troubleshooting/ >>>>>> #manually- deploying-a-manager-daemon >>>>>> >>>>>> because cephadm standalone is working. Crazy as it sounds. >>>>>> >>>>>> What do you think? >>>>>> >>>>>> Best, >>>>>> Malte >>>>>> >>>>>> On 17.10.24 12:49, Eugen Block wrote: >>>>>>> Hi, >>>>>>> >>>>>>> if you just execute cephadm commands, those are issued >>>>>>> locally on the >>>>>>> hosts, they won't confirm an orchestrator issue immediately. >>>>>>> What does the active MGR log? It could show a stack >>>>>>> trace or error >>>>>>> messages which could point to a root cause. >>>>>>> >>>>>>>> What about the cephadm files under /var/lib/ceph/fsid? >>>>>>>> Can I replace >>>>>>>> the latest? >>>>>>> >>>>>>> Those are the cephadm versions the orchestrator actually >>>>>>> uses, it will >>>>>>> just download them again from your registry (or upstream). >>>>>>> Can you share: >>>>>>> >>>>>>> ceph -s >>>>>>> ceph versions >>>>>>> MGR logs (active MGR) >>>>>>> >>>>>>> Thanks, >>>>>>> Eugen >>>>>>> >>>>>>> Zitat von Malte Stroem <malte.stroem@gmail.com>: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I am still struggling here and do not know the root >>>>>>>> cause of this issue. >>>>>>>> >>>>>>>> Searching the list I found lots of people who had the same or a >>>>>>>> similar problem the last years. >>>>>>>> >>>>>>>> However there is no solution four our cluster. >>>>>>>> >>>>>>>> Disabling and enabling the cephadm module does not >>>>>>>> work. There are no >>>>>>>> error messages. When we run "ceph orch..." we get the >>>>>>>> error message: >>>>>>>> >>>>>>>> Error ENOENT: No orchestrator configured (try `ceph >>>>>>>> orch set backend`) >>>>>>>> >>>>>>>> But every single cephadm command works! >>>>>>>> >>>>>>>> cephadm ls for example. >>>>>>>> >>>>>>>> Stopping and restarting the MGRs did not help. Removing >>>>>>>> the .asok >>>>>>>> files did not help. >>>>>>>> >>>>>>>> I think of stopping both MGRs and trying to deploy a >>>>>>>> new MGR like this: >>>>>>>> >>>>>>>> https://docs.ceph.com/en/latest/cephadm/troubleshooting/ >>>>>>>> #manually- >>>>>>>> deploying-a-manager-daemon >>>>>>>> >>>>>>>> How could I find the root cause? Is the cephadm somehow broken? >>>>>>>> >>>>>>>> What about the cephadm files under /var/lib/ceph/fsid? >>>>>>>> Can I replace >>>>>>>> the latest? >>>>>>>> >>>>>>>> Best, >>>>>>>> Malte >>>>>>>> >>>>>>>> On 16.10.24 14:54, Malte Stroem wrote: >>>>>>>>> Hi Laimis, >>>>>>>>> >>>>>>>>> that did not work. Still ceph orch does not work. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Malte >>>>>>>>> >>>>>>>>> On 16.10.24 14:12, Malte Stroem wrote: >>>>>>>>>> Thank you, Laimis. >>>>>>>>>> >>>>>>>>>> And you got the same error message? That's strange. >>>>>>>>>> >>>>>>>>>> In the mean time I try to check for clients >>>>>>>>>> connected. No Kubernetes >>>>>>>>>> and CephFS, but RGWs. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Malte >>>>>>>>>> >>>>>>>>>> On 16.10.24 14:01, Laimis Juzeliūnas wrote: >>>>>>>>>>> Hi Malte, >>>>>>>>>>> >>>>>>>>>>> We have faced this recently when upgrading to Squid >>>>>>>>>>> from latest Reef. >>>>>>>>>>> As a temporary workaround we disabled the balancer with ‘ceph >>>>>>>>>>> balancer off’ and restarted mgr daemons. >>>>>>>>>>> We are suspecting older clients (from Kubernetes RBD >>>>>>>>>>> mounts as well >>>>>>>>>>> as CephFS mounts) on servers with incompatible >>>>>>>>>>> client versions but >>>>>>>>>>> are yet to dig through it. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Laimis J. >>>>>>>>>>> >>>>>>>>>>>> On 16 Oct 2024, at 14:57, Malte Stroem >>>>>>>>>>>> <malte.stroem@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Error ENOENT: No orchestrator configured (try `ceph orch set >>>>>>>>>>>> backend`) >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io