Hello Eugen, thanks a lot. We got our down time today to work on the cluster. However, nothing worked. Even with Ceph 19. All ceph orch commands do not work. Error ENOENT: No orchestrator configured (try `ceph orch set backend`) This has nothing to do with osd_remove_queue. Getting back the MON quorum with three MONs and also three MGRs with Squid did not help at all. I still think this can be fixed somehow. Perhaps with editing the mon store somehow but I don't know where. We decided to deploy a new cluster since backups are available. Thanks again everybody. Best, Malte On 18.10.24 16:37, Eugen Block wrote:
Hi Malte,
so I would only suggest to bring up a new MGR, issue a failover to that MGR and see if you get the orchestrator to work again. It should suffice to change the container_image in the unit.run file (/ var/lib/ceph/{FSID}/mgr.{MGR}/unit.run):
CONTAINER_IMAGE={NEWER IMAGE}
So stop one MGR, change the container image, start it and make sure it takes over as the active MGR.
But I would like to know if I could replace the cephadm on one running node, stop the MGR and deploy a new MGR on that node with this:
https://docs.ceph.com/en/latest/cephadm/troubleshooting/#manually- deploying-a-manager-daemon
cephadm --image <container-image> deploy --fsid <fsid> --name mgr.hostname.smfvfd --config-json config-json.json
This approach probably works as well, but I haven't tried that yet.
And I still do not know what places cephadm... under /var/lib/ceph/fsid.
Does that happen when I enable the orchestrator in the MGR?
And can I replace that cephadm by hand?
The orchestrator would automatically download the respective cephadm image into that directory if you changed the container_image config value(s). But I wouldn't do that because you could break your cluster. If for some reason a MON, OSD or some other Ceph daemon would need to be redeployed, you would basically upgrade it. That's why I would suggest to only start one single MGR daemon with a newer version to see how it goes. In case you get the orchestrator to work again, I would "downgrade" it again and see what happens next.
Zitat von Eugen Block <eblock@nde.ag>:
I’m on a mobile phone right now, I can’t go into much detail right now. But I don’t think it’s necessary to rebuild an entire node, just a mgr. otherwise you risk cluster integrity if you redeploy a mon as well with a newer image. I’ll respond later in more detail.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Well, thank you, Eugen. That is what I planned to do.
Rebuild the broken node and start a MON and a MGR there with the latest images. Then I will stop the other MGRs and have a look if it's working.
But I would like to know if I could replace the cephadm on one running node, stop the MGR and deploy a new MGR on that node with this:
https://docs.ceph.com/en/latest/cephadm/troubleshooting/#manually- deploying-a-manager-daemon
cephadm --image <container-image> deploy --fsid <fsid> --name mgr.hostname.smfvfd --config-json config-json.json
And I still do not know what places cephadm... under /var/lib/ceph/fsid.
Does that happen when I enable the orchestrator in the MGR?
And can I replace that cephadm by hand?
Best, Malte
On 18.10.24 12:11, Eugen Block wrote:
Okay, then I misinterpreted your former statement:
I think there are entries of the OSDs from the broken node we removed.
So the stack trace in the log points to the osd_remove_queue, but I don't understand why it's empty. Is there still some OSD removal going on or something? Did you paste your current cluster status already? You could probably try starting a Squid mgr daemon by replacing the container image in the unit.run file and see how that goes.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hello Eugen,
thanks a lot. However:
ceph config-key get mgr/cephadm/osd_remove_queue
is empty!
Damn.
So should I get a new cephadm with the diff included?
Best, Malte
On 17.10.24 23:48, Eugen Block wrote:
Save the current output to a file:
ceph config-key get mgr/cephadm/osd_remove_queue > remove_queue.json
Then remove the original_weight key from the json and set the modified key again with: ceph config-key set … Then fail the mgr.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
> Hello Frederic, Hello Eugen, > > yes, but I am not sure how to do it. > > The links says: > >> the config-key responsible was mgr/cephadm/osd_remove_queue >> >> This is what it looked like before. After removing the >> original_weight field and setting the variable again, the >> cephadm module loads and orch works. > > So now: Do I remove the value of mgr/cephadm/osd_remove_queue? > > Or: > > What is meant by: > > "After removing the original_weight field and setting the > variable again, the cephadm module loads and orch works." > > I can enter a MGR's container and open the file: > > /usr/share/ceph/mgr/cephadm/services/osd.py > > But what is meant by "removing the original_weight field and > setting the variable again" and what JSON do you mean, Eugen? > >> osd_obj = OSD.from_json(osd, rm_util=self.rm_util) > > Code looks like this: > > def load_from_store(self) -> None: > with self.lock: > for k, v in > self.mgr.get_store_prefix('osd_remove_queue').items(): > for osd in json.loads(v): > logger.debug(f"Loading osd ->{osd} from store") > osd_obj = OSD.from_json(osd, > rm_util=self.rm_util) > if osd_obj is not None: > self.osds.add(osd_obj) > > I am a bit lost here. > > Best, > Malte > > On 17.10.24 21:50, Eugen Block wrote: >> I appreciate your kind words. 😎🙂 >> Frederics link has the correct answer, remove the respective >> field from the json. >> >> Zitat von Malte Stroem <malte.stroem@gmail.com>: >> >>> You're so cool, Eugen. Somehow you seem to find out everything. >>> >>> Yes, this seems to be the issue and I suspected a bug there. >>> >>> Looking here: >>> >>> https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/ >>> services/osd.py >>> >>> The diff is included in the code. >>> >>> What can I do now? Get the latest cephadm and put it on the node? >>> >>> What about the cephadm under /var/lib/ceph/fsid? >>> >>> I am not sure how to continue. >>> >>> I would download the latest cephadm and put it under /usr/sbin. >>> >>> Then disable the module with >>> >>> ceph mgr module disable cephadm >>> >>> and enable it >>> >>> ceph mgr module enable cephadm >>> >>> Best, >>> Malte >>> >>> On 17.10.24 19:20, Eugen Block wrote: >>>> Oh why didn’t you mention earlier that you removed OSDs? 😄 it >>>> sounds like this one: >>>> >>>> https://tracker.ceph.com/issues/67329 >>>> >>>> Zitat von Malte Stroem <malte.stroem@gmail.com>: >>>> >>>>> Hello Redouane, >>>>> >>>>> thank you. Interesting. >>>>> >>>>> ceph config-key dump >>>>> >>>>> shows about 42000 lines. >>>>> >>>>> What can I search for? Something with OSDs. >>>>> >>>>> But there are thousands of entries. >>>>> >>>>> And if I find something, how can I fix that? >>>>> >>>>> I think there are entries of the OSDs from the broken node we >>>>> removed. >>>>> >>>>> Best, >>>>> Malte >>>>> >>>>> On 17.10.24 17:46, Redouane Kachach wrote: >>>>>> So basically it's failing here: >>>>>> >>>>>>> self.to_remove_osds.load_from_store() >>>>>> >>>>>> This function is responsible of loading Specs from the mon- >>>>>> store. The >>>>>> information is stored in json format and it seems the >>>>>> stored json for the OSD(s) is not valid for some reason. You >>>>>> can see what's >>>>>> stored in the mon-store by running: >>>>>> >>>>>>> ceph config-key dump >>>>>> >>>>>> Don't share the information publicly here especially if it's a >>>>>> production cluster as it may have sensitive information >>>>>> about your cluster. >>>>>> >>>>>> Best, >>>>>> Redo. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Oct 17, 2024 at 5:04 PM Malte Stroem >>>>>> <malte.stroem@gmail.com> wrote: >>>>>> >>>>>>> Thanks Eugen & Redouane, >>>>>>> >>>>>>> of course I tried enabling and disabling the cephadm module >>>>>>> for the MGRs. >>>>>>> >>>>>>> Running ceph mgr module enable cephadm produces this output >>>>>>> in the MGR log: >>>>>>> >>>>>>> -1 mgr load Failed to construct class in 'cephadm' >>>>>>> -1 mgr load Traceback (most recent call last): >>>>>>> File "/usr/share/ceph/mgr/cephadm/module.py", line 619, >>>>>>> in __init__ >>>>>>> self.to_remove_osds.load_from_store() >>>>>>> File "/usr/share/ceph/mgr/cephadm/services/osd.py", line >>>>>>> 922, in >>>>>>> load_from_store >>>>>>> for osd in json.loads(v): >>>>>>> File "/lib64/python3.9/json/__init__.py", line 346, in >>>>>>> loads >>>>>>> return _default_decoder.decode(s) >>>>>>> File "/lib64/python3.9/json/decoder.py", line 337, in >>>>>>> decode >>>>>>> obj, end = self.raw_decode(s, idx=_w(s, 0).end()) >>>>>>> File "/lib64/python3.9/json/decoder.py", line 355, in >>>>>>> raw_decode >>>>>>> raise JSONDecodeError("Expecting value", s, err.value) >>>>>>> from None >>>>>>> json.decoder.JSONDecodeError: Expecting value: line 1 >>>>>>> column 1 (char 0) >>>>>>> >>>>>>> >>>>>>> -1 mgr operator() Failed to run module in active mode >>>>>>> ('cephadm') >>>>>>> >>>>>>> This comes from inside the MGR container because it's >>>>>>> Python3.9. On the >>>>>>> hosts it'S Python3.11. >>>>>>> >>>>>>> I think of redeploying an MGR. >>>>>>> >>>>>>> Can I stop the existing MGRs? >>>>>>> >>>>>>> Redeploying with ceph orch does not work of course, but I >>>>>>> think this >>>>>>> will work: >>>>>>> >>>>>>> >>>>>>> https://docs.ceph.com/en/latest/cephadm/troubleshooting/ >>>>>>> #manually- deploying-a-manager-daemon >>>>>>> >>>>>>> because cephadm standalone is working. Crazy as it sounds. >>>>>>> >>>>>>> What do you think? >>>>>>> >>>>>>> Best, >>>>>>> Malte >>>>>>> >>>>>>> On 17.10.24 12:49, Eugen Block wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> if you just execute cephadm commands, those are issued >>>>>>>> locally on the >>>>>>>> hosts, they won't confirm an orchestrator issue immediately. >>>>>>>> What does the active MGR log? It could show a stack trace >>>>>>>> or error >>>>>>>> messages which could point to a root cause. >>>>>>>> >>>>>>>>> What about the cephadm files under /var/lib/ceph/fsid? >>>>>>>>> Can I replace >>>>>>>>> the latest? >>>>>>>> >>>>>>>> Those are the cephadm versions the orchestrator actually >>>>>>>> uses, it will >>>>>>>> just download them again from your registry (or upstream). >>>>>>>> Can you share: >>>>>>>> >>>>>>>> ceph -s >>>>>>>> ceph versions >>>>>>>> MGR logs (active MGR) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Eugen >>>>>>>> >>>>>>>> Zitat von Malte Stroem <malte.stroem@gmail.com>: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I am still struggling here and do not know the root cause >>>>>>>>> of this issue. >>>>>>>>> >>>>>>>>> Searching the list I found lots of people who had the >>>>>>>>> same or a >>>>>>>>> similar problem the last years. >>>>>>>>> >>>>>>>>> However there is no solution four our cluster. >>>>>>>>> >>>>>>>>> Disabling and enabling the cephadm module does not work. >>>>>>>>> There are no >>>>>>>>> error messages. When we run "ceph orch..." we get the >>>>>>>>> error message: >>>>>>>>> >>>>>>>>> Error ENOENT: No orchestrator configured (try `ceph orch >>>>>>>>> set backend`) >>>>>>>>> >>>>>>>>> But every single cephadm command works! >>>>>>>>> >>>>>>>>> cephadm ls for example. >>>>>>>>> >>>>>>>>> Stopping and restarting the MGRs did not help. Removing >>>>>>>>> the .asok >>>>>>>>> files did not help. >>>>>>>>> >>>>>>>>> I think of stopping both MGRs and trying to deploy a new >>>>>>>>> MGR like this: >>>>>>>>> >>>>>>>>> https://docs.ceph.com/en/latest/cephadm/troubleshooting/ >>>>>>>>> #manually- >>>>>>>>> deploying-a-manager-daemon >>>>>>>>> >>>>>>>>> How could I find the root cause? Is the cephadm somehow >>>>>>>>> broken? >>>>>>>>> >>>>>>>>> What about the cephadm files under /var/lib/ceph/fsid? >>>>>>>>> Can I replace >>>>>>>>> the latest? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Malte >>>>>>>>> >>>>>>>>> On 16.10.24 14:54, Malte Stroem wrote: >>>>>>>>>> Hi Laimis, >>>>>>>>>> >>>>>>>>>> that did not work. Still ceph orch does not work. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Malte >>>>>>>>>> >>>>>>>>>> On 16.10.24 14:12, Malte Stroem wrote: >>>>>>>>>>> Thank you, Laimis. >>>>>>>>>>> >>>>>>>>>>> And you got the same error message? That's strange. >>>>>>>>>>> >>>>>>>>>>> In the mean time I try to check for clients connected. >>>>>>>>>>> No Kubernetes >>>>>>>>>>> and CephFS, but RGWs. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Malte >>>>>>>>>>> >>>>>>>>>>> On 16.10.24 14:01, Laimis Juzeliūnas wrote: >>>>>>>>>>>> Hi Malte, >>>>>>>>>>>> >>>>>>>>>>>> We have faced this recently when upgrading to Squid >>>>>>>>>>>> from latest Reef. >>>>>>>>>>>> As a temporary workaround we disabled the balancer >>>>>>>>>>>> with ‘ceph >>>>>>>>>>>> balancer off’ and restarted mgr daemons. >>>>>>>>>>>> We are suspecting older clients (from Kubernetes RBD >>>>>>>>>>>> mounts as well >>>>>>>>>>>> as CephFS mounts) on servers with incompatible client >>>>>>>>>>>> versions but >>>>>>>>>>>> are yet to dig through it. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Laimis J. >>>>>>>>>>>> >>>>>>>>>>>>> On 16 Oct 2024, at 14:57, Malte Stroem >>>>>>>>>>>>> <malte.stroem@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Error ENOENT: No orchestrator configured (try `ceph >>>>>>>>>>>>> orch set >>>>>>>>>>>>> backend`) >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io