Ceph orch doesn't execute commands and doesn't report correct status of daemons
Hi, I have finished the conversion from ceph-ansible to cephadm yesterday. Everything seemed to be working until this morning, I wanted to redeploy rgw service to specify the network to be used. So I deleted the rgw services with ceph orch rm, then I prepared a yml file with the new conf. I applied the file and the new rgw service was started but it was launched with an external image, so I wanted to redeploy using my local image so I did a redeploy ... and then nothing happened, I get the rescheduled message but nothing happened, then I restarted one of the controllers, the orchestrator doesn't seem to be aware that some service have restarted??? PS : I don't fully master the cephadm command line and use. Regards.
There have been bugs in the past where things have gotten "stuck". Usually I'd say check the REFRESHED column in the output of `ceph orch ps`. It should refresh the daemons on each host roughly every 10 minutes, so if you see some value much larger than that, things are probably actually stuck. If they are, the first thing to try is usually a mgr failover (`ceph mgr fail`). Are you using a special RGW image different from the other ceph daemons in the cluster? The typical case is that all the ceph daemons use the same image and then get updated using the `ceph orch upgrade start ...` command. On Fri, Mar 1, 2024 at 4:26 AM wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I have finished the conversion from ceph-ansible to cephadm yesterday. Everything seemed to be working until this morning, I wanted to redeploy rgw service to specify the network to be used.
So I deleted the rgw services with ceph orch rm, then I prepared a yml file with the new conf. I applied the file and the new rgw service was started but it was launched with an external image, so I wanted to redeploy using my local image so I did a redeploy ... and then nothing happened, I get the rescheduled message but nothing happened, then I restarted one of the controllers, the orchestrator doesn't seem to be aware that some service have restarted???
PS : I don't fully master the cephadm command line and use.
Regards. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I'll try the 'ceph mgr fail' and report back. In the meantime, my problem with the images... I am trying to use my local registry to deploy the different services. I don't know how to use the 'apply' and force my cluster to use my local registry. So basically, what I am doing so far is : 1 - ceph orch apply -i rgw-service.yml <---------------- deploy the rgw, and this will pull the image from the internet 2 - ceph orch daemon redeploy rgw.opsrgw.controllera.gtrttj --image 192.168.2.36:4000/ceph/ceph:v17 <------------ Redeploy the demons of that service with my local image. How May I deploy directly from my local registry? Regards.
Okay, it seems like from what you're saying the RGW image itself isn't special compared to the other ceph daemons, it's just that you want to use the image on your local registry. In that case, I would still recommend just using `ceph orch upgrade start --image <image-name>` with the image from your local registry. It will transition all the ceph daemons rather than just RGW to that image, but that's generally how cephadm expects things to be anyway. Assuming that registry is reachable from all the nodes on the cluster, the upgrade should be able to do it. Side note, but I have seen some people in the past have issues with cephadm's use of repo digests when using local registries, so if you're having issues you may want to try setting the mgr/cephadm/use_repo_digest option to false. Just keep in mind that means if you want to upgrade to another image, you'd have to make sure it has a different name (usage of repo digest was added in to support floating tags). On Fri, Mar 1, 2024 at 11:47 AM wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I'll try the 'ceph mgr fail' and report back.
In the meantime, my problem with the images... I am trying to use my local registry to deploy the different services. I don't know how to use the 'apply' and force my cluster to use my local registry. So basically, what I am doing so far is : 1 - ceph orch apply -i rgw-service.yml <---------------- deploy the rgw, and this will pull the image from the internet 2 - ceph orch daemon redeploy rgw.opsrgw.controllera.gtrttj --image 192.168.2.36:4000/ceph/ceph:v17 <------------ Redeploy the demons of that service with my local image.
How May I deploy directly from my local registry?
Regards.
participants (2)
-
Adam King
-
wodel youchi