Cephadm adoption - service reconfiguration changes container image
Hi Everyone, We are looking at migrating all our production clusters from ceph-ansible to cephadm. We are currently experiencing an issue where when reconfiguring a service through ceph orch, it will change the running container image for that service which has led to the mgr services running an earlier version than the rest of the cluster, this has caused cephadm/ceph orch to not be able to manage services in the cluster. Does anyone have any help with this as I cannot find anything in the docs that would correct this. Cheers Iain [root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm --image quay.io/ceph/ceph:v16.2.9 adopt --style legacy --name mon.$(hostname -s) [root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml service_type: mon service_name: mon placement: host_pattern: '*mon*' extra_container_args: - -v - /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.client.admin.keyring [root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm shell -- ceph config get mgr mgr/cephadm/container_image_base quay.io/ceph/ceph [root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon 55fe8bc10476 quay.io/ceph/ceph:v16.2.9 -n mgr.de1-ceph-m... 5 minutes ago Up 5 minutes ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mgr-de1-ceph-mon-ceph-site-a-1 [root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml | cephadm --image quay.io/ceph/ceph:v16.2.9 shell -- ceph orch apply -i - Inferring fsid da9e9837-a3cf-4482-9a13-790a721598cd Scheduled mon update... [root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon ecec1d62c719 docker.io/ceph/daemon-base:latest-pacific-devel -n mon.de1-ceph-m... 25 seconds ago Up 26 seconds ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mon-de1-ceph-mon-ceph-site-a-1 [root@de1-ceph-mon-ceph-site-a-2 ~]# ceph versions { "mon": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "osd": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 8 }, "mds": {}, "rgw": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "overall": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3, "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 14 } } Iain Stott OpenStack Engineer Iain.Stott@thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>
you could maybe try running "ceph config set global container quay.io/ceph/ceph:v16.2.9" before running the adoption. It seems it still thinks it should be deploying mons with the default image ( docker.io/ceph/daemon-base:latest-pacific-devel ) for some reason and maybe that config option is why. On Tue, Aug 15, 2023 at 7:29 AM Iain Stott <Iain.Stott@thehutgroup.com> wrote:
Hi Everyone,
We are looking at migrating all our production clusters from ceph-ansible to cephadm. We are currently experiencing an issue where when reconfiguring a service through ceph orch, it will change the running container image for that service which has led to the mgr services running an earlier version than the rest of the cluster, this has caused cephadm/ceph orch to not be able to manage services in the cluster.
Does anyone have any help with this as I cannot find anything in the docs that would correct this.
Cheers Iain
[root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm --image quay.io/ceph/ceph:v16.2.9 adopt --style legacy --name mon.$(hostname -s)
[root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml service_type: mon service_name: mon placement: host_pattern: '*mon*' extra_container_args: - -v - /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.client.admin.keyring
[root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm shell -- ceph config get mgr mgr/cephadm/container_image_base quay.io/ceph/ceph
[root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon 55fe8bc10476 quay.io/ceph/ceph:v16.2.9 -n mgr.de1-ceph-m... 5 minutes ago Up 5 minutes ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mgr-de1-ceph-mon-ceph-site-a-1
[root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml | cephadm --image quay.io/ceph/ceph:v16.2.9 shell -- ceph orch apply -i - Inferring fsid da9e9837-a3cf-4482-9a13-790a721598cd Scheduled mon update...
[root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon ecec1d62c719 docker.io/ceph/daemon-base:latest-pacific-devel -n mon.de1-ceph-m... 25 seconds ago Up 26 seconds ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mon-de1-ceph-mon-ceph-site-a-1
[root@de1-ceph-mon-ceph-site-a-2 ~]# ceph versions { "mon": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "osd": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 8 }, "mds": {}, "rgw": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "overall": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3, "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 14 } }
Iain Stott OpenStack Engineer Iain.Stott@thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<; https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [ https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Adam, Will give it a try today. Cheers ________________________________ From: Adam King <adking@redhat.com> Sent: 15 August 2023 15:40 To: Iain Stott <Iain.Stott@thehutgroup.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Cephadm adoption - service reconfiguration changes container image CAUTION: This email originates from outside THG ________________________________ you could maybe try running "ceph config set global container quay.io/ceph/ceph:v16.2.9<http://quay.io/ceph/ceph:v16.2.9>" before running the adoption. It seems it still thinks it should be deploying mons with the default image (docker.io/ceph/daemon-base:latest-pacific-devel<http://docker.io/ceph/daemon-base:latest-pacific-devel> ) for some reason and maybe that config option is why. On Tue, Aug 15, 2023 at 7:29 AM Iain Stott <Iain.Stott@thehutgroup.com<mailto:Iain.Stott@thehutgroup.com>> wrote: Hi Everyone, We are looking at migrating all our production clusters from ceph-ansible to cephadm. We are currently experiencing an issue where when reconfiguring a service through ceph orch, it will change the running container image for that service which has led to the mgr services running an earlier version than the rest of the cluster, this has caused cephadm/ceph orch to not be able to manage services in the cluster. Does anyone have any help with this as I cannot find anything in the docs that would correct this. Cheers Iain [root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm --image quay.io/ceph/ceph:v16.2.9<http://quay.io/ceph/ceph:v16.2.9> adopt --style legacy --name mon.$(hostname -s) [root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml service_type: mon service_name: mon placement: host_pattern: '*mon*' extra_container_args: - -v - /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.client.admin.keyring [root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm shell -- ceph config get mgr mgr/cephadm/container_image_base quay.io/ceph/ceph<http://quay.io/ceph/ceph> [root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon 55fe8bc10476 quay.io/ceph/ceph:v16.2.9<http://quay.io/ceph/ceph:v16.2.9> -n mgr.de1-ceph-m... 5 minutes ago Up 5 minutes ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mgr-de1-ceph-mon-ceph-site-a-1 [root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml | cephadm --image quay.io/ceph/ceph:v16.2.9<http://quay.io/ceph/ceph:v16.2.9> shell -- ceph orch apply -i - Inferring fsid da9e9837-a3cf-4482-9a13-790a721598cd Scheduled mon update... [root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon ecec1d62c719 docker.io/ceph/daemon-base:latest-pacific-devel<http://docker.io/ceph/daemon-base:latest-pacific-devel> -n mon.de1-ceph-m... 25 seconds ago Up 26 seconds ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mon-de1-ceph-mon-ceph-site-a-1 [root@de1-ceph-mon-ceph-site-a-2 ~]# ceph versions { "mon": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "osd": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 8 }, "mds": {}, "rgw": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "overall": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3, "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 14 } } Iain Stott OpenStack Engineer Iain.Stott@thehutgroup.com<mailto:Iain.Stott@thehutgroup.com> [THG Ingenuity Logo]<https://www.thg.com<https://www.thg.com>> [https://i.imgur.com/wbpVRW6.png<https://i.imgur.com/wbpVRW6.png>]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk>> [https://i.imgur.com/c3040tr.png<https://i.imgur.com/c3040tr.png>] <https://twitter.com/thgplc?lang=en<https://twitter.com/thgplc?lang=en>> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
That would have been my suggestion as well, set your own container image and override the default. Just one comment, the config option is "container_image" and not "container", that one fails: $ ceph config set global container my-registry:5000/ceph/ceph:16.2.9 Error EINVAL: unrecognized config option 'container' Instead run: $ ceph config set global container_image my-registry:5000/ceph/ceph:16.2.9 Zitat von Adam King <adking@redhat.com>:
you could maybe try running "ceph config set global container quay.io/ceph/ceph:v16.2.9" before running the adoption. It seems it still thinks it should be deploying mons with the default image ( docker.io/ceph/daemon-base:latest-pacific-devel ) for some reason and maybe that config option is why.
On Tue, Aug 15, 2023 at 7:29 AM Iain Stott <Iain.Stott@thehutgroup.com> wrote:
Hi Everyone,
We are looking at migrating all our production clusters from ceph-ansible to cephadm. We are currently experiencing an issue where when reconfiguring a service through ceph orch, it will change the running container image for that service which has led to the mgr services running an earlier version than the rest of the cluster, this has caused cephadm/ceph orch to not be able to manage services in the cluster.
Does anyone have any help with this as I cannot find anything in the docs that would correct this.
Cheers Iain
[root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm --image quay.io/ceph/ceph:v16.2.9 adopt --style legacy --name mon.$(hostname -s)
[root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml service_type: mon service_name: mon placement: host_pattern: '*mon*' extra_container_args: - -v - /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.client.admin.keyring
[root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm shell -- ceph config get mgr mgr/cephadm/container_image_base quay.io/ceph/ceph
[root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon 55fe8bc10476 quay.io/ceph/ceph:v16.2.9 -n mgr.de1-ceph-m... 5 minutes ago Up 5 minutes ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mgr-de1-ceph-mon-ceph-site-a-1
[root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml | cephadm --image quay.io/ceph/ceph:v16.2.9 shell -- ceph orch apply -i - Inferring fsid da9e9837-a3cf-4482-9a13-790a721598cd Scheduled mon update...
[root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon ecec1d62c719 docker.io/ceph/daemon-base:latest-pacific-devel -n mon.de1-ceph-m... 25 seconds ago Up 26 seconds ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mon-de1-ceph-mon-ceph-site-a-1
[root@de1-ceph-mon-ceph-site-a-2 ~]# ceph versions { "mon": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "osd": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 8 }, "mds": {}, "rgw": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "overall": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3, "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 14 } }
Iain Stott OpenStack Engineer Iain.Stott@thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<; https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [ https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yeah this seems to have done the trick. I still need to complete the full cluster adoption, but after mon and mgr reconfigure they have come back up and built from the correct image. Thanks for this.
participants (4)
-
Adam King
-
Eugen Block
-
Iain Stott
-
Iain Stott