Cephadm on mixed architecture hosts
Hi, We are trying to deploy Ceph Reef 18.2.1 using cephadm on mixed architecture hosts using x86_64 for the mons and aarch64 for the OSDs. During deployment we use the following config for the bootstrap process, where $REPOSITORY is our docker repo. [global] container_image = $REPOSITORY/ceph/ceph:v18.2.1 [mgr] mgr/cephadm/container_image_base = $REPOSITORY/ceph/ceph:v18.2.1 mgr/cephadm/container_image_prometheus = $REPOSITORY/ceph/prometheus:v2.33.4 mgr/cephadm/container_image_node_exporter = $REPOSITORY/ceph/node-exporter:v1.3.1 mgr/cephadm/container_image_grafana = $REPOSITORY/ceph/ceph-grafana:8.3.5 mgr/cephadm/container_image_alertmanager = $REPOSITORY/ceph/alertmanager:v0.23.0 [osd] container_image = $REPOSITORY/ceph/ceph:v18.2.1 Once the bootstrap process is complete, if we do a ceph config dump, the container image for global and osd changes from version tag to sha reference, meaning that when deploying the containers on the OSDs they try using the amd64 container image and not the aarch64 image and fail deployment. Is there a config setting we are missing or a workaround for this? Thanks Iain Iain Stott OpenStack Engineer Iain.Stott@thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>
Iain - I’ve seen this same behavior. I’ve not found a work-around, though would agree that it would be a “nice to have” feature.
On Mar 26, 2024, at 7:22 AM, Iain Stott <Iain.Stott@thg.com> wrote:
Hi,
We are trying to deploy Ceph Reef 18.2.1 using cephadm on mixed architecture hosts using x86_64 for the mons and aarch64 for the OSDs.
During deployment we use the following config for the bootstrap process, where $REPOSITORY is our docker repo.
[global] container_image = $REPOSITORY/ceph/ceph:v18.2.1 [mgr] mgr/cephadm/container_image_base = $REPOSITORY/ceph/ceph:v18.2.1 mgr/cephadm/container_image_prometheus = $REPOSITORY/ceph/prometheus:v2.33.4 mgr/cephadm/container_image_node_exporter = $REPOSITORY/ceph/node-exporter:v1.3.1 mgr/cephadm/container_image_grafana = $REPOSITORY/ceph/ceph-grafana:8.3.5 mgr/cephadm/container_image_alertmanager = $REPOSITORY/ceph/alertmanager:v0.23.0 [osd] container_image = $REPOSITORY/ceph/ceph:v18.2.1 Once the bootstrap process is complete, if we do a ceph config dump, the container image for global and osd changes from version tag to sha reference, meaning that when deploying the containers on the OSDs they try using the amd64 container image and not the aarch64 image and fail deployment.
Is there a config setting we are missing or a workaround for this?
Thanks Iain
Iain Stott OpenStack Engineer Iain.Stott@thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tuesday, March 26, 2024 7:22:18 AM EDT Iain Stott wrote:
Hi,
We are trying to deploy Ceph Reef 18.2.1 using cephadm on mixed architecture hosts using x86_64 for the mons and aarch64 for the OSDs.
During deployment we use the following config for the bootstrap process, where $REPOSITORY is our docker repo.
[global] container_image = $REPOSITORY/ceph/ceph:v18.2.1 [mgr] mgr/cephadm/container_image_base = $REPOSITORY/ceph/ceph:v18.2.1 mgr/cephadm/container_image_prometheus = $REPOSITORY/ceph/prometheus:v2.33.4 mgr/cephadm/container_image_node_exporter = $REPOSITORY/ceph/node-exporter:v1.3.1 mgr/cephadm/container_image_grafana = $REPOSITORY/ceph/ceph-grafana:8.3.5 mgr/cephadm/container_image_alertmanager = $REPOSITORY/ceph/alertmanager:v0.23.0 [osd] container_image = $REPOSITORY/ceph/ceph:v18.2.1 Once the bootstrap process is complete, if we do a ceph config dump, the container image for global and osd changes from version tag to sha reference, meaning that when deploying the containers on the OSDs they try using the amd64 container image and not the aarch64 image and fail deployment.
Is there a config setting we are missing or a workaround for this?
Try: `ceph config set mgr mgr/cephadm/use_repo_digest false` This comes up often enough that we should document it. I don't think that option is documented right now because I searched for the option with google and all that came up were other tracker issues and older mailing-list posts. In the longer term we may want to make cephadm arch-aware (volunteers welcome :-) ).
Ooooo thanks John, will give it a try and report back ________________________________ From: John Mulligan <phlogistonjohn@asynchrono.us> Sent: 26 March 2024 14:24 To: ceph-users@ceph.io <ceph-users@ceph.io> Cc: Iain Stott <Iain.Stott@thg.com> Subject: Re: [ceph-users] Cephadm on mixed architecture hosts CAUTION: This email originates from outside THG On Tuesday, March 26, 2024 7:22:18 AM EDT Iain Stott wrote:
Hi,
We are trying to deploy Ceph Reef 18.2.1 using cephadm on mixed architecture hosts using x86_64 for the mons and aarch64 for the OSDs.
During deployment we use the following config for the bootstrap process, where $REPOSITORY is our docker repo.
[global] container_image = $REPOSITORY/ceph/ceph:v18.2.1 [mgr] mgr/cephadm/container_image_base = $REPOSITORY/ceph/ceph:v18.2.1 mgr/cephadm/container_image_prometheus = $REPOSITORY/ceph/prometheus:v2.33.4 mgr/cephadm/container_image_node_exporter = $REPOSITORY/ceph/node-exporter:v1.3.1 mgr/cephadm/container_image_grafana = $REPOSITORY/ceph/ceph-grafana:8.3.5 mgr/cephadm/container_image_alertmanager = $REPOSITORY/ceph/alertmanager:v0.23.0 [osd] container_image = $REPOSITORY/ceph/ceph:v18.2.1 Once the bootstrap process is complete, if we do a ceph config dump, the container image for global and osd changes from version tag to sha reference, meaning that when deploying the containers on the OSDs they try using the amd64 container image and not the aarch64 image and fail deployment.
Is there a config setting we are missing or a workaround for this?
Try: `ceph config set mgr mgr/cephadm/use_repo_digest false` This comes up often enough that we should document it. I don't think that option is documented right now because I searched for the option with google and all that came up were other tracker issues and older mailing-list posts. In the longer term we may want to make cephadm arch-aware (volunteers welcome :-) ).
participants (3)
-
Daniel Brown
-
Iain Stott
-
John Mulligan