On Tuesday, March 26, 2024 7:22:18 AM EDT Iain Stott wrote:
Hi,
We are trying to deploy Ceph Reef 18.2.1 using cephadm on mixed architecture hosts using x86_64 for the mons and aarch64 for the OSDs.
During deployment we use the following config for the bootstrap process, where $REPOSITORY is our docker repo.
[global] container_image = $REPOSITORY/ceph/ceph:v18.2.1 [mgr] mgr/cephadm/container_image_base = $REPOSITORY/ceph/ceph:v18.2.1 mgr/cephadm/container_image_prometheus = $REPOSITORY/ceph/prometheus:v2.33.4 mgr/cephadm/container_image_node_exporter = $REPOSITORY/ceph/node-exporter:v1.3.1 mgr/cephadm/container_image_grafana = $REPOSITORY/ceph/ceph-grafana:8.3.5 mgr/cephadm/container_image_alertmanager = $REPOSITORY/ceph/alertmanager:v0.23.0 [osd] container_image = $REPOSITORY/ceph/ceph:v18.2.1 Once the bootstrap process is complete, if we do a ceph config dump, the container image for global and osd changes from version tag to sha reference, meaning that when deploying the containers on the OSDs they try using the amd64 container image and not the aarch64 image and fail deployment.
Is there a config setting we are missing or a workaround for this?
Try: `ceph config set mgr mgr/cephadm/use_repo_digest false` This comes up often enough that we should document it. I don't think that option is documented right now because I searched for the option with google and all that came up were other tracker issues and older mailing-list posts. In the longer term we may want to make cephadm arch-aware (volunteers welcome :-) ).