Don't know if it helps, but we have also experienced something similar
with osd images. We changed the image tag from version to sha and it did
not happen again.
___________________________________
ceph ambassador DACH
ceph consultant since 2012
Clyso GmbH - Premier Ceph Foundation Member
https://www.clyso.com/
Am 15.05.23 um 14:50 schrieb Adam King:
> I think with the `config set` commands there is logic to notify the
> relevant mgr modules and update their values. That might not exist with
> `config rm`, so it's still using the last set value. Looks like a real bug.
> Curious what happens if the mgr restarts after the `config rm`. Whether it
> goes back to the default image in that case or not. Might take a look later.
>
> On Mon, May 15, 2023 at 7:37 AM Daniel Krambrock <
> krambrock(a)hrz.uni-marburg.de> wrote:
>
>> Hello.
>>
>> I think i found a bug in cephadm/ceph orch:
>> Redeploying a container image (tested with alertmanager) after removing
>> a custom `mgr/cephadm/container_image_alertmanager` value, deploys the
>> previous container image and not the default container image.
>>
>> I'm running `cephadm` from ubuntu 22.04 pkg 17.2.5-0ubuntu0.22.04.3 and
>> `ceph` version 17.2.6.
>>
>> Here is an example. Node clrz20-08 is the node altermanager is running
>> on, clrz20-01 the node I'm controlling ceph from:
>>
>> * Get alertmanager version
>> ```
>> root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name ==
>> "alertmanager")| .container_image_name'
>> "quay.io/prometheus/alertmanager:v0.23.0"
>> ```
>>
>> * Set alertmanager image
>> ```
>> root@clrz20-01:~# ceph config set mgr
>> mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager
>> root@clrz20-01:~# ceph config get mgr
>> mgr/cephadm/container_image_alertmanager
>> quay.io/prometheus/alertmanager
>> ```
>>
>> * redeploy altermanager
>> ```
>> root@clrz20-01:~# ceph orch redeploy alertmanager
>> Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08'
>> ```
>>
>> * Get alertmanager version
>> ```
>> root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name ==
>> "alertmanager")| .container_image_name'
>> "quay.io/prometheus/alertmanager:latest"
>> ```
>>
>> * Remove alertmanager image setting, revert to default:
>> ```
>> root@clrz20-01:~# ceph config rm mgr
>> mgr/cephadm/container_image_alertmanager
>> root@clrz20-01:~# ceph config get mgr
>> mgr/cephadm/container_image_alertmanager
>> quay.io/prometheus/alertmanager:v0.23.0
>> ```
>>
>> * redeploy altermanager
>> ```
>> root@clrz20-01:~# ceph orch redeploy alertmanager
>> Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08'
>> ```
>>
>> * Get alertmanager version
>> ```
>> root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name ==
>> "alertmanager")| .container_image_name'
>> "quay.io/prometheus/alertmanager:latest"
>> ```
>> -> `mgr/cephadm/container_image_alertmanager` is set to
>> `quay.io/prometheus/alertmanager:v0.23.0`
>> <http://quay.io/prometheus/alertmanager:v0.23.0>, but redeploy uses
>> `quay.io/prometheus/alertmanager:latest`
>> <http://quay.io/prometheus/alertmanager:latest>. This looks like a bug.
>>
>> * Set alertmanager image explicitly to the default value
>> ```
>> root@clrz20-01:~# ceph config set mgr
>> mgr/cephadm/container_image_alertmanager
>> quay.io/prometheus/alertmanager:v0.23.0
>> root@clrz20-01:~# ceph config get mgr
>> mgr/cephadm/container_image_alertmanager
>> quay.io/prometheus/alertmanager:v0.23.0
>> ```
>>
>> * redeploy altermanager
>> ```
>> root@clrz20-01:~# ceph orch redeploy alertmanager
>> Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08'
>> ```
>>
>> * Get alertmanager version
>> ```
>> root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name ==
>> "alertmanager")| .container_image_name'
>> "quay.io/prometheus/alertmanager:v0.23.0"
>> ```
>> -> Setting `mgr/cephadm/container_image_alertmanager` to the default
>> setting fixes the issue.
>>
>>
>>
>> Bests,
>> Daniel
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io