To pull quay.io/prometheus/node-exporter:v1.5.0 the nodes would need access to the internet, yes. I don't fully understand the reason for
root@node-01:~# ceph config set mgr mgr/cephadm/container_image_node_exporter quay.io/prometheus/node-exporter:v1.5.0
though. Why not tell it to point to the image on the local repo similar to the one you passed to the upgrade command? Beyond that, it's hard to tell more without getting a snippet from the journal logs of a failed node-exporter daemon and potentially the output of `cep health detail`. I'll also point out that the node-exporter daemons won't automatically be redeployed when changing the config option for their container image. That still has to be triggered by an additional `ceph orch redeploy node-exporter` command once the config setting is correct (see https://docs.ceph.com/en/latest/cephadm/services/monitoring/#using-custom-im...). You can check the /var/lib/ceph/<FSID>/<node-exporter-daemon-name>/unit.run file on a host with a node-exporter daemon to know for sure with which image the daemon attempted to start up. On Mon, Jul 15, 2024 at 1:10 PM Saif Mohammad <samdto987@gmail.com> wrote:
Hello,
We are facing an issue with node-exporter entering an error state while upgrading our cluster in an air-gapped environment. Specifically, we are upgrading from quincyv17.2.0 to reefv18.2.2. To facilitate this upgrade, we have set up a custom repository on a separate machine within the same network and pushed required images to this private repository.
Here are the images that we have pushed:
root@custom-registry:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.1.10:5000/ceph/ceph v18.2.2 3c937764e6f5 7 weeks ago 1.25GB 192.168.1.10:5000/ceph/ceph-grafana 9.4.7 954c08fa6188 7 months ago 633MB 192.168.1.10:5000/prometheus/prometheus v2.43.0 a07b618ecd1d 16 months ago 234MB 192.168.1.10:5000/prometheus/alertmanager v0.25.0 c8568f914cd2 19 months ago 65.1MB 192.168.1.10:5000/prometheus/node-exporter v1.5.0 0da6a335fe13 19 months ago 22.5MB
Since we have configured insecure private registry, we added following lines in "/etc/containers/registries.conf" file on each node of the cluster. [[registry]] location = "192.168.1.10:5000" insecure = true
Ceph image upgradation done by the command (ceph orch upgrade start --image 192.168.1.10:5000/ceph/ceph:v18.2.2) but we encountered issues from the node-exporter image, which fails to start and remains in an error state.
For the remaining images ( except ceph ) we used "ceph-config" command. We are here now getting the new images values by executing the following command that seems to be default ones. root@node-01:~# ceph config get mgr mgr/cephadm/container_image_node_exporter quay.io/prometheus/node-exporter:v1.5.0
Does an internet connection is required to pull these images (Monitoring stack components images)? I am unsure because whenever we attempt to run the following command, node-exporter enters an error state and not coming into the running state even after redeploy. root@node-01:~# ceph config set mgr mgr/cephadm/container_image_node_exporter quay.io/prometheus/node-exporter:v1.5.0
Please advise the best approach to upgrade the cluster including all images in Air-gapped environment?
Any guidance on resolving this issue would be appreciated.
Regards, Mohammad Saif _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io