Hi everyone I just change the subnet of my cluster. The cephfs part seem to working well. But I got many error with Jul 11 10:08:35 hostname ceph-*************** ts=2024-07-11T08:08:35.364Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://OLD_IP:8765/sd/prometheus/sd-config?service=alertmanager\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" I didn't find which service does the discovery and was unable to change the URL. I try a ceph config-key dump |grep OLD_IP and didn't find it. So where this information are store ? Regards JAS -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 11 juil. 2024 10:22:58 CEST
Le 11/07/2024 à 10:27:09+0200, Albert Shih a écrit
Hi everyone
I just change the subnet of my cluster.
The cephfs part seem to working well.
But I got many error with
Jul 11 10:08:35 hostname ceph-*************** ts=2024-07-11T08:08:35.364Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://OLD_IP:8765/sd/prometheus/sd-config?service=alertmanager\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
I didn't find which service does the discovery and was unable to change the URL.
I try a
ceph config-key dump |grep OLD_IP
and didn't find it.
So where this information are store ?
Forget to say I'm running reef 18.2.2 Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 11 juil. 2024 10:30:41 CEST
Do you see it in 'ceph mgr services'? You might need to change the prometheus config as well and redeploy. Zitat von Albert Shih <Albert.Shih@obspm.fr>:
Hi everyone
I just change the subnet of my cluster.
The cephfs part seem to working well.
But I got many error with
Jul 11 10:08:35 hostname ceph-*************** ts=2024-07-11T08:08:35.364Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://OLD_IP:8765/sd/prometheus/sd-config?service=alertmanager\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
I didn't find which service does the discovery and was unable to change the URL.
I try a
ceph config-key dump |grep OLD_IP
and didn't find it.
So where this information are store ?
Regards
JAS -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 11 juil. 2024 10:22:58 CEST _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 11/07/2024 à 08:34:21+0000, Eugen Block a écrit Hi, Sorry I miss the answer to the list.
Do you see it in 'ceph mgr services'? You might need to change the
Yes I did root@cthulhu1:/etc# ceph mgr services { "dashboard": "https://NEW_SUBNET.189.35:8443/", "prometheus": "http://NEW_SUBNET.189.35:9283/" } root@cthulhu1:/etc#
prometheus config as well and redeploy.
I'm not sure, but I don't think the problem is on the «server side» (server = prometheus) because I got every daemon listen on every IP root@cthulhu1:/etc# ceph orch ps|grep pro prometheus.cthulhu1 cthulhu1 *:9095 running (22h) 9m ago 5M 58.8M - 2.43.0 a07b618ecd1d 77abd8ebe0c2 prometheus.cthulhu2 cthulhu2 *:9095 running (22h) 9m ago 5M 55.6M - 2.43.0 a07b618ecd1d a1346925807a prometheus.cthulhu3 cthulhu3 *:9095 running (22h) 9m ago 5M 61.5M - 2.43.0 a07b618ecd1d 2fb558b9d300 prometheus.cthulhu4 cthulhu4 *:9095 running (22h) 9m ago 5M 56.3M - 2.43.0 a07b618ecd1d 119ada9ba6de prometheus.cthulhu5 cthulhu5 *:9095 running (22h) 9m ago 5M 60.6M - 2.43.0 a07b618ecd1d c355ed607fae If I check with ss same result. I'm guessing is on the client side somewhere they are the old IP address...but I can't find where and of course I'm unable to change it. Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 11 juil. 2024 10:51:25 CEST
And how about the prometheus.yml? /var/lib/ceph/{fsid}/prometheus.{node}/etc/prometheus/prometheus.yml It contains an IP address as well: alerting: alertmanagers: - scheme: http http_sd_configs: - url: http://{IP}:8765/sd/prometheus/sd-config?service=alertmanager I misread the line, maybe you need to update alertmanager instead of prometheus. Zitat von Albert Shih <Albert.Shih@obspm.fr>:
Le 11/07/2024 à 08:34:21+0000, Eugen Block a écrit
Hi,
Sorry I miss the answer to the list.
Do you see it in 'ceph mgr services'? You might need to change the
Yes I did
root@cthulhu1:/etc# ceph mgr services { "dashboard": "https://NEW_SUBNET.189.35:8443/", "prometheus": "http://NEW_SUBNET.189.35:9283/" } root@cthulhu1:/etc#
prometheus config as well and redeploy.
I'm not sure, but I don't think the problem is on the «server side» (server = prometheus) because I got every daemon listen on every IP
root@cthulhu1:/etc# ceph orch ps|grep pro prometheus.cthulhu1 cthulhu1 *:9095 running (22h) 9m ago 5M 58.8M - 2.43.0 a07b618ecd1d 77abd8ebe0c2 prometheus.cthulhu2 cthulhu2 *:9095 running (22h) 9m ago 5M 55.6M - 2.43.0 a07b618ecd1d a1346925807a prometheus.cthulhu3 cthulhu3 *:9095 running (22h) 9m ago 5M 61.5M - 2.43.0 a07b618ecd1d 2fb558b9d300 prometheus.cthulhu4 cthulhu4 *:9095 running (22h) 9m ago 5M 56.3M - 2.43.0 a07b618ecd1d 119ada9ba6de prometheus.cthulhu5 cthulhu5 *:9095 running (22h) 9m ago 5M 60.6M - 2.43.0 a07b618ecd1d c355ed607fae
If I check with ss same result.
I'm guessing is on the client side somewhere they are the old IP address...but I can't find where and of course I'm unable to change it.
Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 11 juil. 2024 10:51:25 CEST
Le 11/07/2024 à 09:00:02+0000, Eugen Block a écrit Hi, Thanks, but ... nope.
And how about the prometheus.yml?
/var/lib/ceph/{fsid}/prometheus.{node}/etc/prometheus/prometheus.yml
It contains an IP address as well:
alerting: alertmanagers: - scheme: http http_sd_configs: - url: http://{IP}:8765/sd/prometheus/sd-config?service=alertmanager
nothing....even a cd /var/lib/ceph rg -l OLD_SUBNET got nothing. But no worries...win98 method (reboot the cluster) solve the issue. I'm guessing something/some service need a restart.
I misread the line, maybe you need to update alertmanager instead of prometheus.
Nope either Thanks. Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 11 juil. 2024 11:23:38 CEST
participants (2)
-
Albert Shih
-
Eugen Block