cannot reach telemetry.ceph.com -> HEALTH_WARN after cluster upgrade to 19.2.3
Hi, Does anyone know who to contact to report problems with telemetry.ceph.com? After a reasonably successful update of my cluster, I received a HEALTH_WARN [Telemetry requires re-opt-in] even though I had already re-approved telemetry. While trying to understand the problem, I stumbled across the fact that although the host telemetry.ceph.com has an IPv4 and an IPv6 address, I cannot reach it because IPv6 only has a dropping firewall or something similar attached to it. Now, I can configure a proxy that also reaches the legacy Internet, but it also tried to reach IPv6 first, so that currently no IPv6-enabled Ceph cluster should be visible in the telemetry. My workaround would now be to overlay telemetry.ceph.com on the DNS cache... but that's just a hack. ----- raw shell output for reproduction -----
host telemetry.ceph.com telemetry.ceph.com has address 8.43.84.137 telemetry.ceph.com has IPv6 address 2620:52:3:3:201a:4aff:feab:8409
telnet 2620:52:3:3:201a:4aff:feab:8409 80 Trying 2620:52:3:3:201a:4aff:feab:8409... telnet: Unable to connect to remote host: Connection timed out
telnet 8.43.84.137 80 Trying 8.43.84.137... Connected to 8.43.84.137. Escape character is '^]'. get / HTTP/1.0
HTTP/1.1 302 Found Location: https://telemetry.ceph.com/
Hi Björn, Thanks for reporting this. From which version did you upgrade to 19.2.3? Just double-checking -- did you run `ceph telemetry on` (along with `--license sharing-1-0`) *after* the upgrade? In the meantime, you can mute the health warning. We're looking into the IPv6 issue — thanks for sharing your insights. Thanks, Yaarit On Tue, Aug 26, 2025 at 3:58 AM Björn Lässig <b.laessig@pengutronix.de> wrote:
Hi,
Does anyone know who to contact to report problems with telemetry.ceph.com?
After a reasonably successful update of my cluster, I received a HEALTH_WARN [Telemetry requires re-opt-in] even though I had already re-approved telemetry.
While trying to understand the problem, I stumbled across the fact that although the host telemetry.ceph.com has an IPv4 and an IPv6 address, I cannot reach it because IPv6 only has a dropping firewall or something similar attached to it.
Now, I can configure a proxy that also reaches the legacy Internet, but it also tried to reach IPv6 first, so that currently no IPv6-enabled Ceph cluster should be visible in the telemetry.
My workaround would now be to overlay telemetry.ceph.com on the DNS cache... but that's just a hack.
----- raw shell output for reproduction -----
host telemetry.ceph.com telemetry.ceph.com has address 8.43.84.137 telemetry.ceph.com has IPv6 address 2620:52:3:3:201a:4aff:feab:8409
telnet 2620:52:3:3:201a:4aff:feab:8409 80 Trying 2620:52:3:3:201a:4aff:feab:8409... telnet: Unable to connect to remote host: Connection timed out
telnet 8.43.84.137 80 Trying 8.43.84.137... Connected to 8.43.84.137. Escape character is '^]'. get / HTTP/1.0
HTTP/1.1 302 Found Location: https://telemetry.ceph.com/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Yaarit, On Tue, 2025-08-26 at 12:05 -0400, Yaarit Hatuka wrote:
Thanks for reporting this.
Thank you very much for your reply. The Problem is on my site and i would not have found it, if you did not asked this question:
From which version did you upgrade to 19.2.3?
The version before was 18.2.7. It is an cephadm managed cluster. Upgrade worked like a charm. (Except for some little Problems that have workarounds and are tracked somewhere else). And long before that it was version 17.2.X
Just double-checking -- did you run `ceph telemetry on` (along with `--license sharing-1-0`) after the upgrade?
After the upgrade i did this as it was documented in the upgrade procedure. But maybe it was at the same time, as the Warning appeared. Some time later it went off and everything was nice. (but it never send telemetry)
We're looking into the IPv6 issue — thanks for sharing your insights.
On a 3rd look its totally me. The Telemetry endpoints are on port https/443 and i checked http/80. And these are reachable now. But they were not some years ago. So i configured a proxy years ago. Later I reinstalled the proxy and changed some things … and forgot to tell the ceph-cluster. So my ceph-cluster had an proxy, that did not worked. I just removed the proxy setting with ceph config rm mgr mgr/telemetry/proxy and it should work now. I'm sorry, and I'm grateful. Björn Lässig
participants (2)
-
Björn Lässig
-
Yaarit Hatuka