Re: Grafana dashboards warnings after upgrade to Squid 19.2.4
[ I've left off dev@ from this reply, as it doesn't seem necessarily like a dev issue ] Is it possible that your cluster is overriding the default container images, assuming that you're running containerized with cephadm? root@squid1:~# ceph config dump | grep container_image container_image XX mgr/cephadm/container_image_alertmanager YY mgr/cephadm/container_image_base ZZ mgr/cephadm/container_image_elasticsearch QQ mgr/cephadm/container_image_grafana .. mgr/cephadm/container_image_haproxy .. mgr/cephadm/container_image_jaeger_agent .. mgr/cephadm/container_image_jaeger_collector .. mgr/cephadm/container_image_jaeger_query .. mgr/cephadm/container_image_keepalived .. mgr/cephadm/container_image_loki .. mgr/cephadm/container_image_node_exporter .. mgr/cephadm/container_image_nvmeof .. mgr/cephadm/container_image_prometheus .. mgr/cephadm/container_image_promtail .. mgr/cephadm/container_image_samba .. mgr/cephadm/container_image_snmp_gateway ..
On Jul 1, 2026, at 6:53 AM, Sake via ceph-users <ceph-users@ceph.io> wrote:
Probably something not going correctly, because I just upgraded to 19.2.4 and Grafana is using the grafana image (in the past this was ceph-grafana) version 12.3.1.
On 1 July 2026 11:13:02 CEST, Eugen Block via ceph-users <ceph-users@ceph.io> wrote:
Hi,
I just upgraded our production cluster to 19.2.4 (from 19.2.3), there were no issues reported by our users, so all good. But I noticed that many graphs now show different warnings or strange labels:
- "This panel requires Angular (deprecated)." on multiple graphs - "PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "ceph_pool_wr" (1:6)" also on multiple graphs for different metrics - "OSD Capacity Utilization" just shows: "RangeError: invalid array length" - The "Pool Overview" graph has column names like "Value #F", "Value #G" and so on. And the values within don't make sense either.
I have uploaded screenshots:
https://www.pasteboard.co/A34Zcy9cqHe5.png https://www.pasteboard.co/F_vkpeBjGGWH.png https://www.pasteboard.co/mkCtlBN3zS6v.png https://www.pasteboard.co/T_ByD_gCpslA.png
I compared the prometheus and grafana versions we have currently running with a fresh 19.2.4 bootstrapped cluster and the versions are identical:
ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' 10.4.0
ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' 2.51.0
Is there anything I'm missing?
Thanks! Eugen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Huh, that's kind of interesting... So yes, we do have a container_image override in our production cluster because of a private container registry, but not in that test cluster. I wiped that single host test cluster completely (cephadm rm-cluster) and bootstrapped a fresh one. That's where I previously got that grafana version from. Now I did that again, only this time I also pruned all stored images on that host before bootstrapping again. And now I get a different grafana version: root@ceph0-2:~# ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' 12.3.1 That is unexpected (to me) but good to know not to rely on that. I'll update our grafana image as well, let's see if those issues are resolved. Thanks! Eugen Zitat von Anthony D'Atri <anthony.datri@gmail.com>:
[ I've left off dev@ from this reply, as it doesn't seem necessarily like a dev issue ]
Is it possible that your cluster is overriding the default container images, assuming that you're running containerized with cephadm?
root@squid1:~# ceph config dump | grep container_image container_image XX mgr/cephadm/container_image_alertmanager YY mgr/cephadm/container_image_base ZZ mgr/cephadm/container_image_elasticsearch QQ mgr/cephadm/container_image_grafana .. mgr/cephadm/container_image_haproxy .. mgr/cephadm/container_image_jaeger_agent .. mgr/cephadm/container_image_jaeger_collector .. mgr/cephadm/container_image_jaeger_query .. mgr/cephadm/container_image_keepalived .. mgr/cephadm/container_image_loki .. mgr/cephadm/container_image_node_exporter .. mgr/cephadm/container_image_nvmeof .. mgr/cephadm/container_image_prometheus .. mgr/cephadm/container_image_promtail .. mgr/cephadm/container_image_samba .. mgr/cephadm/container_image_snmp_gateway ..
On Jul 1, 2026, at 6:53 AM, Sake via ceph-users <ceph-users@ceph.io> wrote:
Probably something not going correctly, because I just upgraded to 19.2.4 and Grafana is using the grafana image (in the past this was ceph-grafana) version 12.3.1.
On 1 July 2026 11:13:02 CEST, Eugen Block via ceph-users <ceph-users@ceph.io> wrote:
Hi,
I just upgraded our production cluster to 19.2.4 (from 19.2.3), there were no issues reported by our users, so all good. But I noticed that many graphs now show different warnings or strange labels:
- "This panel requires Angular (deprecated)." on multiple graphs - "PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "ceph_pool_wr" (1:6)" also on multiple graphs for different metrics - "OSD Capacity Utilization" just shows: "RangeError: invalid array length" - The "Pool Overview" graph has column names like "Value #F", "Value #G" and so on. And the values within don't make sense either.
I have uploaded screenshots:
https://www.pasteboard.co/A34Zcy9cqHe5.png https://www.pasteboard.co/F_vkpeBjGGWH.png https://www.pasteboard.co/mkCtlBN3zS6v.png https://www.pasteboard.co/T_ByD_gCpslA.png
I compared the prometheus and grafana versions we have currently running with a fresh 19.2.4 bootstrapped cluster and the versions are identical:
ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' 10.4.0
ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' 2.51.0
Is there anything I'm missing?
Thanks! Eugen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just a quick update: - The warnings about Angular are gone apparently. - The PromQL warnings are still present across multiple views although the graphs seem okay. - The "Pool Overview" table is still confusing, the header is not bold or otherwise highlighted. And the dropdown where you can select a specific view still contains "Value #I" columns among other things. - The OSD Capacity Utilization graph is also fixed now. I just compared the grafana dashboards with a Tentacle (20.2.1) lab cluster with Grafana 12.3.1, the pool overview is quite similar and has the same issues (or let's say "flaws"). Zitat von Eugen Block <eblock@nde.ag>:
Huh, that's kind of interesting... So yes, we do have a container_image override in our production cluster because of a private container registry, but not in that test cluster.
I wiped that single host test cluster completely (cephadm rm-cluster) and bootstrapped a fresh one. That's where I previously got that grafana version from. Now I did that again, only this time I also pruned all stored images on that host before bootstrapping again. And now I get a different grafana version:
root@ceph0-2:~# ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' 12.3.1
That is unexpected (to me) but good to know not to rely on that. I'll update our grafana image as well, let's see if those issues are resolved.
Thanks! Eugen
Zitat von Anthony D'Atri <anthony.datri@gmail.com>:
[ I've left off dev@ from this reply, as it doesn't seem necessarily like a dev issue ]
Is it possible that your cluster is overriding the default container images, assuming that you're running containerized with cephadm?
root@squid1:~# ceph config dump | grep container_image container_image XX mgr/cephadm/container_image_alertmanager YY mgr/cephadm/container_image_base ZZ mgr/cephadm/container_image_elasticsearch QQ mgr/cephadm/container_image_grafana .. mgr/cephadm/container_image_haproxy .. mgr/cephadm/container_image_jaeger_agent .. mgr/cephadm/container_image_jaeger_collector .. mgr/cephadm/container_image_jaeger_query .. mgr/cephadm/container_image_keepalived .. mgr/cephadm/container_image_loki .. mgr/cephadm/container_image_node_exporter .. mgr/cephadm/container_image_nvmeof .. mgr/cephadm/container_image_prometheus .. mgr/cephadm/container_image_promtail .. mgr/cephadm/container_image_samba .. mgr/cephadm/container_image_snmp_gateway ..
On Jul 1, 2026, at 6:53 AM, Sake via ceph-users <ceph-users@ceph.io> wrote:
Probably something not going correctly, because I just upgraded to 19.2.4 and Grafana is using the grafana image (in the past this was ceph-grafana) version 12.3.1.
On 1 July 2026 11:13:02 CEST, Eugen Block via ceph-users <ceph-users@ceph.io> wrote:
Hi,
I just upgraded our production cluster to 19.2.4 (from 19.2.3), there were no issues reported by our users, so all good. But I noticed that many graphs now show different warnings or strange labels:
- "This panel requires Angular (deprecated)." on multiple graphs - "PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "ceph_pool_wr" (1:6)" also on multiple graphs for different metrics - "OSD Capacity Utilization" just shows: "RangeError: invalid array length" - The "Pool Overview" graph has column names like "Value #F", "Value #G" and so on. And the values within don't make sense either.
I have uploaded screenshots:
https://www.pasteboard.co/A34Zcy9cqHe5.png https://www.pasteboard.co/F_vkpeBjGGWH.png https://www.pasteboard.co/mkCtlBN3zS6v.png https://www.pasteboard.co/T_ByD_gCpslA.png
I compared the prometheus and grafana versions we have currently running with a fresh 19.2.4 bootstrapped cluster and the versions are identical:
ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' 10.4.0
ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' 2.51.0
Is there anything I'm missing?
Thanks! Eugen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Anthony D'Atri
-
Eugen Block