RGW Graphs in cephadm setup
We recently upgraded all our clusters to rocky 9.4 and reef 18.2.4. Two of the clusters show the rgw metrics in the ceph dashboard and the other two don't. I made sure the firewalls were open for ceph-exporter and that Prometheus was gathering the stats on all 4 clusters. For the clusters that are not working, when I do a search on the Prometheus web console at port 9095 on the performance monitoring node using "{instance="rgw1"}, I noticed that the instance_id shows a long number. I did the same search on the clusters that show rgw stats in grafana and they have an instance_id that looks like the drop down in the Grafana dashboard. E.g. "rgw.rgw.rgw1.biewae". So it seems that for some reason, on the clusters that are not working, the ceph-exporter running on the rgw nodes is not pulling the proper instance_id rgw instance. Is there a way to fix the instance_id that ceph-exporter is pulling? I can see this instance id when browsing to the servers http://rgw1:9926/metrics portal. If I run a podman ps, I can see the container running and it has the unique id .biewae showing. I used cephadm to enter the ceph-exporter container on the rgw to see what the logs were saying and I see the same thing repeated: 1 pid path is empty; process metrics won't be fetched for: ceph-client.rgw.rgw.rgw1.biewae.2.94759457265864 Its only the rgw's that show this error in the ceph-exporter log file, the mons and osd nodes show no errors in the log and show up in grafana properly with data. I redeployed ceph-exporter and rgw to the same node in that order, but no change. I did set the default image container to 18.2.4 prior to redeploy cause I ran into that issue as well. Regards, -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw
participants (1)
-
brentk@cfl.rr.com