After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages. I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus. Kind regards, Sake
Hi Sake, Do you have the config mgr/cephadm/secure_monitoring_stack to true? If so, this pull request will fix your problem: https://github.com/ceph/ceph/pull/58402 Regards, Frank Sake Ceph wrote:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, That option is set to false (I didnt enabled security for the monitoring stack). Kind regards, Sake
Op 04-09-2024 20:17 CEST schreef Frank de Bot (lists) <lists@searchy.net>:
Hi Sake,
Do you have the config mgr/cephadm/secure_monitoring_stack to true? If so, this pull request will fix your problem: https://github.com/ceph/ceph/pull/58402
Regards,
Frank
Sake Ceph wrote:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
As a workaround you can use: ceph config set mgr mgr/prometheus/exclude_perf_counters false However I understand that deploying a ceph-exporter daemon on each host is the proper fix. You may still be missing some configuration for it? On Thu, 5 Sept 2024 at 08:25, Sake Ceph <ceph@paulusma.eu> wrote:
Hi Frank,
That option is set to false (I didnt enabled security for the monitoring stack).
Kind regards, Sake
Op 04-09-2024 20:17 CEST schreef Frank de Bot (lists) <lists@searchy.net :
Hi Sake,
Do you have the config mgr/cephadm/secure_monitoring_stack to true? If so, this pull request will fix your problem: https://github.com/ceph/ceph/pull/58402
Regards,
Frank
Sake Ceph wrote:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I would like to stay away from using the workaround. First I did a redeployment of prometheus and later ceph-exporter, but still no data. After the deployment of ceph-exporter I saw the following messages (2 times, for each host running prometheus): Reconfiguring daemon prometheus.<host>. You mention missing a configuration, but I can't find any specific configuration options in the docs; https://docs.ceph.com/en/reef/mgr/prometheus/ and https://docs.ceph.com/en/reef/cephadm/services/monitoring/ Kind regards, Sake
Op 05-09-2024 10:00 CEST schreef Pierre Riteau <pierre@stackhpc.com>:
As a workaround you can use: ceph config set mgr mgr/prometheus/exclude_perf_counters false
However I understand that deploying a ceph-exporter daemon on each host is the proper fix. You may still be missing some configuration for it?
On Thu, 5 Sept 2024 at 08:25, Sake Ceph <ceph@paulusma.eu> wrote:
Hi Frank,
That option is set to false (I didnt enabled security for the monitoring stack).
Kind regards, Sake
Op 04-09-2024 20:17 CEST schreef Frank de Bot (lists) <lists@searchy.net :
Hi Sake,
Do you have the config mgr/cephadm/secure_monitoring_stack to true? If so, this pull request will fix your problem: https://github.com/ceph/ceph/pull/58402
Regards,
Frank
Sake Ceph wrote:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
Kind regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
is the ceph-exporter listening on port 9926 (on every host)? ss -tlpn sport 9926 Can you connect via browser? curl localhost:9926/metrics
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
there is a bug, that this exporter does not listens for IPv6. greetings Björn
That is working, but I noticed the firewall isn't opened for that port. Shouldn't cephadm manage this, like it does for all the other ports? Kind regards, Sake
Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>:
Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
is the ceph-exporter listening on port 9926 (on every host)?
ss -tlpn sport 9926
Can you connect via browser?
curl localhost:9926/metrics
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
there is a bug, that this exporter does not listens for IPv6.
greetings Björn _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
After opening port 9926 manually, the Grafana dashboards show the data. So is this a bug? Kind regards, Sake
Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>:
That is working, but I noticed the firewall isn't opened for that port. Shouldn't cephadm manage this, like it does for all the other ports?
Kind regards, Sake
Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>:
Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
is the ceph-exporter listening on port 9926 (on every host)?
ss -tlpn sport 9926
Can you connect via browser?
curl localhost:9926/metrics
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
there is a bug, that this exporter does not listens for IPv6.
greetings Björn _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Do you see anything in the cephadm.log related to the firewall? Zitat von Sake Ceph <ceph@paulusma.eu>:
After opening port 9926 manually, the Grafana dashboards show the data. So is this a bug?
Kind regards, Sake
Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>:
That is working, but I noticed the firewall isn't opened for that port. Shouldn't cephadm manage this, like it does for all the other ports?
Kind regards, Sake
Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>:
Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
is the ceph-exporter listening on port 9926 (on every host)?
ss -tlpn sport 9926
Can you connect via browser?
curl localhost:9926/metrics
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
there is a bug, that this exporter does not listens for IPv6.
greetings Björn _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry, clicked "send" too soon. In a test cluster, cephadm.log shows that it would try to open a port if a firewall was enabled: 2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is not enabled 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable service <ceph-exporter>. firewalld.service is not available Zitat von Eugen Block <eblock@nde.ag>:
Do you see anything in the cephadm.log related to the firewall?
Zitat von Sake Ceph <ceph@paulusma.eu>:
After opening port 9926 manually, the Grafana dashboards show the data. So is this a bug?
Kind regards, Sake
Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>:
That is working, but I noticed the firewall isn't opened for that port. Shouldn't cephadm manage this, like it does for all the other ports?
Kind regards, Sake
Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>:
Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
is the ceph-exporter listening on port 9926 (on every host)?
ss -tlpn sport 9926
Can you connect via browser?
curl localhost:9926/metrics
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
there is a bug, that this exporter does not listens for IPv6.
greetings Björn _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Eugen, Well nothing about enabling port 9926. For example I see the following when deploying Grafanan: 2024-09-05 14:27:30,969 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,334 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:31,350 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,593 7fd2e6583740 DEBUG Non-zero exit code 1 from /bin/firewall-cmd --permanent --query-port 3000/tcp 2024-09-05 14:27:31,594 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout no 2024-09-05 14:27:31,594 7fd2e6583740 INFO Enabling firewalld port 3000/tcp in current zone... 2024-09-05 14:27:31,832 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:32,212 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success But only the following when deploying ceph-exporter: 2024-09-05 12:17:48,897 7f3d7cc0e740 INFO firewalld ready 2024-09-05 12:17:49,269 7f3d7cc0e740 DEBUG /bin/firewall-cmd: stdout success When looking in the deploy configuration, Grafana shows 'ports': [3000], but ceph-exporter shows 'ports': [] Kind regards, Sake
Op 09-09-2024 10:50 CEST schreef Eugen Block <eblock@nde.ag>:
Sorry, clicked "send" too soon. In a test cluster, cephadm.log shows that it would try to open a port if a firewall was enabled:
2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is not enabled 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable service <ceph-exporter>. firewalld.service is not available
Zitat von Eugen Block <eblock@nde.ag>:
Do you see anything in the cephadm.log related to the firewall?
Zitat von Sake Ceph <ceph@paulusma.eu>:
After opening port 9926 manually, the Grafana dashboards show the data. So is this a bug?
Kind regards, Sake
Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>:
That is working, but I noticed the firewall isn't opened for that port. Shouldn't cephadm manage this, like it does for all the other ports?
Kind regards, Sake
Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>:
Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph:
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages.
is the ceph-exporter listening on port 9926 (on every host)?
ss -tlpn sport 9926
Can you connect via browser?
curl localhost:9926/metrics
I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus.
there is a bug, that this exporter does not listens for IPv6.
greetings Björn _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Can you be more specific about "deploy configuration"? Do you have your own spec files for grafana and ceph-exporter? I just ran 'ceph orch apply ceph-exporter' and the resulting config is this one: # ceph orch ls ceph-exporter --export service_type: ceph-exporter service_name: ceph-exporter placement: host_pattern: '*' spec: prio_limit: 5 stats_period: 5 Zitat von Sake Ceph <ceph@paulusma.eu>:
Hello Eugen,
Well nothing about enabling port 9926.
For example I see the following when deploying Grafanan: 2024-09-05 14:27:30,969 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,334 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:31,350 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,593 7fd2e6583740 DEBUG Non-zero exit code 1 from /bin/firewall-cmd --permanent --query-port 3000/tcp 2024-09-05 14:27:31,594 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout no 2024-09-05 14:27:31,594 7fd2e6583740 INFO Enabling firewalld port 3000/tcp in current zone... 2024-09-05 14:27:31,832 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:32,212 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success
But only the following when deploying ceph-exporter: 2024-09-05 12:17:48,897 7f3d7cc0e740 INFO firewalld ready 2024-09-05 12:17:49,269 7f3d7cc0e740 DEBUG /bin/firewall-cmd: stdout success
When looking in the deploy configuration, Grafana shows 'ports': [3000], but ceph-exporter shows 'ports': []
Kind regards, Sake
Op 09-09-2024 10:50 CEST schreef Eugen Block <eblock@nde.ag>:
Sorry, clicked "send" too soon. In a test cluster, cephadm.log shows that it would try to open a port if a firewall was enabled:
2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is not enabled 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable service <ceph-exporter>. firewalld.service is not available
Zitat von Eugen Block <eblock@nde.ag>:
Do you see anything in the cephadm.log related to the firewall?
Zitat von Sake Ceph <ceph@paulusma.eu>:
After opening port 9926 manually, the Grafana dashboards show the data. So is this a bug?
Kind regards, Sake
Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>:
That is working, but I noticed the firewall isn't opened for that port. Shouldn't cephadm manage this, like it does for all the other ports?
Kind regards, Sake
Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>:
Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph: > After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For > example the Osd latency under OSD device details or the Osd Overview > has a lot of No data messages. >
is the ceph-exporter listening on port 9926 (on every host)?
ss -tlpn sport 9926
Can you connect via browser?
curl localhost:9926/metrics
> I deployed ceph-exporter on all hosts, am I missing something? Did > even a redeploy of prometheus.
there is a bug, that this exporter does not listens for IPv6.
greetings Björn _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We're using default :) I'm talking about the deployment configuration which is shown in the log files when deploying grafana/ceph-exporter. I got the same configuration as you for ceph-exporter (the default) when exporting the service. Kind regards, Sake
Op 09-09-2024 12:04 CEST schreef Eugen Block <eblock@nde.ag>:
Can you be more specific about "deploy configuration"? Do you have your own spec files for grafana and ceph-exporter? I just ran 'ceph orch apply ceph-exporter' and the resulting config is this one:
# ceph orch ls ceph-exporter --export service_type: ceph-exporter service_name: ceph-exporter placement: host_pattern: '*' spec: prio_limit: 5 stats_period: 5
Zitat von Sake Ceph <ceph@paulusma.eu>:
Hello Eugen,
Well nothing about enabling port 9926.
For example I see the following when deploying Grafanan: 2024-09-05 14:27:30,969 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,334 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:31,350 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,593 7fd2e6583740 DEBUG Non-zero exit code 1 from /bin/firewall-cmd --permanent --query-port 3000/tcp 2024-09-05 14:27:31,594 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout no 2024-09-05 14:27:31,594 7fd2e6583740 INFO Enabling firewalld port 3000/tcp in current zone... 2024-09-05 14:27:31,832 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:32,212 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success
But only the following when deploying ceph-exporter: 2024-09-05 12:17:48,897 7f3d7cc0e740 INFO firewalld ready 2024-09-05 12:17:49,269 7f3d7cc0e740 DEBUG /bin/firewall-cmd: stdout success
When looking in the deploy configuration, Grafana shows 'ports': [3000], but ceph-exporter shows 'ports': []
Kind regards, Sake
Op 09-09-2024 10:50 CEST schreef Eugen Block <eblock@nde.ag>:
Sorry, clicked "send" too soon. In a test cluster, cephadm.log shows that it would try to open a port if a firewall was enabled:
2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is not enabled 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable service <ceph-exporter>. firewalld.service is not available
Zitat von Eugen Block <eblock@nde.ag>:
Do you see anything in the cephadm.log related to the firewall?
Zitat von Sake Ceph <ceph@paulusma.eu>:
After opening port 9926 manually, the Grafana dashboards show the data. So is this a bug?
Kind regards, Sake
Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>:
That is working, but I noticed the firewall isn't opened for that port. Shouldn't cephadm manage this, like it does for all the other ports?
Kind regards, Sake
> Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>: > > > Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph: > > After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For > > example the Osd latency under OSD device details or the Osd Overview > > has a lot of No data messages. > > > > is the ceph-exporter listening on port 9926 (on every host)? > > ss -tlpn sport 9926 > > Can you connect via browser? > > curl localhost:9926/metrics > > > I deployed ceph-exporter on all hosts, am I missing something? Did > > even a redeploy of prometheus. > > there is a bug, that this exporter does not listens for IPv6. > > greetings > Björn > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Those two daemons are handled differently by cephadm, they're different classes (grafana is "class Monitoring(ContainerDaemonForm)" while ceph-exporter is "class CephExporter(ContainerDaemonForm)"), therefore they have different metadata etc., for example: soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/ceph-exporter.soc9-ceph/unit.meta [] soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/grafana.soc9-ceph/unit.meta [ 3000 ] But that's about all I can provide here. Maybe the host OS plays some role here as well, not sure. Zitat von Sake Ceph <ceph@paulusma.eu>:
We're using default :) I'm talking about the deployment configuration which is shown in the log files when deploying grafana/ceph-exporter.
I got the same configuration as you for ceph-exporter (the default) when exporting the service.
Kind regards, Sake
Op 09-09-2024 12:04 CEST schreef Eugen Block <eblock@nde.ag>:
Can you be more specific about "deploy configuration"? Do you have your own spec files for grafana and ceph-exporter? I just ran 'ceph orch apply ceph-exporter' and the resulting config is this one:
# ceph orch ls ceph-exporter --export service_type: ceph-exporter service_name: ceph-exporter placement: host_pattern: '*' spec: prio_limit: 5 stats_period: 5
Zitat von Sake Ceph <ceph@paulusma.eu>:
Hello Eugen,
Well nothing about enabling port 9926.
For example I see the following when deploying Grafanan: 2024-09-05 14:27:30,969 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,334 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:31,350 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,593 7fd2e6583740 DEBUG Non-zero exit code 1 from /bin/firewall-cmd --permanent --query-port 3000/tcp 2024-09-05 14:27:31,594 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout no 2024-09-05 14:27:31,594 7fd2e6583740 INFO Enabling firewalld port 3000/tcp in current zone... 2024-09-05 14:27:31,832 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:32,212 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success
But only the following when deploying ceph-exporter: 2024-09-05 12:17:48,897 7f3d7cc0e740 INFO firewalld ready 2024-09-05 12:17:49,269 7f3d7cc0e740 DEBUG /bin/firewall-cmd: stdout success
When looking in the deploy configuration, Grafana shows 'ports': [3000], but ceph-exporter shows 'ports': []
Kind regards, Sake
Op 09-09-2024 10:50 CEST schreef Eugen Block <eblock@nde.ag>:
Sorry, clicked "send" too soon. In a test cluster, cephadm.log shows that it would try to open a port if a firewall was enabled:
2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is not enabled 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable service <ceph-exporter>. firewalld.service is not available
Zitat von Eugen Block <eblock@nde.ag>:
Do you see anything in the cephadm.log related to the firewall?
Zitat von Sake Ceph <ceph@paulusma.eu>:
After opening port 9926 manually, the Grafana dashboards show the data. So is this a bug?
Kind regards, Sake > Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>: > > > That is working, but I noticed the firewall isn't opened for that > port. Shouldn't cephadm manage this, like it does for all the > other ports? > > Kind regards, > Sake > >> Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>: >> >> >> Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph: >> > After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For >> > example the Osd latency under OSD device details or the Osd Overview >> > has a lot of No data messages. >> > >> >> is the ceph-exporter listening on port 9926 (on every host)? >> >> ss -tlpn sport 9926 >> >> Can you connect via browser? >> >> curl localhost:9926/metrics >> >> > I deployed ceph-exporter on all hosts, am I missing something? Did >> > even a redeploy of prometheus. >> >> there is a bug, that this exporter does not listens for IPv6. >> >> greetings >> Björn >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We're using RHEL 8 and 9 and on both the port was not open. It's just strange it isn't working for ceph-exporter but just fine for everything else. Kind regards, Sake
Op 09-09-2024 14:03 CEST schreef Eugen Block <eblock@nde.ag>:
Those two daemons are handled differently by cephadm, they're different classes (grafana is "class Monitoring(ContainerDaemonForm)" while ceph-exporter is "class CephExporter(ContainerDaemonForm)"), therefore they have different metadata etc., for example:
soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/ceph-exporter.soc9-ceph/unit.meta []
soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/grafana.soc9-ceph/unit.meta [ 3000 ]
But that's about all I can provide here. Maybe the host OS plays some role here as well, not sure.
Zitat von Sake Ceph <ceph@paulusma.eu>:
We're using default :) I'm talking about the deployment configuration which is shown in the log files when deploying grafana/ceph-exporter.
I got the same configuration as you for ceph-exporter (the default) when exporting the service.
Kind regards, Sake
Op 09-09-2024 12:04 CEST schreef Eugen Block <eblock@nde.ag>:
Can you be more specific about "deploy configuration"? Do you have your own spec files for grafana and ceph-exporter? I just ran 'ceph orch apply ceph-exporter' and the resulting config is this one:
# ceph orch ls ceph-exporter --export service_type: ceph-exporter service_name: ceph-exporter placement: host_pattern: '*' spec: prio_limit: 5 stats_period: 5
Zitat von Sake Ceph <ceph@paulusma.eu>:
Hello Eugen,
Well nothing about enabling port 9926.
For example I see the following when deploying Grafanan: 2024-09-05 14:27:30,969 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,334 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:31,350 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,593 7fd2e6583740 DEBUG Non-zero exit code 1 from /bin/firewall-cmd --permanent --query-port 3000/tcp 2024-09-05 14:27:31,594 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout no 2024-09-05 14:27:31,594 7fd2e6583740 INFO Enabling firewalld port 3000/tcp in current zone... 2024-09-05 14:27:31,832 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:32,212 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success
But only the following when deploying ceph-exporter: 2024-09-05 12:17:48,897 7f3d7cc0e740 INFO firewalld ready 2024-09-05 12:17:49,269 7f3d7cc0e740 DEBUG /bin/firewall-cmd: stdout success
When looking in the deploy configuration, Grafana shows 'ports': [3000], but ceph-exporter shows 'ports': []
Kind regards, Sake
Op 09-09-2024 10:50 CEST schreef Eugen Block <eblock@nde.ag>:
Sorry, clicked "send" too soon. In a test cluster, cephadm.log shows that it would try to open a port if a firewall was enabled:
2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is not enabled 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable service <ceph-exporter>. firewalld.service is not available
Zitat von Eugen Block <eblock@nde.ag>:
Do you see anything in the cephadm.log related to the firewall?
Zitat von Sake Ceph <ceph@paulusma.eu>:
> After opening port 9926 manually, the Grafana dashboards show the data. > So is this a bug? > > Kind regards, > Sake >> Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>: >> >> >> That is working, but I noticed the firewall isn't opened for that >> port. Shouldn't cephadm manage this, like it does for all the >> other ports? >> >> Kind regards, >> Sake >> >>> Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>: >>> >>> >>> Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph: >>> > After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For >>> > example the Osd latency under OSD device details or the Osd Overview >>> > has a lot of No data messages. >>> > >>> >>> is the ceph-exporter listening on port 9926 (on every host)? >>> >>> ss -tlpn sport 9926 >>> >>> Can you connect via browser? >>> >>> curl localhost:9926/metrics >>> >>> > I deployed ceph-exporter on all hosts, am I missing something? Did >>> > even a redeploy of prometheus. >>> >>> there is a bug, that this exporter does not listens for IPv6. >>> >>> greetings >>> Björn >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Seems like a BUG in cephadm, the ceph-exporter when deployed doesn't specify its port that's why it's not being opened automatically. You can see that in the cephadm logs (ports list is empty): 2024-09-09 04:39:48,986 7fc2993d7740 DEBUG Loaded deploy configuration: {'fsid': '250b9d7c-6e65-11ef-8e0e-525400ecf80a', 'name': 'ceph-exporter.ceph-node-0', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_name': 'ceph-exporter', *'ports': [],* 'ip': None, 'deployed_by': [' quay.ceph.io/ceph-ci/ceph@sha256:02ce7c1aa356b524041713a3603da8445c4fe00ed30cb1c1f91532926db20d3c'], 'rank': None, 'rank_generation': None, I opened the following tracker to fix the issue: https://tracker.ceph.com/issues/67975 On Mon, Sep 9, 2024 at 2:54 PM Sake Ceph <ceph@paulusma.eu> wrote:
We're using RHEL 8 and 9 and on both the port was not open. It's just strange it isn't working for ceph-exporter but just fine for everything else.
Kind regards, Sake
Op 09-09-2024 14:03 CEST schreef Eugen Block <eblock@nde.ag>:
Those two daemons are handled differently by cephadm, they're different classes (grafana is "class Monitoring(ContainerDaemonForm)" while ceph-exporter is "class CephExporter(ContainerDaemonForm)"), therefore they have different metadata etc., for example:
soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/ceph-exporter.soc9-ceph/unit.meta []
soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/grafana.soc9-ceph/unit.meta [ 3000 ]
But that's about all I can provide here. Maybe the host OS plays some role here as well, not sure.
Zitat von Sake Ceph <ceph@paulusma.eu>:
We're using default :) I'm talking about the deployment configuration which is shown in the log files when deploying grafana/ceph-exporter.
I got the same configuration as you for ceph-exporter (the default) when exporting the service.
Kind regards, Sake
Op 09-09-2024 12:04 CEST schreef Eugen Block <eblock@nde.ag>:
Can you be more specific about "deploy configuration"? Do you have your own spec files for grafana and ceph-exporter? I just ran 'ceph orch apply ceph-exporter' and the resulting config is this one:
# ceph orch ls ceph-exporter --export service_type: ceph-exporter service_name: ceph-exporter placement: host_pattern: '*' spec: prio_limit: 5 stats_period: 5
Zitat von Sake Ceph <ceph@paulusma.eu>:
Hello Eugen,
Well nothing about enabling port 9926.
For example I see the following when deploying Grafanan: 2024-09-05 14:27:30,969 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,334 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:31,350 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,593 7fd2e6583740 DEBUG Non-zero exit code 1 from /bin/firewall-cmd --permanent --query-port 3000/tcp 2024-09-05 14:27:31,594 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout no 2024-09-05 14:27:31,594 7fd2e6583740 INFO Enabling firewalld port 3000/tcp in current zone... 2024-09-05 14:27:31,832 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:32,212 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success
But only the following when deploying ceph-exporter: 2024-09-05 12:17:48,897 7f3d7cc0e740 INFO firewalld ready 2024-09-05 12:17:49,269 7f3d7cc0e740 DEBUG /bin/firewall-cmd: stdout success
When looking in the deploy configuration, Grafana shows 'ports': [3000], but ceph-exporter shows 'ports': []
Kind regards, Sake
Op 09-09-2024 10:50 CEST schreef Eugen Block <eblock@nde.ag>:
Sorry, clicked "send" too soon. In a test cluster, cephadm.log shows that it would try to open a port if a firewall was enabled:
2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is not enabled 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable service <ceph-exporter>. firewalld.service is not available
Zitat von Eugen Block <eblock@nde.ag>:
> Do you see anything in the cephadm.log related to the firewall? > > Zitat von Sake Ceph <ceph@paulusma.eu>: > >> After opening port 9926 manually, the Grafana dashboards show the data. >> So is this a bug? >> >> Kind regards, >> Sake >>> Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>: >>> >>> >>> That is working, but I noticed the firewall isn't opened for that >>> port. Shouldn't cephadm manage this, like it does for all the >>> other ports? >>> >>> Kind regards, >>> Sake >>> >>>> Op 06-09-2024 16:14 CEST schreef Björn Lässig <b.laessig@pengutronix.de>: >>>> >>>> >>>> Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph: >>>> > After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For >>>> > example the Osd latency under OSD device details or the Osd Overview >>>> > has a lot of No data messages. >>>> > >>>> >>>> is the ceph-exporter listening on port 9926 (on every host)? >>>> >>>> ss -tlpn sport 9926 >>>> >>>> Can you connect via browser? >>>> >>>> curl localhost:9926/metrics >>>> >>>> > I deployed ceph-exporter on all hosts, am I missing something? Did >>>> > even a redeploy of prometheus. >>>> >>>> there is a bug, that this exporter does not listens for IPv6. >>>> >>>> greetings >>>> Björn >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you!
Op 10-09-2024 09:39 CEST schreef Redouane Kachach <rkachach@redhat.com>:
Seems like a BUG in cephadm, the ceph-exporter when deployed doesn't specify its port that's why it's not being opened automatically. You can see that in the cephadm logs (ports list is empty):
2024-09-09 04:39:48,986 7fc2993d7740 DEBUG Loaded deploy configuration: {'fsid': '250b9d7c-6e65-11ef-8e0e-525400ecf80a', 'name': 'ceph-exporter.ceph-node-0', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_name': 'ceph-exporter', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:02ce7c1aa356b524041713a3603da8445c4fe00ed30cb1c1f91532926db20d3c' (http://quay.ceph.io/ceph-ci/ceph@sha256:02ce7c1aa356b524041713a3603da8445c4f...')], 'rank': None, 'rank_generation': None,
I opened the following tracker to fix the issue: https://tracker.ceph.com/issues/67975
On Mon, Sep 9, 2024 at 2:54 PM Sake Ceph <ceph@paulusma.eu> wrote:
We're using RHEL 8 and 9 and on both the port was not open. It's just strange it isn't working for ceph-exporter but just fine for everything else.
Kind regards, Sake
Op 09-09-2024 14:03 CEST schreef Eugen Block <eblock@nde.ag>:
Those two daemons are handled differently by cephadm, they're different classes (grafana is "class Monitoring(ContainerDaemonForm)" while ceph-exporter is "class CephExporter(ContainerDaemonForm)"), therefore they have different metadata etc., for example:
soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/ceph-exporter.soc9-ceph/unit.meta []
soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/grafana.soc9-ceph/unit.meta [ 3000 ]
But that's about all I can provide here. Maybe the host OS plays some role here as well, not sure.
Zitat von Sake Ceph <ceph@paulusma.eu>:
We're using default :) I'm talking about the deployment configuration which is shown in the log files when deploying grafana/ceph-exporter.
I got the same configuration as you for ceph-exporter (the default) when exporting the service.
Kind regards, Sake
Op 09-09-2024 12:04 CEST schreef Eugen Block <eblock@nde.ag>:
Can you be more specific about "deploy configuration"? Do you have your own spec files for grafana and ceph-exporter? I just ran 'ceph orch apply ceph-exporter' and the resulting config is this one:
# ceph orch ls ceph-exporter --export service_type: ceph-exporter service_name: ceph-exporter placement: host_pattern: '*' spec: prio_limit: 5 stats_period: 5
Zitat von Sake Ceph <ceph@paulusma.eu>:
Hello Eugen,
Well nothing about enabling port 9926.
For example I see the following when deploying Grafanan: 2024-09-05 14:27:30,969 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,334 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:31,350 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,593 7fd2e6583740 DEBUG Non-zero exit code 1 from /bin/firewall-cmd --permanent --query-port 3000/tcp 2024-09-05 14:27:31,594 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout no 2024-09-05 14:27:31,594 7fd2e6583740 INFO Enabling firewalld port 3000/tcp in current zone... 2024-09-05 14:27:31,832 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:32,212 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success
But only the following when deploying ceph-exporter: 2024-09-05 12:17:48,897 7f3d7cc0e740 INFO firewalld ready 2024-09-05 12:17:49,269 7f3d7cc0e740 DEBUG /bin/firewall-cmd: stdout success
When looking in the deploy configuration, Grafana shows 'ports': [3000], but ceph-exporter shows 'ports': []
Kind regards, Sake
> Op 09-09-2024 10:50 CEST schreef Eugen Block <eblock@nde.ag>: > > > Sorry, clicked "send" too soon. In a test cluster, cephadm.log shows > that it would try to open a port if a firewall was enabled: > > 2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is not enabled > 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable > service <ceph-exporter>. firewalld.service is not available > > Zitat von Eugen Block <eblock@nde.ag>: > > > Do you see anything in the cephadm.log related to the firewall? > > > > Zitat von Sake Ceph <ceph@paulusma.eu>: > > > >> After opening port 9926 manually, the Grafana dashboards show the data. > >> So is this a bug? > >> > >> Kind regards, > >> Sake > >>> Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>: > >>> > >>> > >>> That is working, but I noticed the firewall isn't opened for that > >>> port. Shouldn't cephadm manage this, like it does for all the > >>> other ports? > >>> > >>> Kind regards, > >>> Sake > >>> > >>>> Op 06-09-2024 16:14 CEST schreef Björn Lässig > <b.laessig@pengutronix.de>: > >>>> > >>>> > >>>> Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph: > >>>> > After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are > empty. For > >>>> > example the Osd latency under OSD device details or the Osd Overview > >>>> > has a lot of No data messages. > >>>> > > >>>> > >>>> is the ceph-exporter listening on port 9926 (on every host)? > >>>> > >>>> ss -tlpn sport 9926 > >>>> > >>>> Can you connect via browser? > >>>> > >>>> curl localhost:9926/metrics > >>>> > >>>> > I deployed ceph-exporter on all hosts, am I missing something? Did > >>>> > even a redeploy of prometheus. > >>>> > >>>> there is a bug, that this exporter does not listens for IPv6. > >>>> > >>>> greetings > >>>> Björn > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users@ceph.io > >>>> To unsubscribe send an email to ceph-users-leave@ceph.io > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users@ceph.io > >>> To unsubscribe send an email to ceph-users-leave@ceph.io > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Björn Lässig
-
Eugen Block
-
Frank de Bot (lists)
-
Pierre Riteau
-
Redouane Kachach
-
Sake Ceph