Thank you!
Op 10-09-2024 09:39 CEST schreef Redouane Kachach <rkachach@redhat.com>:
Seems like a BUG in cephadm, the ceph-exporter when deployed doesn't specify its port that's why it's not being opened automatically. You can see that in the cephadm logs (ports list is empty):
2024-09-09 04:39:48,986 7fc2993d7740 DEBUG Loaded deploy configuration: {'fsid': '250b9d7c-6e65-11ef-8e0e-525400ecf80a', 'name': 'ceph-exporter.ceph-node-0', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_name': 'ceph-exporter', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:02ce7c1aa356b524041713a3603da8445c4fe00ed30cb1c1f91532926db20d3c' (http://quay.ceph.io/ceph-ci/ceph@sha256:02ce7c1aa356b524041713a3603da8445c4f...')], 'rank': None, 'rank_generation': None,
I opened the following tracker to fix the issue: https://tracker.ceph.com/issues/67975
On Mon, Sep 9, 2024 at 2:54 PM Sake Ceph <ceph@paulusma.eu> wrote:
We're using RHEL 8 and 9 and on both the port was not open. It's just strange it isn't working for ceph-exporter but just fine for everything else.
Kind regards, Sake
Op 09-09-2024 14:03 CEST schreef Eugen Block <eblock@nde.ag>:
Those two daemons are handled differently by cephadm, they're different classes (grafana is "class Monitoring(ContainerDaemonForm)" while ceph-exporter is "class CephExporter(ContainerDaemonForm)"), therefore they have different metadata etc., for example:
soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/ceph-exporter.soc9-ceph/unit.meta []
soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/grafana.soc9-ceph/unit.meta [ 3000 ]
But that's about all I can provide here. Maybe the host OS plays some role here as well, not sure.
Zitat von Sake Ceph <ceph@paulusma.eu>:
We're using default :) I'm talking about the deployment configuration which is shown in the log files when deploying grafana/ceph-exporter.
I got the same configuration as you for ceph-exporter (the default) when exporting the service.
Kind regards, Sake
Op 09-09-2024 12:04 CEST schreef Eugen Block <eblock@nde.ag>:
Can you be more specific about "deploy configuration"? Do you have your own spec files for grafana and ceph-exporter? I just ran 'ceph orch apply ceph-exporter' and the resulting config is this one:
# ceph orch ls ceph-exporter --export service_type: ceph-exporter service_name: ceph-exporter placement: host_pattern: '*' spec: prio_limit: 5 stats_period: 5
Zitat von Sake Ceph <ceph@paulusma.eu>:
Hello Eugen,
Well nothing about enabling port 9926.
For example I see the following when deploying Grafanan: 2024-09-05 14:27:30,969 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,334 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:31,350 7fd2e6583740 INFO firewalld ready 2024-09-05 14:27:31,593 7fd2e6583740 DEBUG Non-zero exit code 1 from /bin/firewall-cmd --permanent --query-port 3000/tcp 2024-09-05 14:27:31,594 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout no 2024-09-05 14:27:31,594 7fd2e6583740 INFO Enabling firewalld port 3000/tcp in current zone... 2024-09-05 14:27:31,832 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success 2024-09-05 14:27:32,212 7fd2e6583740 DEBUG /bin/firewall-cmd: stdout success
But only the following when deploying ceph-exporter: 2024-09-05 12:17:48,897 7f3d7cc0e740 INFO firewalld ready 2024-09-05 12:17:49,269 7f3d7cc0e740 DEBUG /bin/firewall-cmd: stdout success
When looking in the deploy configuration, Grafana shows 'ports': [3000], but ceph-exporter shows 'ports': []
Kind regards, Sake
> Op 09-09-2024 10:50 CEST schreef Eugen Block <eblock@nde.ag>: > > > Sorry, clicked "send" too soon. In a test cluster, cephadm.log shows > that it would try to open a port if a firewall was enabled: > > 2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is not enabled > 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable > service <ceph-exporter>. firewalld.service is not available > > Zitat von Eugen Block <eblock@nde.ag>: > > > Do you see anything in the cephadm.log related to the firewall? > > > > Zitat von Sake Ceph <ceph@paulusma.eu>: > > > >> After opening port 9926 manually, the Grafana dashboards show the data. > >> So is this a bug? > >> > >> Kind regards, > >> Sake > >>> Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@paulusma.eu>: > >>> > >>> > >>> That is working, but I noticed the firewall isn't opened for that > >>> port. Shouldn't cephadm manage this, like it does for all the > >>> other ports? > >>> > >>> Kind regards, > >>> Sake > >>> > >>>> Op 06-09-2024 16:14 CEST schreef Björn Lässig > <b.laessig@pengutronix.de>: > >>>> > >>>> > >>>> Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph: > >>>> > After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are > empty. For > >>>> > example the Osd latency under OSD device details or the Osd Overview > >>>> > has a lot of No data messages. > >>>> > > >>>> > >>>> is the ceph-exporter listening on port 9926 (on every host)? > >>>> > >>>> ss -tlpn sport 9926 > >>>> > >>>> Can you connect via browser? > >>>> > >>>> curl localhost:9926/metrics > >>>> > >>>> > I deployed ceph-exporter on all hosts, am I missing something? Did > >>>> > even a redeploy of prometheus. > >>>> > >>>> there is a bug, that this exporter does not listens for IPv6. > >>>> > >>>> greetings > >>>> Björn > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users@ceph.io > >>>> To unsubscribe send an email to ceph-users-leave@ceph.io > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users@ceph.io > >>> To unsubscribe send an email to ceph-users-leave@ceph.io > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io