On Mon, Dec 09, 2019 at 05:01:04PM -0800, Paul Choi wrote:
Hello, Anybody seeing the Prometheus endpoint hanging with the new 13.2.7 release? With 13.2.6 the endpoint would respond with a payload of 15MB in less than 10 seconds.
I'd guess its not the prometheus module itself: $ git diff v13.2.6 v13.2.7 src/pybind/mgr/prometheus diff --git a/src/pybind/mgr/prometheus/module.py b/src/pybind/mgr/prometheus/module.py index 2d4472434a..3a398e0b0c 100644 --- a/src/pybind/mgr/prometheus/module.py +++ b/src/pybind/mgr/prometheus/module.py @@ -142,7 +142,8 @@ class Metric(object): def promethize(path): ''' replace illegal metric name characters ''' - result = path.replace('.', '_').replace('+', '_plus').replace('::', '_') + result = path.replace('.', '_').replace( + '+', '_plus').replace('::', '_').replace(' ', '_') # Hyphens usually turn into underscores, unless they are # trailing @@ -720,7 +721,8 @@ class Module(MgrModule): raise cherrypy.HTTPError(503, 'No MON connection') # Make the cache timeout for collecting configurable - self.collect_timeout = self.get_localized_config('scrape_interval', 5.0) + self.collect_timeout = float(self.get_localized_config( + 'scrape_interval', 5.0)) server_addr = self.get_localized_config('server_addr', DEFAULT_ADDR) server_port = self.get_localized_config('server_port', DEFAULT_PORT) So the mgr would be a likely suspect. If you could open a tracker ticket, ideally with mgr debug logs attached, this can be looked at.
Now, if you restart ceph-mgr, the Prometheus endpoint responds quickly for the first run, then successive runs get slower and slower, until it takes several minutes. I have no customization for the mgr module. Except for the Prometheus module, the "status" module and "Zabbix" module are working fine. This is on Ubuntu 16 LTS: ii ceph-mgr 13.2.7-1xenial amd64 manager for the ceph distributed storage system I'd love to know if there's a way to diagnose this issue - I tried upping the debug ms level but that doesn't seem to yield useful information. I don't know if this useful, but "prometheus self-test" is fine too. $ sudo ceph tell mgr.0 prometheus self-test Self-test OK pchoi@pchoi-desktop:~$ ceph mgr module ls { "enabled_modules": [ "prometheus", "status", "zabbix" ], "disabled_modules": [ { "name": "balancer", "can_run": true, "error_string": "" }, { "name": "crash", "can_run": true, "error_string": "" }, { "name": "dashboard", "can_run": true, "error_string": "" }, { "name": "hello", "can_run": true, "error_string": "" }, { "name": "influx", "can_run": false, "error_string": "influxdb python module not found" }, { "name": "iostat", "can_run": true, "error_string": "" }, { "name": "localpool", "can_run": true, "error_string": "" }, { "name": "restful", "can_run": true, "error_string": "" }, { "name": "selftest", "can_run": true, "error_string": "" }, { "name": "smart", "can_run": true, "error_string": "" }, { "name": "telegraf", "can_run": true, "error_string": "" }, { "name": "telemetry", "can_run": true, "error_string": "" } ] } pchoi@pchoi-desktop:~$ ceph mgr services { "prometheus": "[1]http://woodenbox2:9283/" }
References
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer