Multiple Metric Generation Locations in Ceph

Redouane and Avan came to me with an issue with RGW related metrics that warrants a broader community discussion for all daemons. For more information, the issue is being tracked by https://tracker.ceph.com/issues/64598 Currently, metrics consumed by Prometheus related to the RGW are being generated by combining two parts: 1. The RGW perf counters: these counters are generated by the ceph-exporter by parsing the output of the rgw command `ceph counter dump`. 2. The RGW metadata (daemon, ceph-version, hostname, etc): this information is generated by the prometheus mgr module. To combine the two parts ceph-exporter uses a key field called instance_id, which is generated as follows: 1. On the ceph-exporter side asok admin socket filename is parsed to extract the daemon_id which is used to derive the instance_id. 2. On the prometheus-mgr module side orchestrator (cephadm or rook) is called to get the daemon_id then instance_id is derived from the daemon_id This approach/design suffers from the following issues: 1. It creates a strong dependency between prometheus-mgr module and the orchestrator module (this has already caused issues for Rook environments, ceph v18.2.1 metrics are completely broken because of this) 2. instance_id on the ceph-exporter side mgmt is weak as it relies on socket filename parsing 3. instance_id generation is error-prone as it relies on how daemon_ids are handled by the orchestrator module (which is difference between rook and cephadm) The issue for RGW is that with certain orchestrators, for example in Rook, there is a mismatch between the instance IDs for the metrics emitted by the exporter and the metrics from the prometheus manager module. This has ramifications when running queries in Prometheus when the instance id is the primary key between the metrics in the queries. There are many options for solutions, and I'd be happy to hear the community's thoughts about what they think. Here are ours (Avan, Redouane, and I): 1. We think daemon specific metrics meant for Prometheus should only be emitted from one place, and that place should be the newer ceph-exporter. 2. We discussed having a command you can run on an admin socket that would emit all of the metadata that is currently being sent by the manager module. This way we're not relying on parsing file names anymore. 3. promtheus-mgr module will still exist and will be used to emit cluster wise metrics The command could be something like `ceph who-am-i` that you would expect to work on any daemons admin socket, or something daemon specific like `ceph rgw-info`. In other words, move the metadata source from the mgr-prometheus module to the ceph-exporter and use this new command `ceph who-am-i` to get it. This way, each ceph-daemon will be self-sufficient and able to provide the metadata needed to label/tag its metrics. At this moment this affects at least two daemons: rgw and rbd-mirror, but following the approach above and by introducing the new generic command we can follow the same pattern for other legacy (or new) daemons. Look forward to hearing other thoughts, Ali, Redouane, Avan