Prometheus anomaly in Reef
I finally got brave and migrated from Pacific to Reef, did some banging and hammering and for the first time in a long time got a complete "HEALTH OK" status. However, the dashboard is still not happy. It cannot contact the Prometheus API on port 9095. I have redeployed Prometheus multiple times without result. I'm pretty sure that at one time there were no less than 3 different Prometheus containers running on one of the configured Prometheus servers, but now all I can get is the node-exporter. Worse, if I do: ceph orch reconfig prometheus I get: Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls" But if I do: ceph orch ls I get: prometheus ?:9095 0/1 - 116s ceph02.mousetech.com Suggestions?
The cephadm.log should show some details why it fails to deploy the daemon. If there's not much, look into the daemon logs as well (cephadm logs --name prometheus.ceph02.mousetech.com). Could it be that there's a non-cephadm prometheus already listening on port 9095? Zitat von Tim Holloway <timh@mousetech.com>:
I finally got brave and migrated from Pacific to Reef, did some banging and hammering and for the first time in a long time got a complete "HEALTH OK" status.
However, the dashboard is still not happy. It cannot contact the Prometheus API on port 9095.
I have redeployed Prometheus multiple times without result.
I'm pretty sure that at one time there were no less than 3 different Prometheus containers running on one of the configured Prometheus servers, but now all I can get is the node-exporter.
Worse, if I do:
ceph orch reconfig prometheus
I get:
Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls"
But if I do:
ceph orch ls
I get:
prometheus ?:9095 0/1 - 116s ceph02.mousetech.com
Suggestions?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I don't think there is failure to deploy. For one thing, I did have, as mentioned 3 Prometheus-related containers running at one point on the machine. Also checked for port issues and there are none. Nothing listens on 9095. One thing that does concern me is that the docs sau changes in settings require "restarting prometheus", but not what command does that. Given that there are no systemd units to address and that the orchestrator claims that there is no "prometheus service" even as it shows that there's 1 service, stopped, it's quite frustrating. On 3/26/25 07:26, Eugen Block wrote:
The cephadm.log should show some details why it fails to deploy the daemon. If there's not much, look into the daemon logs as well (cephadm logs --name prometheus.ceph02.mousetech.com). Could it be that there's a non-cephadm prometheus already listening on port 9095?
Zitat von Tim Holloway <timh@mousetech.com>:
I finally got brave and migrated from Pacific to Reef, did some banging and hammering and for the first time in a long time got a complete "HEALTH OK" status.
However, the dashboard is still not happy. It cannot contact the Prometheus API on port 9095.
I have redeployed Prometheus multiple times without result.
I'm pretty sure that at one time there were no less than 3 different Prometheus containers running on one of the configured Prometheus servers, but now all I can get is the node-exporter.
Worse, if I do:
ceph orch reconfig prometheus
I get:
Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls"
But if I do:
ceph orch ls
I get:
prometheus ?:9095 0/1 - 116s ceph02.mousetech.com
Suggestions?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Can you share 'ceph orch ls prometheus --export'? And if it has been deployed successfully but is currently not running, the logs should show why that is the case. To restart prometheus, you can just run this to restart the entire prometheus service (which would include all instances if you had multiple, but currently you only have one in the specs): ceph orch restart prometheus or only a specific daemon: ceph orch daemon restart prometheus.<hostname> And usually, cephadm does create systemd units, for example: cephadm ls --no-detail | grep prometheus "name": "prometheus.nautilus", "systemd_unit": "ceph-201a2fbc-ce7b-44a3-9ed7-39427972083b@prometheus.nautilus" Zitat von Tim Holloway <timh@mousetech.com>:
I don't think there is failure to deploy. For one thing, I did have, as mentioned 3 Prometheus-related containers running at one point on the machine. Also checked for port issues and there are none. Nothing listens on 9095.
One thing that does concern me is that the docs sau changes in settings require "restarting prometheus", but not what command does that. Given that there are no systemd units to address and that the orchestrator claims that there is no "prometheus service" even as it shows that there's 1 service, stopped, it's quite frustrating.
On 3/26/25 07:26, Eugen Block wrote:
The cephadm.log should show some details why it fails to deploy the daemon. If there's not much, look into the daemon logs as well (cephadm logs --name prometheus.ceph02.mousetech.com). Could it be that there's a non-cephadm prometheus already listening on port 9095?
Zitat von Tim Holloway <timh@mousetech.com>:
I finally got brave and migrated from Pacific to Reef, did some banging and hammering and for the first time in a long time got a complete "HEALTH OK" status.
However, the dashboard is still not happy. It cannot contact the Prometheus API on port 9095.
I have redeployed Prometheus multiple times without result.
I'm pretty sure that at one time there were no less than 3 different Prometheus containers running on one of the configured Prometheus servers, but now all I can get is the node-exporter.
Worse, if I do:
ceph orch reconfig prometheus
I get:
Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls"
But if I do:
ceph orch ls
I get:
prometheus ?:9095 0/1 - 116s ceph02.mousetech.com
Suggestions?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
service_type: prometheus service_name: prometheus placement: hosts: - dell02.mousetech.com networks: - 10.0.1.0/24 Can't list daemon logs, run restart usw., because "Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls"" And yet, ceph orch ls shows prometheus as a service. On 3/26/25 11:13, Eugen Block wrote:
ceph orch ls prometheus --export
There’s a service called „prometheus“, which can have multiple daemons, just like any other service (mon, mgr etc). To get the daemon logs you need to provide the daemon name (prometheus.ceph02.andsopn), not just the service name (prometheus). Can you run the cephadm command I provided? It should show something like I pasted in the previous message. Zitat von Tim Holloway <timh@mousetech.com>:
service_type: prometheus service_name: prometheus placement: hosts: - dell02.mousetech.com networks: - 10.0.1.0/24
Can't list daemon logs, run restart usw., because "Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls""
And yet, ceph orch ls shows prometheus as a service.
On 3/26/25 11:13, Eugen Block wrote:
ceph orch ls prometheus --export
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
it returns nothing. I'd already done the same via "systemctl | grep prometheus". There simply isn't a systemd service, even though there should be. On 3/26/25 11:31, Eugen Block wrote:
There’s a service called „prometheus“, which can have multiple daemons, just like any other service (mon, mgr etc). To get the daemon logs you need to provide the daemon name (prometheus.ceph02.andsopn), not just the service name (prometheus).
Can you run the cephadm command I provided? It should show something like I pasted in the previous message.
Zitat von Tim Holloway <timh@mousetech.com>:
service_type: prometheus service_name: prometheus placement: hosts: - dell02.mousetech.com networks: - 10.0.1.0/24
Can't list daemon logs, run restart usw., because "Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls""
And yet, ceph orch ls shows prometheus as a service.
On 3/26/25 11:13, Eugen Block wrote:
ceph orch ls prometheus --export
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Also, here are the currently-installed container images: [root@dell02 ~]# podman image ls REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/ceph/ceph <none> 2bc0b0f4375d 8 months ago 1.25 GB quay.io/ceph/ceph <none> 3c4eff6082ae 10 months ago 1.22 GB quay.io/ceph/ceph-grafana 9.4.7 954c08fa6188 15 months ago 647 MB quay.io/prometheus/alertmanager v0.25.0 c8568f914cd2 2 years ago 66.5 MB quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 2 years ago 23.9 MB quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 2 years ago 571 MB quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 3 years ago 22.3 MB quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 3 years ago 58.9 MB On 3/26/25 11:36, Tim Holloway wrote:
it returns nothing. I'd already done the same via "systemctl | grep prometheus". There simply isn't a systemd service, even though there should be.
On 3/26/25 11:31, Eugen Block wrote:
There’s a service called „prometheus“, which can have multiple daemons, just like any other service (mon, mgr etc). To get the daemon logs you need to provide the daemon name (prometheus.ceph02.andsopn), not just the service name (prometheus).
Can you run the cephadm command I provided? It should show something like I pasted in the previous message.
Zitat von Tim Holloway <timh@mousetech.com>:
service_type: prometheus service_name: prometheus placement: hosts: - dell02.mousetech.com networks: - 10.0.1.0/24
Can't list daemon logs, run restart usw., because "Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls""
And yet, ceph orch ls shows prometheus as a service.
On 3/26/25 11:13, Eugen Block wrote:
ceph orch ls prometheus --export
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Then maybe the deployment did fail and we’re back at looking into the cephadm.log. Zitat von Tim Holloway <timh@mousetech.com>:
it returns nothing. I'd already done the same via "systemctl | grep prometheus". There simply isn't a systemd service, even though there should be.
On 3/26/25 11:31, Eugen Block wrote:
There’s a service called „prometheus“, which can have multiple daemons, just like any other service (mon, mgr etc). To get the daemon logs you need to provide the daemon name (prometheus.ceph02.andsopn), not just the service name (prometheus).
Can you run the cephadm command I provided? It should show something like I pasted in the previous message.
Zitat von Tim Holloway <timh@mousetech.com>:
service_type: prometheus service_name: prometheus placement: hosts: - dell02.mousetech.com networks: - 10.0.1.0/24
Can't list daemon logs, run restart usw., because "Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls""
And yet, ceph orch ls shows prometheus as a service.
On 3/26/25 11:13, Eugen Block wrote:
ceph orch ls prometheus --export
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
One thing I did run into when upgrading was TLS issues pulling images. I had to set HTTP/S_PROXY and pull manually. That may relate to this: 025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme (mgr.18015288) 23874 : cephadm [INF] Saving service prometheus spec with placement dell02.mousetech.com 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme (mgr.18015288) 23875 : cephadm [INF] Saving service node-exporter spec with placement * 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme (mgr.18015288) 23876 : cephadm [INF] Saving service alertmanager spec with placement dell02.mousetech.com 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme (mgr.18015288) 23878 : cephadm [INF] Saving service grafana spec with placement dell02.mousetech.com 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme (mgr.18015288) 23879 : cephadm [INF] Saving service ceph-exporter spec with placement * 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme (mgr.18015288) 23898 : cephadm [INF] Restart service mgr 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi (mgr.18043792) 7 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTING 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi (mgr.18043792) 8 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on http://10.0.1.58:8765 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi (mgr.18043792) 9 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on https://10.0.1.58:7150 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi (mgr.18043792) 10 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTED 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi (mgr.18043792) 11 : cephadm [ERR] [26/Mar/2025:10:53:02] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi (mgr.18043792) 12 : cephadm [INF] Updating ceph03.internal.mousetech.com:/etc/ceph/ceph.conf On 3/26/25 11:39, Eugen Block wrote:
Then maybe the deployment did fail and we’re back at looking into the cephadm.log.
Zitat von Tim Holloway <timh@mousetech.com>:
it returns nothing. I'd already done the same via "systemctl | grep prometheus". There simply isn't a systemd service, even though there should be.
On 3/26/25 11:31, Eugen Block wrote:
There’s a service called „prometheus“, which can have multiple daemons, just like any other service (mon, mgr etc). To get the daemon logs you need to provide the daemon name (prometheus.ceph02.andsopn), not just the service name (prometheus).
Can you run the cephadm command I provided? It should show something like I pasted in the previous message.
Zitat von Tim Holloway <timh@mousetech.com>:
service_type: prometheus service_name: prometheus placement: hosts: - dell02.mousetech.com networks: - 10.0.1.0/24
Can't list daemon logs, run restart usw., because "Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls""
And yet, ceph orch ls shows prometheus as a service.
On 3/26/25 11:13, Eugen Block wrote:
ceph orch ls prometheus --export
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If you need a proxy to pull the images, I suggest to set it in the containers.conf: cat /etc/containers/containers.conf [engine] env = ["http_proxy=<host>:<port>", "https_proxy=<host>:<port>", "no_proxy=<your_no_proxy_list>"] But again, you should be able to see a failed to pull in the cephadm.log on dell02. Or even in 'ceph health detail', usually it warns you if the orchestrator failed to place a daemon. Zitat von Tim Holloway <timh@mousetech.com>:
One thing I did run into when upgrading was TLS issues pulling images. I had to set HTTP/S_PROXY and pull manually.
That may relate to this:
025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme (mgr.18015288) 23874 : cephadm [INF] Saving service prometheus spec with placement dell02.mousetech.com 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme (mgr.18015288) 23875 : cephadm [INF] Saving service node-exporter spec with placement * 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme (mgr.18015288) 23876 : cephadm [INF] Saving service alertmanager spec with placement dell02.mousetech.com 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme (mgr.18015288) 23878 : cephadm [INF] Saving service grafana spec with placement dell02.mousetech.com 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme (mgr.18015288) 23879 : cephadm [INF] Saving service ceph-exporter spec with placement * 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme (mgr.18015288) 23898 : cephadm [INF] Restart service mgr 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi (mgr.18043792) 7 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTING 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi (mgr.18043792) 8 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on http://10.0.1.58:8765 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi (mgr.18043792) 9 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on https://10.0.1.58:7150 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi (mgr.18043792) 10 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTED 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi (mgr.18043792) 11 : cephadm [ERR] [26/Mar/2025:10:53:02] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133)
2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi (mgr.18043792) 12 : cephadm [INF] Updating ceph03.internal.mousetech.com:/etc/ceph/ceph.conf
On 3/26/25 11:39, Eugen Block wrote:
Then maybe the deployment did fail and we’re back at looking into the cephadm.log.
Zitat von Tim Holloway <timh@mousetech.com>:
it returns nothing. I'd already done the same via "systemctl | grep prometheus". There simply isn't a systemd service, even though there should be.
On 3/26/25 11:31, Eugen Block wrote:
There’s a service called „prometheus“, which can have multiple daemons, just like any other service (mon, mgr etc). To get the daemon logs you need to provide the daemon name (prometheus.ceph02.andsopn), not just the service name (prometheus).
Can you run the cephadm command I provided? It should show something like I pasted in the previous message.
Zitat von Tim Holloway <timh@mousetech.com>:
service_type: prometheus service_name: prometheus placement: hosts: - dell02.mousetech.com networks: - 10.0.1.0/24
Can't list daemon logs, run restart usw., because "Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls""
And yet, ceph orch ls shows prometheus as a service.
On 3/26/25 11:13, Eugen Block wrote:
ceph orch ls prometheus --export
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Since the containers are all podman, I found a "systemctl edit podman" command that's recommended to set proxy for that. However, once I did, 2 OSDs went down and cannot be restarted. In any event, before I did that, ceph health detail was returning "HEALTH OK". Now I'm getting this: HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (865 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.3 on ceph06.internal.mousetech.com is in error state daemon osd.2 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (865 > max 560) On 3/26/25 12:07, Eugen Block wrote:
If you need a proxy to pull the images, I suggest to set it in the containers.conf:
cat /etc/containers/containers.conf [engine] env = ["http_proxy=<host>:<port>", "https_proxy=<host>:<port>", "no_proxy=<your_no_proxy_list>"]
But again, you should be able to see a failed to pull in the cephadm.log on dell02. Or even in 'ceph health detail', usually it warns you if the orchestrator failed to place a daemon.
Zitat von Tim Holloway <timh@mousetech.com>:
One thing I did run into when upgrading was TLS issues pulling images. I had to set HTTP/S_PROXY and pull manually.
That may relate to this:
025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme (mgr.18015288) 23874 : cephadm [INF] Saving service prometheus spec with placement dell02.mousetech.com 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme (mgr.18015288) 23875 : cephadm [INF] Saving service node-exporter spec with placement * 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme (mgr.18015288) 23876 : cephadm [INF] Saving service alertmanager spec with placement dell02.mousetech.com 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme (mgr.18015288) 23878 : cephadm [INF] Saving service grafana spec with placement dell02.mousetech.com 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme (mgr.18015288) 23879 : cephadm [INF] Saving service ceph-exporter spec with placement * 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme (mgr.18015288) 23898 : cephadm [INF] Restart service mgr 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi (mgr.18043792) 7 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTING 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi (mgr.18043792) 8 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on http://10.0.1.58:8765 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi (mgr.18043792) 9 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on https://10.0.1.58:7150 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi (mgr.18043792) 10 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTED 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi (mgr.18043792) 11 : cephadm [ERR] [26/Mar/2025:10:53:02] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133)
2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi (mgr.18043792) 12 : cephadm [INF] Updating ceph03.internal.mousetech.com:/etc/ceph/ceph.conf
On 3/26/25 11:39, Eugen Block wrote:
Then maybe the deployment did fail and we’re back at looking into the cephadm.log.
Zitat von Tim Holloway <timh@mousetech.com>:
it returns nothing. I'd already done the same via "systemctl | grep prometheus". There simply isn't a systemd service, even though there should be.
On 3/26/25 11:31, Eugen Block wrote:
There’s a service called „prometheus“, which can have multiple daemons, just like any other service (mon, mgr etc). To get the daemon logs you need to provide the daemon name (prometheus.ceph02.andsopn), not just the service name (prometheus).
Can you run the cephadm command I provided? It should show something like I pasted in the previous message.
Zitat von Tim Holloway <timh@mousetech.com>:
service_type: prometheus service_name: prometheus placement: hosts: - dell02.mousetech.com networks: - 10.0.1.0/24
Can't list daemon logs, run restart usw., because "Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls""
And yet, ceph orch ls shows prometheus as a service.
On 3/26/25 11:13, Eugen Block wrote: > ceph orch ls prometheus --export _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Right, systemctl edit works as well. But I'm confused about the down OSDs. Did you set the proxy on all hosts? Because the down OSDs are on ceph06 while prometheus is supposed to run on dell02. Are you sure those are related? I would recommend to remove the prometheus service entirely and start from scratch: ceph orch rm prometheus ceph mgr module disable prometheus ceph mgr fail Wait a minute, then enable it again and deploy prometheus: ceph orch apply -i prometheus.yaml ceph mgr module enable prometheus Zitat von Tim Holloway <timh@mousetech.com>:
Since the containers are all podman, I found a "systemctl edit podman" command that's recommended to set proxy for that.
However, once I did, 2 OSDs went down and cannot be restarted.
In any event, before I did that, ceph health detail was returning "HEALTH OK".
Now I'm getting this:
HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (865 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.3 on ceph06.internal.mousetech.com is in error state daemon osd.2 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (865 > max 560)
On 3/26/25 12:07, Eugen Block wrote:
If you need a proxy to pull the images, I suggest to set it in the containers.conf:
cat /etc/containers/containers.conf [engine] env = ["http_proxy=<host>:<port>", "https_proxy=<host>:<port>", "no_proxy=<your_no_proxy_list>"]
But again, you should be able to see a failed to pull in the cephadm.log on dell02. Or even in 'ceph health detail', usually it warns you if the orchestrator failed to place a daemon.
Zitat von Tim Holloway <timh@mousetech.com>:
One thing I did run into when upgrading was TLS issues pulling images. I had to set HTTP/S_PROXY and pull manually.
That may relate to this:
025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme (mgr.18015288) 23874 : cephadm [INF] Saving service prometheus spec with placement dell02.mousetech.com 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme (mgr.18015288) 23875 : cephadm [INF] Saving service node-exporter spec with placement * 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme (mgr.18015288) 23876 : cephadm [INF] Saving service alertmanager spec with placement dell02.mousetech.com 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme (mgr.18015288) 23878 : cephadm [INF] Saving service grafana spec with placement dell02.mousetech.com 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme (mgr.18015288) 23879 : cephadm [INF] Saving service ceph-exporter spec with placement * 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme (mgr.18015288) 23898 : cephadm [INF] Restart service mgr 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi (mgr.18043792) 7 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTING 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi (mgr.18043792) 8 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on http://10.0.1.58:8765 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi (mgr.18043792) 9 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on https://10.0.1.58:7150 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi (mgr.18043792) 10 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTED 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi (mgr.18043792) 11 : cephadm [ERR] [26/Mar/2025:10:53:02] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133)
2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi (mgr.18043792) 12 : cephadm [INF] Updating ceph03.internal.mousetech.com:/etc/ceph/ceph.conf
On 3/26/25 11:39, Eugen Block wrote:
Then maybe the deployment did fail and we’re back at looking into the cephadm.log.
Zitat von Tim Holloway <timh@mousetech.com>:
it returns nothing. I'd already done the same via "systemctl | grep prometheus". There simply isn't a systemd service, even though there should be.
On 3/26/25 11:31, Eugen Block wrote:
There’s a service called „prometheus“, which can have multiple daemons, just like any other service (mon, mgr etc). To get the daemon logs you need to provide the daemon name (prometheus.ceph02.andsopn), not just the service name (prometheus).
Can you run the cephadm command I provided? It should show something like I pasted in the previous message.
Zitat von Tim Holloway <timh@mousetech.com>:
> service_type: prometheus > service_name: prometheus > placement: > hosts: > - dell02.mousetech.com > networks: > - 10.0.1.0/24 > > Can't list daemon logs, run restart usw., because "Error > EINVAL: No daemons exist under service name "prometheus". View > currently running services using "ceph orch ls"" > > And yet, ceph orch ls shows prometheus as a service. > > On 3/26/25 11:13, Eugen Block wrote: >> ceph orch ls prometheus --export > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It's strange, but for a while I'd been trying to get prometheus working on ceph08, so I don't know. All I do know is immediately after editing the proxy settings I got indications that those 2 OSDs had gone down. What's REALLY strange is that their logs seem to hint that somehow they shifted from administered to legacy configuration. That is, looking for OSD resources under /var/lib/ceph instead of /var/lib/ceph/<fsid>. Anyway, I'll try yanking and re-deploying prometheus and maybe that will magically cure something. On 3/26/25 12:53, Eugen Block wrote:
Right, systemctl edit works as well. But I'm confused about the down OSDs. Did you set the proxy on all hosts? Because the down OSDs are on ceph06 while prometheus is supposed to run on dell02. Are you sure those are related?
I would recommend to remove the prometheus service entirely and start from scratch:
ceph orch rm prometheus ceph mgr module disable prometheus ceph mgr fail
Wait a minute, then enable it again and deploy prometheus:
ceph orch apply -i prometheus.yaml ceph mgr module enable prometheus
Zitat von Tim Holloway <timh@mousetech.com>:
Since the containers are all podman, I found a "systemctl edit podman" command that's recommended to set proxy for that.
However, once I did, 2 OSDs went down and cannot be restarted.
In any event, before I did that, ceph health detail was returning "HEALTH OK".
Now I'm getting this:
HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (865 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.3 on ceph06.internal.mousetech.com is in error state daemon osd.2 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (865 > max 560)
On 3/26/25 12:07, Eugen Block wrote:
If you need a proxy to pull the images, I suggest to set it in the containers.conf:
cat /etc/containers/containers.conf [engine] env = ["http_proxy=<host>:<port>", "https_proxy=<host>:<port>", "no_proxy=<your_no_proxy_list>"]
But again, you should be able to see a failed to pull in the cephadm.log on dell02. Or even in 'ceph health detail', usually it warns you if the orchestrator failed to place a daemon.
Zitat von Tim Holloway <timh@mousetech.com>:
One thing I did run into when upgrading was TLS issues pulling images. I had to set HTTP/S_PROXY and pull manually.
That may relate to this:
025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme (mgr.18015288) 23874 : cephadm [INF] Saving service prometheus spec with placement dell02.mousetech.com 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme (mgr.18015288) 23875 : cephadm [INF] Saving service node-exporter spec with placement * 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme (mgr.18015288) 23876 : cephadm [INF] Saving service alertmanager spec with placement dell02.mousetech.com 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme (mgr.18015288) 23878 : cephadm [INF] Saving service grafana spec with placement dell02.mousetech.com 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme (mgr.18015288) 23879 : cephadm [INF] Saving service ceph-exporter spec with placement * 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme (mgr.18015288) 23898 : cephadm [INF] Restart service mgr 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi (mgr.18043792) 7 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTING 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi (mgr.18043792) 8 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on http://10.0.1.58:8765 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi (mgr.18043792) 9 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on https://10.0.1.58:7150 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi (mgr.18043792) 10 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTED 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi (mgr.18043792) 11 : cephadm [ERR] [26/Mar/2025:10:53:02] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133)
2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi (mgr.18043792) 12 : cephadm [INF] Updating ceph03.internal.mousetech.com:/etc/ceph/ceph.conf
On 3/26/25 11:39, Eugen Block wrote:
Then maybe the deployment did fail and we’re back at looking into the cephadm.log.
Zitat von Tim Holloway <timh@mousetech.com>:
it returns nothing. I'd already done the same via "systemctl | grep prometheus". There simply isn't a systemd service, even though there should be.
On 3/26/25 11:31, Eugen Block wrote: > There’s a service called „prometheus“, which can have multiple > daemons, just like any other service (mon, mgr etc). To get the > daemon logs you need to provide the daemon name > (prometheus.ceph02.andsopn), not just the service name > (prometheus). > > Can you run the cephadm command I provided? It should show > something like I pasted in the previous message. > > Zitat von Tim Holloway <timh@mousetech.com>: > >> service_type: prometheus >> service_name: prometheus >> placement: >> hosts: >> - dell02.mousetech.com >> networks: >> - 10.0.1.0/24 >> >> Can't list daemon logs, run restart usw., because "Error >> EINVAL: No daemons exist under service name "prometheus". View >> currently running services using "ceph orch ls"" >> >> And yet, ceph orch ls shows prometheus as a service. >> >> On 3/26/25 11:13, Eugen Block wrote: >>> ceph orch ls prometheus --export >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
No change. On 3/26/25 13:01, Tim Holloway wrote:
It's strange, but for a while I'd been trying to get prometheus working on ceph08, so I don't know.
All I do know is immediately after editing the proxy settings I got indications that those 2 OSDs had gone down.
What's REALLY strange is that their logs seem to hint that somehow they shifted from administered to legacy configuration. That is, looking for OSD resources under /var/lib/ceph instead of /var/lib/ceph/<fsid>.
Anyway, I'll try yanking and re-deploying prometheus and maybe that will magically cure something.
On 3/26/25 12:53, Eugen Block wrote:
Right, systemctl edit works as well. But I'm confused about the down OSDs. Did you set the proxy on all hosts? Because the down OSDs are on ceph06 while prometheus is supposed to run on dell02. Are you sure those are related?
I would recommend to remove the prometheus service entirely and start from scratch:
ceph orch rm prometheus ceph mgr module disable prometheus ceph mgr fail
Wait a minute, then enable it again and deploy prometheus:
ceph orch apply -i prometheus.yaml ceph mgr module enable prometheus
Zitat von Tim Holloway <timh@mousetech.com>:
Since the containers are all podman, I found a "systemctl edit podman" command that's recommended to set proxy for that.
However, once I did, 2 OSDs went down and cannot be restarted.
In any event, before I did that, ceph health detail was returning "HEALTH OK".
Now I'm getting this:
HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (865 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.3 on ceph06.internal.mousetech.com is in error state daemon osd.2 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (865 > max 560)
On 3/26/25 12:07, Eugen Block wrote:
If you need a proxy to pull the images, I suggest to set it in the containers.conf:
cat /etc/containers/containers.conf [engine] env = ["http_proxy=<host>:<port>", "https_proxy=<host>:<port>", "no_proxy=<your_no_proxy_list>"]
But again, you should be able to see a failed to pull in the cephadm.log on dell02. Or even in 'ceph health detail', usually it warns you if the orchestrator failed to place a daemon.
Zitat von Tim Holloway <timh@mousetech.com>:
One thing I did run into when upgrading was TLS issues pulling images. I had to set HTTP/S_PROXY and pull manually.
That may relate to this:
025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme (mgr.18015288) 23874 : cephadm [INF] Saving service prometheus spec with placement dell02.mousetech.com 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme (mgr.18015288) 23875 : cephadm [INF] Saving service node-exporter spec with placement * 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme (mgr.18015288) 23876 : cephadm [INF] Saving service alertmanager spec with placement dell02.mousetech.com 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme (mgr.18015288) 23878 : cephadm [INF] Saving service grafana spec with placement dell02.mousetech.com 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme (mgr.18015288) 23879 : cephadm [INF] Saving service ceph-exporter spec with placement * 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme (mgr.18015288) 23898 : cephadm [INF] Restart service mgr 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi (mgr.18043792) 7 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTING 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi (mgr.18043792) 8 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on http://10.0.1.58:8765 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi (mgr.18043792) 9 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on https://10.0.1.58:7150 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi (mgr.18043792) 10 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTED 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi (mgr.18043792) 11 : cephadm [ERR] [26/Mar/2025:10:53:02] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133)
2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi (mgr.18043792) 12 : cephadm [INF] Updating ceph03.internal.mousetech.com:/etc/ceph/ceph.conf
On 3/26/25 11:39, Eugen Block wrote:
Then maybe the deployment did fail and we’re back at looking into the cephadm.log.
Zitat von Tim Holloway <timh@mousetech.com>:
> it returns nothing. I'd already done the same via "systemctl | > grep prometheus". There simply isn't a systemd service, even > though there should be. > > On 3/26/25 11:31, Eugen Block wrote: >> There’s a service called „prometheus“, which can have multiple >> daemons, just like any other service (mon, mgr etc). To get the >> daemon logs you need to provide the daemon name >> (prometheus.ceph02.andsopn), not just the service name >> (prometheus). >> >> Can you run the cephadm command I provided? It should show >> something like I pasted in the previous message. >> >> Zitat von Tim Holloway <timh@mousetech.com>: >> >>> service_type: prometheus >>> service_name: prometheus >>> placement: >>> hosts: >>> - dell02.mousetech.com >>> networks: >>> - 10.0.1.0/24 >>> >>> Can't list daemon logs, run restart usw., because "Error >>> EINVAL: No daemons exist under service name "prometheus". View >>> currently running services using "ceph orch ls"" >>> >>> And yet, ceph orch ls shows prometheus as a service. >>> >>> On 3/26/25 11:13, Eugen Block wrote: >>>> ceph orch ls prometheus --export >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ok, I'll try one last time and ask for cephadm.log output. ;-) And the active MGR's log might help here as well. Zitat von Tim Holloway <timh@mousetech.com>:
No change.
On 3/26/25 13:01, Tim Holloway wrote:
It's strange, but for a while I'd been trying to get prometheus working on ceph08, so I don't know.
All I do know is immediately after editing the proxy settings I got indications that those 2 OSDs had gone down.
What's REALLY strange is that their logs seem to hint that somehow they shifted from administered to legacy configuration. That is, looking for OSD resources under /var/lib/ceph instead of /var/lib/ceph/<fsid>.
Anyway, I'll try yanking and re-deploying prometheus and maybe that will magically cure something.
On 3/26/25 12:53, Eugen Block wrote:
Right, systemctl edit works as well. But I'm confused about the down OSDs. Did you set the proxy on all hosts? Because the down OSDs are on ceph06 while prometheus is supposed to run on dell02. Are you sure those are related?
I would recommend to remove the prometheus service entirely and start from scratch:
ceph orch rm prometheus ceph mgr module disable prometheus ceph mgr fail
Wait a minute, then enable it again and deploy prometheus:
ceph orch apply -i prometheus.yaml ceph mgr module enable prometheus
Zitat von Tim Holloway <timh@mousetech.com>:
Since the containers are all podman, I found a "systemctl edit podman" command that's recommended to set proxy for that.
However, once I did, 2 OSDs went down and cannot be restarted.
In any event, before I did that, ceph health detail was returning "HEALTH OK".
Now I'm getting this:
HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (865 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.3 on ceph06.internal.mousetech.com is in error state daemon osd.2 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (865 > max 560)
On 3/26/25 12:07, Eugen Block wrote:
If you need a proxy to pull the images, I suggest to set it in the containers.conf:
cat /etc/containers/containers.conf [engine] env = ["http_proxy=<host>:<port>", "https_proxy=<host>:<port>", "no_proxy=<your_no_proxy_list>"]
But again, you should be able to see a failed to pull in the cephadm.log on dell02. Or even in 'ceph health detail', usually it warns you if the orchestrator failed to place a daemon.
Zitat von Tim Holloway <timh@mousetech.com>:
One thing I did run into when upgrading was TLS issues pulling images. I had to set HTTP/S_PROXY and pull manually.
That may relate to this:
025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme (mgr.18015288) 23874 : cephadm [INF] Saving service prometheus spec with placement dell02.mousetech.com 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme (mgr.18015288) 23875 : cephadm [INF] Saving service node-exporter spec with placement * 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme (mgr.18015288) 23876 : cephadm [INF] Saving service alertmanager spec with placement dell02.mousetech.com 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme (mgr.18015288) 23878 : cephadm [INF] Saving service grafana spec with placement dell02.mousetech.com 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme (mgr.18015288) 23879 : cephadm [INF] Saving service ceph-exporter spec with placement * 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme (mgr.18015288) 23898 : cephadm [INF] Restart service mgr 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi (mgr.18043792) 7 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTING 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi (mgr.18043792) 8 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on http://10.0.1.58:8765 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi (mgr.18043792) 9 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on https://10.0.1.58:7150 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi (mgr.18043792) 10 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTED 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi (mgr.18043792) 11 : cephadm [ERR] [26/Mar/2025:10:53:02] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133)
2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi (mgr.18043792) 12 : cephadm [INF] Updating ceph03.internal.mousetech.com:/etc/ceph/ceph.conf
On 3/26/25 11:39, Eugen Block wrote: > Then maybe the deployment did fail and we’re back at looking > into the cephadm.log. > > > Zitat von Tim Holloway <timh@mousetech.com>: > >> it returns nothing. I'd already done the same via "systemctl >> | grep prometheus". There simply isn't a systemd service, >> even though there should be. >> >> On 3/26/25 11:31, Eugen Block wrote: >>> There’s a service called „prometheus“, which can have >>> multiple daemons, just like any other service (mon, mgr >>> etc). To get the daemon logs you need to provide the daemon >>> name (prometheus.ceph02.andsopn), not just the service name >>> (prometheus). >>> >>> Can you run the cephadm command I provided? It should show >>> something like I pasted in the previous message. >>> >>> Zitat von Tim Holloway <timh@mousetech.com>: >>> >>>> service_type: prometheus >>>> service_name: prometheus >>>> placement: >>>> hosts: >>>> - dell02.mousetech.com >>>> networks: >>>> - 10.0.1.0/24 >>>> >>>> Can't list daemon logs, run restart usw., because "Error >>>> EINVAL: No daemons exist under service name "prometheus". >>>> View currently running services using "ceph orch ls"" >>>> >>>> And yet, ceph orch ls shows prometheus as a service. >>>> >>>> On 3/26/25 11:13, Eugen Block wrote: >>>>> ceph orch ls prometheus --export >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Well, here's an excerpt from the /var/log/ceph/cephadm.log. I don't know if that's the mechanism or file you mean, though. 2025-03-26 13:11:09,382 7fb2abc38740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:12:10,219 7fc4fd405740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:13:11,502 7f2ef3c76740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:14:12,372 7f3566bef740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:15:13,301 7f660e204740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:15:20,880 7f93b227e740 DEBUG -------------------------------------------------------------------------------- cephadm ['ls'] 2025-03-26 13:15:20,904 7f93b227e740 DEBUG /usr/bin/podman: 5.2.2 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 2149e16fa2ce,11.51MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 65529d6ad1ac,17.69MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 51b1d190dfb9,99.79MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 59a865e3bcc5,6.791MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: dd3203f6f3bb,410.2MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 34177c4e5761,1.764GB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: bfe17e83b288,534.2MB / 33.24GB 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 2149e16fa2ce,0.00% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 65529d6ad1ac,0.26% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 51b1d190dfb9,0.22% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 59a865e3bcc5,0.02% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: dd3203f6f3bb,0.86% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 34177c4e5761,1.67% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: bfe17e83b288,0.25% 2025-03-26 13:15:20,985 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:20,993 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,024 7f93b227e740 DEBUG /usr/bin/podman: 2149e16fa2ce8769bf3be9e6e25eec61b8e027b0e8699f1cb7d5f113fc4aac66,quay.io/prometheus/node-exporter:v1.5.0,0da6a335fe1356545476b749c68f022c897d e3a2139e8f0054f6937349ee2b83,2025-03-25 16:52:31.644234532 -0400 EDT, 2025-03-26 13:15:21,057 7f93b227e740 DEBUG /usr/bin/podman: [quay.io/prometheus/node-exporter@sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c quay.io/prometheus/node-exporter@sh a256:fa8e5700b7762fffe0674e944762f44bb787a7e44d97569fe55348260453bf80] 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959) 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: build user: root@6e7732a7b81b 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: build date: 20221129-18:59:09 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: go version: go1.19.3 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: platform: linux/amd64 2025-03-26 13:15:21,187 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,196 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,228 7f93b227e740 DEBUG /usr/bin/podman: 59a865e3bcc5e86f6caed8278aec0cfed608bf89ff4953dfb48b762138955925,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.731849052 -0400 EDT, 2025-03-26 13:15:21,260 7f93b227e740 DEBUG /usr/bin/podman: [quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace85 53330e454152b82f95a2b6bf33c3f3ec2eeac77] 2025-03-26 13:15:21,385 7f93b227e740 DEBUG /usr/bin/podman: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) :2025-03-26 13:15:21,412 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,421 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,451 7f93b227e740 DEBUG /usr/bin/podman: bfe17e83b28821be0ec399cde79965ade3bc3377c5acf05ef047395ddde4d804,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-26 06:53:07.022104802 -0400 EDT, 2025-03-26 13:15:21,464 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,472 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,504 7f93b227e740 DEBUG /usr/bin/podman: 51b1d190dfb9a1db73b8efda020c54df4c339abce8973b8e0d6de2a2b780aa09,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.726614643 -0400 EDT, 2025-03-26 13:15:21,516 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,524 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,557 7f93b227e740 DEBUG /usr/bin/podman: dd3203f6f3bb3876ea35d8732c01211bb9cc79bff2258a7d63f425eb00e0221d,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.898369305 -0400 EDT, 2025-03-26 13:15:21,570 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,579 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,611 7f93b227e740 DEBUG /usr/bin/podman: 34177c4e5761c9b1e232a7f4a854fa1c8fe187253503265998c9cadd2cb7625c,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:33.635739799 -0400 EDT, 2025-03-26 13:15:21,623 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,632 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,662 7f93b227e740 DEBUG /usr/bin/podman: 65529d6ad1ac3c639ef699c2eed01b6a440e27925d5bccd5fb0eef50b283dab3,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.726574789 -0400 EDT, 2025-03-26 13:16:14,190 7fa738df4740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:17:15,057 7f906b406740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:18:15,951 7f3141a37740 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906', '--no-container-init', '--timeout', '895', 'ls'] 2025-03-26 13:18:17,047 7feb94c28740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:19:18,797 7f23b641f740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:20:19,681 7f270b666740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:21:20,566 7fcd77be8740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'check-host'] 2025-03-26 13:21:20,595 7fcd77be8740 INFO podman (/usr/bin/podman) version 5.2.2 is present 2025-03-26 13:21:20,595 7fcd77be8740 INFO systemctl is present 2025-03-26 13:21:20,596 7fcd77be8740 INFO lvcreate is present 2025-03-26 13:21:20,635 7fcd77be8740 INFO Unit chronyd.service is enabled and running 2025-03-26 13:21:20,635 7fcd77be8740 INFO Host looks OK 2025-03-26 13:21:21,016 7f670722d740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:22:21,860 7f29c27cb740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:23:23,116 7f41cdc5d740 DEBUG -------------------------------------------------------------------------------- : plus more of the same. The mgr log for dell02 isn't very exciting except for frequent exceptions where the dashboard cannot contact prometheus. Is there a place I could post complete files without filling up the mailing list? On 3/26/25 13:23, Eugen Block wrote:
Ok, I'll try one last time and ask for cephadm.log output. ;-) And the active MGR's log might help here as well.
Zitat von Tim Holloway <timh@mousetech.com>:
No change.
On 3/26/25 13:01, Tim Holloway wrote:
It's strange, but for a while I'd been trying to get prometheus working on ceph08, so I don't know.
All I do know is immediately after editing the proxy settings I got indications that those 2 OSDs had gone down.
What's REALLY strange is that their logs seem to hint that somehow they shifted from administered to legacy configuration. That is, looking for OSD resources under /var/lib/ceph instead of /var/lib/ceph/<fsid>.
Anyway, I'll try yanking and re-deploying prometheus and maybe that will magically cure something.
On 3/26/25 12:53, Eugen Block wrote:
Right, systemctl edit works as well. But I'm confused about the down OSDs. Did you set the proxy on all hosts? Because the down OSDs are on ceph06 while prometheus is supposed to run on dell02. Are you sure those are related?
I would recommend to remove the prometheus service entirely and start from scratch:
ceph orch rm prometheus ceph mgr module disable prometheus ceph mgr fail
Wait a minute, then enable it again and deploy prometheus:
ceph orch apply -i prometheus.yaml ceph mgr module enable prometheus
Zitat von Tim Holloway <timh@mousetech.com>:
Since the containers are all podman, I found a "systemctl edit podman" command that's recommended to set proxy for that.
However, once I did, 2 OSDs went down and cannot be restarted.
In any event, before I did that, ceph health detail was returning "HEALTH OK".
Now I'm getting this:
HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (865 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.3 on ceph06.internal.mousetech.com is in error state daemon osd.2 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (865 > max 560)
On 3/26/25 12:07, Eugen Block wrote:
If you need a proxy to pull the images, I suggest to set it in the containers.conf:
cat /etc/containers/containers.conf [engine] env = ["http_proxy=<host>:<port>", "https_proxy=<host>:<port>", "no_proxy=<your_no_proxy_list>"]
But again, you should be able to see a failed to pull in the cephadm.log on dell02. Or even in 'ceph health detail', usually it warns you if the orchestrator failed to place a daemon.
Zitat von Tim Holloway <timh@mousetech.com>:
> One thing I did run into when upgrading was TLS issues pulling > images. I had to set HTTP/S_PROXY and pull manually. > > That may relate to this: > > 025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme (mgr.18015288) > 23874 : cephadm [INF] Saving service prometheus spec with > placement dell02.mousetech.com > 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme (mgr.18015288) > 23875 : cephadm [INF] Saving service node-exporter spec with > placement * > 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme (mgr.18015288) > 23876 : cephadm [INF] Saving service alertmanager spec with > placement dell02.mousetech.com > 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme (mgr.18015288) > 23878 : cephadm [INF] Saving service grafana spec with placement > dell02.mousetech.com > 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme (mgr.18015288) > 23879 : cephadm [INF] Saving service ceph-exporter spec with > placement * > 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme (mgr.18015288) > 23898 : cephadm [INF] Restart service mgr > 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi (mgr.18043792) > 7 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTING > 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi (mgr.18043792) > 8 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on > http://10.0.1.58:8765 > 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi (mgr.18043792) > 9 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Serving on > https://10.0.1.58:7150 > 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi (mgr.18043792) > 10 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE Bus STARTED > 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi (mgr.18043792) > 11 : cephadm [ERR] [26/Mar/2025:10:53:02] ENGINE Error in > HTTPServer.serve > Traceback (most recent call last): > File "/lib/python3.9/site-packages/cheroot/server.py", line > 1823, in serve > self._connections.run(self.expiration_interval) > File "/lib/python3.9/site-packages/cheroot/connections.py", > line 203, in run > self._run(expiration_interval) > File "/lib/python3.9/site-packages/cheroot/connections.py", > line 246, in _run > new_conn = self._from_server_socket(self.server.socket) > File "/lib/python3.9/site-packages/cheroot/connections.py", > line 300, in _from_server_socket > s, ssl_env = self.server.ssl_adapter.wrap(s) > File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", > line 277, in wrap > s = self.context.wrap_socket( > File "/lib64/python3.9/ssl.py", line 501, in wrap_socket > return self.sslsocket_class._create( > File "/lib64/python3.9/ssl.py", line 1074, in _create > self.do_handshake() > File "/lib64/python3.9/ssl.py", line 1343, in do_handshake > self._sslobj.do_handshake() > ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) > (_ssl.c:1133) > > 2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi (mgr.18043792) > 12 : cephadm [INF] Updating > ceph03.internal.mousetech.com:/etc/ceph/ceph.conf > > On 3/26/25 11:39, Eugen Block wrote: >> Then maybe the deployment did fail and we’re back at looking >> into the cephadm.log. >> >> >> Zitat von Tim Holloway <timh@mousetech.com>: >> >>> it returns nothing. I'd already done the same via "systemctl | >>> grep prometheus". There simply isn't a systemd service, even >>> though there should be. >>> >>> On 3/26/25 11:31, Eugen Block wrote: >>>> There’s a service called „prometheus“, which can have >>>> multiple daemons, just like any other service (mon, mgr etc). >>>> To get the daemon logs you need to provide the daemon name >>>> (prometheus.ceph02.andsopn), not just the service name >>>> (prometheus). >>>> >>>> Can you run the cephadm command I provided? It should show >>>> something like I pasted in the previous message. >>>> >>>> Zitat von Tim Holloway <timh@mousetech.com>: >>>> >>>>> service_type: prometheus >>>>> service_name: prometheus >>>>> placement: >>>>> hosts: >>>>> - dell02.mousetech.com >>>>> networks: >>>>> - 10.0.1.0/24 >>>>> >>>>> Can't list daemon logs, run restart usw., because "Error >>>>> EINVAL: No daemons exist under service name "prometheus". >>>>> View currently running services using "ceph orch ls"" >>>>> >>>>> And yet, ceph orch ls shows prometheus as a service. >>>>> >>>>> On 3/26/25 11:13, Eugen Block wrote: >>>>>> ceph orch ls prometheus --export >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That would be the correct log file, but I don't see an attempt to deploy a prometheus instance there. You can use any pastebin you like, e. g. https://pastebin.com/ to upload your logs. Mask any sensitive data before you do that. Zitat von Tim Holloway <timh@mousetech.com>:
Well, here's an excerpt from the /var/log/ceph/cephadm.log. I don't know if that's the mechanism or file you mean, though.
2025-03-26 13:11:09,382 7fb2abc38740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:12:10,219 7fc4fd405740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:13:11,502 7f2ef3c76740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:14:12,372 7f3566bef740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:15:13,301 7f660e204740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:15:20,880 7f93b227e740 DEBUG -------------------------------------------------------------------------------- cephadm ['ls'] 2025-03-26 13:15:20,904 7f93b227e740 DEBUG /usr/bin/podman: 5.2.2 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 2149e16fa2ce,11.51MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 65529d6ad1ac,17.69MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 51b1d190dfb9,99.79MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 59a865e3bcc5,6.791MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: dd3203f6f3bb,410.2MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 34177c4e5761,1.764GB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: bfe17e83b288,534.2MB / 33.24GB 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 2149e16fa2ce,0.00% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 65529d6ad1ac,0.26% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 51b1d190dfb9,0.22% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 59a865e3bcc5,0.02% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: dd3203f6f3bb,0.86% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 34177c4e5761,1.67% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: bfe17e83b288,0.25% 2025-03-26 13:15:20,985 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:20,993 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,024 7f93b227e740 DEBUG /usr/bin/podman: 2149e16fa2ce8769bf3be9e6e25eec61b8e027b0e8699f1cb7d5f113fc4aac66,quay.io/prometheus/node-exporter:v1.5.0,0da6a335fe1356545476b749c68f022c897d e3a2139e8f0054f6937349ee2b83,2025-03-25 16:52:31.644234532 -0400 EDT, 2025-03-26 13:15:21,057 7f93b227e740 DEBUG /usr/bin/podman: [quay.io/prometheus/node-exporter@sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c quay.io/prometheus/node-exporter@sh a256:fa8e5700b7762fffe0674e944762f44bb787a7e44d97569fe55348260453bf80] 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959) 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: build user: root@6e7732a7b81b 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: build date: 20221129-18:59:09 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: go version: go1.19.3 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: platform: linux/amd64 2025-03-26 13:15:21,187 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,196 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,228 7f93b227e740 DEBUG /usr/bin/podman: 59a865e3bcc5e86f6caed8278aec0cfed608bf89ff4953dfb48b762138955925,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.731849052 -0400 EDT, 2025-03-26 13:15:21,260 7f93b227e740 DEBUG /usr/bin/podman: [quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace85 53330e454152b82f95a2b6bf33c3f3ec2eeac77] 2025-03-26 13:15:21,385 7f93b227e740 DEBUG /usr/bin/podman: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) :2025-03-26 13:15:21,412 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,421 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,451 7f93b227e740 DEBUG /usr/bin/podman: bfe17e83b28821be0ec399cde79965ade3bc3377c5acf05ef047395ddde4d804,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-26 06:53:07.022104802 -0400 EDT, 2025-03-26 13:15:21,464 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,472 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,504 7f93b227e740 DEBUG /usr/bin/podman: 51b1d190dfb9a1db73b8efda020c54df4c339abce8973b8e0d6de2a2b780aa09,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.726614643 -0400 EDT, 2025-03-26 13:15:21,516 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,524 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,557 7f93b227e740 DEBUG /usr/bin/podman: dd3203f6f3bb3876ea35d8732c01211bb9cc79bff2258a7d63f425eb00e0221d,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.898369305 -0400 EDT, 2025-03-26 13:15:21,570 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,579 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,611 7f93b227e740 DEBUG /usr/bin/podman: 34177c4e5761c9b1e232a7f4a854fa1c8fe187253503265998c9cadd2cb7625c,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:33.635739799 -0400 EDT, 2025-03-26 13:15:21,623 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,632 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,662 7f93b227e740 DEBUG /usr/bin/podman: 65529d6ad1ac3c639ef699c2eed01b6a440e27925d5bccd5fb0eef50b283dab3,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.726574789 -0400 EDT, 2025-03-26 13:16:14,190 7fa738df4740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:17:15,057 7f906b406740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:18:15,951 7f3141a37740 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906', '--no-container-init', '--timeout', '895', 'ls'] 2025-03-26 13:18:17,047 7feb94c28740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:19:18,797 7f23b641f740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:20:19,681 7f270b666740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:21:20,566 7fcd77be8740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'check-host'] 2025-03-26 13:21:20,595 7fcd77be8740 INFO podman (/usr/bin/podman) version 5.2.2 is present 2025-03-26 13:21:20,595 7fcd77be8740 INFO systemctl is present 2025-03-26 13:21:20,596 7fcd77be8740 INFO lvcreate is present 2025-03-26 13:21:20,635 7fcd77be8740 INFO Unit chronyd.service is enabled and running 2025-03-26 13:21:20,635 7fcd77be8740 INFO Host looks OK 2025-03-26 13:21:21,016 7f670722d740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:22:21,860 7f29c27cb740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:23:23,116 7f41cdc5d740 DEBUG -------------------------------------------------------------------------------- : plus more of the same.
The mgr log for dell02 isn't very exciting except for frequent exceptions where the dashboard cannot contact prometheus.
Is there a place I could post complete files without filling up the mailing list?
On 3/26/25 13:23, Eugen Block wrote:
Ok, I'll try one last time and ask for cephadm.log output. ;-) And the active MGR's log might help here as well.
Zitat von Tim Holloway <timh@mousetech.com>:
No change.
On 3/26/25 13:01, Tim Holloway wrote:
It's strange, but for a while I'd been trying to get prometheus working on ceph08, so I don't know.
All I do know is immediately after editing the proxy settings I got indications that those 2 OSDs had gone down.
What's REALLY strange is that their logs seem to hint that somehow they shifted from administered to legacy configuration. That is, looking for OSD resources under /var/lib/ceph instead of /var/lib/ceph/<fsid>.
Anyway, I'll try yanking and re-deploying prometheus and maybe that will magically cure something.
On 3/26/25 12:53, Eugen Block wrote:
Right, systemctl edit works as well. But I'm confused about the down OSDs. Did you set the proxy on all hosts? Because the down OSDs are on ceph06 while prometheus is supposed to run on dell02. Are you sure those are related?
I would recommend to remove the prometheus service entirely and start from scratch:
ceph orch rm prometheus ceph mgr module disable prometheus ceph mgr fail
Wait a minute, then enable it again and deploy prometheus:
ceph orch apply -i prometheus.yaml ceph mgr module enable prometheus
Zitat von Tim Holloway <timh@mousetech.com>:
Since the containers are all podman, I found a "systemctl edit podman" command that's recommended to set proxy for that.
However, once I did, 2 OSDs went down and cannot be restarted.
In any event, before I did that, ceph health detail was returning "HEALTH OK".
Now I'm getting this:
HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (865 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.3 on ceph06.internal.mousetech.com is in error state daemon osd.2 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (865 > max 560)
On 3/26/25 12:07, Eugen Block wrote: > If you need a proxy to pull the images, I suggest to set it in > the containers.conf: > > cat /etc/containers/containers.conf > [engine] > env = ["http_proxy=<host>:<port>", > "https_proxy=<host>:<port>", "no_proxy=<your_no_proxy_list>"] > > But again, you should be able to see a failed to pull in the > cephadm.log on dell02. Or even in 'ceph health detail', > usually it warns you if the orchestrator failed to place a > daemon. > > Zitat von Tim Holloway <timh@mousetech.com>: > >> One thing I did run into when upgrading was TLS issues >> pulling images. I had to set HTTP/S_PROXY and pull manually. >> >> That may relate to this: >> >> 025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme >> (mgr.18015288) 23874 : cephadm [INF] Saving service >> prometheus spec with placement dell02.mousetech.com >> 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme >> (mgr.18015288) 23875 : cephadm [INF] Saving service >> node-exporter spec with placement * >> 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme >> (mgr.18015288) 23876 : cephadm [INF] Saving service >> alertmanager spec with placement dell02.mousetech.com >> 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme >> (mgr.18015288) 23878 : cephadm [INF] Saving service grafana >> spec with placement dell02.mousetech.com >> 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme >> (mgr.18015288) 23879 : cephadm [INF] Saving service >> ceph-exporter spec with placement * >> 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme >> (mgr.18015288) 23898 : cephadm [INF] Restart service mgr >> 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi >> (mgr.18043792) 7 : cephadm [INF] [26/Mar/2025:10:53:02] >> ENGINE Bus STARTING >> 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi >> (mgr.18043792) 8 : cephadm [INF] [26/Mar/2025:10:53:02] >> ENGINE Serving on http://10.0.1.58:8765 >> 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi >> (mgr.18043792) 9 : cephadm [INF] [26/Mar/2025:10:53:02] >> ENGINE Serving on https://10.0.1.58:7150 >> 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi >> (mgr.18043792) 10 : cephadm [INF] [26/Mar/2025:10:53:02] >> ENGINE Bus STARTED >> 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi >> (mgr.18043792) 11 : cephadm [ERR] [26/Mar/2025:10:53:02] >> ENGINE Error in HTTPServer.serve >> Traceback (most recent call last): >> File "/lib/python3.9/site-packages/cheroot/server.py", line >> 1823, in serve >> self._connections.run(self.expiration_interval) >> File "/lib/python3.9/site-packages/cheroot/connections.py", >> line 203, in run >> self._run(expiration_interval) >> File "/lib/python3.9/site-packages/cheroot/connections.py", >> line 246, in _run >> new_conn = self._from_server_socket(self.server.socket) >> File "/lib/python3.9/site-packages/cheroot/connections.py", >> line 300, in _from_server_socket >> s, ssl_env = self.server.ssl_adapter.wrap(s) >> File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", >> line 277, in wrap >> s = self.context.wrap_socket( >> File "/lib64/python3.9/ssl.py", line 501, in wrap_socket >> return self.sslsocket_class._create( >> File "/lib64/python3.9/ssl.py", line 1074, in _create >> self.do_handshake() >> File "/lib64/python3.9/ssl.py", line 1343, in do_handshake >> self._sslobj.do_handshake() >> ssl.SSLZeroReturnError: TLS/SSL connection has been closed >> (EOF) (_ssl.c:1133) >> >> 2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi >> (mgr.18043792) 12 : cephadm [INF] Updating >> ceph03.internal.mousetech.com:/etc/ceph/ceph.conf >> >> On 3/26/25 11:39, Eugen Block wrote: >>> Then maybe the deployment did fail and we’re back at looking >>> into the cephadm.log. >>> >>> >>> Zitat von Tim Holloway <timh@mousetech.com>: >>> >>>> it returns nothing. I'd already done the same via >>>> "systemctl | grep prometheus". There simply isn't a systemd >>>> service, even though there should be. >>>> >>>> On 3/26/25 11:31, Eugen Block wrote: >>>>> There’s a service called „prometheus“, which can have >>>>> multiple daemons, just like any other service (mon, mgr >>>>> etc). To get the daemon logs you need to provide the >>>>> daemon name (prometheus.ceph02.andsopn), not just the >>>>> service name (prometheus). >>>>> >>>>> Can you run the cephadm command I provided? It should show >>>>> something like I pasted in the previous message. >>>>> >>>>> Zitat von Tim Holloway <timh@mousetech.com>: >>>>> >>>>>> service_type: prometheus >>>>>> service_name: prometheus >>>>>> placement: >>>>>> hosts: >>>>>> - dell02.mousetech.com >>>>>> networks: >>>>>> - 10.0.1.0/24 >>>>>> >>>>>> Can't list daemon logs, run restart usw., because "Error >>>>>> EINVAL: No daemons exist under service name "prometheus". >>>>>> View currently running services using "ceph orch ls"" >>>>>> >>>>>> And yet, ceph orch ls shows prometheus as a service. >>>>>> >>>>>> On 3/26/25 11:13, Eugen Block wrote: >>>>>>> ceph orch ls prometheus --export >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
OK. I couldn't find a quick way to shovel a largish file from an internal server into pastebin, but my own servers can suffice. the URLs are: https://www.mousetech.com/share/cephadm.log https://www.mousetech.com/share/cephadm.log And I don't see a deployment either. On 3/26/25 14:26, Eugen Block wrote:
That would be the correct log file, but I don't see an attempt to deploy a prometheus instance there. You can use any pastebin you like, e. g. https://pastebin.com/ to upload your logs. Mask any sensitive data before you do that.
Zitat von Tim Holloway <timh@mousetech.com>:
Well, here's an excerpt from the /var/log/ceph/cephadm.log. I don't know if that's the mechanism or file you mean, though.
2025-03-26 13:11:09,382 7fb2abc38740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:12:10,219 7fc4fd405740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:13:11,502 7f2ef3c76740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:14:12,372 7f3566bef740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:15:13,301 7f660e204740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:15:20,880 7f93b227e740 DEBUG -------------------------------------------------------------------------------- cephadm ['ls'] 2025-03-26 13:15:20,904 7f93b227e740 DEBUG /usr/bin/podman: 5.2.2 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 2149e16fa2ce,11.51MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 65529d6ad1ac,17.69MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 51b1d190dfb9,99.79MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 59a865e3bcc5,6.791MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: dd3203f6f3bb,410.2MB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: 34177c4e5761,1.764GB / 33.24GB 2025-03-26 13:15:20,939 7f93b227e740 DEBUG /usr/bin/podman: bfe17e83b288,534.2MB / 33.24GB 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 2149e16fa2ce,0.00% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 65529d6ad1ac,0.26% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 51b1d190dfb9,0.22% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 59a865e3bcc5,0.02% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: dd3203f6f3bb,0.86% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: 34177c4e5761,1.67% 2025-03-26 13:15:20,972 7f93b227e740 DEBUG /usr/bin/podman: bfe17e83b288,0.25% 2025-03-26 13:15:20,985 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:20,993 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,024 7f93b227e740 DEBUG /usr/bin/podman: 2149e16fa2ce8769bf3be9e6e25eec61b8e027b0e8699f1cb7d5f113fc4aac66,quay.io/prometheus/node-exporter:v1.5.0,0da6a335fe1356545476b749c68f022c897d e3a2139e8f0054f6937349ee2b83,2025-03-25 16:52:31.644234532 -0400 EDT, 2025-03-26 13:15:21,057 7f93b227e740 DEBUG /usr/bin/podman: [quay.io/prometheus/node-exporter@sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c quay.io/prometheus/node-exporter@sh a256:fa8e5700b7762fffe0674e944762f44bb787a7e44d97569fe55348260453bf80] 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959) 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: build user: root@6e7732a7b81b 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: build date: 20221129-18:59:09 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: go version: go1.19.3 2025-03-26 13:15:21,111 7f93b227e740 DEBUG /usr/bin/podman: platform: linux/amd64 2025-03-26 13:15:21,187 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,196 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,228 7f93b227e740 DEBUG /usr/bin/podman: 59a865e3bcc5e86f6caed8278aec0cfed608bf89ff4953dfb48b762138955925,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.731849052 -0400 EDT, 2025-03-26 13:15:21,260 7f93b227e740 DEBUG /usr/bin/podman: [quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace85 53330e454152b82f95a2b6bf33c3f3ec2eeac77] 2025-03-26 13:15:21,385 7f93b227e740 DEBUG /usr/bin/podman: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) :2025-03-26 13:15:21,412 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,421 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,451 7f93b227e740 DEBUG /usr/bin/podman: bfe17e83b28821be0ec399cde79965ade3bc3377c5acf05ef047395ddde4d804,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-26 06:53:07.022104802 -0400 EDT, 2025-03-26 13:15:21,464 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,472 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,504 7f93b227e740 DEBUG /usr/bin/podman: 51b1d190dfb9a1db73b8efda020c54df4c339abce8973b8e0d6de2a2b780aa09,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.726614643 -0400 EDT, 2025-03-26 13:15:21,516 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,524 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,557 7f93b227e740 DEBUG /usr/bin/podman: dd3203f6f3bb3876ea35d8732c01211bb9cc79bff2258a7d63f425eb00e0221d,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.898369305 -0400 EDT, 2025-03-26 13:15:21,570 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,579 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,611 7f93b227e740 DEBUG /usr/bin/podman: 34177c4e5761c9b1e232a7f4a854fa1c8fe187253503265998c9cadd2cb7625c,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:33.635739799 -0400 EDT, 2025-03-26 13:15:21,623 7f93b227e740 DEBUG systemctl: enabled 2025-03-26 13:15:21,632 7f93b227e740 DEBUG systemctl: active 2025-03-26 13:15:21,662 7f93b227e740 DEBUG /usr/bin/podman: 65529d6ad1ac3c639ef699c2eed01b6a440e27925d5bccd5fb0eef50b283dab3,quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c31 72b0b23b37906,2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,2025-03-25 16:52:31.726574789 -0400 EDT, 2025-03-26 13:16:14,190 7fa738df4740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:17:15,057 7f906b406740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:18:15,951 7f3141a37740 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906', '--no-container-init', '--timeout', '895', 'ls'] 2025-03-26 13:18:17,047 7feb94c28740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:19:18,797 7f23b641f740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:20:19,681 7f270b666740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:21:20,566 7fcd77be8740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'check-host'] 2025-03-26 13:21:20,595 7fcd77be8740 INFO podman (/usr/bin/podman) version 5.2.2 is present 2025-03-26 13:21:20,595 7fcd77be8740 INFO systemctl is present 2025-03-26 13:21:20,596 7fcd77be8740 INFO lvcreate is present 2025-03-26 13:21:20,635 7fcd77be8740 INFO Unit chronyd.service is enabled and running 2025-03-26 13:21:20,635 7fcd77be8740 INFO Host looks OK 2025-03-26 13:21:21,016 7f670722d740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:22:21,860 7f29c27cb740 DEBUG -------------------------------------------------------------------------------- cephadm ['--no-container-init', '--timeout', '895', 'gather-facts'] 2025-03-26 13:23:23,116 7f41cdc5d740 DEBUG -------------------------------------------------------------------------------- : plus more of the same.
The mgr log for dell02 isn't very exciting except for frequent exceptions where the dashboard cannot contact prometheus.
Is there a place I could post complete files without filling up the mailing list?
On 3/26/25 13:23, Eugen Block wrote:
Ok, I'll try one last time and ask for cephadm.log output. ;-) And the active MGR's log might help here as well.
Zitat von Tim Holloway <timh@mousetech.com>:
No change.
On 3/26/25 13:01, Tim Holloway wrote:
It's strange, but for a while I'd been trying to get prometheus working on ceph08, so I don't know.
All I do know is immediately after editing the proxy settings I got indications that those 2 OSDs had gone down.
What's REALLY strange is that their logs seem to hint that somehow they shifted from administered to legacy configuration. That is, looking for OSD resources under /var/lib/ceph instead of /var/lib/ceph/<fsid>.
Anyway, I'll try yanking and re-deploying prometheus and maybe that will magically cure something.
On 3/26/25 12:53, Eugen Block wrote:
Right, systemctl edit works as well. But I'm confused about the down OSDs. Did you set the proxy on all hosts? Because the down OSDs are on ceph06 while prometheus is supposed to run on dell02. Are you sure those are related?
I would recommend to remove the prometheus service entirely and start from scratch:
ceph orch rm prometheus ceph mgr module disable prometheus ceph mgr fail
Wait a minute, then enable it again and deploy prometheus:
ceph orch apply -i prometheus.yaml ceph mgr module enable prometheus
Zitat von Tim Holloway <timh@mousetech.com>:
> Since the containers are all podman, I found a "systemctl edit > podman" command that's recommended to set proxy for that. > > However, once I did, 2 OSDs went down and cannot be restarted. > > In any event, before I did that, ceph health detail was > returning "HEALTH OK". > > Now I'm getting this: > > HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has > failed: gaierror(-2, 'Name or service not known'); too many PGs > per OSD (865 > max 560) > [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) > daemon osd.3 on ceph06.internal.mousetech.com is in error state > daemon osd.2 on ceph08.internal.mousetech.com is in error state > [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: > gaierror(-2, 'Name or service not known') > Module 'prometheus' has failed: gaierror(-2, 'Name or > service not known') > [WRN] TOO_MANY_PGS: too many PGs per OSD (865 > max 560) > > On 3/26/25 12:07, Eugen Block wrote: >> If you need a proxy to pull the images, I suggest to set it in >> the containers.conf: >> >> cat /etc/containers/containers.conf >> [engine] >> env = ["http_proxy=<host>:<port>", "https_proxy=<host>:<port>", >> "no_proxy=<your_no_proxy_list>"] >> >> But again, you should be able to see a failed to pull in the >> cephadm.log on dell02. Or even in 'ceph health detail', usually >> it warns you if the orchestrator failed to place a daemon. >> >> Zitat von Tim Holloway <timh@mousetech.com>: >> >>> One thing I did run into when upgrading was TLS issues pulling >>> images. I had to set HTTP/S_PROXY and pull manually. >>> >>> That may relate to this: >>> >>> 025-03-26T10:52:16.547985+0000 mgr.dell02.zwnrme >>> (mgr.18015288) 23874 : cephadm [INF] Saving service prometheus >>> spec with placement dell02.mousetech.com >>> 2025-03-26T10:52:16.560810+0000 mgr.dell02.zwnrme >>> (mgr.18015288) 23875 : cephadm [INF] Saving service >>> node-exporter spec with placement * >>> 2025-03-26T10:52:16.572380+0000 mgr.dell02.zwnrme >>> (mgr.18015288) 23876 : cephadm [INF] Saving service >>> alertmanager spec with placement dell02.mousetech.com >>> 2025-03-26T10:52:16.583555+0000 mgr.dell02.zwnrme >>> (mgr.18015288) 23878 : cephadm [INF] Saving service grafana >>> spec with placement dell02.mousetech.com >>> 2025-03-26T10:52:16.601713+0000 mgr.dell02.zwnrme >>> (mgr.18015288) 23879 : cephadm [INF] Saving service >>> ceph-exporter spec with placement * >>> 2025-03-26T10:52:44.139886+0000 mgr.dell02.zwnrme >>> (mgr.18015288) 23898 : cephadm [INF] Restart service mgr >>> 2025-03-26T10:53:02.720157+0000 mgr.ceph08.tlocfi >>> (mgr.18043792) 7 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE >>> Bus STARTING >>> 2025-03-26T10:53:02.824138+0000 mgr.ceph08.tlocfi >>> (mgr.18043792) 8 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE >>> Serving on http://10.0.1.58:8765 >>> 2025-03-26T10:53:02.962314+0000 mgr.ceph08.tlocfi >>> (mgr.18043792) 9 : cephadm [INF] [26/Mar/2025:10:53:02] ENGINE >>> Serving on https://10.0.1.58:7150 >>> 2025-03-26T10:53:02.962805+0000 mgr.ceph08.tlocfi >>> (mgr.18043792) 10 : cephadm [INF] [26/Mar/2025:10:53:02] >>> ENGINE Bus STARTED >>> 2025-03-26T10:53:02.964966+0000 mgr.ceph08.tlocfi >>> (mgr.18043792) 11 : cephadm [ERR] [26/Mar/2025:10:53:02] >>> ENGINE Error in HTTPServer.serve >>> Traceback (most recent call last): >>> File "/lib/python3.9/site-packages/cheroot/server.py", line >>> 1823, in serve >>> self._connections.run(self.expiration_interval) >>> File "/lib/python3.9/site-packages/cheroot/connections.py", >>> line 203, in run >>> self._run(expiration_interval) >>> File "/lib/python3.9/site-packages/cheroot/connections.py", >>> line 246, in _run >>> new_conn = self._from_server_socket(self.server.socket) >>> File "/lib/python3.9/site-packages/cheroot/connections.py", >>> line 300, in _from_server_socket >>> s, ssl_env = self.server.ssl_adapter.wrap(s) >>> File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", >>> line 277, in wrap >>> s = self.context.wrap_socket( >>> File "/lib64/python3.9/ssl.py", line 501, in wrap_socket >>> return self.sslsocket_class._create( >>> File "/lib64/python3.9/ssl.py", line 1074, in _create >>> self.do_handshake() >>> File "/lib64/python3.9/ssl.py", line 1343, in do_handshake >>> self._sslobj.do_handshake() >>> ssl.SSLZeroReturnError: TLS/SSL connection has been closed >>> (EOF) (_ssl.c:1133) >>> >>> 2025-03-26T10:53:03.471114+0000 mgr.ceph08.tlocfi >>> (mgr.18043792) 12 : cephadm [INF] Updating >>> ceph03.internal.mousetech.com:/etc/ceph/ceph.conf >>> >>> On 3/26/25 11:39, Eugen Block wrote: >>>> Then maybe the deployment did fail and we’re back at looking >>>> into the cephadm.log. >>>> >>>> >>>> Zitat von Tim Holloway <timh@mousetech.com>: >>>> >>>>> it returns nothing. I'd already done the same via "systemctl >>>>> | grep prometheus". There simply isn't a systemd service, >>>>> even though there should be. >>>>> >>>>> On 3/26/25 11:31, Eugen Block wrote: >>>>>> There’s a service called „prometheus“, which can have >>>>>> multiple daemons, just like any other service (mon, mgr >>>>>> etc). To get the daemon logs you need to provide the daemon >>>>>> name (prometheus.ceph02.andsopn), not just the service name >>>>>> (prometheus). >>>>>> >>>>>> Can you run the cephadm command I provided? It should show >>>>>> something like I pasted in the previous message. >>>>>> >>>>>> Zitat von Tim Holloway <timh@mousetech.com>: >>>>>> >>>>>>> service_type: prometheus >>>>>>> service_name: prometheus >>>>>>> placement: >>>>>>> hosts: >>>>>>> - dell02.mousetech.com >>>>>>> networks: >>>>>>> - 10.0.1.0/24 >>>>>>> >>>>>>> Can't list daemon logs, run restart usw., because "Error >>>>>>> EINVAL: No daemons exist under service name "prometheus". >>>>>>> View currently running services using "ceph orch ls"" >>>>>>> >>>>>>> And yet, ceph orch ls shows prometheus as a service. >>>>>>> >>>>>>> On 3/26/25 11:13, Eugen Block wrote: >>>>>>>> ceph orch ls prometheus --export >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry, duplicated a URL. The mgr log is https://www.mousetech.com/share/ceph-mgr.log
OSD mystery is solved. Both OSDs were LVM-based imported as vdisks for Ceph VMs. Apparently something scrambled either the VM manager or the host disk subsystem as the VM disks were getting I/O errors and even disappearing from the VM. I rebooted the physical machine and that cleared it. All OSDs now happy again. ... Well, it looks like one OSD has been damaged permanently, so I purged it. (: On 3/26/25 15:08, Tim Holloway wrote:
Sorry, duplicated a URL. The mgr log is
https://www.mousetech.com/share/ceph-mgr.log _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It gets worse. It looks like the physical disk backing the 2 failing OSDs is failing. I destroyed the host for one of them - which causes me to flash-back to the nightmare of having a deleted OSD get permanently stuck deleting just like in Pacific. Because I cannot restart the OSD, the deletion could not complete. The deleted host was a backup mds, I needed a new mds so I told the system to create one on the dell02 machine. I got the same behaviour as for prometheus. The dell02 machine shows in ceph orch ls as having an un-started mds, there's an empty mds logfile created, but no systemd units. And nothing in the cephadm log about the creation of the mds. The other cephadm log (/var/log/ceph/<fsid>/ceph.cephadm.log) indicates attempts to decommission the old (ceph06) mds, but that machine cannot be contacted as it no longer exists. I've posted yesterday's and today's ceph.cephadm.log: https://www.mousetech.com/share/ceph.cephadm.log-20250326.gz https://www.mousetech.com/share/ceph.cephadm.log Latest health report is dismal: HEALTH_ERR 1 failed cephadm daemon(s); 1 hosts fail cephadm check; insufficient standby MDS daemons available; 2 mgr modules have failed; too many PGs per OSD (648 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.3 on ceph06.internal.mousetech.com is in error state [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check host ceph06.internal.mousetech.com (10.0.1.56) failed check: Can't communicate with remote host `10.0.1.56`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('10.0.1.56', 22) [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more [ERR] MGR_MODULE_ERROR: 2 mgr modules have failed Module 'cephadm' has failed: 'ceph06.internal.mousetech.com' Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) On 3/26/25 16:55, Tim Holloway wrote:
OSD mystery is solved.
Both OSDs were LVM-based imported as vdisks for Ceph VMs. Apparently something scrambled either the VM manager or the host disk subsystem as the VM disks were getting I/O errors and even disappearing from the VM.
I rebooted the physical machine and that cleared it. All OSDs now happy again.
...
Well, it looks like one OSD has been damaged permanently, so I purged it. (:
On 3/26/25 15:08, Tim Holloway wrote:
Sorry, duplicated a URL. The mgr log is
https://www.mousetech.com/share/ceph-mgr.log _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There's a lot going on with your cluster... You seem to have broken the mgr, which is why you're not seeing any deployment attempts, I assume. Sometimes the MGR can be "broken" if some processes never finish, your missing ceph06 might cause that, it's hard to say since you've tried a lot of different things. According to the mgr log, you've set the dashboard prometheus api host to: http://ceph08.internal.mousetech.com:9095/api/v1 Since you tried to move it to dell02, which failed, I assume it's the reason why the prometheus module is broken. What you did with your OSDs, I don't fully understand, tbh. But we may ignore it for now, I hope. And just to have mentioned it: I strongly recommend to check your host removal procedure. It doesn't seem to be optimal to keep your cluster in a healthy state. If there's no chance to bring back ceph06.internal.mousetech.com, I'd probably remove it from the config-key store: ceph config-key ls | grep host.ceph06.internal.mousetech.com ceph config-key rm mgr/cephadm/host.ceph06.internal.mousetech.com ceph config-key rm mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0 (I'm just assuming that the keys will look like that.) Then I would disable the prometheus mgr module again (ceph mgr module disable prometheus), and I would probably also reset your prometheus-api-host: ceph dashboard reset-prometheus-api-host Then fail the mgr (ceph mgr fail) and wait a minute or two. If you don't mind, share the ceph status after you did those steps. And then we'll go from there. Zitat von Tim Holloway <timh@mousetech.com>:
It gets worse.
It looks like the physical disk backing the 2 failing OSDs is failing. I destroyed the host for one of them - which causes me to flash-back to the nightmare of having a deleted OSD get permanently stuck deleting just like in Pacific. Because I cannot restart the OSD, the deletion could not complete.
The deleted host was a backup mds, I needed a new mds so I told the system to create one on the dell02 machine. I got the same behaviour as for prometheus. The dell02 machine shows in ceph orch ls as having an un-started mds, there's an empty mds logfile created, but no systemd units. And nothing in the cephadm log about the creation of the mds.
The other cephadm log (/var/log/ceph/<fsid>/ceph.cephadm.log) indicates attempts to decommission the old (ceph06) mds, but that machine cannot be contacted as it no longer exists.
I've posted yesterday's and today's ceph.cephadm.log:
https://www.mousetech.com/share/ceph.cephadm.log-20250326.gz
https://www.mousetech.com/share/ceph.cephadm.log
Latest health report is dismal:
HEALTH_ERR 1 failed cephadm daemon(s); 1 hosts fail cephadm check; insufficient standby MDS daemons available; 2 mgr modules have failed; too many PGs per OSD (648 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.3 on ceph06.internal.mousetech.com is in error state [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check host ceph06.internal.mousetech.com (10.0.1.56) failed check: Can't communicate with remote host `10.0.1.56`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('10.0.1.56', 22) [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more [ERR] MGR_MODULE_ERROR: 2 mgr modules have failed Module 'cephadm' has failed: 'ceph06.internal.mousetech.com' Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
On 3/26/25 16:55, Tim Holloway wrote:
OSD mystery is solved.
Both OSDs were LVM-based imported as vdisks for Ceph VMs. Apparently something scrambled either the VM manager or the host disk subsystem as the VM disks were getting I/O errors and even disappearing from the VM.
I rebooted the physical machine and that cleared it. All OSDs now happy again.
...
Well, it looks like one OSD has been damaged permanently, so I purged it. (:
On 3/26/25 15:08, Tim Holloway wrote:
Sorry, duplicated a URL. The mgr log is
https://www.mousetech.com/share/ceph-mgr.log _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your patience. host ceph06 isn't referenced in the config database. I think I've finally purged it. I also reset the dashboard API host address from ceph08 to dell02. But since prometheus isn't running on dell02 either, there's no gain there. I did clear some of that lint out via "ceph mgr fail". So here's the latest. There are strange things happening at the base OS level that keep host ceph08 from running its OSDs anymore. At boot, device /dev/sdb suddenly changes to /dev/sdd (????) and there seem to be I/O errors. It's really strange, but I'm going to replace the physical drive and that will hopefully cure that. The problem is, reef and earlier releases seem to have a lot of trouble in deleting OSDs that aren't running. As I've noted before, they tend to get permanently stuck in the "deleting" state. When I cannot restart the OSD, the only cure for that has been to run around the system and apply brute force until things clear up. I did a dashboard purge of the OSDs on ceph08 and that removed them from the GUI (they'd already drained). I also banged on things until I got them out of the OSD tree display and then did a crush delete on host ceph08. And, incidentally, the OSD tree works on simple host names, not FQDNs like the rest of ceph! So in theory, I'm ready to jack in new drives and add new OSDs to ceph08. Except: # ceph health detail HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.2 on ceph08.internal.mousetech.com is in error state daemon osd.4 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them. I'm going to try removing/re-installing prometheus. since it's now showing up in ceph health. I think last time I had zombie OSDs I had to brute-force delete their corresponding directories under /var/lib/ceph. On 3/27/25 14:01, Eugen Block wrote:
ceph config-key rm mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0
Did you disable the prometheus module? I would expect the warning to clear if you did.
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
Or do you still see those OSDs in 'cephadm ls' output on ceph08? If you do, and if those OSDs are really already drained/purged, you can remove them with 'cephadm rm-daemon --name osd.2'. And I would try to get the MGR into a working state first, before you try to deploy prometheus again. So my recommendation is to get into HEALTH_OK first. And btw, "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is serious, you can end up with inactive PGs during recovery, so I'd also consider checking the pools and their PGs. Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for your patience.
host ceph06 isn't referenced in the config database. I think I've finally purged it. I also reset the dashboard API host address from ceph08 to dell02. But since prometheus isn't running on dell02 either, there's no gain there.
I did clear some of that lint out via "ceph mgr fail".
So here's the latest. There are strange things happening at the base OS level that keep host ceph08 from running its OSDs anymore. At boot, device /dev/sdb suddenly changes to /dev/sdd (????) and there seem to be I/O errors. It's really strange, but I'm going to replace the physical drive and that will hopefully cure that.
The problem is, reef and earlier releases seem to have a lot of trouble in deleting OSDs that aren't running. As I've noted before, they tend to get permanently stuck in the "deleting" state. When I cannot restart the OSD, the only cure for that has been to run around the system and apply brute force until things clear up.
I did a dashboard purge of the OSDs on ceph08 and that removed them from the GUI (they'd already drained). I also banged on things until I got them out of the OSD tree display and then did a crush delete on host ceph08. And, incidentally, the OSD tree works on simple host names, not FQDNs like the rest of ceph!
So in theory, I'm ready to jack in new drives and add new OSDs to ceph08. Except:
# ceph health detail HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.2 on ceph08.internal.mousetech.com is in error state daemon osd.4 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
I'm going to try removing/re-installing prometheus. since it's now showing up in ceph health. I think last time I had zombie OSDs I had to brute-force delete their corresponding directories under /var/lib/ceph.
On 3/27/25 14:01, Eugen Block wrote:
ceph config-key rm mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the info on removing stubborn dead OSDs. The actual syntax required was: cephadm rm-daemon --name osd.2 --fsid <fsid>--force On the "too many pgs", that's because I'm down 2 OSDs. I've got new drives, but they were waiting to clear out the dead stuff. I know it's risky, but I have backups. Recall that the start of this thread was on a HEALTH_OK system and prometheus was not activating. The OSD stuff was just a distraction. I did notice that the attempt to add a new mds did work after I did a "ceph mgr fail", so it's only prometheus that's a permanent problem. Here's the latest health after clearing out the dead OSDs: # ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) And yes, disabling prometheus will make the "name or service not known" errors go away. On 3/28/25 02:49, Eugen Block wrote:
Did you disable the prometheus module? I would expect the warning to clear if you did.
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
Or do you still see those OSDs in 'cephadm ls' output on ceph08? If you do, and if those OSDs are really already drained/purged, you can remove them with 'cephadm rm-daemon --name osd.2'. And I would try to get the MGR into a working state first, before you try to deploy prometheus again. So my recommendation is to get into HEALTH_OK first. And btw, "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is serious, you can end up with inactive PGs during recovery, so I'd also consider checking the pools and their PGs.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for your patience.
host ceph06 isn't referenced in the config database. I think I've finally purged it. I also reset the dashboard API host address from ceph08 to dell02. But since prometheus isn't running on dell02 either, there's no gain there.
I did clear some of that lint out via "ceph mgr fail".
So here's the latest. There are strange things happening at the base OS level that keep host ceph08 from running its OSDs anymore. At boot, device /dev/sdb suddenly changes to /dev/sdd (????) and there seem to be I/O errors. It's really strange, but I'm going to replace the physical drive and that will hopefully cure that.
The problem is, reef and earlier releases seem to have a lot of trouble in deleting OSDs that aren't running. As I've noted before, they tend to get permanently stuck in the "deleting" state. When I cannot restart the OSD, the only cure for that has been to run around the system and apply brute force until things clear up.
I did a dashboard purge of the OSDs on ceph08 and that removed them from the GUI (they'd already drained). I also banged on things until I got them out of the OSD tree display and then did a crush delete on host ceph08. And, incidentally, the OSD tree works on simple host names, not FQDNs like the rest of ceph!
So in theory, I'm ready to jack in new drives and add new OSDs to ceph08. Except:
# ceph health detail HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.2 on ceph08.internal.mousetech.com is in error state daemon osd.4 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
I'm going to try removing/re-installing prometheus. since it's now showing up in ceph health. I think last time I had zombie OSDs I had to brute-force delete their corresponding directories under /var/lib/ceph.
On 3/27/25 14:01, Eugen Block wrote:
ceph config-key rm mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Okay, next I would keep prometheus disabled to see if the mgr works properly. So disable the module again, and also reset the dashboard setting to an empty value: ceph dashboard reset-prometheus-api-host Then see if you get an mds daemon deployed. Or test it by removing and redeploying ceph-exporter or crash or something, anything to test if the mgr is able to remove and deploy other services. Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for the info on removing stubborn dead OSDs. The actual syntax required was:
cephadm rm-daemon --name osd.2 --fsid <fsid>--force
On the "too many pgs", that's because I'm down 2 OSDs. I've got new drives, but they were waiting to clear out the dead stuff. I know it's risky, but I have backups.
Recall that the start of this thread was on a HEALTH_OK system and prometheus was not activating. The OSD stuff was just a distraction.
I did notice that the attempt to add a new mds did work after I did a "ceph mgr fail", so it's only prometheus that's a permanent problem.
Here's the latest health after clearing out the dead OSDs:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
And yes, disabling prometheus will make the "name or service not known" errors go away.
On 3/28/25 02:49, Eugen Block wrote:
Did you disable the prometheus module? I would expect the warning to clear if you did.
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
Or do you still see those OSDs in 'cephadm ls' output on ceph08? If you do, and if those OSDs are really already drained/purged, you can remove them with 'cephadm rm-daemon --name osd.2'. And I would try to get the MGR into a working state first, before you try to deploy prometheus again. So my recommendation is to get into HEALTH_OK first. And btw, "TOO_MANY_PGS: too many PGs per OSD (648
max 560)" is serious, you can end up with inactive PGs during recovery, so I'd also consider checking the pools and their PGs.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for your patience.
host ceph06 isn't referenced in the config database. I think I've finally purged it. I also reset the dashboard API host address from ceph08 to dell02. But since prometheus isn't running on dell02 either, there's no gain there.
I did clear some of that lint out via "ceph mgr fail".
So here's the latest. There are strange things happening at the base OS level that keep host ceph08 from running its OSDs anymore. At boot, device /dev/sdb suddenly changes to /dev/sdd (????) and there seem to be I/O errors. It's really strange, but I'm going to replace the physical drive and that will hopefully cure that.
The problem is, reef and earlier releases seem to have a lot of trouble in deleting OSDs that aren't running. As I've noted before, they tend to get permanently stuck in the "deleting" state. When I cannot restart the OSD, the only cure for that has been to run around the system and apply brute force until things clear up.
I did a dashboard purge of the OSDs on ceph08 and that removed them from the GUI (they'd already drained). I also banged on things until I got them out of the OSD tree display and then did a crush delete on host ceph08. And, incidentally, the OSD tree works on simple host names, not FQDNs like the rest of ceph!
So in theory, I'm ready to jack in new drives and add new OSDs to ceph08. Except:
# ceph health detail HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.2 on ceph08.internal.mousetech.com is in error state daemon osd.4 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
I'm going to try removing/re-installing prometheus. since it's now showing up in ceph health. I think last time I had zombie OSDs I had to brute-force delete their corresponding directories under /var/lib/ceph.
On 3/27/25 14:01, Eugen Block wrote:
ceph config-key rm mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Actually, I did deploy a new mds node yesterday. But I followed your instructions and successfully removed and re-installed ceph-exporter (4 nodes). So that part works. On 3/28/25 07:28, Eugen Block wrote:
Okay, next I would keep prometheus disabled to see if the mgr works properly. So disable the module again, and also reset the dashboard setting to an empty value:
ceph dashboard reset-prometheus-api-host
Then see if you get an mds daemon deployed. Or test it by removing and redeploying ceph-exporter or crash or something, anything to test if the mgr is able to remove and deploy other services.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for the info on removing stubborn dead OSDs. The actual syntax required was:
cephadm rm-daemon --name osd.2 --fsid <fsid>--force
On the "too many pgs", that's because I'm down 2 OSDs. I've got new drives, but they were waiting to clear out the dead stuff. I know it's risky, but I have backups.
Recall that the start of this thread was on a HEALTH_OK system and prometheus was not activating. The OSD stuff was just a distraction.
I did notice that the attempt to add a new mds did work after I did a "ceph mgr fail", so it's only prometheus that's a permanent problem.
Here's the latest health after clearing out the dead OSDs:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
And yes, disabling prometheus will make the "name or service not known" errors go away.
On 3/28/25 02:49, Eugen Block wrote:
Did you disable the prometheus module? I would expect the warning to clear if you did.
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
Or do you still see those OSDs in 'cephadm ls' output on ceph08? If you do, and if those OSDs are really already drained/purged, you can remove them with 'cephadm rm-daemon --name osd.2'. And I would try to get the MGR into a working state first, before you try to deploy prometheus again. So my recommendation is to get into HEALTH_OK first. And btw, "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is serious, you can end up with inactive PGs during recovery, so I'd also consider checking the pools and their PGs.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for your patience.
host ceph06 isn't referenced in the config database. I think I've finally purged it. I also reset the dashboard API host address from ceph08 to dell02. But since prometheus isn't running on dell02 either, there's no gain there.
I did clear some of that lint out via "ceph mgr fail".
So here's the latest. There are strange things happening at the base OS level that keep host ceph08 from running its OSDs anymore. At boot, device /dev/sdb suddenly changes to /dev/sdd (????) and there seem to be I/O errors. It's really strange, but I'm going to replace the physical drive and that will hopefully cure that.
The problem is, reef and earlier releases seem to have a lot of trouble in deleting OSDs that aren't running. As I've noted before, they tend to get permanently stuck in the "deleting" state. When I cannot restart the OSD, the only cure for that has been to run around the system and apply brute force until things clear up.
I did a dashboard purge of the OSDs on ceph08 and that removed them from the GUI (they'd already drained). I also banged on things until I got them out of the OSD tree display and then did a crush delete on host ceph08. And, incidentally, the OSD tree works on simple host names, not FQDNs like the rest of ceph!
So in theory, I'm ready to jack in new drives and add new OSDs to ceph08. Except:
# ceph health detail HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.2 on ceph08.internal.mousetech.com is in error state daemon osd.4 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
I'm going to try removing/re-installing prometheus. since it's now showing up in ceph health. I think last time I had zombie OSDs I had to brute-force delete their corresponding directories under /var/lib/ceph.
On 3/27/25 14:01, Eugen Block wrote:
ceph config-key rm mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
OK! Success of a sort. I removed and re-installed each of the failed services in turn using the "ceph orch rm" command followed by "ceph orch apply". They came up with default settings (1 server), but they did come up. Finally, I tried it with prometheus. This gave me: prometheus ?:9095 0/1 - 10s count:1 However, in order for the dashboard to be happy, I had to supply more info. Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it. Following that, I had to set the following: ceph config set mgr mgr/prometheus/server_addr 10.0.1.58 ceph config set mgr mgr/prometheus/server_port 9095 ceph dashboard set-prometheus-api-host 10.0.1.58 (ceph08) ceph dashboard set-prometheus-api-port 909 Once all of the above were set, the dashboard stopped complaining about being able to access the prometheus API. However, one last wart remains. Despite being up and running (and confirmed listening on ceph09 port 9095), I do get this: # ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) On 3/28/25 08:53, Tim Holloway wrote:
Actually, I did deploy a new mds node yesterday. But I followed your instructions and successfully removed and re-installed ceph-exporter (4 nodes). So that part works.
On 3/28/25 07:28, Eugen Block wrote:
Okay, next I would keep prometheus disabled to see if the mgr works properly. So disable the module again, and also reset the dashboard setting to an empty value:
ceph dashboard reset-prometheus-api-host
Then see if you get an mds daemon deployed. Or test it by removing and redeploying ceph-exporter or crash or something, anything to test if the mgr is able to remove and deploy other services.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for the info on removing stubborn dead OSDs. The actual syntax required was:
cephadm rm-daemon --name osd.2 --fsid <fsid>--force
On the "too many pgs", that's because I'm down 2 OSDs. I've got new drives, but they were waiting to clear out the dead stuff. I know it's risky, but I have backups.
Recall that the start of this thread was on a HEALTH_OK system and prometheus was not activating. The OSD stuff was just a distraction.
I did notice that the attempt to add a new mds did work after I did a "ceph mgr fail", so it's only prometheus that's a permanent problem.
Here's the latest health after clearing out the dead OSDs:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
And yes, disabling prometheus will make the "name or service not known" errors go away.
On 3/28/25 02:49, Eugen Block wrote:
Did you disable the prometheus module? I would expect the warning to clear if you did.
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
Or do you still see those OSDs in 'cephadm ls' output on ceph08? If you do, and if those OSDs are really already drained/purged, you can remove them with 'cephadm rm-daemon --name osd.2'. And I would try to get the MGR into a working state first, before you try to deploy prometheus again. So my recommendation is to get into HEALTH_OK first. And btw, "TOO_MANY_PGS: too many PGs per OSD (648
max 560)" is serious, you can end up with inactive PGs during recovery, so I'd also consider checking the pools and their PGs.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for your patience.
host ceph06 isn't referenced in the config database. I think I've finally purged it. I also reset the dashboard API host address from ceph08 to dell02. But since prometheus isn't running on dell02 either, there's no gain there.
I did clear some of that lint out via "ceph mgr fail".
So here's the latest. There are strange things happening at the base OS level that keep host ceph08 from running its OSDs anymore. At boot, device /dev/sdb suddenly changes to /dev/sdd (????) and there seem to be I/O errors. It's really strange, but I'm going to replace the physical drive and that will hopefully cure that.
The problem is, reef and earlier releases seem to have a lot of trouble in deleting OSDs that aren't running. As I've noted before, they tend to get permanently stuck in the "deleting" state. When I cannot restart the OSD, the only cure for that has been to run around the system and apply brute force until things clear up.
I did a dashboard purge of the OSDs on ceph08 and that removed them from the GUI (they'd already drained). I also banged on things until I got them out of the OSD tree display and then did a crush delete on host ceph08. And, incidentally, the OSD tree works on simple host names, not FQDNs like the rest of ceph!
So in theory, I'm ready to jack in new drives and add new OSDs to ceph08. Except:
# ceph health detail HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.2 on ceph08.internal.mousetech.com is in error state daemon osd.4 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
I'm going to try removing/re-installing prometheus. since it's now showing up in ceph health. I think last time I had zombie OSDs I had to brute-force delete their corresponding directories under /var/lib/ceph.
On 3/27/25 14:01, Eugen Block wrote:
ceph config-key rm mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
ceph orch ps --daemon-type prometheus would show you where it tried to place the daemon. So prometheus is now actually up and running? Just to confirm because you pasted the output of 'ceph orch ls' when it wasn't (yet?).
ceph dashboard set-prometheus-api-port 909
Is this a c&p mistake or did you actually miss the 5 here?
[ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known')
This seems to be a DNS issue, both relevant places in the code for "gaierror" point to either https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... or https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... where one tries a "_dns_lookup" and the other a "resolve_ip". What does 'ceph mgr services' show? Zitat von Tim Holloway <timh@mousetech.com>:
OK! Success of a sort.
I removed and re-installed each of the failed services in turn using the "ceph orch rm" command followed by "ceph orch apply". They came up with default settings (1 server), but they did come up.
Finally, I tried it with prometheus. This gave me:
prometheus ?:9095 0/1 - 10s count:1
However, in order for the dashboard to be happy, I had to supply more info. Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
Following that, I had to set the following:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58
ceph config set mgr mgr/prometheus/server_port 9095
ceph dashboard set-prometheus-api-host 10.0.1.58 (ceph08)
ceph dashboard set-prometheus-api-port 909
Once all of the above were set, the dashboard stopped complaining about being able to access the prometheus API.
However, one last wart remains. Despite being up and running (and confirmed listening on ceph09 port 9095), I do get this:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
On 3/28/25 08:53, Tim Holloway wrote:
Actually, I did deploy a new mds node yesterday. But I followed your instructions and successfully removed and re-installed ceph-exporter (4 nodes). So that part works.
On 3/28/25 07:28, Eugen Block wrote:
Okay, next I would keep prometheus disabled to see if the mgr works properly. So disable the module again, and also reset the dashboard setting to an empty value:
ceph dashboard reset-prometheus-api-host
Then see if you get an mds daemon deployed. Or test it by removing and redeploying ceph-exporter or crash or something, anything to test if the mgr is able to remove and deploy other services.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for the info on removing stubborn dead OSDs. The actual syntax required was:
cephadm rm-daemon --name osd.2 --fsid <fsid>--force
On the "too many pgs", that's because I'm down 2 OSDs. I've got new drives, but they were waiting to clear out the dead stuff. I know it's risky, but I have backups.
Recall that the start of this thread was on a HEALTH_OK system and prometheus was not activating. The OSD stuff was just a distraction.
I did notice that the attempt to add a new mds did work after I did a "ceph mgr fail", so it's only prometheus that's a permanent problem.
Here's the latest health after clearing out the dead OSDs:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
And yes, disabling prometheus will make the "name or service not known" errors go away.
On 3/28/25 02:49, Eugen Block wrote:
Did you disable the prometheus module? I would expect the warning to clear if you did.
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
Or do you still see those OSDs in 'cephadm ls' output on ceph08? If you do, and if those OSDs are really already drained/purged, you can remove them with 'cephadm rm-daemon --name osd.2'. And I would try to get the MGR into a working state first, before you try to deploy prometheus again. So my recommendation is to get into HEALTH_OK first. And btw, "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is serious, you can end up with inactive PGs during recovery, so I'd also consider checking the pools and their PGs.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for your patience.
host ceph06 isn't referenced in the config database. I think I've finally purged it. I also reset the dashboard API host address from ceph08 to dell02. But since prometheus isn't running on dell02 either, there's no gain there.
I did clear some of that lint out via "ceph mgr fail".
So here's the latest. There are strange things happening at the base OS level that keep host ceph08 from running its OSDs anymore. At boot, device /dev/sdb suddenly changes to /dev/sdd (????) and there seem to be I/O errors. It's really strange, but I'm going to replace the physical drive and that will hopefully cure that.
The problem is, reef and earlier releases seem to have a lot of trouble in deleting OSDs that aren't running. As I've noted before, they tend to get permanently stuck in the "deleting" state. When I cannot restart the OSD, the only cure for that has been to run around the system and apply brute force until things clear up.
I did a dashboard purge of the OSDs on ceph08 and that removed them from the GUI (they'd already drained). I also banged on things until I got them out of the OSD tree display and then did a crush delete on host ceph08. And, incidentally, the OSD tree works on simple host names, not FQDNs like the rest of ceph!
So in theory, I'm ready to jack in new drives and add new OSDs to ceph08. Except:
# ceph health detail HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.2 on ceph08.internal.mousetech.com is in error state daemon osd.4 on ceph08.internal.mousetech.com is in error state [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
Somewhere deep inside ceph, those deleted OSDs still exist. Likely because ceph08 hasn't deleted the systemd units that run them.
I'm going to try removing/re-installing prometheus. since it's now showing up in ceph health. I think last time I had zombie OSDs I had to brute-force delete their corresponding directories under /var/lib/ceph.
On 3/27/25 14:01, Eugen Block wrote: > ceph config-key rm > mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
OK. I didn't realize I'd pasted the wrong orch ls output. Yes, it's "1/1" and has been for an hour. And yes, I did mis-type the port. The final complaint appears to arise from something looking for prometheus at the failed deployment location dell02. The dashboard says: The mgr/prometheus module at dell02.mousetech.com:9095 is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer ... ceph mgr services shows: # ceph mgr services { "dashboard":"https://10.0.1.52:8443/", "prometheus":"http://10.0.1.58:9095/" } On 3/28/25 09:55, Eugen Block wrote:
Hi,
Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
ceph orch ps --daemon-type prometheus
would show you where it tried to place the daemon.
So prometheus is now actually up and running? Just to confirm because you pasted the output of 'ceph orch ls' when it wasn't (yet?).
ceph dashboard set-prometheus-api-port 909
Is this a c&p mistake or did you actually miss the 5 here?
[ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known')
This seems to be a DNS issue, both relevant places in the code for "gaierror" point to either
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s...
or
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s...
where one tries a "_dns_lookup" and the other a "resolve_ip". What does 'ceph mgr services' show?
Zitat von Tim Holloway <timh@mousetech.com>:
OK! Success of a sort.
I removed and re-installed each of the failed services in turn using the "ceph orch rm" command followed by "ceph orch apply". They came up with default settings (1 server), but they did come up.
Finally, I tried it with prometheus. This gave me:
prometheus ?:9095 0/1 - 10s count:1
However, in order for the dashboard to be happy, I had to supply more info. Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
Following that, I had to set the following:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58
ceph config set mgr mgr/prometheus/server_port 9095
ceph dashboard set-prometheus-api-host 10.0.1.58 (ceph08)
ceph dashboard set-prometheus-api-port 909
Once all of the above were set, the dashboard stopped complaining about being able to access the prometheus API.
However, one last wart remains. Despite being up and running (and confirmed listening on ceph09 port 9095), I do get this:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
On 3/28/25 08:53, Tim Holloway wrote:
Actually, I did deploy a new mds node yesterday. But I followed your instructions and successfully removed and re-installed ceph-exporter (4 nodes). So that part works.
On 3/28/25 07:28, Eugen Block wrote:
Okay, next I would keep prometheus disabled to see if the mgr works properly. So disable the module again, and also reset the dashboard setting to an empty value:
ceph dashboard reset-prometheus-api-host
Then see if you get an mds daemon deployed. Or test it by removing and redeploying ceph-exporter or crash or something, anything to test if the mgr is able to remove and deploy other services.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for the info on removing stubborn dead OSDs. The actual syntax required was:
cephadm rm-daemon --name osd.2 --fsid <fsid>--force
On the "too many pgs", that's because I'm down 2 OSDs. I've got new drives, but they were waiting to clear out the dead stuff. I know it's risky, but I have backups.
Recall that the start of this thread was on a HEALTH_OK system and prometheus was not activating. The OSD stuff was just a distraction.
I did notice that the attempt to add a new mds did work after I did a "ceph mgr fail", so it's only prometheus that's a permanent problem.
Here's the latest health after clearing out the dead OSDs:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
And yes, disabling prometheus will make the "name or service not known" errors go away.
On 3/28/25 02:49, Eugen Block wrote:
Did you disable the prometheus module? I would expect the warning to clear if you did.
> Somewhere deep inside ceph, those deleted OSDs still exist. > Likely because ceph08 hasn't deleted the systemd units that run > them.
Or do you still see those OSDs in 'cephadm ls' output on ceph08? If you do, and if those OSDs are really already drained/purged, you can remove them with 'cephadm rm-daemon --name osd.2'. And I would try to get the MGR into a working state first, before you try to deploy prometheus again. So my recommendation is to get into HEALTH_OK first. And btw, "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is serious, you can end up with inactive PGs during recovery, so I'd also consider checking the pools and their PGs.
Zitat von Tim Holloway <timh@mousetech.com>:
> Thanks for your patience. > > host ceph06 isn't referenced in the config database. I think > I've finally purged it. I also reset the dashboard API host > address from ceph08 to dell02. But since prometheus isn't > running on dell02 either, there's no gain there. > > I did clear some of that lint out via "ceph mgr fail". > > So here's the latest. There are strange things happening at the > base OS level that keep host ceph08 from running its OSDs > anymore. At boot, device /dev/sdb suddenly changes to /dev/sdd > (????) and there seem to be I/O errors. It's really strange, but > I'm going to replace the physical drive and that will hopefully > cure that. > > The problem is, reef and earlier releases seem to have a lot of > trouble in deleting OSDs that aren't running. As I've noted > before, they tend to get permanently stuck in the "deleting" > state. When I cannot restart the OSD, the only cure for that has > been to run around the system and apply brute force until things > clear up. > > I did a dashboard purge of the OSDs on ceph08 and that removed > them from the GUI (they'd already drained). I also banged on > things until I got them out of the OSD tree display and then did > a crush delete on host ceph08. And, incidentally, the OSD tree > works on simple host names, not FQDNs like the rest of ceph! > > So in theory, I'm ready to jack in new drives and add new OSDs > to ceph08. Except: > > # ceph health detail > HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has > failed: gaierror(-2, 'Name or service not known'); too many PGs > per OSD (648 > max 560) > [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) > daemon osd.2 on ceph08.internal.mousetech.com is in error state > daemon osd.4 on ceph08.internal.mousetech.com is in error state > [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: > gaierror(-2, 'Name or service not known') > Module 'prometheus' has failed: gaierror(-2, 'Name or > service not known') > [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) > > Somewhere deep inside ceph, those deleted OSDs still exist. > Likely because ceph08 hasn't deleted the systemd units that run > them. > > I'm going to try removing/re-installing prometheus. since it's > now showing up in ceph health. I think last time I had zombie > OSDs I had to brute-force delete their corresponding directories > under /var/lib/ceph. > > On 3/27/25 14:01, Eugen Block wrote: >> ceph config-key rm >> mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There's still some misconfiguration, it appears. It can be confusing, but one thing is the mgr module "prometheus" which provides additional cluster data, it runs on default port 9283. The other is "prometheus server", which collects all the provided cluster data, typically on port 9095. So these two configs are wrong: ceph config set mgr mgr/prometheus/server_addr 10.0.1.58 ceph config set mgr mgr/prometheus/server_port 9095 Those should be: ceph config get mgr mgr/prometheus/server_addr 0.0.0.0 ceph config get mgr mgr/prometheus/server_port 9283 I assume that's why the module is still failing. Can you give that a try and report back? Zitat von Tim Holloway <timh@mousetech.com>:
OK. I didn't realize I'd pasted the wrong orch ls output. Yes, it's "1/1" and has been for an hour. And yes, I did mis-type the port.
The final complaint appears to arise from something looking for prometheus at the failed deployment location dell02. The dashboard says:
The mgr/prometheus module at dell02.mousetech.com:9095 is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer ...
ceph mgr services shows:
# ceph mgr services { "dashboard":"https://10.0.1.52:8443/", "prometheus":"http://10.0.1.58:9095/" }
On 3/28/25 09:55, Eugen Block wrote:
Hi,
Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
ceph orch ps --daemon-type prometheus
would show you where it tried to place the daemon.
So prometheus is now actually up and running? Just to confirm because you pasted the output of 'ceph orch ls' when it wasn't (yet?).
ceph dashboard set-prometheus-api-port 909
Is this a c&p mistake or did you actually miss the 5 here?
[ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known')
This seems to be a DNS issue, both relevant places in the code for "gaierror" point to either
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... or
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... where one tries a "_dns_lookup" and the other a "resolve_ip". What does 'ceph mgr services' show?
Zitat von Tim Holloway <timh@mousetech.com>:
OK! Success of a sort.
I removed and re-installed each of the failed services in turn using the "ceph orch rm" command followed by "ceph orch apply". They came up with default settings (1 server), but they did come up.
Finally, I tried it with prometheus. This gave me:
prometheus ?:9095 0/1 - 10s count:1
However, in order for the dashboard to be happy, I had to supply more info. Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
Following that, I had to set the following:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58
ceph config set mgr mgr/prometheus/server_port 9095
ceph dashboard set-prometheus-api-host 10.0.1.58 (ceph08)
ceph dashboard set-prometheus-api-port 909
Once all of the above were set, the dashboard stopped complaining about being able to access the prometheus API.
However, one last wart remains. Despite being up and running (and confirmed listening on ceph09 port 9095), I do get this:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
On 3/28/25 08:53, Tim Holloway wrote:
Actually, I did deploy a new mds node yesterday. But I followed your instructions and successfully removed and re-installed ceph-exporter (4 nodes). So that part works.
On 3/28/25 07:28, Eugen Block wrote:
Okay, next I would keep prometheus disabled to see if the mgr works properly. So disable the module again, and also reset the dashboard setting to an empty value:
ceph dashboard reset-prometheus-api-host
Then see if you get an mds daemon deployed. Or test it by removing and redeploying ceph-exporter or crash or something, anything to test if the mgr is able to remove and deploy other services.
Zitat von Tim Holloway <timh@mousetech.com>:
Thanks for the info on removing stubborn dead OSDs. The actual syntax required was:
cephadm rm-daemon --name osd.2 --fsid <fsid>--force
On the "too many pgs", that's because I'm down 2 OSDs. I've got new drives, but they were waiting to clear out the dead stuff. I know it's risky, but I have backups.
Recall that the start of this thread was on a HEALTH_OK system and prometheus was not activating. The OSD stuff was just a distraction.
I did notice that the attempt to add a new mds did work after I did a "ceph mgr fail", so it's only prometheus that's a permanent problem.
Here's the latest health after clearing out the dead OSDs:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
And yes, disabling prometheus will make the "name or service not known" errors go away.
On 3/28/25 02:49, Eugen Block wrote: > Did you disable the prometheus module? I would expect the > warning to clear if you did. > >> Somewhere deep inside ceph, those deleted OSDs still exist. >> Likely because ceph08 hasn't deleted the systemd units that >> run them. > > Or do you still see those OSDs in 'cephadm ls' output on > ceph08? If you do, and if those OSDs are really already > drained/purged, you can remove them with 'cephadm rm-daemon > --name osd.2'. And I would try to get the MGR into a working > state first, before you try to deploy prometheus again. So my > recommendation is to get into HEALTH_OK first. And btw, > "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is > serious, you can end up with inactive PGs during recovery, so > I'd also consider checking the pools and their PGs. > > Zitat von Tim Holloway <timh@mousetech.com>: > >> Thanks for your patience. >> >> host ceph06 isn't referenced in the config database. I think >> I've finally purged it. I also reset the dashboard API host >> address from ceph08 to dell02. But since prometheus isn't >> running on dell02 either, there's no gain there. >> >> I did clear some of that lint out via "ceph mgr fail". >> >> So here's the latest. There are strange things happening at >> the base OS level that keep host ceph08 from running its OSDs >> anymore. At boot, device /dev/sdb suddenly changes to >> /dev/sdd (????) and there seem to be I/O errors. It's really >> strange, but I'm going to replace the physical drive and that >> will hopefully cure that. >> >> The problem is, reef and earlier releases seem to have a lot >> of trouble in deleting OSDs that aren't running. As I've >> noted before, they tend to get permanently stuck in the >> "deleting" state. When I cannot restart the OSD, the only >> cure for that has been to run around the system and apply >> brute force until things clear up. >> >> I did a dashboard purge of the OSDs on ceph08 and that >> removed them from the GUI (they'd already drained). I also >> banged on things until I got them out of the OSD tree display >> and then did a crush delete on host ceph08. And, >> incidentally, the OSD tree works on simple host names, not >> FQDNs like the rest of ceph! >> >> So in theory, I'm ready to jack in new drives and add new >> OSDs to ceph08. Except: >> >> # ceph health detail >> HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' >> has failed: gaierror(-2, 'Name or service not known'); too >> many PGs per OSD (648 > max 560) >> [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) >> daemon osd.2 on ceph08.internal.mousetech.com is in error state >> daemon osd.4 on ceph08.internal.mousetech.com is in error state >> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >> gaierror(-2, 'Name or service not known') >> Module 'prometheus' has failed: gaierror(-2, 'Name or >> service not known') >> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >> >> Somewhere deep inside ceph, those deleted OSDs still exist. >> Likely because ceph08 hasn't deleted the systemd units that >> run them. >> >> I'm going to try removing/re-installing prometheus. since >> it's now showing up in ceph health. I think last time I had >> zombie OSDs I had to brute-force delete their corresponding >> directories under /var/lib/ceph. >> >> On 3/27/25 14:01, Eugen Block wrote: >>> ceph config-key rm >>> mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0 >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm guessing that the configuration issues come from the dashboard wanting the prometheus API at 9505, versus prometheus itself on 9283. Regardless, that didn't fix the message. As far as I can tell, the "service not known" is coming from something trying to contact prometheus on host dell02, and dell02 isn't running prometheus, since the way I got it working at all was to do a generic "ceph orch apply" without arguments. The problem being that apparently the YAML config is still lurking around in the background even though it never spawned the requested instances. What I'll probably do is remove prometheus, try an "orch apply" with arguments on the command line (deploy 2 hosts to ceph02 and dell02), see what works or breaks, and if that succeeds, try again with the YAML. Which won't totally prove everything's fixed, since it could be drawing on the hidden stuff that won't go away, but at least it would make it superficially clean. On 3/28/25 18:52, Eugen Block wrote:
There's still some misconfiguration, it appears. It can be confusing, but one thing is the mgr module "prometheus" which provides additional cluster data, it runs on default port 9283. The other is "prometheus server", which collects all the provided cluster data, typically on port 9095.
So these two configs are wrong:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58 ceph config set mgr mgr/prometheus/server_port 9095
Those should be:
ceph config get mgr mgr/prometheus/server_addr 0.0.0.0 ceph config get mgr mgr/prometheus/server_port 9283
I assume that's why the module is still failing. Can you give that a try and report back?
Zitat von Tim Holloway <timh@mousetech.com>:
OK. I didn't realize I'd pasted the wrong orch ls output. Yes, it's "1/1" and has been for an hour. And yes, I did mis-type the port.
The final complaint appears to arise from something looking for prometheus at the failed deployment location dell02. The dashboard says:
The mgr/prometheus module at dell02.mousetech.com:9095 is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer ...
ceph mgr services shows:
# ceph mgr services { "dashboard":"https://10.0.1.52:8443/", "prometheus":"http://10.0.1.58:9095/" }
On 3/28/25 09:55, Eugen Block wrote:
Hi,
Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
ceph orch ps --daemon-type prometheus
would show you where it tried to place the daemon.
So prometheus is now actually up and running? Just to confirm because you pasted the output of 'ceph orch ls' when it wasn't (yet?).
ceph dashboard set-prometheus-api-port 909
Is this a c&p mistake or did you actually miss the 5 here?
[ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known')
This seems to be a DNS issue, both relevant places in the code for "gaierror" point to either
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... or
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... where one tries a "_dns_lookup" and the other a "resolve_ip". What does 'ceph mgr services' show?
Zitat von Tim Holloway <timh@mousetech.com>:
OK! Success of a sort.
I removed and re-installed each of the failed services in turn using the "ceph orch rm" command followed by "ceph orch apply". They came up with default settings (1 server), but they did come up.
Finally, I tried it with prometheus. This gave me:
prometheus ?:9095 0/1 - 10s count:1
However, in order for the dashboard to be happy, I had to supply more info. Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
Following that, I had to set the following:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58
ceph config set mgr mgr/prometheus/server_port 9095
ceph dashboard set-prometheus-api-host 10.0.1.58 (ceph08)
ceph dashboard set-prometheus-api-port 909
Once all of the above were set, the dashboard stopped complaining about being able to access the prometheus API.
However, one last wart remains. Despite being up and running (and confirmed listening on ceph09 port 9095), I do get this:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
On 3/28/25 08:53, Tim Holloway wrote:
Actually, I did deploy a new mds node yesterday. But I followed your instructions and successfully removed and re-installed ceph-exporter (4 nodes). So that part works.
On 3/28/25 07:28, Eugen Block wrote:
Okay, next I would keep prometheus disabled to see if the mgr works properly. So disable the module again, and also reset the dashboard setting to an empty value:
ceph dashboard reset-prometheus-api-host
Then see if you get an mds daemon deployed. Or test it by removing and redeploying ceph-exporter or crash or something, anything to test if the mgr is able to remove and deploy other services.
Zitat von Tim Holloway <timh@mousetech.com>:
> Thanks for the info on removing stubborn dead OSDs. The actual > syntax required was: > > cephadm rm-daemon --name osd.2 --fsid <fsid>--force > > On the "too many pgs", that's because I'm down 2 OSDs. I've got > new drives, but they were waiting to clear out the dead stuff. I > know it's risky, but I have backups. > > Recall that the start of this thread was on a HEALTH_OK system > and prometheus was not activating. The OSD stuff was just a > distraction. > > I did notice that the attempt to add a new mds did work after I > did a "ceph mgr fail", so it's only prometheus that's a > permanent problem. > > Here's the latest health after clearing out the dead OSDs: > > # ceph health detail > HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or > service not known'); too many PGs per OSD (648 > max 560) > [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: > gaierror(-2, 'Name or service not known') > Module 'prometheus' has failed: gaierror(-2, 'Name or > service not known') > [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) > > And yes, disabling prometheus will make the "name or service not > known" errors go away. > > > On 3/28/25 02:49, Eugen Block wrote: >> Did you disable the prometheus module? I would expect the >> warning to clear if you did. >> >>> Somewhere deep inside ceph, those deleted OSDs still exist. >>> Likely because ceph08 hasn't deleted the systemd units that >>> run them. >> >> Or do you still see those OSDs in 'cephadm ls' output on >> ceph08? If you do, and if those OSDs are really already >> drained/purged, you can remove them with 'cephadm rm-daemon >> --name osd.2'. And I would try to get the MGR into a working >> state first, before you try to deploy prometheus again. So my >> recommendation is to get into HEALTH_OK first. And btw, >> "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is >> serious, you can end up with inactive PGs during recovery, so >> I'd also consider checking the pools and their PGs. >> >> Zitat von Tim Holloway <timh@mousetech.com>: >> >>> Thanks for your patience. >>> >>> host ceph06 isn't referenced in the config database. I think >>> I've finally purged it. I also reset the dashboard API host >>> address from ceph08 to dell02. But since prometheus isn't >>> running on dell02 either, there's no gain there. >>> >>> I did clear some of that lint out via "ceph mgr fail". >>> >>> So here's the latest. There are strange things happening at >>> the base OS level that keep host ceph08 from running its OSDs >>> anymore. At boot, device /dev/sdb suddenly changes to /dev/sdd >>> (????) and there seem to be I/O errors. It's really strange, >>> but I'm going to replace the physical drive and that will >>> hopefully cure that. >>> >>> The problem is, reef and earlier releases seem to have a lot >>> of trouble in deleting OSDs that aren't running. As I've noted >>> before, they tend to get permanently stuck in the "deleting" >>> state. When I cannot restart the OSD, the only cure for that >>> has been to run around the system and apply brute force until >>> things clear up. >>> >>> I did a dashboard purge of the OSDs on ceph08 and that removed >>> them from the GUI (they'd already drained). I also banged on >>> things until I got them out of the OSD tree display and then >>> did a crush delete on host ceph08. And, incidentally, the OSD >>> tree works on simple host names, not FQDNs like the rest of ceph! >>> >>> So in theory, I'm ready to jack in new drives and add new OSDs >>> to ceph08. Except: >>> >>> # ceph health detail >>> HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has >>> failed: gaierror(-2, 'Name or service not known'); too many >>> PGs per OSD (648 > max 560) >>> [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) >>> daemon osd.2 on ceph08.internal.mousetech.com is in error >>> state >>> daemon osd.4 on ceph08.internal.mousetech.com is in error >>> state >>> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >>> gaierror(-2, 'Name or service not known') >>> Module 'prometheus' has failed: gaierror(-2, 'Name or >>> service not known') >>> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >>> >>> Somewhere deep inside ceph, those deleted OSDs still exist. >>> Likely because ceph08 hasn't deleted the systemd units that >>> run them. >>> >>> I'm going to try removing/re-installing prometheus. since it's >>> now showing up in ceph health. I think last time I had zombie >>> OSDs I had to brute-force delete their corresponding >>> directories under /var/lib/ceph. >>> >>> On 3/27/25 14:01, Eugen Block wrote: >>>> ceph config-key rm >>>> mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0 >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Do you find anything related to dell02 in config dump? ceph config dump | grep -C2 dell02 Zitat von Tim Holloway <timh@mousetech.com>:
I'm guessing that the configuration issues come from the dashboard wanting the prometheus API at 9505, versus prometheus itself on 9283.
Regardless, that didn't fix the message. As far as I can tell, the "service not known" is coming from something trying to contact prometheus on host dell02, and dell02 isn't running prometheus, since the way I got it working at all was to do a generic "ceph orch apply" without arguments. The problem being that apparently the YAML config is still lurking around in the background even though it never spawned the requested instances.
What I'll probably do is remove prometheus, try an "orch apply" with arguments on the command line (deploy 2 hosts to ceph02 and dell02), see what works or breaks, and if that succeeds, try again with the YAML. Which won't totally prove everything's fixed, since it could be drawing on the hidden stuff that won't go away, but at least it would make it superficially clean.
On 3/28/25 18:52, Eugen Block wrote:
There's still some misconfiguration, it appears. It can be confusing, but one thing is the mgr module "prometheus" which provides additional cluster data, it runs on default port 9283. The other is "prometheus server", which collects all the provided cluster data, typically on port 9095.
So these two configs are wrong:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58 ceph config set mgr mgr/prometheus/server_port 9095
Those should be:
ceph config get mgr mgr/prometheus/server_addr 0.0.0.0 ceph config get mgr mgr/prometheus/server_port 9283
I assume that's why the module is still failing. Can you give that a try and report back?
Zitat von Tim Holloway <timh@mousetech.com>:
OK. I didn't realize I'd pasted the wrong orch ls output. Yes, it's "1/1" and has been for an hour. And yes, I did mis-type the port.
The final complaint appears to arise from something looking for prometheus at the failed deployment location dell02. The dashboard says:
The mgr/prometheus module at dell02.mousetech.com:9095 is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer ...
ceph mgr services shows:
# ceph mgr services { "dashboard":"https://10.0.1.52:8443/", "prometheus":"http://10.0.1.58:9095/" }
On 3/28/25 09:55, Eugen Block wrote:
Hi,
Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
ceph orch ps --daemon-type prometheus
would show you where it tried to place the daemon.
So prometheus is now actually up and running? Just to confirm because you pasted the output of 'ceph orch ls' when it wasn't (yet?).
ceph dashboard set-prometheus-api-port 909
Is this a c&p mistake or did you actually miss the 5 here?
[ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known')
This seems to be a DNS issue, both relevant places in the code for "gaierror" point to either
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... or
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... where one tries a "_dns_lookup" and the other a "resolve_ip". What does 'ceph mgr services' show?
Zitat von Tim Holloway <timh@mousetech.com>:
OK! Success of a sort.
I removed and re-installed each of the failed services in turn using the "ceph orch rm" command followed by "ceph orch apply". They came up with default settings (1 server), but they did come up.
Finally, I tried it with prometheus. This gave me:
prometheus ?:9095 0/1 - 10s count:1
However, in order for the dashboard to be happy, I had to supply more info. Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
Following that, I had to set the following:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58
ceph config set mgr mgr/prometheus/server_port 9095
ceph dashboard set-prometheus-api-host 10.0.1.58 (ceph08)
ceph dashboard set-prometheus-api-port 909
Once all of the above were set, the dashboard stopped complaining about being able to access the prometheus API.
However, one last wart remains. Despite being up and running (and confirmed listening on ceph09 port 9095), I do get this:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
On 3/28/25 08:53, Tim Holloway wrote:
Actually, I did deploy a new mds node yesterday. But I followed your instructions and successfully removed and re-installed ceph-exporter (4 nodes). So that part works.
On 3/28/25 07:28, Eugen Block wrote: > Okay, next I would keep prometheus disabled to see if the mgr > works properly. So disable the module again, and also reset > the dashboard setting to an empty value: > > ceph dashboard reset-prometheus-api-host > > Then see if you get an mds daemon deployed. Or test it by > removing and redeploying ceph-exporter or crash or something, > anything to test if the mgr is able to remove and deploy other > services. > > Zitat von Tim Holloway <timh@mousetech.com>: > >> Thanks for the info on removing stubborn dead OSDs. The >> actual syntax required was: >> >> cephadm rm-daemon --name osd.2 --fsid <fsid>--force >> >> On the "too many pgs", that's because I'm down 2 OSDs. I've >> got new drives, but they were waiting to clear out the dead >> stuff. I know it's risky, but I have backups. >> >> Recall that the start of this thread was on a HEALTH_OK >> system and prometheus was not activating. The OSD stuff was >> just a distraction. >> >> I did notice that the attempt to add a new mds did work after >> I did a "ceph mgr fail", so it's only prometheus that's a >> permanent problem. >> >> Here's the latest health after clearing out the dead OSDs: >> >> # ceph health detail >> HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name >> or service not known'); too many PGs per OSD (648 > max 560) >> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >> gaierror(-2, 'Name or service not known') >> Module 'prometheus' has failed: gaierror(-2, 'Name or >> service not known') >> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >> >> And yes, disabling prometheus will make the "name or service >> not known" errors go away. >> >> >> On 3/28/25 02:49, Eugen Block wrote: >>> Did you disable the prometheus module? I would expect the >>> warning to clear if you did. >>> >>>> Somewhere deep inside ceph, those deleted OSDs still exist. >>>> Likely because ceph08 hasn't deleted the systemd units that >>>> run them. >>> >>> Or do you still see those OSDs in 'cephadm ls' output on >>> ceph08? If you do, and if those OSDs are really already >>> drained/purged, you can remove them with 'cephadm rm-daemon >>> --name osd.2'. And I would try to get the MGR into a working >>> state first, before you try to deploy prometheus again. So >>> my recommendation is to get into HEALTH_OK first. And btw, >>> "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is >>> serious, you can end up with inactive PGs during recovery, >>> so I'd also consider checking the pools and their PGs. >>> >>> Zitat von Tim Holloway <timh@mousetech.com>: >>> >>>> Thanks for your patience. >>>> >>>> host ceph06 isn't referenced in the config database. I >>>> think I've finally purged it. I also reset the dashboard >>>> API host address from ceph08 to dell02. But since >>>> prometheus isn't running on dell02 either, there's no gain >>>> there. >>>> >>>> I did clear some of that lint out via "ceph mgr fail". >>>> >>>> So here's the latest. There are strange things happening at >>>> the base OS level that keep host ceph08 from running its >>>> OSDs anymore. At boot, device /dev/sdb suddenly changes to >>>> /dev/sdd (????) and there seem to be I/O errors. It's >>>> really strange, but I'm going to replace the physical drive >>>> and that will hopefully cure that. >>>> >>>> The problem is, reef and earlier releases seem to have a >>>> lot of trouble in deleting OSDs that aren't running. As >>>> I've noted before, they tend to get permanently stuck in >>>> the "deleting" state. When I cannot restart the OSD, the >>>> only cure for that has been to run around the system and >>>> apply brute force until things clear up. >>>> >>>> I did a dashboard purge of the OSDs on ceph08 and that >>>> removed them from the GUI (they'd already drained). I also >>>> banged on things until I got them out of the OSD tree >>>> display and then did a crush delete on host ceph08. And, >>>> incidentally, the OSD tree works on simple host names, not >>>> FQDNs like the rest of ceph! >>>> >>>> So in theory, I'm ready to jack in new drives and add new >>>> OSDs to ceph08. Except: >>>> >>>> # ceph health detail >>>> HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' >>>> has failed: gaierror(-2, 'Name or service not known'); too >>>> many PGs per OSD (648 > max 560) >>>> [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) >>>> daemon osd.2 on ceph08.internal.mousetech.com is in error state >>>> daemon osd.4 on ceph08.internal.mousetech.com is in error state >>>> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >>>> gaierror(-2, 'Name or service not known') >>>> Module 'prometheus' has failed: gaierror(-2, 'Name or >>>> service not known') >>>> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >>>> >>>> Somewhere deep inside ceph, those deleted OSDs still exist. >>>> Likely because ceph08 hasn't deleted the systemd units that >>>> run them. >>>> >>>> I'm going to try removing/re-installing prometheus. since >>>> it's now showing up in ceph health. I think last time I had >>>> zombie OSDs I had to brute-force delete their corresponding >>>> directories under /var/lib/ceph. >>>> >>>> On 3/27/25 14:01, Eugen Block wrote: >>>>> ceph config-key rm >>>>> mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0 >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Only the stuff that defines the rgw daemon on dell02. On 3/28/25 19:23, Eugen Block wrote:
Do you find anything related to dell02 in config dump?
ceph config dump | grep -C2 dell02
Zitat von Tim Holloway <timh@mousetech.com>:
I'm guessing that the configuration issues come from the dashboard wanting the prometheus API at 9505, versus prometheus itself on 9283.
Regardless, that didn't fix the message. As far as I can tell, the "service not known" is coming from something trying to contact prometheus on host dell02, and dell02 isn't running prometheus, since the way I got it working at all was to do a generic "ceph orch apply" without arguments. The problem being that apparently the YAML config is still lurking around in the background even though it never spawned the requested instances.
What I'll probably do is remove prometheus, try an "orch apply" with arguments on the command line (deploy 2 hosts to ceph02 and dell02), see what works or breaks, and if that succeeds, try again with the YAML. Which won't totally prove everything's fixed, since it could be drawing on the hidden stuff that won't go away, but at least it would make it superficially clean.
On 3/28/25 18:52, Eugen Block wrote:
There's still some misconfiguration, it appears. It can be confusing, but one thing is the mgr module "prometheus" which provides additional cluster data, it runs on default port 9283. The other is "prometheus server", which collects all the provided cluster data, typically on port 9095.
So these two configs are wrong:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58 ceph config set mgr mgr/prometheus/server_port 9095
Those should be:
ceph config get mgr mgr/prometheus/server_addr 0.0.0.0 ceph config get mgr mgr/prometheus/server_port 9283
I assume that's why the module is still failing. Can you give that a try and report back?
Zitat von Tim Holloway <timh@mousetech.com>:
OK. I didn't realize I'd pasted the wrong orch ls output. Yes, it's "1/1" and has been for an hour. And yes, I did mis-type the port.
The final complaint appears to arise from something looking for prometheus at the failed deployment location dell02. The dashboard says:
The mgr/prometheus module at dell02.mousetech.com:9095 is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer ...
ceph mgr services shows:
# ceph mgr services { "dashboard":"https://10.0.1.52:8443/", "prometheus":"http://10.0.1.58:9095/" }
On 3/28/25 09:55, Eugen Block wrote:
Hi,
Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
ceph orch ps --daemon-type prometheus
would show you where it tried to place the daemon.
So prometheus is now actually up and running? Just to confirm because you pasted the output of 'ceph orch ls' when it wasn't (yet?).
ceph dashboard set-prometheus-api-port 909
Is this a c&p mistake or did you actually miss the 5 here?
[ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known')
This seems to be a DNS issue, both relevant places in the code for "gaierror" point to either
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... or
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... where one tries a "_dns_lookup" and the other a "resolve_ip". What does 'ceph mgr services' show?
Zitat von Tim Holloway <timh@mousetech.com>:
OK! Success of a sort.
I removed and re-installed each of the failed services in turn using the "ceph orch rm" command followed by "ceph orch apply". They came up with default settings (1 server), but they did come up.
Finally, I tried it with prometheus. This gave me:
prometheus ?:9095 0/1 - 10s count:1
However, in order for the dashboard to be happy, I had to supply more info. Since ceph orch ls wouldn't tell me /where/ the new prometheus was deployed, I used the hosts tab in the dashboard to find it.
Following that, I had to set the following:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58
ceph config set mgr mgr/prometheus/server_port 9095
ceph dashboard set-prometheus-api-host 10.0.1.58 (ceph08)
ceph dashboard set-prometheus-api-port 909
Once all of the above were set, the dashboard stopped complaining about being able to access the prometheus API.
However, one last wart remains. Despite being up and running (and confirmed listening on ceph09 port 9095), I do get this:
# ceph health detail HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service not known'); too many PGs per OSD (648 > max 560) [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') Module 'prometheus' has failed: gaierror(-2, 'Name or service not known') [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
On 3/28/25 08:53, Tim Holloway wrote: > Actually, I did deploy a new mds node yesterday. But I followed > your instructions and successfully removed and re-installed > ceph-exporter (4 nodes). So that part works. > > On 3/28/25 07:28, Eugen Block wrote: >> Okay, next I would keep prometheus disabled to see if the mgr >> works properly. So disable the module again, and also reset the >> dashboard setting to an empty value: >> >> ceph dashboard reset-prometheus-api-host >> >> Then see if you get an mds daemon deployed. Or test it by >> removing and redeploying ceph-exporter or crash or something, >> anything to test if the mgr is able to remove and deploy other >> services. >> >> Zitat von Tim Holloway <timh@mousetech.com>: >> >>> Thanks for the info on removing stubborn dead OSDs. The actual >>> syntax required was: >>> >>> cephadm rm-daemon --name osd.2 --fsid <fsid>--force >>> >>> On the "too many pgs", that's because I'm down 2 OSDs. I've >>> got new drives, but they were waiting to clear out the dead >>> stuff. I know it's risky, but I have backups. >>> >>> Recall that the start of this thread was on a HEALTH_OK system >>> and prometheus was not activating. The OSD stuff was just a >>> distraction. >>> >>> I did notice that the attempt to add a new mds did work after >>> I did a "ceph mgr fail", so it's only prometheus that's a >>> permanent problem. >>> >>> Here's the latest health after clearing out the dead OSDs: >>> >>> # ceph health detail >>> HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name >>> or service not known'); too many PGs per OSD (648 > max 560) >>> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >>> gaierror(-2, 'Name or service not known') >>> Module 'prometheus' has failed: gaierror(-2, 'Name or >>> service not known') >>> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >>> >>> And yes, disabling prometheus will make the "name or service >>> not known" errors go away. >>> >>> >>> On 3/28/25 02:49, Eugen Block wrote: >>>> Did you disable the prometheus module? I would expect the >>>> warning to clear if you did. >>>> >>>>> Somewhere deep inside ceph, those deleted OSDs still exist. >>>>> Likely because ceph08 hasn't deleted the systemd units that >>>>> run them. >>>> >>>> Or do you still see those OSDs in 'cephadm ls' output on >>>> ceph08? If you do, and if those OSDs are really already >>>> drained/purged, you can remove them with 'cephadm rm-daemon >>>> --name osd.2'. And I would try to get the MGR into a working >>>> state first, before you try to deploy prometheus again. So my >>>> recommendation is to get into HEALTH_OK first. And btw, >>>> "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is >>>> serious, you can end up with inactive PGs during recovery, so >>>> I'd also consider checking the pools and their PGs. >>>> >>>> Zitat von Tim Holloway <timh@mousetech.com>: >>>> >>>>> Thanks for your patience. >>>>> >>>>> host ceph06 isn't referenced in the config database. I think >>>>> I've finally purged it. I also reset the dashboard API host >>>>> address from ceph08 to dell02. But since prometheus isn't >>>>> running on dell02 either, there's no gain there. >>>>> >>>>> I did clear some of that lint out via "ceph mgr fail". >>>>> >>>>> So here's the latest. There are strange things happening at >>>>> the base OS level that keep host ceph08 from running its >>>>> OSDs anymore. At boot, device /dev/sdb suddenly changes to >>>>> /dev/sdd (????) and there seem to be I/O errors. It's really >>>>> strange, but I'm going to replace the physical drive and >>>>> that will hopefully cure that. >>>>> >>>>> The problem is, reef and earlier releases seem to have a lot >>>>> of trouble in deleting OSDs that aren't running. As I've >>>>> noted before, they tend to get permanently stuck in the >>>>> "deleting" state. When I cannot restart the OSD, the only >>>>> cure for that has been to run around the system and apply >>>>> brute force until things clear up. >>>>> >>>>> I did a dashboard purge of the OSDs on ceph08 and that >>>>> removed them from the GUI (they'd already drained). I also >>>>> banged on things until I got them out of the OSD tree >>>>> display and then did a crush delete on host ceph08. And, >>>>> incidentally, the OSD tree works on simple host names, not >>>>> FQDNs like the rest of ceph! >>>>> >>>>> So in theory, I'm ready to jack in new drives and add new >>>>> OSDs to ceph08. Except: >>>>> >>>>> # ceph health detail >>>>> HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' >>>>> has failed: gaierror(-2, 'Name or service not known'); too >>>>> many PGs per OSD (648 > max 560) >>>>> [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) >>>>> daemon osd.2 on ceph08.internal.mousetech.com is in >>>>> error state >>>>> daemon osd.4 on ceph08.internal.mousetech.com is in >>>>> error state >>>>> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >>>>> gaierror(-2, 'Name or service not known') >>>>> Module 'prometheus' has failed: gaierror(-2, 'Name or >>>>> service not known') >>>>> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >>>>> >>>>> Somewhere deep inside ceph, those deleted OSDs still exist. >>>>> Likely because ceph08 hasn't deleted the systemd units that >>>>> run them. >>>>> >>>>> I'm going to try removing/re-installing prometheus. since >>>>> it's now showing up in ceph health. I think last time I had >>>>> zombie OSDs I had to brute-force delete their corresponding >>>>> directories under /var/lib/ceph. >>>>> >>>>> On 3/27/25 14:01, Eugen Block wrote: >>>>>> ceph config-key rm >>>>>> mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0 >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
How about this: ceph config-key dump | grep -v history Can you spot any key regarding dell02 that doesn't belong there? Zitat von Tim Holloway <timh@mousetech.com>:
Only the stuff that defines the rgw daemon on dell02.
On 3/28/25 19:23, Eugen Block wrote:
Do you find anything related to dell02 in config dump?
ceph config dump | grep -C2 dell02
Zitat von Tim Holloway <timh@mousetech.com>:
I'm guessing that the configuration issues come from the dashboard wanting the prometheus API at 9505, versus prometheus itself on 9283.
Regardless, that didn't fix the message. As far as I can tell, the "service not known" is coming from something trying to contact prometheus on host dell02, and dell02 isn't running prometheus, since the way I got it working at all was to do a generic "ceph orch apply" without arguments. The problem being that apparently the YAML config is still lurking around in the background even though it never spawned the requested instances.
What I'll probably do is remove prometheus, try an "orch apply" with arguments on the command line (deploy 2 hosts to ceph02 and dell02), see what works or breaks, and if that succeeds, try again with the YAML. Which won't totally prove everything's fixed, since it could be drawing on the hidden stuff that won't go away, but at least it would make it superficially clean.
On 3/28/25 18:52, Eugen Block wrote:
There's still some misconfiguration, it appears. It can be confusing, but one thing is the mgr module "prometheus" which provides additional cluster data, it runs on default port 9283. The other is "prometheus server", which collects all the provided cluster data, typically on port 9095.
So these two configs are wrong:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58 ceph config set mgr mgr/prometheus/server_port 9095
Those should be:
ceph config get mgr mgr/prometheus/server_addr 0.0.0.0 ceph config get mgr mgr/prometheus/server_port 9283
I assume that's why the module is still failing. Can you give that a try and report back?
Zitat von Tim Holloway <timh@mousetech.com>:
OK. I didn't realize I'd pasted the wrong orch ls output. Yes, it's "1/1" and has been for an hour. And yes, I did mis-type the port.
The final complaint appears to arise from something looking for prometheus at the failed deployment location dell02. The dashboard says:
The mgr/prometheus module at dell02.mousetech.com:9095 is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer ...
ceph mgr services shows:
# ceph mgr services { "dashboard":"https://10.0.1.52:8443/", "prometheus":"http://10.0.1.58:9095/" }
On 3/28/25 09:55, Eugen Block wrote:
Hi,
> Since ceph orch ls wouldn't tell me /where/ the new prometheus > was deployed, I used the hosts tab in the dashboard to find it.
ceph orch ps --daemon-type prometheus
would show you where it tried to place the daemon.
So prometheus is now actually up and running? Just to confirm because you pasted the output of 'ceph orch ls' when it wasn't (yet?).
> ceph dashboard set-prometheus-api-port 909
Is this a c&p mistake or did you actually miss the 5 here?
> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: > gaierror(-2, 'Name or service not known')
This seems to be a DNS issue, both relevant places in the code for "gaierror" point to either
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... or
https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... where one tries a "_dns_lookup" and the other a "resolve_ip". What does 'ceph mgr services' show?
Zitat von Tim Holloway <timh@mousetech.com>:
> OK! Success of a sort. > > I removed and re-installed each of the failed services in turn > using the "ceph orch rm" command followed by "ceph orch > apply". They came up with default settings (1 server), but > they did come up. > > Finally, I tried it with prometheus. This gave me: > > prometheus ?:9095 0/1 > - 10s count:1 > > However, in order for the dashboard to be happy, I had to > supply more info. Since ceph orch ls wouldn't tell me /where/ > the new prometheus was deployed, I used the hosts tab in the > dashboard to find it. > > Following that, I had to set the following: > > ceph config set mgr mgr/prometheus/server_addr 10.0.1.58 > > ceph config set mgr mgr/prometheus/server_port 9095 > > ceph dashboard set-prometheus-api-host 10.0.1.58 (ceph08) > > ceph dashboard set-prometheus-api-port 909 > > Once all of the above were set, the dashboard stopped > complaining about being able to access the prometheus API. > > However, one last wart remains. Despite being up and running > (and confirmed listening on ceph09 port 9095), I do get this: > > # ceph health detail > HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name > or service not known'); too many PGs per OSD (648 > max 560) > [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: > gaierror(-2, 'Name or service not known') > Module 'prometheus' has failed: gaierror(-2, 'Name or > service not known') > [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) > > On 3/28/25 08:53, Tim Holloway wrote: >> Actually, I did deploy a new mds node yesterday. But I >> followed your instructions and successfully removed and >> re-installed ceph-exporter (4 nodes). So that part works. >> >> On 3/28/25 07:28, Eugen Block wrote: >>> Okay, next I would keep prometheus disabled to see if the >>> mgr works properly. So disable the module again, and also >>> reset the dashboard setting to an empty value: >>> >>> ceph dashboard reset-prometheus-api-host >>> >>> Then see if you get an mds daemon deployed. Or test it by >>> removing and redeploying ceph-exporter or crash or >>> something, anything to test if the mgr is able to remove and >>> deploy other services. >>> >>> Zitat von Tim Holloway <timh@mousetech.com>: >>> >>>> Thanks for the info on removing stubborn dead OSDs. The >>>> actual syntax required was: >>>> >>>> cephadm rm-daemon --name osd.2 --fsid <fsid>--force >>>> >>>> On the "too many pgs", that's because I'm down 2 OSDs. I've >>>> got new drives, but they were waiting to clear out the dead >>>> stuff. I know it's risky, but I have backups. >>>> >>>> Recall that the start of this thread was on a HEALTH_OK >>>> system and prometheus was not activating. The OSD stuff was >>>> just a distraction. >>>> >>>> I did notice that the attempt to add a new mds did work >>>> after I did a "ceph mgr fail", so it's only prometheus >>>> that's a permanent problem. >>>> >>>> Here's the latest health after clearing out the dead OSDs: >>>> >>>> # ceph health detail >>>> HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, >>>> 'Name or service not known'); too many PGs per OSD (648 > >>>> max 560) >>>> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >>>> gaierror(-2, 'Name or service not known') >>>> Module 'prometheus' has failed: gaierror(-2, 'Name or >>>> service not known') >>>> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >>>> >>>> And yes, disabling prometheus will make the "name or >>>> service not known" errors go away. >>>> >>>> >>>> On 3/28/25 02:49, Eugen Block wrote: >>>>> Did you disable the prometheus module? I would expect the >>>>> warning to clear if you did. >>>>> >>>>>> Somewhere deep inside ceph, those deleted OSDs still >>>>>> exist. Likely because ceph08 hasn't deleted the systemd >>>>>> units that run them. >>>>> >>>>> Or do you still see those OSDs in 'cephadm ls' output on >>>>> ceph08? If you do, and if those OSDs are really already >>>>> drained/purged, you can remove them with 'cephadm >>>>> rm-daemon --name osd.2'. And I would try to get the MGR >>>>> into a working state first, before you try to deploy >>>>> prometheus again. So my recommendation is to get into >>>>> HEALTH_OK first. And btw, "TOO_MANY_PGS: too many PGs per >>>>> OSD (648 > max 560)" is serious, you can end up with >>>>> inactive PGs during recovery, so I'd also consider >>>>> checking the pools and their PGs. >>>>> >>>>> Zitat von Tim Holloway <timh@mousetech.com>: >>>>> >>>>>> Thanks for your patience. >>>>>> >>>>>> host ceph06 isn't referenced in the config database. I >>>>>> think I've finally purged it. I also reset the dashboard >>>>>> API host address from ceph08 to dell02. But since >>>>>> prometheus isn't running on dell02 either, there's no >>>>>> gain there. >>>>>> >>>>>> I did clear some of that lint out via "ceph mgr fail". >>>>>> >>>>>> So here's the latest. There are strange things happening >>>>>> at the base OS level that keep host ceph08 from running >>>>>> its OSDs anymore. At boot, device /dev/sdb suddenly >>>>>> changes to /dev/sdd (????) and there seem to be I/O >>>>>> errors. It's really strange, but I'm going to replace the >>>>>> physical drive and that will hopefully cure that. >>>>>> >>>>>> The problem is, reef and earlier releases seem to have a >>>>>> lot of trouble in deleting OSDs that aren't running. As >>>>>> I've noted before, they tend to get permanently stuck in >>>>>> the "deleting" state. When I cannot restart the OSD, the >>>>>> only cure for that has been to run around the system and >>>>>> apply brute force until things clear up. >>>>>> >>>>>> I did a dashboard purge of the OSDs on ceph08 and that >>>>>> removed them from the GUI (they'd already drained). I >>>>>> also banged on things until I got them out of the OSD >>>>>> tree display and then did a crush delete on host ceph08. >>>>>> And, incidentally, the OSD tree works on simple host >>>>>> names, not FQDNs like the rest of ceph! >>>>>> >>>>>> So in theory, I'm ready to jack in new drives and add new >>>>>> OSDs to ceph08. Except: >>>>>> >>>>>> # ceph health detail >>>>>> HEALTH_ERR 2 failed cephadm daemon(s); Module >>>>>> 'prometheus' has failed: gaierror(-2, 'Name or service >>>>>> not known'); too many PGs per OSD (648 > max 560) >>>>>> [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) >>>>>> daemon osd.2 on ceph08.internal.mousetech.com is in >>>>>> error state >>>>>> daemon osd.4 on ceph08.internal.mousetech.com is in >>>>>> error state >>>>>> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >>>>>> gaierror(-2, 'Name or service not known') >>>>>> Module 'prometheus' has failed: gaierror(-2, 'Name or >>>>>> service not known') >>>>>> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >>>>>> >>>>>> Somewhere deep inside ceph, those deleted OSDs still >>>>>> exist. Likely because ceph08 hasn't deleted the systemd >>>>>> units that run them. >>>>>> >>>>>> I'm going to try removing/re-installing prometheus. since >>>>>> it's now showing up in ceph health. I think last time I >>>>>> had zombie OSDs I had to brute-force delete their >>>>>> corresponding directories under /var/lib/ceph. >>>>>> >>>>>> On 3/27/25 14:01, Eugen Block wrote: >>>>>>> ceph config-key rm >>>>>>> mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0 >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It's rather dense. I get 2 very long lines which come from mgr/cephadm/host.dell02.mousetech.com and mgr/cephadm/host.ceph08.mousetech.com The ceph08 entry references the prometheus on ceph08 and mgr and ceph-exporter on dell02. The dell02 entry references the container image for prometheus node. On 3/29/25 05:13, Eugen Block wrote:
How about this:
ceph config-key dump | grep -v history
Can you spot any key regarding dell02 that doesn't belong there?
Zitat von Tim Holloway <timh@mousetech.com>:
Only the stuff that defines the rgw daemon on dell02.
On 3/28/25 19:23, Eugen Block wrote:
Do you find anything related to dell02 in config dump?
ceph config dump | grep -C2 dell02
Zitat von Tim Holloway <timh@mousetech.com>:
I'm guessing that the configuration issues come from the dashboard wanting the prometheus API at 9505, versus prometheus itself on 9283.
Regardless, that didn't fix the message. As far as I can tell, the "service not known" is coming from something trying to contact prometheus on host dell02, and dell02 isn't running prometheus, since the way I got it working at all was to do a generic "ceph orch apply" without arguments. The problem being that apparently the YAML config is still lurking around in the background even though it never spawned the requested instances.
What I'll probably do is remove prometheus, try an "orch apply" with arguments on the command line (deploy 2 hosts to ceph02 and dell02), see what works or breaks, and if that succeeds, try again with the YAML. Which won't totally prove everything's fixed, since it could be drawing on the hidden stuff that won't go away, but at least it would make it superficially clean.
On 3/28/25 18:52, Eugen Block wrote:
There's still some misconfiguration, it appears. It can be confusing, but one thing is the mgr module "prometheus" which provides additional cluster data, it runs on default port 9283. The other is "prometheus server", which collects all the provided cluster data, typically on port 9095.
So these two configs are wrong:
ceph config set mgr mgr/prometheus/server_addr 10.0.1.58 ceph config set mgr mgr/prometheus/server_port 9095
Those should be:
ceph config get mgr mgr/prometheus/server_addr 0.0.0.0 ceph config get mgr mgr/prometheus/server_port 9283
I assume that's why the module is still failing. Can you give that a try and report back?
Zitat von Tim Holloway <timh@mousetech.com>:
OK. I didn't realize I'd pasted the wrong orch ls output. Yes, it's "1/1" and has been for an hour. And yes, I did mis-type the port.
The final complaint appears to arise from something looking for prometheus at the failed deployment location dell02. The dashboard says:
The mgr/prometheus module at dell02.mousetech.com:9095 is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer ...
ceph mgr services shows:
# ceph mgr services { "dashboard":"https://10.0.1.52:8443/", "prometheus":"http://10.0.1.58:9095/" }
On 3/28/25 09:55, Eugen Block wrote: > Hi, > >> Since ceph orch ls wouldn't tell me /where/ the new prometheus >> was deployed, I used the hosts tab in the dashboard to find it. > > ceph orch ps --daemon-type prometheus > > would show you where it tried to place the daemon. > > So prometheus is now actually up and running? Just to confirm > because you pasted the output of 'ceph orch ls' when it wasn't > (yet?). > >> ceph dashboard set-prometheus-api-port 909 > > Is this a c&p mistake or did you actually miss the 5 here? > >> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >> gaierror(-2, 'Name or service not known') > > This seems to be a DNS issue, both relevant places in the code > for "gaierror" point to either > > https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... > or > > https://github.com/ceph/ceph/blob/be5dba538167f282c4ec74ea3cae958c8bd79830/s... > where one tries a "_dns_lookup" and the other a "resolve_ip". > What does 'ceph mgr services' show? > > Zitat von Tim Holloway <timh@mousetech.com>: > >> OK! Success of a sort. >> >> I removed and re-installed each of the failed services in turn >> using the "ceph orch rm" command followed by "ceph orch apply". >> They came up with default settings (1 server), but they did >> come up. >> >> Finally, I tried it with prometheus. This gave me: >> >> prometheus ?:9095 0/1 - 10s >> count:1 >> >> However, in order for the dashboard to be happy, I had to >> supply more info. Since ceph orch ls wouldn't tell me /where/ >> the new prometheus was deployed, I used the hosts tab in the >> dashboard to find it. >> >> Following that, I had to set the following: >> >> ceph config set mgr mgr/prometheus/server_addr 10.0.1.58 >> >> ceph config set mgr mgr/prometheus/server_port 9095 >> >> ceph dashboard set-prometheus-api-host 10.0.1.58 (ceph08) >> >> ceph dashboard set-prometheus-api-port 909 >> >> Once all of the above were set, the dashboard stopped >> complaining about being able to access the prometheus API. >> >> However, one last wart remains. Despite being up and running >> (and confirmed listening on ceph09 port 9095), I do get this: >> >> # ceph health detail >> HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name >> or service not known'); too many PGs per OSD (648 > max 560) >> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >> gaierror(-2, 'Name or service not known') >> Module 'prometheus' has failed: gaierror(-2, 'Name or >> service not known') >> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >> >> On 3/28/25 08:53, Tim Holloway wrote: >>> Actually, I did deploy a new mds node yesterday. But I >>> followed your instructions and successfully removed and >>> re-installed ceph-exporter (4 nodes). So that part works. >>> >>> On 3/28/25 07:28, Eugen Block wrote: >>>> Okay, next I would keep prometheus disabled to see if the mgr >>>> works properly. So disable the module again, and also reset >>>> the dashboard setting to an empty value: >>>> >>>> ceph dashboard reset-prometheus-api-host >>>> >>>> Then see if you get an mds daemon deployed. Or test it by >>>> removing and redeploying ceph-exporter or crash or something, >>>> anything to test if the mgr is able to remove and deploy >>>> other services. >>>> >>>> Zitat von Tim Holloway <timh@mousetech.com>: >>>> >>>>> Thanks for the info on removing stubborn dead OSDs. The >>>>> actual syntax required was: >>>>> >>>>> cephadm rm-daemon --name osd.2 --fsid <fsid>--force >>>>> >>>>> On the "too many pgs", that's because I'm down 2 OSDs. I've >>>>> got new drives, but they were waiting to clear out the dead >>>>> stuff. I know it's risky, but I have backups. >>>>> >>>>> Recall that the start of this thread was on a HEALTH_OK >>>>> system and prometheus was not activating. The OSD stuff was >>>>> just a distraction. >>>>> >>>>> I did notice that the attempt to add a new mds did work >>>>> after I did a "ceph mgr fail", so it's only prometheus >>>>> that's a permanent problem. >>>>> >>>>> Here's the latest health after clearing out the dead OSDs: >>>>> >>>>> # ceph health detail >>>>> HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, >>>>> 'Name or service not known'); too many PGs per OSD (648 > >>>>> max 560) >>>>> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >>>>> gaierror(-2, 'Name or service not known') >>>>> Module 'prometheus' has failed: gaierror(-2, 'Name or >>>>> service not known') >>>>> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >>>>> >>>>> And yes, disabling prometheus will make the "name or service >>>>> not known" errors go away. >>>>> >>>>> >>>>> On 3/28/25 02:49, Eugen Block wrote: >>>>>> Did you disable the prometheus module? I would expect the >>>>>> warning to clear if you did. >>>>>> >>>>>>> Somewhere deep inside ceph, those deleted OSDs still >>>>>>> exist. Likely because ceph08 hasn't deleted the systemd >>>>>>> units that run them. >>>>>> >>>>>> Or do you still see those OSDs in 'cephadm ls' output on >>>>>> ceph08? If you do, and if those OSDs are really already >>>>>> drained/purged, you can remove them with 'cephadm rm-daemon >>>>>> --name osd.2'. And I would try to get the MGR into a >>>>>> working state first, before you try to deploy prometheus >>>>>> again. So my recommendation is to get into HEALTH_OK first. >>>>>> And btw, "TOO_MANY_PGS: too many PGs per OSD (648 > max >>>>>> 560)" is serious, you can end up with inactive PGs during >>>>>> recovery, so I'd also consider checking the pools and their >>>>>> PGs. >>>>>> >>>>>> Zitat von Tim Holloway <timh@mousetech.com>: >>>>>> >>>>>>> Thanks for your patience. >>>>>>> >>>>>>> host ceph06 isn't referenced in the config database. I >>>>>>> think I've finally purged it. I also reset the dashboard >>>>>>> API host address from ceph08 to dell02. But since >>>>>>> prometheus isn't running on dell02 either, there's no gain >>>>>>> there. >>>>>>> >>>>>>> I did clear some of that lint out via "ceph mgr fail". >>>>>>> >>>>>>> So here's the latest. There are strange things happening >>>>>>> at the base OS level that keep host ceph08 from running >>>>>>> its OSDs anymore. At boot, device /dev/sdb suddenly >>>>>>> changes to /dev/sdd (????) and there seem to be I/O >>>>>>> errors. It's really strange, but I'm going to replace the >>>>>>> physical drive and that will hopefully cure that. >>>>>>> >>>>>>> The problem is, reef and earlier releases seem to have a >>>>>>> lot of trouble in deleting OSDs that aren't running. As >>>>>>> I've noted before, they tend to get permanently stuck in >>>>>>> the "deleting" state. When I cannot restart the OSD, the >>>>>>> only cure for that has been to run around the system and >>>>>>> apply brute force until things clear up. >>>>>>> >>>>>>> I did a dashboard purge of the OSDs on ceph08 and that >>>>>>> removed them from the GUI (they'd already drained). I also >>>>>>> banged on things until I got them out of the OSD tree >>>>>>> display and then did a crush delete on host ceph08. And, >>>>>>> incidentally, the OSD tree works on simple host names, not >>>>>>> FQDNs like the rest of ceph! >>>>>>> >>>>>>> So in theory, I'm ready to jack in new drives and add new >>>>>>> OSDs to ceph08. Except: >>>>>>> >>>>>>> # ceph health detail >>>>>>> HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' >>>>>>> has failed: gaierror(-2, 'Name or service not known'); too >>>>>>> many PGs per OSD (648 > max 560) >>>>>>> [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) >>>>>>> daemon osd.2 on ceph08.internal.mousetech.com is in >>>>>>> error state >>>>>>> daemon osd.4 on ceph08.internal.mousetech.com is in >>>>>>> error state >>>>>>> [ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: >>>>>>> gaierror(-2, 'Name or service not known') >>>>>>> Module 'prometheus' has failed: gaierror(-2, 'Name or >>>>>>> service not known') >>>>>>> [WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560) >>>>>>> >>>>>>> Somewhere deep inside ceph, those deleted OSDs still >>>>>>> exist. Likely because ceph08 hasn't deleted the systemd >>>>>>> units that run them. >>>>>>> >>>>>>> I'm going to try removing/re-installing prometheus. since >>>>>>> it's now showing up in ceph health. I think last time I >>>>>>> had zombie OSDs I had to brute-force delete their >>>>>>> corresponding directories under /var/lib/ceph. >>>>>>> >>>>>>> On 3/27/25 14:01, Eugen Block wrote: >>>>>>>> ceph config-key rm >>>>>>>> mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0 >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Almost forgot to say. I switched out disks and got rid of the OSD errors. I actually found a third independent location, so it should be a lot more failure resistant now. So now it's only the prometheus stuff that's still complaining. Everything else is happy.
Let me close this out by describing where I went afterwards. I did a ceph orch apply with explicit --placement of 2 servers, one being the existing ceph08 and the other being the dell02 that was getting the "unknown service" errors. This caused the dell02 machine to get a working prometheus and shut up about the error. The ceph08 machine reported an error. It claimed that apparently the prometheus port was already in use. As it was, since prometheus was already running. The deployer should properly have understood that and acted in a way that allowed the deployment to process without complaint. To clear the port in use issue, I restarted prometheus on ceph08, then when it persisted did a "ceph mgr fail". That cleared all of the prometheus-related complaints and gave me a "HEALTH OK" status. I wasn't brave enough to attempt the YAML version again, considering that's where the problem started. I also didn't attempt to try an "orch apply" that omitted any running service host, for fear it wouldn't remove the omitted host. So the problem is fixed, but it took a lot of banging and hammering to make it work. Tim On 3/28/25 19:12, Tim Holloway wrote:
Almost forgot to say. I switched out disks and got rid of the OSD errors. I actually found a third independent location, so it should be a lot more failure resistant now.
So now it's only the prometheus stuff that's still complaining. Everything else is happy. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Tim, I'm glad you sorted it out. But I'm wondering, did the prometheus spec file ever work? I had just assumed that it had since you wrote you had prometheus up and running before, so I didn't even question the "networks" parameter in there. Now that you say you only used the "--placement" parameter, I started to wonder. Since it's the only "extra" paramter in your spec file, I must assume that it's the root cause of the failures. That would probably easy to test... if you still have the patience. ;-) Zitat von Tim Holloway <timh@mousetech.com>:
Let me close this out by describing where I went afterwards.
I did a ceph orch apply with explicit --placement of 2 servers, one being the existing ceph08 and the other being the dell02 that was getting the "unknown service" errors.
This caused the dell02 machine to get a working prometheus and shut up about the error. The ceph08 machine reported an error. It claimed that apparently the prometheus port was already in use. As it was, since prometheus was already running. The deployer should properly have understood that and acted in a way that allowed the deployment to process without complaint.
To clear the port in use issue, I restarted prometheus on ceph08, then when it persisted did a "ceph mgr fail". That cleared all of the prometheus-related complaints and gave me a "HEALTH OK" status.
I wasn't brave enough to attempt the YAML version again, considering that's where the problem started. I also didn't attempt to try an "orch apply" that omitted any running service host, for fear it wouldn't remove the omitted host.
So the problem is fixed, but it took a lot of banging and hammering to make it work.
Tim
On 3/28/25 19:12, Tim Holloway wrote:
Almost forgot to say. I switched out disks and got rid of the OSD errors. I actually found a third independent location, so it should be a lot more failure resistant now.
So now it's only the prometheus stuff that's still complaining. Everything else is happy. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, I never used a spec file before now. It was all done directly originally. One thing that came up, however, is that my 14 year old motherboards seem to have been rejecting the extra disks I've been trying to add. That includes a spinning disk, a SATA SSD and even an M.2 PCI adapter. Which, unfortunately, my mobos only support PCIe 2 anyway. So I'm going to be doing some offline hardware testing on a spare box and worry about the spec file some other day. Thanks for all the help! Tim On 4/1/25 02:43, Eugen Block wrote:
Hi Tim,
I'm glad you sorted it out. But I'm wondering, did the prometheus spec file ever work? I had just assumed that it had since you wrote you had prometheus up and running before, so I didn't even question the "networks" parameter in there. Now that you say you only used the "--placement" parameter, I started to wonder. Since it's the only "extra" paramter in your spec file, I must assume that it's the root cause of the failures. That would probably easy to test... if you still have the patience. ;-)
Zitat von Tim Holloway <timh@mousetech.com>:
Let me close this out by describing where I went afterwards.
I did a ceph orch apply with explicit --placement of 2 servers, one being the existing ceph08 and the other being the dell02 that was getting the "unknown service" errors.
This caused the dell02 machine to get a working prometheus and shut up about the error. The ceph08 machine reported an error. It claimed that apparently the prometheus port was already in use. As it was, since prometheus was already running. The deployer should properly have understood that and acted in a way that allowed the deployment to process without complaint.
To clear the port in use issue, I restarted prometheus on ceph08, then when it persisted did a "ceph mgr fail". That cleared all of the prometheus-related complaints and gave me a "HEALTH OK" status.
I wasn't brave enough to attempt the YAML version again, considering that's where the problem started. I also didn't attempt to try an "orch apply" that omitted any running service host, for fear it wouldn't remove the omitted host.
So the problem is fixed, but it took a lot of banging and hammering to make it work.
Tim
On 3/28/25 19:12, Tim Holloway wrote:
Almost forgot to say. I switched out disks and got rid of the OSD errors. I actually found a third independent location, so it should be a lot more failure resistant now.
So now it's only the prometheus stuff that's still complaining. Everything else is happy. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Tim Holloway