- ceph-users - lists.ceph.io

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 27 May '26

27 May '26

Hi! On 26/05/26 18:16, C U via ceph-users wrote: > fqdn on the dashboard needs working certificate validation. > > "Squid did cert work inline → bad cert raised crypto.Error → caught and > rewrapped as ServerConfigException cleanly. Tentacle does cert work via a > subprocess; failures can now come from (a) the fork itself, (b) PyO3 > init crashing in the child (the documented subinterpreter collision), or > (c) > genuine cert errors marshalled back as JSON. When (b) happens, the > dashboard module's cert provisioning never completes, the module ends up > half-initialized, and downstream dashboard→RGW calls fail with > SignatureDoesNotMatch because the dashboard is operating without > properly loaded > credentials/sigv4 plumbing." In this context you mean the validation of the certificate build around the FQDN of the RGW VIP, right? The certificate in question is a public one, the CA is correctly installed in the system paths, an "openssl s_client -showcerts -connect vip.domain.com:443" shows no error (verify: return 1) and has the correct FQDN. I executed the openssl command within the active MGR container. Is there an other location where I the CA should be (if the problem is indeed the certificate validation)? Cheers Iztok > > On 5/23/26 17:54, Eugen Block via ceph-users wrote: >> Hi, >> >> I might be wrong, but I don't think it's a cert issue. If you look in >> the mgr log, do you see more information that just the >> SignatureDoesNotMatch error? >> >> Could it be mismatching dashboard-rgw-api settings? Have you checked >> these settings? >> >> ceph dashboard get-rgw-api-admin-resource >> >> ceph dashboard get-rgw-api-access-key >> >> ceph dashboard get-rgw-api-secret-key >> >> They should match with: >> >> radosgw-admin user info --uid dashboard | jq -r '.keys' >> >> But as I wrote, it might something else, I would expect the mgr log to >> contain more details. >> >> Regards >> Eugen >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> >>> Hi to all! >>> >>> After upgrading my cluster from squid to tentacle (now on 20.2.1) I'm >>> getting the following error when I try to access any 'page' in the >>> Object section of the Ceph Dashboard: >>> >>> Error connecting to Object Gateway: RGW REST API failed request with >>> status code 403 >>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx0000000b5bf42356' b'85174-006a0d95b0-308299-eros","HostId":"308299-zone-default"}') >>> >>> I'm pretty sure that in Squid it was working (I don't usually access >>> the Dashboard, but I did after the upgrade to check if everything is >>> ok). >>> >>> The error 'SignatureDoesNotMatch' leads me to believe that there is a >>> problem somewhere with the SSL certificates. But I put >>> RGW_API_SSL_VERIFY to false... >>> >>> I have a SSL certificate (issued by harica.gr) which has different >>> SANs including the hostname of all the nodes where RGW daemon is >>> running (plus a cluster hostname and a wildcard hostname). I used >>> this certificate for the ingress service and for the rgw daemons. >>> Everything is done with a spec file using ceph orchestrator. I'm >>> using the fullchain for the certificates and the the rgw/ingress >>> services are running without any problem >>> >>> Or I'm completely wrong and the problem itself is with the access >>> permissions (error code 403)? >>> >>> Any ideas? >>> >>> Thanks >>> Iztok >>> >>> -- >>> Iztok Gregori >>> ICT Systems and Services >>> Elettra - Sincrotrone Trieste S.C.p.A. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Eugen Block 27 May '26

27 May '26

I'm not sure about the curl versions. Did you notice the response from "C U" the day before yesterday? He or she commented: > fqdn on the dashboard needs working certificate validation. > "Squid did cert work inline → bad cert raised crypto.Error → caught > and rewrapped as ServerConfigException cleanly. Tentacle does cert > work via a subprocess; failures can now come from (a) the fork itself, (b) PyO3 init crashing in the child (the documented subinterpreter collision), or (c) genuine cert errors marshalled back as JSON. When (b) happens, the dashboard module's cert provisioning never completes, the module ends up half-initialized, and downstream dashboard→RGW calls fail with SignatureDoesNotMatch because the dashboard is operating without properly loaded credentials/sigv4 plumbing." Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > On 27/05/26 12:17, Eugen Block via ceph-users wrote: >> Hi, >> >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> >>> On 27/05/26 09:36, Eugen Block via ceph-users wrote: >>>> The description in this tracker [0] looks identical, but it is >>>> supposed to be fixed. Do you have rgw_dns_name set? >>> >>> Yes, at certain point I configured client.rgw rgw_dns_name to the >>> FQDN belonging to the virtual IP of the ingress service. >> >> does it mean that it works now? > > No, still doesn't work. To try I've also remove it and then again > set it, but without luck. > > I also tried to call the API endpoint directly with curl using the > dashboard credentials and found that from 'older' versions of curl > (provided for example by Rocky 9) the command gives me the same > error ('SignatureDoesNotMatch'), but when I was using newer one > (8.5.0 on Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer my > query. > > Could it be the problem? > >> >>> BTW, how you 'unset' a configuration variable, you just put a >>> empty string or you use the rm option? >>> >>> ceph config set client.rgw rgw_dns_name '' >>> or >>> ceph config rm client.rgw rgw_dns_name >>> ? >> >> That's the correct one (config rm). > > Thank you! I sometimes get lost in the documentation :-) > > Cheers > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 27 May '26

27 May '26

On 27/05/26 12:17, Eugen Block via ceph-users wrote: > Hi, > > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > >> On 27/05/26 09:36, Eugen Block via ceph-users wrote: >>> The description in this tracker [0] looks identical, but it is >>> supposed to be fixed. Do you have rgw_dns_name set? >> >> Yes, at certain point I configured client.rgw rgw_dns_name to the FQDN >> belonging to the virtual IP of the ingress service. > > does it mean that it works now? No, still doesn't work. To try I've also remove it and then again set it, but without luck. I also tried to call the API endpoint directly with curl using the dashboard credentials and found that from 'older' versions of curl (provided for example by Rocky 9) the command gives me the same error ('SignatureDoesNotMatch'), but when I was using newer one (8.5.0 on Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer my query. Could it be the problem? > >> BTW, how you 'unset' a configuration variable, you just put a empty >> string or you use the rm option? >> >> ceph config set client.rgw rgw_dns_name '' >> or >> ceph config rm client.rgw rgw_dns_name >> ? > > That's the correct one (config rm). Thank you! I sometimes get lost in the documentation :-) Cheers Iztok

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Eugen Block 27 May '26

27 May '26

Hi, Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > On 27/05/26 09:36, Eugen Block via ceph-users wrote: >> The description in this tracker [0] looks identical, but it is >> supposed to be fixed. Do you have rgw_dns_name set? > > Yes, at certain point I configured client.rgw rgw_dns_name to the > FQDN belonging to the virtual IP of the ingress service. does it mean that it works now? > BTW, how you 'unset' a configuration variable, you just put a empty > string or you use the rm option? > > ceph config set client.rgw rgw_dns_name '' > or > ceph config rm client.rgw rgw_dns_name > ? That's the correct one (config rm). > > >> >> [0] https://tracker.ceph.com/issues/62396 >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> >>> Hi Eugen! >>> >>> >>> On 23/05/26 23:54, Eugen Block via ceph-users wrote: >>>> Hi, >>>> >>>> I might be wrong, but I don't think it's a cert issue. If you >>>> look in the mgr log, do you see more information that just the >>>> SignatureDoesNotMatch error? >>> >>> On further investigations I'm also convinced that is nothing to do >>> with the certificates but more about the permissions of the >>> 'dashboard' user. This is the exception that is thrown from the >>> mgr logs: >>> >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-01.domain.com, port=4443, ssl=True >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-03.domain.com, port=4443, ssl=True >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-02.domain.com, port=4443, ssl=True >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-04.domain.com, port=4443, ssl=True >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-05.domain.com, port=4443, ssl=True >>>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>>> [0.005s] [admin] [308.0B] /api/health/snapshot >>>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>>> [0.007s] [admin] [22.0B] /api/prometheus/alertgroup >>>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>>> [0.012s] [admin] [376.0B] /api/multi-cluster/get_config >>>> [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] >>>> [0.028s] [admin] [1.2K] /api/summary >>>> [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 >>>> [dashboard ERROR rgw_client] RGW REST API failed request with >>>> status code 403 >>>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >>>> b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') >>>> Traceback (most recent call last): >>>> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", >>>> line 413, in __init__ >>>> self.userid = self._get_user_id(self.admin_path) if >>>> self.got_keys_from_config \ >>>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, >>>> in func_wrapper >>>> return func( >>>> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", >>>> line 448, in _get_user_id >>>> response = request() >>>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, >>>> in __call__ >>>> resp = self.rest_client.do_request(method, self._gen_path(), params, >>>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, >>>> in do_request >>>> raise RequestException( >>>> dashboard.rest_client.RequestException: RGW REST API failed >>>> request with status code 403 >>>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >>>> b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') >>> >>> >>>> >>>> Could it be mismatching dashboard-rgw-api settings? Have you >>>> checked these settings? >>>> >>>> ceph dashboard get-rgw-api-admin-resource >>> >>> It's the default, 'admin'. I also reset the value with >>> "reset-rgw-api- admin-resource", so technically now is 'unset'. >>> Question what should be the value if not 'admin'? >>> >>>> >>>> ceph dashboard get-rgw-api-access-key >>>> >>>> ceph dashboard get-rgw-api-secret-key >>>> >>>> They should match with: >>>> >>>> radosgw-admin user info --uid dashboard | jq -r '.keys' >>> >>> The values are matching, I'see in both get-rgw-api-secret-key and >>> get- rgw-api-access-key something like this: >>> {'<REALM>': '<(ACCESS}SECRET)_KEY>'} >>> >>>> >>>> But as I wrote, it might something else, I would expect the mgr >>>> log to contain more details. >>>> >>> >>> Indeed, I'll tried to backtrace the error in the code and I see >>> that is failing in the rgw_client.py when is trying to get the >>> user_id "of the user that is used to communicate with the RGW >>> Admin Ops API". I suppose that is the 'admin' user from >>> rgw-api-admin-resource, am I right? >>> >>> I didn't had time till now to investigate further. >>> >>> >>> Thanks! >>> Iztok >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 27 May '26

27 May '26

On 27/05/26 09:36, Eugen Block via ceph-users wrote: > The description in this tracker [0] looks identical, but it is supposed > to be fixed. Do you have rgw_dns_name set? Yes, at certain point I configured client.rgw rgw_dns_name to the FQDN belonging to the virtual IP of the ingress service. BTW, how you 'unset' a configuration variable, you just put a empty string or you use the rm option? ceph config set client.rgw rgw_dns_name '' or ceph config rm client.rgw rgw_dns_name ? > > [0] https://tracker.ceph.com/issues/62396 > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > >> Hi Eugen! >> >> >> On 23/05/26 23:54, Eugen Block via ceph-users wrote: >>> Hi, >>> >>> I might be wrong, but I don't think it's a cert issue. If you look in >>> the mgr log, do you see more information that just the >>> SignatureDoesNotMatch error? >> >> On further investigations I'm also convinced that is nothing to do >> with the certificates but more about the permissions of the >> 'dashboard' user. This is the exception that is thrown from the mgr logs: >> >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-01.domain.com, port=4443, ssl=True >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-03.domain.com, port=4443, ssl=True >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-02.domain.com, port=4443, ssl=True >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-04.domain.com, port=4443, ssl=True >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-05.domain.com, port=4443, ssl=True >>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>> [0.005s] [admin] [308.0B] /api/health/snapshot >>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>> [0.007s] [admin] [22.0B] /api/prometheus/alertgroup >>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>> [0.012s] [admin] [376.0B] /api/multi-cluster/get_config >>> [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] >>> [0.028s] [admin] [1.2K] /api/summary >>> [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 >>> [dashboard ERROR rgw_client] RGW REST API failed request with status >>> code 403 >>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >>> b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') >>> Traceback (most recent call last): >>> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line >>> 413, in __init__ >>> self.userid = self._get_user_id(self.admin_path) if >>> self.got_keys_from_config \ >>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in >>> func_wrapper >>> return func( >>> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line >>> 448, in _get_user_id >>> response = request() >>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, in >>> __call__ >>> resp = self.rest_client.do_request(method, self._gen_path(), params, >>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, in >>> do_request >>> raise RequestException( >>> dashboard.rest_client.RequestException: RGW REST API failed request >>> with status code 403 >>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >>> b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') >> >> >>> >>> Could it be mismatching dashboard-rgw-api settings? Have you checked >>> these settings? >>> >>> ceph dashboard get-rgw-api-admin-resource >> >> It's the default, 'admin'. I also reset the value with "reset-rgw-api- >> admin-resource", so technically now is 'unset'. Question what should >> be the value if not 'admin'? >> >>> >>> ceph dashboard get-rgw-api-access-key >>> >>> ceph dashboard get-rgw-api-secret-key >>> >>> They should match with: >>> >>> radosgw-admin user info --uid dashboard | jq -r '.keys' >> >> The values are matching, I'see in both get-rgw-api-secret-key and get- >> rgw-api-access-key something like this: >> {'<REALM>': '<(ACCESS}SECRET)_KEY>'} >> >>> >>> But as I wrote, it might something else, I would expect the mgr log >>> to contain more details. >>> >> >> Indeed, I'll tried to backtrace the error in the code and I see that >> is failing in the rgw_client.py when is trying to get the user_id "of >> the user that is used to communicate with the RGW Admin Ops API". I >> suppose that is the 'admin' user from rgw-api-admin-resource, am I right? >> >> I didn't had time till now to investigate further. >> >> >> Thanks! >> Iztok >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Eugen Block 27 May '26

27 May '26

The description in this tracker [0] looks identical, but it is supposed to be fixed. Do you have rgw_dns_name set? [0] https://tracker.ceph.com/issues/62396 Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi Eugen! > > > On 23/05/26 23:54, Eugen Block via ceph-users wrote: >> Hi, >> >> I might be wrong, but I don't think it's a cert issue. If you look >> in the mgr log, do you see more information that just the >> SignatureDoesNotMatch error? > > On further investigations I'm also convinced that is nothing to do > with the certificates but more about the permissions of the > 'dashboard' user. This is the exception that is thrown from the mgr > logs: > >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-01.domain.com, port=4443, ssl=True >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-03.domain.com, port=4443, ssl=True >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-02.domain.com, port=4443, ssl=True >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-04.domain.com, port=4443, ssl=True >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-05.domain.com, port=4443, ssl=True >> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >> [0.005s] [admin] [308.0B] /api/health/snapshot >> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >> [0.007s] [admin] [22.0B] /api/prometheus/alertgroup >> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >> [0.012s] [admin] [376.0B] /api/multi-cluster/get_config >> [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] >> [0.028s] [admin] [1.2K] /api/summary >> [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 >> [dashboard ERROR rgw_client] RGW REST API failed request with >> status code 403 >> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >> b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') >> Traceback (most recent call last): >> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line >> 413, in __init__ >> self.userid = self._get_user_id(self.admin_path) if >> self.got_keys_from_config \ >> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in >> func_wrapper >> return func( >> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line >> 448, in _get_user_id >> response = request() >> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, in __call__ >> resp = self.rest_client.do_request(method, self._gen_path(), params, >> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, in >> do_request >> raise RequestException( >> dashboard.rest_client.RequestException: RGW REST API failed request >> with status code 403 >> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >> b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') > > >> >> Could it be mismatching dashboard-rgw-api settings? Have you >> checked these settings? >> >> ceph dashboard get-rgw-api-admin-resource > > It's the default, 'admin'. I also reset the value with > "reset-rgw-api-admin-resource", so technically now is 'unset'. > Question what should be the value if not 'admin'? > >> >> ceph dashboard get-rgw-api-access-key >> >> ceph dashboard get-rgw-api-secret-key >> >> They should match with: >> >> radosgw-admin user info --uid dashboard | jq -r '.keys' > > The values are matching, I'see in both get-rgw-api-secret-key and > get-rgw-api-access-key something like this: > {'<REALM>': '<(ACCESS}SECRET)_KEY>'} > >> >> But as I wrote, it might something else, I would expect the mgr log >> to contain more details. >> > > Indeed, I'll tried to backtrace the error in the code and I see that > is failing in the rgw_client.py when is trying to get the user_id > "of the user that is used to communicate with the RGW Admin Ops > API". I suppose that is the 'admin' user from > rgw-api-admin-resource, am I right? > > I didn't had time till now to investigate further. > > > Thanks! > Iztok > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

NVMe-oF (14 msgs, 4 threads) is barely on the radar
by C U 27 May '26

27 May '26

did a little research into nvmeof gateway. tldr, more write throughput. protocol translation overhead, not a concern. mon managed gateway failover, stable. https://github.com/tu503/nvmeof-gateway regards,

1 0

Re: Cluster Dead After Network Failure. Connection Timeout.
by Eugen Block 26 May '26

26 May '26

Hi, this sound a bit like a "classic" quorum loss, resulting in cascading daemon failures. There seems to be one surviving MON (blade3n2), that could be a good starting point for disaster recovery by reducing the monmap to this one MON. This should give you back qourum and a working cluster. You might need to run 'systemctl reset-failed...' to let systemd start the containers. Although the product has been discontinued, the section of the SUSE docs [0] is still relevant ("Restoring the MONs quorum"). Back up the mon store of each node before you do this, just in case. The procedure itself has worked many times for me, but maybe there's an easier way, especially if you're not too familiar with cephadm or this procedure. But before you do that, do you have MON logs with an explanation why they refuse to start? Regarding Ceph images, your cluster uses af0c5903e901 for the Ceph services, what does 'docker images | grep af0c5903e901' show? I doubt it's "ceph-mon:latest", at least I haven't seen those in use with cephadm (I have the impression that this is a "regular" cephadm cluster). Regards, Eugen [0] https://documentation.suse.com/pt-br/ses/7.1/html/ses-all/bp-troubleshootin… Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > Hi, > > The network my 4-node cluster uses broke down after a driver issue a week > ago. Now, as the network resumed normal operation, but my Ceph 19 cluster > first said HEALTH_WARN and informed me of a lengthy recovery process, but > some one hour later, I only found this error message: > > mixtile@blade3n1:~$ sudo ceph -s > [sudo] password for mixtile: > 2026-05-25T13:35:51.685+0200 ffff9701f180 0 monclient(hunting): > authenticate timed out after 300 > [errno 110] RADOS timed out (error connecting to the cluster) > > A restart of all nodes did *not* help. Even worse: The Docker containers > with the various processes (mon, mrg, crash,…) started disappearing one by > one! Here is what remained: > > mixtile@blade3n1:~$ docker ps -a > CONTAINER ID IMAGE COMMAND > CREATED STATUS PORTS NAMES > 2039b18ba392 quay.io/prometheus/node-exporter:v1.7.0 > "/bin/node_exporter …" 47 minutes ago Up 47 minutes > ceph-8aad3073-39a1-11f1-b > f6e-f2704a1efa9b-node-exporter-blade3n1 > 16cb4a6822f2 af0c5903e901 > "/usr/bin/ceph-crash…" 47 minutes ago Up 47 > minutes ceph-8aad3073-39a1-11f1-b > f6e-f2704a1efa9b-crash-blade3n1 > > mixtile@blade3n2:~$ docker ps -a > CONTAINER ID IMAGE COMMAND > CREATED STATUS PORTS NAMES > 42361a694abf quay.io/prometheus/prometheus:v2.51.0 "/bin/prometheus > --c…" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-prometheus-blade3n2 > 88b085d000a8 quay.io/prometheus/node-exporter:v1.7.0 > "/bin/node_exporter …" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-node-exporter-blade3n2 > 7bb17808bdd8 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager > -…" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-alertmanager-blade3n2 > a43cfe36da29 quay.io/ceph/grafana:10.4.0 "/run.sh" > 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-grafana-blade3n2 > a95140b707ee af0c5903e901 > "/usr/bin/ceph-crash…" 2 hours ago Up 2 > hours ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-crash-blade3n2 > 38d4ca4035b0 af0c5903e901 "/usr/bin/ceph-mon > -…" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-mon-blade3n2 > > mixtile@blade3n3:~$ docker ps -a > CONTAINER ID IMAGE COMMAND > CREATED STATUS PORTS NAMES > d664dfe30bd8 af0c5903e901 > "/usr/bin/ceph-crash…" 2 hours ago Up 2 > hours ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-crash-blade3n3 > a64ac00dc28b quay.io/prometheus/node-exporter:v1.7.0 > "/bin/node_exporter …" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-node-exporter-blade3n3 > f7de98403a10 netdata/netdata "/usr/sbin/run.sh" > > mixtile@blade3n4:~$ docker ps -a > CONTAINER ID IMAGE COMMAND > CREATED STATUS PORTS NAMES > d437cec7d6bf quay.io/prometheus/node-exporter:v1.7.0 > "/bin/node_exporter …" 54 minutes ago Up 53 minutes > ceph-8aad3073-39a1-11f1-b > f6e-f2704a1efa9b-node-exporter-blade3n4 > c6d5ac595857 af0c5903e901 > "/usr/bin/ceph-crash…" 54 minutes ago Up 53 > minutes ceph-8aad3073-39a1-11f1-b > f6e-f2704a1efa9b-crash-blade3n4 > > As you can see, there are much less Ceph-related processes than expected. > The rest hasn't only crashed: In fact, the corresponding images have also > disappeared! Pulling the missing containers didn't work: > > mixtile@blade3n1:~$ docker run ceph-mon > Unable to find image 'ceph-mon:latest' locally > docker: Error response from daemon: pull access denied for ceph-mon, > repository does not exist or may require 'docker login' > > This is my exact system and Ceph version BTW: > > mixtile@blade3n1:~$ ceph -v > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > (stable) > > mixtile@blade3n1:~$ uname -a > Linux blade3n1 6.1.0-1027-rockchip #27 SMP Sun Apr 27 01:54:34 UTC 2025 > aarch64 aarch64 aarch64 GNU/Linux > > The drives my data are stored on seem to be still there (as lsblk said), > but as all most OSD processes are gone, I can no longer access them. I've > got four hosts, of which #1 is the admin node. #2 also hosts Ganesha NFS > for external clients. > > So: What can I do to bring my cluster back to life without endangering my > data? Thank you. > > Kind regards > Jacek Rużyczka > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph Days Seattle this week
by Anthony Middleton 26 May '26

26 May '26

Hey Ceph Friends! If you're in the Seattle area this week, please note that Ceph Days Seattle is this Thursday. If you haven't secured your ticket yet, you're still in luck! Check out the event schedule and ticket information here: https://t.ly/TicketsSeattleCeph. Please spread the word within your circles and help me make this Ceph Day a success! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Cluster Dead After Network Failure. Connection Timeout.
by Jacek Rużyczka 26 May '26

26 May '26

Hi, The network my 4-node cluster uses broke down after a driver issue a week ago. Now, as the network resumed normal operation, but my Ceph 19 cluster first said HEALTH_WARN and informed me of a lengthy recovery process, but some one hour later, I only found this error message: mixtile@blade3n1:~$ sudo ceph -s [sudo] password for mixtile: 2026-05-25T13:35:51.685+0200 ffff9701f180 0 monclient(hunting): authenticate timed out after 300 [errno 110] RADOS timed out (error connecting to the cluster) A restart of all nodes did *not* help. Even worse: The Docker containers with the various processes (mon, mrg, crash,…) started disappearing one by one! Here is what remained: mixtile@blade3n1:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2039b18ba392 quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 47 minutes ago Up 47 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-node-exporter-blade3n1 16cb4a6822f2 af0c5903e901 "/usr/bin/ceph-crash…" 47 minutes ago Up 47 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-crash-blade3n1 mixtile@blade3n2:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 42361a694abf quay.io/prometheus/prometheus:v2.51.0 "/bin/prometheus --c…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-prometheus-blade3n2 88b085d000a8 quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-node-exporter-blade3n2 7bb17808bdd8 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager -…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-alertmanager-blade3n2 a43cfe36da29 quay.io/ceph/grafana:10.4.0 "/run.sh" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-grafana-blade3n2 a95140b707ee af0c5903e901 "/usr/bin/ceph-crash…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-crash-blade3n2 38d4ca4035b0 af0c5903e901 "/usr/bin/ceph-mon -…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-mon-blade3n2 mixtile@blade3n3:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d664dfe30bd8 af0c5903e901 "/usr/bin/ceph-crash…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-crash-blade3n3 a64ac00dc28b quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-node-exporter-blade3n3 f7de98403a10 netdata/netdata "/usr/sbin/run.sh" mixtile@blade3n4:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d437cec7d6bf quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 54 minutes ago Up 53 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-node-exporter-blade3n4 c6d5ac595857 af0c5903e901 "/usr/bin/ceph-crash…" 54 minutes ago Up 53 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-crash-blade3n4 As you can see, there are much less Ceph-related processes than expected. The rest hasn't only crashed: In fact, the corresponding images have also disappeared! Pulling the missing containers didn't work: mixtile@blade3n1:~$ docker run ceph-mon Unable to find image 'ceph-mon:latest' locally docker: Error response from daemon: pull access denied for ceph-mon, repository does not exist or may require 'docker login' This is my exact system and Ceph version BTW: mixtile@blade3n1:~$ ceph -v ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) mixtile@blade3n1:~$ uname -a Linux blade3n1 6.1.0-1027-rockchip #27 SMP Sun Apr 27 01:54:34 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux The drives my data are stored on seem to be still there (as lsblk said), but as all most OSD processes are gone, I can no longer access them. I've got four hosts, of which #1 is the admin node. #2 also hosts Ganesha NFS for external clients. So: What can I do to bring my cluster back to life without endangering my data? Thank you. Kind regards Jacek Rużyczka

1 0