- ceph-users - lists.ceph.io

Re: tentacle 20.2.2 QE validation status
by Ilya Dryomov 29 May '26

29 May '26

On Fri, May 29, 2026 at 4:43 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > We are still missing several approvals (rgw, rbd, orch, ceph-volume) rbd and krbd approved. Thanks, Ilya > Please review ASAP > TIA > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/75878#note-6 >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya >> nvmeof: Aviv, Vallari >> >> upgrade: Patrick >> >> upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD >> upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD >> >> perf-basic: PASSED >> >> crimson-rados: N/A >> >> powercycle: PASSED >> >> TIA

1 0

RBD: group snap rollback invalid argument
by Eugen Block 29 May '26

29 May '26

Hi, I wanted to rollback a group snapshot on Ubuntu 24.04 (rbd client version 19.2.1), the Ceph cluster version is 19.2.3. The client fails with "invalid argument": controller02:~# rbd --id <user> group snap rollback --pool images --group test-servers --snap 20260430_start Rolling back to group snapshot: 0% complete...failed. rbd: rollback group to snapshot failed: (22) Invalid argument controller02:~# ceph -v ceph version 19.2.1 (9efac4a81335940925dd17dbf407bfd6d3860d28) squid (stable) But running the same command (just as admin not as <user>) on a Ceph node works: ceph03:~ # rbd group snap rollback --pool images --group test-servers --snap 20260430_start Rolling back to group snapshot: 100% complete...done. ceph03:~ # ceph -v ceph version 16.2.13-66-g54799ee0666 (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) What seems to be the issue here is that I removed an image from the group snapshot. I wonder if it could be this bug [0] which is supposed to be fixed in 19.2.0 according to the "Released In" field of the Squid backport tracker [1]. This seems a little inconsistent to me, could someone please clarify? Thanks! Eugen [0] https://tracker.ceph.com/issues/66300 [1] https://tracker.ceph.com/issues/66589 -- Eugen Block NDE Netzdesign und -entwicklung AG voice: +49 40 5595175 Postfach 61 03 15 e-mail: eblock(a)nde.ag D-22423 Hamburg Vorstand: Jens-U. Mozdzen Aufsichtsratsvorsitzende: Angelika Torlée-Mozdzen Sitz und Registergericht: Hamburg, HRB 90934 USt-IdNr: DE 814 013 983

1 0

Re: tentacle 20.2.2 QE validation status
by Travis Nielsen 28 May '26

28 May '26

Rook approved. As discussed separately, we will follow up after 20.2.2 on this rbd mirror issue. https://tracker.ceph.com/issues/76967 Thanks, Travis On Thu, May 28, 2026 at 2:02 AM Nitzan Mordechai via dev <dev(a)ceph.io> wrote: > Rados approved > Upgrade approved > > > Summary of analysis: > > https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2022-httpstrackercep… > > ------------------------------ > *From:* Vallari Agrawal via dev <dev(a)ceph.io> > *Sent:* Wednesday, May 27, 2026 6:12 PM > *To:* Afreen <afreen23.git(a)gmail.com>; Aviv Caro <acaro(a)redhat.com> > *Cc:* Yuri Weinstein <yweinste(a)redhat.com>; dev <dev(a)ceph.io>; ceph-users > <ceph-users(a)ceph.io> > *Subject:* [EXTERNAL] Re: [ceph-users] Re: tentacle 20.2.2 QE validation > status > > nvmeof approved cc: @Aviv Caro On Tue, May 26, 2026 at 10: 44 PM Afreen > via dev <dev@ ceph. io> wrote: Dashboard approved On Tue, May 26, 2026 at > 10: 00 PM Yuri Weinstein via ceph-users <ceph-users@ ceph. io> wrote: > Resending because of > nvmeof approved > cc: @Aviv Caro <acaro(a)redhat.com> > > On Tue, May 26, 2026 at 10:44 PM Afreen via dev <dev(a)ceph.io> wrote: > > Dashboard approved > > On Tue, May 26, 2026 at 10:00 PM Yuri Weinstein via ceph-users < > ceph-users(a)ceph.io> wrote: > > Resending because of emails problems > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/75878#note-6 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya > > nvmeof: Aviv, Vallari > > > > upgrade: Patrick > > > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > > > perf-basic: PASSED > > > > crimson-rados: N/A > > > > powercycle: PASSED > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: tentacle 20.2.2 QE validation status
by Nitzan Mordechai 28 May '26

28 May '26

Rados approved Upgrade approved Summary of analysis: https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2022-httpstrackercep… ________________________________ From: Vallari Agrawal via dev <dev(a)ceph.io> Sent: Wednesday, May 27, 2026 6:12 PM To: Afreen <afreen23.git(a)gmail.com>; Aviv Caro <acaro(a)redhat.com> Cc: Yuri Weinstein <yweinste(a)redhat.com>; dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] Re: [ceph-users] Re: tentacle 20.2.2 QE validation status nvmeof approved cc: @Aviv Caro On Tue, May 26, 2026 at 10: 44 PM Afreen via dev <dev@ ceph. io> wrote: Dashboard approved On Tue, May 26, 2026 at 10: 00 PM Yuri Weinstein via ceph-users <ceph-users@ ceph. io> wrote: Resending because of nvmeof approved cc: @Aviv Caro<mailto:acaro@redhat.com> On Tue, May 26, 2026 at 10:44 PM Afreen via dev <dev(a)ceph.io<mailto:dev@ceph.io>> wrote: Dashboard approved On Tue, May 26, 2026 at 10:00 PM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> wrote: Resending because of emails problems On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com<mailto:yweinste@redhat.com>> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75878#note-6<https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > TIA > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

Re: Cluster Dead After Network Failure. Connection Timeout.
by Eugen Block 27 May '26

27 May '26

Please don't drop the list from your responses, you'd benefit from more people reading it. The cephadm ls output isn't really helpful, you need to figure out why docker doesn't start. Either syslog, journald or dmesg or whatever should give some clue. And to me it sounds like there has been more going on beside the network outage, maybe some leftovers from previous deployments or tests or something else that "confuses" docker? Maybe you have both podman and docker installed and since cephadm prefers podman, the containers fail to start? Zitat von Jacek Rużyczka <jacek.ruzyczka(a)googlemail.com>: > I can't run the Ceph container on the master node (blade3n1) anymore. It's > not executed anymore without an error message. Here is what cephadm ls says: > > mixtile@blade3n1:~$ sudo cephadm ls > [ > { > "style": "cephadm:v1", > "name": "mon.blade3n1", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)mon.blade3n1" > , > "enabled": true, > "state": "error", > "service_name": "mon", > "memory_request": null, > "memory_limit": null, > "ports": [], > "container_id": null, > "container_image_name": "quay.io/ceph/ceph:v19", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T14:35:47.634066Z", > "deployed": "2026-04-16T14:35:45.414037Z", > "configured": "2026-04-20T17:16:32.722329Z" > }, > { > "style": "cephadm:v1", > "name": "node-exporter.blade3n1", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b@node-exporter > .blade3n1", > "enabled": true, > "state": "error", > "service_name": "node-exporter", > "ports": [ > 9100 > ], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": "quay.io/prometheus/node-exporter:v1.7.0", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-21T12:55:39.731035Z", > "deployed": "2026-04-21T12:55:38.217675Z", > "configured": "2026-04-21T12:55:39.734369Z" > }, > { > "style": "cephadm:v1", > "name": "ceph-exporter.blade3n1", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b@ceph-exporter > .blade3n1", > "enabled": true, > "state": "error", > "service_name": "ceph-exporter", > "ports": [], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe2 > 19dfc8d0c3efc1f05102a753902f33ee16c26b6cee", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T14:37:32.218782Z", > "deployed": "2026-04-16T14:37:30.612094Z", > "configured": "2026-04-20T17:16:36.139048Z" > }, > { > "style": "cephadm:v1", > "name": "mgr.blade3n1.rrlwwv", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)mgr.blade3n1. > rrlwwv", > "enabled": true, > "state": "error", > "service_name": "mgr", > "memory_request": null, > "memory_limit": null, > "ports": [ > 9283, > 8765, > 8443 > ], > "container_id": null, > "container_image_name": "quay.io/ceph/ceph:v19", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T14:35:54.054151Z", > "deployed": "2026-04-16T14:35:52.430796Z", > "configured": "2026-04-20T17:16:37.612403Z" > }, > { > "style": "cephadm:v1", > "name": "crash.blade3n1", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)crash.blade3n > 1", > "enabled": true, > "state": "error", > "service_name": "crash", > "ports": [], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe2 > 19dfc8d0c3efc1f05102a753902f33ee16c26b6cee", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T14:37:36.855510Z", > "deployed": "2026-04-16T14:37:35.268822Z", > "configured": "2026-04-20T17:16:39.025758Z" > }, > { > "style": "cephadm:v1", > "name": "osd.3", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)osd.3", > "enabled": true, > "state": "error", > "service_name": "osd", > "ports": [], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe2 > 19dfc8d0c3efc1f05102a753902f33ee16c26b6cee", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-23T15:05:00.686688Z", > "deployed": "2026-04-23T15:04:59.176667Z", > "configured": "2026-04-23T15:05:00.693355Z" > }, > { > "style": "cephadm:v1", > "name": "mds.data.blade3n1.eczeqc", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)mds.data.blad > e3n1.eczeqc", > "enabled": true, > "state": "error", > "service_name": "mds.data", > "ports": [], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe2 > 19dfc8d0c3efc1f05102a753902f33ee16c26b6cee", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T15:54:13.264224Z", > "deployed": "2026-04-16T15:54:10.870858Z", > "configured": "2026-04-20T17:16:40.499113Z" > } > ] > > > Am Mi., 27. Mai 2026 um 15:07 Uhr schrieb Jacek Rużyczka < > jacek.ruzyczka(a)googlemail.com>: > >> Hi Eugen, >> >> You might need to run 'systemctl reset-failed...' to let systemd start the >>> containers. >> >> >> I've already done that. No use. Even worse: On node #1, Docker no longer >> starts. When trying to restart the daemon, I get errors like this: >> >> docker.service: Failed with result 'core-dump'. >> >> But before you do that, do you have MON logs with an explanation why they >>> refuse to start? >> >> >> Unfortunately no, not even in the syslog. In the meantime, I was able to >> start another MON via Cephadm (because the Docker instance had even deleted >> the image), but now I've got the problem with the one node, where Docker >> refuses to start. >> >> Regarding Ceph images, your cluster uses af0c5903e901 for the Ceph >>> services, what does 'docker images | grep af0c5903e901' show? >> >> >> On the affected node, nothing 'cause the Docker daemon wouldn't even start. >> >> I have the impression that this is a "regular" cephadm cluster >> >> >> True >> >> BTW, when running the test script supplied by the Docker guys >> https://docs.docker.com/engine/daemon/troubleshoot/, I get some warnings: >> >> - Network Drivers: >> - "bridge": >> - sysctl net.ipv4.ip_forward: disabled >> - sysctl net.ipv6.conf.all.forwarding: disabled >> - sysctl net.ipv6.conf.default.forwarding: disabled >> >> On nodes #2 thru #4, net.ipv4.ip_forward is enabled. >> >> Regards >> Jacek Rużyczka >>

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by C U 27 May '26

27 May '26

then my problem was different. v20 did introduce python subprocesses, py03. On 5/27/26 07:13, Iztok Gregori via ceph-users wrote: > Hi! > > On 26/05/26 18:16, C U via ceph-users wrote: >> fqdn on the dashboard needs working certificate validation. >> >> "Squid did cert work inline → bad cert raised crypto.Error → caught >> and rewrapped as ServerConfigException cleanly. Tentacle does cert >> work via a >> subprocess; failures can now come from (a) the fork itself, (b) >> PyO3 init crashing in the child (the documented subinterpreter >> collision), or (c) >> genuine cert errors marshalled back as JSON. When (b) happens, the >> dashboard module's cert provisioning never completes, the module ends up >> half-initialized, and downstream dashboard→RGW calls fail with >> SignatureDoesNotMatch because the dashboard is operating without >> properly loaded >> credentials/sigv4 plumbing." > > In this context you mean the validation of the certificate build > around the FQDN of the RGW VIP, right? > > The certificate in question is a public one, the CA is correctly > installed in the system paths, an "openssl s_client -showcerts > -connect vip.domain.com:443" shows no error (verify: return 1) and has > the correct FQDN. I executed the openssl command within the active MGR > container. > > Is there an other location where I the CA should be (if the problem is > indeed the certificate validation)? > > Cheers > Iztok > > > > > > > > >> >> On 5/23/26 17:54, Eugen Block via ceph-users wrote: >>> Hi, >>> >>> I might be wrong, but I don't think it's a cert issue. If you look >>> in the mgr log, do you see more information that just the >>> SignatureDoesNotMatch error? >>> >>> Could it be mismatching dashboard-rgw-api settings? Have you checked >>> these settings? >>> >>> ceph dashboard get-rgw-api-admin-resource >>> >>> ceph dashboard get-rgw-api-access-key >>> >>> ceph dashboard get-rgw-api-secret-key >>> >>> They should match with: >>> >>> radosgw-admin user info --uid dashboard | jq -r '.keys' >>> >>> But as I wrote, it might something else, I would expect the mgr log >>> to contain more details. >>> >>> Regards >>> Eugen >>> >>> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >>> >>>> Hi to all! >>>> >>>> After upgrading my cluster from squid to tentacle (now on 20.2.1) >>>> I'm getting the following error when I try to access any 'page' in >>>> the Object section of the Ceph Dashboard: >>>> >>>> Error connecting to Object Gateway: RGW REST API failed request >>>> with status code 403 >>>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx0000000b5bf42356' >>>> b'85174-006a0d95b0-308299-eros","HostId":"308299-zone-default"}') >>>> >>>> I'm pretty sure that in Squid it was working (I don't usually >>>> access the Dashboard, but I did after the upgrade to check if >>>> everything is ok). >>>> >>>> The error 'SignatureDoesNotMatch' leads me to believe that there is >>>> a problem somewhere with the SSL certificates. But I put >>>> RGW_API_SSL_VERIFY to false... >>>> >>>> I have a SSL certificate (issued by harica.gr) which has different >>>> SANs including the hostname of all the nodes where RGW daemon is >>>> running (plus a cluster hostname and a wildcard hostname). I used >>>> this certificate for the ingress service and for the rgw daemons. >>>> Everything is done with a spec file using ceph orchestrator. I'm >>>> using the fullchain for the certificates and the the rgw/ingress >>>> services are running without any problem >>>> >>>> Or I'm completely wrong and the problem itself is with the access >>>> permissions (error code 403)? >>>> >>>> Any ideas? >>>> >>>> Thanks >>>> Iztok >>>> >>>> -- >>>> Iztok Gregori >>>> ICT Systems and Services >>>> Elettra - Sincrotrone Trieste S.C.p.A. >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: tentacle 20.2.2 QE validation status
by Vallari Agrawal 27 May '26

27 May '26

nvmeof approved cc: @Aviv Caro <acaro(a)redhat.com> On Tue, May 26, 2026 at 10:44 PM Afreen via dev <dev(a)ceph.io> wrote: > Dashboard approved > > On Tue, May 26, 2026 at 10:00 PM Yuri Weinstein via ceph-users < > ceph-users(a)ceph.io> wrote: > >> Resending because of emails problems >> >> On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> >> wrote: >> >> > Details of this release are summarized here: >> > >> > https://tracker.ceph.com/issues/75878#note-6 >> > >> > Release Notes - TBD >> > LRC upgrade - TBD >> > >> > Seeking approvals/reviews for: >> > >> > smoke: PASSED >> > >> > rados: Radek, Laura, Afreen, Travis >> > rgw: Adam Emerson >> > fs: Venky >> > orch/ceph-volume: Adam King & Guillaume >> > rbd: Ilya >> > krbd: Ilya >> > nvmeof: Aviv, Vallari >> > >> > upgrade: Patrick >> > >> > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD >> > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD >> > >> > perf-basic: PASSED >> > >> > crimson-rados: N/A >> > >> > powercycle: PASSED >> > >> > TIA >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Casey Bodley 27 May '26

27 May '26

On Wed, May 27, 2026 at 6:36 AM Iztok Gregori via ceph-users <ceph-users(a)ceph.io> wrote: > > On 27/05/26 12:17, Eugen Block via ceph-users wrote: > > Hi, > > > > > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > > > >> On 27/05/26 09:36, Eugen Block via ceph-users wrote: > >>> The description in this tracker [0] looks identical, but it is > >>> supposed to be fixed. Do you have rgw_dns_name set? > >> > >> Yes, at certain point I configured client.rgw rgw_dns_name to the FQDN > >> belonging to the virtual IP of the ingress service. > > > > does it mean that it works now? > > No, still doesn't work. To try I've also remove it and then again set > it, but without luck. > > I also tried to call the API endpoint directly with curl using the > dashboard credentials and found that from 'older' versions of curl > (provided for example by Rocky 9) the command gives me the same error > ('SignatureDoesNotMatch'), but when I was using newer one (8.5.0 on > Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer my query. regarding curl versions, i had fixed its sigv4 bugs for curl 8.0.0 with https://github.com/curl/curl/pull/9995 but i believe the dashboard uses its own custom sigv2 implementation in src/pybind/mgr/dashboard/awsauth.py > > Could it be the problem? > > > > >> BTW, how you 'unset' a configuration variable, you just put a empty > >> string or you use the rm option? > >> > >> ceph config set client.rgw rgw_dns_name '' > >> or > >> ceph config rm client.rgw rgw_dns_name > >> ? > > > > That's the correct one (config rm). > > Thank you! I sometimes get lost in the documentation :-) > > Cheers > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Eugen Block 27 May '26

27 May '26

I see what you mean, in our Squid cluster the curl request from within the mgr container fails with the same error. So I should probably expect that we're gonna hit the same dashboard issue once we upgrade to Tentacle, I guess. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > On 27/05/26 12:47, Eugen Block via ceph-users wrote: >> I'm not sure about the curl versions. > > Curl was only a tool and it could be unrelated to the problem I'm > facing, but it was interesting that when I try to query > > # curl "https://rgw.domain.com/admin/metadata/user" --aws-sigv4 > "aws:amz:ZONEGROUP:s3" --user "DASHBOARDKEY:DASHBOARDSECRET" > > it worked fine with newer versions. But the older ones (like the one > shipped with the MGR container) failed with the > 'SignatureDoesNotMatch' error. When I tried with more verbosity I > see that the oldest version is "signing" only 2 headers > (SignedHeaders=host;x-amz-date) and the newer ones 3 > (SignedHeaders=host;x-amz-content-sha256;x-amz-date), but this could > be due to '--aws-sigv4' option in curl. > > >> Did you notice the response from > "C U" the day before yesterday? >> He or she commented: >> >>> fqdn on the dashboard needs working certificate validation. >> >>> "Squid did cert work inline → bad cert raised crypto.Error → >>> caught and rewrapped as ServerConfigException cleanly. Tentacle >>> does cert work via a >> subprocess; failures can now come from (a) the fork itself, (b) >> PyO3 init crashing in the child (the documented subinterpreter >> collision), or (c) >> genuine cert errors marshalled back as JSON. When (b) happens, >> the dashboard module's cert provisioning never completes, the >> module ends up >> half-initialized, and downstream dashboard→RGW calls fail with >> SignatureDoesNotMatch because the dashboard is operating without >> properly loaded >> credentials/sigv4 plumbing." > > I did some tests using openssl s_client from the mgr container to > verify the certificates and I had no problems. Clearly I wasn't > using python, but unless is not using the system paths for > certificates, it shouldn't matter. > > Iztok > >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> >>> On 27/05/26 12:17, Eugen Block via ceph-users wrote: >>>> Hi, >>>> >>>> >>>> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >>>> >>>>> On 27/05/26 09:36, Eugen Block via ceph-users wrote: >>>>>> The description in this tracker [0] looks identical, but it is >>>>>> supposed to be fixed. Do you have rgw_dns_name set? >>>>> >>>>> Yes, at certain point I configured client.rgw rgw_dns_name to >>>>> the FQDN belonging to the virtual IP of the ingress service. >>>> >>>> does it mean that it works now? >>> >>> No, still doesn't work. To try I've also remove it and then again >>> set it, but without luck. >>> >>> I also tried to call the API endpoint directly with curl using the >>> dashboard credentials and found that from 'older' versions of curl >>> (provided for example by Rocky 9) the command gives me the same >>> error ('SignatureDoesNotMatch'), but when I was using newer one >>> (8.5.0 on Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer >>> my query. >>> >>> Could it be the problem? >>> >>>> >>>>> BTW, how you 'unset' a configuration variable, you just put a >>>>> empty string or you use the rm option? >>>>> >>>>> ceph config set client.rgw rgw_dns_name '' >>>>> or >>>>> ceph config rm client.rgw rgw_dns_name >>>>> ? >>>> >>>> That's the correct one (config rm). >>> >>> Thank you! I sometimes get lost in the documentation :-) >>> >>> Cheers >>> Iztok >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 27 May '26

27 May '26

On 27/05/26 12:47, Eugen Block via ceph-users wrote: > I'm not sure about the curl versions. Curl was only a tool and it could be unrelated to the problem I'm facing, but it was interesting that when I try to query # curl "https://rgw.domain.com/admin/metadata/user" --aws-sigv4 "aws:amz:ZONEGROUP:s3" --user "DASHBOARDKEY:DASHBOARDSECRET" it worked fine with newer versions. But the older ones (like the one shipped with the MGR container) failed with the 'SignatureDoesNotMatch' error. When I tried with more verbosity I see that the oldest version is "signing" only 2 headers (SignedHeaders=host;x-amz-date) and the newer ones 3 (SignedHeaders=host;x-amz-content-sha256;x-amz-date), but this could be due to '--aws-sigv4' option in curl. > Did you notice the response from > "C U" the day before yesterday? He or she commented: > >> fqdn on the dashboard needs working certificate validation. > >> "Squid did cert work inline → bad cert raised crypto.Error → caught >> and rewrapped as ServerConfigException cleanly. Tentacle does cert >> work via a > subprocess; failures can now come from (a) the fork itself, (b) PyO3 > init crashing in the child (the documented subinterpreter collision), or > (c) > genuine cert errors marshalled back as JSON. When (b) happens, the > dashboard module's cert provisioning never completes, the module ends up > half-initialized, and downstream dashboard→RGW calls fail with > SignatureDoesNotMatch because the dashboard is operating without > properly loaded > credentials/sigv4 plumbing." I did some tests using openssl s_client from the mgr container to verify the certificates and I had no problems. Clearly I wasn't using python, but unless is not using the system paths for certificates, it shouldn't matter. Iztok > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > >> On 27/05/26 12:17, Eugen Block via ceph-users wrote: >>> Hi, >>> >>> >>> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >>> >>>> On 27/05/26 09:36, Eugen Block via ceph-users wrote: >>>>> The description in this tracker [0] looks identical, but it is >>>>> supposed to be fixed. Do you have rgw_dns_name set? >>>> >>>> Yes, at certain point I configured client.rgw rgw_dns_name to the >>>> FQDN belonging to the virtual IP of the ingress service. >>> >>> does it mean that it works now? >> >> No, still doesn't work. To try I've also remove it and then again set >> it, but without luck. >> >> I also tried to call the API endpoint directly with curl using the >> dashboard credentials and found that from 'older' versions of curl >> (provided for example by Rocky 9) the command gives me the same error >> ('SignatureDoesNotMatch'), but when I was using newer one (8.5.0 on >> Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer my query. >> >> Could it be the problem? >> >>> >>>> BTW, how you 'unset' a configuration variable, you just put a empty >>>> string or you use the rm option? >>>> >>>> ceph config set client.rgw rgw_dns_name '' >>>> or >>>> ceph config rm client.rgw rgw_dns_name >>>> ? >>> >>> That's the correct one (config rm). >> >> Thank you! I sometimes get lost in the documentation :-) >> >> Cheers >> Iztok >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0