Hi Flemming, certmgr at the moment doesn't support fullchain PEM certificates: https://tracker.ceph.com/issues/75710 Can you plz paste the spec you are currently using that is leading to the issue). Right now the format of cert/key has to be something like: service_type: rgw service_id: myrgw spec: ssl: true ssl_key: | -----BEGIN PRIVATE KEY----- MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQCM58IqqXGTCJof ..... QsJ69XOQV9Os37zG2/hELQ1ycBs2Jw== -----END PRIVATE KEY----- ssl_cert: | -----BEGIN CERTIFICATE----- MIIF4TCCA8mgAwIBAgIUI1F5+FeKTeF5J+xYLdkEDP9J0OMwDQYJKoZIhvcNAQEL .... fhTw47C8Dry+2bOaYQPe6qvoFQ+x -----END CERTIFICATE----- Best, Redo. On Fri, Apr 24, 2026 at 2:52 PM Flemming Frandsen via ceph-users < ceph-users@ceph.io> wrote:

I have a tentacle 20.2.1 cluster managed by cephadm, where I have configured a certificate for the rgw using something like:

service_type: ingress service_id: rgw.default service_name: ingress.rgw.default placement: count: 3 label: rgw spec: backend_service: rgw.default first_virtual_router_id: 50 frontend_port: 443 monitor_port: 8080 virtual_ip: 10.0.40.3/24 networks: - 10.0.40.0/24 ssl_cert: | -----BEGIN CERTIFICATE----- *snip* -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- *snip* -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- *snip* -----END RSA PRIVATE KEY-----

This initially worked perfectly, but I have just rotated the certificate by generating the yaml with a fresh private key and associated certificate then installing it with: ceph orch apply -i new.yaml

Clients accessing the rgw ingress get the correct certificate and work perfectly, both before and after the key rotation.

I have a script that does this for me daily, and until the certificate was rotated, everything was fine, but then I got this error:

# ceph orch certmgr cert check

Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')])

After messing with the certificate a bunch I now get a completely different, but also worrying error:

# ceph orch certmgr cert check - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid private key: [('DECODER routines', '', 'unsupported')])

The documentation offers no help: https://docs.ceph.com/en/tentacle/cephadm/certmgr/

The error seems to come from this code:

https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/in...

So the error is an amputated exception from somewhere in openssl but the actual exception is eaten by the code, so there is no way for me to debug the situation:

https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/in...

I have tried installing the old certificate and key, that only leads the the Invalid private key: [('DECODER routines', '', 'unsupported')] error.

I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does nothing.

I have even tried copying the key and cert pem files into the mgr container and verified them using the openssl installed in the container.

The certificates are generated by cert-manager, running in k8s and a cronjob that periodically pushes the new key/cert to the ceph cluster via ssh.

Has anyone seen these types of errors before?

-- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io