Hi All, So we've been using the Ceph (v18.2.4) Dashboard with internally generated TLS Certificates (via our Step-CA CA), one for each of our three Ceph Manager Nodes. Everything was working AOK. The TLS Certificates came up for renewal, which they were successfully renewed. Accordingly, the old Certificates & Keys were overwritten by the new ones and the commands `ceph mgr module disable/enabled dashboard` (respectively) were run. HOWEVER, the Ceph Dashboard stopped working / wouldn't use the renewed Certificates; as per the logs the Dashboard was still using the old Certificates and is now complaining that they have expired, and therefore the Dashboard won't run (unless I disable SSL via the CLI). I've been through the documentation and I can't work out what I've done wrong; according to this page (https://docs.ceph.com/en/latest/mgr/dashboard/#dashboard-ssl-tls-support) - in particular the Blue-Box Note - there's nothing else that needs to be done - EXCEPT that I've done all of that and the logs still tell me the Certificate(s) has expired - ie I can't get the Dashboard to recognise the new Certificates. Any help greatly appreciated. Thanks in advance Dulux-Oz
IIRC, the certificate and key are only read from their files when the commands to specify the file are executed. At that point they are stored somewhere else. Try executing the two commands (one for key, one for cert) again, then restart (disable/enable might be enough, I can't remember). Regards, Chris On 19/12/2024 07:04, duluxoz wrote:
Hi All,
So we've been using the Ceph (v18.2.4) Dashboard with internally generated TLS Certificates (via our Step-CA CA), one for each of our three Ceph Manager Nodes.
Everything was working AOK.
The TLS Certificates came up for renewal, which they were successfully renewed. Accordingly, the old Certificates & Keys were overwritten by the new ones and the commands `ceph mgr module disable/enabled dashboard` (respectively) were run.
HOWEVER, the Ceph Dashboard stopped working / wouldn't use the renewed Certificates; as per the logs the Dashboard was still using the old Certificates and is now complaining that they have expired, and therefore the Dashboard won't run (unless I disable SSL via the CLI).
I've been through the documentation and I can't work out what I've done wrong; according to this page (https://docs.ceph.com/en/latest/mgr/dashboard/#dashboard-ssl-tls-support) - in particular the Blue-Box Note - there's nothing else that needs to be done - EXCEPT that I've done all of that and the logs still tell me the Certificate(s) has expired - ie I can't get the Dashboard to recognise the new Certificates.
Any help greatly appreciated.
Thanks in advance
Dulux-Oz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Chris, Yeah, I did that (sorry I didn't mention that in the original post) - no joy. :-( Any other suggestions? :-) On 19/12/24 21:14, Chris Palmer wrote:
IIRC, the certificate and key are only read from their files when the commands to specify the file are executed. At that point they are stored somewhere else. Try executing the two commands (one for key, one for cert) again, then restart (disable/enable might be enough, I can't remember).
Regards, Chris
On 19/12/2024 07:04, duluxoz wrote:
Hi All,
So we've been using the Ceph (v18.2.4) Dashboard with internally generated TLS Certificates (via our Step-CA CA), one for each of our three Ceph Manager Nodes.
Everything was working AOK.
The TLS Certificates came up for renewal, which they were successfully renewed. Accordingly, the old Certificates & Keys were overwritten by the new ones and the commands `ceph mgr module disable/enabled dashboard` (respectively) were run.
HOWEVER, the Ceph Dashboard stopped working / wouldn't use the renewed Certificates; as per the logs the Dashboard was still using the old Certificates and is now complaining that they have expired, and therefore the Dashboard won't run (unless I disable SSL via the CLI).
I've been through the documentation and I can't work out what I've done wrong; according to this page (https://docs.ceph.com/en/latest/mgr/dashboard/#dashboard-ssl-tls-support) - in particular the Blue-Box Note - there's nothing else that needs to be done - EXCEPT that I've done all of that and the logs still tell me the Certificate(s) has expired - ie I can't get the Dashboard to recognise the new Certificates.
Any help greatly appreciated.
Thanks in advance
Dulux-Oz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I just replaced an expired cert in a 18.2.2 test cluster: ceph config-key set mgr/dashboard/crt -i /tmp/newcert.pem ceph config-key set mgr/dashboard/key -i /tmp/newkey.pem ceph mgr fail And that was it. In our prod Pacific cluster we use per server certificates (mgr/dashboard/{host1}/crt, mgr/dashboard/{host2}/crt and so on). Maybe you have some remainders in the config-keys? I would check all of the dashboard/cert related and remove any expired certs/keys. Zitat von duluxoz <duluxoz@gmail.com>:
Hi Chris,
Yeah, I did that (sorry I didn't mention that in the original post) - no joy. :-(
Any other suggestions? :-)
On 19/12/24 21:14, Chris Palmer wrote:
IIRC, the certificate and key are only read from their files when the commands to specify the file are executed. At that point they are stored somewhere else. Try executing the two commands (one for key, one for cert) again, then restart (disable/enable might be enough, I can't remember).
Regards, Chris
On 19/12/2024 07:04, duluxoz wrote:
Hi All,
So we've been using the Ceph (v18.2.4) Dashboard with internally generated TLS Certificates (via our Step-CA CA), one for each of our three Ceph Manager Nodes.
Everything was working AOK.
The TLS Certificates came up for renewal, which they were successfully renewed. Accordingly, the old Certificates & Keys were overwritten by the new ones and the commands `ceph mgr module disable/enabled dashboard` (respectively) were run.
HOWEVER, the Ceph Dashboard stopped working / wouldn't use the renewed Certificates; as per the logs the Dashboard was still using the old Certificates and is now complaining that they have expired, and therefore the Dashboard won't run (unless I disable SSL via the CLI).
I've been through the documentation and I can't work out what I've done wrong; according to this page (https://docs.ceph.com/en/latest/mgr/dashboard/#dashboard-ssl-tls-support) - in particular the Blue-Box Note - there's nothing else that needs to be done - EXCEPT that I've done all of that and the logs still tell me the Certificate(s) has expired - ie I can't get the Dashboard to recognise the new Certificates.
Any help greatly appreciated.
Thanks in advance
Dulux-Oz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi All, Ongoing saga - Part II: So there's been some progress(?). I'm now getting `Invalid certificate: [('PEM routines', '', 'no start line')]`errors in the log. The thing is, the cert seems AOK to me. Here it is: -----BEGIN CERTIFICATE----- MIIDQjCCAuegAwIBAgIRAJWbTEH2A88mv25vFhnN/kcwCgYIKoZIzj0EAwIwbDEL MAkGA1UEBhMCQVUxHzAdBgNVBAoTFlBFUkVHUklORSBJLlQuIFB0eSBMdGQxCzAJ BgNVBAsTAkhRMS8wLQYDVQQDEyZQRVJFR1JJTkUgSS5ULiBQdHkgTHRkIEludGVy bWVkaWF0ZSBDQTAeFw0yNDEyMjMwNjM5MDRaFw0yNTAzMjMwNjQwMDRaMCExHzAd BgNVBAMTFmNlcGhkYXNoLXNhbjAxLm1qYi5wcmkwggEiMA0GCSqGSIb3DQEBAQUA A4IBDwAwggEKAoIBAQDwCzhzMAr6Jj7ep2uYzBW0GLfvRyWMlnt/hOQlSW8gya/t GzPaZETiLomZL6bnZSQIlkqUkVrFOe/aB+tjAGoq9IEi5ih33/clN7x+fo8A8oop F7B2xynYT54R3PH7matmtTMyHGIg/DxTtKoHQ+n93/0I+R3s6cW+Uu6ImFUQicU1 gPKZ2YP/bgLBEWfWZx6N5Q9JOxrqNgJnXZ8HVEam+LCTukj+BCCWn73qWiPkIP+3 Gm/+KkxFMQH0vNrMaN03FPAp8N3dQ3AYPotVeGkq1PcG2XfyZCAi0eEGkHz6C6k6 +jF4TWLZZNifBb5tx/A5lkt1XmJy/z0y7DDiLkaFAgMBAAGjgekwgeYwDgYDVR0P AQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4E FgQUtnjiVkvVp1oUqAg6pj9lgFAXuzIwHwYDVR0jBBgwFoAUwvqfH2T9UZFQ602f 20RULBTTd+swGAYDVR0RBBEwD4INc2FuMDEubWpiLnByaTBbBgwrBgEEAYKkZMYo QAEESzBJAgEBBBdtamJfcHJpX3Rsc19wcm92aXNpb25lcgQrTk1XWXhNaDEwWDNC UjR6QTVIeFBJTGtXdVJ2ZmRSbWxvV19ZdkpKbEhaNDAKBggqhkjOPQQDAgNJADBG AiEAsLY5zTGh53P5zeTfsxR1TIRD8hZaDRdyZ5ZpIVIWfpoCIQCbFLdjrp8fjQmT zCJQeZuFKM2d9vi7I5H+obRKGMrAnA== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIICbjCCAhWgAwIBAgIQZWkpXMMZwJ/TL9ZyqHZlfzAKBggqhkjOPQQDAjBkMQsw CQYDVQQGEwJBVTEfMB0GA1UEChMWUEVSRUdSSU5FIEkuVC4gUHR5IEx0ZDELMAkG A1UECxMCSFExJzAlBgNVBAMTHlBFUkVHUklORSBJLlQuIFB0eSBMdGQgUm9vdCBD QTAeFw0yNDEyMTQwNDAxNTVaFw0zNDEyMTQxNjAxNDFaMGwxCzAJBgNVBAYTAkFV MR8wHQYDVQQKExZQRVJFR1JJTkUgSS5ULiBQdHkgTHRkMQswCQYDVQQLEwJIUTEv MC0GA1UEAxMmUEVSRUdSSU5FIEkuVC4gUHR5IEx0ZCBJbnRlcm1lZGlhdGUgQ0Ew WTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQHeuLJVqA6gf8bgWFIewaNfb5E0yRf 3pSw9zxQR5NHLwFkVNefTP8UjzgdZ56b/XQqM+0I1kitDoOe4bUqGOEto4GgMIGd MA4GA1UdDwEB/wQEAwIBBjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBTC +p8fZP1RkVDrTZ/bRFQsFNN36zAfBgNVHSMEGDAWgBQDf/dhCop69HvQSdVeoxXv SBPv+zA3BgNVHR8EMDAuMCygKqAohiZodHRwczovL3d3dy5wZXJlZ3JpbmVpdC5u ZXQvY3JsL2NhLmNybDAKBggqhkjOPQQDAgNHADBEAiB7plAbFadeZI1uyM1tdksF FKNDe1+RvVN9harBx/CL6gIgacoqe9tSadLiYvYxMFQqeZ+8G92kr5an8iS9wLlb QMs= -----END CERTIFICATE----- Anyone see anything wrong? (FTR: the second block is the Intermediate-CA) :-) Cheers Dulux-Oz On 20/12/24 19:46, Eugen Block wrote:
I just replaced an expired cert in a 18.2.2 test cluster:
ceph config-key set mgr/dashboard/crt -i /tmp/newcert.pem ceph config-key set mgr/dashboard/key -i /tmp/newkey.pem ceph mgr fail
And that was it. In our prod Pacific cluster we use per server certificates (mgr/dashboard/{host1}/crt, mgr/dashboard/{host2}/crt and so on).
Maybe you have some remainders in the config-keys? I would check all of the dashboard/cert related and remove any expired certs/keys.
Zitat von duluxoz <duluxoz@gmail.com>:
Hi Chris,
Yeah, I did that (sorry I didn't mention that in the original post) - no joy. :-(
Any other suggestions? :-)
On 19/12/24 21:14, Chris Palmer wrote:
IIRC, the certificate and key are only read from their files when the commands to specify the file are executed. At that point they are stored somewhere else. Try executing the two commands (one for key, one for cert) again, then restart (disable/enable might be enough, I can't remember).
Regards, Chris
On 19/12/2024 07:04, duluxoz wrote:
Hi All,
So we've been using the Ceph (v18.2.4) Dashboard with internally generated TLS Certificates (via our Step-CA CA), one for each of our three Ceph Manager Nodes.
Everything was working AOK.
The TLS Certificates came up for renewal, which they were successfully renewed. Accordingly, the old Certificates & Keys were overwritten by the new ones and the commands `ceph mgr module disable/enabled dashboard` (respectively) were run.
HOWEVER, the Ceph Dashboard stopped working / wouldn't use the renewed Certificates; as per the logs the Dashboard was still using the old Certificates and is now complaining that they have expired, and therefore the Dashboard won't run (unless I disable SSL via the CLI).
I've been through the documentation and I can't work out what I've done wrong; according to this page (https://docs.ceph.com/en/latest/mgr/dashboard/#dashboard-ssl-tls-support) - in particular the Blue-Box Note - there's nothing else that needs to be done - EXCEPT that I've done all of that and the logs still tell me the Certificate(s) has expired - ie I can't get the Dashboard to recognise the new Certificates.
Any help greatly appreciated.
Thanks in advance
Dulux-Oz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Chris Palmer
-
duluxoz
-
Eugen Block