Tentacle: Grafana could not load SSL certificate after upgrade to 20.2.5
Hi, I just upgraded one of my test clusters to 20.2.5, the upgrade went well except for Grafana. I have my own certificate and key which worked before the upgrade (from 19.2.7). Now Grafana fails with:
could not load SSL certificate: tls: failed to find any PEM data in certificate input
The certmgr lists the existing cert: host:~ # ceph orch certmgr cert ls grafana_cert scope: host certificates nautilus subject countryName: DE ... commonName: host.domain ... validity remaining_days: 13 But the files in /var/lib/ceph/{FSID}/grafana.host/etc/grafana/certs/ only contain a comment: host:~ # cat /var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/grafana.nautilus/etc/grafana/certs/cert_* # generated by cephadm # generated by cephadm These config-keys are stored: "mgr/cephadm/cert_store.cert.grafana_cert", "mgr/cephadm/cert_store.cert.grafana_key", "mgr/cephadm/cert_store.key.grafana_key", "mgr/cephadm/spec.grafana", I know I could just replace the cert and key in that directory and restart grafana. But if certmgr already knows there's a valid cert, why does it (re-)generate an empty cert + key? I'm not sure if I'm missing something or of this is by design. I couldn't find anything in the release notes for 20.2.5 or in the Tentacle docs that would explain this. Thanks! Eugen
Hi Eugen, There are some bugs around the Grafana certificate(s) upgrade that PR #73234 is meant to fix: https://github.com/ceph/ceph/pull/73234 For now, I recommend re-applying them manually (using certmgr). Best, Redo. On Tue, Nov 3, 2026 at 11:14 AM Eugen Block <ceph-users@ceph.io> wrote:
Hi,
I just upgraded one of my test clusters to 20.2.5, the upgrade went well except for Grafana. I have my own certificate and key which worked before the upgrade (from 19.2.7).
Now Grafana fails with:
could not load SSL certificate: tls: failed to find any PEM data in certificate input
The certmgr lists the existing cert:
host:~ # ceph orch certmgr cert ls grafana_cert scope: host certificates nautilus subject countryName: DE ... commonName: host.domain ... validity remaining_days: 13
But the files in /var/lib/ceph/{FSID}/grafana.host/etc/grafana/certs/ only contain a comment:
host:~ # cat
/var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/grafana.nautilus/etc/grafana/certs/cert_* # generated by cephadm # generated by cephadm
These config-keys are stored:
"mgr/cephadm/cert_store.cert.grafana_cert", "mgr/cephadm/cert_store.cert.grafana_key", "mgr/cephadm/cert_store.key.grafana_key", "mgr/cephadm/spec.grafana",
I know I could just replace the cert and key in that directory and restart grafana. But if certmgr already knows there's a valid cert, why does it (re-)generate an empty cert + key? I'm not sure if I'm missing something or of this is by design. I couldn't find anything in the release notes for 20.2.5 or in the Tentacle docs that would explain this.
Thanks! Eugen
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Migration (the part that affects the upgrade) is being handled/fixed on a different PR: https://github.com/ceph/ceph/pull/73366 On Tue, Nov 3, 2026 at 11:42 AM Redouane Kachach <rkachach@redhat.com> wrote:
Hi Eugen,
There are some bugs around the Grafana certificate(s) upgrade that PR #73234 is meant to fix:
https://github.com/ceph/ceph/pull/73234
For now, I recommend re-applying them manually (using certmgr).
Best, Redo.
On Tue, Nov 3, 2026 at 11:14 AM Eugen Block <ceph-users@ceph.io> wrote:
Hi,
I just upgraded one of my test clusters to 20.2.5, the upgrade went well except for Grafana. I have my own certificate and key which worked before the upgrade (from 19.2.7).
Now Grafana fails with:
could not load SSL certificate: tls: failed to find any PEM data in certificate input
The certmgr lists the existing cert:
host:~ # ceph orch certmgr cert ls grafana_cert scope: host certificates nautilus subject countryName: DE ... commonName: host.domain ... validity remaining_days: 13
But the files in /var/lib/ceph/{FSID}/grafana.host/etc/grafana/certs/ only contain a comment:
host:~ # cat
/var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/grafana.nautilus/etc/grafana/certs/cert_* # generated by cephadm # generated by cephadm
These config-keys are stored:
"mgr/cephadm/cert_store.cert.grafana_cert", "mgr/cephadm/cert_store.cert.grafana_key", "mgr/cephadm/cert_store.key.grafana_key", "mgr/cephadm/spec.grafana",
I know I could just replace the cert and key in that directory and restart grafana. But if certmgr already knows there's a valid cert, why does it (re-)generate an empty cert + key? I'm not sure if I'm missing something or of this is by design. I couldn't find anything in the release notes for 20.2.5 or in the Tentacle docs that would explain this.
Thanks! Eugen
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, I appreciate the quick response. certmgr seemed to reject a certificate that will expire in 13 days: ceph orch certmgr cert set grafana --hostname nautilus -i grafana-combined-cert-key.pem Error EINVAL: Certificate 'grafana (host)' (user-made) is about to expire (remaining days: 13) I just copied the cert and key file to the container location and restarted grafana, it works now. I will experiment more with certmgr when I have more time. ;-) Thanks again! Eugen Zitat von Redouane Kachach <rkachach@redhat.com>:
Migration (the part that affects the upgrade) is being handled/fixed on a different PR:
https://github.com/ceph/ceph/pull/73366
On Tue, Nov 3, 2026 at 11:42 AM Redouane Kachach <rkachach@redhat.com> wrote:
Hi Eugen,
There are some bugs around the Grafana certificate(s) upgrade that PR #73234 is meant to fix:
https://github.com/ceph/ceph/pull/73234
For now, I recommend re-applying them manually (using certmgr).
Best, Redo.
On Tue, Nov 3, 2026 at 11:14 AM Eugen Block <ceph-users@ceph.io> wrote:
Hi,
I just upgraded one of my test clusters to 20.2.5, the upgrade went well except for Grafana. I have my own certificate and key which worked before the upgrade (from 19.2.7).
Now Grafana fails with:
could not load SSL certificate: tls: failed to find any PEM data in certificate input
The certmgr lists the existing cert:
host:~ # ceph orch certmgr cert ls grafana_cert scope: host certificates nautilus subject countryName: DE ... commonName: host.domain ... validity remaining_days: 13
But the files in /var/lib/ceph/{FSID}/grafana.host/etc/grafana/certs/ only contain a comment:
host:~ # cat
/var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/grafana.nautilus/etc/grafana/certs/cert_* # generated by cephadm # generated by cephadm
These config-keys are stored:
"mgr/cephadm/cert_store.cert.grafana_cert", "mgr/cephadm/cert_store.cert.grafana_key", "mgr/cephadm/cert_store.key.grafana_key", "mgr/cephadm/spec.grafana",
I know I could just replace the cert and key in that directory and restart grafana. But if certmgr already knows there's a valid cert, why does it (re-)generate an empty cert + key? I'm not sure if I'm missing something or of this is by design. I couldn't find anything in the release notes for 20.2.5 or in the Tentacle docs that would explain this.
Thanks! Eugen
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Welcome Eugen, That's expected behaviour, and it's a safety check to prevent users from setting ver short-duration (or about-to-expire) certificates. It's controlled by the cephadm config option: Option( 'certificate_renewal_threshold_days', type='int', default=30, desc='Specifies the lead time in days to initiate certificate renewal before expiration.', min=10, max=90 ), Best, Redo. On Tue, Nov 3, 2026 at 12:19 PM Eugen Block <eblock@nde.ag> wrote:
Thanks, I appreciate the quick response. certmgr seemed to reject a certificate that will expire in 13 days:
ceph orch certmgr cert set grafana --hostname nautilus -i grafana-combined-cert-key.pem Error EINVAL: Certificate 'grafana (host)' (user-made) is about to expire (remaining days: 13)
I just copied the cert and key file to the container location and restarted grafana, it works now. I will experiment more with certmgr when I have more time. ;-)
Thanks again! Eugen
Zitat von Redouane Kachach <rkachach@redhat.com>:
Migration (the part that affects the upgrade) is being handled/fixed on a different PR:
https://github.com/ceph/ceph/pull/73366
On Tue, Nov 3, 2026 at 11:42 AM Redouane Kachach <rkachach@redhat.com> wrote:
Hi Eugen,
There are some bugs around the Grafana certificate(s) upgrade that PR #73234 is meant to fix:
https://github.com/ceph/ceph/pull/73234
For now, I recommend re-applying them manually (using certmgr).
Best, Redo.
On Tue, Nov 3, 2026 at 11:14 AM Eugen Block <ceph-users@ceph.io> wrote:
Hi,
I just upgraded one of my test clusters to 20.2.5, the upgrade went well except for Grafana. I have my own certificate and key which worked before the upgrade (from 19.2.7).
Now Grafana fails with:
could not load SSL certificate: tls: failed to find any PEM data in certificate input
The certmgr lists the existing cert:
host:~ # ceph orch certmgr cert ls grafana_cert scope: host certificates nautilus subject countryName: DE ... commonName: host.domain ... validity remaining_days: 13
But the files in /var/lib/ceph/{FSID}/grafana.host/etc/grafana/certs/ only contain a comment:
host:~ # cat
/var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/grafana.nautilus/etc/grafana/certs/cert_*
# generated by cephadm # generated by cephadm
These config-keys are stored:
"mgr/cephadm/cert_store.cert.grafana_cert", "mgr/cephadm/cert_store.cert.grafana_key", "mgr/cephadm/cert_store.key.grafana_key", "mgr/cephadm/spec.grafana",
I know I could just replace the cert and key in that directory and restart grafana. But if certmgr already knows there's a valid cert, why does it (re-)generate an empty cert + key? I'm not sure if I'm missing something or of this is by design. I couldn't find anything in the release notes for 20.2.5 or in the Tentacle docs that would explain this.
Thanks! Eugen
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Redouane Kachach