Hi Redo Yes, that error message was there.... And yes, uploading a second copy of the cert/key under the fqdn has worked. I understand now about the CLI arguments - will just avoid that syntax until it is fixed. Thank you so much for your help. 🙂 Regards, Chris On 29/10/2026 13:23, Redouane Kachach wrote:
Hi Chris,
Thanks for trying it and for the detailed output. Both problems are understood now.
2. The --cert/--key failure I think is due to how the CLI parses the args. The ceph CLI treats "--key <value>" as the client's cephx secret before the command ever reaches the mgr, so your PEM key was used to authenticate to the monitors and they rejected it. It isn't related to auth_allowed_ciphers. Passing a combined PEM with -i, as you did in step 3, is the right way. We'll fix the option name so this can't happen. (This is a BUG that must be fixed on the cephadm certmgr side)
5. This is a bug in 21.2.0. Cephadm looks up the per-host grafana certificate under the host's FQDN, while "cert-key set --hostname c1" stores it under the host name. If c1's FQDN is something else (e.g. c1.example.com <http://c1.example.com>), the lookup misses, and 21.2.0 then deploys grafana with empty certificate files, which is the "failed to find any PEM data" error you see.
Could you plz check the mgr logs to see if contain a line like:
Failed to get cert/key grafana_ssl_cert for service grafana host: <fqdn> from the certmgr store.
The <fqdn> in that line is the name cephadm is looking for. You can also get it by running "cephadm gather-facts" on c1 and reading the "fqdn" value.
Workaround: register the same certificate under that FQDN as well, then redeploy grafana (redeploy rather than reconfig, since the daemon is in error state):
ceph orch certmgr cert-key set grafana --hostname <fqdn> -i x.y.crt-key ceph orch redeploy grafana ceph orch certmgr cert check
Your grafana spec (certificate_source: reference) is correct as it is, so no need to re-apply it. If grafana runs on more hosts, do the same for each one, using each host's FQDN.
Plz give it a try and let me know,
Best, Redo.
On Thu, Oct 29, 2026 at 1:32 PM Chris Palmer <chris@chrispalmer.uk> wrote:
Hi Redo
Thanks for the useful info. I've made some progress, but not there yet...
1. The health warning was indeed stale and re-executing the check cleared it. 2. Executing this fails (we have already done "ceph mon set auth_allowed_ciphers aes256k" before the upgrade, and immediately after as the upgrade reset it). This looks like another bug:
# ceph orch certmgr cert-key set grafana --hostname c1 --cert "$(cat x.y.crt)" --key "$(cat x.y.key)" 2026-10-29T11:49:03.802+0000 7f797c8ac6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:03.802+0000 7f797d0ad6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:06.801+0000 7f797d0ad6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:09.801+0000 7f797d0ad6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:09.801+0000 7f797c8ac6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:09.801+0000 7f797eb376c0 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 95] error connecting to the cluster
3. But loading it from a combined file works:
# ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key Certificate/key pair set correctly
# ceph orch certmgr cert ls --filter-by "name=grafana_ssl_cert" grafana_ssl_cert scope: host certificates c1 subject countryName: xx stateOrProvinceName: xxxxx organizationName: xxxxx commonName: x.y contains_chain: False validity remaining_days: 635
# ceph orch certmgr key ls grafana_ssl_key scope: host keys c1 key_type: RSA key_size: 4096
4. I exported the grafana spec, and changed the certificate source, and issued the reconfig ok.
# cat grafana.yaml service_type: grafana service_name: grafana placement: hosts: - c1 spec: anonymous_access: true certificate_source: reference port: 3000 protocol: https ssl: true
5. But now grafana fails to start:
# ceph health detail HEALTH_WARN 1 failed cephadm daemon(s) [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon grafana.c1 on c1 is in error state
And from journalctl:
[starting module *api.HTTPServer: invalid service state: Failed, expected: Running, failure: could not load SSL certificate: tls: failed to find any PEM data in certificate input]
I'm really not sure where to go from here!
Regards, Chris
On 28/10/2026 20:32, Redouane Kachach wrote:
Hi Chris,
Thanks for the detailed report. I’m not sure yet of the exact cause, but I suspect it’s related to a new spec field introduced in this release. In umbrella, the certificate a service uses is determined by the certificate_source field in its spec, not only by what’s in the certmgr store. If the grafana spec doesn’t set a source, it defaults to cephadm-signed, and cephadm then ignores grafana_ssl_cert/grafana_ssl_key entirely. That would explain why you kept getting the self-signed certificate even after cert-key set succeeded.
The disappearing key smells like a bug that I need to reproduce to investigate what's going on. The HEALTH_ERR you still saw afterwards is most likely stale. The certificate check runs at mgr startup and then once a day by default, and a reconfig doesn't re-trigger it. You can force it by running: ceph orch certmgr cert check
The upgrade also didn't carry over your tentacle-era grafana_cert/grafana_key entries to the new names or switch the spec to use them, so it silently fell back to a cephadm-signed cert. That's a regression on our side, and I'll open a tracker for it, along with the key set bug. So as for your questions:
Q1: How to set a grafana cert/key on umbrella
1. Store the cert/key pair for the host. Either a combined PEM:
ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key
or separate files:
ceph orch certmgr cert-key set grafana --hostname c1 --cert "$(cat x.y.crt)" --key "$(cat x.y.key)"
2. Tell the grafana service to use the stored cert:
ceph orch ls grafana --export > grafana.yaml
and add these under "spec:":
ssl: true certificate_source: reference
then:
ceph orch apply -i grafana.yaml ceph orch reconfig grafana
3. Refresh the certificate health status:
ceph orch certmgr cert check
This should fix it
Q2: Grafana on more than one host
Grafana certificates are host-scoped, and that's fixed for the service type. You can't switch it to global. With certificate_source: reference you need to run the cert-key set once per grafana host, using the same file each time if it's the same cert:
ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key ceph orch certmgr cert-key set grafana --hostname c2 -i x.y.crt-key
I hope this helps to fix the issue.
Best, Redouane
On Wed, Oct 28, 2026 at 7:05 PM Chris Palmer <ceph-users@ceph.io> wrote:
I've just had a go at upgrading a cephadm-managed test cluster from 20.2.4 to 21.2.0. It went pretty well.
One minor thing (easily fixed): the upgrade re-enabled use of insecure keys (AUTH_INSECURE_KEYS_ALLOWED) so I had to put that back (ceph mon set auth_allowed_ciphers aes256k).
And a certificate problem that I can't see how to fix. I have a user-supplied grafana cert. (On this cluster grafana is only deployed on one node c1). On tentacle it was simply:
ceph orch certmgr cert set grafana_cert --hostname c1 -i x.ycrt ceph orch certmgr key set grafana_key --hostname c1 -i x.y.key ceph orch reconfig grafana
With umbrella the bindings have changed: grafana_cert is now grafana_ssl_cert and grafana_key is now grafana_ssl_key (according to ceph orch certmgr bindings ls) . So:
ceph orch certmgr cert set grafana_ssl_cert --hostname c1 -i x.ycrt ceph orch certmgr key set grafana_ssl_key --hostname c1 -i x.y.key ceph orch reconfig grafana
They get accepted, and I can see the key, but as soon as I issue the reconfig the key disappears (the cert remains). I then get grafana running with a cephadm-generated self-signed key, and the error:
root@c1:~# ceph health detail HEALTH_ERR Detected 1 cephadm certificate(s) issues: 1 invalid [ERR] CEPHADM_CERT_ERROR: Detected 1 cephadm certificate(s) issues: 1 invalid Certificate 'grafana_ssl_cert (c1)' (user-made) is not valid (error: missing key)
I also tried explicitly adding "--service_name grafana" to the sets but it made no difference.
I then tried using a combined cert/key file:
root@c1:~# ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key
This was accepted, and after the reconfig the key is still visible! However, I still get exactly the same health error, and it is still using the self-signed cert.
Q1: How can I set a grafana cert/key on umbrella?
Q2: If grafana is on more than one host do I need to install the same cert with each hostname? Is it intrinsically locked to be scope host, or can I set it with scope global?
Thanks, Chris
On 28/10/2026 13:59, Yuri Weinstein wrote: > Umbrella is the 21st stable release of Ceph. > > This is the first stable release of Ceph Umbrella. > We recommend that all users update to this release. > > Warnings > > - Umbrella container images are based on Rocky Linux 10 (el10), which > requires a CPU > supporting the x86-64-v3 microarchitecture level. Deployments on hosts > with older CPUs > will not be able to run the containerized daemons. > - Python 3.9 or newer is required on all hosts for cephadm support. > > Notable Changes > > RADOS / OSD / Erasure Coding > > - Optimized ("Fast") Erasure Coding: a substantial new EC I/O path. > - EC Direct Reads > - CLS support for EC reads > - EC OMAP support > - Add Fast EC CRC-checking for data and first parity > - Numerous correctness fixes to the new path (truncate+write planning, > PDW vs RMW > selection, snapshot scrub handling, peering/recovery corner cases) > Ships with health warnings for deprecated EC plugins/techniques and > for Blaum-Roth EC > profiles whose w+1 isn't prime; non-4k-aligned chunk sizes are now > rejected for > optimized EC. > - Security: CephX authentication hardening -- new AES256KRB5 cipher > (replacing weaker > legacy key types as the default), online key rotation for mon/OSD/MDS > daemons during > upgrades, and fixes for CVE-2025-30156 (AES-CBC misuse enabling > authentication bypass), > CVE-2026-39944, CVE-2026-50152, and CVE-2026-54330 > (https://github.com/ceph/ceph/pull/71172). > - Monitor: new optional RocksDB backup/restore mechanism > (mon_backup_interval, ceph tell > mon.* backup, offline restore via ceph-mon --restore-backup). > - New availability score feature for Monitor/OSD (tracking + health > warnings + config > toggle), off by default. > - Stretch clusters: tiebreaker mon can now be optional, added a max > bucket-weight-diff > threshold, a health warning for colocated monitors, and a restriction > on changing the > election strategy after stretch mode is enabled. > - mgr/DaemonServer: ok-to-upgrade now aggregates and globally sorts OSDs > across CRUSH child > buckets (fixing inaccurate per-chassis/rack ordering), and upgrades can > now be scoped to > a CRUSH bucket. > - pg-upmap-primary support and improved pg-upmap computing speed. > - PG autoscaler: default mon_target_pg_per_osd increased 100 -> 200 and > mon_max_pg_per_osd > increased 250 -> 500, letting the autoscaler compute more > parallelism-friendly pg_num > values and improving the balancer's ability to reach uniform OSD > utilization. Existing > pools may autoscale to more PGs after upgrading as a result; > pg_autoscale_during_upgrade > (https://github.com/ceph/ceph/pull/67694) paces that splitting so it > doesn't disrupt the > upgrade itself. Clusters with a smaller-than-4GiB effective > bluestore_osd_memory_target, > or SSDs split into more than two OSDs, may want to pin the old values > before upgrading > (https://github.com/ceph/ceph/pull/69452). > - BlueStore: BlueFS Spillover Cleaner evolution, faster allocation > recovery, static/runtime > object-fragmentation tracking, full-device discard on mkfs now default, > several > spanning-blob/onode-cache-stall/reallocation-corruption fixes, and > device-type-specific > allocation policy. > - mClock scheduler refactor. Default HDD OSD shard config changed as an > interim fix for > inconsistent client/recovery throughput with multiple shards: > osd_op_num_shards_hdd 5 -> > 1, osd_op_num_threads_per_shard_hdd 1 -> 5 ( > https://tracker.ceph.com/issues/66289). > - Crimson/Seastore: continued heavy investment -- erasure coding support > landed in Crimson > (including recovery), PG splitting, a new 2Q cache replacement > algorithm, broad coroutine > conversions, and a long tail of correctness/perf fixes across the > object-data, > LBA/omap-btree, and cleaner subsystems. > > RGW (RADOS Gateway) > > - Full object dedup -- tech preview. > - Bucket logging -- new feature: journal-mode logging, EC-pool support, > admin commands, and > new per-bucket lifecycle/logging performance monitoring that closes a > long-standing > observability gap around lifecycle processing. > - Cloud transition / cloud restore: restore from Glacier/tape, admin CLI > for restore > list/status, restore-event notifications, persistent restore state, > per-bucket target > options; lifecycle, cloud-restore, and cloud-transition processing > converted to > coroutines for a significant performance improvement. > - Encryption: added AES-256-GCM SSE support; CopyObject now supported for > encrypted > objects. > - IAM/Account: GetAccountSummary, account-wide PublicAccessBlock, Admin > REST APIs for > account quota, S3 Object Ownership (disable ACLs), legacy Keystone > admin-token > requirement removed, Keystone roles injected into IAM policy. > - STS/OIDC: GetCallerIdentity, JWT/JWKS signature validation, OIDC > multisite support. > - SigV4: fixed a regression (present in 19.2.6/20.2.4) that rejected > requests with an > unsigned Content-Type, breaking presigned PUT uploads; > x-amz-content-sha256 is no longer > required in SignedHeaders for S3 requests ( > https://github.com/ceph/ceph/pull/71363). > - Multisite: numerous sync-correctness fixes (full-sync infinite loop, > segfault during > startup, forwarded awsv4/UNSIGNED-PAYLOAD requests, bucket-sync > full-sync locking, > zonegroup rename handling). > - D4N (Redis-backed) cache: continued build-out (bucket check_empty, > connection > pooling/pipelining, dirty-entry handling, crash fixes). > - s3-tests vendored directly into the repo (src/test/rgw/s3-tests); QA > suites run them from > there instead of an external branch reference. > - Timestamp-based epochs for versioned objects; global CORS rule support; > rate-limit > response with Retry-After; standalone RGW build (without RadosStore) > via a new > config-store abstraction -- a significant architectural refactor of > rgw_zone/realm-watching code. > - Lua scripting: new postAuth hook, fresh VM per background script > execution, > package-management fixes. > > Dashboard > > - Large-scale UI migration to the IBM Carbon design system across most > pages (RGW, pools, > OSD flags, multisite, zone/zonegroup, services, upgrade page, and more). > - NVMe/TCP (NVMeoF) management significantly expanded: multi-step > subsystem/gateway/namespace creation wizards, TLS/mTLS, DHCHAP auth, > KMIP support, > consistency-group CRUD APIs, and many CLI/UI alignment fixes. > - SMB cluster management support added. > - RGW role management realigned with Carbon + API routing fixes; global > RGW tenant "Roles" > tab removed in favor of per-account roles. > - CephFS mirroring wizard/entity management UI. > - OAuth2 SSO support (config UI + backend prerequisites). > - Angular upgraded to v19. > > cephadm / Orchestrator > > - Extended certificate management: cephadm now provides comprehensive > certmgr > integration across managed services including mgmt-gateway, monitoring > stack, NFS, > and RGW. Services support auto-generated and renewed cephadm-signed > certificates, > user-provided certificates, and certificate-store references. TLS PEM > handling is now > standardized with support for full certificate chains and automatic > separation of > certificates and private keys, simplifying certificate management > across services. > - SMB/Samba cluster support (the managed service type itself shipped in > Tentacle): new > remote-control client tool, per-share QoS/bandwidth rate limiting, > custom ports, > keybridge/FSCrypt-based share configuration, and support for CephFS > shares served from an > external Ceph cluster. > - NVMe-oF gateway spec gained many fields (TLS, KMIP, Prometheus > frequency, namespace > limits, group assignment). > - NFS-Ganesha: QoS support, active-active/keepalived-only ingress mode, > TLS > configuration support, RDMA handling during upgrades. > - Crimson/seastore OSD deployment support added to cephadm and > ceph-volume. > - cephadm now supports CRUSH bucket-aware upgrades using the new > ok-to-upgrade command, > enabling faster and safe concurrent OSD upgrades while preserving data > availability. > > CephFS / MDS > > - FSCrypt (client-side encryption) userspace implementation. > - Multi-threaded cephfs-mirror, plus a blockdiff-based sync mechanism for > regular-file > transfers. > - cephfs-mirror: new mirroring checkpoints (CLI + cluster-wide status > tracking), mutable > CephFS snapshot metadata (new libcephfs/pybind API), and substantially > expanded > per-directory/per-peer sync metrics. > - New pybind/cephfs bindings: utimensat, chownat, symlinkat, readlinkat, > statxat, mkdirat; > non-recursive rmtree()/cptree(). > - Subvolume group name-mangling support; new subvolume > performance/utilization metrics; > clone-source info retained in subvolume info. > - Root-directory fragmentation disallowed; session-reclaim blocklisting > fix; several > shutdown-hang and scrub-pinning fixes. > > ceph-volume > > - Migrated internal config/sysinfo representation to dataclasses; device > detection switched > to udev data instead of shelling out to LVM/udevadm in several code > paths (perf and > reliability win). > - Added seastore OSD support; fixed inventory failures on hosts with > RAID-mirrored system > volumes, cdrom devices, and missing /dev/vg/lv paths. > > Build / Packaging / QA Infrastructure > > - Ubuntu 24.04 (Noble) and Rocky Linux 10 added as supported distros > across build and QA; > CMake bumped to require 3.22.1 and gained CMake-4 support; C++ standard > raised to C++23. > - The Ceph container's base OS image has changed to rockylinux:10. > > We recommend users to update to this release. > For detailed release notes with links & changelog please refer to the > official blog entry at > https://ceph.io/en/news/blog/2026/v21-2-0-umbrella-released/ > > Getting Ceph > > - Git at git://github.com/ceph/ceph.git <http://github.com/ceph/ceph.git> > - Tarball at https://download.ceph.com/tarballs/ceph-21.2.0.tar.gz > - Containers at https://quay.io/repository/ceph/ceph > - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > - Release git sha1: b060042deb05d33e6cc358970cfb8221240bcf3e > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io