Hi Redo Thanks for the useful info. I've made some progress, but not there yet... 1. The health warning was indeed stale and re-executing the check cleared it. 2. Executing this fails (we have already done "ceph mon set auth_allowed_ciphers aes256k" before the upgrade, and immediately after as the upgrade reset it). This looks like another bug: # ceph orch certmgr cert-key set grafana --hostname c1 --cert "$(cat x.y.crt)" --key "$(cat x.y.key)" 2026-10-29T11:49:03.802+0000 7f797c8ac6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:03.802+0000 7f797d0ad6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:06.801+0000 7f797d0ad6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:09.801+0000 7f797d0ad6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:09.801+0000 7f797c8ac6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] 2026-10-29T11:49:09.801+0000 7f797eb376c0 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 95] error connecting to the cluster 3. But loading it from a combined file works: # ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key Certificate/key pair set correctly # ceph orch certmgr cert ls --filter-by "name=grafana_ssl_cert" grafana_ssl_cert scope: host certificates c1 subject countryName: xx stateOrProvinceName: xxxxx organizationName: xxxxx commonName: x.y contains_chain: False validity remaining_days: 635 # ceph orch certmgr key ls grafana_ssl_key scope: host keys c1 key_type: RSA key_size: 4096 4. I exported the grafana spec, and changed the certificate source, and issued the reconfig ok. # cat grafana.yaml service_type: grafana service_name: grafana placement: hosts: - c1 spec: anonymous_access: true certificate_source: reference port: 3000 protocol: https ssl: true 5. But now grafana fails to start: # ceph health detail HEALTH_WARN 1 failed cephadm daemon(s) [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon grafana.c1 on c1 is in error state And from journalctl: [starting module *api.HTTPServer: invalid service state: Failed, expected: Running, failure: could not load SSL certificate: tls: failed to find any PEM data in certificate input] I'm really not sure where to go from here! Regards, Chris On 28/10/2026 20:32, Redouane Kachach wrote:
Hi Chris,
Thanks for the detailed report. I’m not sure yet of the exact cause, but I suspect it’s related to a new spec field introduced in this release. In umbrella, the certificate a service uses is determined by the certificate_source field in its spec, not only by what’s in the certmgr store. If the grafana spec doesn’t set a source, it defaults to cephadm-signed, and cephadm then ignores grafana_ssl_cert/grafana_ssl_key entirely. That would explain why you kept getting the self-signed certificate even after cert-key set succeeded.
The disappearing key smells like a bug that I need to reproduce to investigate what's going on. The HEALTH_ERR you still saw afterwards is most likely stale. The certificate check runs at mgr startup and then once a day by default, and a reconfig doesn't re-trigger it. You can force it by running: ceph orch certmgr cert check
The upgrade also didn't carry over your tentacle-era grafana_cert/grafana_key entries to the new names or switch the spec to use them, so it silently fell back to a cephadm-signed cert. That's a regression on our side, and I'll open a tracker for it, along with the key set bug. So as for your questions:
Q1: How to set a grafana cert/key on umbrella
1. Store the cert/key pair for the host. Either a combined PEM:
ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key
or separate files:
ceph orch certmgr cert-key set grafana --hostname c1 --cert "$(cat x.y.crt)" --key "$(cat x.y.key)"
2. Tell the grafana service to use the stored cert:
ceph orch ls grafana --export > grafana.yaml
and add these under "spec:":
ssl: true certificate_source: reference
then:
ceph orch apply -i grafana.yaml ceph orch reconfig grafana
3. Refresh the certificate health status:
ceph orch certmgr cert check
This should fix it
Q2: Grafana on more than one host
Grafana certificates are host-scoped, and that's fixed for the service type. You can't switch it to global. With certificate_source: reference you need to run the cert-key set once per grafana host, using the same file each time if it's the same cert:
ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key ceph orch certmgr cert-key set grafana --hostname c2 -i x.y.crt-key
I hope this helps to fix the issue.
Best, Redouane
On Wed, Oct 28, 2026 at 7:05 PM Chris Palmer <ceph-users@ceph.io> wrote:
I've just had a go at upgrading a cephadm-managed test cluster from 20.2.4 to 21.2.0. It went pretty well.
One minor thing (easily fixed): the upgrade re-enabled use of insecure keys (AUTH_INSECURE_KEYS_ALLOWED) so I had to put that back (ceph mon set auth_allowed_ciphers aes256k).
And a certificate problem that I can't see how to fix. I have a user-supplied grafana cert. (On this cluster grafana is only deployed on one node c1). On tentacle it was simply:
ceph orch certmgr cert set grafana_cert --hostname c1 -i x.ycrt ceph orch certmgr key set grafana_key --hostname c1 -i x.y.key ceph orch reconfig grafana
With umbrella the bindings have changed: grafana_cert is now grafana_ssl_cert and grafana_key is now grafana_ssl_key (according to ceph orch certmgr bindings ls) . So:
ceph orch certmgr cert set grafana_ssl_cert --hostname c1 -i x.ycrt ceph orch certmgr key set grafana_ssl_key --hostname c1 -i x.y.key ceph orch reconfig grafana
They get accepted, and I can see the key, but as soon as I issue the reconfig the key disappears (the cert remains). I then get grafana running with a cephadm-generated self-signed key, and the error:
root@c1:~# ceph health detail HEALTH_ERR Detected 1 cephadm certificate(s) issues: 1 invalid [ERR] CEPHADM_CERT_ERROR: Detected 1 cephadm certificate(s) issues: 1 invalid Certificate 'grafana_ssl_cert (c1)' (user-made) is not valid (error: missing key)
I also tried explicitly adding "--service_name grafana" to the sets but it made no difference.
I then tried using a combined cert/key file:
root@c1:~# ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key
This was accepted, and after the reconfig the key is still visible! However, I still get exactly the same health error, and it is still using the self-signed cert.
Q1: How can I set a grafana cert/key on umbrella?
Q2: If grafana is on more than one host do I need to install the same cert with each hostname? Is it intrinsically locked to be scope host, or can I set it with scope global?
Thanks, Chris
On 28/10/2026 13:59, Yuri Weinstein wrote: > Umbrella is the 21st stable release of Ceph. > > This is the first stable release of Ceph Umbrella. > We recommend that all users update to this release. > > Warnings > > - Umbrella container images are based on Rocky Linux 10 (el10), which > requires a CPU > supporting the x86-64-v3 microarchitecture level. Deployments on hosts > with older CPUs > will not be able to run the containerized daemons. > - Python 3.9 or newer is required on all hosts for cephadm support. > > Notable Changes > > RADOS / OSD / Erasure Coding > > - Optimized ("Fast") Erasure Coding: a substantial new EC I/O path. > - EC Direct Reads > - CLS support for EC reads > - EC OMAP support > - Add Fast EC CRC-checking for data and first parity > - Numerous correctness fixes to the new path (truncate+write planning, > PDW vs RMW > selection, snapshot scrub handling, peering/recovery corner cases) > Ships with health warnings for deprecated EC plugins/techniques and > for Blaum-Roth EC > profiles whose w+1 isn't prime; non-4k-aligned chunk sizes are now > rejected for > optimized EC. > - Security: CephX authentication hardening -- new AES256KRB5 cipher > (replacing weaker > legacy key types as the default), online key rotation for mon/OSD/MDS > daemons during > upgrades, and fixes for CVE-2025-30156 (AES-CBC misuse enabling > authentication bypass), > CVE-2026-39944, CVE-2026-50152, and CVE-2026-54330 > (https://github.com/ceph/ceph/pull/71172). > - Monitor: new optional RocksDB backup/restore mechanism > (mon_backup_interval, ceph tell > mon.* backup, offline restore via ceph-mon --restore-backup). > - New availability score feature for Monitor/OSD (tracking + health > warnings + config > toggle), off by default. > - Stretch clusters: tiebreaker mon can now be optional, added a max > bucket-weight-diff > threshold, a health warning for colocated monitors, and a restriction > on changing the > election strategy after stretch mode is enabled. > - mgr/DaemonServer: ok-to-upgrade now aggregates and globally sorts OSDs > across CRUSH child > buckets (fixing inaccurate per-chassis/rack ordering), and upgrades can > now be scoped to > a CRUSH bucket. > - pg-upmap-primary support and improved pg-upmap computing speed. > - PG autoscaler: default mon_target_pg_per_osd increased 100 -> 200 and > mon_max_pg_per_osd > increased 250 -> 500, letting the autoscaler compute more > parallelism-friendly pg_num > values and improving the balancer's ability to reach uniform OSD > utilization. Existing > pools may autoscale to more PGs after upgrading as a result; > pg_autoscale_during_upgrade > (https://github.com/ceph/ceph/pull/67694) paces that splitting so it > doesn't disrupt the > upgrade itself. Clusters with a smaller-than-4GiB effective > bluestore_osd_memory_target, > or SSDs split into more than two OSDs, may want to pin the old values > before upgrading > (https://github.com/ceph/ceph/pull/69452). > - BlueStore: BlueFS Spillover Cleaner evolution, faster allocation > recovery, static/runtime > object-fragmentation tracking, full-device discard on mkfs now default, > several > spanning-blob/onode-cache-stall/reallocation-corruption fixes, and > device-type-specific > allocation policy. > - mClock scheduler refactor. Default HDD OSD shard config changed as an > interim fix for > inconsistent client/recovery throughput with multiple shards: > osd_op_num_shards_hdd 5 -> > 1, osd_op_num_threads_per_shard_hdd 1 -> 5 ( > https://tracker.ceph.com/issues/66289). > - Crimson/Seastore: continued heavy investment -- erasure coding support > landed in Crimson > (including recovery), PG splitting, a new 2Q cache replacement > algorithm, broad coroutine > conversions, and a long tail of correctness/perf fixes across the > object-data, > LBA/omap-btree, and cleaner subsystems. > > RGW (RADOS Gateway) > > - Full object dedup -- tech preview. > - Bucket logging -- new feature: journal-mode logging, EC-pool support, > admin commands, and > new per-bucket lifecycle/logging performance monitoring that closes a > long-standing > observability gap around lifecycle processing. > - Cloud transition / cloud restore: restore from Glacier/tape, admin CLI > for restore > list/status, restore-event notifications, persistent restore state, > per-bucket target > options; lifecycle, cloud-restore, and cloud-transition processing > converted to > coroutines for a significant performance improvement. > - Encryption: added AES-256-GCM SSE support; CopyObject now supported for > encrypted > objects. > - IAM/Account: GetAccountSummary, account-wide PublicAccessBlock, Admin > REST APIs for > account quota, S3 Object Ownership (disable ACLs), legacy Keystone > admin-token > requirement removed, Keystone roles injected into IAM policy. > - STS/OIDC: GetCallerIdentity, JWT/JWKS signature validation, OIDC > multisite support. > - SigV4: fixed a regression (present in 19.2.6/20.2.4) that rejected > requests with an > unsigned Content-Type, breaking presigned PUT uploads; > x-amz-content-sha256 is no longer > required in SignedHeaders for S3 requests ( > https://github.com/ceph/ceph/pull/71363). > - Multisite: numerous sync-correctness fixes (full-sync infinite loop, > segfault during > startup, forwarded awsv4/UNSIGNED-PAYLOAD requests, bucket-sync > full-sync locking, > zonegroup rename handling). > - D4N (Redis-backed) cache: continued build-out (bucket check_empty, > connection > pooling/pipelining, dirty-entry handling, crash fixes). > - s3-tests vendored directly into the repo (src/test/rgw/s3-tests); QA > suites run them from > there instead of an external branch reference. > - Timestamp-based epochs for versioned objects; global CORS rule support; > rate-limit > response with Retry-After; standalone RGW build (without RadosStore) > via a new > config-store abstraction -- a significant architectural refactor of > rgw_zone/realm-watching code. > - Lua scripting: new postAuth hook, fresh VM per background script > execution, > package-management fixes. > > Dashboard > > - Large-scale UI migration to the IBM Carbon design system across most > pages (RGW, pools, > OSD flags, multisite, zone/zonegroup, services, upgrade page, and more). > - NVMe/TCP (NVMeoF) management significantly expanded: multi-step > subsystem/gateway/namespace creation wizards, TLS/mTLS, DHCHAP auth, > KMIP support, > consistency-group CRUD APIs, and many CLI/UI alignment fixes. > - SMB cluster management support added. > - RGW role management realigned with Carbon + API routing fixes; global > RGW tenant "Roles" > tab removed in favor of per-account roles. > - CephFS mirroring wizard/entity management UI. > - OAuth2 SSO support (config UI + backend prerequisites). > - Angular upgraded to v19. > > cephadm / Orchestrator > > - Extended certificate management: cephadm now provides comprehensive > certmgr > integration across managed services including mgmt-gateway, monitoring > stack, NFS, > and RGW. Services support auto-generated and renewed cephadm-signed > certificates, > user-provided certificates, and certificate-store references. TLS PEM > handling is now > standardized with support for full certificate chains and automatic > separation of > certificates and private keys, simplifying certificate management > across services. > - SMB/Samba cluster support (the managed service type itself shipped in > Tentacle): new > remote-control client tool, per-share QoS/bandwidth rate limiting, > custom ports, > keybridge/FSCrypt-based share configuration, and support for CephFS > shares served from an > external Ceph cluster. > - NVMe-oF gateway spec gained many fields (TLS, KMIP, Prometheus > frequency, namespace > limits, group assignment). > - NFS-Ganesha: QoS support, active-active/keepalived-only ingress mode, > TLS > configuration support, RDMA handling during upgrades. > - Crimson/seastore OSD deployment support added to cephadm and > ceph-volume. > - cephadm now supports CRUSH bucket-aware upgrades using the new > ok-to-upgrade command, > enabling faster and safe concurrent OSD upgrades while preserving data > availability. > > CephFS / MDS > > - FSCrypt (client-side encryption) userspace implementation. > - Multi-threaded cephfs-mirror, plus a blockdiff-based sync mechanism for > regular-file > transfers. > - cephfs-mirror: new mirroring checkpoints (CLI + cluster-wide status > tracking), mutable > CephFS snapshot metadata (new libcephfs/pybind API), and substantially > expanded > per-directory/per-peer sync metrics. > - New pybind/cephfs bindings: utimensat, chownat, symlinkat, readlinkat, > statxat, mkdirat; > non-recursive rmtree()/cptree(). > - Subvolume group name-mangling support; new subvolume > performance/utilization metrics; > clone-source info retained in subvolume info. > - Root-directory fragmentation disallowed; session-reclaim blocklisting > fix; several > shutdown-hang and scrub-pinning fixes. > > ceph-volume > > - Migrated internal config/sysinfo representation to dataclasses; device > detection switched > to udev data instead of shelling out to LVM/udevadm in several code > paths (perf and > reliability win). > - Added seastore OSD support; fixed inventory failures on hosts with > RAID-mirrored system > volumes, cdrom devices, and missing /dev/vg/lv paths. > > Build / Packaging / QA Infrastructure > > - Ubuntu 24.04 (Noble) and Rocky Linux 10 added as supported distros > across build and QA; > CMake bumped to require 3.22.1 and gained CMake-4 support; C++ standard > raised to C++23. > - The Ceph container's base OS image has changed to rockylinux:10. > > We recommend users to update to this release. > For detailed release notes with links & changelog please refer to the > official blog entry at > https://ceph.io/en/news/blog/2026/v21-2-0-umbrella-released/ > > Getting Ceph > > - Git at git://github.com/ceph/ceph.git <http://github.com/ceph/ceph.git> > - Tarball at https://download.ceph.com/tarballs/ceph-21.2.0.tar.gz > - Containers at https://quay.io/repository/ceph/ceph > - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > - Release git sha1: b060042deb05d33e6cc358970cfb8221240bcf3e > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io