Hi Chris, Thanks for the detailed report. I’m not sure yet of the exact cause, but I suspect it’s related to a new spec field introduced in this release. In umbrella, the certificate a service uses is determined by the certificate_source field in its spec, not only by what’s in the certmgr store. If the grafana spec doesn’t set a source, it defaults to cephadm-signed, and cephadm then ignores grafana_ssl_cert/grafana_ssl_key entirely. That would explain why you kept getting the self-signed certificate even after cert-key set succeeded. The disappearing key smells like a bug that I need to reproduce to investigate what's going on. The HEALTH_ERR you still saw afterwards is most likely stale. The certificate check runs at mgr startup and then once a day by default, and a reconfig doesn't re-trigger it. You can force it by running: ceph orch certmgr cert check The upgrade also didn't carry over your tentacle-era grafana_cert/grafana_key entries to the new names or switch the spec to use them, so it silently fell back to a cephadm-signed cert. That's a regression on our side, and I'll open a tracker for it, along with the key set bug. So as for your questions: Q1: How to set a grafana cert/key on umbrella 1. Store the cert/key pair for the host. Either a combined PEM: ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key or separate files: ceph orch certmgr cert-key set grafana --hostname c1 --cert "$(cat x.y.crt)" --key "$(cat x.y.key)" 2. Tell the grafana service to use the stored cert: ceph orch ls grafana --export > grafana.yaml and add these under "spec:": ssl: true certificate_source: reference then: ceph orch apply -i grafana.yaml ceph orch reconfig grafana 3. Refresh the certificate health status: ceph orch certmgr cert check This should fix it Q2: Grafana on more than one host Grafana certificates are host-scoped, and that's fixed for the service type. You can't switch it to global. With certificate_source: reference you need to run the cert-key set once per grafana host, using the same file each time if it's the same cert: ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key ceph orch certmgr cert-key set grafana --hostname c2 -i x.y.crt-key I hope this helps to fix the issue. Best, Redouane On Wed, Oct 28, 2026 at 7:05 PM Chris Palmer <ceph-users@ceph.io> wrote:
I've just had a go at upgrading a cephadm-managed test cluster from 20.2.4 to 21.2.0. It went pretty well.
One minor thing (easily fixed): the upgrade re-enabled use of insecure keys (AUTH_INSECURE_KEYS_ALLOWED) so I had to put that back (ceph mon set auth_allowed_ciphers aes256k).
And a certificate problem that I can't see how to fix. I have a user-supplied grafana cert. (On this cluster grafana is only deployed on one node c1). On tentacle it was simply:
ceph orch certmgr cert set grafana_cert --hostname c1 -i x.ycrt ceph orch certmgr key set grafana_key --hostname c1 -i x.y.key ceph orch reconfig grafana
With umbrella the bindings have changed: grafana_cert is now grafana_ssl_cert and grafana_key is now grafana_ssl_key (according to ceph orch certmgr bindings ls) . So:
ceph orch certmgr cert set grafana_ssl_cert --hostname c1 -i x.ycrt ceph orch certmgr key set grafana_ssl_key --hostname c1 -i x.y.key ceph orch reconfig grafana
They get accepted, and I can see the key, but as soon as I issue the reconfig the key disappears (the cert remains). I then get grafana running with a cephadm-generated self-signed key, and the error:
root@c1:~# ceph health detail HEALTH_ERR Detected 1 cephadm certificate(s) issues: 1 invalid [ERR] CEPHADM_CERT_ERROR: Detected 1 cephadm certificate(s) issues: 1 invalid Certificate 'grafana_ssl_cert (c1)' (user-made) is not valid (error: missing key)
I also tried explicitly adding "--service_name grafana" to the sets but it made no difference.
I then tried using a combined cert/key file:
root@c1:~# ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key
This was accepted, and after the reconfig the key is still visible! However, I still get exactly the same health error, and it is still using the self-signed cert.
Q1: How can I set a grafana cert/key on umbrella?
Q2: If grafana is on more than one host do I need to install the same cert with each hostname? Is it intrinsically locked to be scope host, or can I set it with scope global?
Thanks, Chris
Umbrella is the 21st stable release of Ceph.
This is the first stable release of Ceph Umbrella. We recommend that all users update to this release.
Warnings
- Umbrella container images are based on Rocky Linux 10 (el10), which requires a CPU supporting the x86-64-v3 microarchitecture level. Deployments on hosts with older CPUs will not be able to run the containerized daemons. - Python 3.9 or newer is required on all hosts for cephadm support.
Notable Changes
RADOS / OSD / Erasure Coding
- Optimized ("Fast") Erasure Coding: a substantial new EC I/O path. - EC Direct Reads - CLS support for EC reads - EC OMAP support - Add Fast EC CRC-checking for data and first parity - Numerous correctness fixes to the new path (truncate+write
On 28/10/2026 13:59, Yuri Weinstein wrote: planning,
PDW vs RMW selection, snapshot scrub handling, peering/recovery corner cases) Ships with health warnings for deprecated EC plugins/techniques and for Blaum-Roth EC profiles whose w+1 isn't prime; non-4k-aligned chunk sizes are now rejected for optimized EC. - Security: CephX authentication hardening -- new AES256KRB5 cipher (replacing weaker legacy key types as the default), online key rotation for mon/OSD/MDS daemons during upgrades, and fixes for CVE-2025-30156 (AES-CBC misuse enabling authentication bypass), CVE-2026-39944, CVE-2026-50152, and CVE-2026-54330 (https://github.com/ceph/ceph/pull/71172). - Monitor: new optional RocksDB backup/restore mechanism (mon_backup_interval, ceph tell mon.* backup, offline restore via ceph-mon --restore-backup). - New availability score feature for Monitor/OSD (tracking + health warnings + config toggle), off by default. - Stretch clusters: tiebreaker mon can now be optional, added a max bucket-weight-diff threshold, a health warning for colocated monitors, and a restriction on changing the election strategy after stretch mode is enabled. - mgr/DaemonServer: ok-to-upgrade now aggregates and globally sorts OSDs across CRUSH child buckets (fixing inaccurate per-chassis/rack ordering), and upgrades can now be scoped to a CRUSH bucket. - pg-upmap-primary support and improved pg-upmap computing speed. - PG autoscaler: default mon_target_pg_per_osd increased 100 -> 200 and mon_max_pg_per_osd increased 250 -> 500, letting the autoscaler compute more parallelism-friendly pg_num values and improving the balancer's ability to reach uniform OSD utilization. Existing pools may autoscale to more PGs after upgrading as a result; pg_autoscale_during_upgrade (https://github.com/ceph/ceph/pull/67694) paces that splitting so it doesn't disrupt the upgrade itself. Clusters with a smaller-than-4GiB effective bluestore_osd_memory_target, or SSDs split into more than two OSDs, may want to pin the old values before upgrading (https://github.com/ceph/ceph/pull/69452). - BlueStore: BlueFS Spillover Cleaner evolution, faster allocation recovery, static/runtime object-fragmentation tracking, full-device discard on mkfs now default, several spanning-blob/onode-cache-stall/reallocation-corruption fixes, and device-type-specific allocation policy. - mClock scheduler refactor. Default HDD OSD shard config changed as an interim fix for inconsistent client/recovery throughput with multiple shards: osd_op_num_shards_hdd 5 -> 1, osd_op_num_threads_per_shard_hdd 1 -> 5 ( https://tracker.ceph.com/issues/66289). - Crimson/Seastore: continued heavy investment -- erasure coding support landed in Crimson (including recovery), PG splitting, a new 2Q cache replacement algorithm, broad coroutine conversions, and a long tail of correctness/perf fixes across the object-data, LBA/omap-btree, and cleaner subsystems.
RGW (RADOS Gateway)
- Full object dedup -- tech preview. - Bucket logging -- new feature: journal-mode logging, EC-pool support, admin commands, and new per-bucket lifecycle/logging performance monitoring that closes a long-standing observability gap around lifecycle processing. - Cloud transition / cloud restore: restore from Glacier/tape, admin CLI for restore list/status, restore-event notifications, persistent restore state, per-bucket target options; lifecycle, cloud-restore, and cloud-transition processing converted to coroutines for a significant performance improvement. - Encryption: added AES-256-GCM SSE support; CopyObject now supported for encrypted objects. - IAM/Account: GetAccountSummary, account-wide PublicAccessBlock, Admin REST APIs for account quota, S3 Object Ownership (disable ACLs), legacy Keystone admin-token requirement removed, Keystone roles injected into IAM policy. - STS/OIDC: GetCallerIdentity, JWT/JWKS signature validation, OIDC multisite support. - SigV4: fixed a regression (present in 19.2.6/20.2.4) that rejected requests with an unsigned Content-Type, breaking presigned PUT uploads; x-amz-content-sha256 is no longer required in SignedHeaders for S3 requests ( https://github.com/ceph/ceph/pull/71363). - Multisite: numerous sync-correctness fixes (full-sync infinite loop, segfault during startup, forwarded awsv4/UNSIGNED-PAYLOAD requests, bucket-sync full-sync locking, zonegroup rename handling). - D4N (Redis-backed) cache: continued build-out (bucket check_empty, connection pooling/pipelining, dirty-entry handling, crash fixes). - s3-tests vendored directly into the repo (src/test/rgw/s3-tests); QA suites run them from there instead of an external branch reference. - Timestamp-based epochs for versioned objects; global CORS rule support; rate-limit response with Retry-After; standalone RGW build (without RadosStore) via a new config-store abstraction -- a significant architectural refactor of rgw_zone/realm-watching code. - Lua scripting: new postAuth hook, fresh VM per background script execution, package-management fixes.
Dashboard
- Large-scale UI migration to the IBM Carbon design system across most pages (RGW, pools, OSD flags, multisite, zone/zonegroup, services, upgrade page, and more). - NVMe/TCP (NVMeoF) management significantly expanded: multi-step subsystem/gateway/namespace creation wizards, TLS/mTLS, DHCHAP auth, KMIP support, consistency-group CRUD APIs, and many CLI/UI alignment fixes. - SMB cluster management support added. - RGW role management realigned with Carbon + API routing fixes; global RGW tenant "Roles" tab removed in favor of per-account roles. - CephFS mirroring wizard/entity management UI. - OAuth2 SSO support (config UI + backend prerequisites). - Angular upgraded to v19.
cephadm / Orchestrator
- Extended certificate management: cephadm now provides comprehensive certmgr integration across managed services including mgmt-gateway, monitoring stack, NFS, and RGW. Services support auto-generated and renewed cephadm-signed certificates, user-provided certificates, and certificate-store references. TLS PEM handling is now standardized with support for full certificate chains and automatic separation of certificates and private keys, simplifying certificate management across services. - SMB/Samba cluster support (the managed service type itself shipped in Tentacle): new remote-control client tool, per-share QoS/bandwidth rate limiting, custom ports, keybridge/FSCrypt-based share configuration, and support for CephFS shares served from an external Ceph cluster. - NVMe-oF gateway spec gained many fields (TLS, KMIP, Prometheus frequency, namespace limits, group assignment). - NFS-Ganesha: QoS support, active-active/keepalived-only ingress mode, TLS configuration support, RDMA handling during upgrades. - Crimson/seastore OSD deployment support added to cephadm and ceph-volume. - cephadm now supports CRUSH bucket-aware upgrades using the new ok-to-upgrade command, enabling faster and safe concurrent OSD upgrades while preserving data availability.
CephFS / MDS
- FSCrypt (client-side encryption) userspace implementation. - Multi-threaded cephfs-mirror, plus a blockdiff-based sync mechanism for regular-file transfers. - cephfs-mirror: new mirroring checkpoints (CLI + cluster-wide status tracking), mutable CephFS snapshot metadata (new libcephfs/pybind API), and substantially expanded per-directory/per-peer sync metrics. - New pybind/cephfs bindings: utimensat, chownat, symlinkat, readlinkat, statxat, mkdirat; non-recursive rmtree()/cptree(). - Subvolume group name-mangling support; new subvolume performance/utilization metrics; clone-source info retained in subvolume info. - Root-directory fragmentation disallowed; session-reclaim blocklisting fix; several shutdown-hang and scrub-pinning fixes.
ceph-volume
- Migrated internal config/sysinfo representation to dataclasses; device detection switched to udev data instead of shelling out to LVM/udevadm in several code paths (perf and reliability win). - Added seastore OSD support; fixed inventory failures on hosts with RAID-mirrored system volumes, cdrom devices, and missing /dev/vg/lv paths.
Build / Packaging / QA Infrastructure
- Ubuntu 24.04 (Noble) and Rocky Linux 10 added as supported distros across build and QA; CMake bumped to require 3.22.1 and gained CMake-4 support; C++ standard raised to C++23. - The Ceph container's base OS image has changed to rockylinux:10.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v21-2-0-umbrella-released/
Getting Ceph
- Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.2.0.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: b060042deb05d33e6cc358970cfb8221240bcf3e _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io