I've just had a go at upgrading a cephadm-managed test cluster from 20.2.4 to 21.2.0. It went pretty well. One minor thing (easily fixed): the upgrade re-enabled use of insecure keys (AUTH_INSECURE_KEYS_ALLOWED) so I had to put that back (ceph mon set auth_allowed_ciphers aes256k). And a certificate problem that I can't see how to fix. I have a user-supplied grafana cert. (On this cluster grafana is only deployed on one node c1). On tentacle it was simply: ceph orch certmgr cert set grafana_cert --hostname c1 -i x.ycrt ceph orch certmgr key set grafana_key --hostname c1 -i x.y.key ceph orch reconfig grafana With umbrella the bindings have changed: grafana_cert is now grafana_ssl_cert and grafana_key is now grafana_ssl_key (according to ceph orch certmgr bindings ls) . So: ceph orch certmgr cert set grafana_ssl_cert --hostname c1 -i x.ycrt ceph orch certmgr key set grafana_ssl_key --hostname c1 -i x.y.key ceph orch reconfig grafana They get accepted, and I can see the key, but as soon as I issue the reconfig the key disappears (the cert remains). I then get grafana running with a cephadm-generated self-signed key, and the error: root@c1:~# ceph health detail HEALTH_ERR Detected 1 cephadm certificate(s) issues: 1 invalid [ERR] CEPHADM_CERT_ERROR: Detected 1 cephadm certificate(s) issues: 1 invalid Certificate 'grafana_ssl_cert (c1)' (user-made) is not valid (error: missing key) I also tried explicitly adding "--service_name grafana" to the sets but it made no difference. I then tried using a combined cert/key file: root@c1:~# ceph orch certmgr cert-key set grafana --hostname c1 -i x.y.crt-key This was accepted, and after the reconfig the key is still visible! However, I still get exactly the same health error, and it is still using the self-signed cert. Q1: How can I set a grafana cert/key on umbrella? Q2: If grafana is on more than one host do I need to install the same cert with each hostname? Is it intrinsically locked to be scope host, or can I set it with scope global? Thanks, Chris On 28/10/2026 13:59, Yuri Weinstein wrote:
Umbrella is the 21st stable release of Ceph.
This is the first stable release of Ceph Umbrella. We recommend that all users update to this release.
Warnings
- Umbrella container images are based on Rocky Linux 10 (el10), which requires a CPU supporting the x86-64-v3 microarchitecture level. Deployments on hosts with older CPUs will not be able to run the containerized daemons. - Python 3.9 or newer is required on all hosts for cephadm support.
Notable Changes
RADOS / OSD / Erasure Coding
- Optimized ("Fast") Erasure Coding: a substantial new EC I/O path. - EC Direct Reads - CLS support for EC reads - EC OMAP support - Add Fast EC CRC-checking for data and first parity - Numerous correctness fixes to the new path (truncate+write planning, PDW vs RMW selection, snapshot scrub handling, peering/recovery corner cases) Ships with health warnings for deprecated EC plugins/techniques and for Blaum-Roth EC profiles whose w+1 isn't prime; non-4k-aligned chunk sizes are now rejected for optimized EC. - Security: CephX authentication hardening -- new AES256KRB5 cipher (replacing weaker legacy key types as the default), online key rotation for mon/OSD/MDS daemons during upgrades, and fixes for CVE-2025-30156 (AES-CBC misuse enabling authentication bypass), CVE-2026-39944, CVE-2026-50152, and CVE-2026-54330 (https://github.com/ceph/ceph/pull/71172). - Monitor: new optional RocksDB backup/restore mechanism (mon_backup_interval, ceph tell mon.* backup, offline restore via ceph-mon --restore-backup). - New availability score feature for Monitor/OSD (tracking + health warnings + config toggle), off by default. - Stretch clusters: tiebreaker mon can now be optional, added a max bucket-weight-diff threshold, a health warning for colocated monitors, and a restriction on changing the election strategy after stretch mode is enabled. - mgr/DaemonServer: ok-to-upgrade now aggregates and globally sorts OSDs across CRUSH child buckets (fixing inaccurate per-chassis/rack ordering), and upgrades can now be scoped to a CRUSH bucket. - pg-upmap-primary support and improved pg-upmap computing speed. - PG autoscaler: default mon_target_pg_per_osd increased 100 -> 200 and mon_max_pg_per_osd increased 250 -> 500, letting the autoscaler compute more parallelism-friendly pg_num values and improving the balancer's ability to reach uniform OSD utilization. Existing pools may autoscale to more PGs after upgrading as a result; pg_autoscale_during_upgrade (https://github.com/ceph/ceph/pull/67694) paces that splitting so it doesn't disrupt the upgrade itself. Clusters with a smaller-than-4GiB effective bluestore_osd_memory_target, or SSDs split into more than two OSDs, may want to pin the old values before upgrading (https://github.com/ceph/ceph/pull/69452). - BlueStore: BlueFS Spillover Cleaner evolution, faster allocation recovery, static/runtime object-fragmentation tracking, full-device discard on mkfs now default, several spanning-blob/onode-cache-stall/reallocation-corruption fixes, and device-type-specific allocation policy. - mClock scheduler refactor. Default HDD OSD shard config changed as an interim fix for inconsistent client/recovery throughput with multiple shards: osd_op_num_shards_hdd 5 -> 1, osd_op_num_threads_per_shard_hdd 1 -> 5 ( https://tracker.ceph.com/issues/66289). - Crimson/Seastore: continued heavy investment -- erasure coding support landed in Crimson (including recovery), PG splitting, a new 2Q cache replacement algorithm, broad coroutine conversions, and a long tail of correctness/perf fixes across the object-data, LBA/omap-btree, and cleaner subsystems.
RGW (RADOS Gateway)
- Full object dedup -- tech preview. - Bucket logging -- new feature: journal-mode logging, EC-pool support, admin commands, and new per-bucket lifecycle/logging performance monitoring that closes a long-standing observability gap around lifecycle processing. - Cloud transition / cloud restore: restore from Glacier/tape, admin CLI for restore list/status, restore-event notifications, persistent restore state, per-bucket target options; lifecycle, cloud-restore, and cloud-transition processing converted to coroutines for a significant performance improvement. - Encryption: added AES-256-GCM SSE support; CopyObject now supported for encrypted objects. - IAM/Account: GetAccountSummary, account-wide PublicAccessBlock, Admin REST APIs for account quota, S3 Object Ownership (disable ACLs), legacy Keystone admin-token requirement removed, Keystone roles injected into IAM policy. - STS/OIDC: GetCallerIdentity, JWT/JWKS signature validation, OIDC multisite support. - SigV4: fixed a regression (present in 19.2.6/20.2.4) that rejected requests with an unsigned Content-Type, breaking presigned PUT uploads; x-amz-content-sha256 is no longer required in SignedHeaders for S3 requests ( https://github.com/ceph/ceph/pull/71363). - Multisite: numerous sync-correctness fixes (full-sync infinite loop, segfault during startup, forwarded awsv4/UNSIGNED-PAYLOAD requests, bucket-sync full-sync locking, zonegroup rename handling). - D4N (Redis-backed) cache: continued build-out (bucket check_empty, connection pooling/pipelining, dirty-entry handling, crash fixes). - s3-tests vendored directly into the repo (src/test/rgw/s3-tests); QA suites run them from there instead of an external branch reference. - Timestamp-based epochs for versioned objects; global CORS rule support; rate-limit response with Retry-After; standalone RGW build (without RadosStore) via a new config-store abstraction -- a significant architectural refactor of rgw_zone/realm-watching code. - Lua scripting: new postAuth hook, fresh VM per background script execution, package-management fixes.
Dashboard
- Large-scale UI migration to the IBM Carbon design system across most pages (RGW, pools, OSD flags, multisite, zone/zonegroup, services, upgrade page, and more). - NVMe/TCP (NVMeoF) management significantly expanded: multi-step subsystem/gateway/namespace creation wizards, TLS/mTLS, DHCHAP auth, KMIP support, consistency-group CRUD APIs, and many CLI/UI alignment fixes. - SMB cluster management support added. - RGW role management realigned with Carbon + API routing fixes; global RGW tenant "Roles" tab removed in favor of per-account roles. - CephFS mirroring wizard/entity management UI. - OAuth2 SSO support (config UI + backend prerequisites). - Angular upgraded to v19.
cephadm / Orchestrator
- Extended certificate management: cephadm now provides comprehensive certmgr integration across managed services including mgmt-gateway, monitoring stack, NFS, and RGW. Services support auto-generated and renewed cephadm-signed certificates, user-provided certificates, and certificate-store references. TLS PEM handling is now standardized with support for full certificate chains and automatic separation of certificates and private keys, simplifying certificate management across services. - SMB/Samba cluster support (the managed service type itself shipped in Tentacle): new remote-control client tool, per-share QoS/bandwidth rate limiting, custom ports, keybridge/FSCrypt-based share configuration, and support for CephFS shares served from an external Ceph cluster. - NVMe-oF gateway spec gained many fields (TLS, KMIP, Prometheus frequency, namespace limits, group assignment). - NFS-Ganesha: QoS support, active-active/keepalived-only ingress mode, TLS configuration support, RDMA handling during upgrades. - Crimson/seastore OSD deployment support added to cephadm and ceph-volume. - cephadm now supports CRUSH bucket-aware upgrades using the new ok-to-upgrade command, enabling faster and safe concurrent OSD upgrades while preserving data availability.
CephFS / MDS
- FSCrypt (client-side encryption) userspace implementation. - Multi-threaded cephfs-mirror, plus a blockdiff-based sync mechanism for regular-file transfers. - cephfs-mirror: new mirroring checkpoints (CLI + cluster-wide status tracking), mutable CephFS snapshot metadata (new libcephfs/pybind API), and substantially expanded per-directory/per-peer sync metrics. - New pybind/cephfs bindings: utimensat, chownat, symlinkat, readlinkat, statxat, mkdirat; non-recursive rmtree()/cptree(). - Subvolume group name-mangling support; new subvolume performance/utilization metrics; clone-source info retained in subvolume info. - Root-directory fragmentation disallowed; session-reclaim blocklisting fix; several shutdown-hang and scrub-pinning fixes.
ceph-volume
- Migrated internal config/sysinfo representation to dataclasses; device detection switched to udev data instead of shelling out to LVM/udevadm in several code paths (perf and reliability win). - Added seastore OSD support; fixed inventory failures on hosts with RAID-mirrored system volumes, cdrom devices, and missing /dev/vg/lv paths.
Build / Packaging / QA Infrastructure
- Ubuntu 24.04 (Noble) and Rocky Linux 10 added as supported distros across build and QA; CMake bumped to require 3.22.1 and gained CMake-4 support; C++ standard raised to C++23. - The Ceph container's base OS image has changed to rockylinux:10.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v21-2-0-umbrella-released/
Getting Ceph
- Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.2.0.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: b060042deb05d33e6cc358970cfb8221240bcf3e _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io