v21.1.1 Umbrella (RC1) released
This is the second release candidate for Umbrella. All community feedback is welcome. Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie) Containers are built on a Rocky 10 base image, available at: https://quay.io/repository/ceph/ceph?tab=tags Please note: the full draft release notes for v21.2.0 are available at https://github.com/ceph/ceph/pull/70240 Notable changes from 21.1.0, the first release candidate: - Fixes for CVE-2025-30156, CVE-2026-39944,CVE-2026-50152, and CVE-2026-54330 included. (Please see upgrade notes for v20.2.4/v19.2.6.) - MGR: The default values of ``mon_target_pg_per_osd`` and ``mon_max_pg_per_osd`` have been increased from 100 and 250 to 200 and 500, respectively. These values enhance the PG autoscaler's ability to calculate reasonable ``pg_num`` values for pools under its control. These in turn increase parallelism and performance, and improve the ``balancer`` module's ability to acheive uniform OSD utilization. [This concomitant PR](https://github.com/ceph/ceph/pull/67694) ensures that upgrades are not delayed or disrupted by PG splitting as a result of this change. Clusters with effective ``bluestore_osd_memory_target`` values smaller than the default 4GiB may wish to explicitly set the previous default values in the central config store before upgrading. This will reduce the potential for increased memory starvation. Clusters with SSDs split into more than two OSDs may also wish to pin their current values in advance, or redeploy those SSDs with at most two OSDs per. - RGW: In multisite deployments, zone endpoints configured in the zonegroup (e.g. ``https://zone-a.example.com``) can now be resolved to all IP addresses returned by DNS, with inter-zone traffic distributed across them using round-robin and per-IP health tracking. This enables DNS-based service discovery for inter-zone traffic without an external load balancer. This feature is disabled by default: to opt in, set ``rgw_rest_conn_connect_to_resolved_ips = true``. The per-IP retry-after-failure timeout is controlled by ``rgw_rest_conn_ip_fail_timeout_secs`` (default: 2 seconds). Deployments that work around libcurl's single-address behavior by repeating the same endpoint multiple times in zone configuration should review that setup before enabling, as round-robin distribution makes such duplication unnecessary. - RGW: The rgw_gc_max_deferred and rgw_gc_max_deferred_entries_size options have been removed, as they did not do anything. - Updated the default for rgw_gc_max_queue_size to account for the extra space from removing rgw_gc_max_deferred_entries_size. - RGW: The default value of ``rgw_thread_pool_size`` has been reduced from 512 to 128. Benchmarks showed that the lower thread count improves throughput and reduces latency on NVMe and HDD based environments. Users with specific workload requirements can tune this value; see also ``rgw_max_concurrent_requests``. - CephFS Mirroring: The ``fs mirror peer status`` admin socket command reports additional per-directory sync metrics (sync mode, throughput, crawl and data-sync queue timing, bytes/files progress, and ETA). Output is grouped under ``metrics/<mirrored-dir-path>/peer/<peer-uuid>``. For more information, see https://tracker.ceph.com/issues/73453 - CephFS Mirroring: Per-directory snapshot sync progress is exposed as labeled perf counters in the ``cephfs_mirror_directory`` group (``counter dump`` on the mirror daemon admin socket, exportable via ``ceph-exporter``). Counters mirror ``fs mirror peer status`` fields (directory state, current sync, last sync, and snap summary) and are labeled by peer UUID and mirrored directory path. See https://tracker.ceph.com/issues/73457 - RADOS: PG autoscaler allows overlapping roots. Each root receives a PG target based on its OSDs, with OSDs shared across multiple roots contributing proportionally less to each root's allocation. - CephFS Mirroring: Introduced snapshot checkpoints feature that allows users to mark specific snapshots as important milestones and track their replication status to remote sites. Checkpoints automatically detect if a snapshot has already been synced and provide visibility into disaster recovery readiness. The feature includes four new CLI commands: ``ceph fs snapshot mirror checkpoint add`` to mark a snapshot, ``ceph fs snapshot mirror checkpoint now`` to checkpoint the latest snapshot, ``ceph fs snapshot mirror checkpoint ls`` to list all checkpoints with their status (created/complete/failed), and ``ceph fs snapshot mirror checkpoint remove`` to remove a checkpoint. Checkpoint metadata is persistent across daemon restarts. This feature is useful for compliance auditing, application consistency verification, and SLA tracking. For more information, see https://tracker.ceph.com/issues/73454 - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.1.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: 815bc6b6f9b4e0fd98daed552ec8e7123a0a8f84 - Distribution matrix: https://docs.ceph.com/en/latest/start/os-recommendations/ -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi, I just tried to deploy the v21 RC1 as a VM but failed with: Fatal glibc error: CPU does not support x86-64-v3 I am aware that the release notes will contain a note about this: https://github.com/ceph/ceph/pull/70240 But I'm wondering if it's really necessary to drop v2 support (already). Thanks, Eugen Zitat von Patrick Donnelly <ceph-users@ceph.io>:
This is the second release candidate for Umbrella. All community feedback is welcome.
Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie)
Containers are built on a Rocky 10 base image, available at: https://quay.io/repository/ceph/ceph?tab=tags
Please note: the full draft release notes for v21.2.0 are available at https://github.com/ceph/ceph/pull/70240
Notable changes from 21.1.0, the first release candidate:
- Fixes for CVE-2025-30156, CVE-2026-39944,CVE-2026-50152, and CVE-2026-54330 included. (Please see upgrade notes for v20.2.4/v19.2.6.) - MGR: The default values of ``mon_target_pg_per_osd`` and ``mon_max_pg_per_osd`` have been increased from 100 and 250 to 200 and 500, respectively. These values enhance the PG autoscaler's ability to calculate reasonable ``pg_num`` values for pools under its control. These in turn increase parallelism and performance, and improve the ``balancer`` module's ability to acheive uniform OSD utilization.
[This concomitant PR](https://github.com/ceph/ceph/pull/67694) ensures that upgrades are not delayed or disrupted by PG splitting as a result of this change.
Clusters with effective ``bluestore_osd_memory_target`` values smaller than the default 4GiB may wish to explicitly set the previous default values in the central config store before upgrading. This will reduce the potential for increased memory starvation. Clusters with SSDs split into more than two OSDs may also wish to pin their current values in advance, or redeploy those SSDs with at most two OSDs per. - RGW: In multisite deployments, zone endpoints configured in the zonegroup (e.g. ``https://zone-a.example.com``) can now be resolved to all IP addresses returned by DNS, with inter-zone traffic distributed across them using round-robin and per-IP health tracking. This enables DNS-based service discovery for inter-zone traffic without an external load balancer. This feature is disabled by default: to opt in, set ``rgw_rest_conn_connect_to_resolved_ips = true``. The per-IP retry-after-failure timeout is controlled by ``rgw_rest_conn_ip_fail_timeout_secs`` (default: 2 seconds). Deployments that work around libcurl's single-address behavior by repeating the same endpoint multiple times in zone configuration should review that setup before enabling, as round-robin distribution makes such duplication unnecessary. - RGW: The rgw_gc_max_deferred and rgw_gc_max_deferred_entries_size options have been removed, as they did not do anything. - Updated the default for rgw_gc_max_queue_size to account for the extra space from removing rgw_gc_max_deferred_entries_size. - RGW: The default value of ``rgw_thread_pool_size`` has been reduced from 512 to 128. Benchmarks showed that the lower thread count improves throughput and reduces latency on NVMe and HDD based environments. Users with specific workload requirements can tune this value; see also ``rgw_max_concurrent_requests``. - CephFS Mirroring: The ``fs mirror peer status`` admin socket command reports additional per-directory sync metrics (sync mode, throughput, crawl and data-sync queue timing, bytes/files progress, and ETA). Output is grouped under ``metrics/<mirrored-dir-path>/peer/<peer-uuid>``. For more information, see https://tracker.ceph.com/issues/73453 - CephFS Mirroring: Per-directory snapshot sync progress is exposed as labeled perf counters in the ``cephfs_mirror_directory`` group (``counter dump`` on the mirror daemon admin socket, exportable via ``ceph-exporter``). Counters mirror ``fs mirror peer status`` fields (directory state, current sync, last sync, and snap summary) and are labeled by peer UUID and mirrored directory path. See https://tracker.ceph.com/issues/73457 - RADOS: PG autoscaler allows overlapping roots. Each root receives a PG target based on its OSDs, with OSDs shared across multiple roots contributing proportionally less to each root's allocation. - CephFS Mirroring: Introduced snapshot checkpoints feature that allows users to mark specific snapshots as important milestones and track their replication status to remote sites. Checkpoints automatically detect if a snapshot has already been synced and provide visibility into disaster recovery readiness. The feature includes four new CLI commands: ``ceph fs snapshot mirror checkpoint add`` to mark a snapshot, ``ceph fs snapshot mirror checkpoint now`` to checkpoint the latest snapshot, ``ceph fs snapshot mirror checkpoint ls`` to list all checkpoints with their status (created/complete/failed), and ``ceph fs snapshot mirror checkpoint remove`` to remove a checkpoint. Checkpoint metadata is persistent across daemon restarts. This feature is useful for compliance auditing, application consistency verification, and SLA tracking. For more information, see https://tracker.ceph.com/issues/73454 - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store.
- Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.1.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: 815bc6b6f9b4e0fd98daed552ec8e7123a0a8f84 - Distribution matrix: https://docs.ceph.com/en/latest/start/os-recommendations/
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eugen Block <ceph-users@ceph.io> writes: Hi,
I just tried to deploy the v21 RC1 as a VM but failed with:
Fatal glibc error: CPU does not support x86-64-v3
Are you using Rocky 10 for that? Not that the error comes from glibc. The support for anything older has been dropped in RedHat 10, so I don’t think this is specific to Ceph: https://access.redhat.com/solutions/7122807 -- Stefan
I tried on openSUSE Leap 16 as well as debian 13 and Ubuntu 26 VMs. I thought maybe this could be just a build flag when building containers or so. But I'm not familiar with the build process or anything so I just wanted to ask. Zitat von Stefan Möding <ceph-users@ceph.io>:
Eugen Block <ceph-users@ceph.io> writes:
Hi,
I just tried to deploy the v21 RC1 as a VM but failed with:
Fatal glibc error: CPU does not support x86-64-v3
Are you using Rocky 10 for that?
Not that the error comes from glibc. The support for anything older has been dropped in RedHat 10, so I don’t think this is specific to Ceph: https://access.redhat.com/solutions/7122807
-- Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Sep 30, 2026 at 4:30 AM Eugen Block <ceph-users@ceph.io> wrote:
Hi,
I just tried to deploy the v21 RC1 as a VM but failed with:
Fatal glibc error: CPU does not support x86-64-v3
I am aware that the release notes will contain a note about this:
https://github.com/ceph/ceph/pull/70240
But I'm wondering if it's really necessary to drop v2 support (already).
This is caused by adopting the Rocky 10 container as the base image (and follows upstream RHEL 10). This is not something Ceph did. It is already slated to be in the release notes for Umbrella. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Tue, Sep 29, 2026 at 8:40 AM Patrick Donnelly <ceph-users@ceph.io> wrote:
This is the second release candidate for Umbrella. All community feedback is welcome.
Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie)
Is there a reason we aren't building Umbrella for the current Ubuntu LTS release (resolute/26.04) that came out 6 months ago? My home cluster is running the latest Tentacle release (20.2.4) and last night I tried to mount my CephFS filesystem on a new Ubuntu 26.04.1 VM, but the ceph-common in resolute (20.2.0-0ubuntu2.1) doesn't understand the new aes256k key type. Normally I would switch to the upstream Ceph repositories to fix this, but there aren't any 26.04/resolute packages for Tentacle either. This kind of makes sense since 26.04/resolute wasn't out when tentacle was released, but it seems like we should be building 26.04/resolute packages for Umbrella. For anyone else that finds this message, I was able to workaround this problem by mounting the filesystem using only the kernel with: mount -t ceph 192.168.0.23:6789,192.168.0.25:6789,192.168.0.26:6789:/ /mnt/cephfs -o name=media,rbytes,secret=$(cat /etc/ceph/ceph.client.media.secret) Thanks, Bryan
On Thu, Oct 8, 2026 at 7:20 PM Bryan Stillwell <ceph-users@ceph.io> wrote:
On Tue, Sep 29, 2026 at 8:40 AM Patrick Donnelly <ceph-users@ceph.io> wrote:
This is the second release candidate for Umbrella. All community feedback is welcome.
Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie)
Is there a reason we aren't building Umbrella for the current Ubuntu LTS release (resolute/26.04) that came out 6 months ago?
My home cluster is running the latest Tentacle release (20.2.4) and last night I tried to mount my CephFS filesystem on a new Ubuntu 26.04.1 VM, but the ceph-common in resolute (20.2.0-0ubuntu2.1) doesn't understand the new aes256k key type. Normally I would switch to the upstream Ceph repositories to fix this, but there aren't any 26.04/resolute packages for Tentacle either. This kind of makes sense since 26.04/resolute wasn't out when tentacle was released, but it seems like we should be building 26.04/resolute packages for Umbrella.
For anyone else that finds this message, I was able to workaround this problem by mounting the filesystem using only the kernel with:
mount -t ceph 192.168.0.23:6789,192.168.0.25:6789,192.168.0.26:6789:/ /mnt/cephfs -o name=media,rbytes,secret=$(cat /etc/ceph/ceph.client.media.secret)
The simple answer is that it wasn't planned for Umbrella. Umbrella is already delayed enough from various circumstances that adding resolute support is simply not going to happen. Planned support is here: https://docs.ceph.com/en/latest/start/os-recommendations/#platforms -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 10/9/26 01:19, Bryan Stillwell wrote:
Is there a reason we aren't building Umbrella for the current Ubuntu LTS release (resolute/26.04) that came out 6 months ago?
My home cluster is running the latest Tentacle release (20.2.4) and last night I tried to mount my CephFS filesystem on a new Ubuntu 26.04.1 VM, but the ceph-common in resolute (20.2.0-0ubuntu2.1) doesn't understand the new aes256k key type. Normally I would switch to the upstream Ceph repositories to fix this, but there aren't any 26.04/resolute packages for Tentacle either. This kind of makes sense since 26.04/resolute wasn't out when tentacle was released, but it seems like we should be building 26.04/resolute packages for Umbrella.
I've hit a similar problem on one of our squid clusters with mixed ceph+libvirt-kvm setup. I've updated from Debian bookworm to trixie - then found out that my installed ceph-common and ceph-librbd packages are not available anymore because the official squid repo don't provide trixie packages. Long story short: I've created a build environment and published various deb packages, covering bookworm -> forky & jammy -> resolute for squid, tentacle and umbrella (pre-release). Repo & Instructions: https://github.com/cytrinox/ceph-apt Only amd64 is available so far, but I'm working on arm builds, too. It's pretty fresh work, but if you want to try it out, I'm happy about feedback. -- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher
participants (5)
-
Bryan Stillwell
-
Daniel Vogelbacher
-
Eugen Block
-
Patrick Donnelly
-
Stefan Möding