- ceph-users - lists.ceph.io

Tentacle: rgw zonegroup modify failed: Module 'rgw' is not enabled
by Eugen Block 15 Oct '26

15 Oct '26

Hi all, today we stumbled across an rgw issue. This issue [0] (and PR [1]) seems to address it but to me it looks like not all cases are covered. - cephadm deployment - Ceph v20.2.4 - Ubuntu hosts (22.04) - no multi-site For a specific reason we gave a customer the instructions to pre-create a realm, zonegroup and zone before applying the rgw service. It worked well in previous versions (Squid < 19.2.6, haven't tested other versions yet). Now on 20.2.4 we try the same procedure (create realm, zonegroup and zone beforehand) with this rgw spec: service_type: rgw service_id: s3 service_name: s3 placement: label: rgw spec: ssl: true rgw_frontend_port: 7480 generate_cert: true wildcard_enabled: true zonegroup_hostnames: - s3.example.com What we didn't have in previous tests was the requirement to use a generated certificate (generate_cert). And we currently don't have our own certificates in this case (I know, we could just generate some and probably will). Anyway, applying this spec results in working RGW daemons (they successfully show ListAllMyBuckets) but this Ceph warning can't be cleared: [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): rgw.s3 rgw.s3: rgw zonegroup modify failed: Module 'rgw' is not enabled (required by command 'rgw zonegroup modify'): use `ceph mgr module enable rgw` to enable it retval: -95 Is the mentioned fix incomplete? Or how could we utilize generate_cert without issuing a health warning? I guess the simple workaround would be to create our own certs and use the same procedure as before. Thanks, Eugen [0] https://tracker.ceph.com/issues/75264 [1] https://github.com/ceph/ceph/pull/67598

1 0

Octopus PG not going active
by Michael Thomas 15 Oct '26

15 Oct '26

The background: We are preparing to migrate our Octopus cluster to Quincy (as a first step on the way to Tentacle). The primary use of the cluster is cephfs, with separate data pools on HDD and SSD. During the migration we accidentally reformatted 6 OSDs in a SSD 6+2 EC pool. Fortunately, all PGs recovered, except for one. Three of the lost OSDs contributed to this PG, so we are accepting that it is gone and any files that were placed on this PG are lost. We identified the lost files with cephfs-data-scan and found that all can be recovered from our non-ceph backup. The problem: Now that we've determined that all lost data can be recovered from backup, we'd like to get the PG active again so we can start this recovery. Unfortunately, the PG is stuck in the incomplete state. We've tried a few things to get it going again, some of which may have made matters worse: ceph osd reweight $osd 0.0 ceph osd lost $osd This left the PG blocked from peering due to 'history_les_bound'. After about an hour we tried: ceph tell $osd config set osd_find_best_info_ignore_history_les 1 This didn't get it peering again, so a couple of hours later we tried forcing a repeer: ceph pg repeer 8.7b That didn't get it going either. Over the next couple of days we then tried a combination of repeer and restarting OSDs. During this we tried doing a force-create on the pg: ceph osd force-create-pg 8.7b However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions? --Mike

3 5

Simultaneous increasing numbers of PGs and OSDs
by Loïc Tortay 14 Oct '26

14 Oct '26

Hello, For one of our CephFS clusters, we need to increase the number of PGs in the main data pool from 8192 to 16384. We have also received new nodes to increase the capacity of this cluster, an almost 50% increase in capacity/number of nodes. We plan to use "upmap-remapped.py" as described in https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ and https://docs.clyso.com/docs/kb/rados/pg-splitting-with-upmap-remapped/ Is is safe/OK to combine the two operations ? That is: - disable rebalance/the balancer - "apply" the OSDs YAML, the new hosts are already added to the cluster - run upmap-remapped.py a few times until the number of misplaced objects is low - increase the number of PGs in the pool - run upmap-remapped.py until "pg_num" has reached the new value - re-enable rebalance/the balancer Or is it safer/recommended to do this in two distinct operations ? Loïc. -- | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

2 1

Autoscaler second-guessing itself on number of PGs required for pool
by Martin Andersen 12 Oct '26

12 Oct '26

Hi all, We have a 19.2.4 cluster with a Cinder pool (autoscale=warn) that keeps asking for num_pgs to be adjusted. This was recently set to 2048 (from 4096), and over this weekend it wanted to increase it again to 4096 - this was also done. However, before the rebalance has even finished, it again informs that the num_pgs value is now too high, and asks for it to be reduced. This is within the span of under one hour. ceph health detail HEALTH_WARN 1 pools have too few placement groups [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups Pool cinder-volumes has 2048 placement groups, should have 4096 ceph osd pool set cinder-volumes pg_num 4096 set pool 4 pg_num to 4096 ceph health detail HEALTH_WARN 1 pools have too many placement groups [WRN] POOL_TOO_MANY_PGS: 1 pools have too many placement groups Pool cinder-volumes has 4096 placement groups, should have 2048 Anyone else experiencing this behavior? We can obviously set the autoscaler to off for this pool, but there seems to either be a misconfiguration or errant default setting that should be addressed. This is a long-running pool that has had no recent config changes to pg assignment. Also, for reference - ceph config dump | grep pg global advanced mon_max_pg_per_osd 250 global advanced mon_pg_warn_max_object_skew 11.000000 global advanced mon_pg_warn_min_per_osd 10 global advanced mon_target_pg_per_osd 101 mgr advanced mgr/pg_autoscaler/log_level info --Martin

2 2

v21.1.2 Umbrella RC2 released
by Yuri Weinstein 12 Oct '26

12 Oct '26

This is the third release candidate for Umbrella. All community feedback is welcome. It is our intention that this is the final release candidate for Umbrella. If no blockers are identified, we plan to release Umbrella around October 21st. Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie) Containers are built on a Rocky 10 base image, available at: https://quay.io/repository/ceph/ceph?tab=tags Please note: the full draft release notes for v21.2.0 are available at https://github.com/ceph/ceph/pull/70240 Compatibility requirements for this release: - CPU: Umbrella containers are based on el10 and require a CPU supporting x86-64-v3. - Python: Python 3.9+ is required on all host operating systems for cephadm. Notable changes from 21.1.1, the second release candidate: Security / CephX: - auth/cephx: harden service ticket verification (pr#72126) - cephadm: rotate OSD/MDS keyrings at upgrade time and based on health warnings (pr#72240) - rgw: bind GCM object keys to the bucket marker (pr#71038) - mgr/rbd_support: make schedule phase offset calculation FIPS compliant (pr#71167) RADOS / OSD: - OSDMap::calc_pg_maps: fix balancing for some important corner cases (pr#71916) - osd/scheduler: classify subOp reads by op priority for mClock, and bound high-priority drain to prevent starvation (pr#69761, pr#72179) - mgr/osd/mon: remove pg_upmap_primary mappings for PG merge targets (pr#71050) - osd/PeeringState: don't clear the PG rebuild latch on same-primary interval restart (pr#70990) - Reverted the ProtocolV2 connection speedup and netsplit grace-period changes (pr#71957) RGW: - Compress/decompress objects during lifecycle transitions (pr#70687) - Multisite: redirect to the endpoint of the bucket's zonegroup (pr#71524) - Honor CopyObject tagging-directive (pr#70681); read user usage log in S3 GetUsage (pr#71102) - S3: reject an empty x-amz-server-side-encryption header (pr#71536); allow completing a multipart upload without SSE-C headers (pr#71841) - Fix conditional writes on versioning-suspended buckets (pr#72048) - Fix use-after-free in shard_io async_reads (pr#71136); infinite loop in role-list pagination (pr#71312) - radosgw-admin: 'account rm --purge-data' removes OIDC by account id (pr#71532); 'bi list' honors --max-entries (pr#71043) cephadm / Orchestrator: - certmgr: support fullchain PEM (pr#71513) - Allow removing OSDs stuck in error state (pr#71729); zap raw OSD devices on rm-cluster (pr#71730); "ok-to-stop" check for OSD redeploy/reconfig (pr#72032) - Fix staggered upgrade (pr#69724); hardware monitoring shows "NO DATA" for missing categories (pr#71725); node-proxy no longer crashes on empty Redfish logout (pr#71727) Dashboard: - Angular 20 upgrade and HTML linter fixes (pr#70871) - Skip RGW daemons whose frontend is unreachable (pr#72047) CephFS: - cephfs-mirror bug-fix backports (pr#70957) - Reverted "mds: enable osdc op throttles" (pr#72186) - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.2.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: b6f98ed07f44274e393138ffbae442b7a3f7ec84 - Distribution matrix: https://docs.ceph.com/en/latest/start/os-recommendations/

1 0

Ceph Community Survey 2026
by Anthony Middleton 12 Oct '26

12 Oct '26

Hey Ceph friends, It's Cephalopod Awareness Week, and we're taking a cue from the octopus. The Ceph Foundation needs to hear from every arm of the community. The 2026 Ceph Community Survey is now open and covers: - Foundation priorities for the next three years - What would make it easier to contribute to Ceph - Cephalocon and Ceph Days - How you use AI in your Ceph work An octopus’s eight arms sense and act independently but move it in one direction. The Ceph community works similarly: developers, operators, vendors, and users contribute distinct strengths, advancing the project together. Your answers shape how the Foundation allocates its resources, including the events we run and the tools we build. It takes about 10 minutes. Take the survey: https://t.ly/Ceph-Survey-2026 Open until: November 13th, 2026 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph> Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

v19.2.7 Squid released
by Yuri Weinstein 12 Oct '26

12 Oct '26

This is the seventh backport release in the Squid series. We recommend that all deployments update to this release. ## Notable Changes CephFS ------ - Fixed a crash caused by an invalid iterator in `_readdir_cache_cb`. - Fixed a deadlock in the client when the OSD is full. - Fixed client I/O stalling due to a buffer list exceeding `INT_MAX`. - Fixed client cache trimming removing dentries still attached to an open directory inode. - Fixed `join_fscid` being incorrectly reset for an active MDS during filesystem removal. - MDS no longer path-traverses a damaged dirfrag. - Fixed rank 0 being marked damaged if stopping fails after log flush and trim. - Fixed an incorrect snap check for directories with parent snaps. - Added a `ceph.dir.subvolume` get vxattr. - `mgr/snap_schedule` now handles volume deletion correctly. - Fixed `fs status` JSON output. - `mgr/vol` now handles a failed non-atomic operation correctly. RADOS ----- - Removed `pg_upmap_primary` mappings for PG merge targets. - Added `pg-upmap-primary` support to `clean_pg_upmaps`. - Fixed a Neorados `CephContext` leak. - Fixed a BlueStore onode-cache issue causing stalls. - Stretch mode: added a max bucket weight diff threshold. - `mgr` now handles `SIGTERM`/`SIGINT` in standby to avoid a spurious `CEPHADM_FAILED_DAEMON`. RBD (RADOS Block Device) ------------------------ - It's possible to specify the source cluster's `mon_host` and `key` for `native` format migration via the migration spec now. This eliminates the dependency on a `<cluster-name>.conf` file in a known location, which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - Fixed a use-after-free in trash purge on image open error. - Fixed a use-after-free releasing object map locks during deep copy. - Fixed memory leaks in PWL cache discard operations. - Fixed a race between `dispatch_deferred_writes()` and `~AbstractWriteLog()` in the SSD PWL cache. - Fixed strict weak ordering in `rbd-mirror`'s `PeerSpec::operator<`. - `rbd-mirror` now prunes obsolete primary mirror snapshots after relocation. - Fixed `mgr/rbd_support` perf iostat pool-spec filtering by data pool. RGW (RADOS Gateway) ------------------- - Fixed conditional Delete, MultiDelete, and Put handling. - `x-amz-content-sha256` is no longer required in signed headers when verifying client auth. - The AWS v4 signer now always includes the `x-amz-content-sha256` header when signing. - `rgw_gc_max_objs` is now startup-only. - Added a warning against changing `rgw_lc_max_objs` once lifecycle is in use. - `x-amz-expiration` is now reported only for the current object version. - Multisite: bucket `obj_lock` settings are now synced. - Fixed a race between bucket reshard and bucket index update. - Fixed a crash on realm reload. - Fixed a frontend crash in `abort_early()` on client disconnect. - Fixed the lifecycle transition of encrypted multipart objects. - Content-Type is no longer always required to be present in the signature. - Fixed a shutdown hang caused by a race between the async processor and multisite sync threads. - Fixed cloud-tier multipart resume starting at part number 0. - `complete-multipart` now returns an ETag header on success. - Fixed a crash when an admin tries to assume a nonexistent role. ceph-volume ----------- - Fixed raw activation when the device path is stale. - Fixed OSD re-deployment issues with disk selection filters and DB devices. - Detects rotational media under dm-crypt for workqueue bypass. - Includes LVM mapper devices in `get_devices()`. - Retries `lvs` after an empty result or a "devices file is missing" error. - Skips virtual CD-ROM devices during inventory. Dashboard --------- - Fixed a pool profile issue. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v19-2-7-squid-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 9ec952b9f9668e9fa97e29a4c2b51e951674df2a

1 0

Ceph Days Chicago CFP Reminder
by Anthony Middleton 09 Oct '26

09 Oct '26

Hey Ceph Users, Just one more week to submit your talk(s) for Ceph Days Chicago 2026! Join us in Chicago to share your Ceph experience, technical insights, or community story with developers, operators, and enthusiasts at this in-person event. CFP deadline: Friday, Oct 15 Event date: Monday, Nov 16 Submit your talk here: https://t.ly/CDChicago26 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

v21.1.1 Umbrella (RC1) released
by Patrick Donnelly 09 Oct '26

09 Oct '26

This is the second release candidate for Umbrella. All community feedback is welcome. Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie) Containers are built on a Rocky 10 base image, available at: https://quay.io/repository/ceph/ceph?tab=tags Please note: the full draft release notes for v21.2.0 are available at https://github.com/ceph/ceph/pull/70240 Notable changes from 21.1.0, the first release candidate: - Fixes for CVE-2025-30156, CVE-2026-39944,CVE-2026-50152, and CVE-2026-54330 included. (Please see upgrade notes for v20.2.4/v19.2.6.) - MGR: The default values of ``mon_target_pg_per_osd`` and ``mon_max_pg_per_osd`` have been increased from 100 and 250 to 200 and 500, respectively. These values enhance the PG autoscaler's ability to calculate reasonable ``pg_num`` values for pools under its control. These in turn increase parallelism and performance, and improve the ``balancer`` module's ability to acheive uniform OSD utilization. [This concomitant PR](https://github.com/ceph/ceph/pull/67694) ensures that upgrades are not delayed or disrupted by PG splitting as a result of this change. Clusters with effective ``bluestore_osd_memory_target`` values smaller than the default 4GiB may wish to explicitly set the previous default values in the central config store before upgrading. This will reduce the potential for increased memory starvation. Clusters with SSDs split into more than two OSDs may also wish to pin their current values in advance, or redeploy those SSDs with at most two OSDs per. - RGW: In multisite deployments, zone endpoints configured in the zonegroup (e.g. ``https://zone-a.example.com``) can now be resolved to all IP addresses returned by DNS, with inter-zone traffic distributed across them using round-robin and per-IP health tracking. This enables DNS-based service discovery for inter-zone traffic without an external load balancer. This feature is disabled by default: to opt in, set ``rgw_rest_conn_connect_to_resolved_ips = true``. The per-IP retry-after-failure timeout is controlled by ``rgw_rest_conn_ip_fail_timeout_secs`` (default: 2 seconds). Deployments that work around libcurl's single-address behavior by repeating the same endpoint multiple times in zone configuration should review that setup before enabling, as round-robin distribution makes such duplication unnecessary. - RGW: The rgw_gc_max_deferred and rgw_gc_max_deferred_entries_size options have been removed, as they did not do anything. - Updated the default for rgw_gc_max_queue_size to account for the extra space from removing rgw_gc_max_deferred_entries_size. - RGW: The default value of ``rgw_thread_pool_size`` has been reduced from 512 to 128. Benchmarks showed that the lower thread count improves throughput and reduces latency on NVMe and HDD based environments. Users with specific workload requirements can tune this value; see also ``rgw_max_concurrent_requests``. - CephFS Mirroring: The ``fs mirror peer status`` admin socket command reports additional per-directory sync metrics (sync mode, throughput, crawl and data-sync queue timing, bytes/files progress, and ETA). Output is grouped under ``metrics/<mirrored-dir-path>/peer/<peer-uuid>``. For more information, see https://tracker.ceph.com/issues/73453 - CephFS Mirroring: Per-directory snapshot sync progress is exposed as labeled perf counters in the ``cephfs_mirror_directory`` group (``counter dump`` on the mirror daemon admin socket, exportable via ``ceph-exporter``). Counters mirror ``fs mirror peer status`` fields (directory state, current sync, last sync, and snap summary) and are labeled by peer UUID and mirrored directory path. See https://tracker.ceph.com/issues/73457 - RADOS: PG autoscaler allows overlapping roots. Each root receives a PG target based on its OSDs, with OSDs shared across multiple roots contributing proportionally less to each root's allocation. - CephFS Mirroring: Introduced snapshot checkpoints feature that allows users to mark specific snapshots as important milestones and track their replication status to remote sites. Checkpoints automatically detect if a snapshot has already been synced and provide visibility into disaster recovery readiness. The feature includes four new CLI commands: ``ceph fs snapshot mirror checkpoint add`` to mark a snapshot, ``ceph fs snapshot mirror checkpoint now`` to checkpoint the latest snapshot, ``ceph fs snapshot mirror checkpoint ls`` to list all checkpoints with their status (created/complete/failed), and ``ceph fs snapshot mirror checkpoint remove`` to remove a checkpoint. Checkpoint metadata is persistent across daemon restarts. This feature is useful for compliance auditing, application consistency verification, and SLA tracking. For more information, see https://tracker.ceph.com/issues/73454 - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.1.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: 815bc6b6f9b4e0fd98daed552ec8e7123a0a8f84 - Distribution matrix: https://docs.ceph.com/en/latest/start/os-recommendations/ -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

5 7

alertmanager connecting to secure mail server?
by listy 09 Oct '26

09 Oct '26

Hi guys. Is there a way to add an external CA so ceph could use secure mail? I get: ... [WRN] ALERTS_SMTP_ERROR: unable to send alert email [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:998) ... many thanks, L.

1 0