- ceph-users - lists.ceph.io

Persistent “PGs not scrubbed in time” on Reef 18.2.7 cluster — seeking tuning guidance
by Zachary Wright-Faust 30 Nov '25

30 Nov '25

Hi all, I’m running a 10-node Ceph cluster (Reef 18.2.7) that has been having persistent issues keeping PG scrubs up to date as we migrate workloads from our previous production cluster. This environment provides CephFS only. The main CephFS data pool is an EC 4+2 pool backed by HDD OSDs, and the metadata pool is a REP 3 pool on SSD OSDs. The hardware is relatively old (mostly Dell PowerEdge R630s with SAS-2 attached JBODs). This cluster was deployed as an interim platform while we rebuild our production environment following unrecoverable MDS corruption. The scrub backlog started to accumulate as soon as we began migrating data onto this interim cluster, and although I’ve been able to mitigate it at times, it has been growing again now that the majority of our clients have shifted over to it. Below is the current cluster status: > cluster: > id: 150735d4-4449-11f0-a328-a0369ff049b4 > health: HEALTH_WARN > 126 pgs not scrubbed in time > > services: > mon: 5 daemons, quorum > sio-nautiloid01,sio-nautiloid02,sio-nautiloid03,sio-nautiloid07,sio-nautiloid06 > (age 4w) > mgr: sio-nautiloid02.jbayyi(active, since 5h), standbys: > sio-nautiloid01.ssmngh > mds: 1/1 daemons up, 1 standby > osd: 248 osds: 248 up (since 4w), 248 in (since 9w) > > data: > volumes: 1/1 healthy > pools: 6 pools, 8417 pgs > objects: 1.04G objects, 1.3 PiB > usage: 1.9 PiB used, 1.7 PiB / 3.6 PiB avail > pgs: 6913 active+clean > 669 active+clean+scrubbing+deep > 408 active+clean+scrubbing > 405 active+clean+snaptrim_wait > 22 active+clean+snaptrim > > io: > client: 37 MiB/s rd, 4.3 MiB/s wr, 1.11k op/s rd, 53 op/s wr > > Current scrub-related configuration: mgr advanced > osd_deep_scrub_interval 5184000.000000 > > mgr advanced > osd_scrub_max_interval 2419200.000000 > > mgr advanced > osd_scrub_min_interval 1209600.000000 > > osd advanced > osd_deep_scrub_interval 4838400.000000 > > osd advanced > osd_deep_scrub_randomize_ratio 0.150000 > > osd advanced > osd_max_scrubs 3 > > osd advanced > osd_scrub_chunk_max 25 > > osd advanced > osd_scrub_load_threshold 0.900000 > > osd advanced > osd_scrub_max_interval 2419200.000000 > > osd advanced > osd_scrub_min_interval 1209600.000000 > > osd advanced > osd_scrub_priority 10 > > osd advanced > osd_scrub_sleep 0.100000 > These intervals mirror what we had configured on our previous production cluster (Quincy 17.2.7). I introduced osd_scrub_sleep after observing high commit and apply latency in ceph osd perf. I’ve also experimented with higher scrub priority and a higher load threshold, but the number of overdue scrubs continues to grow. At this point I’m looking for guidance on: - Additional tuning options that might help scrub throughput - Whether my current interval choices are counterproductive - How to identify the primary bottlenecks (OSD load, disk latency, stride size, concurrency, etc.) Any insights would be greatly appreciated. Thanks, Zach

1 0

Re: cephadm without containers
by Anthony D'Atri 29 Nov '25

29 Nov '25

> Thus displacing the problem from developers to the admins as now admins have to maintain the whole container support environment, including the "slightly varying library packages etc." This whole exercise is pointless as Ceph cluster storage nodes are natively dedicated to Ceph There are LOTS of converged clusters, notably Proxmox and Rook within K8s. > thus the bare-metal machine is the real Ceph container. Just install the specific OS supported by Ceph Fleets don’t care for that approach. They don’t want to have to maintain automation and package and provisioning systems for multiple OSes, multiply the infosec burden, etc. With Podman and log-to-file, in addition to the cephadm orchestrator, one can still manage individual daemons with systemctl. > and you've got your Ceph "container" made out of metal. > > You can make an argument that monitor and/or manager daemons could run in containers to allow for migration to alternative nodes. Indeed, this is a big advantage. > Properly sized Ceph clusters come with a number of dedicated monitor nodes anyway In the Luminous days I felt that way, since mons were subtle and quick to anger. I liked to decouple them from OSDs, which were more likely to need node reboots. I no longer due. I’ve had a pre-cephadm cluster with 5x mon/RGW nodes, two of which broke, and it took me *18 months* to convince the TAC to fix them. The cluster was one more failure from an outage, where with a converged cluster the full quorum would be maintained automagically. Defining the service spec to abstract based on a “mon” host label, and judiciously applying that label to one or at most two nodes per rack / failure domain maintains prudent anti-affinity. > therefore migration usually means add a new monitor node to the cluster and remove the old one, thus making the monitor containers point moot, too. Adding a new node to the cluster is far from trivial when the DC is overseas and unstaffed. It would also mean having a warm / hot spare node, which with your reasoning would be a different spec than the OSD nodes. The fewer chassis types in a DC the better, from an interchangeability, servicing, and sparing perspective. And, yes, I’ve run fleets of as many as 15000 nodes. ymmv. — aad

1 0

[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag
by Reto Gysi 29 Nov '25

29 Nov '25

Hi I tried to enable the new ec optimisation flag on one of the erasure encoded pools for rbd on my recently upgraded to Tentacle 20.2.0 version ceph cluster. ceph osd pool set rbd_ecpool allow_ec_optimizations true At first everything seemed normal. After a while when I went back to the computer I saw that osd.1 and osd.3 were always crashing. After a reboot I then saw errors and warnings about inconsistent pg's. OSD_SCRUB_ERRORS: 116116 scrub errors OSD_TOO_MANY_REPAIRS: Too many repaired reads on 4 OSDs PG_DAMAGED: Possible data damage: 3 pgs inconsistent I tried to run a manual ceph pg repair <pgid> on the affected pg's. While the affected pg's disappeared from the inconsistent pg list after a deep-scrub+repair new pgid appeared on the incosistent pg list and the amount of scrub errors increased. When I run a manual repair on a pgid, I see a lot of errors in the pg's primary osd log like: - 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : candidate size 1400832 info size 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : candidate size 1400832 info siz e 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : size 1400832 != size 1396736 f rom auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39(237963'11927123 client.40141308.0:1785630 dirty s 4194304 uv 11910481 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : size 1400832 != size 1396736 from auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39(237963'11927123 client.40141308.0:1785630 dirty s 4194304 uv 11910481 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : candidate size 1400832 info size 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : candidate size 1400832 info siz e 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : size 1400832 != size 1396736 f rom auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45(238043'11948425 client.40186030.0:433080 dirty s 4194304 uv 11933347 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : size 1400832 != size 1396736 from auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45(238043'11948425 client.40186030.0:433080 dirty s 4194304 uv 11933347 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : candidate size 1400832 info size 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : candidate size 1400832 info siz e 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : size 1400832 != size 1396736 f rom auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d(238319'11972769 client.40186030.0:1294524 dirty|data_digest s 4194304 uv 11955052 dd 2cef26e9 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : size 1400832 != size 1396736 The size mismatch always seem to be the same: 1400832 != size 1396736 And looking at the rbd namespace like example rbd_data.7.6f5c183c7dc42b It looks like so far that images that have been in use/mounted during enablement of allow_ec_pool_optimization are affected. So my questions are: 1. Are these errors expected/explainable? 2. Will a manual pg repair on all the pools pgids eventually fix the problem? 3. Is there any chance that data has been lost/inconsistent after pg repair on all the pools pgids? Thanks & best regards, Reto

1 1

v20.2.0 Tentacle released
by Yuri Weinstein 28 Nov '25

28 Nov '25

We're very happy to announce the first stable release of the Tentacle series. We express our gratitude to all members of the Ceph community who contributed by proposing pull requests, testing this release, providing feedback, and offering valuable suggestions. We would like to especially thank some of our community members who helped us test upgrades for the pre-released version of 20.2.0. Your feedback and effort are greatly appreciated! Note from the Ceph Infrastructure Team: Part of our Standard Operating Procedure for Ceph releases is typically to upgrade the Ceph cluster in the Ceph lab to the latest version before announcing the release. This is a production cluster, which we colloquially call the Long Running Cluster (LRC), as it has existed and been updated for the past decade or even longer. For this release, we have decided to hold off on updating the LRC until we have migrated some of the labs from Red Hat data centers to IBM. We plan to update the LRC to Tentacle in the next calendar year. We believe this decision should not be viewed as a reflection of our confidence in the project, but rather as an attempt to focus on other priorities before the migration. ---Highlights--- CephFS * Directories may now be configured with case-insensitive or normalized directory entry names. * Modifying the FS setting variable ``max_mds`` when a cluster is unhealthy now requires users to pass the confirmation flag (``--yes-i-really-mean-it``). * ``EOPNOTSUPP`` (Operation not supported) is now returned by the CephFS FUSE client for ``fallocate`` for the default case (i.e. ``mode == 0``). Crimson * SeaStore Tech Preview: SeaStore object store is now deployable alongside Crimson-OSD, mainly for early testing and experimentation. Community feedback is encouraged to help with future improvements. Dashboard * Support has been added for NVMe/TCP gateway groups and multiple namespaces, multi-cluster management, OAuth 2.0 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication. Integrated SMB support * Ceph clusters now offer an SMB Manager module that works like the existing NFS subsystem. The new SMB support allows the Ceph cluster to automatically create Samba-backed SMB file shares connected to CephFS. The ``smb`` module can configure both basic Active Directory domain or standalone user authentication. The Ceph cluster can host one or more virtual SMB clusters which can be truly clustered using Samba's CTDB technology. The ``smb`` module requires a cephadm-enabled Ceph cluster and deploys container images provided by the ``samba-container`` project. The Ceph dashboard can be used to configure SMB clusters and shares. A new ``cephfs-proxy`` daemon is automatically deployed to improve scalability and memory usage when connecting Samba to CephFS. MGR * Users now have the ability to force-disable always-on modules. * The ``restful`` and ``zabbix`` modules (deprecated since 2020) have been officially removed. RADOS * FastEC: Long-anticipated performance and space amplification optimizations are added for erasure-coded pools. * BlueStore: Improved compression and a new, faster WAL (write-ahead-log). * Data Availability Score: Users can now track a data availability score for each pool in their cluster. * OMAP: All components have been switched to the faster OMAP iteration interface, which improves RGW bucket listing and scrub operations. RBD * New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats. * There is now support for RBD namespace remapping while mirroring between Ceph clusters. * Several commands related to group and group snap info were added or improved, and ``rbd device map`` command now defaults to ``msgr2``. RGW * Added support for S3 ``GetObjectAttributes``. * For compatibility with AWS S3, ``LastModified`` timestamps are now truncated to the second. Note that during upgrade, users may observe these timestamps moving backwards as a result. * Bucket resharding now does most of its processing before it starts to block write operations. This should significantly reduce the client-visible impact of resharding on large buckets. * The User Account feature introduced in Squid provides first-class support for IAM APIs and policy. Our preliminary STS support was based on tenants, and exposed some IAM APIs to admins only. This tenant-level IAM functionality is now deprecated in favor of accounts. While we'll continue to support the tenant feature itself for namespace isolation, the following features will be removed no sooner than the V release: - Tenant-level IAM APIs including CreateRole, PutRolePolicy and PutUserPolicy, - Use of tenant names instead of accounts in IAM policy documents, - Interpretation of IAM policy without cross-account policy evaluation, - S3 API support for cross-tenant names such as `Bucket='tenant:bucketname'` - STS Lite and `sts:GetSessionToken`. We encourage you to read the full release notes at https://ceph.io/en/news/blog/2025/v20-2-0-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_20.2.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 69f84cc2651aa259a15bc192ddaabd3baba07489 Did you know? Every Ceph release is built and tested on resources funded directly by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now https://ceph.io/en/foundation/.

6 7

cephadm cluster - logging redudancy
by Francesco Di Nucci 27 Nov '25

27 Nov '25

Hi all, I am deploying a new cluster with cephadm, I used "--with-centralized-logging" option at bootstrap and it set up graphana, loki and prometheus with count: 1 as placement. Now I'd like to increase redundancy/availability of the logging part, so that even if the host that has one of these services goes down the logging keeps on working; are there any guidelines/suggestions on this? Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

1 0

Meaning of RGW sync status full sync: objects completed
by Adam Prycki 26 Nov '25

26 Nov '25

Hello, I have a question about rgw synchronization mechanic. When we are trying to initialize full bucket sync on existing bucket, or when try to synchonize delete bucket we see "full sync" in bucket sync status on our clusters. For example: realm b6ca44da-921e-42da-b377-80e9fae83b61 (s3-TEST.kmd4.pl) zonegroup c41b36c6-2031-407b-bab4-36cd0b3e56da (poznan-1) zone 91a17d0c-84a0-4706-a40d-1b6df1996a10 (zone-poznan-1-kielman) bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) current time 2025-11-24T13:11:48Z source zone 8e5c54a9-7eb2-4521-b327-081a1a647205 (zone-poznan-1-pcss-bst) source bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) full sync: 15221490 objects completed In this case we have deleted bucket with broken synchronization on the secondary zone. After RGW was started up again synchronization should have tarted from zero. Bucket 1d2848a2-9ca7-402c-9976-a6dc775547bb has 1163263 10MiB s3 objects. bucket sync status is right now at 15221490 and still increasing. That's 13x bigger number than object count in this bucket. In case of some other bucket we saw that "full sync: objects completed" number grew to about 20x s3 object count in the bucket. One bucket with 10 000 000 objects has running full sync has "full sync: 605952706 objects completed" Can someone explain what this number means and why it gets so big? How big it's supposed to get? Can we predict how much of the synchronization is "done" from this number? Best regards Adam Prycki

1 0

Ceph Developer Meetup next week
by Anthony Middleton 26 Nov '25

26 Nov '25

Hey Ceph friends, Please join us for the* Ceph Developer Meetup next week on December 3 at 11 am EST / 8 am PDT*. The agenda is in the meetup announcement. Please *add your topics* in advance. *RSVP *here: https://www.meetup.com/ceph-user-group/events/312181316/?utm_medium=referra…

1 0

RGW Signature does not match
by GLE, Vivien 26 Nov '25

26 Nov '25

Hi, I'm trying to setup a VM RGW in a cephadm managed test cluster. Everything is working fine except 2 things: - RGW didn't show up in the object section of the ceph dashboard and this message shows up The Object Gateway Service is not configured Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx000002bfe4c78d8a' b'4b666-006926c06c-42063466-<zone>","HostId":"42063466-<zone>-<groupzone>"}') The daemon is active in ceph -s and other functionalities are working fine I read that it might be a date error, as my hosts are on CET, but the containers are set to UTC is there a way to correct that ? - The second thing is that I need to radosgw-admin --name client.ceph-rgw (which is working fine) instead of only radosgw-admin otherwise it seems that radosgw-admin didn't find my ceph.conf and output is looking like this : -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin (2) No such file or directory failed to fetch mon config (--no-mon-config to skip) If I rename my client.keyring file this output is still showing up => failed to fetch mon config (--no-mon-config to skip) cat /etc/ceph/ceph.conf [global] fsid = <fsid> mon_host = [v2:192.168.219.12:3300/0,v1:192.168.219.12:6789/0] [v2:192.168.219.14:3300/0,v1:192.168.219.14:6789/0] [v2:192.168.219.15:3300/0,v1:192.168.219.15:6789/0] auth_client_required = cephx auth_service_required = cephx auth_supported = cephx [client.ceph-rgw] host = ceph-rgw rgw_dns_name = ceph-rgw keyring = /etc/ceph/ceph.client.ceph-rgw.keyring rgw_frontends = "beast endpoint=192.168.219.19:8080 ssl_endpoint=192.168.219.19:8443 ssl_certificate=/etc/ceph/certs/ceph-rgw.pem" log_file = /var/log/ceph/ceph-rgw-ceph-rgw.log rgw_thread_pool_size = 512 cat /etc/ceph/ceph.client.ceph-rgw.keyring [client.ceph-rgw] key = <key> ceph-radosgw(a)ceph-rgw.service - Ceph rados gateway Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; preset: enabled) Active: active (running) since Tue 2025-11-25 16:33:08 CET; 19h ago Main PID: 90607 (radosgw) Tasks: 621 Memory: 114.6M (peak: 115.1M) CPU: 2min 2.665s CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw(a)ceph-rgw.service └─90607 /usr/bin/radosgw -f --cluster ceph --name client.ceph-rgw --setuser ceph --setgroup ceph Thanks Vivien

1 0

Changing the crush rule of an existing pool
by Huseyin Cotuk 26 Nov '25

26 Nov '25

Hello Cephers, I just want to get confirmation on two things. I'm going to change the crush rule of an existing pool so that it uses SSDs instead of HDDs. 1. Will changing the rule cause any problems with the current workload? 2. Does Ceph manage the migration of data currently in use when the existing pool's rule is changed? If so, does it also delete the data on the old disks after the migration? Thanks in advance. BR, Huseyin Cotuk hcotuk(a)gmail.com

1 0

New Cluster Ceph
by GLE, Vivien 25 Nov '25

25 Nov '25

Hi, We plan to buy hardware for a new cluster ceph and would like some approbation on what we choose. 6 nodes DELL 6715 + 6 Powervault MD2412 enclosure For each node => 1 CPU AMD EPYC 9475F 3,65 GHz, 48C/96T, 256M Cache (400 W) RAM 16Gox16 = 256Go 4x 100GB NVIDIA MELLANOX 100GB HBA465e (externe, 22,5GB/s) 2 NVMe mixed 6,4To DBWAL 6 NVMe mixed 6,4To 5 NVMe read 15,36To for each enclosure => 12 HDD 20To HDD will be a replica 3 pool with the dbwal NVME mixed and read will be 2 differents pools (we will test replica 3 and EC to see which performance/storage efficency satisfy us the most) This cluster will be mostly use to store block (VM proxmox/kubernetes) and S3. The crushmap will be like so : 3 rooms with 2 nodes per room. So the point of failure can be for replica 3 => rooms and if we use EC 4+2 => host We saw that the thread needs for NVMe OSD are very expensive, does this CPU good enough to carry them ? MON and MGR will be spread across the cluster and RGW on virtual machine Thanks for your answer ! Vivien

3 5