- ceph-users - lists.ceph.io

[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag
by Reto Gysi 29 Nov '25

29 Nov '25

Hi I tried to enable the new ec optimisation flag on one of the erasure encoded pools for rbd on my recently upgraded to Tentacle 20.2.0 version ceph cluster. ceph osd pool set rbd_ecpool allow_ec_optimizations true At first everything seemed normal. After a while when I went back to the computer I saw that osd.1 and osd.3 were always crashing. After a reboot I then saw errors and warnings about inconsistent pg's. OSD_SCRUB_ERRORS: 116116 scrub errors OSD_TOO_MANY_REPAIRS: Too many repaired reads on 4 OSDs PG_DAMAGED: Possible data damage: 3 pgs inconsistent I tried to run a manual ceph pg repair <pgid> on the affected pg's. While the affected pg's disappeared from the inconsistent pg list after a deep-scrub+repair new pgid appeared on the incosistent pg list and the amount of scrub errors increased. When I run a manual repair on a pgid, I see a lot of errors in the pg's primary osd log like: - 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : candidate size 1400832 info size 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : candidate size 1400832 info siz e 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : size 1400832 != size 1396736 f rom auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39(237963'11927123 client.40141308.0:1785630 dirty s 4194304 uv 11910481 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : size 1400832 != size 1396736 from auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39(237963'11927123 client.40141308.0:1785630 dirty s 4194304 uv 11910481 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : candidate size 1400832 info size 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : candidate size 1400832 info siz e 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : size 1400832 != size 1396736 f rom auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45(238043'11948425 client.40186030.0:433080 dirty s 4194304 uv 11933347 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : size 1400832 != size 1396736 from auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45(238043'11948425 client.40186030.0:433080 dirty s 4194304 uv 11933347 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : candidate size 1400832 info size 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : candidate size 1400832 info siz e 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : size 1400832 != size 1396736 f rom auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d(238319'11972769 client.40186030.0:1294524 dirty|data_digest s 4194304 uv 11955052 dd 2cef26e9 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : size 1400832 != size 1396736 The size mismatch always seem to be the same: 1400832 != size 1396736 And looking at the rbd namespace like example rbd_data.7.6f5c183c7dc42b It looks like so far that images that have been in use/mounted during enablement of allow_ec_pool_optimization are affected. So my questions are: 1. Are these errors expected/explainable? 2. Will a manual pg repair on all the pools pgids eventually fix the problem? 3. Is there any chance that data has been lost/inconsistent after pg repair on all the pools pgids? Thanks & best regards, Reto

1 1

v20.2.0 Tentacle released
by Yuri Weinstein 28 Nov '25

28 Nov '25

We're very happy to announce the first stable release of the Tentacle series. We express our gratitude to all members of the Ceph community who contributed by proposing pull requests, testing this release, providing feedback, and offering valuable suggestions. We would like to especially thank some of our community members who helped us test upgrades for the pre-released version of 20.2.0. Your feedback and effort are greatly appreciated! Note from the Ceph Infrastructure Team: Part of our Standard Operating Procedure for Ceph releases is typically to upgrade the Ceph cluster in the Ceph lab to the latest version before announcing the release. This is a production cluster, which we colloquially call the Long Running Cluster (LRC), as it has existed and been updated for the past decade or even longer. For this release, we have decided to hold off on updating the LRC until we have migrated some of the labs from Red Hat data centers to IBM. We plan to update the LRC to Tentacle in the next calendar year. We believe this decision should not be viewed as a reflection of our confidence in the project, but rather as an attempt to focus on other priorities before the migration. ---Highlights--- CephFS * Directories may now be configured with case-insensitive or normalized directory entry names. * Modifying the FS setting variable ``max_mds`` when a cluster is unhealthy now requires users to pass the confirmation flag (``--yes-i-really-mean-it``). * ``EOPNOTSUPP`` (Operation not supported) is now returned by the CephFS FUSE client for ``fallocate`` for the default case (i.e. ``mode == 0``). Crimson * SeaStore Tech Preview: SeaStore object store is now deployable alongside Crimson-OSD, mainly for early testing and experimentation. Community feedback is encouraged to help with future improvements. Dashboard * Support has been added for NVMe/TCP gateway groups and multiple namespaces, multi-cluster management, OAuth 2.0 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication. Integrated SMB support * Ceph clusters now offer an SMB Manager module that works like the existing NFS subsystem. The new SMB support allows the Ceph cluster to automatically create Samba-backed SMB file shares connected to CephFS. The ``smb`` module can configure both basic Active Directory domain or standalone user authentication. The Ceph cluster can host one or more virtual SMB clusters which can be truly clustered using Samba's CTDB technology. The ``smb`` module requires a cephadm-enabled Ceph cluster and deploys container images provided by the ``samba-container`` project. The Ceph dashboard can be used to configure SMB clusters and shares. A new ``cephfs-proxy`` daemon is automatically deployed to improve scalability and memory usage when connecting Samba to CephFS. MGR * Users now have the ability to force-disable always-on modules. * The ``restful`` and ``zabbix`` modules (deprecated since 2020) have been officially removed. RADOS * FastEC: Long-anticipated performance and space amplification optimizations are added for erasure-coded pools. * BlueStore: Improved compression and a new, faster WAL (write-ahead-log). * Data Availability Score: Users can now track a data availability score for each pool in their cluster. * OMAP: All components have been switched to the faster OMAP iteration interface, which improves RGW bucket listing and scrub operations. RBD * New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats. * There is now support for RBD namespace remapping while mirroring between Ceph clusters. * Several commands related to group and group snap info were added or improved, and ``rbd device map`` command now defaults to ``msgr2``. RGW * Added support for S3 ``GetObjectAttributes``. * For compatibility with AWS S3, ``LastModified`` timestamps are now truncated to the second. Note that during upgrade, users may observe these timestamps moving backwards as a result. * Bucket resharding now does most of its processing before it starts to block write operations. This should significantly reduce the client-visible impact of resharding on large buckets. * The User Account feature introduced in Squid provides first-class support for IAM APIs and policy. Our preliminary STS support was based on tenants, and exposed some IAM APIs to admins only. This tenant-level IAM functionality is now deprecated in favor of accounts. While we'll continue to support the tenant feature itself for namespace isolation, the following features will be removed no sooner than the V release: - Tenant-level IAM APIs including CreateRole, PutRolePolicy and PutUserPolicy, - Use of tenant names instead of accounts in IAM policy documents, - Interpretation of IAM policy without cross-account policy evaluation, - S3 API support for cross-tenant names such as `Bucket='tenant:bucketname'` - STS Lite and `sts:GetSessionToken`. We encourage you to read the full release notes at https://ceph.io/en/news/blog/2025/v20-2-0-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_20.2.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 69f84cc2651aa259a15bc192ddaabd3baba07489 Did you know? Every Ceph release is built and tested on resources funded directly by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now https://ceph.io/en/foundation/.

6 7

cephadm cluster - logging redudancy
by Francesco Di Nucci 27 Nov '25

27 Nov '25

Hi all, I am deploying a new cluster with cephadm, I used "--with-centralized-logging" option at bootstrap and it set up graphana, loki and prometheus with count: 1 as placement. Now I'd like to increase redundancy/availability of the logging part, so that even if the host that has one of these services goes down the logging keeps on working; are there any guidelines/suggestions on this? Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

1 0

Meaning of RGW sync status full sync: objects completed
by Adam Prycki 26 Nov '25

26 Nov '25

Hello, I have a question about rgw synchronization mechanic. When we are trying to initialize full bucket sync on existing bucket, or when try to synchonize delete bucket we see "full sync" in bucket sync status on our clusters. For example: realm b6ca44da-921e-42da-b377-80e9fae83b61 (s3-TEST.kmd4.pl) zonegroup c41b36c6-2031-407b-bab4-36cd0b3e56da (poznan-1) zone 91a17d0c-84a0-4706-a40d-1b6df1996a10 (zone-poznan-1-kielman) bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) current time 2025-11-24T13:11:48Z source zone 8e5c54a9-7eb2-4521-b327-081a1a647205 (zone-poznan-1-pcss-bst) source bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) full sync: 15221490 objects completed In this case we have deleted bucket with broken synchronization on the secondary zone. After RGW was started up again synchronization should have tarted from zero. Bucket 1d2848a2-9ca7-402c-9976-a6dc775547bb has 1163263 10MiB s3 objects. bucket sync status is right now at 15221490 and still increasing. That's 13x bigger number than object count in this bucket. In case of some other bucket we saw that "full sync: objects completed" number grew to about 20x s3 object count in the bucket. One bucket with 10 000 000 objects has running full sync has "full sync: 605952706 objects completed" Can someone explain what this number means and why it gets so big? How big it's supposed to get? Can we predict how much of the synchronization is "done" from this number? Best regards Adam Prycki

1 0

Ceph Developer Meetup next week
by Anthony Middleton 26 Nov '25

26 Nov '25

Hey Ceph friends, Please join us for the* Ceph Developer Meetup next week on December 3 at 11 am EST / 8 am PDT*. The agenda is in the meetup announcement. Please *add your topics* in advance. *RSVP *here: https://www.meetup.com/ceph-user-group/events/312181316/?utm_medium=referra…

1 0

RGW Signature does not match
by GLE, Vivien 26 Nov '25

26 Nov '25

Hi, I'm trying to setup a VM RGW in a cephadm managed test cluster. Everything is working fine except 2 things: - RGW didn't show up in the object section of the ceph dashboard and this message shows up The Object Gateway Service is not configured Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx000002bfe4c78d8a' b'4b666-006926c06c-42063466-<zone>","HostId":"42063466-<zone>-<groupzone>"}') The daemon is active in ceph -s and other functionalities are working fine I read that it might be a date error, as my hosts are on CET, but the containers are set to UTC is there a way to correct that ? - The second thing is that I need to radosgw-admin --name client.ceph-rgw (which is working fine) instead of only radosgw-admin otherwise it seems that radosgw-admin didn't find my ceph.conf and output is looking like this : -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin (2) No such file or directory failed to fetch mon config (--no-mon-config to skip) If I rename my client.keyring file this output is still showing up => failed to fetch mon config (--no-mon-config to skip) cat /etc/ceph/ceph.conf [global] fsid = <fsid> mon_host = [v2:192.168.219.12:3300/0,v1:192.168.219.12:6789/0] [v2:192.168.219.14:3300/0,v1:192.168.219.14:6789/0] [v2:192.168.219.15:3300/0,v1:192.168.219.15:6789/0] auth_client_required = cephx auth_service_required = cephx auth_supported = cephx [client.ceph-rgw] host = ceph-rgw rgw_dns_name = ceph-rgw keyring = /etc/ceph/ceph.client.ceph-rgw.keyring rgw_frontends = "beast endpoint=192.168.219.19:8080 ssl_endpoint=192.168.219.19:8443 ssl_certificate=/etc/ceph/certs/ceph-rgw.pem" log_file = /var/log/ceph/ceph-rgw-ceph-rgw.log rgw_thread_pool_size = 512 cat /etc/ceph/ceph.client.ceph-rgw.keyring [client.ceph-rgw] key = <key> ceph-radosgw(a)ceph-rgw.service - Ceph rados gateway Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; preset: enabled) Active: active (running) since Tue 2025-11-25 16:33:08 CET; 19h ago Main PID: 90607 (radosgw) Tasks: 621 Memory: 114.6M (peak: 115.1M) CPU: 2min 2.665s CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw(a)ceph-rgw.service └─90607 /usr/bin/radosgw -f --cluster ceph --name client.ceph-rgw --setuser ceph --setgroup ceph Thanks Vivien

1 0

Changing the crush rule of an existing pool
by Huseyin Cotuk 26 Nov '25

26 Nov '25

Hello Cephers, I just want to get confirmation on two things. I'm going to change the crush rule of an existing pool so that it uses SSDs instead of HDDs. 1. Will changing the rule cause any problems with the current workload? 2. Does Ceph manage the migration of data currently in use when the existing pool's rule is changed? If so, does it also delete the data on the old disks after the migration? Thanks in advance. BR, Huseyin Cotuk hcotuk(a)gmail.com

1 0

New Cluster Ceph
by GLE, Vivien 25 Nov '25

25 Nov '25

Hi, We plan to buy hardware for a new cluster ceph and would like some approbation on what we choose. 6 nodes DELL 6715 + 6 Powervault MD2412 enclosure For each node => 1 CPU AMD EPYC 9475F 3,65 GHz, 48C/96T, 256M Cache (400 W) RAM 16Gox16 = 256Go 4x 100GB NVIDIA MELLANOX 100GB HBA465e (externe, 22,5GB/s) 2 NVMe mixed 6,4To DBWAL 6 NVMe mixed 6,4To 5 NVMe read 15,36To for each enclosure => 12 HDD 20To HDD will be a replica 3 pool with the dbwal NVME mixed and read will be 2 differents pools (we will test replica 3 and EC to see which performance/storage efficency satisfy us the most) This cluster will be mostly use to store block (VM proxmox/kubernetes) and S3. The crushmap will be like so : 3 rooms with 2 nodes per room. So the point of failure can be for replica 3 => rooms and if we use EC 4+2 => host We saw that the thread needs for NVMe OSD are very expensive, does this CPU good enough to carry them ? MON and MGR will be spread across the cluster and RGW on virtual machine Thanks for your answer ! Vivien

3 5

Tentacle: bootstrap fails setting orch backend
by Eugen Block 25 Nov '25

25 Nov '25

Hi, I tried bootstrapping a fresh test cluster on a virtual machine (openSUSE Leap 16), and it fails: ---snip--- Setting orchestrator backend to cephadm... Non-zero exit code 95 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v20.2.0 -e NODE_NAME=tentacle -v /var/log/ceph/c31e1b94-c921-11f0-b1cf-fa163e7037b6:/var/log/ceph:z -v /tmp/ceph-tmpqdv5t793:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp615joe66:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.0 orch set backend cephadm /usr/bin/ceph: stderr Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date /usr/bin/ceph: stderr Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v20.2.0 -e NODE_NAME=tentacle -v /var/log/ceph/c31e1b94-c921-11f0-b1cf-fa163e7037b6:/var/log/ceph:z -v /tmp/ceph-tmpqdv5t793:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp615joe66:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.0 orch set backend cephadm: Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it ---snip--- Setting it afterwards succeeds, but then you end up with a semi-broken cluster: [ceph: root@tentacle /]# ceph orch set backend cephadm [ceph: root@tentacle /]# ceph orch host ls 0 hosts in cluster I created a tracker without checking if anything similar already exists: https://tracker.ceph.com/issues/73974 I haven't tried bootstrapping with other distros yet, have others experienced something similar on a different OS? Thanks, Eugen

1 1

CSC Meeting Notes: 2025-11-24
by Adam Emerson 24 Nov '25

24 Nov '25

For the meeting on Monday, 24 November 2025. S390/PPC Testing Upstream? ========================== Radoslaw noted a recent PR (https://github.com/ceph/ceph/pull/64894) that is S390/Power specific. We'd like to merge it, it looks reasonable. In general, we don't test S390 and PowerPC upstream. Since we're moving the lab, is this on our plate/plans at all? Greg points out that we don't just lack machines, we don't really have anything in place to support those architectures upstream. Perhaps we should, but we'd need to convince some corporation to support it and donate hardware. The answer for new architectures is always that we are happy to accept hardware, we will not make your github checks required, and you have to do the work. (Especially since many of them are very expensive.) Ilya says that IBM has some virtual machines that can be carved out. Several upstream projects are using that to test on S390. It's hardware made available by IBM in a cloud-like setting. Probably not sufficient for our needed testing. We might put a github check there on a low frequency, maybe a couple times a week on main. Someone needs to watch it and address its output for it to be any use. We would need continuous time investment to do the work. Casey thinks downstreams should set up nightly/weekly builds on their own hardware and fix things when they break. building/testing upstream puts the maintenance burden on us. Lab Shutdown - 15 December ========================== David Galloway reminds us that 15 December is when we will start shutting the Sepia lab down. A truck will be arriving two days later to pick everything up and take it tou Poughkeepsie. It is increasingly looking like we will not have access to the stuff in Poughkeepsie at the end of the month as we were told. So it is looking increasingly like there might just be a time where the upstream lab is not available. Floated the idea of pushing this back through the management chain and that did not go well. Unfortunately things did not work out and there will be some downtime. Hopefully the downtime will last no more than a month. Will likely go to Poughkeepsie to to the wiring. While the physical network infrastructure is there and ready, the main networking person there is on medical leave and has not been backfilled yet. There is also Security Stuff to deal with. There are a number of hurdles. It may be feasible to run some Jenkins builders in the cloud. Release order documentation =========================== From Joseph Mundackal (ref: https://ceph-storage.slack.com/archives/C05R975114L/p1763681398991599) Since we were doing a reef point release even though it's EoL, got asked what our criteria for deciding what release to do next were. Do we have that documented anywhere? According to Ilya, the reef release was a result of our n − 2 policy and tentacle being delayed. Expecting the delay, we did the work to get the PRs together and put a Reef out. The listed EoL date is when we expect the next major release to come out. Joseph asked if there would be interest in a PR to resolve the confusion between listed EoL date and actual release activity. Ilya suggested removing the column for expected EoL and replacing it with an explanatory note and agrees the current situation is probably confusing to users. Joseph will raise a PR to update the documentation. Reef Release ============ Yuri did a set of runs for the Reef release and sent an email to the leads asking to approve Erroneous Spam Marking ====================== Laura notes that when you find one of the emails in your spam filter you can mark it as Not Spam. Greg says you need to set up a folder and tell it never to send emails to that list to Spam. David Galloway mentions a few possible tweaks for deliverability next year.

1 0