- ceph-users - lists.ceph.io

Ceph Repository Signing Key and EL10
by Whitehouse, Dan 15 Oct '25

15 Oct '25

Hi, We are currently doing some testing of Rocky 10 and I have noticed that EL10 seems to have tighter standards for repository signing keys: Importing GPG key 0x460F3994: Userid : "" Fingerprint: 08B7 3419 AC32 B4E9 66C1 A330 E84A C2C0 460F 3994 From : https://download.ceph.com/keys/release.asc Is this ok [y/N]: y error: Certificate E84AC2C0460F3994: Policy rejects E84AC2C0460F3994: No binding signature at time 2025-10-15T09:43:58Z Key import failed (code 2). Failing package is: librados2-2:20.1.1-0.el10.x86_64 GPG Keys are configured as: https://download.ceph.com/keys/release.asc Many thanks, Dan Whitehouse

1 0

Announcing go-ceph v0.36.0
by John Mulligan 14 Oct '25

14 Oct '25

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.36.0 Changes include additions to the rbd, rgw, and cephfs packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project.

1 0

How to speed up the backfill on replicated pool?
by Jan Kasprzak 14 Oct '25

14 Oct '25

Hello, Ceph users, on my new cluster which I filled with testing data two weeks ago there are many repmapped PGs in backfill_wait state, probably as result of autoscaling the number of PGs per pool. But the recovery speed is quite low, in order of small MB/s and < 10 obj/s according to ceph -s. The cluster is otherwise idle, no client traffic after initial import, so I wonder why the backfill does not progress faster. Also, it seems like more pgs are getting remapped as existing ones get successfully backfilled - the percentage of misplaced objects is steadily around 6 % for the last two weeks. The PGs waiting for backfill all belong to the biggest pool I have according to "ceph pg dump | grep backfill", no surprise here. The pool has 229 TB of data and currently 128 PGs. It is replicated with k=4 m=2. The second biggest pool has only 23 TB of data: rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR pool_with_backfill 229 TiB 10086940 0 60521640 0 0 0 0 0 B 72545009 54 TiB 0 B 0 B second_biggest_pool 23 TiB 1153174 0 6919044 0 0 0 0 0 B 38506397 16 TiB 0 B 0 B [...] I tried to do "ceph osd pool force-backfill $pool", it helped to speed things up a bit, but it still runs at 50-200 MB/s and 4-20 obj/s. The initial data import ran at around 600 MB/s. Is it normal or can I speed the recovery up a bit somehow? Output of ceph -s: cluster: id: ... health: HEALTH_WARN 2 large omap objects services: mon: 3 daemons, quorum istor11,istor21,istor31 (age 13d) mgr: istor31(active, since 3w), standbys: istor21, istor11 osd: 36 osds: 36 up (since 2w), 36 in (since 3w); 14 remapped pgs data: pools: 45 pools, 1505 pgs objects: 13.39M objects, 198 TiB usage: 303 TiB used, 421 TiB / 724 TiB avail pgs: 5335074/80345832 objects misplaced (6.640%) 1449 active+clean 34 active+clean+scrubbing 11 active+remapped+backfill_wait+forced_backfill 8 active+clean+scrubbing+deep 2 active+remapped+forced_backfill 1 active+remapped+backfilling+forced_backfill io: recovery: 69 MiB/s, 4 objects/s The OSDs are HDD-based with metadata on NVMe, 4 OSDs per node, and all the nodes have load average somewhere between 0.3 and 0.6. Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

6 8

Ceph User+Dev October Meetup Cancelled
by Anthony Middleton 14 Oct '25

14 Oct '25

Hey friends, we're cancelling tomorrow's Ceph User+Dev Meetup to shift our focus to Cephalocon. Please feel free to share any thoughts in the thread.

1 0

Slow deep-scrubs
by Sven Grottke 13 Oct '25

13 Oct '25

Hi, we are operating a Ceph cluster with mostly replicated pools on SSDs, but one of our pools is erasure-coded on 16TB HDDs. Two weeks ago, we upgraded from Reef to Squid (19.2.3). Since then, there has been an ever growing number of PGs that have not been deep-scrubbed or scrubbed in time. All these PGs are located on the HDD pool. I would like to increase the scrubbing and deep-scrubbing intervals for the HDDs, but not for the SSDs. Is there any way to do that other than setting it on a per-OSD base, such as per device class or per sub-tree of the crush map? Thanks in advance for any advice! Ciao, Sven

4 5

ceph pop-up chat / water cooler / happy hour -- today at 12:05 Pacific / 21:05 Europe
by Dan van der Ster 10 Oct '25

10 Oct '25

Fellow Cephalopods, I'm hosting a spontaneous Ceph water cooler chat today at 12:05 Pacific / 21:05 Europe. Wanna look at some code together? Chat about osdmaps? Preview the Cephalocon agenda? ... Grab a sandwich or your beverage of choice and connect here: https://meet.jit.si/ephemeral-quorum Cheers, Dan -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO https://clyso.com | dan.vanderster(a)clyso.com

1 0

IO500 SC25 list call for submission
by IO500 Committee 09 Oct '25

09 Oct '25

Call for Submission Stabilization Period: Sep 29th - Oct 3rd, 2025 Submission Deadline: Nov 3rd, 2025 AoE The IO500 will soon be accepting and encouraging submissions for the upcoming 17th semi-annual IO500 Production and Research lists, in conjunction with SC'25. Once again, we will also be accepting submissions to the 10 Client Node Challenge to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Submissions Webpage. The new ranked lists will be announced at the "IO500: High Performance Storage Community" BoF. We hope to see many new results. The tag io500-sc25 should be used for submissions to this version of the benchmark list. Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance, and the documentation of how the numbers are achieved. Support small to extreme-scale Research and Production HPC systems using flexible storage APIs Maximizing simplicity in running the benchmark suite Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly prescribed parameters in an attempt to determine a lower performance bound. New in this release is a random 4KB read phase, to better understand the performance of non-sequential IO patterns used in AI workloads. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. The goals of the community are also multi-fold: Gather historical data for the sake of analysis and to aid predictions of storage futures Collect tuning information to share valuable performance optimizations across the community Encourage vendors and designers to optimize for workloads beyond "hero runs" Establish bounded expectations for users, procurers, and administrators Understand and be able to reproduce performance on storage systems Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. The IO500 follows a two-staged approach. First, there will be a two-week stabilization period during which we encourage the community to verify that the benchmark runs properly on a variety of storage systems. During this period the benchmark may be updated based upon feedback from the community. The final benchmark will then be released. We expect that runs compliant with the rules made during the stabilization period will be valid as a final submission unless a significant defect is found. 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather Once again, we encourage you to submit to join our community, and to attend the SC25 BoF where we will announce the new IO500 Production and Research lists and their 10 client node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. We will also highlight new and interesting results with invited talk(s) at the BoF.

1 0

Re: Read-Only LUN
by Kardos László 09 Oct '25

09 Oct '25

Hello, Thanks for your response. We must use it currently. We would like to mount the block device as read only for Veeam block based backup. We would like to be make it Read-Only because of security reasons. We found the write_protection option under configFS, but the write_protection option getting overwritten time to time. Best Regards, Laszlo Kardos -----Original Message----- From: Anthony D'Atri <aad(a)dreamsnake.net> Sent: Thursday, October 9, 2025 3:46 PM To: Kardos László <laszlo.kardos(a)acetelecom.hu> Subject: Re: [ceph-users] Read-Only LUN Don't use the iSCSI gateway unless you really really have to. It is not maintained. The NVMeoF gateway is a better idea. Are you providing storage to VMware? > On Oct 9, 2025, at 9:03 AM, Kardos László <laszlo.kardos(a)acetelecom.hu> > wrote: > > Hello, > > > > I would like to ask whether there is a possibility to configure a LUN > and iSCSI initiator to be Read-Only? > > > > > > Best Regards, > > Laszlo Kardos > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

Read-Only LUN
by Kardos László 09 Oct '25

09 Oct '25

Hello, I would like to ask whether there is a possibility to configure a LUN and iSCSI initiator to be Read-Only? Best Regards, Laszlo Kardos

1 0

Ceph PG stuck at degraded and very slow
by Sa Pham 09 Oct '25

09 Oct '25

*Hello everyone,* I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG). - Accessing objects from this PG is *extremely slow*. - Even running ceph pg <pg_id> takes a very long time. - The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs. Current ceph version is reef 18.2.7. Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it? Thanks in advance!

5 11