- ceph-users - lists.ceph.io

Performance Related Issue
by Vishnu Bhaskar 09 Oct '26

09 Oct '26

Hi, I am currently working on building an *all-flash Ceph setup for high-performance workloads*, and I am testing the performance using *8K write I/O*. I have tested the *same hardware* with PetaSAN as well. Since PetaSAN is Ceph-based, I expected the performance to be relatively similar, but I am seeing a significant difference. Performance Observed With *PetaSAN*, I am getting approximately *120K IOPS* from the iSCSI workload. With my standalone *Ceph setup*, I am getting only around *30K IOPS* from the same type of workload. Test Environment - All-flash storage - 100 GbE network - Jumbo frames enabled on both setups - FIO workload running over an *iSCSI share* - 8K write workload - Same/similar hardware used for both PetaSAN and Ceph - Comparing the performance of PetaSAN vs. my Ceph deployment I also checked the OSD-level benchmark to make sure the OSD itself is not showing a significant performance difference. PetaSAN / OSD Benchmark For 4K: root@node3:~# ceph tell osd.0 bench 12288000 4096 4194304 100 { "bytes_written": 12288000, "blocksize": 4096, "elapsed_sec": 0.054923286, "bytes_per_sec": 223730240.75799105, "iops": 54621.640810056408 } For 8K: root@node3:~# ceph tell osd.0 bench 12288000 8192 4194304 100 { "bytes_written": 12288000, "blocksize": 8192, "elapsed_sec": 0.028371557, "bytes_per_sec": 433109821.92482424, "iops": 52869.851309182646 } Standalone Ceph OSD Benchmark For 4K: root@c100node2:~# ceph tell osd.0 bench 12288000 4096 4194304 100 { "bytes_written": 12288000, "blocksize": 4096, "elapsed_sec": 0.060264142, "bytes_per_sec": 203902347.10385489, "iops": 49780.846460902074 } For 8K: root@c100node2:~# ceph tell osd.0 bench 12288000 8192 4194304 100 { "bytes_written": 12288000, "blocksize": 8192, "elapsed_sec": 0.031226269, "bytes_per_sec": 393514832.01531374, "iops": 48036.478517494354 } Interestingly, the OSD-level benchmark results are fairly close between the two environments (~48–53K IOPS), but the *end-to-end iSCSI FIO performance differs significantly*, with PetaSAN reaching around *120K IOPS* while my Ceph setup reaches only around *30K IOPS*. Any suggestions on how to isolate where the additional ~90K IOPS difference is coming from would be very helpful Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

3 2

We dodged a bullet
by Stefan Kooman 07 Oct '26

07 Oct '26

Hi, Yeah, so that is a bit of a clickbait title, but I do think it's true. In the hope this might be useful for someone, or some cluster, I decided to write this down. So what happened? First some context: subject at hand is a Ceph pacific cluster that has been running fine for over nine years (Went through 12 -> 13 -> 14 -> 15 -> 16 with many point releases in between). All OSDs were recreated in 16.2.11 to make use of the RocksDB column families without the need to do resharding (which would have taken forever with the amount of OMAP we have). This cluster has been provisioned with a home build Ansible playbook based on the manual procedure [1], and is basically an automation of all these manual steps. For OSDs we used the "short form" variant [2]. The first issue we encountered was when we added new storage nodes / OSDs to the cluster. The deployment of the first storage node took longer than expected (PCIe NVMe devices that were in use for OS were not correctly filtered and turned up as eligible for OSD creation). After we fixed that we could succesfully proceed. However, the first OSD that was added hit the "out" timer (we did not have noout flag set) and was set "out" by the cluster. We issued the "ceph osd in osd.$id" command to get the OSD back in. The state for all new OSDs was "down" and "in" apart from this OSD, which was in state "down" and "out". This immediately resulted in a assert of the monitors (num_down_in_osds <= num_in_osds). Different monitors crashed in quick succession but the quorum was never lost, and three mons (out of five) were always online. It also did not impact production traffic as now data was stored on them yet. The "ceph osd in osd.$id" that was hanging (due to crashing monitors) was aborted and shortly after that the monitors stopped crashing. The storage node got rebooted and the OSDs were all "up" and "in", apart from one. But this time we could successfully set the osd "in" ("up" and "in" state). We tried to reproduce this in a (virtual) test cluster that is as close a match to production but did not manage to reproduce. We figured this might have been a race condition of some kind (while the command was issued new OSDs might also have been added). The storage nodes were rebalanced and the new nodes were now also taking production workload. Somewhat later we decided to test a change (disable bypass workqueue flags of the encrypted (dmcrypt) devices) and shutdown the OSDs. Immediately after that several mons hit the same assert again. Although the OSDs had gracefully shut itself down ... because of the asserting mons no new osdmaps were created and the peer OSDs were not informed that there peers were down. Thus all involved OSDs had to figure this out themselves which they directly did ... but of course this did result in a lot of slow OSD operations. The OSDs were quickly started again and only until all OSDs were "up" and "in" again the monitors stopped crashing. The monitors did not loose quorum although we were of course afraid that they would. As it turned out this was likely not a race condition we started to look into this further. A search query on Ceph's issue tracker [3] resulted in a lot of hits. Turned out this issues has been hit by clusters provisioned by ROOK, manual procedure (ours) and potentially others and that the issue had been seen on clusters from 16.x, 17.x as well as 19.x and 20.x. This note gives a good summary [4]. It was fixed by this PR [5] and backported to Squid and Tentacle. But the reason *why* we hit this issue was not clear to us. Until we saw this report [6] wherein a user reported that their cluster had a lot of their OSDs with a "NEW" bit and when they would set an OSD "out" and "in" this bit would disappear and so did their issues (not hitting the assert anymore). We dumped the osdmap of the cluster and found out that *all* of our OSDs had the "NEW" bit active apart from one ... the one osd that had been through the OSD "out" / "in" state before. We only expected a "NEW" bit to be enabled for truely new OSDs, not OSDs that have been in the cluster for many years already. We decided not to wait until we would hit a scenario like the following: - one or more OSDs go down (broken or otherwise unable to start again) - have the monitors hit the assert - unable to get out of this situation easily as at that point you cannot issue any ceph commands as the monitors are asserting and the OSDs that went down cannot be brought back up. - hit a lot of slow ops in the meantime until the monitors would set the OSDs down ... or out eventually by itself (down out timer) which could have taken a long time in our case - have the risk of the monitors loosing the quorum and experience unavailability for an extended period of time. We backported the fix to the OSDMap.cc and created our own packages in case we would need them. After that we went ahead and set every OSD "out" and "in" to get rid of the NEW bit and that worked fine. The problem did not trigger anymore, as expected. Today I dived a bit deeper in the OSDMap.cc code responsible for this behavior ... turns out that there is a quicker way to fix this. Relevant code snippet: for (const auto &weight : inc.new_weight) { set_weight(weight.first, weight.second); // if we are marking in, clear the AUTOOUT and NEW bits, and clear // xinfo old_weight. if (weight.second) { osd_state[weight.first] &= ~(CEPH_OSD_AUTOOUT | CEPH_OSD_NEW); osd_xinfo[weight.first].old_weight = 0; } } If you perform a osd reweight, like "ceph osd reweight osd.0 1" it will remove the NEW bit. This is a way quicker process than "out" / "in". We added a reweight step to get rid of the "NEW" bit in our playbook if we would need to provision / replace OSDs again in the future. The reason why we hit this issue now, and not ever before, with the NEW bit OSDs in them and the bug still present is still unclear. There must be some other condition(s) that we hit this time and not before. I've checked our Cephadm based clusters and those don't have any NEW bits present. I'm not sure what steps Cephadm executes under the hood when it creates new OSDs but I guess this involves a step were it performs a reweight. For clusters out there that have been manually provisioned in the past, or created with Ansible playbooks that follow a similar procedure there might be OSDs present with this NEW bit that sooner or later have the risk of hitting this assert (until they run a version in which this is fixed). I advise you to check your clusters and get rid of these NEW bits because AFAIK their only purpose is to have this bit set when they are indeed, new, and they might harm your cluster(s) in the future. Questions we still have: - What other condition(s) are needed to trigger this assert - Would the manual install procedure need to include this reweight step, or is there another way to have the NEW bit cleared? Gr. Stefan [1]: https://docs.ceph.com/en/latest/install/manual-deployment [2]: https://docs.ceph.com/en/latest/install/manual-deployment/#short-form [3]: https://tracker.ceph.com/search?page=3&q=ceph_assert%28num_down_in_osds+%3C… [4]: https://tracker.ceph.com/issues/70869#note-9 [5]: https://github.com/ceph/ceph/pull/64009 [6]: https://tracker.ceph.com/issues/52535#note-12

3 2

Google Summer Internship recording
by Anthony Middleton 07 Oct '26

07 Oct '26

Hey Ceph friends, In case you missed it, here's the recording of yesterday's Google Summer of Code intern showcase: https://youtu.be/jUrze3sZMhA Project descriptions are on the event landing page: https://ceph.io/en/community/events/2026/GSoC2026/ Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

19.2.7 QE validation status
by Yuri Weinstein 06 Oct '26

06 Oct '26

Details of this release are summarized here: https://tracker.ceph.com/issues/78483#note-7 Release Notes (drafts) - https://github.com/ceph/ceph/pull/71539 https://github.com/ceph/ceph.io/pull/1198 LRC/SRC upgrade - TBD (Patrick,Dan / The Core Team) PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: Laura rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson, Eric fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/quincy-x: PASSED upgrade/reef-x: PASSED upgrade/telemetry-upgrade: PASSED upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: FAILED (suspect https://tracker.ceph.com/issues/79948) upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED powercycle: PASSED TIA

2 2

Ceph Developer Meetup Wednesday, Oct 7th, at 8 am PDT
by Anthony Middleton 05 Oct '26

05 Oct '26

Hey Ceph Devs, The next Ceph Developer Meetup is this Wednesday, Oct 7th, at 8 am PDT / 15:00 UTC. Add your discussion topics or general questions to the agenda here: https://tracker.ceph.com/projects/ceph/wiki/CDM_07_OCT_2026 Please reach out to Laura Flores with any questions about this session. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph Community: Google Summer of Code internet showcase
by Anthony Middleton 05 Oct '26

05 Oct '26

Happy Monday, everyone! Just a reminder that the Ceph Community: Google Summer of Code internet showcase is tomorrow, Oct 6th, at 8 am PDT/11 am EDT. During this session, we'll hear from 3 mentees who spent their summer tackling real-world Ceph problems: - RGW Admin UX & Docs — auto-generated radosgw-admin docs - Kafka Security — Kerberos/JWT/mTLS support for RGW's Kafka notifications - Ceph Dashboard — completing the Dashboard's move to Carbon Each mentee will walk through their project, the challenges they faced, how they overcame them, and what they'd tell the next GSoC contributor. 🕗 8:00 am PDT / 11:00 am EDT Join here: https://meet.jit.si/CephGSOC2026 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph OSD Jaeger tracing: errors and no end-to-end traces
by Kathirvel T 05 Oct '26

05 Oct '26

Hi Ceph community, We enabled the native Jaeger tracing on the OSDs of our Ceph 19.2.3 cluster to get distributed tracing. Issues: 1. Every OSD continuously logs this following error, [Error] File: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/src/recordable.cc:95[TRACE JAEGER Exporter] SetAttribute() failed, attribute type not supported 2. In Jaeger we only see two standalone spans per op (op requested/created and enqueue). They are not connected to each other or to any client-side trace. Questions 1. Is the SetAttribute() error a known issue, and is there a fix or workaround? 2. What is the intended use of these two spans? Is there a way to get full end-to-end traces in current releases? If anyone has faced this or knows a fix, please let us know. Thanks & Regards, Kathirvel T Associate R&D Engineer 9025610912 mailto:manoj.kumar@zybisys.com kathirvel.t(a)zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 2

Ceph Tech Talk Recording
by Anthony Middleton 01 Oct '26

01 Oct '26

Hey Ceph friends, Here is the recording of yesterday's Ceph Tech Talk: https://youtu.be/7lY-yJNkPhc Thank you to Anthony D'Atri for leading the talk, and to everyone who joined and contributed to the discussion. Our next tech talk is scheduled for *Monday, Oct 19th at 12 pm EDT* and will feature Tadas Sutkaitis of VEXXHOST. Tadas will present on *OIDC Web Identity for Ceph RGW*. More details to follow! Hope to see you there! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph MDS issues!
by Erich Weiler 01 Oct '26

01 Oct '26

Hey All, I’m hoping someone can help with an issue I’m having… I have Ceph 19.2.3 on Rocky 9.3, and I have a CephFS filesystem. It’s been working for years OK, through several upgrades. Just today it went offline, with all MDS daemons (3 of them) stuck in “replay” forever. I tried restarting them, etc, but nothing helped. I tried manually failing ranks 2 and 3 to make it a simple one MDS daemon, then set "ceph fs set slugfs max_mds 1" and it seems to be stuck in “resolve” state after restarting that one daemon: [root@pr-md-01 ~]# ceph fs status slugfs - 30 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 resolve slugfs.pr-md-01.umxlxt 679k 557k 66.4k 0 1 failed 2 failed POOL TYPE USED AVAIL cephfs_metadata metadata 797G 3026G cephfs_md_and_data data 0 36.2T cephfs_data data 1449T 72.5T MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@pr-md-01 ~]# cephadm enter --name mds.slugfs.pr-md-01.umxlxt -- ceph daemon /var/run/ceph/ceph-mds.slugfs.pr-md-01.umxlxt.asok status Inferring fsid 58bde08a-d7ed-11ee-9098-506b4b4da440 { "cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", "whoami": 0, "id": 19345540, "want_state": "up:resolve", "state": "up:resolve", "fs_name": "slugfs", "resolve_status": { "resolve_gather": "", "resolve_ack_gather": "" }, "rank_uptime": 1538.6363653870001, "mdsmap_epoch": 745780, "osdmap_epoch": 76803, "osdmap_epoch_barrier": 0, "uptime": 1538.823382218 } Ranks 1 and 2 are manually failed for now just to try to get things going again. I ran these commands to remove the “failed” ranks (which I probably should not have done, in retrospect): ceph mds fail slugfs:1 --yes-i-really-mean-it ceph mds fail slugfs:2 --yes-i-really-mean-it ceph mds rmfailed slugfs:1 --yes-i-really-mean-it ceph mds rmfailed slugfs:2 --yes-i-really-mean-it Any ideas how I can get the MDS back? I am up against a wall… I did dump the journals. I think I just need to get ranks 1 and 2 back online in addition to rank 0, then get the MDS's to move from "resolving" to "up"... Thanks for any advice!! cheers, erich

2 3

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Frank Ederveen 01 Oct '26

01 Oct '26

> If you upgrade to 20.2.4 the key rotation for the Ceph services will > be done by the orchestrator automatically. But only for the Ceph > services, no client keyrings will be touched by cephadm. So you can > safely upgrade and fix the CVEs but can still use clients with the aes > format keys. I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both. It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains. All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd. Is there going to be a new release soon, or some scripting which fixes things? health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ... Thanks, Frank ________________________________ From: Eugen Block <ceph-users(a)ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4 If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. Zitat von Brent Kennedy <ceph-users(a)ceph.io>: > Question, sort of related. If upgrading from squid to tentacle, > will it automatically try to rotate keys are part of the cluster > upgrade or is that still a manual process? We would like to avoid > the key rotation for now as we have some proxmox nodes using ceph > storage in some of the clusters. > > Regards, > -Brent > > Existing Clusters: > US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > > > > -----Original Message----- > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: Saturday, August 29, 2026 10:38 AM > To: Saif Mohammad <samdto987(a)gmail.com> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Clarification on Client Key Rotation > During Upgrade to 19.2.6/20.2.4 > > Hello, > > On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: >> >> Hi, >> >> While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these >> versions introduce the secure key type, I have a clarification >> regarding client key rotation. >> For the client.admin key, we need to rotate the key and then >> import/update the keyring. >> For the other client keys, do we only need to rotate the key and >> verify that the key has changed? Is there no need to import/update >> the keyring for the other clients, since there is no local keyring >> available? > > Step (9) of [1] says: > > "Then copy and import the key to each machine using that client.$ID > credential." > > [1] > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 6