- ceph-users - lists.ceph.io

Birds of a Feather Proposal for SuperComputing 2026
by Anthony Middleton 15 May '26

15 May '26

Hey Ceph friends, I hope all is well. This year, the Ceph Foundation plans to submit a Birds of a Feather (BoF) proposal for SuperComputing 2026 (SC26), and we want this to be a genuinely community-driven session, shaped by the people who are deploying, developing, and using Ceph in HPC environments. A BoF is an open, interactive session at SC where practitioners and developers gather to share experiences, discuss challenges, and influence future direction. It's less of a talk and more of a structured community conversation. We'd love your input on two things: 1. TOPIC IDEAS: What should we discuss? We're looking for the most pressing topics, questions, and challenges facing Ceph in HPC right now. Some starting points: - CephFS performance and tuning for HPC workloads - NVMe/SSD tiering and all-flash deployments - Data encryption and security at scale - Interoperability with HPC job schedulers and parallel filesystems - Operational challenges at petabyte scale - What's missing in Ceph for HPC use cases? What would YOU want to talk about? Reply to this thread or fill out this short form: https://forms.gle/SEhgjYz9pda8SgT88 2. PARTICIPATION: Are you attending SC26? We're looking for: - Lightning talk speakers (5–10 min) who can share a real deployment story or lesson learned - Co-session leaders from different organizations (especially HPC centers, national labs, or universities) - Community members who want to show up and join the discussion SC26 takes place November 15–20, 2026, in Chicago, IL. The BoF proposal deadline is July 2nd. This session works best when it reflects a wide range of voices and organizations within the Ceph community, so the more input we receive, the stronger the proposal and the session will be. Please reply here, reach out directly, or share this with anyone in your network who works with Ceph in HPC contexts. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Ceph Tech Talk Recording for May and June Announcement!
by Konold, Martin 14 May '26

14 May '26

Am 2026-05-12 17:01, schrieb Anthony Middleton via ceph-users: Hi, YouTube gives me "This video has been removed by the uploader". Regards --martin > Hello Ceph Friends! > > I hope your week is off to a great start! You can find the recording of > yesterday's Ceph Tech Talk on Containerized Deployment of Ceph with > KCLI > here: https://youtu.be/vPOYyNtpHp4. -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

1 0

CFS IO500 ISC'26 Deadline 12 June 2026
by IO500 Committee 13 May '26

13 May '26

https://io500.org/submission Deadline 12 June 2026 AoE The IO500 is now accepting and encouraging submissions for the upcoming semi-annual IO500 Production and Research lists, in conjunction with ISC HPC 26. Once again, we are also accepting submissions to the 10 Client Node Challenge to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Submissions Webpage. The new ranked lists will be announced at the "IO500: High Performance Storage Community" BoF. We hope to see many new results. The GitHub repository has a new tag io500-isc26 for the current version of the benchmark for this list. Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community-accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes, giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: - Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance, and the documentation of how the numbers are achieved. - Support small to extreme-scale Research and Production HPC systems using flexible storage APIs - Maximizing simplicity in running the benchmark suite - Specifically, the benchmark suite includes a hero run of both IOR and mdtest, configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly prescribed parameters in an attempt to determine a lower performance bound. New in this release is a random 4KB read phase to better understand the performance of non-sequential IO patterns used in AI workloads. Finally, it includes a namespace search, as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. - The goals of the community are also multi-fold: - Gather historical data for the sake of analysis and to aid predictions of storage futures - Collect tuning information to share valuable performance optimizations across the community - Encourage vendors and designers to optimize for workloads beyond "hero runs" - Establish bounded expectations for users, procurers, and administrators - Understand and be able to reproduce performance on storage systems - Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. - The IO500 follows a two-stage approach. First, there will be a two-week stabilization period during which we encourage the community to verify that the benchmark runs properly on a variety of storage systems. During this period, the benchmark may be updated based upon feedback from the community. The final benchmark will then be released. We expect that runs compliant with the rules made during the stabilization period will be valid as a final submission unless a significant defect is found. 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather Once again, we encourage you to submit to join our community, and to attend the ISC26 BoF on ISC26, where we will announce the new IO500 Production and Research lists and their 10 client node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage features customizable sorting, allowing you to submit on a small system and still achieve a very good per-client score, for example. We will also highlight new and interesting results at the BoF.

1 0

Ceph Tech Talk May
by Anthony Middleton 12 May '26

12 May '26

There was an issue with the video, so I deleted it and reuploaded it. Sorry for the confusion. Please enjoy Shraddha's presentation - https://www.youtube.com/watch?v=_zPR0s5H468 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph Tech Talk Recording for May and June Announcement!
by Anthony Middleton 12 May '26

12 May '26

Hello Ceph Friends! I hope your week is off to a great start! You can find the recording of yesterday's Ceph Tech Talk on Containerized Deployment of Ceph with KCLI here: https://youtu.be/vPOYyNtpHp4. Our next session is scheduled for June 9th. The topic will be "Building Ultra-Resilient Ceph Clusters with Xinnor and OSNexus," featuring Steven Umbehocker, CEO of OSNexus, and Dmitry Livshits, CEO of Xinnor. RSVP here: https://www.linkedin.com/events/7459977670458757120?viewAsMember=true Subscribed to the Ceph Community Calendar <https://calendar.google.com/calendar/b/1?cid=OXRzOWM3bHQ3dTF2aWMyaWp2dnFxbG…> for more details. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

RGW/Beast logs show the proxy IP instead of the real client IP
by farhad kh 12 May '26

12 May '26

Hi, i try to enable RGW access log for calculation byte_send, byte_reseved, operation_count, audit_detail, request_tracing and ... with set these options ceph config set client.rgw.vanak rgw_enable_usage_log true ceph config set client.rgw rgw_enable_usage_log true ceph config set client.rgw.vanak rgw_ops_log_rados true RGW Beast access logs show the HAProxy node IP instead of the real client IP, despite X-Forwarded-For being correctly set by the cephadm ingress service. Root Cause Analysis: 1. Logging Mechanism: By design, the Beast frontend logs the TCP source address (Layer 4) rather than parsing the X-Forwarded-For HTTP header (Layer 7). This ensures audit logs reflect the trusted transport peer. 2. Missing PROXY Protocol: To preserve the real client IP at the transport layer, PROXY protocol must be used (send-proxy-v2 in HAProxy and proxy_protocol=true in RGW). 3. Orchestration Blocker: The current version of cephadm ingress (e.g., Reef) does not support the enable_proxy_protocol parameter in its service specification. Consequently, it cannot generate the required send-proxy-v2 directive in the managed HAProxy configuration. Technical Impact: - Observability: RGW logs are insufficient for client-level auditing, forensics, or per-IP bandwidth accounting. - Workaround Needed: Client attribution must currently be performed at the HAProxy log layer rather than the RGW log layer. Proposed Resolution / Solution Paths: 1. Upgrade: Verify and move to a Ceph version where cephadm ingress supports enable_proxy_protocol. 2. Manual Ingress: Replace the cephadm-managed ingress with a manual HAProxy deployment configured with send-proxy-v2 to satisfy RGW’s proxy_protocol=true requirement. 3. Upstream Change: Propose a feature update for cephadm to include first-class PROXY protocol support in the ingress spec to ensure parity with Beast’s logging architecture. Captured evidence: - tcpdump confirms X-Forwarded-For is present but no PROXY preamble is sent. - RGW is configured for proxy_protocol=true but receives plain HTTP streams. 07:09:44.247960 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [.], ack 1, win 502, options [nop,nop,TS val 735749891 ecr 4199462555], length 0 E..4..@.@.;..........,.@.C@f... ....#...... +....N.. 07:09:44.248068 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [P.], seq 1:902, ack 1, win 502, options [nop,nop,TS val 735749891 ecr 4199462555], length 901 E.....@.@.83.........,.@.C@f... .....G..... +....N..PUT /test/test2 HTTP/1.1 host: 172.29.244.8 accept-encoding: identity user-agent: aws-cli/2.31.18 md/awscrt#0.27.6 ua/2.1 os/linux#6.8.0-86-generic md/arch#x86_64 lang/python#3.13.7 md/pyimpl#CPython m/Z,E,n,b,W,N cfg/retry-mode#standard md/installer#exe md/distr ib#ubuntu.24 md/prompt#off md/command#s3api.put-object expect: 100-continue x-amz-checksum-crc64nvme: DxVUd/EiBAY= x-amz-sdk-checksum-algorithm: CRC64NVME x-amz-date: 20260510T070943Z x-amz-content-sha256: e951419ac0d10cc76f3e5eb415e68f7d7d6dd99e016d1463f0cb1ce18680d8e1 authorization: AWS4-HMAC-SHA256 Credential=MPQOYU9DYBV012QXHQ0H/20260510/tehran/s3/aws4_request, SignedHeaders=host;x-amz-checksum-crc64nvme;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-a lgorithm, Signature=59da15f49a75490adf3e90829616416ed0abac757272146692c3a6149849e1c3 content-length: 52428800 x-forwarded-for: 172.29.244.19 connection: close 07:09:44.248088 ens160 Out IP ceph-csb-02.8000 > 172.29.244.7.44332: Flags [.], ack 902, win 503, options [nop,nop,TS val 4199462556 ecr 735749891], length 0 E..4I.@.@............@.,... .CC.....@p..... .N..+... 07:09:44.253519 ens160 Out IP ceph-csb-02.8000 > 172.29.244.7.44332: Flags [P.], seq 1:26, ack 902, win 503, options [nop,nop,TS val 4199462561 ecr 735749891], length 25 E..MI.@.@............@.,... .CC.....@...... .N..+...HTTP/1.1 100 CONTINUE 07:09:44.253912 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [.], ack 26, win 502, options [nop,nop,TS val 735749897 ecr 4199462561], length 0 E..4..@.@.;..........,.@.CC....9........... +.. .N.. 07:09:44.255059 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [P.], seq 902:8142, ack 26, win 502, options [nop,nop,TS val 735749898 ecr 4199462561], length 7240 E..|>.@.@..n.........,.@.CC....9....\...... +.. .N.._>.z..?....b.T}g1Wa^yc..wWq$...S.....(..0...yf''...`..O....P.....Y.H.....?#.:f...z..v....{w_fjx..a....v...>+.............J;...M.......e...OE.....#.. .h...U.UW.`...d.,..6...2e .i(.;.... + my client_ip is '172.29.244.19' but show in log haproxy ip '192.29.244.7' :| {"log":"debug 2026-05-09T11:09:27.398+0000 7fc897ac4640 1 beast: 0x7fc7b18d6310: 172.29.244.7 - farhad$root [09/May/2026:11:09:27.394 +0000] \"GET / HTTP/1.1\" 200 510 - \"aws-cli/1.22.34 Python/3.10.12 Linux/5.15.0-164-generic botocore/1.23.34\" - latency=0.004000204s\n","stream":"stdout","time":"2026-05-09T11:09:27.405683471Z"} how can i fix this problem ? anyone have solution?

1 0

Exploring Interest- Ceph Days Chicago 2026
by Lee-Ann Pullar 11 May '26

11 May '26

Hey All, I’m reaching out to start some initial planning discussions around a potential Ceph Days Chicago event, tentatively scheduled for Monday, November 16, 2026. We would like to collaborate with the OpenInfra community and, as the event coincides with the Supercomputing Conference, bring together attendees from across the Ceph, OpenInfra, and SC26 communities. Before moving further with planning, we’d love to gauge interest from the community. If you have a moment, please take this quick survey. https://forms.gle/jzrP7A4hkgiTL98X6 Your feedback will help shape the size, scope, and direction of the event. If there’s enough interest in the date and location, we’ll move forward with planning to make this a great community event! Thank you, Lee-Ann Pullar

1 0

OSDs and RGW busy without traffic since Tentacle Upgrade
by Phillip Schichtel 10 May '26

10 May '26

Hi everyone, I've previously posted about this in Slack, but it was suggested to bring this to the mailing list, so here I am. The post in slack: [1]. I've recently upgraded a small Squid cluster to Tentacle (from 19.2.3 straight to 20.2.1). The cluster has 3 nodes (Debian Bookworm, 2x EPYC 7371, 256gig memory) and is primarily used for RBD (VM images) and RGW/S3. There are 43 OSDs, 31 Intel SATA SSDs (10 per node, but the first node has 1 extra) and 12 Seagate SAS SSDs (4 per node). The SATA ssds have device class "ssd" with ~50% average utilization, the SAS SSDs have device class "sas-ssd" with ~15% average utilization and serve different pools. RGW only utilizes the ssd class. The cluster has been continuously upgraded from (I think) Mimic or Nautilus. It was originally deployed using ceph-deploy and switched to cephadm as soon as that was stable. Under Squid the cluster was usually running around 1-3% CPU most of the time, since the applications aren't super IO heavy. During off-hours we only see about 5-8k iops usually. With the upgrade to Tentacle the nodes jumped to about 60% CPU utilization and 150k iops (Hosts Overall Performance dashboard). All that while ceph status and all the metrics I could find in Grafana/Prometheus don't report increased throughput, iops or even latency. Checking on the nodes directly (e.g. with htop's IO view) all the OSDs in the ssd class consistently generate quite a bit throughput. The iops are generated by the OSDs, the CPU load is generated both by the OSDs and RGW. RGW is usually deployed with 3 instances (1 per node), but the issue also appears when running with just 1. It is currently single-site, but I have toyed around with multi-site a while back on Squid. Increasing the logging (debug_rgw 20/5 & debug_ms 1/5) showed that it often attempted to get data_log.* objects but failed to do so. The .rgw.log pool only contained a data_log.0 object, blindly creating empty data_log.{1..127} objects also didn't silence the log line. An example log: osd.2 v2:_._._._:6843/2369122728 33152999 ==== osd_op_reply(286992957 data_log.20 [call] v109598'11820244296 uv0 ondisk = -61 ((61) No data available)) ==== 155+0+0 (crc 0 0 0) 0x55e65cff3400 con 0x55e656ad7800 When I restart the single instance, the load returns back to normal for exactly 20 minutes, after which it returns to roughly the same level. This pointed me to the `rgw_sync_log_trim_interval` config. Both the log lines and this setting seem to be related to multi-site replication logs. Changing the option to another value also changes the delay until the issue appears, which makes me pretty certain the issue is triggered by one of the coroutines spawned by RGWSyncLogTrimThread. Luckily setting the interval to 0 doesn't start that thread, which I have set now to prevent RGW from burning through my flash cells and to keep power usage in check. I've also temporarily swapped out the RGW container image for quay.ceph.io/ceph- ci/ceph:tentacle-debug@sha256:1b89786ce81c900227a2077be33dadda8722006b0 45b409e91e1ed89d7547be1 (I hope that was the right image to get the latest backports for tentacle), but that version also has this issue. I've also created a flamegraph with perf (executed perf outside the container and then converted to svg inside the ceph container), but it doesn't seem super useful to me since the interesting symbols seems to be missing. I've attached it anyway. Should I go ahead and report this to tracker.ceph.com or does anyone have ideas on what to do here? Thanks in advance for any input! ~ Phillip [1]: https://ceph-storage.slack.com/archives/C1HFU4JK1/p1776029060801259

1 0

Re: InfluxDB deprecated?
by Anthony D'Atri 10 May '26

10 May '26

This may be a function of packages available on the underlying EL distributions used as the base for package builds. I believe that there may be an RPM in EPEL, but I personally am hesitant to rely on EPEL for anything. Installation via pip3 should work. I'll forward this to someone involved in releases / builds. ceph.spec.in has %package mgr Summary: Ceph Manager Daemon %if 0%{?suse_version} Group: System/Filesystems %endif Requires: ceph-base = %{_epoch_prefix}%{version}-%{release} Requires: ceph-mgr-modules-core = %{_epoch_prefix}%{version}-%{release} Requires: libcephsqlite = %{_epoch_prefix}%{version}-%{release} %if 0%{?weak_deps} Recommends: ceph-mgr-dashboard = %{_epoch_prefix}%{version}-%{release} Recommends: ceph-mgr-diskprediction-local = %{_epoch_prefix}%{version}-%{release} Recommends: ceph-mgr-k8sevents = %{_epoch_prefix}%{version}-%{release} Recommends: ceph-mgr-cephadm = %{_epoch_prefix}%{version}-%{release} Recommends: python%{python3_pkgversion}-influxdb %endif I see an RPM package in EPEL9 but not EPEL10, so I suspect that's the root cause. I suspect that 18.2.7 might still work but do not have the wherewithal to test. The EPEL 9 RPM may well have been outdated. Possibly the build just needs to install via pip3. I'll see what I can do. > On May 7, 2026, at 3:02 PM, Robert Sander via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > Going from 16.2.15 in packages to cephadm managed containers and > upgrading to 18.2.8 we find that the InfluxDB MGR module does > not work any more: > > # ceph mgr module enable influx > Error ENOENT: module 'influx' reports that it cannot run on the active manager daemon: influxdb python module not found (pass --force to force enablement) > > The container image does not contain the necessary Python module. > > There is a tracker issue open for 18 months now: > https://tracker.ceph.com/issues/68892 > > Has the InfluxDB support been deprecated? I do not believe so, though the cool kids skate Prometheus ;) > Why is it still in the documentation for recent Ceph versions? See above. Oversights for soft dependencies on rarely-used bits can happen. > > Regards > -- > Robert Sander > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > http://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 220009 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Ceph deployment for HPC and research workflows
by Anthony D'Atri 09 May '26

09 May '26

> On May 6, 2026, at 9:55 AM, Max Breitmeyer via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello all, > At UMBC we were given a grant to run a ceph cluster as our primary research > storage for an HPC facility. The ceph cluster consists of the following > hardware: > Mon” nodes x3: > • 2x 25Gb interfaces > • 192GB RAM > • Storage: > -- 0 (none) OSDs/drives Why nodes dedicated only to mons? I suggest considering a deployment where mons are colocated on OSD or MDS nodes, either abstracting client config through DNS, by constraining placement to ensure that the orchestrator is failure-domain aware when scheduling, or possibly even just specifying all of the mons on the client mounts for resilience. The latter for sure doesn't scale well and is prone to not keeping up with changes. Clients need to be able to contact at least one mon to download the current monmap. > “HDD” nodes x16: > • 2x 25Gb interfaces > • 192GB RAM > • Storage: > -- 12 OSDs w/ 20TB HDDs > -- 2 local Write-Access-Logs (journal) w/ 8TB NVMe drives WAL+DB? > “NVMe” nodes x3: > • 2x 100Gb interfaces > • 384GB RAM > • Storage: > -- 16x OSDs w/ 8TB NVMe drives These could easily have colocated mons, sparing the dedicated mon nodes. > “MDS” nodes x3: > • 2x 100Gb interfaces > • 192GB RAM > • Storage: > -- 8x OSDs w/ 1TB NVMe drives A few thoughts: * I recommend 5 mon daemons vs only 3 * Since the MDS node spec above mentions 1TB (960 GB) SSDs and the NVMe node spec mentions 8 TB (7.6 TB) SSDs, I might infer that you constrain -- or intended to constrain -- the MDS metadata pool to only these 24x OSDs? And that the 48x 7.6 T SSD nodes host OSDs used for a separate CephFS filesystem or specific files/directories via layout xattrs? I might suggest a rearchitecture, which could be done incrementally without downtime: * Colocate 5 mon daemons on other nodes and either use these 3 systems for something else, locate some RGWs (if in use) there. If those nodes were ordered or could be made NVMe-capable, you might repurpose them as additional OSD nodes. If they have SATA bays, you could deploy HDD OSDs on them too. * It's best to not have OSDs within a given device class not be very different sizes, but Ceph can handle it. Setting mgr/balancer/upmap_max_deviation 1 will help the balancer do well with the smaller OSDs. * If the 1T and 7.6T SSD OSDs aren't in the same device class, consider consolidating them. Your workload -- and availability -- will benefit. The fine folks at OSNexus could help you with these ideas. "to use the full 192GB" There ya go. Note that the MDS cache size config value (default is 4GiB I think?) is like osd_memory_target: advisory, not a hard limit. Under heavy load an MDS can and will attempt to use more than the configured amount. You also need to allow some margin for other MDS internal uses, other daemons (node_exporter, the OSDs I presume you're running, etc), and the usual OS tasks. I might suggest with 192 GiB of physmem to never configure more than 128 GiB at most for MDS cache. Individual MDSes with sizes > 64 GiB may also be very slow to failover. So without seeing your actual deployment details this is speculation, but one thought is that the too-large MDS cache size DoSed or OOMed the colocated OSDs. > > In the six months since moving from a different storage solution (isilon), > we've had multiple crashes on the system that have completely taken down > our system as a result of multiple thousands of jobs being run at the same > time, causing an overload on the active MDS nodes. Has the use of ceph as a > primary research storage been used in an HPC with many dozens of different > workflows and filetypes being used at once? Or is a better to design to > come up with a tiered approach in combination with a second storage > solution? > > -- > V/R, > Maxwell Breitmeyer > UMBC HPCF Specialist > Graduate Student > (443) 835-8250 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0