- ceph-users - lists.ceph.io

Ceph reef ingress service - v4v6_flag undefined
by Steven Vacaroaia 08 May '25

08 May '25

Hi, I am unable to deploy ingress service because "v4v6_flag" is undefined I couldn't find any information about this flag The ingress.yaml file used is similar with this one Any help would be greatly appreciated Steven service_type: ingress service_id: rgw placement: hosts: - ceph-node-06.cephlab.com - ceph-node-07.cephlab.com spec: backend_service: rgw.client virtual_ip: 192.168.122.152/24 frontend_port: 80 monitor_port: 1967 use_tcp_mode_over_rgw: True

3 8

Request for Recommendations: Tiering/Archiving from NetApp to Ceph (with stub file support)
by sacawulu 07 May '25

07 May '25

Hi all, We're exploring solutions to offload large volumes of data (on the order of petabytes) from our NetApp all-flash storage to our more cost-effective, HDD-based Ceph storage cluster, based on criteria such as: last access time older than X years. Ideally, we would like to leave behind a 'stub' or placeholder file on the NetApp side to preserve the original directory structure and potentially enable some sort of transparent access or recall if needed. This kind of setup is commonly supported by solutions like DataCore/FileFly, but as far as we can tell, FileFly doesn’t support Ceph as a backend and instead favors its own Swarm object store. Has anyone here implemented a similar tiering/archive/migration solution involving NetApp and Ceph? We’re specifically looking for: * Enterprise-grade tooling * Stub file support or similar metadata-preserving offload * Support and reliability (given the scale, we can’t afford data loss or inconsistency) * Either commercial or well-supported open source solutions Any do’s/don’ts, war stories, or product recommendations would be greatly appreciated. We’re open to paying for software or services if it brings us the reliability and integration we need. Thanks in advance! MJ

5 5

Update of MDS (non-cephadm cluster)
by Massimo Sgaravatto 07 May '25

07 May '25

I have done the update of the ceph software several times, but this is the first time I have to do an update (from quincy to reef) considering also ceph-fs I am not using cephadm I have 3 MDSs (2 active and one in standby) I'm looking at point 5 of: https://ceph.io/en/news/blog/2023/v18-2-0-reef-released/ If I have got it right It says to temporarily set max_mds to 1 and when you have only one active: 1 - turn off the MDS service on the 2 inactive MDS nodes 2- update and restart MDS on the active instance 3- restart (after updating the software: this is not written in the doc but it seems obvious to me) MDS on the other 2 instances 4 put max_mds back to 2 Did I understand correctly? But doesn't this mean to have a short period of time when the cluster is in error state (in 2, during the restart) and therefore with possible problems on the cephfs clients ? Thanks, Massimo

9 24

Need help with MDS stuck in readonly state
by Maxim Goncharov 07 May '25

07 May '25

Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool. I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again? Thank you, --Max

4 10

CFS IO500 ISC'25 Deadline 2 June 2025
by IO500 Committee 07 May '25

07 May '25

Call for Submission Submission Deadline: June 2nd, 2025 AoE The IO500 is now accepting and encouraging submissions for the upcoming 16th semi-annual IO500 Production and Research lists, in conjunction with ISC25. We are also accepting submissions to both the Production and Research 10 Client Node Challenges to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Webpage. The new ranked lists will be announced at the BoF [1]. We look forward to seeing many new results! Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance and the documentation of how the numbers are achieved. Support small to extreme-scale Research and Production HPC systems using flexible storage APIs Maximizing simplicity in running the benchmark suite while offering tunable parameters Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes runs with highly prescribed parameters in an attempt to determine a lower performance bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported Storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. The goals of the community are also multi-fold: Gather historical data for the sake of analysis and to aid predictions of storage futures Collect tuning information to share valuable performance optimizations across the community Encourage vendors and designers to optimize for workloads beyond "hero runs" Establish bounded expectations for users, procurers, and administrators Understand and be able to reproduce performance on storage systems Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. Benchmark Changes This year, we have added a 4KiB random read test phase at the end of the benchmark suite. By default, this phase will also run, but will not be included in the score nor affect if the score is valid or not. If this causes problems, it is possible to turn this off in the configuration file. Entries without this result will not be penalized. We will have the traditional two-week stabilization period. Unless a serious flaw is discovered, results submitted during stabilization will be considered for this event. The detailed information about the new phase can be read here: 4KiB Random Read [3] 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather We encourage you to submit [2] to join our community, and to attend the ISC'25 BoF [1] on Tuesday, Nov 19, 2024 at 12:15pm - 1:15pm EST, where we will announce the new IO500 Production and Research lists and their 10 Client Node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. We will also highlight new and interesting results with invited talk(s) at the BoF. [1] https://io500.org/pages/bof-isc25 [2] https://io500.org/submission [3] https://docs.google.com/document/d/1zr88byIhNIhVXp6KN6v4zGB828oNs2LDZSbIj-3… -- The IO500 Committee

1 0

reef 18.2.7 hotfix QE validation status
by Yuri Weinstein 06 May '25

06 May '25

Details of this release are summarized here: https://tracker.ceph.com/issues/71166 Release Notes - https://github.com/ceph/ceph/pull/63090 LRC upgrade - N/A Seeking approvals/reviews for: smoke - same as in 18.2.5 rados - Laura, Mark N, Igor approved? upgrade/quincy-x (reef) - Laura approved? upgrade/reef-p2p - unexpected failure in `test_librbd_python.sh` Ilya PTL (it did pass in 18.2.5 https://tracker.ceph.com/issues/70563 and was not ran in 18.2.6)

3 3

BlueFS race condition between truncate() and unlink()
by Frédéric Nass 06 May '25

06 May '25

Hi everyone, I've recently become aware of this bug [1] that affects Reef and Squid. When the bug is triggered, OSD will fail to start with "bluefs mount failed to replay log: (5) Input/output error" after crashing with "bluestore/AvlAllocator.cc: 173: FAILED ceph_assert(rs->start <= start)". The Red Hat KB solution [2] recommends redeploying the OSD as soon as possible to avoid having to deal with multiple crashed OSDs in the cluster. Reef v18.2.7 will fix this issue via PR #62840 [3]. Squid v19.2.3 should fix this issue via PR #62839 [4]. Thought it might be worth sharing. Regards, Frédéric. [1] https://tracker.ceph.com/issues/70747 [2] https://access.redhat.com/solutions/7113657 [3] https://github.com/ceph/ceph/pull/62840 [4] https://github.com/ceph/ceph/pull/62839

1 0

Reef: Dashboard still freezing on RGW overview after upgrade to 18.2.6
by Eugen Block 06 May '25

06 May '25

Hi, I just updated these two trackers: https://tracker.ceph.com/issues/69755 https://tracker.ceph.com/issues/67009 But I'm not sure if that has any impact on closed trackers. I just want to raise awareness that the issue still persists. Shoud I create a new tracker or will one of those above be reopened? Thanks! Eugen

2 2

Grafana vulnerability - cephadm deployment
by Michael Worsham 06 May '25

06 May '25

With cephadm reef (v18.2.4), is there a way to upgrade the Grafana that is incorporated into the docker build? Currently it's showing Grafana v9.4.7 (quay.io/ceph/ceph-grafana:9.4.7) and that is actually an open vulnerability per CVE-2023-1387 (Grafana 9.1.0 < 9.2.17, 9.3.x < 9.3.13, 9.4.x < 9.4.9 Information Disclosure Vulnerability). Grafana shows that it's fixed with version 9.4.9 and above (https://grafana.com/security/security-advisories/cve-2023-1387/) -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

7 14

Heads up -- 18.2.6 regression
by Dan van der Ster 06 May '25

06 May '25

Hi all, Just a quick heads up -- 18.2.6 has a bluefs regression [1] which was not present in 18.2.4. Please avoid upgrading until further notice. Regards, Dan [1] Introduced in https://tracker.ceph.com/issues/65356 and fixed in https://tracker.ceph.com/issues/69764 -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO https://clyso.com | dan.vanderster(a)clyso.com

9 18