- ceph-users - lists.ceph.io

Re: Unknown daemon type ceph-exporter
by John Mulligan 28 Jun '26

28 Jun '26

On Thursday, June 25, 2026 9:22:58 AM Eastern Daylight Time Eugen Block via ceph-users wrote: > Hi, > > is this a known issue that ceph-exporter is not recognized by cephadm? > > host1:~ # cephadm --image ceph:v20.2.2 ls | jq -r '.[] | > select(.name=="ceph-exporter.host1") | "\(.name) \(.version)"' > version for unknown daemon type ceph-exporter > ceph-exporter.host1 null > > > I don't see it in the daemon_types listed here: > > https://github.com/ceph/ceph/blob/main/src/cephadm/cephadmlib/listing_update > rs.py > The issue is that a lot of things in cephadm are implemented as a sequence of if statements (per service or service type) and as opposed to using common interfaces and so if something is forgotten or overlooked in the sequence of ifs it fails. Other parts of cephadm should handle ceph-exporter fine, it's just that this version gathering function doesn't handle it properly. It's a bug, please file a tracker. > And one more thing: why is the version empty in 'cephadm ls' output > while 'ceph orch ps' is capable of recognizing the version: > > host1:~ # ceph orch ps --daemon-type ceph-exporter -f json | jq -r > '.[] | select(.daemon_name=="ceph-exporter.host1") | "\(.version)"' > 20.2.2 > I'm not sure about this but it's clearly part of the bug where the system is following different code paths.

2 2

Recovery using OSDs after total mon store loss
by Eugen Block 28 Jun '26

28 Jun '26

Hi, although there is already a section in the upstream docs about mon store recovery using OSDs [0], I decided to write a blog post [1] with more details especially wrt cephadm managed clusters. I tested the procedure multiple times on two different lab clusters and it worked every time. Although I hope that nobody will have to use it (I doubt it though), I do hope that this will help speed up recovery. Maybe the upstream docs could be extended a bit as well. Although I have tested it extensively I may have forgotten something or made a mistake somewhere. I'd appreciate comments if you find anything like that. Have a great rest of the weekend! Eugen [0] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… [1] https://heiterbiswolkig.blogs.nde.ag/2026/06/28/cephadm-recover-mon-store-u…

1 0

mgmt-gateway: hostnames in nginx configuration
by Robert Sander 26 Jun '26

26 Jun '26

Hi, all upstream configuration sections in nginx.conf of the mgmt-gateway service use the IP addresses of the hosts in question. Except of the dashboard section which uses the hostname. Why is that implemented this way? We stumbled across this as the hostname resolves to an IP address different from the Ceph public network. This means port 8443 is not reachable via this IP. If the dashboard section would also use the IP address from the orchestrator's host list this would work. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Tentacle 20.2.1 - All MGRs failed
by Chris Palmer 26 Jun '26

26 Jun '26

Due to a number of power and heat events, I had to power down a small ceph cluster last night. There was only time to do a operating system shutdown of all nodes. When the systems rebooted a few hours later, all was ok except that all 3 MGRs would abort and restart until they reached the restart limit. This left the system with no MGRs and no orchestration. All OSD, MDS, RGW & monitoring subsystems were working as normally as could be expected without an MGR. Analysis of the log files showed each MGR aborting with the assertion below. Every one was in the telemetry thread. I disabled telemetry and the MGRs could be restarted normally. Questions: * What caused the assertion error? Was it related to the total shutdown? * Aborting the entire MGR seems overkill. Disabling the telemetry module and raising a warning seems more proportionate. * When running with no active MGR the client IO stats in ceph -s were frozen (at plausible non-zero values). Is that expected with no active MGR? * With no active MGR a large number of deep-scrubs started - roughly 3 times the maximum number we ever see were actively deep-scrubbing, with none in the scrub wait state, and response times were horrible. Is this expected with no MGR to control it properly? I had to temporarily disable scrubs and deep-scrubs as a workaround. I haven't yet tried 20.2.2, but I couldn't see anything obvious in the release notes. (Although there were about 3 PRs that I thought might have had the potential to upset exception handling...). Thanks, Chris { "crash_id": "2026-06-26T01:46:00.265207Z_b0579062-d65c-4a51-8dbc-d9837209aa80", "timestamp": "2026-06-26T01:46:00.265207Z", "process_name": "ceph-mgr", "entity_name": "mgr.ceph2", "ceph_version": "20.2.1", "utsname_hostname": "ceph2", "utsname_sysname": "Linux", "utsname_release": "7.0.12-201.fc44.x86_64", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Jun 11 01:30:16 UTC 2026", "utsname_machine": "x86_64", "os_name": "CentOS Stream", "os_id": "centos", "os_version_id": "9", "os_version": "9", "assert_condition": "cursor != root", "assert_func": "virtual void PyFormatter::close_section()", "assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h", "assert_line": 84, "assert_thread_name": "telemetry", "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: In function 'virtual void PyFormatter::close_section()' thread 7f752186e640 time 2026-06-26T01:46:00.263380+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: 84: FAILED ceph_assert(cursor != root)\n", "backtrace": [ "/lib64/libc.so.6(+0x3fc30) [0x7f75a9eb8c30]", "/lib64/libc.so.6(+0x8d05c) [0x7f75a9f0605c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f75aa9ed95b]", "/usr/bin/ceph-mgr(+0x6f7a5) [0x55ac462367a5]", "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x15a3) [0x55ac462d3143]", "/usr/bin/ceph-mgr(+0x12ba12) [0x55ac462f2a12]", "/lib64/libpython3.9.so.1.0(+0x15b4b7) [0x7f75ab5e34b7]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", "/lib64/libpython3.9.so.1.0(+0x14fa85) [0x7f75ab5d7a85]", "/lib64/libpython3.9.so.1.0(+0x138725) [0x7f75ab5c0725]", "_PyObject_CallMethod_SizeT()", "(PyModuleRunner::serve()+0x66) [0x55ac463759e6]", "(PyModuleRunner::PyModuleRunnerThread::entry()+0x139) [0x55ac463761e9]", "/lib64/libc.so.6(+0x8b31a) [0x7f75a9f0431a]" ] }

1 0

Q2 2026 Ceph Foundation Newsletter
by Anthony Middleton 25 Jun '26

25 Jun '26

Hey Ceph Friends, Here is the Q2 2026 Ceph Foundation newsletter. During the second quarter, the Ceph Foundation focused on strengthening collaboration across the project, expanding community programs, supporting documentation efforts, and investing in regional events. This edition highlights recent governance discussions, community elections, new contributor resources, Ceph Days events, and opportunities for community members to get involved. If you have an idea for a Ceph-related event, outreach effort, or project, we encourage you to submit a funding request <https://form.asana.com/?k=7aCHVRhp0x1Ga1nOCXlckQ&d=9283783873717> and work with the Foundation to help bring it to life. In This Issue - Board and CSC Alignment on Community Priorities - 2026 CSC Election Recap - Meet the New Ceph Technical Writer - Ceph Days Continue to Grow - Latest Ceph Releases - Community Committees and New Ways to Contribute Board and CSC Alignment on Community Priorities The Ceph Governing Board and Community Steering Committee (CSC) met this quarter to discuss the long-term health of the project, governance, documentation, community growth, and Foundation-supported initiatives. The discussion reaffirmed the distinct roles of each organization within the Ceph ecosystem. The CSC continues to provide technical governance for the project, while the Ceph Foundation focuses on community growth, events, outreach, and funding. The Linux Foundation provides operational, legal, and financial support that enables both groups to operate effectively. A significant area of focus was improving coordination between technical priorities and Foundation-supported programs. Current documentation initiatives include improvements to the documentation backport process, development of a new OSD operations guide, clarification of package and container documentation, and the removal of outdated content. The Board and CSC also discussed the future of community events and the importance of establishing clear goals and success metrics for Foundation-funded programs. Both groups reaffirmed their commitment to supporting collaboration between users, developers, operators, and organizations across the Ceph ecosystem. Key Takeaways - Continued alignment between the Board and CSC on project priorities. - Improved coordination around documentation and community outreach efforts. - Ongoing work to define success metrics for Foundation-funded programs and events. - Shared commitment to strengthening collaboration across the community. 2026 CSC Election Recap This year's CSC election introduced several improvements, including ranked-choice voting and candidate statements, helping create a more transparent and participatory election process. The election also highlighted opportunities to improve ballot communication and voter outreach in future cycles. Lessons learned from this year's process will help strengthen future elections and improve the overall voter experience. 2026 CSC Co-Chairs - Patrick Donnelly - Neha Ojha - Dan van der Ster Thank you to all candidates who participated and to every community member who cast a ballot. Read the full election recap for additional details. <https://pad.ceph.com/p/CSC_Election_April_2026> Meet the New Ceph Technical Writer Earlier this year, the Foundation announced its search for a Technical Writer to help improve the quality and accessibility of Ceph documentation. Following the interview process, the hiring committee selected Emmanuel Ameh for the role. Emmanuel will work closely with contributors across the community to improve documentation, reduce onboarding friction for new users, and help make Ceph documentation easier to navigate and maintain. If you are interested in joining the Ceph Tech Writing committee, please contact Emmanuel at eameh(a)contractor.linuxfoundation.org. Read Emmanuel's introduction on the Ceph blog <https://ceph.io/en/news/blog/2026/from-laminated-pages-to-petabytes/>to learn more about his background and goals for the role. Ceph Days Continue to Grow Community events remain one of the most effective ways for Ceph users and contributors to connect, share experiences, and learn from one another. During the second quarter, the community gathered for two Ceph Days events: Ceph Days Seattle 2026 Date: May 28, 2026 Sponsors: OSNexus and CLYSO Attendance: 30 community members John Shubeck, a Senior Storage Technical Specialist for IBM Object Storage, shared a detailed recap of the recent Ceph Days event in a new blog post <https://ceph.io/en/news/blog/2026/cd-seattle-cloud-blog/>. Ceph Days London 2026 Date: June 17, 2026 Lead Sponsor: IBM Attendance: 120 Here are a few of the photos of the thriving Ceph community in London <https://drive.google.com/drive/folders/1qKCSZyephZp42BBqaxw-mQ1iuGnUW6sO?us…> ! These events featured technical presentations, real-world deployment stories, community networking, and discussions about the future of Ceph. The Foundation continues to focus on supporting regional events, meetups, and community-led initiatives that help expand Ceph's reach and introduce new users to the project. Check the upcoming and tentative Ceph Days events for the rest of 2026. Ceph, OpenInfra, and KCD Days Korea - Lead Sponsors: Cloud Native Computing Foundation and OpenInfra - Date: Q3 - URL: TBD Ceph Days Zurich 2026 - Lead Sponsor: CLYSO - Date: Oct 14, 2026 - URL: https://ceph.io/en/community/events/2026/ceph-days-zurich/ Ceph Days New York - Lead Sponsor: Bloombreg - Date: Q3 - URL: TBD Ceph Days Chicago - Lead Sponsor: CLYSO - Date: Q4 - URL: TBD Get Involved Interested in organizing a meetup, hosting a Ceph Day, speaking at an event, or promoting Ceph in your region? Community participation is what makes these events successful. If you need support for a Ceph event, meetup, travel, or community campaign, submit a funding request <https://form.asana.com/?k=7aCHVRhp0x1Ga1nOCXlckQ&d=9283783873717>to the Ceph Foundation. Requests are reviewed by the Governing Board and Linux Foundation staff and considered based on available budget and community impact. Latest Ceph Releases The community published maintenance releases for both the Squid and Tentacle release series during the quarter. - v19.2.4 Squid Released <https://ceph.io/en/news/blog/2026/v19-2-4-squid-released/> - v20.2.1 Tentacle Released <https://ceph.io/en/news/blog/2026/v20-2-1-tentacle-released/> - v20.2.2 Tentacle Released <https://ceph.io/en/news/blog/2026/v20-2-2-tentacle-released/> Thank you to the many contributors, maintainers, testers, and reviewers who continue to support Ceph releases. Community Committees: More Ways to Contribute Community committees are coming soon. These committee will provide opportunities for contributors to help shape the future of Ceph beyond code contributions alone. Whether your interests include events, documentation, outreach, onboarding, or community programs, there are many ways to participate. Committee participation will be shared through our social channel and mailing list <https://ceph.io/en/community/connect/>. Watch upcoming community communications for additional opportunities to join working groups and contribute to Foundation-led initiatives. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Unknown daemon type ceph-exporter
by Eugen Block 25 Jun '26

25 Jun '26

Hi, is this a known issue that ceph-exporter is not recognized by cephadm? host1:~ # cephadm --image ceph:v20.2.2 ls | jq -r '.[] | select(.name=="ceph-exporter.host1") | "\(.name) \(.version)"' version for unknown daemon type ceph-exporter ceph-exporter.host1 null I don't see it in the daemon_types listed here: https://github.com/ceph/ceph/blob/main/src/cephadm/cephadmlib/listing_updat… And one more thing: why is the version empty in 'cephadm ls' output while 'ceph orch ps' is capable of recognizing the version: host1:~ # ceph orch ps --daemon-type ceph-exporter -f json | jq -r '.[] | select(.daemon_name=="ceph-exporter.host1") | "\(.version)"' 20.2.2 Thanks! Eugen

1 0

Re: PSA: be aware of osd_heartbeat_min_healthy_ratio
by Alex Gorbachev 25 Jun '26

25 Jun '26

> > since I just stumbled across 'ceph cephadm config-check', is it > enabled in this cluster? Could the MGR have warned you with the health > check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before > deploying OSDs? It seems that it's not enabled by default. This plausibly could have warned after hosts were added, before OSD deployment, assuming the hosts had already been added to cephadm and the check was enabled. And one more question: could 'osd_crush_initial_weight = 0' have > prevented this from happening? We have that set in our own cluster to > carefully check if all OSDs are deployed correctly before reweighting. I doubt this would have helped because the CRUSH weight just controls data placement, but osd heartbeat comes at a lower level than that, and that would have fired anyway. -- Alex Gorbachev ISS/Storcium On Wed, Jun 24, 2026 at 5:50 AM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > since I just stumbled across 'ceph cephadm config-check', is it > enabled in this cluster? Could the MGR have warned you with the health > check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before > deploying OSDs? > > It seems that it's not enabled by default. > > And one more question: could 'osd_crush_initial_weight = 0' have > prevented this from happening? We have that set in our own cluster to > carefully check if all OSDs are deployed correctly before reweighting. > > Thanks, > Eugen > > Zitat von Alex Gorbachev via ceph-users <ceph-users(a)ceph.io>: > > > osd_heartbeat_min_healthy_ratio determines the minimum percentage of > peers > > an OSD must be able to reach before it can accurately mark itself up or > > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > > heartbeat decisions are made independently of monitor quorum health. An > OSD > > can begin reporting peers as down based on heartbeat failures even when > > monitor quorum remains healthy. > > I was adding a second rack of OSD servers to an existing installation, > > assuming that the worst that could happen is that those will be > > unreachable. I also missed setting up the cluster_network VLAN on one > side > > (in the 4x100GbE bond). > > > > I added all the new OSDs at once. The moment that happened, the existing > > OSDs saw 50% unreachable, and all production OSDs promptly shut down. > > > > The take away in my recommendation is to add OSDs in such a way that > > they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time > to > > fix connectivity issues before dealing with down OSDs, as I experienced. > > -- > > Alex Gorbachev > > ISS/Storcium > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Announcing go-ceph v0.40.0
by John Mulligan via dev 24 Jun '26

24 Jun '26

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.40.0 Enhancements include additions to the smb admin package, and a new command builder package. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. PS. The actual release was created June 16th but we did not announce it on the mailing list until now. Apologies for the delayed announcement.

1 0

Re: OSD don't ant to start
by Gilles Mocellin 24 Jun '26

24 Jun '26

Hi ! Of course, I reset thoses settings right after starting the OSD. When I restart 20 times, it was with the default values (40). I only tried once with 1000, and as it didn't worked, I posted my first mail here... If I had tried a second time, it would have probably worked. Le 24 juin 2026 08:42:21 GMT+02:00, Eugen Block via ceph-users <ceph- users(a)ceph.io> a écrit : >Although I wouldn't have expected it to take 20 restarts, it's still good that it helped resolving the issue. I'm not entirely sure of the implications having increased osd_map_share_max_epochs that much. Did you set it back to the default? If not, I recommend to monitor disk usage closely (monitor store), and maybe OSD memory usage as well. > >Zitat von Gilles Mocellin via ceph-users <ceph-users(a)ceph.io>: > >> Le mardi 23 juin 2026, 11:18:45 heure d’été d’Europe centrale Eugen Block via >> ceph-users a écrit : >>> Hi, >>> >>> this sounds familiar: >>> >>> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ Y2YKHQZSBYA7 >>> GS4D4YSPSNMWNE5QMETH/#ZTUQ2CE5547LOPBGQOXHUF63F6UU3RKO >>> >>> You could set the OSD "in" and then retry starting it a couple of times. >>> >>> Regards, >>> Eugen >> >> Yes that sound really that. >> There's a RedHat solution about that (behind a closed wall), they suggest to >> set to 1000 thoses settings : >> osd_map_share_max_epochs and another (I can't access it from here...). >> >> It didn't work at first so I thought it wasn't it. >> But I certainly was far behind more than 1000... >> My OSD stayed down/out for at least 4 days. >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: PSA: be aware of osd_heartbeat_min_healthy_ratio
by Alex Gorbachev 24 Jun '26

24 Jun '26

Hi Wannes, The 33% threshold is not applied against the raw total number of OSDs in the cluster. It is applied against the OSD’s heartbeat peer set. With the cluster network VLAN missing, public-network connectivity to MONs could still work, while OSD heartbeats, replication, recovery, and peering over the cluster network failed. -- Alex Gorbachev ISS/Storcium On Wed, Jun 24, 2026 at 1:25 AM Wannes Smet <smet(a)icsense.com> wrote: > Hi Alex, > > Lessons learned are always valuable information to share! Thanks. > > Pretty sure I'm missing out on a subtlety here. I can't really put my > finger on it why only osd_heartbeat_min_healthy_ratio caused this, so some > questions from my side. > > At first glance I'd think 50% is still larger than 33%? I also assume OSDs > can be peers even if they never were able to heartbeat each other before > due to crush. > > Maybe this happened because that peering sets aren't perfectly balanced? > So you might have gotten slightly below that 50%, but still above 33%. > Meaning you're <17% away from 33%, "critical mass". But then some > additional peering/activity or the like happened due to the "shock change", > and the whole effectively dropped below 33% cluster wide due to some > cascading effects? > > Thanks, > > Wannes > ------------------------------ > *From:* Alex Gorbachev via ceph-users <ceph-users(a)ceph.io> > *Sent:* Tuesday, June 23, 2026 18:10 > *To:* ceph-users <ceph-users(a)ceph.io> > *Subject:* [ceph-users] PSA: be aware of osd_heartbeat_min_healthy_ratio > > osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers > an OSD must be able to reach before it can accurately mark itself up or > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > heartbeat decisions are made > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers > an OSD must be able to reach before it can accurately mark itself up or > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > heartbeat decisions are made independently of monitor quorum health. An OSD > can begin reporting peers as down based on heartbeat failures even when > monitor quorum remains healthy. > I was adding a second rack of OSD servers to an existing installation, > assuming that the worst that could happen is that those will be > unreachable. I also missed setting up the cluster_network VLAN on one side > (in the 4x100GbE bond). > > I added all the new OSDs at once. The moment that happened, the existing > OSDs saw 50% unreachable, and all production OSDs promptly shut down. > > The take away in my recommendation is to add OSDs in such a way that > they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to > fix connectivity issues before dealing with down OSDs, as I experienced. > -- > Alex Gorbachev > ISS/Storcium > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0