- ceph-users - lists.ceph.io

Ceph Steering Committee Meeting 2026-08-03
by Ilya Dryomov 03 Aug '26

03 Aug '26

Hello, - first public foundation board meeting recording is available (Neha) - https://www.youtube.com/watch?v=gPyu9KLVD64 - CDS is starting this week! (Anthony) - https://ceph.io/en/community/events/2026/ceph-developer-summit-vampire/ - performance session led by Mark Nelson is yet to be scheduled, the post will be updated when that happens - upcoming Ceph Days PSA (Patrick) - Zurich: https://ceph.io/en/community/events/2026/ceph-days-zurich/ - Chicago: https://ceph.io/en/community/events/2026/ceph-days-chicago/ - merge window process (Ilya, Patrick) - the date in the calendar is the "branch closes on" not the "PR must be ready for QA by" date - this is in large part due to the fact that the time that it takes for the PR to be picked up into a QA run and the time that it takes for the test results to be reviewed are currently outside of the release manager's control - Patrick will be introducing a new needs-qa alike automated label for backport PRs to formalize the "PR is ready for QA" requirements and make such PRs more discoverable - component leads are free to do their own QA runs when needed - 21.1.1 status - merge window closes in two days - barring negative user feedback or last minute blockers this would be the last umbrella RC - "happy path" tentative release date remains to be Aug 31 - squid EOL plan - at least one merge window/release after umbrella is out Thanks, Ilya

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Wannes Smet 03 Aug '26

03 Aug '26

Hi, Also make sure every node has an _admin label. (ceph orch host label add <hostname> _admin. Verify with ceph orch host ls. Not proud of it, but I once chown-ed /var/lib/ceph on mulptiple nodes. Due to that, our Ceph cluster lost quorum because all the containers crashed. A learning experience I'll never forget #facepalm. I'll definitely think twice before I manupulate /var/lib/ceph ever again ;) . Only if your cluster is 100% test with no production workload on it: Why not try to shut down one node and see what happens on the surviving nodes. Do you get back to HEALTH_OK after you restart the downed node? It should and if it doesn't you're lucky to find out before you went in production. Also, do a "rolling" reboot of all of your nodes. Do you get back to HEALTH_OK after a reboot has finished? This for all the nodes? Anything else breaks if you shut down one node? Wannes. ________________________________ From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, July 21, 2026 22:31 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: How to administer Ceph from any node without bootstrap dependency Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of the admin keyring via orchestrator and the _admin label [0]. That enables you to manage the cluster from any admin node. Not sure what you mean by "bootstrap file" though... If you'd synced the /var/lib/ceph directory, you'd overwrite all the daemons but one, including keyring etc. So please don't do that, that would break your cluster. Regards, Eugen [0] https://urldefense.com/v3/__https://docs.ceph.com/en/latest/cephadm/operati… Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > Hi all, > > I am currently deploying a Ceph cluster with cephadm on 3 nodes with > MON, MGR, and MDS roles. OSDs will be added later. > > My goal is for these 3 nodes to be fully interchangeable, with the > ability to administer the cluster from any of them without risk if a > node fails. > > I have identified that the solution involves synchronizing > /var/lib/ceph/ (excluding the bootstrap file) across all nodes. > > However: > > - Is this the right approach to achieve this goal? > > - Which tool should I use for reliable bidirectional synchronization? > > - What are the potential risks (conflicts, corruption) and precautions > to take? > > Have you implemented such a configuration in production before? > > Thank you for your advice. > > -- > Michel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Guidance Required for Changing Ceph Squid Development Version to Stable Release
by Eugen Block 03 Aug '26

03 Aug '26

Hi, my first recommendation would be to not use a dev release in a production cluster, but that's not really your question. ;-) I haven't used 19.3.0 ever so I'm not sure how it's going to react if you tried to adopt with 19.2.3. Since downgrading Ceph never has been supported this might either just fail or maybe even cause an inconsistency, hard to tell. So my recommendation would be to upgrade to 20 first, then adopt. Regards, Eugen Zitat von Kathirvel T via ceph-users <ceph-users(a)ceph.io>: > Hello Ceph Community, > > We are looking for guidance regarding our existing Ceph cluster. > > Our cluster was deployed using ceph-ansible and is currently running > the Squid development release (v19.3.0). We are planning to adopt > the cluster into cephadm using the cephadm-adopt playbook provided > by ceph-ansible. > > Before proceeding, we would like to understand the recommended > approach for changing our Ceph version from the Squid development > release (v19.3.0) to the stable Squid release (v19.2.3). > > Cluster Details > > Deployment: ceph-ansible > > Current Version: Squid Development (v19.3.0) > > Health: HEALTH_OK > > Hosts: 4 > > MONs: 3 > > MGRs: 1 Active, 2 Standby > > OSDs: 15 (15 Up / 15 In) > > RGW: 1 daemon > > Pools: 15 > > Objects: ~1.64 million > > Usage: 17 TiB used, 9.2 TiB / 26 TiB avail > > > > > We have the following questions: > > Is it possible to change from the Squid development version > (v19.3.0) to the stable Squid release (v19.2.3)? > > Is this a supported approach, or is there a recommended alternative? > > Should we change to the stable release before adopting the cluster > into cephadm, or is it better to perform the cephadm adoption first > and then update the version? > > Are there any known limitations, prerequisites, or best practices we > should be aware of before proceeding? > > > Any guidance, recommendations, or references to relevant > documentation would be greatly appreciated. > > > > Thanks & Regards, > > > > > > > > > > > > > > Kathirvel T > > Associate R&D Engineer mailto:manoj.kumar@zybisys.com > > kathirvel.t(a)zybisys.com > > https://www.zybisys.com > > > > > > https://in.linkedin.com/company/zybisys > https://www.facebook.com/Zybisys/ > https://www.instagram.com/ZyBiSys/ > https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ > Disclaimer : The content of this email and any files transmitted > with it are confidential and intended solely for the use of the > individual or entity to which they are addressed. If you have > received this email in error, please notify the sender and remove > the messages from your system. If you are not the named addressee, > it is strictly forbidden for you to share, circulate, distribute or > copy any part of this e-mail to any third party without the written > consent of the sender. > > > > E-mail transmission cannot be guaranteed to be secured or error free > as information could be intercepted, corrupted, lost, destroyed, > arrive late, incomplete, or may contain viruses. Therefore, we do > not accept liability for any errors or omissions in the contents of > this message, which arise as a result of e-mail transmission. The > recipient should check this e-mail and any attachments for the > presence of viruses. The company accepts no liability for any damage > caused by any virus transmitted by this email." > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Marek Szuba 02 Aug '26

02 Aug '26

Thank you for all the responses! OK, so it's about possible resource contention and not some sort of scary data-corruption problem. That's fine by me, as we only use CephFS mounted directly on storage nodes for operations requiring elevated privileges - nothing resource-intensive. Cheers, -- MS

1 0

Re: Many empty objects in CephFS data pool
by Eugen Block 31 Jul '26

31 Jul '26

Hi, are you using snapshots on CephFS as well? Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Hi, > > Am 31.07.26 um 9:11 AM schrieb Dario Graña: > >> here is an snippet from the documentation[0] that explains this behavior: >> /The data pool used to create the file system is the “default” data >> pool and the location for storing all inode backtrace information, >> used for hard link management and disaster recovery. For this >> reason, all inodes created in CephFS have at least one object in >> the default data pool. If erasure-coded pools are planned for the >> file system, it is usually better to use a replicated pool for the >> default data pool to improve small-object write and read >> performance for updating backtraces./ > > Thanks for the hint. I already suspected something like this. > > It is so strange as there are many more empty objects in the first > replicated data pool than there are "real" file objects in the EC > data pool. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Many empty objects in CephFS data pool
by Dario Graña 31 Jul '26

31 Jul '26

Hello, here is an snippet from the documentation[0] that explains this behavior: *The data pool used to create the file system is the “default” data pool and the location for storing all inode backtrace information, used for hard link management and disaster recovery. For this reason, all inodes created in CephFS have at least one object in the default data pool. If erasure-coded pools are planned for the file system, it is usually better to use a replicated pool for the default data pool to improve small-object write and read performance for updating backtraces.* [0] https://docs.ceph.com/en/latest/cephfs/createfs/ On Thu, Jul 30, 2026 at 12:04 PM Loïc Tortay via ceph-users < ceph-users(a)ceph.io> wrote: > On 30/07/2026 11:42, Robert Sander via ceph-users wrote: > > Hi, > > > > we have a CephFS with a replicated data pool and an additional EC data > > pool. All files store their data in the EC pool. > > > > But "ceph df" shows that the replicated data pool contains over 200 > > million objects with a total size of 700 bytes. So the vast majority are > > empty objects. > > > > How can this be? > > > Hello, > The replicated pool is the "default pool" and contains files metadata ? > (unlike the "metadata" pool) > > Like in this filesystem with ~1.4 billion files and 85 million directories: > POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL > cephfs_EC_data 3 8192 6.6 PiB 3.00G 8.3 PiB 79.90 1.7 PiB > cephfs_metadata 4 256 778 GiB 84.70M 2.3 TiB 3.38 22 TiB > cephfs_default 5 128 0 B 1.37G 0 B 0 22 TiB > > > Loïc. > -- > | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre | > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: v20.2.3 tentacle QE validation status
by Venky Shankar 31 Jul '26

31 Jul '26

Hey Yuri, On Mon, Jul 27, 2026 at 7:37 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > https://tracker.ceph.com/issues/77256#note-5 > > Release Notes - TBD > LRC upgrade - TBD > > PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky fs approved -- https://tracker.ceph.com/issues/77256#note-15 > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > krbd: Ilya > > nvmeof: Aviv, Vallari > > upgrade: Kamoltat, Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > Please reply with your approval or flag any concerns. -- Cheers, Venky

1 0

Ceph Developer Summit 2026
by Anthony Middleton 30 Jul '26

30 Jul '26

Hey Ceph friends, The Ceph Developer Summit begins next week! From August 4–13, the Ceph community comes together online to shape the next major release, Vampire, coming in Spring 2027. Nine sessions across two weeks will cover RADOS, RBD, RGW, CephFS, Dashboard, Cephadm, Telemetry/Ceph-MGR, NVMe-oF gateway, and Crimson/SeaStore — led by the engineers building each of these components. Whether you're a core contributor, an operator running Ceph in production, or just curious about what's next for the project, every session is open to join. Bring your questions, your feedback, and your ideas for where these features should go. Each session will be recorded and shared on our YouTube channel. Full schedule, speakers, and session links below: https://t.ly/CDS-Vampire Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: v20.2.3 tentacle QE validation status
by Kamoltat Sirivadhna 30 Jul '26

30 Jul '26

upgrade approved (Note that the many failures we saw was due to a known issue regarding stress split and mds upgrade) On Mon, Jul 27, 2026 at 10:09 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Details of this release are summarized here: > https://tracker.ceph.com/issues/77256#note-5 > > Release Notes - TBD > LRC upgrade - TBD > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > krbd: Ilya > > nvmeof: Aviv, Vallari > > upgrade: Kamoltat, Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > Please reply with your approval or flag any concerns. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Kamoltat (Junior) Sirivadhna (HE/HIM) Senior SoftWare Engineer - IBM Ceph Storage ksirivad(a)redhat.com T: (857) <(919)716-5348>253-8927

1 0

Using CephFS for encrypted Kubernetes volumes
by Stefan Möding 30 Jul '26

30 Jul '26

Hi, we are trying to use Ceph (Tentacle 20.2.2) as storage provider for multiple Kubernetes clusters. We are trying to implement the following security measures: * Each Kubernetes cluster should use a dedicated RADOS namespace for its objects. * The volumes should use fscrypt encryption. We can deploy pods successfully as long as we don’t try to enable fscrypt. With encryption enabled the pod shows the following logs: AttachVolume.Attach succeeded for volume "pvc-0923605d-3993-4d99-a4a3-d41749ead831" MountVolume.MountDevice failed for volume "pvc-0923605d-3993-4d99-a4a3-d41749ead831" : rpc error: code = Internal desc = failed to lock volume ID 64ab25ab-9d66-4508-be74-13c02825f40e: rados: ret=-1, Operation not permitted There is a closed issue that mentions a length limitation when creating a lock to initialize the encryption: https://github.com/ceph/ceph-csi/issues/5419 I also tried the mentioned workaround but that does not help. There has also been a discussion that the ’x’ permission is needed in the OSD capabilities, which we have set: mon_caps: 'allow r fsname=root' mgr_caps: 'allow rw' osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, allow rw namespace=k8s-test tag cephfs data=root' mds_caps: 'allow r fsname=root path=/volumes, allow rws fsname=root path=/volumes/k8s-test' Does anybody have a working setup with encryption and RADOS namespaces? -- Stefan

1 0