- ceph-users - lists.ceph.io

Re: Guidance Required for Changing Ceph Squid Development Version to Stable Release
by Eugen Block 03 Aug '26

03 Aug '26

Hi, my first recommendation would be to not use a dev release in a production cluster, but that's not really your question. ;-) I haven't used 19.3.0 ever so I'm not sure how it's going to react if you tried to adopt with 19.2.3. Since downgrading Ceph never has been supported this might either just fail or maybe even cause an inconsistency, hard to tell. So my recommendation would be to upgrade to 20 first, then adopt. Regards, Eugen Zitat von Kathirvel T via ceph-users <ceph-users(a)ceph.io>: > Hello Ceph Community, > > We are looking for guidance regarding our existing Ceph cluster. > > Our cluster was deployed using ceph-ansible and is currently running > the Squid development release (v19.3.0). We are planning to adopt > the cluster into cephadm using the cephadm-adopt playbook provided > by ceph-ansible. > > Before proceeding, we would like to understand the recommended > approach for changing our Ceph version from the Squid development > release (v19.3.0) to the stable Squid release (v19.2.3). > > Cluster Details > > Deployment: ceph-ansible > > Current Version: Squid Development (v19.3.0) > > Health: HEALTH_OK > > Hosts: 4 > > MONs: 3 > > MGRs: 1 Active, 2 Standby > > OSDs: 15 (15 Up / 15 In) > > RGW: 1 daemon > > Pools: 15 > > Objects: ~1.64 million > > Usage: 17 TiB used, 9.2 TiB / 26 TiB avail > > > > > We have the following questions: > > Is it possible to change from the Squid development version > (v19.3.0) to the stable Squid release (v19.2.3)? > > Is this a supported approach, or is there a recommended alternative? > > Should we change to the stable release before adopting the cluster > into cephadm, or is it better to perform the cephadm adoption first > and then update the version? > > Are there any known limitations, prerequisites, or best practices we > should be aware of before proceeding? > > > Any guidance, recommendations, or references to relevant > documentation would be greatly appreciated. > > > > Thanks & Regards, > > > > > > > > > > > > > > Kathirvel T > > Associate R&D Engineer mailto:manoj.kumar@zybisys.com > > kathirvel.t(a)zybisys.com > > https://www.zybisys.com > > > > > > https://in.linkedin.com/company/zybisys > https://www.facebook.com/Zybisys/ > https://www.instagram.com/ZyBiSys/ > https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ > Disclaimer : The content of this email and any files transmitted > with it are confidential and intended solely for the use of the > individual or entity to which they are addressed. If you have > received this email in error, please notify the sender and remove > the messages from your system. If you are not the named addressee, > it is strictly forbidden for you to share, circulate, distribute or > copy any part of this e-mail to any third party without the written > consent of the sender. > > > > E-mail transmission cannot be guaranteed to be secured or error free > as information could be intercepted, corrupted, lost, destroyed, > arrive late, incomplete, or may contain viruses. Therefore, we do > not accept liability for any errors or omissions in the contents of > this message, which arise as a result of e-mail transmission. The > recipient should check this e-mail and any attachments for the > presence of viruses. The company accepts no liability for any damage > caused by any virus transmitted by this email." > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Marek Szuba 02 Aug '26

02 Aug '26

Thank you for all the responses! OK, so it's about possible resource contention and not some sort of scary data-corruption problem. That's fine by me, as we only use CephFS mounted directly on storage nodes for operations requiring elevated privileges - nothing resource-intensive. Cheers, -- MS

1 0

Re: Many empty objects in CephFS data pool
by Eugen Block 31 Jul '26

31 Jul '26

Hi, are you using snapshots on CephFS as well? Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Hi, > > Am 31.07.26 um 9:11 AM schrieb Dario Graña: > >> here is an snippet from the documentation[0] that explains this behavior: >> /The data pool used to create the file system is the “default” data >> pool and the location for storing all inode backtrace information, >> used for hard link management and disaster recovery. For this >> reason, all inodes created in CephFS have at least one object in >> the default data pool. If erasure-coded pools are planned for the >> file system, it is usually better to use a replicated pool for the >> default data pool to improve small-object write and read >> performance for updating backtraces./ > > Thanks for the hint. I already suspected something like this. > > It is so strange as there are many more empty objects in the first > replicated data pool than there are "real" file objects in the EC > data pool. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Many empty objects in CephFS data pool
by Dario Graña 31 Jul '26

31 Jul '26

Hello, here is an snippet from the documentation[0] that explains this behavior: *The data pool used to create the file system is the “default” data pool and the location for storing all inode backtrace information, used for hard link management and disaster recovery. For this reason, all inodes created in CephFS have at least one object in the default data pool. If erasure-coded pools are planned for the file system, it is usually better to use a replicated pool for the default data pool to improve small-object write and read performance for updating backtraces.* [0] https://docs.ceph.com/en/latest/cephfs/createfs/ On Thu, Jul 30, 2026 at 12:04 PM Loïc Tortay via ceph-users < ceph-users(a)ceph.io> wrote: > On 30/07/2026 11:42, Robert Sander via ceph-users wrote: > > Hi, > > > > we have a CephFS with a replicated data pool and an additional EC data > > pool. All files store their data in the EC pool. > > > > But "ceph df" shows that the replicated data pool contains over 200 > > million objects with a total size of 700 bytes. So the vast majority are > > empty objects. > > > > How can this be? > > > Hello, > The replicated pool is the "default pool" and contains files metadata ? > (unlike the "metadata" pool) > > Like in this filesystem with ~1.4 billion files and 85 million directories: > POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL > cephfs_EC_data 3 8192 6.6 PiB 3.00G 8.3 PiB 79.90 1.7 PiB > cephfs_metadata 4 256 778 GiB 84.70M 2.3 TiB 3.38 22 TiB > cephfs_default 5 128 0 B 1.37G 0 B 0 22 TiB > > > Loïc. > -- > | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre | > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: v20.2.3 tentacle QE validation status
by Venky Shankar 31 Jul '26

31 Jul '26

Hey Yuri, On Mon, Jul 27, 2026 at 7:37 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > https://tracker.ceph.com/issues/77256#note-5 > > Release Notes - TBD > LRC upgrade - TBD > > PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky fs approved -- https://tracker.ceph.com/issues/77256#note-15 > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > krbd: Ilya > > nvmeof: Aviv, Vallari > > upgrade: Kamoltat, Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > Please reply with your approval or flag any concerns. -- Cheers, Venky

1 0

Ceph Developer Summit 2026
by Anthony Middleton 30 Jul '26

30 Jul '26

Hey Ceph friends, The Ceph Developer Summit begins next week! From August 4–13, the Ceph community comes together online to shape the next major release, Vampire, coming in Spring 2027. Nine sessions across two weeks will cover RADOS, RBD, RGW, CephFS, Dashboard, Cephadm, Telemetry/Ceph-MGR, NVMe-oF gateway, and Crimson/SeaStore — led by the engineers building each of these components. Whether you're a core contributor, an operator running Ceph in production, or just curious about what's next for the project, every session is open to join. Bring your questions, your feedback, and your ideas for where these features should go. Each session will be recorded and shared on our YouTube channel. Full schedule, speakers, and session links below: https://t.ly/CDS-Vampire Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: v20.2.3 tentacle QE validation status
by Kamoltat Sirivadhna 30 Jul '26

30 Jul '26

upgrade approved (Note that the many failures we saw was due to a known issue regarding stress split and mds upgrade) On Mon, Jul 27, 2026 at 10:09 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Details of this release are summarized here: > https://tracker.ceph.com/issues/77256#note-5 > > Release Notes - TBD > LRC upgrade - TBD > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > krbd: Ilya > > nvmeof: Aviv, Vallari > > upgrade: Kamoltat, Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > Please reply with your approval or flag any concerns. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Kamoltat (Junior) Sirivadhna (HE/HIM) Senior SoftWare Engineer - IBM Ceph Storage ksirivad(a)redhat.com T: (857) <(919)716-5348>253-8927

1 0

Using CephFS for encrypted Kubernetes volumes
by Stefan Möding 30 Jul '26

30 Jul '26

Hi, we are trying to use Ceph (Tentacle 20.2.2) as storage provider for multiple Kubernetes clusters. We are trying to implement the following security measures: * Each Kubernetes cluster should use a dedicated RADOS namespace for its objects. * The volumes should use fscrypt encryption. We can deploy pods successfully as long as we don’t try to enable fscrypt. With encryption enabled the pod shows the following logs: AttachVolume.Attach succeeded for volume "pvc-0923605d-3993-4d99-a4a3-d41749ead831" MountVolume.MountDevice failed for volume "pvc-0923605d-3993-4d99-a4a3-d41749ead831" : rpc error: code = Internal desc = failed to lock volume ID 64ab25ab-9d66-4508-be74-13c02825f40e: rados: ret=-1, Operation not permitted There is a closed issue that mentions a length limitation when creating a lock to initialize the encryption: https://github.com/ceph/ceph-csi/issues/5419 I also tried the mentioned workaround but that does not help. There has also been a discussion that the ’x’ permission is needed in the OSD capabilities, which we have set: mon_caps: 'allow r fsname=root' mgr_caps: 'allow rw' osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, allow rw namespace=k8s-test tag cephfs data=root' mds_caps: 'allow r fsname=root path=/volumes, allow rws fsname=root path=/volumes/k8s-test' Does anybody have a working setup with encryption and RADOS namespaces? -- Stefan

1 0

Re: Many empty objects in CephFS data pool
by Loïc Tortay 30 Jul '26

30 Jul '26

On 30/07/2026 11:42, Robert Sander via ceph-users wrote: > Hi, > > we have a CephFS with a replicated data pool and an additional EC data > pool. All files store their data in the EC pool. > > But "ceph df" shows that the replicated data pool contains over 200 > million objects with a total size of 700 bytes. So the vast majority are > empty objects. > > How can this be? > Hello, The replicated pool is the "default pool" and contains files metadata ? (unlike the "metadata" pool) Like in this filesystem with ~1.4 billion files and 85 million directories: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_EC_data 3 8192 6.6 PiB 3.00G 8.3 PiB 79.90 1.7 PiB cephfs_metadata 4 256 778 GiB 84.70M 2.3 TiB 3.38 22 TiB cephfs_default 5 128 0 B 1.37G 0 B 0 22 TiB Loïc. -- | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Re: Many empty objects in CephFS data pool
by Janne Johansson 30 Jul '26

30 Jul '26

> > > we have a CephFS with a replicated data pool and an additional EC data > pool. All files store their data in the EC pool. > > But "ceph df" shows that the replicated data pool contains over 200 > million objects with a total size of 700 bytes. So the vast majority are > empty objects. > > How can this be? > I don't do cephfs, but at least radosgw likes to stash information into zero-sized objects, so the sum of the sizes of those objects was small, but the data usage would of course be larger than zero on those rgw pools. Perhaps cephfs does the same? "ceph df detail" on one of our rgw clusters show this: POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR default.rgw.buckets.index 7 512 51 GiB 0 B 51 GiB 9.57k 154 GiB 0 B 154 GiB 0.02 225 TiB N/A N/A N/A 0 B 0 B the "DATA" column is zero, even though there are 9.57k objects in there, but OMAP is lots more, probably meaning it stashed information into the metadata of said zero-sized objs. If we stat one of the .dir.xxxx objects, it says: rados stat -p default.rgw.buckets.index ".dir.b9865dc1-e648-4479-abcd-ed812345678.144664284.33737.1.20" default.rgw.buckets.index/.dir.b9865dc1-e648-4479-abcd-ed87654321.144664284.33737.1.20 mtime 2026-07-30T09:48:05.000000+0000, size 0 -- May the most significant bit of your life be positive.

1 0