- ceph-users - lists.ceph.io

v21.1.0 Umbrella RC0 released
by Yuri Weinstein 20 Jul '26

20 Jul '26

This is the first release candidate for Umbrella. Ubuntu 22.04, 24.04, CentOS 9, Rocky Linux 10, Debian 12 (Bookworm) and Debian 13 (Trixie) were built for this RC0. Please note: these release notes are a draft and incomplete. Feature highlights: - RADOS/OSD: New Optimized ("Fast") Erasure Coding I/O path (direct/sync reads, OMAP support, deep scrub); mon RocksDB backup/restore; stretch cluster improvements; mClock scheduler refactor; continued Crimson/Seastore EC and perf work. - RGW: Full object dedup (tech preview); bucket logging; cloud restore from Glacier/tape; AES-256-GCM encryption + security fixes; expanded IAM/Account and STS/OIDC support. - Dashboard: Large-scale migration to IBM Carbon design; expanded NVMe/TCP management; SMB cluster support; OAuth2 SSO; Angular 19. - cephadm/Orchestrator: New unified certmgr for TLS cert management; SMB/Samba managed service; expanded NVMe-oF gateway spec; new hardware monitoring (Atollon BMCs); NFS-Ganesha active-active mode. - CephFS/MDS: FSCrypt client-side encryption; multi-threaded cephfs-mirror; new python bindings; subvolume improvements. - ceph-volume: Migrated to dataclasses + udev-based device detection (perf/reliability); seastore OSD support. - Build/Packaging/QA: Ubuntu 24.04 (Noble) and Rocky Linux 10 support added; the Ceph container's base OS image has changed to rockylinux:10; CMake 3.22.1+/CMake-4; C++23. - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.0.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: f7eb5f7c8097648828a6b31f4590c3ee3661c95a

1 0

Re: Survey: Is anyone using 32-bit builds of the CephFS kernel client?
by Xiubo Li 17 Jul '26

17 Jul '26

Just forward this since I wasn't in the ceph-users mail list and it was reject yesterday. Thanks Xiubo Li On Thu, 16 Jul 2026 at 16:53, Xiubo Li <xiubo.li(a)clyso.com> wrote: > > Hi everyone, > > We are considering a performance improvement for the CephFS kernel > client (kclient) that would be significantly simpler and cleaner to > implement as 64-bit-only. Supporting 32-bit in this case adds > non-trivial complexity and ongoing maintenance burden, so we'd like to > understand whether continuing 32-bit support is still necessary. > > Before making any decision, we want to hear from the community: > > - Are you currently using, or planning to use, 32-bit (arm/armhf, > i386/i686) builds of the CephFS kernel client? > - If yes, is this for production workloads? Or legacy/testing purposes? > - What is your platform (distribution, kernel version, architecture)? > > If there are no active 32-bit users, we would proceed with a > 64-bit-only implementation and clearly document the deprecation in the > release notes with appropriate warnings. > > If 32-bit support is still needed, we will explore alternative > approaches — for example, keeping the current code path for 32-bit > while the new fast path is 64-bit-only, or finding a way to cleanly > abstract the difference. > > Please reply by August 6, 2026 so we can factor your feedback into the plan. > > Thanks! > Xiubo Li

1 0

Re: Performance issue after tentacle upgrade
by Frédéric Nass 16 Jul '26

16 Jul '26

Hi Moti, Anything suspicious on 'ceph osd perf' output? > NVMe disks' iops went from something around 5-6K constant io per second to almost zero (less than a hundred per second, on all host/OSDs, with some non-frequent high peaks), That would be pretty good news, actually. This would likely be the result of the new 'bluefs_wal_envelope_mode' [1]. Its own description says it "reduces by ~50% the amount of fdatasync syscalls" on the RocksDB WAL. :-) I don't expect this to explain the performance drop at this time, though. I'd recommend that you check the new/old settings and default values between both versions using our online Config Diff tool here [2]. It might ring a bell. The tool highlights that the default values for 'osd_op_num_shards_hdd' and 'osd_op_num_threads_per_shard_hdd' were swapped between both versions. I'm not sure what the impact could be on your cluster in terms of performance, but if you're using mClock, reverting this change or trying 'wpq' osd scheduler instead might help. You can run 'ceph config set osd osd_op_queue wpq' and restart all OSDs to figure out if mClock has anything to do with what you observe. If swapping values or changing the osd scheduler to wpq doesn't help, you could try disabling 'bluefs_wal_envelope_mode' and running 'ceph-bluestore-tool --command downgrade-wal-to-v1' (make sure to test this out on a single OSD first). Best regards, Frédéric [1] https://github.com/ceph/ceph/blob/708737d3ec0669a8e01265d9699d6f939debec13/… [2] https://analyzer.clyso.com/#/analyzer/config-diff?source=v18.2.4&target=v20… Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> frederic.nass(a)clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Le jeu. 9 juil. 2026 à 15:33, Moti via ceph-users <ceph-users(a)ceph.io> a écrit : > Hello everyone! > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing as > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET and > HEAD requests. > > There is no resource cap or significant increase in usage is visible. > Neither fast ec nor any other new features are enabled yet, just did the > upgrade and left it as is. For measuring latency I have a services that > probes the cluster with consistent load so it's not a difference in load > either. PG status history does not show any pg operation being heavily in > process. > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db block. > Both ec and replicated pools exist. > The strange hint was: NVMe disks' iops went from something around 5-6K > constant io per second to almost zero (less than a hundred per second, on > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > not much changed. db blocks on NVMe are still attached to OSDs with no > issue. > > I can't find a clue to explain this. Got the same results on 3 separate > clusters (all went version 18 --> 20, same hybrid build and configs). > Nothing in config dump changes (changed defaults) catches my attention to > have caused this. > > Anyone had similar issues? Any tips and ideas on what to check? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: umbrella 21.1.0 RC QE validation status
by Venky Shankar 14 Jul '26

14 Jul '26

Hi Yuri, On Tue, Jul 14, 2026 at 1:46 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Waiting for approvals/reviews for: > > rados: Travis > > rgw: Adam Emerson > fs: Venky fs approved - failures are https://tracker.ceph.com/projects/cephfs/wiki/Umbrella#Umbrella-v2110-rc1 > orch/ceph-volume: Adam King & Guillaume > > upgrade: Kamoltat, Patrick > > powercycle: Brad (I saw your tracker, but pls approve this) > > On Mon, Jul 6, 2026 at 8:00 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/76757 >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! >> >> Seeking approvals/reviews for: >> >> smoke: Radek, Laura, Nitzan >> >> rados: Radek, Laura, Notzan, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya >> >> nvmeof: PASSED >> >> upgrade: Laura, Patrick >> >> upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: TBD (https://github.com/ceph/ceph/pull/69888) >> upgrade-clients:client-upgrade-squid-umbrella-umbrella: TBD (https://github.com/ceph/ceph/pull/69887) >> >> perf-basic: PASSED >> >> crimson-rados: Matan >> >> powercycle: Brad >> >> TIA -- Cheers, Venky

1 0

v19.2.5 Squid released
by Yuri Weinstein 14 Jul '26

14 Jul '26

This is the fifth backport release in the Squid series. We recommend all users update to this release. ## Notable Changes MDS (Metadata Server) --------------------- - Fixed a segmentation fault relating to MDSContext completion and request queuing. OSD (Object Storage Daemon) --------------------------- - Rollback & Vector Fixes: Corrected rollback logic for partial write object information (OI) and optimized Erasure Coding (EC) by ensuring Twiddle creates a full-sized vector. RGW (RADOS Gateway) ------------------- - TLS 1.3 Ciphersuites: Introduced the ssl_ciphersuites configuration option for the Beast front-end to support TLS 1.3 cipher customizations. - S3 PutObject ACLs Fix: Fixed a bug where PutObject requests were erroneously rejected under BlockPublicAcls due to an incorrect comparison return value from canned_acl.compare(). - Multisite Sync Robustness: Resolved an infinite loop scenario in RGWBucketFullSyncCR when a source bucket was deleted mid-sync by introducing a mechanism to clear stateful data in reused bucket_list_result objects. - SNS Policy Evaluation: Updated ListTopics to use the account root ARN for policy evaluation when invoked by a non-root account user, preventing unexpected implicit denies from identity-based policies. - Presigned URLs: Fixed an internal server error occurring during the authentication process of presigned URLs. - Prerequest Hook: Restructured the order of execution so that the prerequest hook now runs after the authorization process has completed. - Object Tagging: Added handling for plain-text object tags during execution of RGWObjTags::decode(). - Beast Frontend Stability: Applied a strand executor to the Beast timeout timer to avoid concurrent socket access and potential race conditions. - Bucket Statistics: Added a fix for bucket stats execution when the underlying bucket index does not exist. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v19-2-5-squid-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.5.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: abc7aa7f2701e5d46878fd5e6bb7e2955f1a395a

1 0

Re: Performance issue after tentacle upgrade
by Eugen Block 14 Jul '26

14 Jul '26

Hi, I'm not sure if it could be related to summer vacation, but I feel like there should have been several reports similar to yours by now. I don't have upgraded any production cluster to 20 yet, so I can't really deny or confirm anything. Did you upgrade the OS as well? Maybe there were some relevant changes there. Or some non-persistent configs were overwritten, either in the OS or Ceph. Was there more deep-scrubbing after the upgrade which could have impacted the IO? Regards, Eugen Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > Hello everyone! > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing as > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET and > HEAD requests. > > There is no resource cap or significant increase in usage is visible. > Neither fast ec nor any other new features are enabled yet, just did the > upgrade and left it as is. For measuring latency I have a services that > probes the cluster with consistent load so it's not a difference in load > either. PG status history does not show any pg operation being heavily in > process. > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db block. > Both ec and replicated pools exist. > The strange hint was: NVMe disks' iops went from something around 5-6K > constant io per second to almost zero (less than a hundred per second, on > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > not much changed. db blocks on NVMe are still attached to OSDs with no > issue. > > I can't find a clue to explain this. Got the same results on 3 separate > clusters (all went version 18 --> 20, same hybrid build and configs). > Nothing in config dump changes (changed defaults) catches my attention to > have caused this. > > Anyone had similar issues? Any tips and ideas on what to check? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: umbrella 21.1.0 RC QE validation status
by Brad Hubbard 13 Jul '26

13 Jul '26

On Tue, Jul 14, 2026 at 6:18 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Waiting for approvals/reviews for: > > rados: Travis > > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > > upgrade: Kamoltat, Patrick > > powercycle: Brad (I saw your tracker, but pls approve this) > Approved. > > On Mon, Jul 6, 2026 at 8:00 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/76757 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > > Seeking approvals/reviews for: > > > > smoke: Radek, Laura, Nitzan > > > > rados: Radek, Laura, Notzan, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya > > > > nvmeof: PASSED > > > > upgrade: Laura, Patrick > > > > upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: TBD ( > > https://github.com/ceph/ceph/pull/69888) > > upgrade-clients:client-upgrade-squid-umbrella-umbrella: TBD ( > > https://github.com/ceph/ceph/pull/69887) > > > > perf-basic: PASSED > > > > crimson-rados: Matan > > > > powercycle: Brad > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Cheers, Brad

1 0

Re: umbrella 21.1.0 RC QE validation status
by Adam Emerson 13 Jul '26

13 Jul '26

On 06/07/2026, Yuri Weinstein via ceph-users wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/76757 > > Release Notes - TBD > LRC upgrade - TBD > > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > Seeking approvals/reviews for: > > smoke: Radek, Laura, Nitzan > > rados: Radek, Laura, Notzan, Afreen, Travis > rgw: Adam Emerson RGW Approved.

1 0

Re: Argonaut: an open-source terminal UI for Ceph administration
by Sinan Polat 13 Jul '26

13 Jul '26

Hi Thomas, Thanks for taking the time to write all that out, it is genuinely useful. Let me deal with the AI thing first since you raised it. I did lean on AI a lot while building this and I want to be clear I was never trying to hide that. I have been open about it elsewhere (I said as much on Reddit) and I am actually working on a blog post about how the whole thing came together, so it's not something I'm shy about. It wasn't "prompt and ship." I acted as the architect and reviewer, I made every design, UX, and safety decision, set the hard constraints (strict separation between UI, logic, and a single isolated Ceph transport; every destructive action confirms and previews the exact ceph command before it runs; version-aware JSON decoding in one place), reviewed the changes, and maybe importantly I myself validated it against real Reef, Squid, and Tentacle clusters (the decoding layer is covered by golden fixtures captured from live clusters), plus a full audit pass before release. On the single commit: that is a fair read and I owe an explanation. The project does have a real, iterative development history, it lives in a private dev repo. The public repo is a clean snapshot because some early commits contained infrastructure credentials I didn't want to publish. I can see how one squashed commit looks like an opaque code-dump. That was absolutely not the intent. And your point about not handing cluster access to a tool you haven't read, I'd do exactly the same :-) It authenticates the same way the ceph CLI does, with your keyring and your permissions, and there is a mock mode (--client mock) that runs on fake in-memory data if you just want to click around without touching a real cluster. About your feedback, which I really appreciate: The connected-clients overview is a great idea! The same goes for a view with the real recovery ETA. The hung-but-still-marked-up OSDs sound nasty! I will have to think about how you'd even detect that reliably. Perhaps you can share that part of your script? Unrelated note, based on other feedback about the name the project is now called Siphon. Repo here: https://github.com/cinpol/siphon Thanks again for the honest feedback :) Cheers, Sinan Op ma 13 jul 2026 om 16:58 schreef Thomas Schneider via ceph-users < ceph-users(a)ceph.io>: > Because some users won't trust tools that have limited or no human > oversight. > Having a repository that is written by "AI" with one single large commit > calls into question > * the maintainability of the code > * the understanding of the author of the code > * the longevity of the project > * the safety > > If the software was read only it _could_ be ok, but I would never give a > vibe coded piece of software that I didn't read completely access to my > cluster. > If a human maintains a piece of software I can look at the commit > history, understand the design, be reasonable sure to follow the code > and could in theory follow the development. LLM code-dumps that span > thousands of lines and aren't even completely shown in the github diff > view are a nightmare. > > Oh and I find it intransparent and questionable if someone isn't open > about using AI. If a project is transparent about it's usage I'm more > willing to give them the benefit of the doubt as if they don't mention > it at all. > > Kind regards, > Thomas > > On 2026-07-13 16:43, Anthony D'Atri wrote: > > Why does that matter? > > > >> On Jul 13, 2026, at 10:41 AM, Thomas Schneider via ceph-users > >> <ceph-users(a)ceph.io> wrote: > >> Looking at the README and "commit history" this kind of screams vibe > >> coded which turns me off from even trying to use it tbh. If you used > >> LLMs you should be transparent about it. > > -- > Thomas Schneider > IT.SERVICES > Wissenschaftliche Informationsversorgung > Ruhr-Universität Bochum | 44780 Bochum > Telefon: +49 234 32 23939 > http://www.it-services.rub.de/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

umbrella 21.1.0 RC QE validation status
by Yuri Weinstein 13 Jul '26

13 Jul '26

Details of this release are summarized here: https://tracker.ceph.com/issues/76757 Release Notes - TBD LRC upgrade - TBD *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* Seeking approvals/reviews for: smoke: Radek, Laura, Nitzan rados: Radek, Laura, Notzan, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: PASSED upgrade: Laura, Patrick upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: TBD ( https://github.com/ceph/ceph/pull/69888) upgrade-clients:client-upgrade-squid-umbrella-umbrella: TBD ( https://github.com/ceph/ceph/pull/69887) perf-basic: PASSED crimson-rados: Matan powercycle: Brad TIA

1 1