- ceph-users - lists.ceph.io

Is there an efficient way to list all namespaces in a pool?
by Florian Haas 19 Dec '25

19 Dec '25

Hello everyone, I've been wondering if there is a way of listing all namespaces in a RADOS pool in a way that is both human- and machine-readable, and is more efficient than this: rados -p <pool> ls --all -f json \ | jq -r "[.[] | .namespace] | unique" There are of course several variations of this approach, but they all seem to boil down to enumerating *all objects* and then picking out their namespaces — which is, shall we say, impractical if a pool contains millions of objects. Does a fundamentally different option exist? Cheers, Florian

1 0

Announcing go-ceph v0.37.0
by Anoop C S 19 Dec '25

19 Dec '25

Hi, I'm happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.37.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S.

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Frédéric Nass 19 Dec '25

19 Dec '25

Hi Văn Trần, For msg30399.html <https://www.mail-archive.com/ceph-users@ceph.io/msg30399.html>, the root cause is unknown and likely specific to the user's environment. For msg31238.html <https://www.mail-archive.com/ceph-users@ceph.io/msg31238.html>, the impact is noticeable on clusters with a high density of OSDs per node. With 7 OSDs per node, you likely won't notice it. It's being investigated here [1]. For tracker #70390 <https://tracker.ceph.com/issues/70390>, it only impacts OSDs created in Squid, so you're safe. Just make sure to disable bluestore_elastic_shared_blobs with 'ceph config set osd bluestore_elastic_shared_blobs 0' right after the upgrade; this ensures that newly created or recreated OSDs will be safe as well. Regarding Squid critical bugs, I recommend reading this list, which we will update periodically [2]. Best regards, Frédéric. [1] https://tracker.ceph.com/issues/73107 [2] https://docs.clyso.com/docs/kb/known-bugs/squid/ -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Check our online Config Diff Tool, it's great! https://analyzer.clyso.com/#/analyzer/config-diff https://clyso.com | frederic.nass(a)clyso.com Le jeu. 18 déc. 2025 à 08:08, Văn Trần Quốc via ceph-users < ceph-users(a)ceph.io> a écrit : > Hi Ceph Users, > > I am planning a major upgrade for our production cluster from Reef (18.2.7) > to Squid (19.2.3) and would like to seek advice regarding stability and > potential risks. > > Infrastructure Overview: > > + Deployment: Cephadm. > + Cluster Size: 7 Nodes total. > + Hardware/Virtualization: Each node is a Virtual Machine hosted on > Proxmox. > + OSD Layout: 7 OSDs total (7 OSD per node). > + Other Daemons: 5 MONs, 7 MDSs, 3 MGRs. Services include RGW, CephFS, and > Block Devices. > + Pool Type: Replicated. > > The Concern: We are currently running stable on Reef 18.2.7. However, I > have been following recent discussions on the mailing list and tracker > regarding critical failures when upgrading to Squid, specifically > concerning OSD crashes and data corruption. > I am particularly worried about the issues reported in these threads, where > users experienced failures during or after the upgrade: > + https://www.mail-archive.com/ceph-users@ceph.io/msg30399.html > + https://www.mail-archive.com/ceph-users@ceph.io/msg31238.html > + https://tracker.ceph.com/issues/70390 > > Given our deployment topology and the jump to a new major version (Squid), > I have a few questions: > + Stability & Success Stories: Is Ceph Squid 19.2.3 considered safe > regarding the OSD crash/corruption bugs mentioned in the links above? Has > anyone in the community successfully completed the upgrade from 18.2.7 to > 19.2.3 without issues? Confirmation of a clean upgrade path would be very > reassuring. > + Upgrade Path: Are there any known regressions or critical "gotchas" when > moving from 18.2.7 directly to 19.2.3 in a virtualized environment? > > Any experiences or warnings from those running Squid in similar > environments would be greatly appreciated. > > Thank you, > Van Tran > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Eugen Block 18 Dec '25

18 Dec '25

Just adding the list again. Văn Trần <vantran15122004(a)gmail.com> schrieb am Do. 18. Dez. 2025 um 12:41: > Hi Eugen, > > Thank you very much for the clarification. > > It is very reassuring to know that you are successfully running a mix of > Replicated and EC pools on Squid, and that disabling > bluestore_elastic_shared_blobs was the only necessary precaution. > > I will follow your lead and apply this flag as a safeguard before > proceeding with my upgrade and future expansion. > > Appreciate your time and help! > > Best regards, > Van Tran >

1 0

Re: Problems with orchestrator
by Redouane Kachach 18 Dec '25

18 Dec '25

Hi Cartsen, Plz find the answers inline: Best, Redo. On Wed, Dec 17, 2025 at 3:12 PM Carsten Götze via ceph-users < ceph-users(a)ceph.io> wrote: > Hi! > > Thank you for the pointers. > > Unfortunately restarting the active mgr daemon didn't help. > > However looking at cephadm.log gave me at least a clue what was going on. > It seems that installing and deinstalling systemd-firewalld has completely > messed up the firewall rules. And restarting one of the nodes has made > things even worse. The osd's on that node died after a few minutes. I had > to restart the osd's systemd services, only to see them tumbling down a few > minutes later. Only after I finally flushed all the firewall rules from > every single node things went back to "normal". However, orchestrator is > still out of sync with reality. > > So a few more questions: > Where does orchestrator store its state information? > [Redo] Orchestrator (cephadm in this case) use the mon-store to store the services/daemon status information. In order to get a "fresh" view of the services state you can run: > ceph orch ps --refresh > Is firewalld required for tentacle? I didn't need it for squid, but It > sure is now required for the nfs module.Do the other modules also use it to > control the firewall? > [Redo] Firewalld isn’t required for cephadm-managed services to run, but it’s an important operational security layer. Whether you enable it depends on your org/company security model and requirements. In many environments, host firewall rules (often combined with network segmentation) help ensure Ceph ports and services are only reachable from the intended networks and hosts, reducing the risk of accidental exposure and limiting blast radius if something is misconfigured or bcz some service is exposing exploited ports. With firewalld installed, would restarting the nodes help to restore the > firewall rules? > [Redo] Firewalld is an external service. It stores its configuration on the local host. Restarting the node will just get you back to the same status. If you want to reset/restore any configuration you have to use firewalld commands. > What ports have to be open in the firewall? So far I have: 3000, 3300, > 5666, 6789, 6800:7568, 8443, 9093, 9094, 9095, 9100. Anything else? > [Redo] When firewalld is enabled cephadm will automatically open the corresponding port for the service, you don't have to worry about opening the port manually. > Would it be safe to downgrade to squid again? > [Redo] No, unfortunately you can't downgrade the ceph version (bcz of several technical reasons). > > With best regards > Carsten Götze > > > Am 16.12.2025 um 14:10 schrieb Robert Sander via ceph-users < > ceph-users(a)ceph.io>: > > > > Hi, > > > > Am 16.12.25 um 2:03 PM schrieb Carsten Götze via ceph-users: > > > >> Is there a way to force the orchestrator to sync its state information > with the nodes? > > > > Try to restart the active MGR daemon with systemctl on its node. > > > >> Where do I find meaningful logs for the orchestrator? > > > > The journal of the active MGR contains cephadm output > > > > Regards > > -- > > Robert Sander > > Linux Consultant > > > > Heinlein Consulting GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > https://www.heinlein-support.de > > > > Tel: +49 30 405051 - 0 > > Fax: +49 30 405051 - 19 > > > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Eugen Block 18 Dec '25

18 Dec '25

Hi, we upgraded two months ago from 18.2.7 to 19.2.3 without any issues. We use RBD, RGW and CephFS. The only thing I made sure of was to disable bluestore_elastic_shared_blobs as Wannes already mentioned because we needed to deploy additional OSDs. Regards, Eugen Am Do., 18. Dez. 2025 um 09:07 Uhr schrieb Wannes Smet via ceph-users < ceph-users(a)ceph.io>: > I can't directly answer your question, but I'm currently redeploying all > my OSDs with `bluestore_elastic_shared_blobs` disabled. I don't know for > your situation, but you might want to investigate if that's a good idea > before upgrading from 18.2.7 to 19.2.3. > > AFAIK the issue has also only be reported on EC pools. Not on replicated > pools that I'm aware of. > > Wannes > ________________________________ > From: Văn Trần Quốc via ceph-users <ceph-users(a)ceph.io> > Sent: Thursday, December 18, 2025 08:09 > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) > on Proxmox VMs > > Hi Ceph Users, I am planning a major upgrade for our production cluster > from Reef (18. 2. 7) to Squid (19. 2. 3) and would like to seek advice > regarding stability and potential risks. Infrastructure Overview: + > Deployment: Cephadm. + Cluster Size: > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > Hi Ceph Users, > > I am planning a major upgrade for our production cluster from Reef (18.2.7) > to Squid (19.2.3) and would like to seek advice regarding stability and > potential risks. > > Infrastructure Overview: > > + Deployment: Cephadm. > + Cluster Size: 7 Nodes total. > + Hardware/Virtualization: Each node is a Virtual Machine hosted on > Proxmox. > + OSD Layout: 7 OSDs total (7 OSD per node). > + Other Daemons: 5 MONs, 7 MDSs, 3 MGRs. Services include RGW, CephFS, and > Block Devices. > + Pool Type: Replicated. > > The Concern: We are currently running stable on Reef 18.2.7. However, I > have been following recent discussions on the mailing list and tracker > regarding critical failures when upgrading to Squid, specifically > concerning OSD crashes and data corruption. > I am particularly worried about the issues reported in these threads, where > users experienced failures during or after the upgrade: > + > https://urldefense.com/v3/__https://www.mail-archive.com/ceph-users@ceph.io… > + > https://urldefense.com/v3/__https://www.mail-archive.com/ceph-users@ceph.io… > + > https://urldefense.com/v3/__https://tracker.ceph.com/issues/70390__;!!Ftrht… > > Given our deployment topology and the jump to a new major version (Squid), > I have a few questions: > + Stability & Success Stories: Is Ceph Squid 19.2.3 considered safe > regarding the OSD crash/corruption bugs mentioned in the links above? Has > anyone in the community successfully completed the upgrade from 18.2.7 to > 19.2.3 without issues? Confirmation of a clean upgrade path would be very > reassuring. > + Upgrade Path: Are there any known regressions or critical "gotchas" when > moving from 18.2.7 directly to 19.2.3 in a virtualized environment? > > Any experiences or warnings from those running Squid in similar > environments would be greatly appreciated. > > Thank you, > Van Tran > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Wannes Smet 18 Dec '25

18 Dec '25

I can't directly answer your question, but I'm currently redeploying all my OSDs with `bluestore_elastic_shared_blobs` disabled. I don't know for your situation, but you might want to investigate if that's a good idea before upgrading from 18.2.7 to 19.2.3. AFAIK the issue has also only be reported on EC pools. Not on replicated pools that I'm aware of. Wannes ________________________________ From: Văn Trần Quốc via ceph-users <ceph-users(a)ceph.io> Sent: Thursday, December 18, 2025 08:09 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs Hi Ceph Users, I am planning a major upgrade for our production cluster from Reef (18. 2. 7) to Squid (19. 2. 3) and would like to seek advice regarding stability and potential risks. Infrastructure Overview: + Deployment: Cephadm. + Cluster Size: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Ceph Users, I am planning a major upgrade for our production cluster from Reef (18.2.7) to Squid (19.2.3) and would like to seek advice regarding stability and potential risks. Infrastructure Overview: + Deployment: Cephadm. + Cluster Size: 7 Nodes total. + Hardware/Virtualization: Each node is a Virtual Machine hosted on Proxmox. + OSD Layout: 7 OSDs total (7 OSD per node). + Other Daemons: 5 MONs, 7 MDSs, 3 MGRs. Services include RGW, CephFS, and Block Devices. + Pool Type: Replicated. The Concern: We are currently running stable on Reef 18.2.7. However, I have been following recent discussions on the mailing list and tracker regarding critical failures when upgrading to Squid, specifically concerning OSD crashes and data corruption. I am particularly worried about the issues reported in these threads, where users experienced failures during or after the upgrade: + https://urldefense.com/v3/__https://www.mail-archive.com/ceph-users@ceph.io… + https://urldefense.com/v3/__https://www.mail-archive.com/ceph-users@ceph.io… + https://urldefense.com/v3/__https://tracker.ceph.com/issues/70390__;!!Ftrht… Given our deployment topology and the jump to a new major version (Squid), I have a few questions: + Stability & Success Stories: Is Ceph Squid 19.2.3 considered safe regarding the OSD crash/corruption bugs mentioned in the links above? Has anyone in the community successfully completed the upgrade from 18.2.7 to 19.2.3 without issues? Confirmation of a clean upgrade path would be very reassuring. + Upgrade Path: Are there any known regressions or critical "gotchas" when moving from 18.2.7 directly to 19.2.3 in a virtualized environment? Any experiences or warnings from those running Squid in similar environments would be greatly appreciated. Thank you, Van Tran _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Văn Trần Quốc 18 Dec '25

18 Dec '25

Hi Ceph Users, I am planning a major upgrade for our production cluster from Reef (18.2.7) to Squid (19.2.3) and would like to seek advice regarding stability and potential risks. Infrastructure Overview: + Deployment: Cephadm. + Cluster Size: 7 Nodes total. + Hardware/Virtualization: Each node is a Virtual Machine hosted on Proxmox. + OSD Layout: 7 OSDs total (7 OSD per node). + Other Daemons: 5 MONs, 7 MDSs, 3 MGRs. Services include RGW, CephFS, and Block Devices. + Pool Type: Replicated. The Concern: We are currently running stable on Reef 18.2.7. However, I have been following recent discussions on the mailing list and tracker regarding critical failures when upgrading to Squid, specifically concerning OSD crashes and data corruption. I am particularly worried about the issues reported in these threads, where users experienced failures during or after the upgrade: + https://www.mail-archive.com/ceph-users@ceph.io/msg30399.html + https://www.mail-archive.com/ceph-users@ceph.io/msg31238.html + https://tracker.ceph.com/issues/70390 Given our deployment topology and the jump to a new major version (Squid), I have a few questions: + Stability & Success Stories: Is Ceph Squid 19.2.3 considered safe regarding the OSD crash/corruption bugs mentioned in the links above? Has anyone in the community successfully completed the upgrade from 18.2.7 to 19.2.3 without issues? Confirmation of a clean upgrade path would be very reassuring. + Upgrade Path: Are there any known regressions or critical "gotchas" when moving from 18.2.7 directly to 19.2.3 in a virtualized environment? Any experiences or warnings from those running Squid in similar environments would be greatly appreciated. Thank you, Van Tran

1 0

Re: Proposal to end support for some erasure code plugins
by Jeremy Austin 17 Dec '25

17 Dec '25

Jamie, I run a single cluster using CLAY, and have quietly and successfully for 6+ years. Initially, I was concerned about network bandwidth during recovery. I have slowly upgraded network speeds and now that is not a primary consideration. I've read the relevant blog post and conclusions, and am willing to switch to a recommend Fast EC-supported plugin and technique. That's one datum, I guess. /Jeremy On Tue, Dec 16, 2025 at 11:42 AM Jamie Pryde via ceph-users < ceph-users(a)ceph.io> wrote: > Hi all, > > I want to make you aware of the Fast EC team's proposal to end support for > some erasure code plugins and techniques. > > We want to do this for a few reasons: > 1. Continuing to support a large list of plugins and techniques requires > development and test effort. We don't think this effort is valuable because > 2. Telemetry data shows that most Ceph clusters that are using EC are > using Jerasure reed_sol_van or ISA-L reed_sol_van, and > 3. Benchmarking shows that reed_sol_van has very good performance on > modern CPUs. > > We are proposing that the following list of plugins and techniques should > be deprecated in the Umbrella release, and support be removed in the V > release: > > * > Jerasure (reed_sol_r6_op, cauchy_orig, cauchy_good, liberation, > blaum_roth, liber8tion) > * > SHEC > * > CLAY > > The following list of plugins and techniques would continue to be > supported in the V release: > > * > Jerasure (reed_sol_van) > * > ISA-L (reed_sol_van, cauchy) > * > LRC (Although Fast EC doesn’t currently support LRC and we wouldn't > recommend using LRC yet, we think we will be able to use it in future to > improve support for erasure coded pools in stretched clusters.) > > I have written a post for the ceph.io blog which contains more details > and shows some of the data behind the proposal. It also talks about the > pool migration feature we are planning to add in Umbrella. This can be used > to migrate data from a pool to a new EC pool that uses a supported > plugin/technique. You can find the blog PR here: > https://github.com/ceph/ceph.io/pull/988 > > We are interested to hear if anyone has any objections to this plan. > Please reply to this mail or leave a comment on the blog PR if you do have > any objections or concerns. We plan to publish the blog post sometime > mid-January. > > > Thanks, > > > > -- > > > > Jamie Pryde > > Software Engineer > > IBM UK > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > Winchester, Hampshire SO21 2JN > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Jeremy Austin jhaustin(a)gmail.com

1 0

Test email
by Eugen Block 17 Dec '25

17 Dec '25

Test

1 1