- ceph-users - lists.ceph.io

Ceph upgrade OSD unsafe to stop
by GLE, Vivien 18 Aug '25

18 Aug '25

Hi, I'm trying to update my cluster (19.2.2 -> 19.2.3), mon and mgr upgrade goes well but I had some issue with OSD : Upgrade: unsafe to stop osd(s) at this time (165 PGs are or would become offline) Cluster is in health_ok All pools are replica 3 and pgs active+clean autoscaler is off following the ceph docs Does ceph osd ok-to-stop lead to lost data ? The only rules used in the cluster is replicated_rule : root@ceph-monitor-1:/# ceph osd crush rule dump replicated_rule { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "inist" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } root@ceph-monitor-1:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 6.63879 root inist -15 0.90970 host ceph-monitor-1 6 hdd 0.90970 osd.6 up 1.00000 1.00000 -21 5.72910 datacenter bat1 -20 2.72910 room room01 -19 2.72910 row left -18 2.72910 rack 10 -3 0.90970 host ceph-node-1 2 hdd 0.90970 osd.2 up 1.00000 1.00000 -5 0.90970 host ceph-node-2 1 hdd 0.90970 osd.1 up 1.00000 1.00000 -9 0.90970 host ceph-node-3 5 hdd 0.90970 osd.5 up 1.00000 1.00000 -36 3.00000 room room03 -35 3.00000 row left06 -34 3.00000 rack 08 -7 1.00000 host ceph-node-4 0 hdd 1.00000 osd.0 up 1.00000 1.00000 -13 1.00000 host ceph-node-5 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph-node-6 4 hdd 1.00000 osd.4 up 1.00000 1.00000 Thanks ! Vivien

4 5

ceph.io certificate expired :-/
by Dan O'Brien 17 Aug '25

17 Aug '25

Looks like the Let's Encrypt auto-renew caught a snag. Issued On Monday, May 19, 2025 at 6:27:52 AM Expires On Sunday, August 17, 2025 at 6:27:51 AM

1 0

Replicas
by Shawn Heil 16 Aug '25

16 Aug '25

I recently set up a 5-node Proxmox/Ceph cluster with 10 OSDs each node for a total for 50 nodes (approximately 135TB). All the nodes have 768GB RAM and a single 72-Core (144 thread) Epyx CPU. I set the ceph pool size to: 3 and the minimum size: 2. The performance seemed good and everything was happy in cephland! The other day I spoke to a ceph consultant and he recommended I change the pool size from 3/2 to 2/1. He cited several valid points: more usable storage space, faster rebuilds, better performance... I followed his advice and changed it. The benchmark performance was about the same, but the recovery time when I took a node down was improved. I really do like the idea of the extra storage space! So now I am confused whether I should leave it or go back to 3 replicas. Anyone have any thoughts or compelling reasons I should leave it or change it back? Shawn Shawn Heil Phone: (608) 836-7041 | Direct: 6084102333 Email: shawn.heil(a)brucecompany.com<mailto:shawn.heil@brucecompany.com> | Website: www.brucecompany.com<http://www.brucecompany.com> [cid:SocialLink_Facebook_32x32_24bd5ab0-6667-4cda-9838-e5caaf24a024.png]Find us on Facebook<https://www.facebook.com/pages/The-Bruce-Company/113279807065?ref=hl> [cid:BruceCompanyHomepageLogo(2025)_429ac410-efd6-49ca-8739-0317666483cd.jpg]

3 2

OSD's are Moving back from custom bucket...
by Devender Singh 16 Aug '25

16 Aug '25

Hello all Why my buckets are moving back when restarting osd…how to make them persistent.. After move command crush shows as below… host custom { id -9 # do not change unnecessarily id -10 class hdd # do not change unnecessarily # weight 0.01469 alg straw2 hash 0 # rjenkins1 item osd.3 weight 0.00490 item osd.4 weight 0.00490 item osd.5 weight 0.00490 } root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd crush move osd.3 host=custom root=default ceph osd crush move osd.4 host=custom root=default ceph osd crush move osd.5 host=custom root=default no need to move item id 3 name 'osd.3' to location {host=custom,root=default} in crush map no need to move item id 4 name 'osd.4' to location {host=custom,root=default} in crush map no need to move item id 5 name 'osd.5' to location {host=custom,root=default} in crush map root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 up 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph orch daemon restart osd.5 Scheduled to restart osd.5 on host 'ceph-01.tinihub.com' root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 up 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.00980 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 -1 0.09279 root default -5 0.03419 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.00980 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 -1 0.09279 root default -5 0.03419 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 5 hdd 0.00490 osd.5 up 0 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000

2 1

How to test run the new EC Erasure Coding Enhancements Optimizations?
by Chris Lawsonn 15 Aug '25

15 Aug '25

Hi, I am new ignorant user and want to practice test any of the new EC Optimizations that are being worked on for tentacle. How can I test install and setup using any of the new code that includes the EC Optimizations? What would be easiest steps to standup a test environment and how to run pre-release code? Thanks

5 5

August User / Dev Meeting
by Anthony Middleton 15 Aug '25

15 Aug '25

Hey everyone, Just a quick update: there won't be a User + Dev meeting on Wednesday, August 20th, since many team leads are busy. Our next meeting will be on Wednesday, September 17th. Start adding topics to the agenda beforehand: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes. Please continue to ask questions and share your concerns on the #ceph-devel Slack and mailing list. Have a great weekend! Best, Anthony Middleton Ceph Community Manager community-manager(a)ceph.io

1 0

Re: Default firewall zone
by Sake Ceph 15 Aug '25

15 Aug '25

Found a solution: sudo firewall-cmd --set-default-zone=ZONE_NAME Now need to find a way to make sure the company default aren't being applied again. > Op 14-08-2025 23:48 CEST schreef Anthony D'Atri <anthony.datri(a)gmail.com>: > > > There are all kinds of firewalls in use, so this would be difficult. > > > On Aug 14, 2025, at 1:41 AM, Sake Ceph <ceph(a)paulusma.eu> wrote: > > > > Is it possible to set the firewall zone to use for the orchestrator/cephadm to use? If we don't provide the zone when configuring the firewall, the default zone is drop in our company (don't ask me why). > > > > Kind regards, > > Sake > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

SuSE stops building Ceph packages for its distributions
by Robert Sander 15 Aug '25

15 Aug '25

Hi, SuSE will not build even the Ceph client packages for its distributions any more: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org/threa… https://www.suse.com/releasenotes/x86_64/SUSE-SLES/15-SP6/index.html#jsc-PE… This means that this distribution cannot consume Ceph storage any more, if no other party will build these packages. If you are using SuSE Linux Enterprise Server or OpenSuSE as a client for Ceph storage you need to plan for this. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 4

Safe Procedure to Increase PG Number in Cache Pool
by Vishnu Bhaskar 15 Aug '25

15 Aug '25

Hi Team, I have an OpenStack Ceph cluster where the Cinder volumes pool uses a cache pool named *volumes_cache*. Due to performance issues, I observed that the cache pool has fewer PGs and a high PG-to-object ratio, approximately 10x higher than normal. For instance, one PG contains around 35,000 objects. I am planning to increase the PG number for the cache pool. For the base pool, I was able to increase the PG count without any issues. However, for the cache pool, I encountered a warning indicating that a force argument is required. In my lab environment, I found that changing the cache pool mode to *none* allowed me to modify the PG number. Since this is my production setup, I would like to know if there is a safe and recommended procedure to increase the PG number of a *cache pool* without impacting the environment. Kindly advise on the best approach. Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

3 5

tentacle 20.1.0 RC QE validation status
by Yuri Weinstein 14 Aug '25

14 Aug '25

Details of this release are summarized here: https://tracker.ceph.com/issues/72316#note-1 Release Notes - TBD LRC upgrade - TBD Dev Leads, please review the list of suites for completeness, as this is a new release. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venly orch: Adam King rbd: Ilya krbd: Ilya upgrade/squid-x: Laura upgrade/reef-x: Laura upgrade/telemetry-upgrade: PASSED upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: N/A upgrade-clients:client-upgrade-squid-tentacle-tentacle: N/A perf-basic: PASSED crimson-rados: PASSED powercycle: PASSED ceph-volume - Guillaume TIA

8 11