- ceph-users - lists.ceph.io

ceph NFS reef - cephadm reconfigure haproxy
by Steven Vacaroaia 05 May '25

05 May '25

Hi I am testing NFS trying to make sure that, deploying it as below will give me redundancy ceph nfs cluster create main "1 ceph-01,ceph-02" --port 2049 --ingress --virtual-ip 10.90.0.90 For what I have read so far ( and some of the posts on this list) the only way to get the NFS "surviving " a node reboot is to actually reconfigure haproxy since the config is missing the second entry under "backend" and the "check" The method to do that is as below using the template from the github link ceph config-key set mgr/cephadm/services/ingress/haproxy.cfg -i <custom-haproxy-config> ceph orch redeploy <ingress-service-name> ceph orch restart ingress-service-name https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/templates/ser… However, after doing the above , I was checking the actual haproxy.cfg on the main node (/var/lib/ceph/FSID/SERVICENAME/haproxy/haproxy.cfg ) and it does not change restart the NFS service (ceph orch restart servicename) did not help either What am I missing ?

1 0

Process to update images used by cephadm
by Joshua Blanch 05 May '25

05 May '25

Hello All, Is there a process to update images used by cephadm such as HAproxy? For example on main right now it is pointing to v2.3 tag https://github.com/ceph/ceph/blob/94876852b462c2ac15b447becf4d78071c529fb1/… and on quay, there is a more recent (albeit outdated) image version https://quay.io/repository/ceph/haproxy?tab=tags There are more recent releases for this image, is there a process to push a new image to quay and update which one cephadm pulls the default image. This question also goes for other images like grafana, etc. -- Joshua Blanch Clyso GmbH p: +49 89 215252722 | a: Victoria, Canada w: https://clyso.com | e: joshua.blanch(a)clyso.com

2 1

Notes from Ceph Steering Committee Weekly 2025-05-05
by Dan van der Ster 05 May '25

05 May '25

Hi all, Here are the summary notes from today's Ceph Steering Committee Weekly meeting: * CI/Testing: Push to reduce CI load by disabling flaky tests, parallelizing checks, and possibly allowlisting known infra issues; Will be coordinated further in the Infrastructure calls. * 18.2.7 Hotfix: Test summary sent, awaiting core team review and approval. Quincy will get a hotfix with minimal / critical PRs. Coordinating on Slack. * Major Point-Release Release Process: Clarified the process (<release> copied to <release>-release at the start, and that post-fork fix PRs need manual inclusion; proposals for better coordination via milestones and automation. * Telemetry Bot: Crash reports are not being reported in Redmine; the team is working on it: the collected reports will be ticketed soon. As usual the full notes are available at https://pad.ceph.com/p/csc-weekly-minutes -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO https://clyso.com | dan.vanderster(a)clyso.com

1 0

singular PG is snaptrimming for ever since pacific -> reef 18.2.6 upgrade
by Boris 05 May '25

05 May '25

Hi, we have one cluster where one PG that seems to be snaptrimming forever. It sometimes seem to hinder other PGs to on the same main OSD to snaptrim as well. We've synced out the OSD but it still keeps happening. It was gone sometime in between but now it is back again with the same PG. And the constant factor is always PG 3.c1c. Anyone got an idea what I can do? Is it possible to "sync out and recreate" a PG, like I can do with an OSD? Here is some output from the pg list Cheers Boris # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:12:21 PM UTC 2025 3.1b active+clean+snaptrim_wait 3m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 3m [156,117,32]p156 3.65d active+clean+snaptrim_wait 3m [156,263,9]p156 3.721 active+clean+snaptrim_wait 3m [156,23,190]p156 3.849 active+clean+snaptrim_wait 3m [156,135,31]p156 3.abb active+clean+snaptrim_wait 3m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 3m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 3m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 3m [156,266,281]p156 3.103f active+clean+snaptrim_wait 3m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 3m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 3m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 3m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 3m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 3m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 3m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 3m [156,212,134]p156 3.1e74 active+clean+snaptrim 3m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 3m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 3m [156,190,283]p156 # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:47:38 PM UTC 2025 3.1b active+clean+snaptrim_wait 2m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 2m [156,117,32]p156 3.65d active+clean+snaptrim_wait 2m [156,263,9]p156 3.721 active+clean+snaptrim_wait 2m [156,23,190]p156 3.849 active+clean+snaptrim_wait 2m [156,135,31]p156 3.abb active+clean+snaptrim_wait 2m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 2m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 2m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 2m [156,266,281]p156 3.103f active+clean+snaptrim_wait 2m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 2m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 2m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 2m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 2m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 2m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 2m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 2m [156,212,134]p156 3.1e74 active+clean+snaptrim 2m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 2m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 2m [156,190,283]p156 # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 04:24:55 PM UTC 2025 3.1b active+clean+snaptrim_wait 55s [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 55s [156,117,32]p156 3.65d active+clean+snaptrim_wait 55s [156,263,9]p156 3.721 active+clean+snaptrim_wait 55s [156,23,190]p156 3.849 active+clean+snaptrim_wait 55s [156,135,31]p156 3.abb active+clean+snaptrim_wait 55s [156,70,262]p156 3.c1c active+clean+snaptrim 34s [156,111,11]p156 3.f83 active+clean+snaptrim_wait 55s [156,119,97]p156 3.f8a active+clean+snaptrim_wait 55s [156,266,281]p156 3.103f active+clean+snaptrim_wait 55s [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 55s [156,135,280]p156 3.14ea active+clean+snaptrim_wait 55s [156,118,103]p156 3.1688 active+clean+snaptrim_wait 55s [156,221,98]p156 3.1848 active+clean+snaptrim_wait 55s [156,268,68]p156 3.18ca active+clean+snaptrim_wait 55s [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 55s [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 55s [156,212,134]p156 3.1e74 active+clean+snaptrim_wait 34s [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 55s [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 55s [156,190,283]p156 # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Mon May 5 06:32:53 AM UTC 2025 3.c1c active+clean+snaptrim 41h [165,111,25]p165 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 3

Ceph usage doubled after update
by Yunus Emre Sarıpınar 05 May '25

05 May '25

I have a ceph cluster created with nautilus and I upgraded to octopus later. I was have 24x node and I added 8x new nodes to the cluster. Balancer is upmap enabled. I increased my PG count from 8192 to 16384. I had to set reweight 0.8 on new OSD's to solve full usage (There was only 1TB left because of the faulted balance) Now cluster health is OK and distribution is balanced by my manual fix. data: pools: 4 pools, 16545 pgs objects: 173.32M objects, 82 TiB usage: 436 TiB used, 235 TiB / 671 TiB avail pgs: 15930 active+clean 579 active+clean+snaptrim_wait 36 active+clean+snaptrim Before adding 8 nodes to the cluster, the usage was 215 TiB used. I did the update about 2 months ago and my usage area still hasn't decreased. Why did the usage double, how can I solve it?

3 2

Excited to Support the Ceph User Community!
by Anthony Middleton 03 May '25

03 May '25

Hello Ceph Users, I'm Anthony Middleton, and I'm excited to introduce myself as the new Ceph Community Manager. I'm based in Chicago, IL, and bring seven years of experience supporting users across tech communities, including Confluent and Hedera Hashgraph, focusing on continually improving user experience and engagement. My goal is to ensure that every Ceph user feels heard, supported, and connected. I'll be working closely with you to gather feedback, strengthen support systems, and build programs that make the community even stronger. Outside of work, I'm active in the Chicago Gay Men's Chorus and the Chicago Dragons Rugby Club — two amazing communities that remind me every day of the value of connection and teamwork. Let's connect — I'd love to hear your ideas and experiences. Connect with me on LinkedIn <https://www.linkedin.com/in/middleton-anthony/>and Bluesky <https://bsky.app/profile/anthonymicmidd.bsky.social>! Looking forward to growing with you, Best, *Anthony Middleton*

2 1

Adding crush rules and pool crush rules to prometheus
by Boris 02 May '25

02 May '25

Hi, is it possible to have the crush rules, the one you get from ceph osd crush rule dump, and which pool uses which crush rule into the prometheus exporter? It would help us to monitor this. Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Optimal number of placement groups per OSD
by gagan tiwari 01 May '25

01 May '25

Hi Guys, I have 7 OSD nodes with 10X15T NVME disk on each OSD node. To start with , I want to use only 8X15T disk on each osd node and keep 2X15 disk spare in case of any disk failure and recovery event. I am going to use the 4X2 EC CephFS data pool to store data. So, with the above set-up, what will be the optimal number of placement groups per OSD. As per the PG calculator :- ( 8X7X100 ) / 6 = 933.33 nearest power of 2 is 1024. With 1024 placement groups distributed across 56 OSDs, that evaluates to approximately 18 placement groups per OSD. I don't think its optimal as Ceph doc recommends 50-100 PG per OSD. So, am I doing something wrong? or missing something white calculating the number of PG per OSD. Also, will it be best practice to keep 2X15T disk spare on each OSD or should I use all of them. Also, I am going to deploy 7 OSD nodes across 4 Racks and will be using the failure domain as "rack" to have Ceph handle the entire rack failure. I hope this will provide more protection. Please advise. Thanks, Gagan Thanks, Gagan.

2 4

Doubled numbers of PGs from 8192 to 16384 - backfill bottlenecked
by Torkil Svensgaard 01 May '25

01 May '25

Hi 19.2.2 with WPQ, osd-max-backfills and osd-recovery-max-active are set to 1. In order to get our PG sizes better aligned we doubled the number of PGs on the pool with the largest PG size. The pool is HDD with DB/WAL on SATA SSD and HDD sizes between 2TB and 20TB and PG size was ~140GB before the doubling. All went well but looking at this: " cluster: id: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-flash1,ceph-flash2,ceph-flash3,ceph-e3s2,ceph-e3s3 (age 2d) mgr: ceph-flash2.utlhuz(active, since 4d), standbys: ceph-flash3.ciudre, ceph-flash1.erhakb mds: 1/1 daemons up, 2 standby osd: 576 osds: 576 up (since 2h), 576 in (since 3d); 8767 remapped pgs data: volumes: 1/1 healthy pools: 18 pools, 25249 pgs objects: 683.85M objects, 1.6 PiB usage: 2.7 PiB used, 1.9 PiB / 4.5 PiB avail pgs: 842769842/3951610673 objects misplaced (21.327%) 16481 active+clean 8762 active+remapped+backfill_wait 6 active+remapped+backfilling io: client: 374 MiB/s rd, 14 MiB/s wr, 2.86k op/s rd, 410 op/s wr recovery: 153 MiB/s, 38 objects/s " The balancer was running and seemingly making very small changes: " [root@lazy ~]# ceph balancer status { "active": true, "last_optimize_duration": "0:00:01.012679", "last_optimize_started": "Mon Apr 28 10:01:24 2025", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Optimization plan created successfully", "plans": [] } " I tried the pgremapper and that gave a nice throughput in the beginning but now it seems to be about back to scratch: " cluster: id: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-flash1,ceph-flash2,ceph-flash3,ceph-e3s2,ceph-e3s3 (age 7h) mgr: ceph-flash2.utlhuz(active, since 5d), standbys: ceph-flash3.ciudre, ceph-flash1.erhakb mds: 1/1 daemons up, 2 standby osd: 577 osds: 577 up (since 5h), 577 in (since 31h); 7392 remapped pgs data: volumes: 1/1 healthy pools: 18 pools, 25249 pgs objects: 683.96M objects, 1.6 PiB usage: 2.7 PiB used, 1.9 PiB / 4.5 PiB avail pgs: 459889823/3952312686 objects misplaced (11.636%) 17857 active+clean 7384 active+remapped+backfill_wait 8 active+remapped+backfilling io: client: 70 MiB/s rd, 15 MiB/s wr, 689 op/s rd, 378 op/s wr recovery: 294 MiB/s, 75 objects/s " Balancer is still running and making small changes. This is going to take a while, any tips on how to escape the apparent bottleneck? Is having many PGs misplaced actually counter productive, so it would be better to throttle via target_max_misplaced_ratio? I was thinking it was better to let the balancer balance all it could, as that would make all the moves available and decrease the risk of bottlenecking. Thanks. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

6 13

Understanding CRUSH rule changes to existing pools
by Conrad Hoffmann 30 Apr '25

30 Apr '25

Hello, I am trying to get a better understanding of how Ceph handles changes to CRUSH rules, such as changing the failure domain. I performed this (maybe somewhat academic, sorry) excercise, and would love to verify my conclusions (or get a good explanation of my observations): Starting point: - 2 Nodes - 6 OSDs each - osd_crush_chooseleaf_type = 0 (OSD) - pool size = 3 - pool min size = 2 As expected, the default CRUSH rule would distribute each PG across three random OSDs, without regard for the host they are on. I then added the following CRUSH rule (initially unused): rule replicated_three_over_two { id 1 type replicated step take default step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } Tests with `crushtool` gave me confidence that it worked as I expected: unlike the default rule, this one always made sure at least one copy was on a different host. When I applied this to rule to an existing pool, for a moment I got a HEALTH_WARN "Reduced data availability: 6 pgs peering" and the PGs being shown as inactive (cluster then proceeded to heal itself). This sort of surprised me. I have read threads like [1], where it seems even larger remappings go down without a hitch. I changed back to the default rule, this time with `osd set norebalance` set. But the outcome was the same (brief reduced availability, then healed). Having read more about peering, I now assume that a very brief interruption due to peering is likely unavoidable? But, more importantly, the length of the interruption probably does not scale with the size of the pool (I don't have much data in there right now). So changing the rule for a big pool would still mean only a brief interruption, is that correct? Trying to wrap my head around what exactly the lower-level implications of changing the CRUSH rule could be, I remembered some forum thread I had read, but initially dismissed as somewhat incoherent. Someone claimed that one should set the pool's min_size to 1 when changing rules. For science, I gave this a try. Interestingly enough, the warning about reduced data availability didn't happen (even though there were again inactive/peering PGs). I arrived at the following mental model of what happens on a CRUSH rule update: the lead OSD remains steady, but the placement of the two replicas is re-calculated. In case they need to be moved, peering needs to happen, briefly blocking. But, in case of min_size = 1, the lead OSD, I don't know, says YOLO and accepts writes without having finished peering? If so, why would the PG be considered inactive, though? And, out of curiosity: besides the obvious possibility of corrupting your only copy of an object, are there any implications to setting min_size to 1 before changing a rule? Like, could a write in the right moment cause the peering process to permanently fail or something? [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/UUXT4EBPZVO… Thanks in advance for any enlightenment :) Conrad

3 2