- ceph-users - lists.ceph.io

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Alfonso Escribano Merino 22 Jan '26

22 Jan '26

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Joshua Blanch 21 Jan '26

21 Jan '26

We ran into that allow_ec_optimization on a legacy pool and scrubbing issue recently on 20.2.0. Wrote down some notes here: https://docs.clyso.com/blog/2026/04/13/tentacle-v20.2.0-ec-bug/ Specifically it is these trackers which will be in the v20.2.1 release allow_ec issue: https://tracker.ceph.com/issues/71642 scrubbing issue: https://tracker.ceph.com/issues/73184 https://tracker.ceph.com/issues/73260 On Wed, 21 Jan 2026 at 11:17, Eric Dold via ceph-users <ceph-users(a)ceph.io> wrote: > Hi Reto, hi Bill > > I found this PR that was merged after v20.2.0 was tagged. > https://github.com/ceph/ceph/pull/66021 > > I'm running Tentacle on Ubuntu Jammy without cephadm, with EC 6+2 and > FastEC enabled. > > I'm getting inconsistent PGs with errors like these: > 3.336s0 shard 25(5) soid 3:6cc82c0a:::10002107475.0000002c:head : size > 700416 != size 696320 from auth oi > 3:6cc82c0a:::10002107475.0000002c:head(2830345'2540087 > osd.41.2825431:3886213 dirty|data_digest s 4194304 uv 17841689 dd 8f22c13e > alloc_hint [0 0 0]) > 3.336s0 shard 13(4) soid 3:6cc1c47e:::10002585665.0000045e:10279 : size > 700416 != size 696320 from auth oi > 3:6cc1c47e:::10002585665.0000045e:10279(2905787'3123318 > osd.41.2896446:3729885 dirty|data_digest s 4194304 uv 23445132 dd 365ee4af > alloc_hint [0 0 0] shard_versions={1: 2904402'3115959, 2: 2904402'3115959, > 3: 2904402'3115959, 4: 2904402'3115959, 5: 2904402'3115959}) > 3.336s0 shard 13(4) soid 3:6cc1f637:::20000007ea9.000008fb:bd : size 700416 > != size 696320 from auth oi > 3:6cc1f637:::20000007ea9.000008fb:bd(45314'680679 client.124476.0:17244371 > dirty|data_digest s 4194304 uv 274244 dd 9f754009 alloc_hint [0 0 0]) > 3.336 shard 13(4) soid 3:6cc82c71:::1000098dda0.00000237:head : candidate > size 700416 info size 696320 mismatch > 3.336 shard 60(1) soid 3:6cde2c17:::20001b3c880.00000000:head : candidate > size 4096 info size 0 mismatch > 3.336 shard 13(4) soid 3:6cde2eb0:::100000d68b9.0000000b:head : candidate > size 622592 info size 618496 mismatch > > All these errors have something to do with size. > > When I restart the OSDs the cluster is healthy until some scrub picks this > up again. > The data seems ok. > > It seems like the fix is not in the tentacle release. > Is there a future release with fixes for this planned? > > Regards > Eric > > > On Fri, 5 Dec 2025 at 16:34, Reto Gysi via ceph-users <ceph-users(a)ceph.io> > wrote: > > > Hi Bill > > > > Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < > > bill_scales(a)uk.ibm.com>: > > > > > Hi Reto, > > > > > > Sorry to hear about your problems with turning on ec optimizations. > I've > > > led the team that developed this feature, so we are keen to help > > understand > > > what happened to your system. > > > > > > Your configuration looks fine as far as being supported with ec > > > optimizations. The daemons (mons, osds and mgr) need to be running > > tentacle > > > code to use this feature, there is no requirement to update any of your > > > clients. > > > > > > Do you have any more examples of the inconsistent objects logs that you > > > can share with me? > > > > > > There was a bug that we fixed late in the development cycle which was > > > scrubbing incorrectly reporting a size mismatch for objects written > prior > > > to turning on optimizations. This is because prior to turning on > > > optimizations objects are padded to a multiple of the stripe_width, > > > afterwards objects are no longer padded. The scrub code was > occasionally > > > getting confused and incorrectly reporting an inconsistency. In this > case > > > the scrub was reporting false positive errors - there was nothing wrong > > > with the data and no problems accessing the data. I notice the log you > > > shared in the email was for a size mismatch, I'm interested whether all > > the > > > logs were for mismatched sizes. We will do some further work to confirm > > > that the fix is in the 20.2.0 tentacle release and that there are no > > > problems with the back port. > > > > > > > As far I could see the errors where all about the mismatched sizes. > > > > > > > > > > You also mention that you saw some OSD crashes. Do you have any further > > > information about these? > > > > > > > I just could cause another crash with starting up a windows VM, and > doing a > > WIndows file history backup to a drive that is on an RBD Image on pool > > rbd_ecpool where I had the allow_ec_optimization flag enabled. > > > > <disk type="network" device="disk"> > > <driver name="qemu" type="raw" cache="writethrough" io="threads" > > discard="unmap" detect_zeroes="unmap"/> > > <auth username="admin"> > > <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> > > </auth> > > <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> > > <host name="zephir" port="6789"/> > > </source> > > <target dev="vdd" bus="virtio"/> > > <alias name="virtio-disk3"/> > > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > > function="0x0"/> > > </disk> > > > > root@zephir:~# ceph -s > > cluster: > > id: 27923302-87a5-11ec-ac5b-976d21a49941 > > health: HEALTH_WARN > > 2 osds down > > Reduced data availability: 49 pgs inactive > > Degraded data redundancy: 15722192/79848340 objects degraded > > (19.690%), 246 pgs degraded, 247 pgs undersized > > > > services: > > mon: 3 daemons, quorum zephir,debian,raspi (age 20h) > [leader: > > zephir] > > mgr: zephir.enywvy(active, since 21h), standbys: > debian.nusuye > > mds: 3/3 daemons up, 3 standby > > osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped > > pgs > > cephfs-mirror: 2 daemons active (2 hosts) > > rbd-mirror: 2 daemons active (2 hosts) > > rgw: 1 daemon active (1 hosts, 1 zones) > > tcmu-runner: 5 portals active (2 hosts) > > > > data: > > volumes: 3/3 healthy > > pools: 25 pools, 450 pgs > > objects: 17.52M objects, 62 TiB > > usage: 116 TiB used, 63 TiB / 179 TiB avail > > pgs: 10.889% pgs not active > > 15722192/79848340 objects degraded (19.690%) > > 197 active+undersized+degraded > > 183 active+clean > > 49 undersized+degraded+peered > > 15 active+clean+scrubbing > > 5 active+clean+scrubbing+deep > > 1 active+undersized > > > > io: > > client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr > > > > root@zephir:~# ceph osd status > > ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE > > 0 debian 11.2T 5289G 0 0 0 0 exists,up > > 1 zephir 11.6T 4846G 0 0 0 0 exists > > 2 zephir 10.7T 5827G 0 0 2 4 exists,up > > 3 zephir 11.0T 5497G 0 0 0 0 exists > > 4 zephir 159G 1376G 0 0 2 0 exists,up > > 5 zephir 120G 1415G 0 0 0 0 exists,up > > 6 debian 11.3T 5200G 0 0 1 0 exists,up > > 7 zephir 305G 1230G 0 0 0 0 exists,up > > 8 zephir 10.9T 5594G 0 0 0 0 exists,up > > 9 zephir 11.9T 4616G 0 0 0 0 exists,up > > 10 zephir 235G 1300G 0 0 0 0 exists,up > > 11 zephir 223G 1312G 0 0 0 0 exists,up > > 12 zephir 11.0T 5479G 0 0 0 0 exists,up > > 13 debian 118G 631G 0 0 3 8 exists,up > > 14 debian 345G 1190G 0 0 0 0 exists,up > > 15 debian 11.4T 5079G 0 0 0 0 exists,up > > 16 debian 554G 3029G 0 0 3 8 exists,up > > 17 debian 12.5T 5874G 0 0 1 0 exists,up > > root@zephir:~# > > > > I attached the log of osd.1 to > > https://filebin.net/jwvs6kuqrc7hx8id > > > > Best Regards, > > > > Reto > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Eric Dold 21 Jan '26

21 Jan '26

Hi Reto, hi Bill I found this PR that was merged after v20.2.0 was tagged. https://github.com/ceph/ceph/pull/66021 I'm running Tentacle on Ubuntu Jammy without cephadm, with EC 6+2 and FastEC enabled. I'm getting inconsistent PGs with errors like these: 3.336s0 shard 25(5) soid 3:6cc82c0a:::10002107475.0000002c:head : size 700416 != size 696320 from auth oi 3:6cc82c0a:::10002107475.0000002c:head(2830345'2540087 osd.41.2825431:3886213 dirty|data_digest s 4194304 uv 17841689 dd 8f22c13e alloc_hint [0 0 0]) 3.336s0 shard 13(4) soid 3:6cc1c47e:::10002585665.0000045e:10279 : size 700416 != size 696320 from auth oi 3:6cc1c47e:::10002585665.0000045e:10279(2905787'3123318 osd.41.2896446:3729885 dirty|data_digest s 4194304 uv 23445132 dd 365ee4af alloc_hint [0 0 0] shard_versions={1: 2904402'3115959, 2: 2904402'3115959, 3: 2904402'3115959, 4: 2904402'3115959, 5: 2904402'3115959}) 3.336s0 shard 13(4) soid 3:6cc1f637:::20000007ea9.000008fb:bd : size 700416 != size 696320 from auth oi 3:6cc1f637:::20000007ea9.000008fb:bd(45314'680679 client.124476.0:17244371 dirty|data_digest s 4194304 uv 274244 dd 9f754009 alloc_hint [0 0 0]) 3.336 shard 13(4) soid 3:6cc82c71:::1000098dda0.00000237:head : candidate size 700416 info size 696320 mismatch 3.336 shard 60(1) soid 3:6cde2c17:::20001b3c880.00000000:head : candidate size 4096 info size 0 mismatch 3.336 shard 13(4) soid 3:6cde2eb0:::100000d68b9.0000000b:head : candidate size 622592 info size 618496 mismatch All these errors have something to do with size. When I restart the OSDs the cluster is healthy until some scrub picks this up again. The data seems ok. It seems like the fix is not in the tentacle release. Is there a future release with fixes for this planned? Regards Eric On Fri, 5 Dec 2025 at 16:34, Reto Gysi via ceph-users <ceph-users(a)ceph.io> wrote: > Hi Bill > > Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < > bill_scales(a)uk.ibm.com>: > > > Hi Reto, > > > > Sorry to hear about your problems with turning on ec optimizations. I've > > led the team that developed this feature, so we are keen to help > understand > > what happened to your system. > > > > Your configuration looks fine as far as being supported with ec > > optimizations. The daemons (mons, osds and mgr) need to be running > tentacle > > code to use this feature, there is no requirement to update any of your > > clients. > > > > Do you have any more examples of the inconsistent objects logs that you > > can share with me? > > > > There was a bug that we fixed late in the development cycle which was > > scrubbing incorrectly reporting a size mismatch for objects written prior > > to turning on optimizations. This is because prior to turning on > > optimizations objects are padded to a multiple of the stripe_width, > > afterwards objects are no longer padded. The scrub code was occasionally > > getting confused and incorrectly reporting an inconsistency. In this case > > the scrub was reporting false positive errors - there was nothing wrong > > with the data and no problems accessing the data. I notice the log you > > shared in the email was for a size mismatch, I'm interested whether all > the > > logs were for mismatched sizes. We will do some further work to confirm > > that the fix is in the 20.2.0 tentacle release and that there are no > > problems with the back port. > > > > As far I could see the errors where all about the mismatched sizes. > > > > > > You also mention that you saw some OSD crashes. Do you have any further > > information about these? > > > > I just could cause another crash with starting up a windows VM, and doing a > WIndows file history backup to a drive that is on an RBD Image on pool > rbd_ecpool where I had the allow_ec_optimization flag enabled. > > <disk type="network" device="disk"> > <driver name="qemu" type="raw" cache="writethrough" io="threads" > discard="unmap" detect_zeroes="unmap"/> > <auth username="admin"> > <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> > </auth> > <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> > <host name="zephir" port="6789"/> > </source> > <target dev="vdd" bus="virtio"/> > <alias name="virtio-disk3"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x0"/> > </disk> > > root@zephir:~# ceph -s > cluster: > id: 27923302-87a5-11ec-ac5b-976d21a49941 > health: HEALTH_WARN > 2 osds down > Reduced data availability: 49 pgs inactive > Degraded data redundancy: 15722192/79848340 objects degraded > (19.690%), 246 pgs degraded, 247 pgs undersized > > services: > mon: 3 daemons, quorum zephir,debian,raspi (age 20h) [leader: > zephir] > mgr: zephir.enywvy(active, since 21h), standbys: debian.nusuye > mds: 3/3 daemons up, 3 standby > osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped > pgs > cephfs-mirror: 2 daemons active (2 hosts) > rbd-mirror: 2 daemons active (2 hosts) > rgw: 1 daemon active (1 hosts, 1 zones) > tcmu-runner: 5 portals active (2 hosts) > > data: > volumes: 3/3 healthy > pools: 25 pools, 450 pgs > objects: 17.52M objects, 62 TiB > usage: 116 TiB used, 63 TiB / 179 TiB avail > pgs: 10.889% pgs not active > 15722192/79848340 objects degraded (19.690%) > 197 active+undersized+degraded > 183 active+clean > 49 undersized+degraded+peered > 15 active+clean+scrubbing > 5 active+clean+scrubbing+deep > 1 active+undersized > > io: > client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr > > root@zephir:~# ceph osd status > ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE > 0 debian 11.2T 5289G 0 0 0 0 exists,up > 1 zephir 11.6T 4846G 0 0 0 0 exists > 2 zephir 10.7T 5827G 0 0 2 4 exists,up > 3 zephir 11.0T 5497G 0 0 0 0 exists > 4 zephir 159G 1376G 0 0 2 0 exists,up > 5 zephir 120G 1415G 0 0 0 0 exists,up > 6 debian 11.3T 5200G 0 0 1 0 exists,up > 7 zephir 305G 1230G 0 0 0 0 exists,up > 8 zephir 10.9T 5594G 0 0 0 0 exists,up > 9 zephir 11.9T 4616G 0 0 0 0 exists,up > 10 zephir 235G 1300G 0 0 0 0 exists,up > 11 zephir 223G 1312G 0 0 0 0 exists,up > 12 zephir 11.0T 5479G 0 0 0 0 exists,up > 13 debian 118G 631G 0 0 3 8 exists,up > 14 debian 345G 1190G 0 0 0 0 exists,up > 15 debian 11.4T 5079G 0 0 0 0 exists,up > 16 debian 554G 3029G 0 0 3 8 exists,up > 17 debian 12.5T 5874G 0 0 1 0 exists,up > root@zephir:~# > > I attached the log of osd.1 to > https://filebin.net/jwvs6kuqrc7hx8id > > Best Regards, > > Reto > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

rgw - migrate data from replicated to ec pool
by Boris 21 Jan '26

21 Jan '26

Hi, we want to migrate the data from a replicated to an EC pool. Is there a way to do it without downtime? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Jan Kasprzak 20 Jan '26

20 Jan '26

Andreas, Anthony, Andreas Eriksson via ceph-users wrote: > I've done exactly this recently. > > Changing crush rule to "step take default class hdd". And with 'nobackfill' > set, I ran "pgremapper cancel-backfill --yes" two or three times > to make the misplaced objects reduce from 95% -> 18%. OK, this helped somewhat. Both pgremapper and upmap-remapped.py failed for me, I had to do this: ceph osd set-require-min-compat-client octopus Then I edited the crush rule for my biggest replicated pool adding "class hdd" to the line "step take default", compiled and set a new crush map, and ran pgremapper. Then there were no remapped pgs. So far good. Then I added "class hdd" to the crush rules of all the other pools as well and set a new crush map. After that, pgremapper printed the following type errors: WARNING: pg 77.1b: conflicting mapping 34->9 found when trying to map 39->34 WARNING: pg 71.12: conflicting mapping 8->21 found when trying to map 6->8 WARNING: pg 68.11: conflicting mapping 25->23 found when trying to map 37->25 WARNING: pg 68.11: conflicting mapping 4->0 found when trying to map 11->4 (many more probably for all PGs of these three pools, with pools 68, 71, and 77 being my erasure-coded pools). ./upmap-remapped.py | sh -x apparently does not print this type of error. Now I have 56 out of ~650 pgs remapped, and about 6 % of objects misplaced. Interestingly enough, most of the remapped pgs are from my main replicated pool, which were not remapped after the first run of pgremapper. But 6 % this is small enough, so I am letting it remap itself now. Thanks! -Yenya > > This was on a Quincy (17.2.x) cluster. > > On 1/19/26 5:48 PM, Anthony D'Atri via ceph-users wrote: > > I’m surprised that this would cause data to move. > > > > > How can I pin the PGs to HDD-based OSDs where they are, without the cluster > > > rebalancing itself? > > You can use upmap-remapped.py, though the balancer will progressively still move them if that’s what CRUSH really wants. > > > > I have an RFE in to at the very least raise a warning if any rules/pools do not specify a device class, ideally setting one by default. > > > > > The whole CRUSH map can be seen here: > > > > > > https://www.fi.muni.cz/~kas/tmp/crushmap.txt > > > > > > Thanks! > > > > > > -Yenya > > > > > > Jan Kasprzak via ceph-users wrote: > > > > Hello, Ceph users, > > > > > > > > how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, > > > > with all replicas on different hosts? > > > > > > > > One of my clusters has been upgraded to two brand new ~6TB NVMe disks > > > > together with previously used two 12TB HDDs per host. > > > > Right now I have two HDD-based OSDs per host, each with ~200GB partition > > > > on NVMe for metadata. The rest of NVMe is free, so I can now create > > > > two additional ~6TB NVMe OSDs per host. > > > > > > > > My question is how to create a CRUSH rule which would have three replicas > > > > on three different hosts, with one of these replicas on NVMe, and the > > > > remaining two on HDD. > > > > > > > > In the docs here: > > > > https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… > > > > there is an example of a SSD-first rule, which is close to what I would like > > > > to do, but it seems the SSD replica in that rule can end up on the same host > > > > as one of the HDD replicas. How can I avoid that? > > > > > > > > And a noob question -- how can I create a crush rule from the > > > > "rule something { ... }" text in that example? > > > > > > > > Thanks! > > > > > > > > -Yenya > > > > > > > > -- > > > > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > > > > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > > > > I don't like Python; its lack of inline, anonymous, multi-statement > > > > functions makes me sad. --Eric Wastl > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > -- > > > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > > > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > > > I don't like Python; its lack of inline, anonymous, multi-statement > > > functions makes me sad. --Eric Wastl > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: MDS tuning for large production cluster
by Devin A. Bougie 20 Jan '26

20 Jan '26

Thanks so much, Elia! I greatly appreciate your reply and pointer to the CephFS dashboard! We continue to get a few “MDS_CLIENT_RECALL” "clients failing to respond to cache pressure” messages each day, but it’s much quieter than it used to be and overall everything seems to be working well. I’m afraid our workload is *very* unpredictable and varies greatly between clients and even processes within clients, so maybe a few MDS_CLIENT_RECALL’s are to be expected. I’ll continue to play with things a bit, but again appreciate you taking the time to glance at our tuning and share what you’ve done. Sincerely, Devin > On Jan 16, 2026, at 6:12 AM, Oggian Elia via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello Devin, > > I struggled a lot with MDS tuning on a full NVMe SSD cephfs only cluster. > > To give you an idea of the HW and sizing we have 3 MON nodes (128 cores / 128GB) and 7 OSD nodes (48cores / 256GB RAM ) with 8x 7.68TB NVMe SSD (2x OSDs per device) > > Obviously the MDS tuning and CephFS usage in general heavily depends on the workload and I/O patterns that are using it. > > To provide some context, we have ~150 nodes of Alps (https://www.cscs.ch/computers/alps ) and ~20 nodes of a Kubernetes cluster (running on VMs) mounting cephfs and doing I/O in predictable paths and directory structures. The filesystem is used as scratch file system for jobs, each jobs creates a number of directories in a known path, which makes directory pinning easier. > > With a single MDS it was not able to cope with the load and using this very useful dashboard (https://grafana.com/grafana/dashboards/9340-ceph-cephfs/) I was able to see MDS accumulating log segments and finally crash. One crash even led to FS corruption that I was not able to fix and I had to nuke the whole fs. > > That said, after upgrading to Squid I decided to scale UP MDSs to 6 and to configure ephemeral pinning on the parent directory of the jobs. This resulted in decent spread of load on the MDS servers and good overall behavior of the filesystem, avg reply latencies in the order of a few ms with some spikes up to ~100ms under heavy load, no more log segments accumulation, stable operations. > > This is our current MDS configuration: > > mds_cache_memory_limit 68719476736 > mds_cache_mid 0.700000 > mds_cache_reservation 0.050000 > mds_cache_trim_decay_rate 0.800000 > mds_cache_trim_threshold 524288 > mds_cap_revoke_eviction_timeout 300.000000 > mds_health_cache_threshold 1.500000 > mds_log_max_segments 256 > mds_max_caps_per_client 50000 > mds_recall_global_max_decay_threshold 131072 > mds_recall_max_caps 30000 > mds_recall_max_decay_rate 1.500000 > mds_recall_max_decay_threshold 131072 > mds_recall_warning_decay_rate 60.000000 > mds_recall_warning_threshold 262144 > > mds_beacon_grace 120.000000 > mon_mds_skip_sanity true > mds_client_delegate_inos_pct 0 > mds_session_blocklist_on_evict false > mds_session_blocklist_on_timeout false > > > I don’t consider myself an expert but I hope this can be helpful. > > Best Regards, > > > ELIA OGGIAN > SYSTEM ENGINEER > > CSCS > Centro Svizzero di Calcolo Scientifico > Swiss National Supercomputing Centre > elia.oggian(a)cscs.ch<mailto:elia.oggian@cscs.ch> > > > From: Devin A. Bougie via ceph-users <ceph-users(a)ceph.io> > Date: Thursday, 15 January 2026 at 23:37 > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] MDS tuning for large production cluster > > We have a 19.2.3 cluster managed by cephadm with five management nodes and 21 OSD nodes. We have roughly 300 linux clients across several subnets doing cephfs kernel mounts running a wide range of applications. > > Each management node has 760GB of memory. We’re currently using a single active MDS daemon, but have experimented with multiple MDS daemons and directory pinning. > > Each OSD for the cephfs data pool uses hdd (SATA) drives, with the DB and WAL on nvme's internal to the storage node. The cephfs metadata pool is on nvme drives internal to the management nodes. > > After a lot of testing in attempts to quiet down persistent MDS_CLIENT_LATE_RELEASE "clients failing to respond to capability release” and MDS_CLIENT_RECALL "clients failing to respond to cache pressure” warnings, we’ve ended up with the following settings. This seems to be working well. We still get periodic MDS_CLIENT_RECALL warnings, but not nearly as many as we were seeing and they clear relatively quickly. > > I’d greatly appreciate any suggestions for further improvements, or any concerns anyone has with these. > > Many thanks, > Devin > > ——— > cephfs session_timeout 120 > mds mds_cache_memory_limit 549755813888 mds mds_cache_mid 0.700000 mds mds_cache_reservation 0.100000 mds mds_cache_trim_decay_rate 0.900000 mds mds_cache_trim_threshold 524288 mds mds_cap_revoke_eviction_timeout 30.000000 mds mds_health_cache_threshold 2.000000 mds mds_log_max_segments 256 mds mds_max_caps_per_client 5000000 mds mds_recall_global_max_decay_threshold 1048576 mds mds_recall_max_caps 5000 mds mds_recall_max_decay_rate 1.500000 mds mds_recall_max_decay_threshold 524288 mds mds_recall_warning_decay_rate 120.000000 mds mds_recall_warning_threshold 262144 > ——— > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Metrics for multi-site sync delay?
by Casey Bodley 20 Jan '26

20 Jan '26

for the tentacle release, "rgw_sync_delta" perf counters were added to track the "oldest incremental change not applied" in https://github.com/ceph/ceph/pull/61418. i don't believe they've been exposed by the dashboard yet On Mon, Jan 19, 2026 at 11:39 AM Matthew Vernon via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > The RGW Sync Overview dashboard has graphs for replication (objects and > byes/s) from source, polling request latency, and unsuccessful object > replications. But it doesn't have anything like "age of oldest > incremental change not applied". > > i.e. if I do radosgw-admin sync status and replication is not fully > caught up, I get lines like: > > data is behind on 1 shards > oldest incremental change not applied: 2026-01-19T13:51:27.418794+0000 [39] > > but neither is graphed (or maybe even in the output metrics). Is there a > handy way of getting these in machine-readable manner? We're thinking we > might want to e.g. alert on a high number of behind shards and/or the > age of the oldest not applied change. > > We run Reef / Cephadm. > > Thanks, > > Matthew > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Andreas Eriksson 20 Jan '26

20 Jan '26

I've done exactly this recently. Changing crush rule to "step take default class hdd". And with 'nobackfill' set, I ran "pgremapper cancel-backfill --yes" two or three times to make the misplaced objects reduce from 95% -> 18%. This was on a Quincy (17.2.x) cluster. On 1/19/26 5:48 PM, Anthony D'Atri via ceph-users wrote: > I’m surprised that this would cause data to move. > >> How can I pin the PGs to HDD-based OSDs where they are, without the cluster >> rebalancing itself? > You can use upmap-remapped.py, though the balancer will progressively still move them if that’s what CRUSH really wants. > > I have an RFE in to at the very least raise a warning if any rules/pools do not specify a device class, ideally setting one by default. > >> The whole CRUSH map can be seen here: >> >> https://www.fi.muni.cz/~kas/tmp/crushmap.txt >> >> Thanks! >> >> -Yenya >> >> Jan Kasprzak via ceph-users wrote: >>> Hello, Ceph users, >>> >>> how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, >>> with all replicas on different hosts? >>> >>> One of my clusters has been upgraded to two brand new ~6TB NVMe disks >>> together with previously used two 12TB HDDs per host. >>> Right now I have two HDD-based OSDs per host, each with ~200GB partition >>> on NVMe for metadata. The rest of NVMe is free, so I can now create >>> two additional ~6TB NVMe OSDs per host. >>> >>> My question is how to create a CRUSH rule which would have three replicas >>> on three different hosts, with one of these replicas on NVMe, and the >>> remaining two on HDD. >>> >>> In the docs here: >>> https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… >>> there is an example of a SSD-first rule, which is close to what I would like >>> to do, but it seems the SSD replica in that rule can end up on the same host >>> as one of the HDD replicas. How can I avoid that? >>> >>> And a noob question -- how can I create a crush rule from the >>> "rule something { ... }" text in that example? >>> >>> Thanks! >>> >>> -Yenya >>> >>> -- >>> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >>> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >>> I don't like Python; its lack of inline, anonymous, multi-statement >>> functions makes me sad. --Eric Wastl >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> -- >> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >> I don't like Python; its lack of inline, anonymous, multi-statement >> functions makes me sad. --Eric Wastl >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Anthony D'Atri 19 Jan '26

19 Jan '26

I’m surprised that this would cause data to move. > How can I pin the PGs to HDD-based OSDs where they are, without the cluster > rebalancing itself? You can use upmap-remapped.py, though the balancer will progressively still move them if that’s what CRUSH really wants. I have an RFE in to at the very least raise a warning if any rules/pools do not specify a device class, ideally setting one by default. > > The whole CRUSH map can be seen here: > > https://www.fi.muni.cz/~kas/tmp/crushmap.txt > > Thanks! > > -Yenya > > Jan Kasprzak via ceph-users wrote: >> Hello, Ceph users, >> >> how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, >> with all replicas on different hosts? >> >> One of my clusters has been upgraded to two brand new ~6TB NVMe disks >> together with previously used two 12TB HDDs per host. >> Right now I have two HDD-based OSDs per host, each with ~200GB partition >> on NVMe for metadata. The rest of NVMe is free, so I can now create >> two additional ~6TB NVMe OSDs per host. >> >> My question is how to create a CRUSH rule which would have three replicas >> on three different hosts, with one of these replicas on NVMe, and the >> remaining two on HDD. >> >> In the docs here: >> https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… >> there is an example of a SSD-first rule, which is close to what I would like >> to do, but it seems the SSD replica in that rule can end up on the same host >> as one of the HDD replicas. How can I avoid that? >> >> And a noob question -- how can I create a crush rule from the >> "rule something { ... }" text in that example? >> >> Thanks! >> >> -Yenya >> >> -- >> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >> I don't like Python; its lack of inline, anonymous, multi-statement >> functions makes me sad. --Eric Wastl >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Metrics for multi-site sync delay?
by Matthew Vernon 19 Jan '26

19 Jan '26

Hi, The RGW Sync Overview dashboard has graphs for replication (objects and byes/s) from source, polling request latency, and unsuccessful object replications. But it doesn't have anything like "age of oldest incremental change not applied". i.e. if I do radosgw-admin sync status and replication is not fully caught up, I get lines like: data is behind on 1 shards oldest incremental change not applied: 2026-01-19T13:51:27.418794+0000 [39] but neither is graphed (or maybe even in the output metrics). Is there a handy way of getting these in machine-readable manner? We're thinking we might want to e.g. alert on a high number of behind shards and/or the age of the oldest not applied change. We run Reef / Cephadm. Thanks, Matthew

1 0