- ceph-users - lists.ceph.io

Re: Unable to start scrub on PG with unfound object
by Konstantin Shalygin 19 Jan '26

19 Jan '26

Hi, > On 18 Jan 2026, at 21:04, Michal Strnad via ceph-users <ceph-users(a)ceph.io> wrote: > > State of PG: > > pg 133.186 is stuck undersized for 6d, current state active+backfill_unfound+undersized+degraded+remapped+forced_backfill, last acting Seems this is EC 4+2, right? > [NONE,233,768,1202,NONE,423] > > Our ceph version: 18.2.1 (Reef) > > Do you have any ideas on how to get the scrub to run, or on alternative steps we should take to verify that only this single object is lost? You was try's list all PG's objects and read them? k

1 0

Re: Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Eugen Block 19 Jan '26

19 Jan '26

Hi, check out the --reclassify parameter for crushtool, works like a charm. https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/L3XPGQMJ2Y3… Zitat von Jan Kasprzak via ceph-users <ceph-users(a)ceph.io>: > Hello, Ceph users, > > In my Ceph cluster I have only OSDs of class "hdd" now. In preparation of > adding also NVMe-based OSDs I wanted to prevent cluster rebalancing > as soon as I start adding the new OSDs. So I thought adding "class hdd" > to all my CRUSH rules would do what I want - i.e. pin all the PGs where > they are, on HDD-based OSDs. I did the following: > > ceph osd getcrushmap -o /var/tmp/crushmap.bin > crushtool -d /var/tmp/crushmap.bin > /var/tmp/crushmap.txt > vi /var/tmp/crushmap.txt > > edit: > - step take default > + step take default class hdd > > crushtool -c /var/tmp/crushmap.txt -o /var/tmp/crushmap-new.bin > ceph osd setcrushmap -i /var/tmp/crushmap-new.bin > > However, after doing this, my cluster reports 90 % PGs as remapped, > and wants to completely rebalance itself. After reverting the crush map > with > > ceph osd setcrushmap -i /var/tmp/crushmap.bin > > it quickly went back to a steady state. > > How can I pin the PGs to HDD-based OSDs where they are, without the cluster > rebalancing itself? > > The whole CRUSH map can be seen here: > > https://www.fi.muni.cz/~kas/tmp/crushmap.txt > > Thanks! > > -Yenya > > Jan Kasprzak via ceph-users wrote: >> Hello, Ceph users, >> >> how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, >> with all replicas on different hosts? >> >> One of my clusters has been upgraded to two brand new ~6TB NVMe disks >> together with previously used two 12TB HDDs per host. >> Right now I have two HDD-based OSDs per host, each with ~200GB partition >> on NVMe for metadata. The rest of NVMe is free, so I can now create >> two additional ~6TB NVMe OSDs per host. >> >> My question is how to create a CRUSH rule which would have three replicas >> on three different hosts, with one of these replicas on NVMe, and the >> remaining two on HDD. >> >> In the docs here: >> https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… >> there is an example of a SSD-first rule, which is close to what I would like >> to do, but it seems the SSD replica in that rule can end up on the same host >> as one of the HDD replicas. How can I avoid that? >> >> And a noob question -- how can I create a crush rule from the >> "rule something { ... }" text in that example? >> >> Thanks! >> >> -Yenya >> >> -- >> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >> I don't like Python; its lack of inline, anonymous, multi-statement >> functions makes me sad. --Eric Wastl >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Jan Kasprzak 19 Jan '26

19 Jan '26

Hello, Ceph users, In my Ceph cluster I have only OSDs of class "hdd" now. In preparation of adding also NVMe-based OSDs I wanted to prevent cluster rebalancing as soon as I start adding the new OSDs. So I thought adding "class hdd" to all my CRUSH rules would do what I want - i.e. pin all the PGs where they are, on HDD-based OSDs. I did the following: ceph osd getcrushmap -o /var/tmp/crushmap.bin crushtool -d /var/tmp/crushmap.bin > /var/tmp/crushmap.txt vi /var/tmp/crushmap.txt edit: - step take default + step take default class hdd crushtool -c /var/tmp/crushmap.txt -o /var/tmp/crushmap-new.bin ceph osd setcrushmap -i /var/tmp/crushmap-new.bin However, after doing this, my cluster reports 90 % PGs as remapped, and wants to completely rebalance itself. After reverting the crush map with ceph osd setcrushmap -i /var/tmp/crushmap.bin it quickly went back to a steady state. How can I pin the PGs to HDD-based OSDs where they are, without the cluster rebalancing itself? The whole CRUSH map can be seen here: https://www.fi.muni.cz/~kas/tmp/crushmap.txt Thanks! -Yenya Jan Kasprzak via ceph-users wrote: > Hello, Ceph users, > > how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, > with all replicas on different hosts? > > One of my clusters has been upgraded to two brand new ~6TB NVMe disks > together with previously used two 12TB HDDs per host. > Right now I have two HDD-based OSDs per host, each with ~200GB partition > on NVMe for metadata. The rest of NVMe is free, so I can now create > two additional ~6TB NVMe OSDs per host. > > My question is how to create a CRUSH rule which would have three replicas > on three different hosts, with one of these replicas on NVMe, and the > remaining two on HDD. > > In the docs here: > https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… > there is an example of a SSD-first rule, which is close to what I would like > to do, but it seems the SSD replica in that rule can end up on the same host > as one of the HDD replicas. How can I avoid that? > > And a noob question -- how can I create a crush rule from the > "rule something { ... }" text in that example? > > Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: Unable to start scrub on PG with unfound object
by Eugen Block 19 Jan '26

19 Jan '26

Hi, is this an EC pool, maybe k4 m2? If it is, have you tried setting min_size to k for this pool? And if you do that, please don't forget to set it back to k + 1. Do you have the 'ceph pg 133.186 query' output for this PG? What's the health status of the rest of the cluster? I'm not entirely sure in which state a PG has to be to be able to deep-scrub, but this one is in forced_backfill state. Maybe if you cancelled that (ceph pg cancel-force-backfill) a scrub could be scheduled? What's the scrub status of that PG anyway ('ceph pg dump' has that information)? And one more thing, according to this: [NONE,233,768,1202,NONE,423] OSD.233 is not the primary OSD but the secondary. Regards, Eugen Zitat von Michal Strnad via ceph-users <ceph-users(a)ceph.io>: > Hi. > > We have a single unfound object. All OSDs are up, so a simple “put > down to up” of an OSD is unlikely to recover it. > > Before we notify the object owner and mark the object as lost, I > would like to scrub the PG to confirm that only this one object is > missing. However, we are unable to force either a normal scrub or a > deep scrub on the affected PG (e.g., ceph pg scrub 133.186 or ceph > pg deep‑scrub 133.186). We even increased osd_max_scrubs from 1 to 3 > on all OSDs (which are participating on PG 133.186): > > > for i in 328 233 768 1202 84 423; do > ceph config set osd.$i osd_max_scrubs 3 > done > > Despite this, the scrub still does not start. We have set > osd_scrub_during_recovery to true. I am attaching the configuration > of the primary OSD (osd.233); the other OSDs have identical settings. > > WHO MASK LEVEL OPTION VALUE RO > global basic device_failure_prediction_mode > local osd advanced osd_max_backfills > 10 osd.233 advanced osd_max_scrubs > 3 osd advanced osd_mclock_override_recovery_settings > false osd advanced osd_op_queue > wpq * > global advanced osd_pool_default_pg_autoscale_mode warn > osd advanced osd_repair_during_recovery true > osd advanced osd_scrub_auto_repair true > osd advanced osd_scrub_during_recovery true > osd.233 advanced osd_scrub_load_threshold > 10.000000 > > > State of PG: > > pg 133.186 is stuck undersized for 6d, current state > active+backfill_unfound+undersized+degraded+remapped+forced_backfill, last > acting [NONE,233,768,1202,NONE,423] > > Our ceph version: 18.2.1 (Reef) > > Do you have any ideas on how to get the scrub to run, or on > alternative steps we should take to verify that only this single > object is lost? > > Best regards, > Michal

1 0

Unable to start scrub on PG with unfound object
by Michal Strnad 18 Jan '26

18 Jan '26

Hi. We have a single unfound object. All OSDs are up, so a simple “put down to up” of an OSD is unlikely to recover it. Before we notify the object owner and mark the object as lost, I would like to scrub the PG to confirm that only this one object is missing. However, we are unable to force either a normal scrub or a deep scrub on the affected PG (e.g., ceph pg scrub 133.186 or ceph pg deep‑scrub 133.186). We even increased osd_max_scrubs from 1 to 3 on all OSDs (which are participating on PG 133.186): for i in 328 233 768 1202 84 423; do ceph config set osd.$i osd_max_scrubs 3 done Despite this, the scrub still does not start. We have set osd_scrub_during_recovery to true. I am attaching the configuration of the primary OSD (osd.233); the other OSDs have identical settings. WHO MASK LEVEL OPTION VALUE RO global basic device_failure_prediction_mode local osd advanced osd_max_backfills 10 osd.233 advanced osd_max_scrubs 3 osd advanced osd_mclock_override_recovery_settings false osd advanced osd_op_queue wpq * global advanced osd_pool_default_pg_autoscale_mode warn osd advanced osd_repair_during_recovery true osd advanced osd_scrub_auto_repair true osd advanced osd_scrub_during_recovery true osd.233 advanced osd_scrub_load_threshold 10.000000 State of PG: pg 133.186 is stuck undersized for 6d, current state active+backfill_unfound+undersized+degraded+remapped+forced_backfill, last acting [NONE,233,768,1202,NONE,423] Our ceph version: 18.2.1 (Reef) Do you have any ideas on how to get the scrub to run, or on alternative steps we should take to verify that only this single object is lost? Best regards, Michal

1 0

mailing list's owners/admins ?
by lejeczek 18 Jan '26

18 Jan '26

Hi guys. Does anybody know if this mailing lists admins are actually humans and if so, then how to contact them? I emailed - ceph-users-owner(a)ceph.io - already, trying to get subscription (not this account) but have not heard back, it's a week. many thanks, L.

1 0

Community Manager Updates Early January
by Anthony Middleton 16 Jan '26

16 Jan '26

Hello, Ceph Friends! I hope the New Year is treating you well! I wanted to share a few updates and ongoing projects with you all before we log off for the weekend. 1. *Call for Proposals (CFPs) *are open for two upcoming Ceph Days: - Ceph Days India: *The deadline has been extended to January 18th at 11:59 PM IST*. More details can be found here: https://ceph.io/en/community/events/2026/ceph-days-india/ - Ceph Days Raleigh: *The deadline is February 16th*. For more information, visit: https://ceph.io/en/community/events/2026/ceph-days-raleigh/ 2. The recording of our latest *Ceph Tech Talk, "Running Teuthology Outside of the Sepia Lab,"* is now live. You can watch it here: https://youtu.be/6ovdJ79AqbM. If you have ideas for future tech talks, please share them by responding to this email or contacting community-manager(a)ceph.io. 3. Good news! The *original Ceph subreddit* has been released from Reddit jail. Just kidding! The group was temporarily banned due to a bot attack and insufficient monitoring. Thanks to the efforts of several community members, the account is now unbanned. Please help us keep this account active and report any suspicious activity. If you notice anything unusual, feel free to reach out to me. *Join here: https://www.reddit.com/r/ceph/ <https://www.reddit.com/r/ceph/>.* 4. I've set up a *Slack channel called #getting-started* for Ceph newcomers. If you're new to Ceph and want a welcoming space to ask questions, please join. Mentors and tech leads are also encouraged to participate. Your input would be valuable as we continue to grow our community. *Ceph Slack: https://join.slack.com/t/ceph-storage/shared_invite/zt-3jlvf8f6e-45tyKGpqkk… <https://join.slack.com/t/ceph-storage/shared_invite/zt-3jlvf8f6e-45tyKGpqkk…>* 5. Enjoy this highlight video from *Cephalocon 2025*: https://www.youtube.com/watch?v=cYfls0m9h7Y. That's all for now. If you need anything, you can always email or message me on Slack. Take care and have a great weekend! Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

RBD: exclusive lock vs. advisory locks vs. fencing
by Christian Theune 16 Jan '26

16 Jan '26

Hey, I’m currently preparing a long overdue move from the traditional advisory locks to the exclusive-lock feature and am a bit perplexed by the situation. I’d love some feedback whether I’m missing something or not. We currently use the advisory locks to determine whether it’s safe to start a VM or not. We use exclusive advisory locks on the RBD images that belong to a VM so that we get guaranteed atomicity: if we get the lock, then we’re good to go. The exclusive locks are incompatible with this as they build on the same infrastructure but with different semantics. As the docs say: exclusive locks are used to protect the RBD-internal objects (so the data on the RADOS layer) from being inconsistently manipulated but they will only serialize RBD-level concurrent access. There are some parts of the tool chain that allow having a single client (map --exclusive) but this isn’t consistently exposed and not useful as it AFAICT will undermine the ability for concurrent internal tasks like mirrorin/migration/… Now, the docs also talk about fencing - however - that seems to be missing the original atomicity promise: I can fence (blocklist) a known other client, but I can not be sure that no other client appeared in between. Without atomicity I can’t safely start using this volume. This issue can be solved on a higher level orchestrator, but at that point one loses the ability for consistent low level access to volumes (e.g. for backups, debugging or other tasks). Am I describing the situation accurately? Is this the way it is on purpose or is there room for improvement here? Thanks, Christian -- Christian Theune · ct(a)flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

1 0

Re: MDS tuning for large production cluster
by Oggian Elia 16 Jan '26

16 Jan '26

Hello Devin, I struggled a lot with MDS tuning on a full NVMe SSD cephfs only cluster. To give you an idea of the HW and sizing we have 3 MON nodes (128 cores / 128GB) and 7 OSD nodes (48cores / 256GB RAM ) with 8x 7.68TB NVMe SSD (2x OSDs per device) Obviously the MDS tuning and CephFS usage in general heavily depends on the workload and I/O patterns that are using it. To provide some context, we have ~150 nodes of Alps (https://www.cscs.ch/computers/alps ) and ~20 nodes of a Kubernetes cluster (running on VMs) mounting cephfs and doing I/O in predictable paths and directory structures. The filesystem is used as scratch file system for jobs, each jobs creates a number of directories in a known path, which makes directory pinning easier. With a single MDS it was not able to cope with the load and using this very useful dashboard (https://grafana.com/grafana/dashboards/9340-ceph-cephfs/) I was able to see MDS accumulating log segments and finally crash. One crash even led to FS corruption that I was not able to fix and I had to nuke the whole fs. That said, after upgrading to Squid I decided to scale UP MDSs to 6 and to configure ephemeral pinning on the parent directory of the jobs. This resulted in decent spread of load on the MDS servers and good overall behavior of the filesystem, avg reply latencies in the order of a few ms with some spikes up to ~100ms under heavy load, no more log segments accumulation, stable operations. This is our current MDS configuration: mds_cache_memory_limit 68719476736 mds_cache_mid 0.700000 mds_cache_reservation 0.050000 mds_cache_trim_decay_rate 0.800000 mds_cache_trim_threshold 524288 mds_cap_revoke_eviction_timeout 300.000000 mds_health_cache_threshold 1.500000 mds_log_max_segments 256 mds_max_caps_per_client 50000 mds_recall_global_max_decay_threshold 131072 mds_recall_max_caps 30000 mds_recall_max_decay_rate 1.500000 mds_recall_max_decay_threshold 131072 mds_recall_warning_decay_rate 60.000000 mds_recall_warning_threshold 262144 mds_beacon_grace 120.000000 mon_mds_skip_sanity true mds_client_delegate_inos_pct 0 mds_session_blocklist_on_evict false mds_session_blocklist_on_timeout false I don’t consider myself an expert but I hope this can be helpful. Best Regards, ELIA OGGIAN SYSTEM ENGINEER CSCS Centro Svizzero di Calcolo Scientifico Swiss National Supercomputing Centre elia.oggian(a)cscs.ch<mailto:elia.oggian@cscs.ch> www.cscs.ch<http://www.cscs.ch> From: Devin A. Bougie via ceph-users <ceph-users(a)ceph.io> Date: Thursday, 15 January 2026 at 23:37 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] MDS tuning for large production cluster We have a 19.2.3 cluster managed by cephadm with five management nodes and 21 OSD nodes. We have roughly 300 linux clients across several subnets doing cephfs kernel mounts running a wide range of applications. Each management node has 760GB of memory. We’re currently using a single active MDS daemon, but have experimented with multiple MDS daemons and directory pinning. Each OSD for the cephfs data pool uses hdd (SATA) drives, with the DB and WAL on nvme's internal to the storage node. The cephfs metadata pool is on nvme drives internal to the management nodes. After a lot of testing in attempts to quiet down persistent MDS_CLIENT_LATE_RELEASE "clients failing to respond to capability release” and MDS_CLIENT_RECALL "clients failing to respond to cache pressure” warnings, we’ve ended up with the following settings. This seems to be working well. We still get periodic MDS_CLIENT_RECALL warnings, but not nearly as many as we were seeing and they clear relatively quickly. I’d greatly appreciate any suggestions for further improvements, or any concerns anyone has with these. Many thanks, Devin ——— cephfs session_timeout 120 mds mds_cache_memory_limit 549755813888 mds mds_cache_mid 0.700000 mds mds_cache_reservation 0.100000 mds mds_cache_trim_decay_rate 0.900000 mds mds_cache_trim_threshold 524288 mds mds_cap_revoke_eviction_timeout 30.000000 mds mds_health_cache_threshold 2.000000 mds mds_log_max_segments 256 mds mds_max_caps_per_client 5000000 mds mds_recall_global_max_decay_threshold 1048576 mds mds_recall_max_caps 5000 mds mds_recall_max_decay_rate 1.500000 mds mds_recall_max_decay_threshold 524288 mds mds_recall_warning_decay_rate 120.000000 mds mds_recall_warning_threshold 262144 ——— _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

MDS tuning for large production cluster
by Devin A. Bougie 15 Jan '26

15 Jan '26

We have a 19.2.3 cluster managed by cephadm with five management nodes and 21 OSD nodes. We have roughly 300 linux clients across several subnets doing cephfs kernel mounts running a wide range of applications. Each management node has 760GB of memory. We’re currently using a single active MDS daemon, but have experimented with multiple MDS daemons and directory pinning. Each OSD for the cephfs data pool uses hdd (SATA) drives, with the DB and WAL on nvme's internal to the storage node. The cephfs metadata pool is on nvme drives internal to the management nodes. After a lot of testing in attempts to quiet down persistent MDS_CLIENT_LATE_RELEASE "clients failing to respond to capability release” and MDS_CLIENT_RECALL "clients failing to respond to cache pressure” warnings, we’ve ended up with the following settings. This seems to be working well. We still get periodic MDS_CLIENT_RECALL warnings, but not nearly as many as we were seeing and they clear relatively quickly. I’d greatly appreciate any suggestions for further improvements, or any concerns anyone has with these. Many thanks, Devin ——— cephfs session_timeout 120 mds mds_cache_memory_limit 549755813888 mds mds_cache_mid 0.700000 mds mds_cache_reservation 0.100000 mds mds_cache_trim_decay_rate 0.900000 mds mds_cache_trim_threshold 524288 mds mds_cap_revoke_eviction_timeout 30.000000 mds mds_health_cache_threshold 2.000000 mds mds_log_max_segments 256 mds mds_max_caps_per_client 5000000 mds mds_recall_global_max_decay_threshold 1048576 mds mds_recall_max_caps 5000 mds mds_recall_max_decay_rate 1.500000 mds mds_recall_max_decay_threshold 524288 mds mds_recall_warning_decay_rate 120.000000 mds mds_recall_warning_threshold 262144 ———

1 0