- ceph-users - lists.ceph.io

Re: Disable cache tier
by Eugen Block 05 Feb '26

05 Feb '26

Hi, I've been searching the list for my own report from last year maybe, when we finally got rid of our cache tier as well. I can't really find it though, so I'm gonna summarize again how we managed to get out of it. After setting the cache mode to proxy and failing to flush/evict (also with "no such file or directory" messages), I gradually reduced the cache pool size from 260 GB in steps of 40 GB or so, not sure anymore .Suddenly Ceph was able to flush and evict again. I reduced the target_max_bytes to 1000000, only rbd_header objects remained in the pool, but those couldn't be flushed/evicted anymore: failed to evict /rbd_header.133ba5e7513046: (16) Device or resource busy rbd_header.8fbda012200854 Unfortunately, we didn't find a promsing way to get rid of them while the images were in use. I had played around in a test cluster and ignored the header objects when remocing the cache tier, and it seemed to work (since the rbd_header objects should be identical in the backing pool as well), but it didn't feel safe and we were really uncomfortable trying it in production. So in the end, we shut down our VMs to be able to flush/evict everything and safely got rid of the cache tier. I know this isn't really helpful, but maybe next time I find my comment on the list. :-) Regards, Eugen Am Do., 5. Feb. 2026 um 15:16 Uhr schrieb Kees Meijs | Nefos via ceph-users <ceph-users(a)ceph.io>: > Hello list, > > After running cache tiers for years and years, we decided to turn off > caching on a mostly dormant cluster. > > Given an installation using Pacific, we followed the instructions as > dictated on: > > https://docs.ceph.com/en/pacific/rados/operations/cache-tiering/#removing-a… > > So, we set the "hot" tier to cache-mode proxy (as the documentation > mentions) and simply waited for a bit. Since nothing really seemed to > happen, we did a manual flush using cache-flush-evict-all (also as the > documentation mentions). So far, so good. > > The end result is a cache tier with still quite some (>81k) objects > within. Running cache-flush-evict-all again renders errors such as: > > failed to flush /rbd_data.d9d54166ef438d.0000000000020899: (2) No such > file or directory > rbd_data.d9d54166ef438d.000000000001a511 > > This is not what is expected. > > What are we looking at? And, how to move forward in removing the cache? > > Thanks for your help and wisdom! > > Cheers, > Kees > > P.S. The introduction text speaks of readproxy and not proxy. This > confusing since the instruction text speaks of proxy. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

CephFS Metadata loss, data present, inodes in pool, scan_links finishes instantly - Squid
by Enzo Van Thorre 05 Feb '26

05 Feb '26

Hello Ceph users and readers, I am facing a really unknown situation for me (being kinda new) with a CephFS cluster (Squid 19.2.2, Cephadm) where the data pool is intact (7.2 TB used, 4.9TB stored), inode objects exist in the recovered metadata pool, but when i try to mount the filesystem, its empty. What initially happened is that I had two groups of drives: nvme’s and spinning. I had separate pools for both, and I eventually deleted the nvme’s pool and then removed the nvmes from the machine. I never thought that they had included metadata for spinning rust’s pool, but here we are. Version: 19.2.2 Squid Metadata pool ID 1 (cephfs.cephfs.meta), Size=1 (No Replicas :(, yes I know I messed up). Data Pool ID 6 (ec_data), erasure coded with k=2 m=1. All OSDs are up. 256 PGs are active+clean. there’s then quite a bit that are inconsistent, degraded, undersized, remapped, and other stuff. Here’s what I’ve done since losing the metadata pool due to OSD purges. I tried to rebuild metadata from the data pool by doing: 1. cephfs-data-scan init --force-init 2. cephfs-data-scan scan_extents 3. cephfs-data-scan scan_inodes ceph df shows 7.2 TiB stored in pool ec_data (which is the spinning rust pool) Inodes do exist. running rados -p cephfs.cephfs.meta ls | grep "100." returns tons of objects (ex. 1000005f1f3.00000000). If i try to mount via ceph-fuse, it shows an empty root. there is no lost+found folder either. A big problem is that when I try to link the orphaned inodes to lost+found, the command finishes instantly with no output, as if it believes the cluster is clean. I run: cephfs-data-scan scan_links --filesystem cephfs # and it returns code 0, no output, and runs in <1 second So my final question is since scan_inodes has populated the metadata pool with objects (100.xxxx), but scan_links ignores them, how do I force a re-scan or just do SOMETHING so scan_links will actually link these files into lost+found? All The Best, Enzo

1 0

Re: How to disable an alertmanager alert
by Ankush Behl 05 Feb '26

05 Feb '26

Hi Chris, do you mean they show up in overview dashboard? Can you show via some screenshots where all they are coming up? On Thu, 5 Feb 2026 at 6:39 PM, Chris Palmer via ceph-users < ceph-users(a)ceph.io> wrote: > cephadm (squid & tentacle) deploys alertmanager with some default > alerts. One is CephNodeInconsistentMTU which checks if the MTU of each > interface with the same name is the same across the cluster. > > However if your cluster nodes are not identical, and have differing > interface names, we end up with lots of permanent alerts. Although I can > silence the alert (matching the alert name) for a long time, they still > show up in the dashboard. > > The alerts are different permutations of: > > Node ceph3 has a different MTU size (1500) than the median of > devices named enp4s0. > > Since that check is inappropriate in a heterogeneous cluster, what's the > recommended way of disabling it, which won't get overwritten with > redeployment of the monitoring stack? > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Disable cache tier
by Kees Meijs | Nefos 05 Feb '26

05 Feb '26

Hello list, After running cache tiers for years and years, we decided to turn off caching on a mostly dormant cluster. Given an installation using Pacific, we followed the instructions as dictated on: https://docs.ceph.com/en/pacific/rados/operations/cache-tiering/#removing-a… So, we set the "hot" tier to cache-mode proxy (as the documentation mentions) and simply waited for a bit. Since nothing really seemed to happen, we did a manual flush using cache-flush-evict-all (also as the documentation mentions). So far, so good. The end result is a cache tier with still quite some (>81k) objects within. Running cache-flush-evict-all again renders errors such as: failed to flush /rbd_data.d9d54166ef438d.0000000000020899: (2) No such file or directory rbd_data.d9d54166ef438d.000000000001a511 This is not what is expected. What are we looking at? And, how to move forward in removing the cache? Thanks for your help and wisdom! Cheers, Kees P.S. The introduction text speaks of readproxy and not proxy. This confusing since the instruction text speaks of proxy.

1 0

How to disable an alertmanager alert
by Chris Palmer 05 Feb '26

05 Feb '26

cephadm (squid & tentacle) deploys alertmanager with some default alerts. One is CephNodeInconsistentMTU which checks if the MTU of each interface with the same name is the same across the cluster. However if your cluster nodes are not identical, and have differing interface names, we end up with lots of permanent alerts. Although I can silence the alert (matching the alert name) for a long time, they still show up in the dashboard. The alerts are different permutations of: Node ceph3 has a different MTU size (1500) than the median of devices named enp4s0. Since that check is inappropriate in a heterogeneous cluster, what's the recommended way of disabling it, which won't get overwritten with redeployment of the monitoring stack? Thanks, Chris

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Kirby Haze 04 Feb '26

04 Feb '26

When reweight the drive down to 0, does the behavior change if you set it 'out' first? The only other special thing that happens with reweight 0 is that upmaps get removed for said OSD. If you think it is at the pool level you could run a simple rados bench to a pool while you reweight an OSD to see if your issue still holds. Also could be some issue with osd_op_queue being mclock if you have that, might be worth A/B testing with one OSD if you still see it. On Wed, Feb 4, 2026 at 12:31 PM Rok Jaklič via ceph-users < ceph-users(a)ceph.io> wrote: > If one HDD drive fails and smartctl shows errors and then I decide to drain > it with crush reweight 0, would Ceph try to copy/move data/pgs from data > failed disk anyway? > > Because we noticed on some non ceph clusters (raid setup, actually mail > servers), that one failed drive may hog up "app/OS" because "app" fails to > read/write to failed disk because "some queue" fills up (since app/OS is > unable for data to be read/written)? > > Could something similar happen in Ceph? > > Rok > > On Wed, Feb 4, 2026 at 9:52 AM Rok Jaklič <rjaklic(a)gmail.com> wrote: > > > > > On Wed, Feb 4, 2026 at 2:59 AM Anthony D'Atri <anthony.datri(a)gmail.com> > > wrote: > > > >> Are these rear bay drives, hence the limit of 2? Or > >> You might consider an M.2 AIC adapter card with bifurcation. M.2 > >> enterprise SSDs are sunsetting but for retrofits you should be able to > find > >> Micron 6450 units. > >> > >> What’s your workload like? > >> > > > > On average 10-50MB/s of write, with spikes up to a few hundred MB/s > during > > evening/night time; it went up to 1GB/s during tests without a problem. > All > > these are S3 workloads/tests. > > > > I would have to check that on site, RM does not show, however we are just > > about to migrate to new machines, which have 4 NVMe slots ... so I am > > really considering moving WAL/DB to NVMe, however I am still a little bit > > hesitant, since I am not really sure this will solve the problem of why > > radosgw/s3 stops after some time when setting crush reweight to 0 on one > > failed disk. We are doing the same thing on HPC where radosgw/s3 is not > > used and we are not experiencing this problem there. If we move WAL/DB to > > NVMe, and if one NVMe fails and we have to recover 10 OSDs for example, > it > > would take much longer than if just 1 OSD has to be recovered (while > users > > being unable to access s3). > > > > --- > > > > My suspicion is that when we set crush reweight of the failed disk to 0, > > all other affected disks from that pool disables some write (because of > > recovery) and some queue fills up which then stops/hangs radosgw... > > > > Rok > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

rpm update to tentacle
by Christopher Durham 04 Feb '26

04 Feb '26

I updated squid 19.2.2 to tentacle 2.2.0 using rpms on rocky9 linux. I see that the release was allegedly compiled with 'RelWithDebInfo' (some level of debug) as evidenced with: ceph versions "overall": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 37 } Is this intentional? I have never seen this before on a production release... -Chris

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Anthony D'Atri 04 Feb '26

04 Feb '26

>> . Not sure where you get your number from (10 DB devices >> max per NVMe), but the docs [0] state to not have more than 15 OSDs per NVMe: Huh, I thought it said 10. That was the rule of thumb 10 years ago when NVMe NAND SSDs were on the order of the P3600 and PCIe was what, gen 2 or 3? These days 15 probably is fine with an appropriate SSD model and PCIe gen 4+. These guidelines are there because we’re expected to provide some, but in the end there are variables including the workload. >> But you're correct about the SPOF, if one NVMe dies, all OSDs that have >> their DB/WAL on that NVMe die as well. In a sufficiently large cluster that isn’t a big deal. Even so, any cluster should be able to survive the loss of an entire host, so is this blast radius any scarier? Some admins choose to mirror offload SSDs because of this FUD, but consider: * Ceph is already handling distributed redundancy. Does it make sense to double the space amp for wall+ DB ? * If HDDs are used for the perception † that they cost less, consider that doubling them up costs more, narrowing the capex gap. My sense is that most admins elect to distribute OSDs across unmirrored offload SSDs. Costs less and burns less endurance. † With thoughtful sourcing, model selection appropriate to the workload, and TCO analysis you might be surprised. >> >> [0] >> >> https://docs.ceph.com/en/latest/start/hardware-recommendations/#minimum-har… >> >> Am Di., 3. Feb. 2026 um 15:00 Uhr schrieb Rok Jaklič via ceph-users < >> ceph-users(a)ceph.io>: >> >>> We have 28 OSDs per host and we can only have 2 NVMe per host (one being >>> used for OS) . Are these rear bay drives, hence the limit of 2? Or You might consider an M.2 AIC adapter card with bifurcation. M.2 enterprise SSDs are sunsetting but for retrofits you should be able to find Micron 6450 units. What’s your workload like? >>> .. and if I remember correctly there is max 10 OSDs/NVMe >>> recommended, that's why we decided to go just for HDD based clusters at >> the >>> beinging. >>> >>> We have 2 clusters this way, one being for HPC (no radosgw/s3) and other >>> for "users" (radosgw/s3), running over 4 years now ... works ok, >>> performance is ok, just we have this problem where we have to do gentle >>> reweight of a failed OSDs. With modern releases the upmap / balancer strategy is better, you might be using the name of the old script to describe it. >>> >>> Thanks for the info, we will consider NVMe ... although then there is >> SPOF >>> for those OSDs which have DB on NVMe? Ceph can handle it. Enterprise SSDs have lower AFR than spinners. Keep the firmware updated. >>> >>> Rok >>> >>> On Tue, Feb 3, 2026 at 2:35 PM Robert Sander via ceph-users < >>> ceph-users(a)ceph.io> wrote: >>> >>>> Am 03.02.26 um 2:31 PM schrieb Rok Jaklič: >>>>> On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users <ceph- >>>>> users(a)ceph.io <mailto:ceph-users@ceph.io>> wrote: >>>>> >>>>>> Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: >>>>>> >>>>>>> 2 OSD(s) experiencing slow operations in >> BlueStore >>>>>>> 2 OSD(s) experiencing stalled read in db device I see this when updating firmware live. The default lifetime of the warning is 24 hours, so usually this is far more ephemeral than one might think and less of a big deal. >> of >>>>>> BlueFS >>>>>> >>>>>> Are your OSDs HDD only? >>>>> >>>> >>>>> Yes. >>>>> >>>>> Does not affect users much. Usually those messages appear when we are >>>>> reweighting and changing failed disks. >>>> >>>> These HDDs will be maxed out with the recovery work and cannot serve >>>> anything else any more. >>>> >>>> I have seen HDD only clusters going into the "spiral of death" because >>>> the HDDs cannot answer fast enough. OSDs randomly dropping out making >>>> the whole system unstable. >>>> >>>> The RocksDB is such a random IO application that it is not suitable for >>>> HDDs. It should always be put on flash storage (SSD/NVMe). >>>> >>>> Regards >>>> -- >>>> Robert Sander >>>> Linux Consultant >>>> >>>> Heinlein Consulting GmbH >>>> Schwedter Str. 8/9b, 10119 Berlin >>>> >>>> https://www.heinlein-support.de >>>> >>>> Tel: +49 30 405051 - 0 >>>> Fax: +49 30 405051 - 19 >>>> >>>> Amtsgericht Berlin-Charlottenburg - HRB 220009 B >>>> Geschäftsführer: Peer Heinlein - Sitz: Berlin >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: Limiting number of object versions per key in RGW (hotspot / bucket index shard growth)
by Frédéric Nass 04 Feb '26

04 Feb '26

Hi Ansgar, Since Squid (v19) and PR #54152 [1], the RGW Lifecycle supports the 'NoncurrentDays' and 'NewerNoncurrentVersions' filters. Would that work for your use case? You can find examples of how to use these filters here [2]. Note that PR also added support for ObjectSizeGreater(Less)Than filers to transition objects based on their size. Best regards, Frédéric Nass [1] https://github.com/ceph/ceph/pull/54152 [2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configurati… -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Squishing Squids - A Ceph Compression Guide <https://www.eventbrite.com/e/squishing-squids-a-ceph-compression-guide-tick…>, February 25th, 9am PST. https://clyso.com | frederic.nass(a)clyso.com Le mer. 4 févr. 2026 à 13:23, Ansgar Jazdzewski via ceph-users < ceph-users(a)ceph.io> a écrit : > Hi Cephers, > > we are running Ceph RGW with bucket versioning enabled (currently on > Reef, but we have also observed the same behavior on other releases). > > We are seeing an operational issue where a single object key can > accumulate extremely large numbers of versions (hundreds of thousands > up to millions). Because all versions of the same key are stored on > the same bucket index shard, this results in very large bucket-index > entries / omap growth and shard hotspots. > > In our case, one object with millions of versions caused significant > bucket index pressure and related performance/maintenance concerns. > > As far as we can tell, RGW currently does not provide a way to enforce > a limit such as “keep only the last N versions per object key,” and > existing quotas only apply at bucket/user level, not per object key. > > We wanted to ask: > * Have others seen similar cases with very high version counts for > individual keys? > * How do you handle or mitigate this operationally today > > Any advice or shared experience would be very appreciated. > > Thank you, > Ansgar Jazdzewski > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Limiting number of object versions per key in RGW (hotspot / bucket index shard growth)
by Ansgar Jazdzewski 04 Feb '26

04 Feb '26

Hi Cephers, we are running Ceph RGW with bucket versioning enabled (currently on Reef, but we have also observed the same behavior on other releases). We are seeing an operational issue where a single object key can accumulate extremely large numbers of versions (hundreds of thousands up to millions). Because all versions of the same key are stored on the same bucket index shard, this results in very large bucket-index entries / omap growth and shard hotspots. In our case, one object with millions of versions caused significant bucket index pressure and related performance/maintenance concerns. As far as we can tell, RGW currently does not provide a way to enforce a limit such as “keep only the last N versions per object key,” and existing quotas only apply at bucket/user level, not per object key. We wanted to ask: * Have others seen similar cases with very high version counts for individual keys? * How do you handle or mitigate this operationally today Any advice or shared experience would be very appreciated. Thank you, Ansgar Jazdzewski

1 0