February 2026 - ceph-users

Re: reef 18.2.8 QE validation status
by Brad Hubbard 10 Feb '26

10 Feb '26

On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < ceph-users(a)ceph.io> wrote: > On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < > ceph-users(a)ceph.io> wrote: > > > We are continuing on this release. > > > > > > powercycle - FAILED Brad, approved? > > > > Unfortunately, I don't see any logs anywhere where I have access to them so > I have no way of checking this? > OK, I have the latest logs. The three failing jobs appear to have all timed out after running for about six hours. At this time I don't know why but I continue to investigate. > > > > ceph-volume - Approved Guillaume? > > > > Patrick was reviewing some tests, please see his comments in the same > > tracker. > > TIA > > > > > > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> > > > wrote: > > > > > > > > This test branch was built loner then 2.5 weeks ago > > > > I have to rebuild it first and then will schedule krbd > > > > > > krbd approved based on > > > > > > > > > > > > https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… > > > > > > > > > https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… > > > > > > Thanks, > > > > > > Ilya > > > > > > > > > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > > > ceph-users(a)ceph.io> wrote: > > > >> > > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> > > wrote: > > > >> > > > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > > > >> > <ceph-users(a)ceph.io> wrote: > > > >> > > > > > >> > > Hi Yuri, > > > >> > > > > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < > > yweinste(a)redhat.com> > > > wrote: > > > >> > > > > > > >> > > > Note: this is an attempt to test and publish post EOL reef > > point > > > release > > > >> > > > > > > >> > > > Details of this release are summarized here: > > > >> > > > > > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > > > >> > > > Release Notes - TBD > > > >> > > > LRC upgrade - N/A > > > >> > > > > > > >> > > > distro => jammy focal centos9 windows bookworm > > > >> > > > > > > >> > > > Seeking approvals/reviews for: > > > >> > > > > > > >> > > > smoke - Laura approved? > > > >> > > > > > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > > > approved? > > > >> > > > (rados is also running on build 2 == build 1 + > > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > >> > > > > > > >> > > > rgw - Adam E approved? > > > >> > > > > > > >> > > > fs - Venky approved? > > > >> > > > > > >> > > There are lot many failures in the fs suite resembling > > > >> > > > > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: > wrong > > fs > > > >> > > type, bad option ... > > > >> > > > > > >> > > which is an issue with the testing kernel. Adding Alex who was > > > looking > > > >> > > into this issue. > > > >> > > > > > >> > > As far as the run validation is concerned, it's risky to ignore > > the > > > >> > > large number of failures. We have two options: > > > >> > > > > > >> > > - fix the testing kernel issue and rerun > > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the > > > >> > > teuthology-suite schedule command) > > > >> > > > > > >> > > The latter would be using an older kernel branch, but I think it > > > >> > > should suffice for this case. > > > >> > > > > >> > Hi Venky, > > > >> > > > > >> > With your report of the fs suite being affected as well, I carved > > out > > > >> > some time to take a look. > > > >> > > > > >> > I think it's happening because XFS maintainers disabled support > for > > V4 > > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel > > it's > > > >> > no longer compiled in by default. The proper fix would be to > create > > > an > > > >> > image with/move to V5 on-disk format; I have opened [1] to try to > > > >> > achieve that. We can also re-enable V4 on-disk format in our > kernel > > > >> > builds as a workaround. > > > >> > > > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > > >> > > > >> This should be fixed now. Yuri, please schedule a rerun for krbd. > > > >> > > > >> Thanks, > > > >> > > > >> Ilya > > > >> _______________________________________________ > > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > _______________________________________________ > > > > Dev mailing list -- dev(a)ceph.io > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > -- > Cheers, > Brad > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Cheers, Brad

1 1

tentacle and libvirt
by Piotr Pisz 09 Feb '26

09 Feb '26

Hello, We upgraded Ceph to Tentacle, and everything went fine, but all hosts running Ubuntu 22 and 24 lost connectivity to RBD at the libvirt level. What has changed in the tentacle? Best regards, Piotr

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 09 Feb '26

09 Feb '26

Still waiting for dev leads review/approval reply. On Fri, Feb 6, 2026 at 6:48 AM Ilya Dryomov via Dev <dev(a)ceph.io> wrote: > On Thu, Feb 5, 2026 at 10:59 PM Yuri Weinstein via Dev <dev(a)ceph.io> > wrote: > > > > We are continuing on this release. > > > > In the tracker https://tracker.ceph.com/issues/73906 please see runs > with the comment "on Build 5". > > This build and runs were done after the lab move. > > > > Seeking approvals for: > > > > smoke - FAILED, was reviewed and approved by Laura and Patrick, see > Slack thread for details > > rados - FAILED, Afreen, Travis, Laura, Radek approved? > > > > rgw - FAILED, Adam E approved? > > > > rbd, krbd - FAILED, Ilya approved? > > krbd approved. > > For rbd, I did a rerun [1]. The qemu-iotests.sh failure is known and > can be ignored, but the crashes in test_librbd_python.sh are concerning. > Patrick and I discussed them briefly -- they seem to be specific to > 20.04, but I don't remember seeing them before. There were multiple > passes on 20.04 in the run that I approved last year [2], so it looks > like something around Python, Cython or the build process has changed > in the interim. > > [1] > https://pulpito.ceph.com/dis-2026-02-02_19:37:16-rbd-reef-release-distro-de… > [2] > https://pulpito.ceph.com/yuriw-2025-11-18_23:00:18-rbd-reef-release-distro-… > > Thanks, > > Ilya > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

1 0

Debian Packages for Trixie
by Andrew 09 Feb '26

09 Feb '26

Hello Team, I can see that there are official packages for Debian Old-Stable (Bookworm) available on the official site: https://download.ceph.com/debian-19.2.3/dists/ Does anyone know if there is an approximate (or likely) ETA for the Debian Packages to be released for current stable version (Trixie)? Many thanks, Andrew

6 6

RGW/multisite: Segmentation fault during full sync
by Vahideh Alinouri 09 Feb '26

09 Feb '26

Hi folks, I have one realm, one zonegroup, and four zones, all running version 19.2.3. One zone was recently added to the zonegroup while it was on version 18.2.7. The newly added zone can perform data sync with the other secondary zones without issues. However, when syncing with the master zone, it gets stuck in the init state. The master zone, on the other hand, can sync with it successfully. All zones were recently upgraded from versions 18.2.4 and 18.2.7 to 19.2.3 but the problem still persists. sync status on master zone (dc07): radosgw-admin sync status realm 710cf69b-7382-47d2-aca6-03d991b00d1f (s3-cdn) zonegroup 7c01d60f-88c6-4192-baf7-d725260bf05d (s3-cdn-group) zone 03f6a8ec-008c-4cbf-8efc-d70a6013066f (s3-cdn-dc07) current time 2026-02-09T08:57:47Z zonegroup features enabled: disabled: compress-encrypted, notification_v2, resharding metadata sync: no sync (zone is master) data sync source: 1a6e33b9-8ece-4b9c-a9a5-961fa97c42c8 (s3-cdn-dc05) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shard behind shards: [93] oldest incremental change not applied: 2026-02-09T08:57:44.779833+0000 [93] source: 367dbfe9-a5f8-4101-a271-9749f25ba09c *(s3-cdn-dc10)* *syncing* full sync: 0/128 shards incremental sync: 128/128 shards 1 shard is recovering recovering shards: [125] source: 40122a7c-e594-43b7-89bb-e7ada37991c5 (s3-cdn-dc06) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source sync status on problematic zone (dc10): radosgw-admin sync status realm 710cf69b-7382-47d2-aca6-03d991b00d1f (s3-cdn) zonegroup 7c01d60f-88c6-4192-baf7-d725260bf05d (s3-cdn-group) zone 367dbfe9-a5f8-4101-a271-9749f25ba09c (s3-cdn-dc10) current time 2026-02-09T09:00:17Z zonegroup features enabled: disabled: compress-encrypted, notification_v2, resharding metadata sync: syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 03f6a8ec-008c-4cbf-8efc-d70a6013066f *(s3-cdn-dc07)* *init* full sync: 128/128 shards full sync: 0 buckets to sync incremental sync: 0/128 shards data is behind on 128 shards behind shards: [0–127] source: 1a6e33b9-8ece-4b9c-a9a5-961fa97c42c8 (s3-cdn-dc05) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shard behind shards: [93] oldest incremental change not applied: 2026-02-09T08:59:44.783203+0000 [93] 1 shard is recovering recovering shards: [41] source: 40122a7c-e594-43b7-89bb-e7ada37991c5 (s3-cdn-dc06) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source I see the following logs in the RGW sync enabled service: Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xb4) [0x5602e312a6f4] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 5: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4d7) [0x5602e3598d97] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 6: /usr/bin/radosgw(+0x837248) [0x5602e32eb248] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 7: (RGWRadosThread::Worker::entry()+0xbd) [0x5602e32ec83d] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 8: /lib64/libc.so.6(+0x8a4da) [0x7fbd2b2394da] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 9: clone() Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: debug -9999> 2026-02-09T09:05:46.708+0000 7fbbf4c19640 -1 *** Caught signal (Segmentation fault) ** Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: in thread 7fbbf4c19640 thread_name:data-sync Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 1: /lib64/libc.so.6(+0x3ebf0) [0x7fbd2b1edbf0] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 2: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x37) [0x5602e3127c97] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 3: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x481) [0x5602e3129901] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xb4) [0x5602e312a6f4] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 5: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4d7) [0x5602e3598d97] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 6: /usr/bin/radosgw(+0x837248) [0x5602e32eb248] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 7: (RGWRadosThread::Worker::entry()+0xbd) [0x5602e32ec83d] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 8: /lib64/libc.so.6(+0x8a4da) [0x7fbd2b2394da] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 9: clone() Additionally, when I restart the RGW sync enabled service on the newly added zone, the daemon crashes. If I restart the RGW sync enabled daemon directly, it also crashes. To restart RGW sync enabled successfully, I have to run: ceph config set client.rgw.s3-cdn-colocate.ceph-mon10001.rrbzdg rgw_run_sync_thread false ceph orch daemon restart rgw.s3-cdn-colocate.ceph-mon10001.rrbzdg ceph config set client.rgw.s3-cdn-colocate.ceph-mon10001.rrbzdg rgw_run_sync_thread true I saw that this bug (https://tracker.ceph.com/issues/63378) was resolved and backported to 19.2.3. However, I am still observing the same behavior.

1 0

Re: Disable cache tier
by Kees Meijs | Nefos 09 Feb '26

09 Feb '26

Moin' and thanks! I'm going to take a further look at the objects within the pool and report back my findings. Cheers, Kees On 05/02/2026 20:36, Eugen Block via ceph-users wrote: > I've been searching the list for my own report from last year maybe, when > we finally got rid of our cache tier as well. I can't really find it > though, so I'm gonna summarize again how we managed to get out of it. > > After setting the cache mode to proxy and failing to flush/evict (also with > "no such file or directory" messages), I gradually reduced the cache pool > size from 260 GB in steps of 40 GB or so, not sure anymore .Suddenly Ceph > was able to flush and evict again. > I reduced the target_max_bytes to 1000000, only rbd_header objects remained > in the pool, but those couldn't be flushed/evicted anymore: > > failed to evict /rbd_header.133ba5e7513046: (16) Device or resource busy > rbd_header.8fbda012200854 > > Unfortunately, we didn't find a promsing way to get rid of them while the > images were in use. I had played around in a test cluster and ignored the > header objects when remocing the cache tier, and it seemed to work (since > the rbd_header objects should be identical in the backing pool as well), > but it didn't feel safe and we were really uncomfortable trying it in > production. So in the end, we shut down our VMs to be able to flush/evict > everything and safely got rid of the cache tier. > > I know this isn't really helpful, but maybe next time I find my comment on > the list. 🙂

1 0

MGR Restful not accessible when doing mgr failover (squid v19.2.3)
by Andreas Eriksson 08 Feb '26

08 Feb '26

Hi, I'm doing cluster monitoring using the restful api on the ceph-mgr on port 8003. The requests are working fine as long as I have my *mgr-0* as primary. Doing a `ceph mgr fail` so that the *mgr-1* becomes active, the mgr daemon doesn't listen on port 8003. Is there any workaround for this?

1 0

Repeated crashes of mgr module in Tentacle
by Daniel Brown 08 Feb '26

08 Feb '26

Greetings — Have been seeing repeated crashes on my mgr module. Seems to run for about 45 to 50 seconds and then boom. Cephadm setup here. Did (try to) enable a couple modules lately. iostat, stats diskprediction_local, but have toggled them back off - unfortunately it hasn’t fixed the issue. I was seeing “no mgr” with ceph -s — to get around that I’ve tweaked the “StartLimitInterval” setting in the ceph-[CLUSTER-UID]@.service file down to 1m, and have 4x mgr’s setup so that I can get a couple commands run before mgr crashes and another starts. The 30m default there seems… high, imo - I was having intervals with no mgr which makes it tough to do much with the cluster - even “ceph -s” was hanging. Everything else in the cluster seems “normal” - still serving data. One other note — which I think is generally unrelated — I did upgrade one of my cluster nodes from “Plucky Puffin" (25.04) ubuntu, to “Questing Quokka” (25.10) ubuntu. After the upgrade, cephadm managed containers didn’t want to start. I tracked that down to having the ceph user userid in /etc/passwd set at 64045, but the container seeming to want userid 167. Most things under /var/lib/ceph/[CLUSTER UID]/ … appear to be owned by user/group 167:167 — I assume this is a default inside the container. Workaround here was to manually change the UID/GID for ceph in /etc/passwd and /etc/group. I’m going to imagine this is some collision between cephadm managed deployments, and how Ubuntu / apt installs cephadm. The aforementioned mgr Crashes look like: { "assert_condition": "cursor != root", "assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h", "assert_func": "virtual void PyFormatter::close_section()", "assert_line": 84, "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h: In function 'virtual void PyFormatter::close_section()' thread ffff34e55700 time 2026-02-08T13:10:07.526894+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h: 84: FAILED ceph_assert(cursor != root)\n", "assert_thread_name": "telemetry", "backtrace": [ "__kernel_rt_sigreturn()", "/lib64/libc.so.6(+0x82a78) [0xffff83603a78]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x190) [0xffff84039874]", "/usr/bin/ceph-mgr(+0xcf540) [0xaaaacdf2f540]", "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xf6c) [0xaaaacdf46ec0]", "/usr/bin/ceph-mgr(+0x105528) [0xaaaacdf65528]", "/lib64/libpython3.9.so.1.0(+0xf7bfc) [0xffff84ae9bfc]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xda3f0) [0xffff84acc3f0]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xda3f0) [0xffff84acc3f0]", "/lib64/libpython3.9.so.1.0(+0xea93c) [0xffff84adc93c]", "/lib64/libpython3.9.so.1.0(+0xcf304) [0xffff84ac1304]", "/lib64/libpython3.9.so.1.0(+0x197d78) [0xffff84b89d78]", "_PyObject_CallMethod_SizeT()", "(PyModuleRunner::serve()+0x6c) [0xaaaacdfdf6cc]", "(PyModuleRunner::PyModuleRunnerThread::entry()+0x148) [0xaaaacdfdff08]" ], "ceph_version": "20.2.0", "crash_id": "2026-02-08T13:10:07.528739Z_f18e6b74-438b-47db-9438-5a3861fdef2d", "entity_name": "mgr.hc-945901a5cad1b6e3.mtijbv", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mgr", "stack_sig": "319d76a0d71a4644f9d65f592f6e621cca918d9205fc759ab7acf6944bc77fdd", "timestamp": "2026-02-08T13:10:07.528739Z", "utsname_hostname": "hc-945901a5cad1b6e3", "utsname_machine": "aarch64", "utsname_release": "6.14.0-1010-raspi", "utsname_sysname": "Linux", "utsname_version": "#10-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 15 19:09:05 UTC 2025" }

1 0

Re: Limiting number of object versions per key in RGW (hotspot / bucket index shard growth)
by Frédéric Nass 07 Feb '26

07 Feb '26

Hi Ansgar, You're absolutely right; it makes sense to protect the infrastructure as a whole rather than relying on users to properly manage their bucket's data. This concern was discussed at Cephalocon '24 [1]. A tracker [2] had been created a couple of months before the conference to implement this feature, and the last two comments show that your message is getting some attention. This slightly different one that Sage created in 2011 is also worth mentioning [3]. Hopefully, this will be implemented soon. Cheers, Frédéric. [1] https://pad.ceph.com/p/cds24-powerusers#L128 [2] https://tracker.ceph.com/issues/68066 [3] https://tracker.ceph.com/issues/1424 -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Squishing Squids - A Ceph Compression Guide <https://www.eventbrite.com/e/squishing-squids-a-ceph-compression-guide-tick…>, February 25th, 9am PST. https://clyso.com | frederic.nass(a)clyso.com Le mer. 4 févr. 2026 à 18:53, Ansgar Jazdzewski via ceph-users < ceph-users(a)ceph.io> a écrit : > Hello Frédéric, > > thank you, that is very helpful information. > > Support for NewerNoncurrentVersions in Squid lifecycle definitely > addresses the issue for customers who configure lifecycle rules > properly. > > However, as an object storage service provider our concern is more > about cluster protection against unbounded version growth in general: > > lifecycle policies are optional and customer-controlled > * a tenant can enable versioning and never configure expiration > * a single hot key can still accumulate millions of versions and > create a bucket-index shard hotspot > * So while Squid lifecycle makes it possible for customers to > self-mitigate, we still see a gap in terms of "provider-side > guardrails" (e.g., a hard or automatic limit on versions per key). > > Do you know if there is any discussion about enforcing such limits > globally or at least providing admin-side defaults/mandatory lifecycle > policies? > > Best regards, > Ansgar > > Am Mi., 4. Feb. 2026 um 14:23 Uhr schrieb Frédéric Nass > <frederic.nass(a)clyso.com>: > > > > Hi Ansgar, > > > > Since Squid (v19) and PR #54152 [1], the RGW Lifecycle supports the > 'NoncurrentDays' and 'NewerNoncurrentVersions' filters. Would that work for > your use case? > > You can find examples of how to use these filters here [2]. > > > > Note that PR also added support for ObjectSizeGreater(Less)Than filers > to transition objects based on their size. > > > > Best regards, > > Frédéric Nass > > > > [1] https://github.com/ceph/ceph/pull/54152 > > [2] > https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configurati… > > > > -- > > Frédéric Nass > > Ceph Ambassador France | Senior Ceph Engineer @ CLYSO > > > > Squishing Squids - A Ceph Compression Guide, February 25th, 9am PST. > > https://clyso.com | frederic.nass(a)clyso.com > > > > > > Le mer. 4 févr. 2026 à 13:23, Ansgar Jazdzewski via ceph-users < > ceph-users(a)ceph.io> a écrit : > >> > >> Hi Cephers, > >> > >> we are running Ceph RGW with bucket versioning enabled (currently on > >> Reef, but we have also observed the same behavior on other releases). > >> > >> We are seeing an operational issue where a single object key can > >> accumulate extremely large numbers of versions (hundreds of thousands > >> up to millions). Because all versions of the same key are stored on > >> the same bucket index shard, this results in very large bucket-index > >> entries / omap growth and shard hotspots. > >> > >> In our case, one object with millions of versions caused significant > >> bucket index pressure and related performance/maintenance concerns. > >> > >> As far as we can tell, RGW currently does not provide a way to enforce > >> a limit such as “keep only the last N versions per object key,” and > >> existing quotas only apply at bucket/user level, not per object key. > >> > >> We wanted to ask: > >> * Have others seen similar cases with very high version counts for > >> individual keys? > >> * How do you handle or mitigate this operationally today > >> > >> Any advice or shared experience would be very appreciated. > >> > >> Thank you, > >> Ansgar Jazdzewski > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Ilya Dryomov 06 Feb '26

06 Feb '26

On Thu, Feb 5, 2026 at 10:59 PM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > > We are continuing on this release. > > In the tracker https://tracker.ceph.com/issues/73906 please see runs with the comment "on Build 5". > This build and runs were done after the lab move. > > Seeking approvals for: > > smoke - FAILED, was reviewed and approved by Laura and Patrick, see Slack thread for details > rados - FAILED, Afreen, Travis, Laura, Radek approved? > > rgw - FAILED, Adam E approved? > > rbd, krbd - FAILED, Ilya approved? krbd approved. For rbd, I did a rerun [1]. The qemu-iotests.sh failure is known and can be ignored, but the crashes in test_librbd_python.sh are concerning. Patrick and I discussed them briefly -- they seem to be specific to 20.04, but I don't remember seeing them before. There were multiple passes on 20.04 in the run that I approved last year [2], so it looks like something around Python, Cython or the build process has changed in the interim. [1] https://pulpito.ceph.com/dis-2026-02-02_19:37:16-rbd-reef-release-distro-de… [2] https://pulpito.ceph.com/yuriw-2025-11-18_23:00:18-rbd-reef-release-distro-… Thanks, Ilya

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 06 Feb '26

06 Feb '26

I tried yesterday, if I stop OSD and just set OSD to out (without reweight), then radosgw does not stop/hangs and rebalance finishes, of course with a little bit more traffic, ... So, one should not reweight a failed disk, but just stop and set failed OSD to out and leave ceph to rebalance. Thx to all. Rok On Thu, Feb 5, 2026 at 1:08 AM Kirby Haze via ceph-users <ceph-users(a)ceph.io> wrote: > When reweight the drive down to 0, does the behavior change if you set it > 'out' first? The only other special thing that happens with reweight 0 is > that upmaps get removed for said OSD. > > If you think it is at the pool level you could run a simple rados bench to > a pool while you reweight an OSD to see if your issue still holds. > > Also could be some issue with osd_op_queue being mclock if you have that, > might be worth A/B testing with one OSD if you still see it. > > On Wed, Feb 4, 2026 at 12:31 PM Rok Jaklič via ceph-users < > ceph-users(a)ceph.io> wrote: > > > If one HDD drive fails and smartctl shows errors and then I decide to > drain > > it with crush reweight 0, would Ceph try to copy/move data/pgs from data > > failed disk anyway? > > > > Because we noticed on some non ceph clusters (raid setup, actually mail > > servers), that one failed drive may hog up "app/OS" because "app" fails > to > > read/write to failed disk because "some queue" fills up (since app/OS is > > unable for data to be read/written)? > > > > Could something similar happen in Ceph? > > > > Rok > > > > On Wed, Feb 4, 2026 at 9:52 AM Rok Jaklič <rjaklic(a)gmail.com> wrote: > > > > > > > > On Wed, Feb 4, 2026 at 2:59 AM Anthony D'Atri <anthony.datri(a)gmail.com > > > > > wrote: > > > > > >> Are these rear bay drives, hence the limit of 2? Or > > >> You might consider an M.2 AIC adapter card with bifurcation. M.2 > > >> enterprise SSDs are sunsetting but for retrofits you should be able to > > find > > >> Micron 6450 units. > > >> > > >> What’s your workload like? > > >> > > > > > > On average 10-50MB/s of write, with spikes up to a few hundred MB/s > > during > > > evening/night time; it went up to 1GB/s during tests without a problem. > > All > > > these are S3 workloads/tests. > > > > > > I would have to check that on site, RM does not show, however we are > just > > > about to migrate to new machines, which have 4 NVMe slots ... so I am > > > really considering moving WAL/DB to NVMe, however I am still a little > bit > > > hesitant, since I am not really sure this will solve the problem of why > > > radosgw/s3 stops after some time when setting crush reweight to 0 on > one > > > failed disk. We are doing the same thing on HPC where radosgw/s3 is not > > > used and we are not experiencing this problem there. If we move WAL/DB > to > > > NVMe, and if one NVMe fails and we have to recover 10 OSDs for example, > > it > > > would take much longer than if just 1 OSD has to be recovered (while > > users > > > being unable to access s3). > > > > > > --- > > > > > > My suspicion is that when we set crush reweight of the failed disk to > 0, > > > all other affected disks from that pool disables some write (because of > > > recovery) and some queue fills up which then stops/hangs radosgw... > > > > > > Rok > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Brad Hubbard 05 Feb '26

05 Feb '26

On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > We are continuing on this release. > > > powercycle - FAILED Brad, approved? > Unfortunately, I don't see any logs anywhere where I have access to them so I have no way of checking this? > ceph-volume - Approved Guillaume? > > Patrick was reviewing some tests, please see his comments in the same > tracker. > TIA > > > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < > ceph-users(a)ceph.io> wrote: > > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> > > wrote: > > > > > > This test branch was built loner then 2.5 weeks ago > > > I have to rebuild it first and then will schedule krbd > > > > krbd approved based on > > > > > > > https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… > > > > > https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… > > > > Thanks, > > > > Ilya > > > > > > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > > ceph-users(a)ceph.io> wrote: > > >> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> > wrote: > > >> > > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > > >> > <ceph-users(a)ceph.io> wrote: > > >> > > > > >> > > Hi Yuri, > > >> > > > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < > yweinste(a)redhat.com> > > wrote: > > >> > > > > > >> > > > Note: this is an attempt to test and publish post EOL reef > point > > release > > >> > > > > > >> > > > Details of this release are summarized here: > > >> > > > > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > > >> > > > Release Notes - TBD > > >> > > > LRC upgrade - N/A > > >> > > > > > >> > > > distro => jammy focal centos9 windows bookworm > > >> > > > > > >> > > > Seeking approvals/reviews for: > > >> > > > > > >> > > > smoke - Laura approved? > > >> > > > > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > > approved? > > >> > > > (rados is also running on build 2 == build 1 + > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > >> > > > > > >> > > > rgw - Adam E approved? > > >> > > > > > >> > > > fs - Venky approved? > > >> > > > > >> > > There are lot many failures in the fs suite resembling > > >> > > > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong > fs > > >> > > type, bad option ... > > >> > > > > >> > > which is an issue with the testing kernel. Adding Alex who was > > looking > > >> > > into this issue. > > >> > > > > >> > > As far as the run validation is concerned, it's risky to ignore > the > > >> > > large number of failures. We have two options: > > >> > > > > >> > > - fix the testing kernel issue and rerun > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the > > >> > > teuthology-suite schedule command) > > >> > > > > >> > > The latter would be using an older kernel branch, but I think it > > >> > > should suffice for this case. > > >> > > > >> > Hi Venky, > > >> > > > >> > With your report of the fs suite being affected as well, I carved > out > > >> > some time to take a look. > > >> > > > >> > I think it's happening because XFS maintainers disabled support for > V4 > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel > it's > > >> > no longer compiled in by default. The proper fix would be to create > > an > > >> > image with/move to V5 on-disk format; I have opened [1] to try to > > >> > achieve that. We can also re-enable V4 on-disk format in our kernel > > >> > builds as a workaround. > > >> > > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > >> > > >> This should be fixed now. Yuri, please schedule a rerun for krbd. > > >> > > >> Thanks, > > >> > > >> Ilya > > >> _______________________________________________ > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > _______________________________________________ > > > Dev mailing list -- dev(a)ceph.io > > > To unsubscribe send an email to dev-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Cheers, Brad

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 05 Feb '26

05 Feb '26

We are continuing on this release. In the tracker https://tracker.ceph.com/issues/73906 please see runs with the comment "on Build 5". This build and runs were done after the lab move. Seeking approvals for: smoke - FAILED, was reviewed and approved by Laura and Patrick, see Slack thread for details rados - FAILED, Afreen, Travis, Laura, Radek approved? rgw - FAILED, Adam E approved? rbd, krbd - FAILED, Ilya approved? fs - FAILED, Venky approved? orch - FAILED, Adam K approved? upgrade/pacific-x (reef) - FAILED Laura, approved? upgrade/quincy-x (reef) - FAILED Laura, approved? upgrade/reef-p2p - FAILED Laura, approved? powercycle - FAILED Brad, approved? ceph-volume - Approved Guillaume? Patrick was reviewing some tests, please see his comments in the same tracker. TIA On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < ceph-users(a)ceph.io> wrote: > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> > wrote: > > > > This test branch was built loner then 2.5 weeks ago > > I have to rebuild it first and then will schedule krbd > > krbd approved based on > > > https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… > > https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… > > Thanks, > > Ilya > > > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > ceph-users(a)ceph.io> wrote: > >> > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: > >> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > >> > <ceph-users(a)ceph.io> wrote: > >> > > > >> > > Hi Yuri, > >> > > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > >> > > > > >> > > > Note: this is an attempt to test and publish post EOL reef point > release > >> > > > > >> > > > Details of this release are summarized here: > >> > > > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > >> > > > Release Notes - TBD > >> > > > LRC upgrade - N/A > >> > > > > >> > > > distro => jammy focal centos9 windows bookworm > >> > > > > >> > > > Seeking approvals/reviews for: > >> > > > > >> > > > smoke - Laura approved? > >> > > > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > approved? > >> > > > (rados is also running on build 2 == build 1 + > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > >> > > > > >> > > > rgw - Adam E approved? > >> > > > > >> > > > fs - Venky approved? > >> > > > >> > > There are lot many failures in the fs suite resembling > >> > > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > >> > > type, bad option ... > >> > > > >> > > which is an issue with the testing kernel. Adding Alex who was > looking > >> > > into this issue. > >> > > > >> > > As far as the run validation is concerned, it's risky to ignore the > >> > > large number of failures. We have two options: > >> > > > >> > > - fix the testing kernel issue and rerun > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the > >> > > teuthology-suite schedule command) > >> > > > >> > > The latter would be using an older kernel branch, but I think it > >> > > should suffice for this case. > >> > > >> > Hi Venky, > >> > > >> > With your report of the fs suite being affected as well, I carved out > >> > some time to take a look. > >> > > >> > I think it's happening because XFS maintainers disabled support for V4 > >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's > >> > no longer compiled in by default. The proper fix would be to create > an > >> > image with/move to V5 on-disk format; I have opened [1] to try to > >> > achieve that. We can also re-enable V4 on-disk format in our kernel > >> > builds as a workaround. > >> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > >> > >> This should be fixed now. Yuri, please schedule a rerun for krbd. > >> > >> Thanks, > >> > >> Ilya > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > > Dev mailing list -- dev(a)ceph.io > > To unsubscribe send an email to dev-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Regression for kernel mounts with "read_from_replica=localize"
by Gregory Farnum 05 Feb '26

05 Feb '26

The Ceph project has discovered a regression with the tentacle release for kernel CephFS users who mount their filesystems using the "read_from_replica={localize|balance}" configuration option. Unless you are explicitly setting this option (which you generally shouldn't) or are using a system (such as Rook) to set up "stretch mode" and handle client mounts, you are not subject to this regression. Ceph-CSI users are subject to it if using the --enable-read-affinity flag. Due to cleanup changes in the OSD for tentacle, it is now dropping write operations with these flags set -- which the kernel client is sending on every operation, including writes, when it is set to use them for reads. You can see the initial bug report at https://tracker.ceph.com/issues/73997. The first point release of tentacle will fix this regression, but due to delays from the lab migration we do not currently have an ETA. The relevant PR is https://github.com/ceph/ceph/pull/66611. We are also adding additional tests to make sure we detect problems like this ahead of time in the future. https://tracker.ceph.com/issues/74190 The Rook report: https://github.com/rook/rook/issues/16839 Our apologies for this miss. -The Ceph team

1 0

Re: Disable cache tier
by Eugen Block 05 Feb '26

05 Feb '26

Hi, I've been searching the list for my own report from last year maybe, when we finally got rid of our cache tier as well. I can't really find it though, so I'm gonna summarize again how we managed to get out of it. After setting the cache mode to proxy and failing to flush/evict (also with "no such file or directory" messages), I gradually reduced the cache pool size from 260 GB in steps of 40 GB or so, not sure anymore .Suddenly Ceph was able to flush and evict again. I reduced the target_max_bytes to 1000000, only rbd_header objects remained in the pool, but those couldn't be flushed/evicted anymore: failed to evict /rbd_header.133ba5e7513046: (16) Device or resource busy rbd_header.8fbda012200854 Unfortunately, we didn't find a promsing way to get rid of them while the images were in use. I had played around in a test cluster and ignored the header objects when remocing the cache tier, and it seemed to work (since the rbd_header objects should be identical in the backing pool as well), but it didn't feel safe and we were really uncomfortable trying it in production. So in the end, we shut down our VMs to be able to flush/evict everything and safely got rid of the cache tier. I know this isn't really helpful, but maybe next time I find my comment on the list. :-) Regards, Eugen Am Do., 5. Feb. 2026 um 15:16 Uhr schrieb Kees Meijs | Nefos via ceph-users <ceph-users(a)ceph.io>: > Hello list, > > After running cache tiers for years and years, we decided to turn off > caching on a mostly dormant cluster. > > Given an installation using Pacific, we followed the instructions as > dictated on: > > https://docs.ceph.com/en/pacific/rados/operations/cache-tiering/#removing-a… > > So, we set the "hot" tier to cache-mode proxy (as the documentation > mentions) and simply waited for a bit. Since nothing really seemed to > happen, we did a manual flush using cache-flush-evict-all (also as the > documentation mentions). So far, so good. > > The end result is a cache tier with still quite some (>81k) objects > within. Running cache-flush-evict-all again renders errors such as: > > failed to flush /rbd_data.d9d54166ef438d.0000000000020899: (2) No such > file or directory > rbd_data.d9d54166ef438d.000000000001a511 > > This is not what is expected. > > What are we looking at? And, how to move forward in removing the cache? > > Thanks for your help and wisdom! > > Cheers, > Kees > > P.S. The introduction text speaks of readproxy and not proxy. This > confusing since the instruction text speaks of proxy. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

CephFS Metadata loss, data present, inodes in pool, scan_links finishes instantly - Squid
by Enzo Van Thorre 05 Feb '26

05 Feb '26

Hello Ceph users and readers, I am facing a really unknown situation for me (being kinda new) with a CephFS cluster (Squid 19.2.2, Cephadm) where the data pool is intact (7.2 TB used, 4.9TB stored), inode objects exist in the recovered metadata pool, but when i try to mount the filesystem, its empty. What initially happened is that I had two groups of drives: nvme’s and spinning. I had separate pools for both, and I eventually deleted the nvme’s pool and then removed the nvmes from the machine. I never thought that they had included metadata for spinning rust’s pool, but here we are. Version: 19.2.2 Squid Metadata pool ID 1 (cephfs.cephfs.meta), Size=1 (No Replicas :(, yes I know I messed up). Data Pool ID 6 (ec_data), erasure coded with k=2 m=1. All OSDs are up. 256 PGs are active+clean. there’s then quite a bit that are inconsistent, degraded, undersized, remapped, and other stuff. Here’s what I’ve done since losing the metadata pool due to OSD purges. I tried to rebuild metadata from the data pool by doing: 1. cephfs-data-scan init --force-init 2. cephfs-data-scan scan_extents 3. cephfs-data-scan scan_inodes ceph df shows 7.2 TiB stored in pool ec_data (which is the spinning rust pool) Inodes do exist. running rados -p cephfs.cephfs.meta ls | grep "100." returns tons of objects (ex. 1000005f1f3.00000000). If i try to mount via ceph-fuse, it shows an empty root. there is no lost+found folder either. A big problem is that when I try to link the orphaned inodes to lost+found, the command finishes instantly with no output, as if it believes the cluster is clean. I run: cephfs-data-scan scan_links --filesystem cephfs # and it returns code 0, no output, and runs in <1 second So my final question is since scan_inodes has populated the metadata pool with objects (100.xxxx), but scan_links ignores them, how do I force a re-scan or just do SOMETHING so scan_links will actually link these files into lost+found? All The Best, Enzo

1 0

Re: How to disable an alertmanager alert
by Ankush Behl 05 Feb '26

05 Feb '26

Hi Chris, do you mean they show up in overview dashboard? Can you show via some screenshots where all they are coming up? On Thu, 5 Feb 2026 at 6:39 PM, Chris Palmer via ceph-users < ceph-users(a)ceph.io> wrote: > cephadm (squid & tentacle) deploys alertmanager with some default > alerts. One is CephNodeInconsistentMTU which checks if the MTU of each > interface with the same name is the same across the cluster. > > However if your cluster nodes are not identical, and have differing > interface names, we end up with lots of permanent alerts. Although I can > silence the alert (matching the alert name) for a long time, they still > show up in the dashboard. > > The alerts are different permutations of: > > Node ceph3 has a different MTU size (1500) than the median of > devices named enp4s0. > > Since that check is inappropriate in a heterogeneous cluster, what's the > recommended way of disabling it, which won't get overwritten with > redeployment of the monitoring stack? > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Disable cache tier
by Kees Meijs | Nefos 05 Feb '26

05 Feb '26

Hello list, After running cache tiers for years and years, we decided to turn off caching on a mostly dormant cluster. Given an installation using Pacific, we followed the instructions as dictated on: https://docs.ceph.com/en/pacific/rados/operations/cache-tiering/#removing-a… So, we set the "hot" tier to cache-mode proxy (as the documentation mentions) and simply waited for a bit. Since nothing really seemed to happen, we did a manual flush using cache-flush-evict-all (also as the documentation mentions). So far, so good. The end result is a cache tier with still quite some (>81k) objects within. Running cache-flush-evict-all again renders errors such as: failed to flush /rbd_data.d9d54166ef438d.0000000000020899: (2) No such file or directory rbd_data.d9d54166ef438d.000000000001a511 This is not what is expected. What are we looking at? And, how to move forward in removing the cache? Thanks for your help and wisdom! Cheers, Kees P.S. The introduction text speaks of readproxy and not proxy. This confusing since the instruction text speaks of proxy.

1 0

How to disable an alertmanager alert
by Chris Palmer 05 Feb '26

05 Feb '26

cephadm (squid & tentacle) deploys alertmanager with some default alerts. One is CephNodeInconsistentMTU which checks if the MTU of each interface with the same name is the same across the cluster. However if your cluster nodes are not identical, and have differing interface names, we end up with lots of permanent alerts. Although I can silence the alert (matching the alert name) for a long time, they still show up in the dashboard. The alerts are different permutations of: Node ceph3 has a different MTU size (1500) than the median of devices named enp4s0. Since that check is inappropriate in a heterogeneous cluster, what's the recommended way of disabling it, which won't get overwritten with redeployment of the monitoring stack? Thanks, Chris

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Kirby Haze 05 Feb '26

05 Feb '26

When reweight the drive down to 0, does the behavior change if you set it 'out' first? The only other special thing that happens with reweight 0 is that upmaps get removed for said OSD. If you think it is at the pool level you could run a simple rados bench to a pool while you reweight an OSD to see if your issue still holds. Also could be some issue with osd_op_queue being mclock if you have that, might be worth A/B testing with one OSD if you still see it. On Wed, Feb 4, 2026 at 12:31 PM Rok Jaklič via ceph-users < ceph-users(a)ceph.io> wrote: > If one HDD drive fails and smartctl shows errors and then I decide to drain > it with crush reweight 0, would Ceph try to copy/move data/pgs from data > failed disk anyway? > > Because we noticed on some non ceph clusters (raid setup, actually mail > servers), that one failed drive may hog up "app/OS" because "app" fails to > read/write to failed disk because "some queue" fills up (since app/OS is > unable for data to be read/written)? > > Could something similar happen in Ceph? > > Rok > > On Wed, Feb 4, 2026 at 9:52 AM Rok Jaklič <rjaklic(a)gmail.com> wrote: > > > > > On Wed, Feb 4, 2026 at 2:59 AM Anthony D'Atri <anthony.datri(a)gmail.com> > > wrote: > > > >> Are these rear bay drives, hence the limit of 2? Or > >> You might consider an M.2 AIC adapter card with bifurcation. M.2 > >> enterprise SSDs are sunsetting but for retrofits you should be able to > find > >> Micron 6450 units. > >> > >> What’s your workload like? > >> > > > > On average 10-50MB/s of write, with spikes up to a few hundred MB/s > during > > evening/night time; it went up to 1GB/s during tests without a problem. > All > > these are S3 workloads/tests. > > > > I would have to check that on site, RM does not show, however we are just > > about to migrate to new machines, which have 4 NVMe slots ... so I am > > really considering moving WAL/DB to NVMe, however I am still a little bit > > hesitant, since I am not really sure this will solve the problem of why > > radosgw/s3 stops after some time when setting crush reweight to 0 on one > > failed disk. We are doing the same thing on HPC where radosgw/s3 is not > > used and we are not experiencing this problem there. If we move WAL/DB to > > NVMe, and if one NVMe fails and we have to recover 10 OSDs for example, > it > > would take much longer than if just 1 OSD has to be recovered (while > users > > being unable to access s3). > > > > --- > > > > My suspicion is that when we set crush reweight of the failed disk to 0, > > all other affected disks from that pool disables some write (because of > > recovery) and some queue fills up which then stops/hangs radosgw... > > > > Rok > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

rpm update to tentacle
by Christopher Durham 04 Feb '26

04 Feb '26

I updated squid 19.2.2 to tentacle 2.2.0 using rpms on rocky9 linux. I see that the release was allegedly compiled with 'RelWithDebInfo' (some level of debug) as evidenced with: ceph versions "overall": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 37 } Is this intentional? I have never seen this before on a production release... -Chris

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Anthony D'Atri 04 Feb '26

04 Feb '26

>> . Not sure where you get your number from (10 DB devices >> max per NVMe), but the docs [0] state to not have more than 15 OSDs per NVMe: Huh, I thought it said 10. That was the rule of thumb 10 years ago when NVMe NAND SSDs were on the order of the P3600 and PCIe was what, gen 2 or 3? These days 15 probably is fine with an appropriate SSD model and PCIe gen 4+. These guidelines are there because we’re expected to provide some, but in the end there are variables including the workload. >> But you're correct about the SPOF, if one NVMe dies, all OSDs that have >> their DB/WAL on that NVMe die as well. In a sufficiently large cluster that isn’t a big deal. Even so, any cluster should be able to survive the loss of an entire host, so is this blast radius any scarier? Some admins choose to mirror offload SSDs because of this FUD, but consider: * Ceph is already handling distributed redundancy. Does it make sense to double the space amp for wall+ DB ? * If HDDs are used for the perception † that they cost less, consider that doubling them up costs more, narrowing the capex gap. My sense is that most admins elect to distribute OSDs across unmirrored offload SSDs. Costs less and burns less endurance. † With thoughtful sourcing, model selection appropriate to the workload, and TCO analysis you might be surprised. >> >> [0] >> >> https://docs.ceph.com/en/latest/start/hardware-recommendations/#minimum-har… >> >> Am Di., 3. Feb. 2026 um 15:00 Uhr schrieb Rok Jaklič via ceph-users < >> ceph-users(a)ceph.io>: >> >>> We have 28 OSDs per host and we can only have 2 NVMe per host (one being >>> used for OS) . Are these rear bay drives, hence the limit of 2? Or You might consider an M.2 AIC adapter card with bifurcation. M.2 enterprise SSDs are sunsetting but for retrofits you should be able to find Micron 6450 units. What’s your workload like? >>> .. and if I remember correctly there is max 10 OSDs/NVMe >>> recommended, that's why we decided to go just for HDD based clusters at >> the >>> beinging. >>> >>> We have 2 clusters this way, one being for HPC (no radosgw/s3) and other >>> for "users" (radosgw/s3), running over 4 years now ... works ok, >>> performance is ok, just we have this problem where we have to do gentle >>> reweight of a failed OSDs. With modern releases the upmap / balancer strategy is better, you might be using the name of the old script to describe it. >>> >>> Thanks for the info, we will consider NVMe ... although then there is >> SPOF >>> for those OSDs which have DB on NVMe? Ceph can handle it. Enterprise SSDs have lower AFR than spinners. Keep the firmware updated. >>> >>> Rok >>> >>> On Tue, Feb 3, 2026 at 2:35 PM Robert Sander via ceph-users < >>> ceph-users(a)ceph.io> wrote: >>> >>>> Am 03.02.26 um 2:31 PM schrieb Rok Jaklič: >>>>> On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users <ceph- >>>>> users(a)ceph.io <mailto:ceph-users@ceph.io>> wrote: >>>>> >>>>>> Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: >>>>>> >>>>>>> 2 OSD(s) experiencing slow operations in >> BlueStore >>>>>>> 2 OSD(s) experiencing stalled read in db device I see this when updating firmware live. The default lifetime of the warning is 24 hours, so usually this is far more ephemeral than one might think and less of a big deal. >> of >>>>>> BlueFS >>>>>> >>>>>> Are your OSDs HDD only? >>>>> >>>> >>>>> Yes. >>>>> >>>>> Does not affect users much. Usually those messages appear when we are >>>>> reweighting and changing failed disks. >>>> >>>> These HDDs will be maxed out with the recovery work and cannot serve >>>> anything else any more. >>>> >>>> I have seen HDD only clusters going into the "spiral of death" because >>>> the HDDs cannot answer fast enough. OSDs randomly dropping out making >>>> the whole system unstable. >>>> >>>> The RocksDB is such a random IO application that it is not suitable for >>>> HDDs. It should always be put on flash storage (SSD/NVMe). >>>> >>>> Regards >>>> -- >>>> Robert Sander >>>> Linux Consultant >>>> >>>> Heinlein Consulting GmbH >>>> Schwedter Str. 8/9b, 10119 Berlin >>>> >>>> https://www.heinlein-support.de >>>> >>>> Tel: +49 30 405051 - 0 >>>> Fax: +49 30 405051 - 19 >>>> >>>> Amtsgericht Berlin-Charlottenburg - HRB 220009 B >>>> Geschäftsführer: Peer Heinlein - Sitz: Berlin >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: Limiting number of object versions per key in RGW (hotspot / bucket index shard growth)
by Frédéric Nass 04 Feb '26

04 Feb '26

Hi Ansgar, Since Squid (v19) and PR #54152 [1], the RGW Lifecycle supports the 'NoncurrentDays' and 'NewerNoncurrentVersions' filters. Would that work for your use case? You can find examples of how to use these filters here [2]. Note that PR also added support for ObjectSizeGreater(Less)Than filers to transition objects based on their size. Best regards, Frédéric Nass [1] https://github.com/ceph/ceph/pull/54152 [2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configurati… -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Squishing Squids - A Ceph Compression Guide <https://www.eventbrite.com/e/squishing-squids-a-ceph-compression-guide-tick…>, February 25th, 9am PST. https://clyso.com | frederic.nass(a)clyso.com Le mer. 4 févr. 2026 à 13:23, Ansgar Jazdzewski via ceph-users < ceph-users(a)ceph.io> a écrit : > Hi Cephers, > > we are running Ceph RGW with bucket versioning enabled (currently on > Reef, but we have also observed the same behavior on other releases). > > We are seeing an operational issue where a single object key can > accumulate extremely large numbers of versions (hundreds of thousands > up to millions). Because all versions of the same key are stored on > the same bucket index shard, this results in very large bucket-index > entries / omap growth and shard hotspots. > > In our case, one object with millions of versions caused significant > bucket index pressure and related performance/maintenance concerns. > > As far as we can tell, RGW currently does not provide a way to enforce > a limit such as “keep only the last N versions per object key,” and > existing quotas only apply at bucket/user level, not per object key. > > We wanted to ask: > * Have others seen similar cases with very high version counts for > individual keys? > * How do you handle or mitigate this operationally today > > Any advice or shared experience would be very appreciated. > > Thank you, > Ansgar Jazdzewski > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Limiting number of object versions per key in RGW (hotspot / bucket index shard growth)
by Ansgar Jazdzewski 04 Feb '26

04 Feb '26

Hi Cephers, we are running Ceph RGW with bucket versioning enabled (currently on Reef, but we have also observed the same behavior on other releases). We are seeing an operational issue where a single object key can accumulate extremely large numbers of versions (hundreds of thousands up to millions). Because all versions of the same key are stored on the same bucket index shard, this results in very large bucket-index entries / omap growth and shard hotspots. In our case, one object with millions of versions caused significant bucket index pressure and related performance/maintenance concerns. As far as we can tell, RGW currently does not provide a way to enforce a limit such as “keep only the last N versions per object key,” and existing quotas only apply at bucket/user level, not per object key. We wanted to ask: * Have others seen similar cases with very high version counts for individual keys? * How do you handle or mitigate this operationally today Any advice or shared experience would be very appreciated. Thank you, Ansgar Jazdzewski

1 0

RadosGW delete operations seem to hang
by Janek Bevendorff 04 Feb '26

04 Feb '26

Hi, I deleted (or tried to delete) a few hundred TB of data in our S3 storage the last couple of days. Unfortunately, all delete operations seem to hang somewhat. I tried deleting the buckets with radosgw-admin bucket rm --bucket=bucketname --purge-objects --bypass-gc, but it seems to hang forever without any visible progress. Even when I remove --bypass-gc, it just hangs. I've since managed to clear my buckets by doing an s3 sync with an empty directory, but that of course didn't actually free any space yet. So, I started radosgw-admin gc process --include-all yesterday. This process is still running without much visible progress (we are about 10M objects down, nothing significant). The processes are not completely dead. They hang in futex calls with 0% CPU utilization. Every now and then, there's a small spike of ca. 0.6% CPU with a bunch of other calls in strace, but most of the time seems to be spent in interruptible sleep. Is there any way I can see what's actually happening or why it is taking so long? Could this be related to our aging hard disks? Deleting old snapshots in CephFS starts freeing storage almost instantly and the 10M objects are deleted in about half an hour, so that seems like an unlikely cause? We urgently need to free some space to better balance out the data. I found this ML thread, but there's no solution in it. https://www.mail-archive.com/ceph-users@ceph.io/msg11571.html Upping --max-concurrent-ios didn't help. Janek

1 0

Ceph Developer Meetup - Feb 4th at 16:00 UTC
by Anthony Middleton 03 Feb '26

03 Feb '26

Hey Ceph Friends, Join us tomorrow, *Feb 4th at 16:00 UTC* for this month's *Ceph Developer Meetup*, a long-running staple of the Ceph community where developers discuss ideas and seek feedback. This month's topics include: - Connecting the central config store with revision control - Discussing high-level implementation of small object packing - and more! Whether you are looking for advice on a pull request, have a development idea to pitch to a broader audience, or are simply interested in hearing what other developers are working on, you are welcome to join! RSVP here: https://t.ly/DevUserMeetupJan Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: [squid] buckets not in bucket stats in multisite setup when create in a non master zonegroup
by Casey Bodley 03 Feb '26

03 Feb '26

thanks Boris, there was some previous discussion in related issue https://tracker.ceph.com/issues/74231, and i've proposed a fix for that On Tue, Feb 3, 2026 at 6:07 AM Boris via ceph-users <ceph-users(a)ceph.io> wrote: > > created a tracker with additional information: > https://tracker.ceph.com/issues/74729 > > Am Di., 3. Feb. 2026 um 01:51 Uhr schrieb Boris <bb(a)kervyn.de>: > > > Hi, > > > > we just encountered a strange problem: > > > > When we run radosgw-admin bucket stats on the master zonegroup we get some > > of these errors > > error getting bucket stats bucket=rregergf ret=-2 > > > > This is the debug_rgw=20 output: > > error getting bucket stats bucket=rregergf > > ret=2026-02-03T00:50:32.347+0000 722007981a80 20 cls_bucket_head: > > CLSRGWIssueGetDirHeader() returned -2 > > -2 > > failure: (2) No such file or directory: > > > > But the bucketname is listed in a bucket list command. > > > > After some digging we found that the bucket was created after the upgrade > > to squid in a non master zonegroup. > > > > Doing the radosgw-admin bucket stats command in any zonegroup, except the > > master, yields results. > > > > We checked, and it is every bucket created after the upgrade, that was not > > created in the master zonegroup. > > > > We only sync metadata between the zonegrouops, and I just ran a fresh > > metadata sync on all zonegroups. > > > > Any ideas what this could be? > > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

ACT NOW! Ceph Days Raleigh CFP Closes Feb. 16th
by Anthony Middleton 03 Feb '26

03 Feb '26

Hey Ceph Friends, Just a quick reminder that the CFP for Ceph Days Raleigh is open! Please submit your proposal by February 16. You can do it here: https://docs.google.com/forms/d/e/1FAIpQLSfHUyLahd9EFAIJCF0niveMhszt_F5lgTU… If you have any questions, feel free to reach out to Laura Flores <lflores(a)ibm.com> or me <anthony(a)amicmid.com>. Thanks! -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

Question is then why this is happening only on s3 cluster and not also on NO s3 cluster while write io during recovery/backfilling is actually much higher than on s3? On Tue, Feb 3, 2026 at 3:27 PM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > I concur with Robert's statement, having HDDs only could explain what > you're describing. Not sure where you get your number from (10 DB devices > max per NVMe), but the docs [0] state to not have more than 15 OSDs per > NVMe: > > > DB/WAL offload (optional) > > 1x SSD partition per HDD OSD 4-5x HDD OSDs per DB/WAL SATA SSD <= 15 HDD > OSDs per DB/WAL NVMe SSD > But you're correct about the SPOF, if one NVMe dies, all OSDs that have > their DB/WAL on that NVMe die as well. > > [0] > > https://docs.ceph.com/en/latest/start/hardware-recommendations/#minimum-har… > > Am Di., 3. Feb. 2026 um 15:00 Uhr schrieb Rok Jaklič via ceph-users < > ceph-users(a)ceph.io>: > > > We have 28 OSDs per host and we can only have 2 NVMe per host (one being > > used for OS) ... and if I remember correctly there is max 10 OSDs/NVMe > > recommended, that's why we decided to go just for HDD based clusters at > the > > beinging. > > > > We have 2 clusters this way, one being for HPC (no radosgw/s3) and other > > for "users" (radosgw/s3), running over 4 years now ... works ok, > > performance is ok, just we have this problem where we have to do gentle > > reweight of a failed OSDs. > > > > Thanks for the info, we will consider NVMe ... although then there is > SPOF > > for those OSDs which have DB on NVMe? > > > > Rok > > > > On Tue, Feb 3, 2026 at 2:35 PM Robert Sander via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > > > Am 03.02.26 um 2:31 PM schrieb Rok Jaklič: > > > > On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users <ceph- > > > > users(a)ceph.io <mailto:ceph-users@ceph.io>> wrote: > > > > > > > >> Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > > > >> > > > >> > 2 OSD(s) experiencing slow operations in > BlueStore > > > >> > 2 OSD(s) experiencing stalled read in db device > of > > > >> BlueFS > > > >> > > > >> Are your OSDs HDD only? > > > > > > > > > > > Yes. > > > > > > > > Does not affect users much. Usually those messages appear when we are > > > > reweighting and changing failed disks. > > > > > > These HDDs will be maxed out with the recovery work and cannot serve > > > anything else any more. > > > > > > I have seen HDD only clusters going into the "spiral of death" because > > > the HDDs cannot answer fast enough. OSDs randomly dropping out making > > > the whole system unstable. > > > > > > The RocksDB is such a random IO application that it is not suitable for > > > HDDs. It should always be put on flash storage (SSD/NVMe). > > > > > > Regards > > > -- > > > Robert Sander > > > Linux Consultant > > > > > > Heinlein Consulting GmbH > > > Schwedter Str. 8/9b, 10119 Berlin > > > > > > https://www.heinlein-support.de > > > > > > Tel: +49 30 405051 - 0 > > > Fax: +49 30 405051 - 19 > > > > > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Eugen Block 03 Feb '26

03 Feb '26

I concur with Robert's statement, having HDDs only could explain what you're describing. Not sure where you get your number from (10 DB devices max per NVMe), but the docs [0] state to not have more than 15 OSDs per NVMe: > DB/WAL offload (optional) 1x SSD partition per HDD OSD 4-5x HDD OSDs per DB/WAL SATA SSD <= 15 HDD OSDs per DB/WAL NVMe SSD But you're correct about the SPOF, if one NVMe dies, all OSDs that have their DB/WAL on that NVMe die as well. [0] https://docs.ceph.com/en/latest/start/hardware-recommendations/#minimum-har… Am Di., 3. Feb. 2026 um 15:00 Uhr schrieb Rok Jaklič via ceph-users < ceph-users(a)ceph.io>: > We have 28 OSDs per host and we can only have 2 NVMe per host (one being > used for OS) ... and if I remember correctly there is max 10 OSDs/NVMe > recommended, that's why we decided to go just for HDD based clusters at the > beinging. > > We have 2 clusters this way, one being for HPC (no radosgw/s3) and other > for "users" (radosgw/s3), running over 4 years now ... works ok, > performance is ok, just we have this problem where we have to do gentle > reweight of a failed OSDs. > > Thanks for the info, we will consider NVMe ... although then there is SPOF > for those OSDs which have DB on NVMe? > > Rok > > On Tue, Feb 3, 2026 at 2:35 PM Robert Sander via ceph-users < > ceph-users(a)ceph.io> wrote: > > > Am 03.02.26 um 2:31 PM schrieb Rok Jaklič: > > > On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users <ceph- > > > users(a)ceph.io <mailto:ceph-users@ceph.io>> wrote: > > > > > >> Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > > >> > > >> > 2 OSD(s) experiencing slow operations in BlueStore > > >> > 2 OSD(s) experiencing stalled read in db device of > > >> BlueFS > > >> > > >> Are your OSDs HDD only? > > > > > > > > Yes. > > > > > > Does not affect users much. Usually those messages appear when we are > > > reweighting and changing failed disks. > > > > These HDDs will be maxed out with the recovery work and cannot serve > > anything else any more. > > > > I have seen HDD only clusters going into the "spiral of death" because > > the HDDs cannot answer fast enough. OSDs randomly dropping out making > > the whole system unstable. > > > > The RocksDB is such a random IO application that it is not suitable for > > HDDs. It should always be put on flash storage (SSD/NVMe). > > > > Regards > > -- > > Robert Sander > > Linux Consultant > > > > Heinlein Consulting GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > https://www.heinlein-support.de > > > > Tel: +49 30 405051 - 0 > > Fax: +49 30 405051 - 19 > > > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

We have 28 OSDs per host and we can only have 2 NVMe per host (one being used for OS) ... and if I remember correctly there is max 10 OSDs/NVMe recommended, that's why we decided to go just for HDD based clusters at the beinging. We have 2 clusters this way, one being for HPC (no radosgw/s3) and other for "users" (radosgw/s3), running over 4 years now ... works ok, performance is ok, just we have this problem where we have to do gentle reweight of a failed OSDs. Thanks for the info, we will consider NVMe ... although then there is SPOF for those OSDs which have DB on NVMe? Rok On Tue, Feb 3, 2026 at 2:35 PM Robert Sander via ceph-users < ceph-users(a)ceph.io> wrote: > Am 03.02.26 um 2:31 PM schrieb Rok Jaklič: > > On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users <ceph- > > users(a)ceph.io <mailto:ceph-users@ceph.io>> wrote: > > > >> Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > >> > >> > 2 OSD(s) experiencing slow operations in BlueStore > >> > 2 OSD(s) experiencing stalled read in db device of > >> BlueFS > >> > >> Are your OSDs HDD only? > > > > > Yes. > > > > Does not affect users much. Usually those messages appear when we are > > reweighting and changing failed disks. > > These HDDs will be maxed out with the recovery work and cannot serve > anything else any more. > > I have seen HDD only clusters going into the "spiral of death" because > the HDDs cannot answer fast enough. OSDs randomly dropping out making > the whole system unstable. > > The RocksDB is such a random IO application that it is not suitable for > HDDs. It should always be put on flash storage (SSD/NVMe). > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

Yes. Does not affect users much. Usually those messages appear when we are reweighting and changing failed disks. Rok On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users < ceph-users(a)ceph.io> wrote: > Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > > > 2 OSD(s) experiencing slow operations in BlueStore > > 2 OSD(s) experiencing stalled read in db device of BlueFS > > Are your OSDs HDD only? > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Robert Sander 03 Feb '26

03 Feb '26

Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > 2 OSD(s) experiencing slow operations in BlueStore > 2 OSD(s) experiencing stalled read in db device of BlueFS Are your OSDs HDD only? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

Probably I did not use the right words to explain current situtaion... --- Cluster starts to recover/backfilling once I set crush reweight of a failed osd to 0 (here s3 stops responding after some time). If I set it to a little bit lower than the previous value, then s3 does not stop working after some time. For example, right now its io says it is in a recovery state? (that is why I said it is in recovering state): Every 2.0s: ceph -s ctplmon1.arnes.si: Tue Feb 3 13:55:19 2026 cluster: id: 0a6e5422-ac75-4093-af20-528ee00cc847 health: HEALTH_ERR 2 OSD(s) experiencing slow operations in BlueStore 2 OSD(s) experiencing stalled read in db device of BlueFS 5 scrub errors Possible data damage: 3 pgs inconsistent services: mon: 3 daemons, quorum ctplmon1,ctplmon3,ctplmon2 (age 10d) mgr: ctplmon1(active, since 10d), standbys: ctplmon3 mds: 1/1 daemons up osd: 240 osds: 240 up (since 42h), 240 in (since 42h); 13 remapped pgs rgw: 6 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 6353 pgs objects: 360.61M objects, 421 TiB usage: 708 TiB used, 623 TiB / 1.3 PiB avail pgs: 1544002/1802985000 objects misplaced (0.086%) 6328 active+clean 13 active+remapped+backfilling 9 active+clean+scrubbing+deep 3 active+clean+inconsistent io: client: 97 KiB/s rd, 1.2 KiB/s wr, 18 op/s rd, 3 op/s wr recovery: 110 MiB/s, 90 objects/s --- I do not use orchestrator. Crush failure domain is set to host and EC 3+2 is used (historical reasons), even though we would like to change this to 8+3 in the near future. For every new failed disk different PGs are probably used, not the same? Rok On Tue, Feb 3, 2026 at 1:35 PM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > So the cluster is currently recovering and then you wanted to remove > another OSD? I'm just trying to get the full picture here. Let's assume the > cluster is backfilling because you removed one OSD (and it's not done yet), > then you decide to remove another one. If we assume a replicated pool (size > 3, min_size 2), this could cause inactive PGs until backfill has finished. > Depending on the failure domain (host?) you need to be careful when > removing OSDs from multiple failure domains at the same time. That's why > some background here would help to better understand what really happens. > And does it happen every time you replace a disk? Does it always impact the > same PGs? Does the cluster report inactive PGs for more than a few seconds? > > I was asking for RGW logs during that outage, not the "normal" behaviour > (starting new request), so some error logs. The reshardlogs messages are > also nothing unusual. > > Regarding the replacement procedure, it looks like you don't use the > orchestrator, right? > > > Am Di., 3. Feb. 2026 um 10:30 Uhr schrieb Rok Jaklič via ceph-users < > ceph-users(a)ceph.io>: > > > ceph -s is in recovery state, moving around 200-300MB/s of data. > > > > [root@ctplmon1 ~]# ceph osd ok-to-stop 239 > > > > > {"ok_to_stop":true,"osds":[239],"num_ok_pgs":94,"num_not_ok_pgs":0,"ok_become_degraded":["9.e","9.40","9.4a","9.4e","9.69","9.7a","9.87","9.bc","9.c5"," > > 9.cf > > > > > ","9.dd","9.128","9.172","9.1b6","9.1c7","9.227","9.244","9.246","9.24e","9.2b3","9.2d3","9.2ee","9.2f1","9.31c","9.335","9.361","9.364","9.38b","9.3b0","9.3d9","9.408","9.41b","9.483","9.49a","9.4c1","9.4dc","9.4ea","9.4fd","9.553","9.554","9.55b","9.57a","9.586","9.5c7","9.61a","9.627","9.68b","9.69d","9.711","9.71d","9.737","9.784","10.12","10.82","10.18c","10.1c7","10.290","10.310","10.390","10.434","10.4aa","10.4e1","10.525","10.558","10.57f","10.5e2","10.5e7","10.62f","10.736","10.842","10.8d9","10.904","10.964","10.977","10.9eb","10.a1a","10.a40","10.b80","10.bbf","10.be2","10.c7c","10.c9b","10.cb0","10.cb3","10.d7a","10.dc8","10.dcf","10.dde","10.e5b","10.e6d","10.e78","10.ece","10.ef1","10.f60"]} > > > > rgw logs is usually something like: > > 2026-02-03T09:48:32.050+0100 7f34ff13a640 1 ====== starting new request > > req=0x7f34b8f0c640 ===== > > 2026-02-03T09:48:35.901+0100 7f34cb0d2640 1 ====== starting new request > > req=0x7f34b8e8b640 ===== > > 2026-02-03T09:49:41.408+0100 7f34e00fc640 1 ====== starting new request > > req=0x7f34b8d08640 ===== > > 2026-02-03T09:49:46.267+0100 7f35511de640 1 ====== starting new request > > req=0x7f34b8c87640 ===== > > > > while I can see also: > > 2026-02-03T09:49:50.912+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000000 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.919+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000002 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.926+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000004 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.934+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000006 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.953+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000008 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.960+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000010 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.968+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000012 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.975+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000014 to be held by another RGW process; > > skipping for now > > > > but do not know if it is related. > > > > ... > > > > Should I use some different strategy to "replace" failed disks? > > > > Right now I am doing something like: > > ceph osd crush reweight osd.{osdid} 4 3 2 1 ... in steps > > > > while ! ceph osd safe-to-destroy osd.{osdid} ; do sleep 10 ; done > > > > systemctl stop ceph-osd@{id} > > ceph osd down osd.{id} > > ceph osd destroy {id} --yes-i-really-mean-it > > > > ceph-volume lvm prepare --osd-id {id} --data /dev/sdX > > ceph-volume lvm list > > ceph-volume lvm activate {id} {osd fsid} > > > > > > > > On Tue, Feb 3, 2026 at 10:21 AM Eugen Block <eblock82(a)gmail.com> wrote: > > > > > Hi, > > > > > > what does 'ceph osd ok-to-stop <OSD_ID>' show? What do the rgw logs > > > reveal? What's the 'ceph status' during that time? > > > > > > Am Di., 3. Feb. 2026 um 10:18 Uhr schrieb Rok Jaklič via ceph-users < > > > ceph-users(a)ceph.io>: > > > > > >> Hi, > > >> > > >> we currently have 240 OSDs in the cluster, but when I set the reweight > > of > > >> just one failed OSD to 0 (from 5.45798), radosgw stops working for > > >> some reason. > > >> > > >> If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), > > it > > >> works ok. > > >> > > >> Any ideas why? > > >> > > >> Running on reef ceph version 18.2.7. > > >> > > >> Kind regards, > > >> Rok > > >> _______________________________________________ > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Eugen Block 03 Feb '26

03 Feb '26

So the cluster is currently recovering and then you wanted to remove another OSD? I'm just trying to get the full picture here. Let's assume the cluster is backfilling because you removed one OSD (and it's not done yet), then you decide to remove another one. If we assume a replicated pool (size 3, min_size 2), this could cause inactive PGs until backfill has finished. Depending on the failure domain (host?) you need to be careful when removing OSDs from multiple failure domains at the same time. That's why some background here would help to better understand what really happens. And does it happen every time you replace a disk? Does it always impact the same PGs? Does the cluster report inactive PGs for more than a few seconds? I was asking for RGW logs during that outage, not the "normal" behaviour (starting new request), so some error logs. The reshardlogs messages are also nothing unusual. Regarding the replacement procedure, it looks like you don't use the orchestrator, right? Am Di., 3. Feb. 2026 um 10:30 Uhr schrieb Rok Jaklič via ceph-users < ceph-users(a)ceph.io>: > ceph -s is in recovery state, moving around 200-300MB/s of data. > > [root@ctplmon1 ~]# ceph osd ok-to-stop 239 > > {"ok_to_stop":true,"osds":[239],"num_ok_pgs":94,"num_not_ok_pgs":0,"ok_become_degraded":["9.e","9.40","9.4a","9.4e","9.69","9.7a","9.87","9.bc","9.c5"," > 9.cf > > ","9.dd","9.128","9.172","9.1b6","9.1c7","9.227","9.244","9.246","9.24e","9.2b3","9.2d3","9.2ee","9.2f1","9.31c","9.335","9.361","9.364","9.38b","9.3b0","9.3d9","9.408","9.41b","9.483","9.49a","9.4c1","9.4dc","9.4ea","9.4fd","9.553","9.554","9.55b","9.57a","9.586","9.5c7","9.61a","9.627","9.68b","9.69d","9.711","9.71d","9.737","9.784","10.12","10.82","10.18c","10.1c7","10.290","10.310","10.390","10.434","10.4aa","10.4e1","10.525","10.558","10.57f","10.5e2","10.5e7","10.62f","10.736","10.842","10.8d9","10.904","10.964","10.977","10.9eb","10.a1a","10.a40","10.b80","10.bbf","10.be2","10.c7c","10.c9b","10.cb0","10.cb3","10.d7a","10.dc8","10.dcf","10.dde","10.e5b","10.e6d","10.e78","10.ece","10.ef1","10.f60"]} > > rgw logs is usually something like: > 2026-02-03T09:48:32.050+0100 7f34ff13a640 1 ====== starting new request > req=0x7f34b8f0c640 ===== > 2026-02-03T09:48:35.901+0100 7f34cb0d2640 1 ====== starting new request > req=0x7f34b8e8b640 ===== > 2026-02-03T09:49:41.408+0100 7f34e00fc640 1 ====== starting new request > req=0x7f34b8d08640 ===== > 2026-02-03T09:49:46.267+0100 7f35511de640 1 ====== starting new request > req=0x7f34b8c87640 ===== > > while I can see also: > 2026-02-03T09:49:50.912+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000000 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.919+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000002 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.926+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000004 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.934+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000006 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.953+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000008 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.960+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000010 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.968+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000012 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.975+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000014 to be held by another RGW process; > skipping for now > > but do not know if it is related. > > ... > > Should I use some different strategy to "replace" failed disks? > > Right now I am doing something like: > ceph osd crush reweight osd.{osdid} 4 3 2 1 ... in steps > > while ! ceph osd safe-to-destroy osd.{osdid} ; do sleep 10 ; done > > systemctl stop ceph-osd@{id} > ceph osd down osd.{id} > ceph osd destroy {id} --yes-i-really-mean-it > > ceph-volume lvm prepare --osd-id {id} --data /dev/sdX > ceph-volume lvm list > ceph-volume lvm activate {id} {osd fsid} > > > > On Tue, Feb 3, 2026 at 10:21 AM Eugen Block <eblock82(a)gmail.com> wrote: > > > Hi, > > > > what does 'ceph osd ok-to-stop <OSD_ID>' show? What do the rgw logs > > reveal? What's the 'ceph status' during that time? > > > > Am Di., 3. Feb. 2026 um 10:18 Uhr schrieb Rok Jaklič via ceph-users < > > ceph-users(a)ceph.io>: > > > >> Hi, > >> > >> we currently have 240 OSDs in the cluster, but when I set the reweight > of > >> just one failed OSD to 0 (from 5.45798), radosgw stops working for > >> some reason. > >> > >> If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), > it > >> works ok. > >> > >> Any ideas why? > >> > >> Running on reef ceph version 18.2.7. > >> > >> Kind regards, > >> Rok > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

[squid] buckets not in bucket stats in multisite setup when create in a non master zonegroup
by Boris 03 Feb '26

03 Feb '26

Hi, we just encountered a strange problem: When we run radosgw-admin bucket stats on the master zonegroup we get some of these errors error getting bucket stats bucket=rregergf ret=-2 This is the debug_rgw=20 output: error getting bucket stats bucket=rregergf ret=2026-02-03T00:50:32.347+0000 722007981a80 20 cls_bucket_head: CLSRGWIssueGetDirHeader() returned -2 -2 failure: (2) No such file or directory: But the bucketname is listed in a bucket list command. After some digging we found that the bucket was created after the upgrade to squid in a non master zonegroup. Doing the radosgw-admin bucket stats command in any zonegroup, except the master, yields results. We checked, and it is every bucket created after the upgrade, that was not created in the master zonegroup. We only sync metadata between the zonegrouops, and I just ran a fresh metadata sync on all zonegroups. Any ideas what this could be?

1 1

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Eugen Block 03 Feb '26

03 Feb '26

Hi, what does 'ceph osd ok-to-stop <OSD_ID>' show? What do the rgw logs reveal? What's the 'ceph status' during that time? Am Di., 3. Feb. 2026 um 10:18 Uhr schrieb Rok Jaklič via ceph-users < ceph-users(a)ceph.io>: > Hi, > > we currently have 240 OSDs in the cluster, but when I set the reweight of > just one failed OSD to 0 (from 5.45798), radosgw stops working for > some reason. > > If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), it > works ok. > > Any ideas why? > > Running on reef ceph version 18.2.7. > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 2

radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

Hi, we currently have 240 OSDs in the cluster, but when I set the reweight of just one failed OSD to 0 (from 5.45798), radosgw stops working for some reason. If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), it works ok. Any ideas why? Running on reef ceph version 18.2.7. Kind regards, Rok

1 0

Re: upmap-read balancer with CephFS kernel clients
by Eugen Block 03 Feb '26

03 Feb '26

Ah, forgot to mention that our compute and control nodes (OpenStack) also mount the CephFS for live migration purposes (Nova) and image to volume conversion (Cinder), and they apparently also report their feature set as "luminous". Am Di., 3. Feb. 2026 um 08:15 Uhr schrieb Eugen Block via ceph-users < ceph-users(a)ceph.io>: > Hi Janek, > > this 'ceph features' output that you're probably referring to (or maybe > 'ceph tell mon.X sessions') comes up every now and then, for example [0], > [1], [2]. I also still don't have a definitive answer, unfortunately. But > from my understanding those are compatibility features that don't > necessarily reflect the actual ceph version. For example, I have a test > cluster with tentacle: > > tentacle:~ # ceph versions > { > "mon": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 1 > }, > "mgr": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 2 > }, > "osd": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 3 > }, > "mds": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 1 > }, > "overall": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 7 > } > } > > but 'ceph features' shows squid: > > tentacle:~ # ceph features > { > "mon": [ > { > "features": "0x3f07fffffffdffff", > "release": "squid", > "num": 1 > } > ], > "mds": [ > { > "features": "0x3f07fffffffdffff", > "release": "squid", > "num": 1 > } > ], > "osd": [ > { > "features": "0x3f07fffffffdffff", > "release": "squid", > "num": 3 > } > ], > ... > > And this cluster also has "luminous" clients connected although I enforced > "squid" with 'ceph osd set-require-min-compat-client squid'. In our > production cluster I haven't enforced that, it is still running with > luminous as min-compat-client because we actually do have a couple of older > clients and don't want to risk anything, but we're going to have to deal > with them at some point. > > Anyway, I'm not recommending to enforce the require-min-compat-client > update, I'd rather suggest to set up a test cluster with a couple of your > typical clients maybe, and then see if they successfully connect. I did the > same in my lab virtual environment that resembles our prod. cluster > (OpenStack with Ceph), there I enforced "reef" as min-compat-client and new > mounts from Ubuntu machines (24.04, kernel 6.8.0) with Ceph 19.2.1 packages > still report luminous on a CephFS mount. But the mount works, so I assume > that everything is okay in the lab. But I'm still not feeling confident > enough to recommend you to update it. ;-) > Hopefully someone else chimes in and contradicts or confirms my > statement(s). > > Regards, > Eugen > > [0] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/A4IWXINDBM6… > [1] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RVND3Y3Z7FD… > [2] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WSADGYBR7PL… > > Am Mi., 28. Jan. 2026 um 09:16 Uhr schrieb Janek Bevendorff via ceph-users > < > ceph-users(a)ceph.io>: > > > I just noticed that this applies to RBD clients as well. How is this > > supposed to work? Can you use only radosgw and ceph-fuse if you want to > > use upmap-read? It seems strange to me that the most central Ceph > > systems don't work with these new features. > > > > > > On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: > > > Hi, > > > > > > Is there a way to use the new upmap-read balancer mode when I have > > > CephFS kernel clients connected? Our cluster runs Squid and all > > > connected clients are on kernel 6.11-6.14, so they should be new > > > enough. However, they identify themselves as luminous (I could not > > > find clear documentation about this, but this seems to be hard-coded > > > and by design), so I cannot run ceph osd set-require-min-compat-client > > > reef, which is required for the balancer module to work. > > > > > > I have 500 clients connected and migrating them all to FUSE would be a > > > headache. Is there a way around that? Some of my pools have extremely > > > high read balance scores and we're also struggling with performance > > > issues as a result. > > > > > > Janek > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > Bauhaus-Universität Weimar > > Bauhausstr. 9a, R308 > > 99423 Weimar, Germany > > > > Phone: +49 3643 58 3577 > > www.webis.de > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: upmap-read balancer with CephFS kernel clients
by Eugen Block 03 Feb '26

03 Feb '26

Hi Janek, this 'ceph features' output that you're probably referring to (or maybe 'ceph tell mon.X sessions') comes up every now and then, for example [0], [1], [2]. I also still don't have a definitive answer, unfortunately. But from my understanding those are compatibility features that don't necessarily reflect the actual ceph version. For example, I have a test cluster with tentacle: tentacle:~ # ceph versions { "mon": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 1 }, "mgr": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 2 }, "osd": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 3 }, "mds": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 1 }, "overall": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 7 } } but 'ceph features' shows squid: tentacle:~ # ceph features { "mon": [ { "features": "0x3f07fffffffdffff", "release": "squid", "num": 1 } ], "mds": [ { "features": "0x3f07fffffffdffff", "release": "squid", "num": 1 } ], "osd": [ { "features": "0x3f07fffffffdffff", "release": "squid", "num": 3 } ], ... And this cluster also has "luminous" clients connected although I enforced "squid" with 'ceph osd set-require-min-compat-client squid'. In our production cluster I haven't enforced that, it is still running with luminous as min-compat-client because we actually do have a couple of older clients and don't want to risk anything, but we're going to have to deal with them at some point. Anyway, I'm not recommending to enforce the require-min-compat-client update, I'd rather suggest to set up a test cluster with a couple of your typical clients maybe, and then see if they successfully connect. I did the same in my lab virtual environment that resembles our prod. cluster (OpenStack with Ceph), there I enforced "reef" as min-compat-client and new mounts from Ubuntu machines (24.04, kernel 6.8.0) with Ceph 19.2.1 packages still report luminous on a CephFS mount. But the mount works, so I assume that everything is okay in the lab. But I'm still not feeling confident enough to recommend you to update it. ;-) Hopefully someone else chimes in and contradicts or confirms my statement(s). Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/A4IWXINDBM6… [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RVND3Y3Z7FD… [2] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WSADGYBR7PL… Am Mi., 28. Jan. 2026 um 09:16 Uhr schrieb Janek Bevendorff via ceph-users < ceph-users(a)ceph.io>: > I just noticed that this applies to RBD clients as well. How is this > supposed to work? Can you use only radosgw and ceph-fuse if you want to > use upmap-read? It seems strange to me that the most central Ceph > systems don't work with these new features. > > > On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: > > Hi, > > > > Is there a way to use the new upmap-read balancer mode when I have > > CephFS kernel clients connected? Our cluster runs Squid and all > > connected clients are on kernel 6.11-6.14, so they should be new > > enough. However, they identify themselves as luminous (I could not > > find clear documentation about this, but this seems to be hard-coded > > and by design), so I cannot run ceph osd set-require-min-compat-client > > reef, which is required for the balancer module to work. > > > > I have 500 clients connected and migrating them all to FUSE would be a > > headache. Is there a way around that? Some of my pools have extremely > > high read balance scores and we're also struggling with performance > > issues as a result. > > > > Janek > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Bauhaus-Universität Weimar > Bauhausstr. 9a, R308 > 99423 Weimar, Germany > > Phone: +49 3643 58 3577 > www.webis.de > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

CSC minutes 2026-02-02
by Yaarit Hatuka 03 Feb '26

03 Feb '26

- [Dan] Ganesha caretaker: what is involved, JoshH (Rubin) may be able to help. - https://github.com/ceph/ceph/pull/66098 <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ceph_ceph_p…> - David G - happy to help with CI/github actions - we need mapping of ceph versions to ganesha versions - current test coverage: qa/suites/fs/nfs/ - [jmundack] - Reminder - CSC Charter Vote!!. - Link to charter - https://docs.google.com/document/d/1Jekr58LYSppznIZOZ3c136WhRYCISfSA9sH_ySJ… <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_docume…> - Check if your email is accurate/up to date at https://docs.google.com/spreadsheets/d/1eQKJnCYUTgfqOlG7x695sWDNw6t422JdyPW… <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_spread…> - Please look for an e-mail with subject "Vote: Approve Ceph Technical Charter" (the link is specific to a voter) - Current status: - total members - 37 - Request to be removed - 1 - emails bounced - 2 - voted - 25! - to vote - 9 more! - Election extended to 2/11 - Q from Greg - Did our existing code of conduct get approved along the way for this process, or are we going to need to deal with that? Section 4(b) - rocky 10 work in progress - most components are testing the new changes - next we need to focus on ubuntu to allow backport to tentacle and squid (to allow upgrade path) - tentacle 20.2.1 - started QA validation - reef's next point release to be reviewed and released this week

1 0

Help recovering OSD: rocksdb: submit_common error: Corruption: block checksum mismatch
by Theo Cabrerizo Diem 02 Feb '26

02 Feb '26

Hello all, I've been using ceph 19.2.2 on alpine in a small cluster: 3 mons + 4 OSDs (12Tb each). Due a planning mistake the OSDs have been running on 2 nodes only, which I was working to get spread to 4 nodes. My luck not being the best, I had a power loss which made 2 OSDs (one on each node) fail to start, which of course led to 23% of the PGs not being available. One of the OSDs (id 0) seems to be gone for good due hardware fault, so I'm putting efforts to fix the other OSD (id 2). When I attempt to use 'ceph-bluestore-tool fsck ...' on 19.2.2 it core dumps almost immediately. So I've tried to use it from the official v20 container. Running "ceph-bluestore-tool fsck --path /var/lib/ceph/osd" (where activate is exposing the osd) outputs: 2026-02-02T18:12:18.895+0000 7f0c543eac00 -1 bluestore(/var/lib/ceph/osd) fsck error: free extent 0x1714c521000~978b26df000 intersects allocatedblocks fsck status: remaining 1 error(s) and warning(s) # And running it with repair instead: 2026-02-02T22:55:53.404+0000 7f8203afdc00 -1 bluestore(/var/lib/ceph/osd) fsck error: free extent 0x1714c521000~978b26df000 intersects allocatedblocks 2026-02-02T22:55:54.799+0000 7f8203afdc00 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 code = Rocksdb transaction: MergeCF( prefix = b key = 0x000001714D100000 value size = 16) MergeCF( prefix = b key = 0x000001714D100000 value size = 16) MergeCF( prefix = b key = 0x000001714D100000 value size = 16) <..... several times the same message...> MergeCF( prefix = b key = 0x000001714D100000 value size = 16) MergeCF( prefix = b key = 0x000001714D100000 value size = 16) /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function 'unsigned int BlueStoreRepairer::apply(KeyValueDB*)' thread 7f8203afdc00 time 2026-02-02T22:55:55.024481+0000 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 19514: FAILED ceph_assert(ok) ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12f) [0x7f8204b6b8f8] 2: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 3: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 4: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 5: main() 6: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 7: __libc_start_main() 8: _start() 2026-02-02T22:55:55.047+0000 7f8203afdc00 -1 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function 'unsigned int BlueStoreRepairer::apply(KeyValueDB*)' thread 7f8203afdc00 time 2026-02-02T22:55:55.024481+0000 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 19514: FAILED ceph_assert(ok) ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12f) [0x7f8204b6b8f8] 2: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 3: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 4: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 5: main() 6: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 7: __libc_start_main() 8: _start() *** Caught signal (Aborted) ** in thread 7f8203afdc00 thread_name:ceph-bluestore- ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: /lib64/libc.so.6(+0x3fc30) [0x7f8203f78c30] 2: /lib64/libc.so.6(+0x8d03c) [0x7f8203fc603c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f8204b6b957] 6: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 7: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 8: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 9: main() 10: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 11: __libc_start_main() 12: _start() 2026-02-02T22:55:55.082+0000 7f8203afdc00 -1 *** Caught signal (Aborted) ** in thread 7f8203afdc00 thread_name:ceph-bluestore- ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: /lib64/libc.so.6(+0x3fc30) [0x7f8203f78c30] 2: /lib64/libc.so.6(+0x8d03c) [0x7f8203fc603c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f8204b6b957] 6: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 7: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 8: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 9: main() 10: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 11: __libc_start_main() 12: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -499> 2026-02-02T22:55:53.404+0000 7f8203afdc00 -1 bluestore(/var/lib/ceph/osd) fsck error: free extent 0x1714c521000~978b26df000 intersects allocated blocks -498> 2026-02-02T22:55:54.799+0000 7f8203afdc00 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 code = ☻ Rocksdb transaction: Trying to spin up OSD throws this output: <... snipped for brevity ..> -27> 2026-01-24T01:50:57.764+0000 7eff73af5ed0 0 osd.2 2691 crush map has features 432629308056666112 was 8705, adjusting msgr requires for mons -26> 2026-01-24T01:50:57.764+0000 7eff73af5ed0 0 osd.2 2691 crush map has features 3314933069573799936, adjusting msgr requires for osds -25> 2026-01-24T01:50:57.764+0000 7eff73af5ed0 1 osd.2 2691 check_osdmap_features require_osd_release unknown -> squid -24> 2026-01-24T01:51:02.717+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.133333 -23> 2026-01-24T01:51:02.717+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0240964 -22> 2026-01-24T01:51:02.717+0000 7eff72019b30 5 bluestore.MempoolThread(0x7eff7397ebe0) _resize_shards cache_size: 6496138035 kv_alloc: 2785017856 kv_used: 6271162 kv_onode_alloc: 503316480 kv_onode_used: 115387532 meta_alloc: 2717908992 meta_used: 57075 data_alloc: 436207616 data_used: 20480 -21> 2026-01-24T01:51:05.010+0000 7eff725b7b30 3 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:3496] Compaction error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 -20> 2026-01-24T01:51:05.010+0000 7eff725b7b30 3 rocksdb: [db/error_handler.cc:397] Background IO error Corruption: block checksum mismatch:stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 -19> 2026-01-24T01:51:05.010+0000 7eff725b7b30 4 rocksdb: [db/error_handler.cc:285] ErrorHandler: Set regular background error -18> 2026-01-24T01:51:05.010+0000 7eff725b7b30 4 rocksdb: (Original Log Time 2026/01/24-01:51:05.011167) [db/compaction/compaction_job.cc:865] [default] compacted to: files[8 3 0 0 0 0 0] max score 0.01, MB/sec: 16.4 rd, 2.2 wr, level 1, files in(8, 1) out(1 +0 blob) MB in(0.0, 114.7 +0.0 blob) out(15.5 +0.0 blob), read-write-amplify(13694.2) write-amplify(1631.5) Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455, records in: 20619364, records dropped: 17423737 output_compression: LZ4 -17> 2026-01-24T01:51:05.010+0000 7eff725b7b30 4 rocksdb: (Original Log Time 2026/01/24-01:51:05.011226) EVENT_LOG_v1 {"time_micros": 1769219465011193, "job": 3, "event": "compaction_finished", "compaction_time_micros": 7334898, "compaction_time_cpu_micros": 7009405, "output_level": 1, "num_output_files": 1, "total_output_size": 16264458, "num_input_records": 20619364, "num_output_records": 3195627, "num_subcompactions": 1, "output_compression": "LZ4", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [8, 3, 0, 0, 0, 0, 0]} -16> 2026-01-24T01:51:05.010+0000 7eff725b7b30 2 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:2986] Waiting after background compaction error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455,Accumulated background error counts: 1 -15> 2026-01-24T01:51:06.032+0000 7eff725b7b30 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file db/170263.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 -14> 2026-01-24T01:51:06.032+0000 7eff725b7b30 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1769219466032857, "job": 3, "event": "table_file_deletion", "file_number": 170263} -13> 2026-01-24T01:51:07.746+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.117647 -12> 2026-01-24T01:51:07.746+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0243902 -11> 2026-01-24T01:51:07.746+0000 7eff72019b30 5 bluestore.MempoolThread(0x7eff7397ebe0) _resize_shards cache_size: 6496138035 kv_alloc: 2751463424 kv_used: 3155996 kv_onode_alloc: 570425344 kv_onode_used: 183341998 meta_alloc: 2684354560 meta_used: 57075 data_alloc: 436207616 data_used: 20480 -10> 2026-01-24T01:51:12.768+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.111111 -9> 2026-01-24T01:51:12.768+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0246914 -8> 2026-01-24T01:51:12.768+0000 7eff72019b30 5 bluestore.MempoolThread(0x7eff7397ebe0) _resize_shards cache_size: 6496138035 kv_alloc: 2717908992 kv_used: 3155996 kv_onode_alloc: 603979776 kv_onode_used: 226502426 meta_alloc: 2650800128 meta_used: 57075 data_alloc: 436207616 data_used: 20480 -7> 2026-01-24T01:51:15.044+0000 7eff73af5ed0 0 osd.2 2691 load_pgs -6> 2026-01-24T01:51:15.108+0000 7eff73af5ed0 5 osd.2 pg_epoch: 2691 pg[6.c(unlocked)] enter Initial -5> 2026-01-24T01:51:15.108+0000 7eff73af5ed0 5 osd.2 pg_epoch: 2691 pg[6.c( empty local-lis/les=2690/2691 n=0 ec=196/142 lis/c=2690/2690 les/c/f=2691/2691/0 sis=2690) [1,3]/[1,3,2] r=2 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000424 0 0.000000 -4> 2026-01-24T01:51:15.108+0000 7eff73af5ed0 5 osd.2 pg_epoch: 2691 pg[6.c( empty local-lis/les=2690/2691 n=0 ec=196/142 lis/c=2690/2690 les/c/f=2691/2691/0 sis=2690) [1,3]/[1,3,2] r=2 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset -3> 2026-01-24T01:51:15.108+0000 7eff72065b30 5 bluestore(/var/lib/ceph/osd) _kv_sync_thread utilization: idle 17.420808792s of 17.420808792s, submitted: 0 -2> 2026-01-24T01:51:15.108+0000 7eff72065b30 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 code = Rocksdb transaction: PutCF( prefix = O key = 0x7F800000000000000630000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F value size = 34) PutCF( prefix = S key = 'nid_max' value size = 8) PutCF( prefix = S key = 'blobid_max' value size = 8) -1> 2026-01-24T01:51:15.108+0000 7eff72065b30 -1 /home/buildozer/aports/community/ceph19/src/ceph-19.2.2/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7eff72065b30 time 2026-01-24T01:51:15.109433+0000 /home/buildozer/aports/community/ceph19/src/ceph-19.2.2/src/os/bluestore/BlueStore.cc: 14045: FAILED ceph_assert(r == 0) ceph version Development (no_version) squid (stable) 0> 2026-01-24T01:51:15.109+0000 7eff72065b30 -1 *** Caught signal (Aborted) ** in thread 7eff72065b30 thread_name: Any help getting the OSD back running would be greatly appreciated. There's only a handful of folders from cephfs I would like to recover from the system which weren't backed up. The system was far from full (OSDs were 30% full). Of course I can post full logs if needed, just let me know if I should attach them in an email in the mailing list or if preferable to use something like pastebin or another way. Many thanks, Theo

1 0

Re: CephFS “rookfs” marked damaged after cephfs-journal-tool reset on Rook/Kubernetes (Reef 18.2.2)
by Eugen Block 02 Feb '26

02 Feb '26

Hi, you need to fix the stale PGs first. Do they belong to one of the CephFS pools? If that is the case your journal issue most likely roots in those stale PGs. Anth All via ceph-users <ceph-users(a)ceph.io> schrieb am Mo. 2. Feb. 2026 um 19:58: > Hi all, > I’m running a Ceph cluster managed by Rook on Kubernetes, and my CephFS > metadata/journal appears to be in a bad state. I’d like to get advice > before I attempt any destructive metadata repair operations. > Below is a summary of the situation. > *Environment* > > - Orchestrator: Rook/Ceph on Kubernetes (namespace: rook-ceph) > - Ceph version: 18.2.2 (Reef, stable) for mon/mgr/osd/mds > - Filesystem: > - Name: rookfs > - One metadata pool > - One data pool > > > *Cluster health* > ceph status shows: > > - Health: HEALTH_WARN > - Warnings: > - 1 MDS reports slow metadata IOs > - 1 MDS reports slow requests > - Reduced data availability: some PGs in stale state > - A large number of daemons have recently crashed > > MDS section reports: > > - 1/1 MDS daemons up, 1 hot standby > > > CephFS state and MDS status > ceph fs dump for rookfs shows > > - Filesystem rookfs is marked damaged. > - max_mds = 1. > - in set is empty. > - up set is {0=<mds_id>}. > - Flags mention allow_standby_replay. > > > So, the filesystem is marked damaged in the fsmap, while one MDS is still > up:active on rank 0. > ceph tell mds.* status confirms that the MDS for rookfs is: > > - state: up:active > - fs_name: rookfs > - whoami: 0 > > > > I ran the following commands on the CephFS journal: > > 1. Journal reset: > > bash > cephfs-journal-tool --rank=rookfs:0 journal reset > > This completed and indicated a new journal start offset. > 2. Journal inspection: > > bash > cephfs-journal-tool --rank=rookfs:0 journal inspect > > Output : > - Bad entry start ptr (...) at certain offsets > - Overall journal integrity: DAMAGED > - Corrupt regions reported, including a range up to ffffffffffffffff > > So, even after the reset, cephfs-journal-tool reports the journal as > DAMAGED with > corrupt regions. > Listing the metadata pool shows at least the mds_snaptable object, so the > metadata pool is not empty. > > > *Current behaviour* > > - ceph fs status is sometimes very slow or appears to hang. > - Ceph health reports: > - “MDSs report slow metadata IOs” > - “MDSs report slow requests” > - Stale PGs in the cluster > - The filesystem rookfs is marked damaged in ceph fs dump, but the > MDS is still up:active on rank 0. > > Any guidance or best practices for handling this kind of journal corruption > and damaged filesystem in a Rook/Kubernetes setup would be greatly > appreciated, including precautions you would strongly recommend before > running the heavy-repair commands. > Best regards, > Anthony > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

CephFS “rookfs” marked damaged after cephfs-journal-tool reset on Rook/Kubernetes (Reef 18.2.2)
by Anth All 02 Feb '26

02 Feb '26

Hi all, I’m running a Ceph cluster managed by Rook on Kubernetes, and my CephFS metadata/journal appears to be in a bad state. I’d like to get advice before I attempt any destructive metadata repair operations. Below is a summary of the situation. *Environment* - Orchestrator: Rook/Ceph on Kubernetes (namespace: rook-ceph) - Ceph version: 18.2.2 (Reef, stable) for mon/mgr/osd/mds - Filesystem: - Name: rookfs - One metadata pool - One data pool *Cluster health* ceph status shows: - Health: HEALTH_WARN - Warnings: - 1 MDS reports slow metadata IOs - 1 MDS reports slow requests - Reduced data availability: some PGs in stale state - A large number of daemons have recently crashed MDS section reports: - 1/1 MDS daemons up, 1 hot standby CephFS state and MDS status ceph fs dump for rookfs shows - Filesystem rookfs is marked damaged. - max_mds = 1. - in set is empty. - up set is {0=<mds_id>}. - Flags mention allow_standby_replay. So, the filesystem is marked damaged in the fsmap, while one MDS is still up:active on rank 0. ceph tell mds.* status confirms that the MDS for rookfs is: - state: up:active - fs_name: rookfs - whoami: 0 I ran the following commands on the CephFS journal: 1. Journal reset: bash cephfs-journal-tool --rank=rookfs:0 journal reset This completed and indicated a new journal start offset. 2. Journal inspection: bash cephfs-journal-tool --rank=rookfs:0 journal inspect Output : - Bad entry start ptr (...) at certain offsets - Overall journal integrity: DAMAGED - Corrupt regions reported, including a range up to ffffffffffffffff So, even after the reset, cephfs-journal-tool reports the journal as DAMAGED with corrupt regions. Listing the metadata pool shows at least the mds_snaptable object, so the metadata pool is not empty. *Current behaviour* - ceph fs status is sometimes very slow or appears to hang. - Ceph health reports: - “MDSs report slow metadata IOs” - “MDSs report slow requests” - Stale PGs in the cluster - The filesystem rookfs is marked damaged in ceph fs dump, but the MDS is still up:active on rank 0. Any guidance or best practices for handling this kind of journal corruption and damaged filesystem in a Rook/Kubernetes setup would be greatly appreciated, including precautions you would strongly recommend before running the heavy-repair commands. Best regards, Anthony

1 0

Re: [ceph-user][ceph-ansible] Quincy: Module 'devicehealth' has failed: unknown operation
by Frédéric Nass 02 Feb '26

02 Feb '26

Hi Danish, You seem to be hitting this bug [1] that was fixed in Quincy v17.2.8 by [2]. If you're using packages (.rpm,.deb), you could upgrade to v17.2.9 that has the fix. Do not use v17.2.8 due to a critical BlueStore bug. If you're using cephadm and containers, you'll have to upgrade to Reef as v17.2.9 had no container images. Best regards, Frédéric. [1] https://tracker.ceph.com/issues/55606 [2] https://github.com/ceph/ceph/pull/52461 -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Squishing Squids - A Ceph Compression Guide <https://www.eventbrite.com/e/squishing-squids-a-ceph-compression-guide-tick…>, February 25th, 9am PST. https://clyso.com | frederic.nass(a)clyso.com Le sam. 31 janv. 2026 à 09:09, Danish Khan via ceph-users < ceph-users(a)ceph.io> a écrit : > Dear Team, > > I am keep getting below error after every few days: > > Error : > Module 'devicehealth' has failed: unknown operation > > Actual error from MGR log: > > 2026-01-30T05:50:59.338+0530 7f314da26640 0 [devicehealth ERROR root] > Caught fatal database error: unknown operation > 2026-01-30T05:50:59.338+0530 7f314da26640 -1 log_channel(cluster) log [ERR] > : Unhandled exception from module 'devicehealth' while running on > mgr.controller3: unknown operation > 2026-01-30T05:50:59.338+0530 7f314da26640 -1 devicehealth.serve: > 2026-01-30T05:50:59.338+0530 7f314da26640 -1 Traceback (most recent call > last): > File "/usr/share/ceph/mgr/devicehealth/module.py", line 408, in serve > self._do_serve() > File "/usr/share/ceph/mgr/mgr_module.py", line 513, in check > return func(self, *args, **kwargs) > File "/usr/share/ceph/mgr/devicehealth/module.py", line 399, in _do_serve > self.scrape_all() > File "/usr/share/ceph/mgr/devicehealth/module.py", line 449, in > scrape_all > self.put_device_metrics(device, data) > File "/usr/share/ceph/mgr/devicehealth/module.py", line 525, in > put_device_metrics > self._create_device(devid) > File "/usr/share/ceph/mgr/devicehealth/module.py", line 511, in > _create_device > cursor = self.db.execute(SQL, (devid,)) > sqlite3.InternalError: unknown operation > > 2026-01-30T05:50:59.366+0530 7f3155335640 0 log_channel(cluster) log [DBG] > : pgmap v116150: 6977 pgs: 92 peering, 40 stale+active+clean, 191 > active+clean+laggy, 30 active+undersized, 1026 active+undersized+degraded, > 5598 active+clean; 48 TiB data, 153 TiB used, 1.2 PiB / 1.4 PiB avail; 44 > MiB/s wr, 1.00k op/s; 8043806/154077270 objects degraded (5.221%); 4.4 > GiB/s, 8 keys/s, 530 objects/s recovering > > And I get one warning for mgr daemon crash but all MGR Service > actually doesn't get failed or restarted automatically. > > I guess it is something related to sqlite3 which checks ceph > health internally. > > Although this is not causing any issue and error disappears after > restarting ceph-mgr service or doing a failover of MGR to another > controller. > > Does anyone know how we can resolve this issue permanently? > > Similar issue: > https://github.com/rook/rook/issues/12349 > > Regards, > Danish > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Experiences with Tentacle 20.2.0
by Chris Palmer 02 Feb '26

02 Feb '26

Thanks Wannes. Those are the caveats around EC optimisation that I alluded to below. Since we don't have any EC on this cluster that should not be relevant... Regards, Chris On 02/02/2026 09:30, Wannes Smet via ceph-users wrote: > This story jumps to mind: https://docs.clyso.com/blog/2026/01/13/tentacle-v20.2.0-ec-bug/ > ________________________________ > From: Chris Palmer via ceph-users <ceph-users(a)ceph.io> > Sent: Monday, February 2, 2026 10:16 > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Experiences with Tentacle 20.2.0 > > I'd like to know whether anyone has any good or bad experiences with Tentacle 20. 2. 0 in a production environment. I have the opportunity to upgrade a small production cluster from 19. 2. 3 to 20. 2. 0. It uses Cephadm/RBD/CephFS/RGW with R3 only > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > I'd like to know whether anyone has any good or bad experiences with > Tentacle 20.2.0 in a production environment. > > I have the opportunity to upgrade a small production cluster from 19.2.3 > to 20.2.0. It uses Cephadm/RBD/CephFS/RGW with R3 only (no EC, so the > known caveats around EC optimisations won't be relevant). > > My experiences trying it in a lab test environment have been positive. > But it would be good to know if anyone is encountering any problems with > it in real life. > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Experiences with Tentacle 20.2.0
by Wannes Smet 02 Feb '26

02 Feb '26

This story jumps to mind: https://docs.clyso.com/blog/2026/01/13/tentacle-v20.2.0-ec-bug/ ________________________________ From: Chris Palmer via ceph-users <ceph-users(a)ceph.io> Sent: Monday, February 2, 2026 10:16 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Experiences with Tentacle 20.2.0 I'd like to know whether anyone has any good or bad experiences with Tentacle 20. 2. 0 in a production environment. I have the opportunity to upgrade a small production cluster from 19. 2. 3 to 20. 2. 0. It uses Cephadm/RBD/CephFS/RGW with R3 only ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I'd like to know whether anyone has any good or bad experiences with Tentacle 20.2.0 in a production environment. I have the opportunity to upgrade a small production cluster from 19.2.3 to 20.2.0. It uses Cephadm/RBD/CephFS/RGW with R3 only (no EC, so the known caveats around EC optimisations won't be relevant). My experiences trying it in a lab test environment have been positive. But it would be good to know if anyone is encountering any problems with it in real life. Thanks, Chris _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Experiences with Tentacle 20.2.0
by Chris Palmer 02 Feb '26

02 Feb '26

I'd like to know whether anyone has any good or bad experiences with Tentacle 20.2.0 in a production environment. I have the opportunity to upgrade a small production cluster from 19.2.3 to 20.2.0. It uses Cephadm/RBD/CephFS/RGW with R3 only (no EC, so the known caveats around EC optimisations won't be relevant). My experiences trying it in a lab test environment have been positive. But it would be good to know if anyone is encountering any problems with it in real life. Thanks, Chris

1 0