- ceph-users - lists.ceph.io

RGW/multisite: Segmentation fault during full sync
by Vahideh Alinouri 09 Feb '26

09 Feb '26

Hi folks, I have one realm, one zonegroup, and four zones, all running version 19.2.3. One zone was recently added to the zonegroup while it was on version 18.2.7. The newly added zone can perform data sync with the other secondary zones without issues. However, when syncing with the master zone, it gets stuck in the init state. The master zone, on the other hand, can sync with it successfully. All zones were recently upgraded from versions 18.2.4 and 18.2.7 to 19.2.3 but the problem still persists. sync status on master zone (dc07): radosgw-admin sync status realm 710cf69b-7382-47d2-aca6-03d991b00d1f (s3-cdn) zonegroup 7c01d60f-88c6-4192-baf7-d725260bf05d (s3-cdn-group) zone 03f6a8ec-008c-4cbf-8efc-d70a6013066f (s3-cdn-dc07) current time 2026-02-09T08:57:47Z zonegroup features enabled: disabled: compress-encrypted, notification_v2, resharding metadata sync: no sync (zone is master) data sync source: 1a6e33b9-8ece-4b9c-a9a5-961fa97c42c8 (s3-cdn-dc05) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shard behind shards: [93] oldest incremental change not applied: 2026-02-09T08:57:44.779833+0000 [93] source: 367dbfe9-a5f8-4101-a271-9749f25ba09c *(s3-cdn-dc10)* *syncing* full sync: 0/128 shards incremental sync: 128/128 shards 1 shard is recovering recovering shards: [125] source: 40122a7c-e594-43b7-89bb-e7ada37991c5 (s3-cdn-dc06) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source sync status on problematic zone (dc10): radosgw-admin sync status realm 710cf69b-7382-47d2-aca6-03d991b00d1f (s3-cdn) zonegroup 7c01d60f-88c6-4192-baf7-d725260bf05d (s3-cdn-group) zone 367dbfe9-a5f8-4101-a271-9749f25ba09c (s3-cdn-dc10) current time 2026-02-09T09:00:17Z zonegroup features enabled: disabled: compress-encrypted, notification_v2, resharding metadata sync: syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 03f6a8ec-008c-4cbf-8efc-d70a6013066f *(s3-cdn-dc07)* *init* full sync: 128/128 shards full sync: 0 buckets to sync incremental sync: 0/128 shards data is behind on 128 shards behind shards: [0–127] source: 1a6e33b9-8ece-4b9c-a9a5-961fa97c42c8 (s3-cdn-dc05) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shard behind shards: [93] oldest incremental change not applied: 2026-02-09T08:59:44.783203+0000 [93] 1 shard is recovering recovering shards: [41] source: 40122a7c-e594-43b7-89bb-e7ada37991c5 (s3-cdn-dc06) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source I see the following logs in the RGW sync enabled service: Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xb4) [0x5602e312a6f4] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 5: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4d7) [0x5602e3598d97] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 6: /usr/bin/radosgw(+0x837248) [0x5602e32eb248] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 7: (RGWRadosThread::Worker::entry()+0xbd) [0x5602e32ec83d] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 8: /lib64/libc.so.6(+0x8a4da) [0x7fbd2b2394da] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 9: clone() Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: debug -9999> 2026-02-09T09:05:46.708+0000 7fbbf4c19640 -1 *** Caught signal (Segmentation fault) ** Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: in thread 7fbbf4c19640 thread_name:data-sync Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 1: /lib64/libc.so.6(+0x3ebf0) [0x7fbd2b1edbf0] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 2: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x37) [0x5602e3127c97] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 3: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x481) [0x5602e3129901] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xb4) [0x5602e312a6f4] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 5: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4d7) [0x5602e3598d97] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 6: /usr/bin/radosgw(+0x837248) [0x5602e32eb248] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 7: (RGWRadosThread::Worker::entry()+0xbd) [0x5602e32ec83d] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 8: /lib64/libc.so.6(+0x8a4da) [0x7fbd2b2394da] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 9: clone() Additionally, when I restart the RGW sync enabled service on the newly added zone, the daemon crashes. If I restart the RGW sync enabled daemon directly, it also crashes. To restart RGW sync enabled successfully, I have to run: ceph config set client.rgw.s3-cdn-colocate.ceph-mon10001.rrbzdg rgw_run_sync_thread false ceph orch daemon restart rgw.s3-cdn-colocate.ceph-mon10001.rrbzdg ceph config set client.rgw.s3-cdn-colocate.ceph-mon10001.rrbzdg rgw_run_sync_thread true I saw that this bug (https://tracker.ceph.com/issues/63378) was resolved and backported to 19.2.3. However, I am still observing the same behavior.

1 0

Re: Disable cache tier
by Kees Meijs | Nefos 09 Feb '26

09 Feb '26

Moin' and thanks! I'm going to take a further look at the objects within the pool and report back my findings. Cheers, Kees On 05/02/2026 20:36, Eugen Block via ceph-users wrote: > I've been searching the list for my own report from last year maybe, when > we finally got rid of our cache tier as well. I can't really find it > though, so I'm gonna summarize again how we managed to get out of it. > > After setting the cache mode to proxy and failing to flush/evict (also with > "no such file or directory" messages), I gradually reduced the cache pool > size from 260 GB in steps of 40 GB or so, not sure anymore .Suddenly Ceph > was able to flush and evict again. > I reduced the target_max_bytes to 1000000, only rbd_header objects remained > in the pool, but those couldn't be flushed/evicted anymore: > > failed to evict /rbd_header.133ba5e7513046: (16) Device or resource busy > rbd_header.8fbda012200854 > > Unfortunately, we didn't find a promsing way to get rid of them while the > images were in use. I had played around in a test cluster and ignored the > header objects when remocing the cache tier, and it seemed to work (since > the rbd_header objects should be identical in the backing pool as well), > but it didn't feel safe and we were really uncomfortable trying it in > production. So in the end, we shut down our VMs to be able to flush/evict > everything and safely got rid of the cache tier. > > I know this isn't really helpful, but maybe next time I find my comment on > the list. 🙂

1 0

MGR Restful not accessible when doing mgr failover (squid v19.2.3)
by Andreas Eriksson 08 Feb '26

08 Feb '26

Hi, I'm doing cluster monitoring using the restful api on the ceph-mgr on port 8003. The requests are working fine as long as I have my *mgr-0* as primary. Doing a `ceph mgr fail` so that the *mgr-1* becomes active, the mgr daemon doesn't listen on port 8003. Is there any workaround for this?

1 0

Repeated crashes of mgr module in Tentacle
by Daniel Brown 08 Feb '26

08 Feb '26

Greetings — Have been seeing repeated crashes on my mgr module. Seems to run for about 45 to 50 seconds and then boom. Cephadm setup here. Did (try to) enable a couple modules lately. iostat, stats diskprediction_local, but have toggled them back off - unfortunately it hasn’t fixed the issue. I was seeing “no mgr” with ceph -s — to get around that I’ve tweaked the “StartLimitInterval” setting in the ceph-[CLUSTER-UID]@.service file down to 1m, and have 4x mgr’s setup so that I can get a couple commands run before mgr crashes and another starts. The 30m default there seems… high, imo - I was having intervals with no mgr which makes it tough to do much with the cluster - even “ceph -s” was hanging. Everything else in the cluster seems “normal” - still serving data. One other note — which I think is generally unrelated — I did upgrade one of my cluster nodes from “Plucky Puffin" (25.04) ubuntu, to “Questing Quokka” (25.10) ubuntu. After the upgrade, cephadm managed containers didn’t want to start. I tracked that down to having the ceph user userid in /etc/passwd set at 64045, but the container seeming to want userid 167. Most things under /var/lib/ceph/[CLUSTER UID]/ … appear to be owned by user/group 167:167 — I assume this is a default inside the container. Workaround here was to manually change the UID/GID for ceph in /etc/passwd and /etc/group. I’m going to imagine this is some collision between cephadm managed deployments, and how Ubuntu / apt installs cephadm. The aforementioned mgr Crashes look like: { "assert_condition": "cursor != root", "assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h", "assert_func": "virtual void PyFormatter::close_section()", "assert_line": 84, "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h: In function 'virtual void PyFormatter::close_section()' thread ffff34e55700 time 2026-02-08T13:10:07.526894+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h: 84: FAILED ceph_assert(cursor != root)\n", "assert_thread_name": "telemetry", "backtrace": [ "__kernel_rt_sigreturn()", "/lib64/libc.so.6(+0x82a78) [0xffff83603a78]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x190) [0xffff84039874]", "/usr/bin/ceph-mgr(+0xcf540) [0xaaaacdf2f540]", "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xf6c) [0xaaaacdf46ec0]", "/usr/bin/ceph-mgr(+0x105528) [0xaaaacdf65528]", "/lib64/libpython3.9.so.1.0(+0xf7bfc) [0xffff84ae9bfc]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xda3f0) [0xffff84acc3f0]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0xda3f0) [0xffff84acc3f0]", "/lib64/libpython3.9.so.1.0(+0xea93c) [0xffff84adc93c]", "/lib64/libpython3.9.so.1.0(+0xcf304) [0xffff84ac1304]", "/lib64/libpython3.9.so.1.0(+0x197d78) [0xffff84b89d78]", "_PyObject_CallMethod_SizeT()", "(PyModuleRunner::serve()+0x6c) [0xaaaacdfdf6cc]", "(PyModuleRunner::PyModuleRunnerThread::entry()+0x148) [0xaaaacdfdff08]" ], "ceph_version": "20.2.0", "crash_id": "2026-02-08T13:10:07.528739Z_f18e6b74-438b-47db-9438-5a3861fdef2d", "entity_name": "mgr.hc-945901a5cad1b6e3.mtijbv", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mgr", "stack_sig": "319d76a0d71a4644f9d65f592f6e621cca918d9205fc759ab7acf6944bc77fdd", "timestamp": "2026-02-08T13:10:07.528739Z", "utsname_hostname": "hc-945901a5cad1b6e3", "utsname_machine": "aarch64", "utsname_release": "6.14.0-1010-raspi", "utsname_sysname": "Linux", "utsname_version": "#10-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 15 19:09:05 UTC 2025" }

1 0

Re: Limiting number of object versions per key in RGW (hotspot / bucket index shard growth)
by Frédéric Nass 07 Feb '26

07 Feb '26

Hi Ansgar, You're absolutely right; it makes sense to protect the infrastructure as a whole rather than relying on users to properly manage their bucket's data. This concern was discussed at Cephalocon '24 [1]. A tracker [2] had been created a couple of months before the conference to implement this feature, and the last two comments show that your message is getting some attention. This slightly different one that Sage created in 2011 is also worth mentioning [3]. Hopefully, this will be implemented soon. Cheers, Frédéric. [1] https://pad.ceph.com/p/cds24-powerusers#L128 [2] https://tracker.ceph.com/issues/68066 [3] https://tracker.ceph.com/issues/1424 -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Squishing Squids - A Ceph Compression Guide <https://www.eventbrite.com/e/squishing-squids-a-ceph-compression-guide-tick…>, February 25th, 9am PST. https://clyso.com | frederic.nass(a)clyso.com Le mer. 4 févr. 2026 à 18:53, Ansgar Jazdzewski via ceph-users < ceph-users(a)ceph.io> a écrit : > Hello Frédéric, > > thank you, that is very helpful information. > > Support for NewerNoncurrentVersions in Squid lifecycle definitely > addresses the issue for customers who configure lifecycle rules > properly. > > However, as an object storage service provider our concern is more > about cluster protection against unbounded version growth in general: > > lifecycle policies are optional and customer-controlled > * a tenant can enable versioning and never configure expiration > * a single hot key can still accumulate millions of versions and > create a bucket-index shard hotspot > * So while Squid lifecycle makes it possible for customers to > self-mitigate, we still see a gap in terms of "provider-side > guardrails" (e.g., a hard or automatic limit on versions per key). > > Do you know if there is any discussion about enforcing such limits > globally or at least providing admin-side defaults/mandatory lifecycle > policies? > > Best regards, > Ansgar > > Am Mi., 4. Feb. 2026 um 14:23 Uhr schrieb Frédéric Nass > <frederic.nass(a)clyso.com>: > > > > Hi Ansgar, > > > > Since Squid (v19) and PR #54152 [1], the RGW Lifecycle supports the > 'NoncurrentDays' and 'NewerNoncurrentVersions' filters. Would that work for > your use case? > > You can find examples of how to use these filters here [2]. > > > > Note that PR also added support for ObjectSizeGreater(Less)Than filers > to transition objects based on their size. > > > > Best regards, > > Frédéric Nass > > > > [1] https://github.com/ceph/ceph/pull/54152 > > [2] > https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configurati… > > > > -- > > Frédéric Nass > > Ceph Ambassador France | Senior Ceph Engineer @ CLYSO > > > > Squishing Squids - A Ceph Compression Guide, February 25th, 9am PST. > > https://clyso.com | frederic.nass(a)clyso.com > > > > > > Le mer. 4 févr. 2026 à 13:23, Ansgar Jazdzewski via ceph-users < > ceph-users(a)ceph.io> a écrit : > >> > >> Hi Cephers, > >> > >> we are running Ceph RGW with bucket versioning enabled (currently on > >> Reef, but we have also observed the same behavior on other releases). > >> > >> We are seeing an operational issue where a single object key can > >> accumulate extremely large numbers of versions (hundreds of thousands > >> up to millions). Because all versions of the same key are stored on > >> the same bucket index shard, this results in very large bucket-index > >> entries / omap growth and shard hotspots. > >> > >> In our case, one object with millions of versions caused significant > >> bucket index pressure and related performance/maintenance concerns. > >> > >> As far as we can tell, RGW currently does not provide a way to enforce > >> a limit such as “keep only the last N versions per object key,” and > >> existing quotas only apply at bucket/user level, not per object key. > >> > >> We wanted to ask: > >> * Have others seen similar cases with very high version counts for > >> individual keys? > >> * How do you handle or mitigate this operationally today > >> > >> Any advice or shared experience would be very appreciated. > >> > >> Thank you, > >> Ansgar Jazdzewski > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Ilya Dryomov 06 Feb '26

06 Feb '26

On Thu, Feb 5, 2026 at 10:59 PM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > > We are continuing on this release. > > In the tracker https://tracker.ceph.com/issues/73906 please see runs with the comment "on Build 5". > This build and runs were done after the lab move. > > Seeking approvals for: > > smoke - FAILED, was reviewed and approved by Laura and Patrick, see Slack thread for details > rados - FAILED, Afreen, Travis, Laura, Radek approved? > > rgw - FAILED, Adam E approved? > > rbd, krbd - FAILED, Ilya approved? krbd approved. For rbd, I did a rerun [1]. The qemu-iotests.sh failure is known and can be ignored, but the crashes in test_librbd_python.sh are concerning. Patrick and I discussed them briefly -- they seem to be specific to 20.04, but I don't remember seeing them before. There were multiple passes on 20.04 in the run that I approved last year [2], so it looks like something around Python, Cython or the build process has changed in the interim. [1] https://pulpito.ceph.com/dis-2026-02-02_19:37:16-rbd-reef-release-distro-de… [2] https://pulpito.ceph.com/yuriw-2025-11-18_23:00:18-rbd-reef-release-distro-… Thanks, Ilya

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 06 Feb '26

06 Feb '26

I tried yesterday, if I stop OSD and just set OSD to out (without reweight), then radosgw does not stop/hangs and rebalance finishes, of course with a little bit more traffic, ... So, one should not reweight a failed disk, but just stop and set failed OSD to out and leave ceph to rebalance. Thx to all. Rok On Thu, Feb 5, 2026 at 1:08 AM Kirby Haze via ceph-users <ceph-users(a)ceph.io> wrote: > When reweight the drive down to 0, does the behavior change if you set it > 'out' first? The only other special thing that happens with reweight 0 is > that upmaps get removed for said OSD. > > If you think it is at the pool level you could run a simple rados bench to > a pool while you reweight an OSD to see if your issue still holds. > > Also could be some issue with osd_op_queue being mclock if you have that, > might be worth A/B testing with one OSD if you still see it. > > On Wed, Feb 4, 2026 at 12:31 PM Rok Jaklič via ceph-users < > ceph-users(a)ceph.io> wrote: > > > If one HDD drive fails and smartctl shows errors and then I decide to > drain > > it with crush reweight 0, would Ceph try to copy/move data/pgs from data > > failed disk anyway? > > > > Because we noticed on some non ceph clusters (raid setup, actually mail > > servers), that one failed drive may hog up "app/OS" because "app" fails > to > > read/write to failed disk because "some queue" fills up (since app/OS is > > unable for data to be read/written)? > > > > Could something similar happen in Ceph? > > > > Rok > > > > On Wed, Feb 4, 2026 at 9:52 AM Rok Jaklič <rjaklic(a)gmail.com> wrote: > > > > > > > > On Wed, Feb 4, 2026 at 2:59 AM Anthony D'Atri <anthony.datri(a)gmail.com > > > > > wrote: > > > > > >> Are these rear bay drives, hence the limit of 2? Or > > >> You might consider an M.2 AIC adapter card with bifurcation. M.2 > > >> enterprise SSDs are sunsetting but for retrofits you should be able to > > find > > >> Micron 6450 units. > > >> > > >> What’s your workload like? > > >> > > > > > > On average 10-50MB/s of write, with spikes up to a few hundred MB/s > > during > > > evening/night time; it went up to 1GB/s during tests without a problem. > > All > > > these are S3 workloads/tests. > > > > > > I would have to check that on site, RM does not show, however we are > just > > > about to migrate to new machines, which have 4 NVMe slots ... so I am > > > really considering moving WAL/DB to NVMe, however I am still a little > bit > > > hesitant, since I am not really sure this will solve the problem of why > > > radosgw/s3 stops after some time when setting crush reweight to 0 on > one > > > failed disk. We are doing the same thing on HPC where radosgw/s3 is not > > > used and we are not experiencing this problem there. If we move WAL/DB > to > > > NVMe, and if one NVMe fails and we have to recover 10 OSDs for example, > > it > > > would take much longer than if just 1 OSD has to be recovered (while > > users > > > being unable to access s3). > > > > > > --- > > > > > > My suspicion is that when we set crush reweight of the failed disk to > 0, > > > all other affected disks from that pool disables some write (because of > > > recovery) and some queue fills up which then stops/hangs radosgw... > > > > > > Rok > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Brad Hubbard 05 Feb '26

05 Feb '26

On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > We are continuing on this release. > > > powercycle - FAILED Brad, approved? > Unfortunately, I don't see any logs anywhere where I have access to them so I have no way of checking this? > ceph-volume - Approved Guillaume? > > Patrick was reviewing some tests, please see his comments in the same > tracker. > TIA > > > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < > ceph-users(a)ceph.io> wrote: > > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> > > wrote: > > > > > > This test branch was built loner then 2.5 weeks ago > > > I have to rebuild it first and then will schedule krbd > > > > krbd approved based on > > > > > > > https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… > > > > > https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… > > > > Thanks, > > > > Ilya > > > > > > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > > ceph-users(a)ceph.io> wrote: > > >> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> > wrote: > > >> > > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > > >> > <ceph-users(a)ceph.io> wrote: > > >> > > > > >> > > Hi Yuri, > > >> > > > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < > yweinste(a)redhat.com> > > wrote: > > >> > > > > > >> > > > Note: this is an attempt to test and publish post EOL reef > point > > release > > >> > > > > > >> > > > Details of this release are summarized here: > > >> > > > > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > > >> > > > Release Notes - TBD > > >> > > > LRC upgrade - N/A > > >> > > > > > >> > > > distro => jammy focal centos9 windows bookworm > > >> > > > > > >> > > > Seeking approvals/reviews for: > > >> > > > > > >> > > > smoke - Laura approved? > > >> > > > > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > > approved? > > >> > > > (rados is also running on build 2 == build 1 + > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > >> > > > > > >> > > > rgw - Adam E approved? > > >> > > > > > >> > > > fs - Venky approved? > > >> > > > > >> > > There are lot many failures in the fs suite resembling > > >> > > > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong > fs > > >> > > type, bad option ... > > >> > > > > >> > > which is an issue with the testing kernel. Adding Alex who was > > looking > > >> > > into this issue. > > >> > > > > >> > > As far as the run validation is concerned, it's risky to ignore > the > > >> > > large number of failures. We have two options: > > >> > > > > >> > > - fix the testing kernel issue and rerun > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the > > >> > > teuthology-suite schedule command) > > >> > > > > >> > > The latter would be using an older kernel branch, but I think it > > >> > > should suffice for this case. > > >> > > > >> > Hi Venky, > > >> > > > >> > With your report of the fs suite being affected as well, I carved > out > > >> > some time to take a look. > > >> > > > >> > I think it's happening because XFS maintainers disabled support for > V4 > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel > it's > > >> > no longer compiled in by default. The proper fix would be to create > > an > > >> > image with/move to V5 on-disk format; I have opened [1] to try to > > >> > achieve that. We can also re-enable V4 on-disk format in our kernel > > >> > builds as a workaround. > > >> > > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > >> > > >> This should be fixed now. Yuri, please schedule a rerun for krbd. > > >> > > >> Thanks, > > >> > > >> Ilya > > >> _______________________________________________ > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > _______________________________________________ > > > Dev mailing list -- dev(a)ceph.io > > > To unsubscribe send an email to dev-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Cheers, Brad

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 05 Feb '26

05 Feb '26

We are continuing on this release. In the tracker https://tracker.ceph.com/issues/73906 please see runs with the comment "on Build 5". This build and runs were done after the lab move. Seeking approvals for: smoke - FAILED, was reviewed and approved by Laura and Patrick, see Slack thread for details rados - FAILED, Afreen, Travis, Laura, Radek approved? rgw - FAILED, Adam E approved? rbd, krbd - FAILED, Ilya approved? fs - FAILED, Venky approved? orch - FAILED, Adam K approved? upgrade/pacific-x (reef) - FAILED Laura, approved? upgrade/quincy-x (reef) - FAILED Laura, approved? upgrade/reef-p2p - FAILED Laura, approved? powercycle - FAILED Brad, approved? ceph-volume - Approved Guillaume? Patrick was reviewing some tests, please see his comments in the same tracker. TIA On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < ceph-users(a)ceph.io> wrote: > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> > wrote: > > > > This test branch was built loner then 2.5 weeks ago > > I have to rebuild it first and then will schedule krbd > > krbd approved based on > > > https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… > > https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… > > Thanks, > > Ilya > > > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > ceph-users(a)ceph.io> wrote: > >> > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: > >> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > >> > <ceph-users(a)ceph.io> wrote: > >> > > > >> > > Hi Yuri, > >> > > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > >> > > > > >> > > > Note: this is an attempt to test and publish post EOL reef point > release > >> > > > > >> > > > Details of this release are summarized here: > >> > > > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > >> > > > Release Notes - TBD > >> > > > LRC upgrade - N/A > >> > > > > >> > > > distro => jammy focal centos9 windows bookworm > >> > > > > >> > > > Seeking approvals/reviews for: > >> > > > > >> > > > smoke - Laura approved? > >> > > > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > approved? > >> > > > (rados is also running on build 2 == build 1 + > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > >> > > > > >> > > > rgw - Adam E approved? > >> > > > > >> > > > fs - Venky approved? > >> > > > >> > > There are lot many failures in the fs suite resembling > >> > > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > >> > > type, bad option ... > >> > > > >> > > which is an issue with the testing kernel. Adding Alex who was > looking > >> > > into this issue. > >> > > > >> > > As far as the run validation is concerned, it's risky to ignore the > >> > > large number of failures. We have two options: > >> > > > >> > > - fix the testing kernel issue and rerun > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the > >> > > teuthology-suite schedule command) > >> > > > >> > > The latter would be using an older kernel branch, but I think it > >> > > should suffice for this case. > >> > > >> > Hi Venky, > >> > > >> > With your report of the fs suite being affected as well, I carved out > >> > some time to take a look. > >> > > >> > I think it's happening because XFS maintainers disabled support for V4 > >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's > >> > no longer compiled in by default. The proper fix would be to create > an > >> > image with/move to V5 on-disk format; I have opened [1] to try to > >> > achieve that. We can also re-enable V4 on-disk format in our kernel > >> > builds as a workaround. > >> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > >> > >> This should be fixed now. Yuri, please schedule a rerun for krbd. > >> > >> Thanks, > >> > >> Ilya > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > > Dev mailing list -- dev(a)ceph.io > > To unsubscribe send an email to dev-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Regression for kernel mounts with "read_from_replica=localize"
by Gregory Farnum 05 Feb '26

05 Feb '26

The Ceph project has discovered a regression with the tentacle release for kernel CephFS users who mount their filesystems using the "read_from_replica={localize|balance}" configuration option. Unless you are explicitly setting this option (which you generally shouldn't) or are using a system (such as Rook) to set up "stretch mode" and handle client mounts, you are not subject to this regression. Ceph-CSI users are subject to it if using the --enable-read-affinity flag. Due to cleanup changes in the OSD for tentacle, it is now dropping write operations with these flags set -- which the kernel client is sending on every operation, including writes, when it is set to use them for reads. You can see the initial bug report at https://tracker.ceph.com/issues/73997. The first point release of tentacle will fix this regression, but due to delays from the lab migration we do not currently have an ETA. The relevant PR is https://github.com/ceph/ceph/pull/66611. We are also adding additional tests to make sure we detect problems like this ahead of time in the future. https://tracker.ceph.com/issues/74190 The Rook report: https://github.com/rook/rook/issues/16839 Our apologies for this miss. -The Ceph team

1 0