- ceph-users - lists.ceph.io

Re: Repeated crashes of mgr module in Tentacle
by Eugen Block 10 Feb '26

10 Feb '26

Hi, > even “ceph -s” was hanging that command is contacting the MONs, are you sure the cluster is healthy? If ' ceph -s' hangs, I suspect either a network or a MON (quorum) issue. The default ceph uid/gid has always been 167 since I've been working with Ceph, so more than 10 years now, and it's the same within containers. And yes, there is a difference between cephadm setup and package based installs. Did you ensure that you don't have any other ceph packages installed except cephadm and maybe ceph-common? There can be issues when both deb packages and containers are running for the same daemon. So if only the MGR is affected, I recommend to ensure that no ceph-mgr package is installed. It looks like the telemetry module is responsible for the crash, can you turn that off? In latest releases you can force to turn off a module: ceph mgr module force disable telemetry Can you check if that doesn't crash the MGR? Also some debug (debug_mgr) logs might be helpful for the devs. I don't find an existing tracker issue for this, I'd suggest to create a new one. Regards, Eugen Am So., 8. Feb. 2026 um 14:46 Uhr schrieb Daniel Brown via ceph-users < ceph-users(a)ceph.io>: > > > Greetings — > > Have been seeing repeated crashes on my mgr module. Seems to run for about > 45 to 50 seconds and then boom. Cephadm setup here. Did (try to) enable a > couple modules lately. iostat, stats diskprediction_local, but have toggled > them back off - unfortunately it hasn’t fixed the issue. > > I was seeing “no mgr” with ceph -s — to get around that I’ve tweaked the > “StartLimitInterval” setting in the ceph-[CLUSTER-UID]@.service file down > to 1m, and have 4x mgr’s setup so that I can get a couple commands run > before mgr crashes and another starts. The 30m default there seems… high, > imo - I was having intervals with no mgr which makes it tough to do much > with the cluster - even “ceph -s” was hanging. Everything else in the > cluster seems “normal” - still serving data. > > > One other note — which I think is generally unrelated — I did upgrade one > of my cluster nodes from “Plucky Puffin" (25.04) ubuntu, to “Questing > Quokka” (25.10) ubuntu. After the upgrade, cephadm managed containers > didn’t want to start. I tracked that down to having the ceph user userid in > /etc/passwd set at 64045, but the container seeming to want userid 167. > Most things under /var/lib/ceph/[CLUSTER UID]/ … appear to be owned by > user/group 167:167 — I assume this is a default inside the container. > Workaround here was to manually change the UID/GID for ceph in /etc/passwd > and /etc/group. I’m going to imagine this is some collision between cephadm > managed deployments, and how Ubuntu / apt installs cephadm. > > > > The aforementioned mgr Crashes look like: > > > { > "assert_condition": "cursor != root", > "assert_file": > "/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h", > "assert_func": "virtual void PyFormatter::close_section()", > "assert_line": 84, > "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h: > In function 'virtual void PyFormatter::close_section()' thread ffff34e55700 > time > 2026-02-08T13:10:07.526894+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h: > 84: FAILED ceph_assert(cursor != root)\n", > "assert_thread_name": "telemetry", > "backtrace": [ > "__kernel_rt_sigreturn()", > "/lib64/libc.so.6(+0x82a78) [0xffff83603a78]", > "raise()", > "abort()", > "(ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x190) [0xffff84039874]", > "/usr/bin/ceph-mgr(+0xcf540) [0xaaaacdf2f540]", > > "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&)+0xf6c) [0xaaaacdf46ec0]", > "/usr/bin/ceph-mgr(+0x105528) [0xaaaacdf65528]", > "/lib64/libpython3.9.so.1.0(+0xf7bfc) [0xffff84ae9bfc]", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xda3f0) [0xffff84acc3f0]", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xda3f0) [0xffff84acc3f0]", > "/lib64/libpython3.9.so.1.0(+0xea93c) [0xffff84adc93c]", > "/lib64/libpython3.9.so.1.0(+0xcf304) [0xffff84ac1304]", > "/lib64/libpython3.9.so.1.0(+0x197d78) [0xffff84b89d78]", > "_PyObject_CallMethod_SizeT()", > "(PyModuleRunner::serve()+0x6c) [0xaaaacdfdf6cc]", > "(PyModuleRunner::PyModuleRunnerThread::entry()+0x148) > [0xaaaacdfdff08]" > ], > "ceph_version": "20.2.0", > "crash_id": > "2026-02-08T13:10:07.528739Z_f18e6b74-438b-47db-9438-5a3861fdef2d", > "entity_name": "mgr.hc-945901a5cad1b6e3.mtijbv", > "os_id": "centos", > "os_name": "CentOS Stream", > "os_version": "9", > "os_version_id": "9", > "process_name": "ceph-mgr", > "stack_sig": > "319d76a0d71a4644f9d65f592f6e621cca918d9205fc759ab7acf6944bc77fdd", > "timestamp": "2026-02-08T13:10:07.528739Z", > "utsname_hostname": "hc-945901a5cad1b6e3", > "utsname_machine": "aarch64", > "utsname_release": "6.14.0-1010-raspi", > "utsname_sysname": "Linux", > "utsname_version": "#10-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 15 19:09:05 > UTC 2025" > } > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: MGR Restful not accessible when doing mgr failover (squid v19.2.3)
by Eugen Block 10 Feb '26

10 Feb '26

Hi, did you configure the additional MGR(s) as well with server_addr, server_port and certs [0]? I haven't used the restful module before, but it works for me on Squid after creating a self-signed cert and a user/key. Then I used the existing crt (it's a test environment) and set the config-key for the other mgr as well: # get cert and key ceph config-key get mgr/restful/nautilus/crt > nautilus-restful.crt ceph config-key get mgr/restful/nautilus/key > nautilus-restful.key # set cert and key ceph config-key set mgr/restful/nautilus2/crt -i nautilus-restful.crt ceph config-key set mgr/restful/nautilus2/key -i nautilus-restful.key # configure per host server_addr and server_port ceph config set mgr mgr/restful/nautilus/server_addr 192.168.168.111 ceph config set mgr mgr/restful/nautilus/server_port 8003 ceph config set mgr mgr/restful/nautilus2/server_addr 192.168.168.112 ceph config set mgr mgr/restful/nautilus2/server_port 8003 # restart restful if necessary ceph restful restart ceph mgr services | grep rest "restful": "https://192.168.168.111:8003/" # after failover ceph mgr services | grep rest "restful": "https://192.168.168.112:8003/" Hope this helps! Eugen [0] https://docs.ceph.com/en/squid/mgr/restful/#configuring-ip-and-port Am So., 8. Feb. 2026 um 19:43 Uhr schrieb Andreas Eriksson via ceph-users < ceph-users(a)ceph.io>: > Hi, > > I'm doing cluster monitoring using the restful api on the ceph-mgr on > port 8003. > The requests are working fine as long as I have my *mgr-0* as primary. > Doing a `ceph mgr fail` so that the *mgr-1* becomes active, the mgr > daemon doesn't listen on port 8003. > > Is there any workaround for this? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Missing changelog in Tentacle release notes?
by Eugen Block 10 Feb '26

10 Feb '26

Hi, I was wondering why the Tentacle release notes [0] don't have a changelog section like the previous releases do (Reef [1], Squid [2]). To prepare an upgrade I usually scan the changelog for anything interesting/important. Has that process changed from v20 in general and there won't be any detailed changelogs anymore or was this an oversight? The highlights are great to get an overview, but I kind of miss the details. I'd appreciate any insghts. Thanks! Eugen [0] https://docs.ceph.com/en/latest/releases/tentacle/ [1] https://docs.ceph.com/en/latest/releases/squid/#changelog [2] https://docs.ceph.com/en/latest/releases/reef/#changelog

1 0

RGW period map issues and zone name collision
by Greg O'Neill 10 Feb '26

10 Feb '26

Hey folks, I'm having an issue with name collision of rgw zones across different realms in three multi-site clusters. Clusters are setup with bi-directional multi-site peering in addition to a local realm in each cluster for container images and thanos metrics. All clusters are running Reef v18.2.2 The local realm's `default` zone from the primary cluster's multi-site realm `cloud` was added to the period map by mistake and it propogated to the replica clusters. The zone name collided with the existing `default` zone in the `default` realm. I see the default zone in both pcloud and default realms with the same UUID. When I run `radosgw-admin period get --rgw-realm default` on the replica clusters, it returns with no zones or zonegroups when it should have one each named `default`. In a test cluster, I tried renaming or deleting the `default` zone from the multi-site realm but it knocks out any zones named `default` regardless of realm. ### Primary cluster (west coast) goneill@us-west-2-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm default | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 37929bb2-2eb3-4827-952a-6b5ca05627a6 default http://us-west-2-ceph-mon1:8080 goneill@us-west-2-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm savagebucket | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 250a2a20-5e11-486b-bae4-e29b173d2556 thanos 107d29a0-b732-4bf1-a26e-1f64f820e839 us-east-1-prod http://us-east-1-ceph-rgw1:8080 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 us-west-2-corp http://us-west-2-ceph-rgw1:8080 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 us-west-1-prod http://us-west-1-ceph-rgw1:8080 37929bb2-2eb3-4827-952a-6b5ca05627a6 default ### Replica cluster (west coast) goneill@us-west-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm default | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' goneill@us-west-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm savagebucket | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 250a2a20-5e11-486b-bae4-e29b173d2556 thanos 107d29a0-b732-4bf1-a26e-1f64f820e839 us-east-1-prod http://us-east-1-ceph-rgw1:8080 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 us-west-2-corp http://us-west-2-ceph-rgw1:8080 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 us-west-1-prod http://us-west-1-ceph-rgw1:8080 37929bb2-2eb3-4827-952a-6b5ca05627a6 default ### Replica cluster (east coast) goneill@us-east-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm default | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' goneill@us-east-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm savagebucket | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 250a2a20-5e11-486b-bae4-e29b173d2556 thanos 107d29a0-b732-4bf1-a26e-1f64f820e839 us-east-1-prod http://us-east-1-ceph-rgw1:8080 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 us-west-2-corp http://us-west-2-ceph-rgw1:8080 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 us-west-1-prod http://us-west-1-ceph-rgw1:8080 37929bb2-2eb3-4827-952a-6b5ca05627a6 default

1 1

Re: reef 18.2.8 QE validation status
by Brad Hubbard 10 Feb '26

10 Feb '26

On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < ceph-users(a)ceph.io> wrote: > On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < > ceph-users(a)ceph.io> wrote: > > > We are continuing on this release. > > > > > > powercycle - FAILED Brad, approved? > > > > Unfortunately, I don't see any logs anywhere where I have access to them so > I have no way of checking this? > OK, I have the latest logs. The three failing jobs appear to have all timed out after running for about six hours. At this time I don't know why but I continue to investigate. > > > > ceph-volume - Approved Guillaume? > > > > Patrick was reviewing some tests, please see his comments in the same > > tracker. > > TIA > > > > > > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> > > > wrote: > > > > > > > > This test branch was built loner then 2.5 weeks ago > > > > I have to rebuild it first and then will schedule krbd > > > > > > krbd approved based on > > > > > > > > > > > > https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… > > > > > > > > > https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… > > > > > > Thanks, > > > > > > Ilya > > > > > > > > > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > > > ceph-users(a)ceph.io> wrote: > > > >> > > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> > > wrote: > > > >> > > > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > > > >> > <ceph-users(a)ceph.io> wrote: > > > >> > > > > > >> > > Hi Yuri, > > > >> > > > > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < > > yweinste(a)redhat.com> > > > wrote: > > > >> > > > > > > >> > > > Note: this is an attempt to test and publish post EOL reef > > point > > > release > > > >> > > > > > > >> > > > Details of this release are summarized here: > > > >> > > > > > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > > > >> > > > Release Notes - TBD > > > >> > > > LRC upgrade - N/A > > > >> > > > > > > >> > > > distro => jammy focal centos9 windows bookworm > > > >> > > > > > > >> > > > Seeking approvals/reviews for: > > > >> > > > > > > >> > > > smoke - Laura approved? > > > >> > > > > > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > > > approved? > > > >> > > > (rados is also running on build 2 == build 1 + > > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > >> > > > > > > >> > > > rgw - Adam E approved? > > > >> > > > > > > >> > > > fs - Venky approved? > > > >> > > > > > >> > > There are lot many failures in the fs suite resembling > > > >> > > > > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: > wrong > > fs > > > >> > > type, bad option ... > > > >> > > > > > >> > > which is an issue with the testing kernel. Adding Alex who was > > > looking > > > >> > > into this issue. > > > >> > > > > > >> > > As far as the run validation is concerned, it's risky to ignore > > the > > > >> > > large number of failures. We have two options: > > > >> > > > > > >> > > - fix the testing kernel issue and rerun > > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the > > > >> > > teuthology-suite schedule command) > > > >> > > > > > >> > > The latter would be using an older kernel branch, but I think it > > > >> > > should suffice for this case. > > > >> > > > > >> > Hi Venky, > > > >> > > > > >> > With your report of the fs suite being affected as well, I carved > > out > > > >> > some time to take a look. > > > >> > > > > >> > I think it's happening because XFS maintainers disabled support > for > > V4 > > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel > > it's > > > >> > no longer compiled in by default. The proper fix would be to > create > > > an > > > >> > image with/move to V5 on-disk format; I have opened [1] to try to > > > >> > achieve that. We can also re-enable V4 on-disk format in our > kernel > > > >> > builds as a workaround. > > > >> > > > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > > >> > > > >> This should be fixed now. Yuri, please schedule a rerun for krbd. > > > >> > > > >> Thanks, > > > >> > > > >> Ilya > > > >> _______________________________________________ > > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > _______________________________________________ > > > > Dev mailing list -- dev(a)ceph.io > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > -- > Cheers, > Brad > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Cheers, Brad

1 1

tentacle and libvirt
by Piotr Pisz 09 Feb '26

09 Feb '26

Hello, We upgraded Ceph to Tentacle, and everything went fine, but all hosts running Ubuntu 22 and 24 lost connectivity to RBD at the libvirt level. What has changed in the tentacle? Best regards, Piotr

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 09 Feb '26

09 Feb '26

Still waiting for dev leads review/approval reply. On Fri, Feb 6, 2026 at 6:48 AM Ilya Dryomov via Dev <dev(a)ceph.io> wrote: > On Thu, Feb 5, 2026 at 10:59 PM Yuri Weinstein via Dev <dev(a)ceph.io> > wrote: > > > > We are continuing on this release. > > > > In the tracker https://tracker.ceph.com/issues/73906 please see runs > with the comment "on Build 5". > > This build and runs were done after the lab move. > > > > Seeking approvals for: > > > > smoke - FAILED, was reviewed and approved by Laura and Patrick, see > Slack thread for details > > rados - FAILED, Afreen, Travis, Laura, Radek approved? > > > > rgw - FAILED, Adam E approved? > > > > rbd, krbd - FAILED, Ilya approved? > > krbd approved. > > For rbd, I did a rerun [1]. The qemu-iotests.sh failure is known and > can be ignored, but the crashes in test_librbd_python.sh are concerning. > Patrick and I discussed them briefly -- they seem to be specific to > 20.04, but I don't remember seeing them before. There were multiple > passes on 20.04 in the run that I approved last year [2], so it looks > like something around Python, Cython or the build process has changed > in the interim. > > [1] > https://pulpito.ceph.com/dis-2026-02-02_19:37:16-rbd-reef-release-distro-de… > [2] > https://pulpito.ceph.com/yuriw-2025-11-18_23:00:18-rbd-reef-release-distro-… > > Thanks, > > Ilya > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

1 0

Debian Packages for Trixie
by Andrew 09 Feb '26

09 Feb '26

Hello Team, I can see that there are official packages for Debian Old-Stable (Bookworm) available on the official site: https://download.ceph.com/debian-19.2.3/dists/ Does anyone know if there is an approximate (or likely) ETA for the Debian Packages to be released for current stable version (Trixie)? Many thanks, Andrew

6 6

RGW/multisite: Segmentation fault during full sync
by Vahideh Alinouri 09 Feb '26

09 Feb '26

Hi folks, I have one realm, one zonegroup, and four zones, all running version 19.2.3. One zone was recently added to the zonegroup while it was on version 18.2.7. The newly added zone can perform data sync with the other secondary zones without issues. However, when syncing with the master zone, it gets stuck in the init state. The master zone, on the other hand, can sync with it successfully. All zones were recently upgraded from versions 18.2.4 and 18.2.7 to 19.2.3 but the problem still persists. sync status on master zone (dc07): radosgw-admin sync status realm 710cf69b-7382-47d2-aca6-03d991b00d1f (s3-cdn) zonegroup 7c01d60f-88c6-4192-baf7-d725260bf05d (s3-cdn-group) zone 03f6a8ec-008c-4cbf-8efc-d70a6013066f (s3-cdn-dc07) current time 2026-02-09T08:57:47Z zonegroup features enabled: disabled: compress-encrypted, notification_v2, resharding metadata sync: no sync (zone is master) data sync source: 1a6e33b9-8ece-4b9c-a9a5-961fa97c42c8 (s3-cdn-dc05) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shard behind shards: [93] oldest incremental change not applied: 2026-02-09T08:57:44.779833+0000 [93] source: 367dbfe9-a5f8-4101-a271-9749f25ba09c *(s3-cdn-dc10)* *syncing* full sync: 0/128 shards incremental sync: 128/128 shards 1 shard is recovering recovering shards: [125] source: 40122a7c-e594-43b7-89bb-e7ada37991c5 (s3-cdn-dc06) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source sync status on problematic zone (dc10): radosgw-admin sync status realm 710cf69b-7382-47d2-aca6-03d991b00d1f (s3-cdn) zonegroup 7c01d60f-88c6-4192-baf7-d725260bf05d (s3-cdn-group) zone 367dbfe9-a5f8-4101-a271-9749f25ba09c (s3-cdn-dc10) current time 2026-02-09T09:00:17Z zonegroup features enabled: disabled: compress-encrypted, notification_v2, resharding metadata sync: syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 03f6a8ec-008c-4cbf-8efc-d70a6013066f *(s3-cdn-dc07)* *init* full sync: 128/128 shards full sync: 0 buckets to sync incremental sync: 0/128 shards data is behind on 128 shards behind shards: [0–127] source: 1a6e33b9-8ece-4b9c-a9a5-961fa97c42c8 (s3-cdn-dc05) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shard behind shards: [93] oldest incremental change not applied: 2026-02-09T08:59:44.783203+0000 [93] 1 shard is recovering recovering shards: [41] source: 40122a7c-e594-43b7-89bb-e7ada37991c5 (s3-cdn-dc06) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source I see the following logs in the RGW sync enabled service: Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xb4) [0x5602e312a6f4] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 5: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4d7) [0x5602e3598d97] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 6: /usr/bin/radosgw(+0x837248) [0x5602e32eb248] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 7: (RGWRadosThread::Worker::entry()+0xbd) [0x5602e32ec83d] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 8: /lib64/libc.so.6(+0x8a4da) [0x7fbd2b2394da] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 9: clone() Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: debug -9999> 2026-02-09T09:05:46.708+0000 7fbbf4c19640 -1 *** Caught signal (Segmentation fault) ** Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: in thread 7fbbf4c19640 thread_name:data-sync Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 1: /lib64/libc.so.6(+0x3ebf0) [0x7fbd2b1edbf0] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 2: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x37) [0x5602e3127c97] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 3: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x481) [0x5602e3129901] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xb4) [0x5602e312a6f4] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 5: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4d7) [0x5602e3598d97] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 6: /usr/bin/radosgw(+0x837248) [0x5602e32eb248] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 7: (RGWRadosThread::Worker::entry()+0xbd) [0x5602e32ec83d] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 8: /lib64/libc.so.6(+0x8a4da) [0x7fbd2b2394da] Feb 09 09:05:46 ceph-mon10001.dc10.maas.etraveli.io bash[2404887]: 9: clone() Additionally, when I restart the RGW sync enabled service on the newly added zone, the daemon crashes. If I restart the RGW sync enabled daemon directly, it also crashes. To restart RGW sync enabled successfully, I have to run: ceph config set client.rgw.s3-cdn-colocate.ceph-mon10001.rrbzdg rgw_run_sync_thread false ceph orch daemon restart rgw.s3-cdn-colocate.ceph-mon10001.rrbzdg ceph config set client.rgw.s3-cdn-colocate.ceph-mon10001.rrbzdg rgw_run_sync_thread true I saw that this bug (https://tracker.ceph.com/issues/63378) was resolved and backported to 19.2.3. However, I am still observing the same behavior.

1 0

Re: Disable cache tier
by Kees Meijs | Nefos 09 Feb '26

09 Feb '26

Moin' and thanks! I'm going to take a further look at the objects within the pool and report back my findings. Cheers, Kees On 05/02/2026 20:36, Eugen Block via ceph-users wrote: > I've been searching the list for my own report from last year maybe, when > we finally got rid of our cache tier as well. I can't really find it > though, so I'm gonna summarize again how we managed to get out of it. > > After setting the cache mode to proxy and failing to flush/evict (also with > "no such file or directory" messages), I gradually reduced the cache pool > size from 260 GB in steps of 40 GB or so, not sure anymore .Suddenly Ceph > was able to flush and evict again. > I reduced the target_max_bytes to 1000000, only rbd_header objects remained > in the pool, but those couldn't be flushed/evicted anymore: > > failed to evict /rbd_header.133ba5e7513046: (16) Device or resource busy > rbd_header.8fbda012200854 > > Unfortunately, we didn't find a promsing way to get rid of them while the > images were in use. I had played around in a test cluster and ignored the > header objects when remocing the cache tier, and it seemed to work (since > the rbd_header objects should be identical in the backing pool as well), > but it didn't feel safe and we were really uncomfortable trying it in > production. So in the end, we shut down our VMs to be able to flush/evict > everything and safely got rid of the cache tier. > > I know this isn't really helpful, but maybe next time I find my comment on > the list. 🙂

1 0