- ceph-users - lists.ceph.io

Missing directories on download.ceph.com?
by Oliver Freyermuth 25 Sep '25

25 Sep '25

Dear Cephalopodians, it seems that recently some directories were emptied / became inaccessible on download.ceph.com, for example (some randomly selected ones): https://download.ceph.com/rpm-octopus https://download.ceph.com/rpm-15.2.16 As older releases such as: https://download.ceph.com/rpm-mimic/ are still there, I presume this was not intentional, so I am alerting here in case there might for example be a file system problem on the download mirror. Cheers, Oliver -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

1 0

Object name in Lua postrequest script
by Rok Jaklič 25 Sep '25

25 Sep '25

Hi, Is it possible to get an object name in Lua postrequest script when I am deleting an object? Request.Object.Name does not return anything probably because it's not "there" anymore. Kind regards, Rok

1 0

Re: tentacle 20.1.1 RC1 QE validation status
by Travis Nielsen 24 Sep '25

24 Sep '25

Rook tests look good for tentacle. Thanks, Travis On Wed, Sep 24, 2025 at 5:29 PM Yuri Weinstein <yweinste(a)redhat.com> wrote: > pls reply to all > > ---------- Forwarded message --------- > From: Yuri Weinstein <yweinste(a)redhat.com> > Date: Mon, Sep 22, 2025 at 7:45 AM > Subject: tentacle 20.1.1 RC1 QE validation status > To: dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> > > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/73123#note-1 > > Release Notes - TBD > LRC upgrade - TBD > > Dev Leads, please review the list of suites for completeness, as this is a > new release. > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > nvmeof: Aviv > > upgrade/squid-x: Laura > upgrade/reef-x: Laura > > upgrade/telemetry-upgrade: PASSED > > upgrade/tentacle-p2p: N/A > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > crimson-rados: PASSED > powercycle: PASSED > > TIA >

1 0

CephFS shares with multiple access rights: Use subvolumes or multiple filesystems?
by Sophonet 24 Sep '25

24 Sep '25

Hi list, for multiple project-level file shares (with individual access rights) I am planning to use CephFS. Technically this can be implemented both with multiple toplevel cephfs or with a single cephfs in the cluster and subvolumes. What is the preferred choice? I have not found any guidance in docs.ceph.com <http://docs.ceph.com/>. The only location that suggests to use subvolumes is https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=systems-cephfs-volumes…. However, how can I avoid that only one MDS is responsible for serving all subvolumes? Is there some current literature (books or web docs) that contain recommendations and examples? A couple of ceph-related books are available in well-known online book stores, but many of them are rather old (6 years or even more). Thanks a lot, Sophonet

2 5

[ceph-user][ceph-ansible] octopus repository removed from ceph
by Danish Khan 24 Sep '25

24 Sep '25

Dear Team, It seems the octopus repository is either removed or not working. https://download.ceph.com/debian-octopus Do we have any alternative or work around to install octopus packages? Regards, Danish

1 0

Alerts to a mail server?
by lejeczek 24 Sep '25

24 Sep '25

Hi guys. I'm trying to set 'alerts' and with _ssl_ I get: "ceph SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1147)" and without ssl: "ceph SMTP AUTH extension not supported by server." I'm browsing the net but failing to find a clear "fix" for it. Anybody care to share some ideas? Mail server/system - postfix - is otherwise serving a ok. many thanks, L.

1 0

Re: SQUID ganesha failures
by Patrick Bégou 24 Sep '25

24 Sep '25

Hi, nobody reach with this ganesha failure yet ? I've ran some additional configuration tests and discovered that the crashes of ganesha.nfsd only occurs when setting "--ingress-mode haproxy-protocol". Ganesha.nfsd is stable with "--ingress-mode keepalive-only" and "--ingress-mode haproxy-standard". The crashes with "--ingress-mode haproxy-protocol" occurs randomly on the servers, even if no client mount the NFS partition. Patrick Le 21/09/2025 à 17:27, Patrick Bégou a écrit : > Hi, > > just to provide some additional information, it is the ganesha.nfsd > process which core-dumps : > > sept. 19 17:06:12 whitaker02-ceph > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn[85697]: > 19/09/2025 15:06:12 : epoch 68cd5fb6 : whitaker02-ceph : > ganesha.nfsd-2[svc_8] rpc :TIRPC :EVENT :svc_vc_recv: 0x7f99c400a6b0 > fd 50 proxy header rest len failed header rlen = % (will set dead) > sept. 19 17:06:12 whitaker02-ceph > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn[85697]: > 19/09/2025 15:06:12 : epoch 68cd5fb6 : whitaker02-ceph : > ganesha.nfsd-2[reaper] nfs_try_lift_grace :STATE :EVENT :check > grace:reclaim complete(0) clid count(0) > sept. 19 17:06:12 whitaker02-ceph > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn[85697]: > 19/09/2025 15:06:12 : epoch 68cd5fb6 : whitaker02-ceph : > ganesha.nfsd-2[svc_10] rpc :TIRPC :EVENT :svc_vc_recv: 0x7f99b0004060 > fd 50 proxy ignored for local > sept. 19 17:06:13 whitaker02-ceph systemd-coredump[120951]: Process > 85701 (ganesha.nfsd) of user 0 dumped core. > > Stack > trace of thread 77: > #0 > 0x00007f9a939ad32e n/a (/usr/lib64/libntirpc.so.5.8 + 0x2232e) > ELF object > binary architecture: AMD x86-64 > sept. 19 17:06:14 whitaker02-ceph podman[120958]: 2025-09-19 > 17:06:14.123651716 +0200 CEST m=+0.029310490 container died > 5cb71f9d8b8986292dd8916ad0f610b9c12f5c08a1de505cd6f7a11137446263 > (image=quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6, > name=ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn, > org.opencontainers.image.authors=Ceph Release Team > <ceph-maintainers(a)ceph.io>, > CEPH_SHA1=0eceb0defba60152a8182f7bd87d164b639885b8, > org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, > org.label-schema.build-date=20250303, io.buildah.version=1.39.3, > CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=squid, > FROM_IMAGE=quay.io/centos/centos:stream9, > org.opencontainers.image.documentation=https://docs.ceph.com/, > org.label-schema.license=GPLv2, > GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, > org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) > sept. 19 17:06:14 whitaker02-ceph podman[120958]: 2025-09-19 > 17:06:14.139752858 +0200 CEST m=+0.045411632 container remove > 5cb71f9d8b8986292dd8916ad0f610b9c12f5c08a1de505cd6f7a11137446263 > (image=quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6, > name=ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn, > FROM_IMAGE=quay.io/centos/centos:stream9, > org.label-schema.vendor=CentOS, > CEPH_SHA1=0eceb0defba60152a8182f7bd87d164b639885b8, CEPH_REF=squid, > org.opencontainers.image.documentation=https://docs.ceph.com/, > org.label-schema.name=CentOS Stream 9 Base Image, > org.label-schema.license=GPLv2, > CEPH_GIT_REPO=https://github.com/ceph/ceph.git, > GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, > OSD_FLAVOR=default, org.label-schema.build-date=20250303, > org.opencontainers.image.authors=Ceph Release Team > <ceph-maintainers(a)ceph.io>, io.buildah.version=1.39.3, > org.label-schema.schema-version=1.0) > sept. 19 17:06:14 whitaker02-ceph systemd[1]: > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10(a)nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn.service: > Main process exited, code=exited, status=139/n/a > sept. 19 17:06:14 whitaker02-ceph systemd[1]: > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10(a)nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn.service: > Failed with result 'exit-code'. > sept. 19 17:06:14 whitaker02-ceph systemd[1]: > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10(a)nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn.service: > Consumed 3.485s CPU time. > sept. 19 17:06:24 whitaker02-ceph systemd[1]: > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10(a)nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn.service: > Scheduled restart job, restart counter is at 8. > > Patrick >

1 0

Why does recovering objects take much longer than the outage that caused them?
by Niklas Hambüchen 23 Sep '25

23 Sep '25

I noticed that for my clusters, even a short 5-minute network outage or single-host reboot can cause pgs: 5586988/366684639 objects misplaced (1.524%) which at the speed of recovery: 2.2 GiB/s, 676 objects/s can take hours to recover. I don't understand how this can be. If it's down for so short, how can rebalancing can take this long? I'm using Ceph 19.2.2 on HDDs with SSDs as BlueStore "db" device. Is this perhaps that writes of new files are written linearly to HDD (fast) but recovery seeks around on my HDDs in random order (slow)? In any case, this asymmetry is quite annoying. Could anything be done against it? Thanks! Niklas

6 9

radosgw-admin usage show reports doubled totals for some uids. [reef]
by 游峻權 23 Sep '25

23 Sep '25

Hi, We are running Ceph Reef 18.2.7, and we noticed an inconsistency when checking RGW usage statistics with radosgw-admin usage show. Cluster environment: - 3 MONs, 36 OSDs (HDD + SSD mixed) - RGW daemons: 3 - Deployment method: cephadm When querying a specific UID, the totals look correct: # radosgw-admin usage show --uid 0249bd92a6804133bf8e81f41da928cb | grep total -A5 "total": { "bytes_sent": 468098, "bytes_received": 35998026, "ops": 80, "successful_ops": 77 } But when dumping all usage and filtering for the same UID, the totals are exactly doubled: # radosgw-admin usage show | grep 0249bd92a6804133bf8e81f41da928cb -A100 | grep total -A5 "total": { "bytes_sent": 936196, "bytes_received": 71996052, "ops": 160, "successful_ops": 154 } Interestingly, if we specify a date filter, the numbers return to normal again: # radosgw-admin usage show --date=2025-09-24 | grep 0249bd92a6804133bf8e81f41da928cb -A100 | grep total -A5 "total": { "bytes_sent": 468098, "bytes_received": 35998026, "ops": 80, "successful_ops": 77 } This issue does not affect all UIDs, only some of them. Why does radosgw-admin usage show report doubled totals for certain UIDs when dumping all usage, while --uid or --date shows correct values? Is this expected behavior or a bug? Thanks, Jack

1 0

Recheck for "large omap objects" warning
by Jan Kasprzak 23 Sep '25

23 Sep '25

Hello, Ceph users, on my Ceph cluster with my custom rgw-like front-end client storing metadata to omap I got "1 large omap objects" warning. According to cluster log, the large object is osd.39 (osd.39) 2292 : cluster [WRN] Large omap object found. Object: 76:0bfe63c7:::idx.00000000004:head PG: 76.e3c67fd0 (76.10) Key count: 259173 Size (bytes): 12440304 I have increased the limit using ceph config set osd osd_deep_scrub_large_omap_object_key_threshold 1000000 but the warning still remains. How can I get rid of the warning after increasing the size limit? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2