- ceph-users - lists.ceph.io

mgmt-gateway and oauth2-proxy for radosgw?
by Florian Haas 12 Feb '26

12 Feb '26

Hi everyone, a question about the mgmt-gateway and oauth2-proxy services, because we want to incorporate one or both of them into our Ceph training course. I understand that as of Tentacle, mgmt-gateway exists as an administrative simplification for access to the Dashboard and the orchestrated monitoring stack,[0] and that it has OpenID Connect (OIDC) support via oauth2-proxy. The docs[1] assert that oauth2-proxy serves as "an advanced method for managing authentication and access control for Ceph applications". We've also had OIDC support in radosgw since Octopus (I believe), however it is tied to an implementation of a subset of STS[2], configuration is rather non-trivial[3], and as far as I understand it is presently not integrated with oauth2-proxy. Now my question is: will oauth2-proxy eventually be integrated with radosgw to replace and remove the STS dependency, or are the two OIDC integrations expected to coexist in parallel? Thanks! Cheers, Florian [0] https://docs.ceph.com/en/latest/cephadm/services/mgmt-gateway/ [1] https://docs.ceph.com/en/latest/cephadm/services/oauth2-proxy/ [2] https://docs.ceph.com/en/latest/radosgw/oidc/ [3] https://community.ibm.com/community/user/blogs/deepak-thorat/2024/03/20/cep…

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 12 Feb '26

12 Feb '26

Still seeking approvals for: (smoke - FAILED, was reviewed and approved by Laura and Patrick, see Slack thread for details) rados - FAILED, Afreen, Travis? ( Laura approved for core) rgw - FAILED, Adam E approved? (rbd, krbd - FAILED, Ilya approved) fs - FAILED, Venky approved? orch - FAILED, Adam K approved? upgrade/pacific-x (reef) - FAILED Laura, approved? upgrade/quincy-x (reef) - FAILED Laura, approved? upgrade/reef-p2p - FAILED Laura, approved? (powercycle - FAILED Brad, approved) ceph-volume - Approved Guillaume? On Wed, Feb 11, 2026 at 12:10 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > The rados suite is approved: > https://tracker.ceph.com/projects/rados/wiki/REEF#reef-v1828-httpstrackerce… > > On Tue, Feb 10, 2026 at 6:33 PM Brad Hubbard via Dev <dev(a)ceph.io> wrote: > >> Powercycle approved. >> >> On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <dev(a)ceph.io> >> wrote: >> >>> >>> >>> On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <bhubbard(a)redhat.com> >>> wrote: >>> >>>> >>>> >>>> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < >>>> ceph-users(a)ceph.io> wrote: >>>> >>>>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < >>>>> ceph-users(a)ceph.io> wrote: >>>>> >>>>> > We are continuing on this release. >>>>> > >>>>> > >>>>> > powercycle - FAILED Brad, approved? >>>>> > >>>>> >>>>> Unfortunately, I don't see any logs anywhere where I have access to >>>>> them so >>>>> I have no way of checking this? >>>>> >>>> >>>> OK, I have the latest logs. The three failing jobs appear to have all >>>> timed out after running for about six hours. At this time I don't know why >>>> but I continue to investigate. >>>> >>> >>> One point of commonality of the three tests that failed is that they are >>> all running tests using ceph fuse. I'll reach out to the FS guys to see if >>> they can help me get to the bottom of these failures. >>> >>> >>>> >>>> >>>>> >>>>> >>>>> > ceph-volume - Approved Guillaume? >>>>> > >>>>> > Patrick was reviewing some tests, please see his comments in the same >>>>> > tracker. >>>>> > TIA >>>>> > >>>>> > >>>>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < >>>>> > ceph-users(a)ceph.io> wrote: >>>>> > >>>>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev < >>>>> dev(a)ceph.io> >>>>> > > wrote: >>>>> > > > >>>>> > > > This test branch was built loner then 2.5 weeks ago >>>>> > > > I have to rebuild it first and then will schedule krbd >>>>> > > >>>>> > > krbd approved based on >>>>> > > >>>>> > > >>>>> > > >>>>> > >>>>> https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… >>>>> > > >>>>> > > >>>>> > >>>>> https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… >>>>> > > >>>>> > > Thanks, >>>>> > > >>>>> > > Ilya >>>>> > > >>>>> > > > >>>>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < >>>>> > > ceph-users(a)ceph.io> wrote: >>>>> > > >> >>>>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com >>>>> > >>>>> > wrote: >>>>> > > >> > >>>>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users >>>>> > > >> > <ceph-users(a)ceph.io> wrote: >>>>> > > >> > > >>>>> > > >> > > Hi Yuri, >>>>> > > >> > > >>>>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < >>>>> > yweinste(a)redhat.com> >>>>> > > wrote: >>>>> > > >> > > > >>>>> > > >> > > > Note: this is an attempt to test and publish post EOL >>>>> reef >>>>> > point >>>>> > > release >>>>> > > >> > > > >>>>> > > >> > > > Details of this release are summarized here: >>>>> > > >> > > > >>>>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 >>>>> > > >> > > > Release Notes - TBD >>>>> > > >> > > > LRC upgrade - N/A >>>>> > > >> > > > >>>>> > > >> > > > distro => jammy focal centos9 windows bookworm >>>>> > > >> > > > >>>>> > > >> > > > Seeking approvals/reviews for: >>>>> > > >> > > > >>>>> > > >> > > > smoke - Laura approved? >>>>> > > >> > > > >>>>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam >>>>> King >>>>> > > approved? >>>>> > > >> > > > (rados is also running on build 2 == build 1 + >>>>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's >>>>> request) >>>>> > > >> > > > >>>>> > > >> > > > rgw - Adam E approved? >>>>> > > >> > > > >>>>> > > >> > > > fs - Venky approved? >>>>> > > >> > > >>>>> > > >> > > There are lot many failures in the fs suite resembling >>>>> > > >> > > >>>>> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: >>>>> wrong >>>>> > fs >>>>> > > >> > > type, bad option ... >>>>> > > >> > > >>>>> > > >> > > which is an issue with the testing kernel. Adding Alex who >>>>> was >>>>> > > looking >>>>> > > >> > > into this issue. >>>>> > > >> > > >>>>> > > >> > > As far as the run validation is concerned, it's risky to >>>>> ignore >>>>> > the >>>>> > > >> > > large number of failures. We have two options: >>>>> > > >> > > >>>>> > > >> > > - fix the testing kernel issue and rerun >>>>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the >>>>> > > >> > > teuthology-suite schedule command) >>>>> > > >> > > >>>>> > > >> > > The latter would be using an older kernel branch, but I >>>>> think it >>>>> > > >> > > should suffice for this case. >>>>> > > >> > >>>>> > > >> > Hi Venky, >>>>> > > >> > >>>>> > > >> > With your report of the fs suite being affected as well, I >>>>> carved >>>>> > out >>>>> > > >> > some time to take a look. >>>>> > > >> > >>>>> > > >> > I think it's happening because XFS maintainers disabled >>>>> support for >>>>> > V4 >>>>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 >>>>> kernel >>>>> > it's >>>>> > > >> > no longer compiled in by default. The proper fix would be to >>>>> create >>>>> > > an >>>>> > > >> > image with/move to V5 on-disk format; I have opened [1] to >>>>> try to >>>>> > > >> > achieve that. We can also re-enable V4 on-disk format in our >>>>> kernel >>>>> > > >> > builds as a workaround. >>>>> > > >> > >>>>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 >>>>> > > >> >>>>> > > >> This should be fixed now. Yuri, please schedule a rerun for >>>>> krbd. >>>>> > > >> >>>>> > > >> Thanks, >>>>> > > >> >>>>> > > >> Ilya >>>>> > > >> _______________________________________________ >>>>> > > >> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> > > > >>>>> > > > _______________________________________________ >>>>> > > > Dev mailing list -- dev(a)ceph.io >>>>> > > > To unsubscribe send an email to dev-leave(a)ceph.io >>>>> > > _______________________________________________ >>>>> > > ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> > > >>>>> > _______________________________________________ >>>>> > ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> > >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> Brad >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> Brad >>>> >>> >>> >>> -- >>> Cheers, >>> Brad >>> _______________________________________________ >>> Dev mailing list -- dev(a)ceph.io >>> To unsubscribe send an email to dev-leave(a)ceph.io >>> >> >> >> -- >> Cheers, >> Brad >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >> > > > -- > > Laura Flores > > She/Her/Hers > > Software Engineer, Ceph Storage <https://ceph.io> > > Chicago, IL > > lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> > M: +17087388804 > > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

1 0

Re: How to disable an alertmanager alert
by Ankush Behl 12 Feb '26

12 Feb '26

Thanks for info, it would need a fix and we need to filter the suppressed alerts. On Thu, Feb 5, 2026 at 10:11 PM Chris Palmer via ceph-users < ceph-users(a)ceph.io> wrote: > Hi > > As a result of applying a Silence, they correctly appear on the > Observability\Alerts screen as "suppressed". > > > However they still appear on the home page Dashboard, in the top-centre > Status panel: > > Thanks, Chris > > > On 05/02/2026 16:10, Ankush Behl wrote: > > Hi Chris, do you mean they show up in overview dashboard? Can you show > > via some screenshots where all they are coming up? > > > > > > On Thu, 5 Feb 2026 at 6:39 PM, Chris Palmer via ceph-users > > <ceph-users(a)ceph.io> wrote: > > > > cephadm (squid & tentacle) deploys alertmanager with some default > > alerts. One is CephNodeInconsistentMTU which checks if the MTU of > > each > > interface with the same name is the same across the cluster. > > > > However if your cluster nodes are not identical, and have differing > > interface names, we end up with lots of permanent alerts. Although > > I can > > silence the alert (matching the alert name) for a long time, they > > still > > show up in the dashboard. > > > > The alerts are different permutations of: > > > > Node ceph3 has a different MTU size (1500) than the median of > > devices named enp4s0. > > > > Since that check is inappropriate in a heterogeneous cluster, > > what's the > > recommended way of disabling it, which won't get overwritten with > > redeployment of the monitoring stack? > > > > Thanks, Chris > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

[rgw multiple zonegroups] ERROR: S3 error: 404 (NoSuchKey) while copy
by Alexander NA 12 Feb '26

12 Feb '26

Hello everyone. I'm trying to set up one realm ("r1"), two zone groups ("zg1" - master, "zg2"), and one zone in each zone group ("z1" - master and "z2", respectively). The first zone group and its zone are masters. Endpoint "zg1" - 10.1.1.1 Endpoint "zg2" - 10.1.1.2 "zg1" has bucket "buc1" "zg2" has bucket "buc2" If I copy the file like this: s3cmd put test_file s3://buc1 --region=zg2 --host=10.1.1.2 I get the correct error code (as per the documentation): "ERROR: S3 error: 301 (Moved Permanently)" BUT, if I copy it like this: s3cmd put test_file s3://buc2 --region=zg1 --host=10.1.1.1 I get this code: ERROR: S3 error: 404 (NoSuchKey) Can you point me to something that might be wrong? Thanks

1 0

Re: Debian Packages for Trixie
by Albert Shih 12 Feb '26

12 Feb '26

Le 09/02/2026 à 16:16:30+0000, Stephane DUGRAVOT via ceph-users a écrit Hi, > ----- Le 13 Nov 25, à 17:32, Albert Shih <Albert.Shih(a)obspm.fr> a écrit : > > > Le 28/08/2025 à 18:52:05+1000, Andrew a écrit > > Hi, > > > > > I can see that there are official packages for Debian Old-Stable (Bookworm) > > > available on the official site: > > > https://download.ceph.com/debian-19.2.3/dists/ > > > > Does anyone know if there is an approximate (or likely) ETA for the Debian > > > Packages to be released for current stable version (Trixie)? > > > > Juste notice it's still not there.... > > > It would be super nice if the pkg for Debian 13 are available. > Just to notice, we installed 18.2.7 on Debian 13, then after upgrade with cephadm : > > ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.0 > > and 19.2.3, etc ... until 20.1.1. On a test plaform. > It might not be that easy to add nodes :-/ ;-) ;-) In fact the point I'm not very sure is what difference between the package ceph-common from Debian and the one from ceph.com. Because the ceph «himself» (at least with cephadm) are in container so should not have big link with the version of the OS. In other term what append if we install the ceph-common from Debian 13 (18.2.7) with cephadm 20.x.y (or whatever which version) in container. Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 12 févr. 2026 14:26:02 CET

1 0

Re: Data is still actively corrupted!
by dominik.baack 12 Feb '26

12 Feb '26

Hi, thanks for your reply. Mounting was done without the 'root_squash' option, here is the corresponding fstab entry: 192.168.251.2,192.168.251.3,192.168.251.4,192.168.251.5,192.168.251.6,192.168.251.7:/ /cephfs ceph name=gpu01,secretfile=/etc/ceph/gpu01.key,noatime,_netdev,recover_session=clean,read_from_replica=balance 0 0 Dominik Am 2026-02-12 09:51, schrieb goetze: > Hi ! > > Have you mounted cephfs with the 'root_squash' option set? If so, > remove that option. I may be wrong here, but as far as I know, this is > still considered unsafe and can lead to data corruption since the > necessary code changes have not yet made it into the mainstream linux > kernel. > > Carsten > ------------------------------------------------------------------ > Carsten Goetze > Computer Graphics tel: +49 531 391-2109 > TU Braunschweig fax: +49 531 391-2103 > Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de > D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze > >> Am 12.02.2026 um 07:15 schrieb Dominik Baack via ceph-users >> <ceph-users(a)ceph.io>: >> >> Hi, >> >> I noticed now that files are still actively corrupted / replaced by >> empty files when open saved. >> >> Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. >> Ceph servers are still running 20.2 Tentacle (deployed via cephadm) >> with an Ubuntu 24.04 host. >> >> Options currently set are to noout, norebalance, noscrub and >> nodeep-scrub >> >> I am currently setting up a read only mount to copy all existing >> data for a backup. >> >> I have currently no clue whats going on, I currently was able to >> observe this behavior only on the nodes. As those were upgraded as >> well (MLNX driver, nvidia-fs,...) can it be network? >> >> Any Idea how to recover from this? >> >> Cheers >> Dominik >> >> Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users: >> >>> Hi, >>> >>> after an controlled shutdown of the whole cluster do to external >>> circumstances we decided to update from 19.2 to 20.2 after the >>> restart. The system was health before and after the update. >>> The nodes mounting the filesystem were not equally lucky and were >>> partially shutdown hard. Storage was kept running additional >>> ~30min after node shutdown, all inflight operations should have >>> finished. >>> >>> Now we discover the some of the user files seem to be replaced >>> with zeros. For example: >>> >>> stat .gitignore >>> File: .gitignore >>> Size: 4429 Blocks: 9 IO Block: 4194304 >>> regular file >>> Device: 0,48 Inode: 1100241384598 Links: 1 >>> >>> hexdump -C .gitignore >>> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >>> |................| >>> * >>> 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| >>> 0000114d >>> >>> Scanning for files containing only zeros show several issues of >>> files that were likely accessed before or during the shutdown of >>> the nodes. >>> >>> How should I progress from here? >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: tentacle and libvirt
by Eugen Block 12 Feb '26

12 Feb '26

I upgraded my Ceph test cluster to Tentacle last night and OpenStack (Ubuntu 24) still works as before. So there must be something else going on with your cluster, I assume. Am Di., 10. Feb. 2026 um 00:06 Uhr schrieb Piotr Pisz via ceph-users < ceph-users(a)ceph.io>: > Hello, > > > > We upgraded Ceph to Tentacle, and everything went fine, but all hosts > running Ubuntu 22 and 24 lost connectivity to RBD at the libvirt level. > > What has changed in the tentacle? > > > > Best regards, > > Piotr > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Data is still actively corrupted!
by Carsten Götze 12 Feb '26

12 Feb '26

Hi ! Have you mounted cephfs with the 'root_squash' option set? If so, remove that option. I may be wrong here, but as far as I know, this is still considered unsafe and can lead to data corruption since the necessary code changes have not yet made it into the mainstream linux kernel. Carsten ------------------------------------------------------------------ Carsten Goetze Computer Graphics tel: +49 531 391-2109 TU Braunschweig fax: +49 531 391-2103 Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze > Am 12.02.2026 um 07:15 schrieb Dominik Baack via ceph-users <ceph-users(a)ceph.io>: > > Hi, > > I noticed now that files are still actively corrupted / replaced by empty files when open saved. > > Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. > Ceph servers are still running 20.2 Tentacle (deployed via cephadm) with an Ubuntu 24.04 host. > > Options currently set are to noout, norebalance, noscrub and nodeep-scrub > > I am currently setting up a read only mount to copy all existing data for a backup. > > > I have currently no clue whats going on, I currently was able to observe this behavior only on the nodes. As those were upgraded as well (MLNX driver, nvidia-fs,...) can it be network? > > Any Idea how to recover from this? > > Cheers > Dominik > > > Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users: >> Hi, >> >> after an controlled shutdown of the whole cluster do to external circumstances we decided to update from 19.2 to 20.2 after the restart. The system was health before and after the update. >> The nodes mounting the filesystem were not equally lucky and were partially shutdown hard. Storage was kept running additional ~30min after node shutdown, all inflight operations should have finished. >> >> Now we discover the some of the user files seem to be replaced with zeros. For example: >> >> stat .gitignore >> File: .gitignore >> Size: 4429 Blocks: 9 IO Block: 4194304 regular file >> Device: 0,48 Inode: 1100241384598 Links: 1 >> >> >> hexdump -C .gitignore >> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| >> * >> 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| >> 0000114d >> >> >> Scanning for files containing only zeros show several issues of files that were likely accessed before or during the shutdown of the nodes. >> >> How should I progress from here? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: tentacle and libvirt
by Alfonso Escribano Merino 12 Feb '26

12 Feb '26

1 0

Re: Data is still actively corrupted!
by Dominik Baack 12 Feb '26

12 Feb '26

Hi, I noticed now that files are still actively corrupted / replaced by empty files when open saved. Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. Ceph servers are still running 20.2 Tentacle (deployed via cephadm) with an Ubuntu 24.04 host. Options currently set are to noout, norebalance, noscrub and nodeep-scrub I am currently setting up a read only mount to copy all existing data for a backup. I have currently no clue whats going on, I currently was able to observe this behavior only on the nodes. As those were upgraded as well (MLNX driver, nvidia-fs,...) can it be network? Any Idea how to recover from this? Cheers Dominik Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users: > Hi, > > after an controlled shutdown of the whole cluster do to external > circumstances we decided to update from 19.2 to 20.2 after the > restart. The system was health before and after the update. > The nodes mounting the filesystem were not equally lucky and were > partially shutdown hard. Storage was kept running additional ~30min > after node shutdown, all inflight operations should have finished. > > Now we discover the some of the user files seem to be replaced with > zeros. For example: > > stat .gitignore > File: .gitignore > Size: 4429 Blocks: 9 IO Block: 4194304 regular file > Device: 0,48 Inode: 1100241384598 Links: 1 > > > hexdump -C .gitignore > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > * > 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| > 0000114d > > > Scanning for files containing only zeros show several issues of files > that were likely accessed before or during the shutdown of the nodes. > > How should I progress from here?

1 0