- ceph-users - lists.ceph.io

Re: Cephfs kernel client bug report
by Alain Bertrand 18 Mar '26

18 Mar '26

Slava, I just put the details in the bug report https://tracker.ceph.com/issues/75609 . The kernels I was able to reproduce the issue on were the proxmox supplied 6.17.13-1-pve, and 6.14.11-5-pve, and the Pop!_OS kernel version 6.18.7-76061807-generic. The problem didn't happen on a different machine running CachyOS kernel 6.19.6-2-cachyos. All I needed to do to reproduce the problem was go to the cephfs mountpoint (/mnt/ceph) and run find -maxdepth 3 -type d -exec ls "{}/.snap/" \; which should have listed snapshots from about 1015 directories. This wasn't the smartest way to find snapshots, it was just the first one I thought of. Each time the effect is a little different. Sometimes it immediately panics, sometimes the machine just gets slow for a few minutes with lots of errors being logged before it panics. Thanks, Alain On 2026-03-18 13:07, Viacheslav Dubeyko wrote: > Hi Alain, > > The ceph-devel(a)vger.kernel.org is the email list for sharing issues in CephFS > kernel client. > > I remember that we had the fix for likewise issue recently. Could you please > share which kernel version you have? Which particular call trace do you have for > this kernel panic? How anybody can reproduce the issue? > > Thanks, > Slava. > > On Wed, 2026-03-18 at 15:17 +0530, Venky Shankar wrote: >> cc Alex/Slava (cephfs kernel driver maintainers) >> >> On Wed, Mar 18, 2026 at 3:35 AM Alain Bertrand via ceph-users >> <ceph-users(a)ceph.io> wrote: >>> Hi. >>> >>> I'm new here. I have been playing with ceph for a few years (nothing >>> dramatic, just using it in my proxmox cluster to store VM images and >>> host a cephfs with a lot of shared files). I think I found a bug in the >>> cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a >>> lot of directories quickly causes a kernel panic. I tried to create an >>> account on the bug tracker at tracker.ceph.com a week ago, but still >>> can't log in there. Do I put details here, or report somewhere else? >>> What's the next step I should take? >>> >>> >>> Thanks, >>> >>> Alain >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

Re: Cephfs kernel client bug report
by Venky Shankar 18 Mar '26

18 Mar '26

cc Alex/Slava (cephfs kernel driver maintainers) On Wed, Mar 18, 2026 at 3:35 AM Alain Bertrand via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi. > > I'm new here. I have been playing with ceph for a few years (nothing > dramatic, just using it in my proxmox cluster to store VM images and > host a cephfs with a lot of shared files). I think I found a bug in the > cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a > lot of directories quickly causes a kernel panic. I tried to create an > account on the bug tracker at tracker.ceph.com a week ago, but still > can't log in there. Do I put details here, or report somewhere else? > What's the next step I should take? > > > Thanks, > > Alain > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Cheers, Venky

1 0

Restricting subuser to its bucket
by Rok Jaklič 18 Mar '26

18 Mar '26

Hi, I am trying to restrict subuser to its bucket (so he/she can work only on a bucket which has permission) with disabling ACLs and setting policy on a bucket: Using ceph version 19.2.2 bucket1-policy.json contains { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::tenant1:user/parent1:user1" }, "Action": "s3:*", "Resource": [ "arn:aws:s3:::bucket1", "arn:aws:s3:::bucket1/*" ] } ] } --- echo "lala" > test.txt export RGW_ENDPOINT="" radosgw-admin user create --tenant=tenant1 --uid=parent1 --key-type=s3 --display-name="tenant1:parent1" for sub in user1 user2 user3; do radosgw-admin subuser create --tenant=tenant1 --uid=parent1 --subuser=$sub --key-type=s3 --access=full; done export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user1").access_key') export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user1").secret_key') aws s3api create-bucket --bucket bucket1 --object-ownership BucketOwnerEnforced --endpoint-url $RGW_ENDPOINT aws s3api put-bucket-policy --bucket bucket1 --policy file://bucket1-policy.json --endpoint-url $RGW_ENDPOINT aws s3api put-public-access-block --bucket bucket1 --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true --endpoint-url $RGW_ENDPOINT aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url $RGW_ENDPOINT aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT # verify export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user2").access_key') export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user2").secret_key') aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT --- Script stops at aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url $RGW_ENDPOINT with an error like: upload failed: ./test.txt to s3://bucket1/test.txt argument of type 'NoneType' is not iterable If I comment the line with aws s3api put-public-access-block... I can successfully upload some file, but I can also see the contents of bucket1 with user2, which is not what I want. Any ideas how would I restrict the subuser to its bucket only? I was actually able to restrict subuser to its bucket by explicitly denying access to bucket with following policy: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/AFNERFMNSC… But it doesn't feel the right way. Wouldn't be better to "restrict" everything for everyone but its user in the first place and then allow others to access? Kind regards, Rok

1 0

Re: Cephfs kernel client bug report
by Wannes Smet 18 Mar '26

18 Mar '26

Hi Alain, You could talk to Anthony Middleton. Usually he can speed up your bug account approval request. In case you're active on Slack, you can find him there. Wannes ________________________________ From: MARTEL Arnaud via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, March 18, 2026 08:58 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Cephfs kernel client bug report Hi Alain, It seems to be related to this kernel bug: https: //urldefense. com/v3/__https: //bugzilla. kernel. org/show_bug. cgi?id=220807__;!!FtrhtPsWDhZ6tw!GXjnqm57m9uDNp3YBpMOJiB1tmvYHyMzZ0h2Fppd1HSPnGMLKlU3L3M--WSNMLIMhoDxCFX_qMTW7Rkc$[bugzilla[. ]kernel[. ]org] ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Alain, It seems to be related to this kernel bug: https://urldefense.com/v3/__https://bugzilla.kernel.org/show_bug.cgi?id=220… Upgrading your kernel should be a solution on Debian 13 (kernel 6.12.73-1 is OK) Arnaud Le 17/03/2026 23:07, « Alain Bertrand via ceph-users » <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> a écrit : Hi. I'm new here. I have been playing with ceph for a few years (nothing dramatic, just using it in my proxmox cluster to store VM images and host a cephfs with a lot of shared files). I think I found a bug in the cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a lot of directories quickly causes a kernel panic. I tried to create an account on the bug tracker at tracker.ceph.com a week ago, but still can't log in there. Do I put details here, or report somewhere else? What's the next step I should take? Thanks, Alain _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Cephfs kernel client bug report
by MARTEL Arnaud 18 Mar '26

18 Mar '26

Hi Alain, It seems to be related to this kernel bug: https://bugzilla.kernel.org/show_bug.cgi?id=220807 Upgrading your kernel should be a solution on Debian 13 (kernel 6.12.73-1 is OK) Arnaud Le 17/03/2026 23:07, « Alain Bertrand via ceph-users » <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> a écrit : Hi. I'm new here. I have been playing with ceph for a few years (nothing dramatic, just using it in my proxmox cluster to store VM images and host a cephfs with a lot of shared files). I think I found a bug in the cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a lot of directories quickly causes a kernel panic. I tried to create an account on the bug tracker at tracker.ceph.com a week ago, but still can't log in there. Do I put details here, or report somewhere else? What's the next step I should take? Thanks, Alain _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 0

Windows CEPH driver Re: Re: CSC minutes 2026-03-16
by Takeshi Nishimura 18 Mar '26

18 Mar '26

On Tue, Mar 17, 2026 at 7:46 PM Martin Wege via Dev <dev(a)ceph.io> wrote: > > On Mon, Mar 16, 2026 at 6:46 PM Takeshi Nishimura via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > On Mon, Mar 16, 2026 at 4:41 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > > > > > > 2026/03/16 > > > > > > [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe > > ... > > > Ceph on Windows discussion > > > Weekly meeting was set up on the Community Calendar on Tuesdays > > > Alexandre Jardon, Andy Herron involved > > > > > > Will come back to CSC next week to ask opinion on plan of action > > > Goal will be to improve understanding/direction around Windows > > > > Could you please consider getting Roland Mainz > > <roland.mainz(a)nrubsig.org> involved? > > > > Mr Mainz wrote ms-nfs41-client (link: > > https://lore.kernel.org/linux-nfs/CANH4o6M1J2FW1dgvA4JHz+EXzMsyguuAT_TfbUy5… > > and https://github.com/kofemann/ms-nfs41-client) and should have > > sufficient expertise to provide comments. > > ms-nfs41-client is a Windows NFSv4.2 filesystem client. How should it > be able to help with Ceph? I explained that already: 1. IMO Mr Mainz has the expertise to comment on the subject of Windows kernel filesystem drivers 2. This specific driver has a kernel part and a userland daemon, and can be used as a template for a Windows CEPH driver. Replace the NFS parts with CEPH parts. 3. You wrote in https://www.spinics.net/lists/dev-ceph/msg06898.html that one of the better choices for a CEPH Windows driver is a RDBSS driver. ms-nfs41-client uses RDBSS. -- Internationalization&localization dev / 大阪大学 Takeshi Nishimura <takeshi.nishimura.linux(a)gmail.com>

1 0

Re: Cephfs kernel client bug report
by John Mulligan 17 Mar '26

17 Mar '26

On Tuesday, March 17, 2026 6:09:29 PM Eastern Daylight Time Alain Bertrand via ceph-users wrote: > Hi. > > I'm new here. I have been playing with ceph for a few years (nothing > dramatic, just using it in my proxmox cluster to store VM images and > host a cephfs with a lot of shared files). I think I found a bug in the > cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a > lot of directories quickly causes a kernel panic. I tried to create an > account on the bug tracker at tracker.ceph.com a week ago, but still > can't log in there. Do I put details here, or report somewhere else? > What's the next step I should take? > Tracker is the right place. Tracker approval requires manual review and unfortunately that makes the sign-up process very slow. You can join ceph slack and ask for updates the #sepia channel - for now I will do that so you don't have to jump through more hoops right now.

1 0

Cephfs kernel client bug report
by Alain Bertrand 17 Mar '26

17 Mar '26

Hi. I'm new here. I have been playing with ceph for a few years (nothing dramatic, just using it in my proxmox cluster to store VM images and host a cephfs with a lot of shared files). I think I found a bug in the cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a lot of directories quickly causes a kernel panic. I tried to create an account on the bug tracker at tracker.ceph.com a week ago, but still can't log in there. Do I put details here, or report somewhere else? What's the next step I should take? Thanks, Alain

1 0

Re: CSC minutes 2026-03-16
by Martin Wege 17 Mar '26

17 Mar '26

On Mon, Mar 16, 2026 at 6:46 PM Takeshi Nishimura via ceph-users <ceph-users(a)ceph.io> wrote: > > On Mon, Mar 16, 2026 at 4:41 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > > > > 2026/03/16 > > > > [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe > ... > > Ceph on Windows discussion > > Weekly meeting was set up on the Community Calendar on Tuesdays > > Alexandre Jardon, Andy Herron involved > > > > Will come back to CSC next week to ask opinion on plan of action > > Goal will be to improve understanding/direction around Windows > > Could you please consider getting Roland Mainz > <roland.mainz(a)nrubsig.org> involved? > > Mr Mainz wrote ms-nfs41-client (link: > https://lore.kernel.org/linux-nfs/CANH4o6M1J2FW1dgvA4JHz+EXzMsyguuAT_TfbUy5… > and https://github.com/kofemann/ms-nfs41-client) and should have > sufficient expertise to provide comments. ms-nfs41-client is a Windows NFSv4.2 filesystem client. How should it be able to help with Ceph? Thanks, Martin

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 17 Mar '26

17 Mar '26

This point release is ready We need to finish reviewing and approving release notes and add notable changes. https://github.com/ceph/ceph/pull/67746 On Tue, Mar 10, 2026 at 8:53 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Patrick approved fs and we will continue with this release > > On Wed, Feb 25, 2026 at 7:43 AM Patrick Donnelly <pdonnell(a)redhat.com> > wrote: > > > The `fs` and `powercycle` suites will need rerun with [1]. I believe > > you're already doing that but I'm documenting that for the list. > > > > I'm going to look into the upgrade issues today [2]. > > > > [1] https://github.com/ceph/ceph/pull/67495 > > [2] https://tracker.ceph.com/issues/75034 > > > > On Mon, Feb 23, 2026 at 4:29 PM Yuri Weinstein via ceph-users > > <ceph-users(a)ceph.io> wrote: > > > > > > Patrick, > > > > > > The only outstanding issues are the FS approval and upgrade issues, > > > reported by Laura. > > > > > > What do you suggest we do next here? > > > > > > On Mon, Feb 23, 2026 at 8:54 AM Adam King via Dev <dev(a)ceph.io> wrote: > > > > > > > orch approved. Run looks ugly due to us no having a proper ignorelist > > for > > > > the orch suite on reef, but the actual results are okay > > > > > > > > On Sat, Nov 29, 2025 at 12:09 PM Yuri Weinstein via ceph-users < > > > > ceph-users(a)ceph.io> wrote: > > > > > > > >> Rerun scheduled for krbd > > > >> > > > >> Still waiting for replies from dev leads > > > >> > > > >> On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> > > wrote: > > > >> > > > >> > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein < > > yweinste(a)redhat.com> > > > >> > wrote: > > > >> > > > > > >> > > Note: this is an attempt to test and publish post EOL reef > point > > > >> release > > > >> > > > > > >> > > Details of this release are summarized here: > > > >> > > > > > >> > > https://tracker.ceph.com/issues/73906#note-1 > > > >> < > > > https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F… > > > > > > >> > > Release Notes - TBD > > > >> > > LRC upgrade - N/A > > > >> > > > > > >> > > distro => jammy focal centos9 windows bookworm > > > >> > > > > > >> > > Seeking approvals/reviews for: > > > >> > > > > > >> > > smoke - Laura approved? > > > >> > > > > > >> > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > > > >> approved? > > > >> > > (rados is also running on build 2 == build 1 + > > > >> > > https://github.com/ceph/ceph/pull/66056 > > > >> < > > > https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fgithub.com%2Fceph%2… > > > > > > >> per Igor's request) > > > >> > > > > > >> > > rgw - Adam E approved? > > > >> > > > > > >> > > fs - Venky approved? > > > >> > > > > > >> > > orch - Adam King approved? > > > >> > > > > > >> > > rbd - Ilya approved? > > > >> > > krbd - Ilya approved? > > > >> > > > > >> > rbd approved, krbd needs a rerun as all jobs failed while > > > >> > going through cephlab.yml playbook (not related to RBD): > > > >> > > > > >> > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > type, > > > >> > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing > > > >> > codepage or helper program, or other error.'] > > > >> > > > > >> > Thanks, > > > >> > > > > >> > Ilya > > > >> > > > > >> > > > > >> _______________________________________________ > > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > >> > > > > _______________________________________________ > > > > Dev mailing list -- dev(a)ceph.io > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > -- > > Patrick Donnelly, Ph.D. > > He / Him / His > > Red Hat Partner Engineer > > IBM, Inc. > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0