- ceph-users - lists.ceph.io

Merging two clusters
by Nico Schottelius 19 Mar '26

19 Mar '26

Hello, I was wondering if there is any (recent, in general) method to merge to clusters. We have several older nautilus based clusters that are natively deployed. We also have several newer rook/kubernetes/squid based clusters. In each data center we have both, alongside each other. While the clusters are running in separate networks, we could open the firewalls between them for a merge/migration. My question is, is there any reasonable way of merging an old nautilus cluster into another existing squid cluster? In theory, what would be amazing is something like a fluid process in which the newer cluster could retrieve items from the older cluster and once it has them, decommission them from the old cluster. The live migration on https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ seems to go a bit into the direction, but I am not sure if that works in our case. We are looking at migrating around 500 rbd images in total. Any hints appreciated! Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Recommended approach for hybrid devices
by Ryan Sleeth 19 Mar '26

19 Mar '26

I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes. My initial plan was to split the NVMes into 5x 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 HDDs creating hybrid devices). My hope was that for performance, the one NVMe data partition could then be used for 4x replicated SSD-only metadata pools. The hybrid devices would be for 4+2 EC cephfs. Previously I had tested this successfully on Squid using a series of manual device adds, but on Tentacle it is seeing pre-created LVM partitions as in use by the file system where Squid did not (ceph orch device ls ceph01). My understanding is a manual device add approach, while possible, removes the benefits of orch (creating "unmanaged" devices) and should be avoided. But would it even be possible for orch to be fed a yaml with the NVMes operating as both a data and db device? The obvious answer is to have a third NVMe per node operating as data for the 4x replicated metadata pool, but I am not sure if my budget permits that. -- Ryan Sleeth

1 0

Re: Cephfs kernel client bug report
by Alain Bertrand 18 Mar '26

18 Mar '26

Slava, I just put the details in the bug report https://tracker.ceph.com/issues/75609 . The kernels I was able to reproduce the issue on were the proxmox supplied 6.17.13-1-pve, and 6.14.11-5-pve, and the Pop!_OS kernel version 6.18.7-76061807-generic. The problem didn't happen on a different machine running CachyOS kernel 6.19.6-2-cachyos. All I needed to do to reproduce the problem was go to the cephfs mountpoint (/mnt/ceph) and run find -maxdepth 3 -type d -exec ls "{}/.snap/" \; which should have listed snapshots from about 1015 directories. This wasn't the smartest way to find snapshots, it was just the first one I thought of. Each time the effect is a little different. Sometimes it immediately panics, sometimes the machine just gets slow for a few minutes with lots of errors being logged before it panics. Thanks, Alain On 2026-03-18 13:07, Viacheslav Dubeyko wrote: > Hi Alain, > > The ceph-devel(a)vger.kernel.org is the email list for sharing issues in CephFS > kernel client. > > I remember that we had the fix for likewise issue recently. Could you please > share which kernel version you have? Which particular call trace do you have for > this kernel panic? How anybody can reproduce the issue? > > Thanks, > Slava. > > On Wed, 2026-03-18 at 15:17 +0530, Venky Shankar wrote: >> cc Alex/Slava (cephfs kernel driver maintainers) >> >> On Wed, Mar 18, 2026 at 3:35 AM Alain Bertrand via ceph-users >> <ceph-users(a)ceph.io> wrote: >>> Hi. >>> >>> I'm new here. I have been playing with ceph for a few years (nothing >>> dramatic, just using it in my proxmox cluster to store VM images and >>> host a cephfs with a lot of shared files). I think I found a bug in the >>> cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a >>> lot of directories quickly causes a kernel panic. I tried to create an >>> account on the bug tracker at tracker.ceph.com a week ago, but still >>> can't log in there. Do I put details here, or report somewhere else? >>> What's the next step I should take? >>> >>> >>> Thanks, >>> >>> Alain >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

Re: Cephfs kernel client bug report
by Venky Shankar 18 Mar '26

18 Mar '26

cc Alex/Slava (cephfs kernel driver maintainers) On Wed, Mar 18, 2026 at 3:35 AM Alain Bertrand via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi. > > I'm new here. I have been playing with ceph for a few years (nothing > dramatic, just using it in my proxmox cluster to store VM images and > host a cephfs with a lot of shared files). I think I found a bug in the > cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a > lot of directories quickly causes a kernel panic. I tried to create an > account on the bug tracker at tracker.ceph.com a week ago, but still > can't log in there. Do I put details here, or report somewhere else? > What's the next step I should take? > > > Thanks, > > Alain > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Cheers, Venky

1 0

Restricting subuser to its bucket
by Rok Jaklič 18 Mar '26

18 Mar '26

Hi, I am trying to restrict subuser to its bucket (so he/she can work only on a bucket which has permission) with disabling ACLs and setting policy on a bucket: Using ceph version 19.2.2 bucket1-policy.json contains { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::tenant1:user/parent1:user1" }, "Action": "s3:*", "Resource": [ "arn:aws:s3:::bucket1", "arn:aws:s3:::bucket1/*" ] } ] } --- echo "lala" > test.txt export RGW_ENDPOINT="" radosgw-admin user create --tenant=tenant1 --uid=parent1 --key-type=s3 --display-name="tenant1:parent1" for sub in user1 user2 user3; do radosgw-admin subuser create --tenant=tenant1 --uid=parent1 --subuser=$sub --key-type=s3 --access=full; done export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user1").access_key') export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user1").secret_key') aws s3api create-bucket --bucket bucket1 --object-ownership BucketOwnerEnforced --endpoint-url $RGW_ENDPOINT aws s3api put-bucket-policy --bucket bucket1 --policy file://bucket1-policy.json --endpoint-url $RGW_ENDPOINT aws s3api put-public-access-block --bucket bucket1 --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true --endpoint-url $RGW_ENDPOINT aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url $RGW_ENDPOINT aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT # verify export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user2").access_key') export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user2").secret_key') aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT --- Script stops at aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url $RGW_ENDPOINT with an error like: upload failed: ./test.txt to s3://bucket1/test.txt argument of type 'NoneType' is not iterable If I comment the line with aws s3api put-public-access-block... I can successfully upload some file, but I can also see the contents of bucket1 with user2, which is not what I want. Any ideas how would I restrict the subuser to its bucket only? I was actually able to restrict subuser to its bucket by explicitly denying access to bucket with following policy: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/AFNERFMNSC… But it doesn't feel the right way. Wouldn't be better to "restrict" everything for everyone but its user in the first place and then allow others to access? Kind regards, Rok

1 0

Re: Cephfs kernel client bug report
by Wannes Smet 18 Mar '26

18 Mar '26

Hi Alain, You could talk to Anthony Middleton. Usually he can speed up your bug account approval request. In case you're active on Slack, you can find him there. Wannes ________________________________ From: MARTEL Arnaud via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, March 18, 2026 08:58 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Cephfs kernel client bug report Hi Alain, It seems to be related to this kernel bug: https: //urldefense. com/v3/__https: //bugzilla. kernel. org/show_bug. cgi?id=220807__;!!FtrhtPsWDhZ6tw!GXjnqm57m9uDNp3YBpMOJiB1tmvYHyMzZ0h2Fppd1HSPnGMLKlU3L3M--WSNMLIMhoDxCFX_qMTW7Rkc$[bugzilla[. ]kernel[. ]org] ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Alain, It seems to be related to this kernel bug: https://urldefense.com/v3/__https://bugzilla.kernel.org/show_bug.cgi?id=220… Upgrading your kernel should be a solution on Debian 13 (kernel 6.12.73-1 is OK) Arnaud Le 17/03/2026 23:07, « Alain Bertrand via ceph-users » <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> a écrit : Hi. I'm new here. I have been playing with ceph for a few years (nothing dramatic, just using it in my proxmox cluster to store VM images and host a cephfs with a lot of shared files). I think I found a bug in the cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a lot of directories quickly causes a kernel panic. I tried to create an account on the bug tracker at tracker.ceph.com a week ago, but still can't log in there. Do I put details here, or report somewhere else? What's the next step I should take? Thanks, Alain _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Cephfs kernel client bug report
by MARTEL Arnaud 18 Mar '26

18 Mar '26

Hi Alain, It seems to be related to this kernel bug: https://bugzilla.kernel.org/show_bug.cgi?id=220807 Upgrading your kernel should be a solution on Debian 13 (kernel 6.12.73-1 is OK) Arnaud Le 17/03/2026 23:07, « Alain Bertrand via ceph-users » <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> a écrit : Hi. I'm new here. I have been playing with ceph for a few years (nothing dramatic, just using it in my proxmox cluster to store VM images and host a cephfs with a lot of shared files). I think I found a bug in the cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a lot of directories quickly causes a kernel panic. I tried to create an account on the bug tracker at tracker.ceph.com a week ago, but still can't log in there. Do I put details here, or report somewhere else? What's the next step I should take? Thanks, Alain _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 0

Windows CEPH driver Re: Re: CSC minutes 2026-03-16
by Takeshi Nishimura 17 Mar '26

17 Mar '26

On Tue, Mar 17, 2026 at 7:46 PM Martin Wege via Dev <dev(a)ceph.io> wrote: > > On Mon, Mar 16, 2026 at 6:46 PM Takeshi Nishimura via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > On Mon, Mar 16, 2026 at 4:41 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > > > > > > 2026/03/16 > > > > > > [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe > > ... > > > Ceph on Windows discussion > > > Weekly meeting was set up on the Community Calendar on Tuesdays > > > Alexandre Jardon, Andy Herron involved > > > > > > Will come back to CSC next week to ask opinion on plan of action > > > Goal will be to improve understanding/direction around Windows > > > > Could you please consider getting Roland Mainz > > <roland.mainz(a)nrubsig.org> involved? > > > > Mr Mainz wrote ms-nfs41-client (link: > > https://lore.kernel.org/linux-nfs/CANH4o6M1J2FW1dgvA4JHz+EXzMsyguuAT_TfbUy5… > > and https://github.com/kofemann/ms-nfs41-client) and should have > > sufficient expertise to provide comments. > > ms-nfs41-client is a Windows NFSv4.2 filesystem client. How should it > be able to help with Ceph? I explained that already: 1. IMO Mr Mainz has the expertise to comment on the subject of Windows kernel filesystem drivers 2. This specific driver has a kernel part and a userland daemon, and can be used as a template for a Windows CEPH driver. Replace the NFS parts with CEPH parts. 3. You wrote in https://www.spinics.net/lists/dev-ceph/msg06898.html that one of the better choices for a CEPH Windows driver is a RDBSS driver. ms-nfs41-client uses RDBSS. -- Internationalization&localization dev / 大阪大学 Takeshi Nishimura <takeshi.nishimura.linux(a)gmail.com>

1 0

Re: Cephfs kernel client bug report
by John Mulligan 17 Mar '26

17 Mar '26

On Tuesday, March 17, 2026 6:09:29 PM Eastern Daylight Time Alain Bertrand via ceph-users wrote: > Hi. > > I'm new here. I have been playing with ceph for a few years (nothing > dramatic, just using it in my proxmox cluster to store VM images and > host a cephfs with a lot of shared files). I think I found a bug in the > cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a > lot of directories quickly causes a kernel panic. I tried to create an > account on the bug tracker at tracker.ceph.com a week ago, but still > can't log in there. Do I put details here, or report somewhere else? > What's the next step I should take? > Tracker is the right place. Tracker approval requires manual review and unfortunately that makes the sign-up process very slow. You can join ceph slack and ask for updates the #sepia channel - for now I will do that so you don't have to jump through more hoops right now.

1 0

Cephfs kernel client bug report
by Alain Bertrand 17 Mar '26

17 Mar '26

Hi. I'm new here. I have been playing with ceph for a few years (nothing dramatic, just using it in my proxmox cluster to store VM images and host a cephfs with a lot of shared files). I think I found a bug in the cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a lot of directories quickly causes a kernel panic. I tried to create an account on the bug tracker at tracker.ceph.com a week ago, but still can't log in there. Do I put details here, or report somewhere else? What's the next step I should take? Thanks, Alain

1 0