- ceph-users - lists.ceph.io

Re: Performance issue after tentacle upgrade
by Moti 13 Aug '26

13 Aug '26

Yes we do have load balancer, but the service that monitors latency, probes rgw endpoints directly as well as the loadbalancer endpoint, and latency jump is similar for all those endpoints, not only loadbalancer. Also 9 clusters were upgraded from 18 to 20 now, on vastly different times, and they all started latency jump only after upgrade, so it's unlikely to be an independent coincidence in loadbalancer. Two new things I found: - The only consistent latency jump through all 9 clusters is in HEAD method, other methods differ cluster to cluster but HEAD latency is vividly and significantly worse on all of them - I removed and re-created the bucket that prober service is using, it fixed the HEAD latency for several hours and then it again jumped back up. This also happens consistently on all clusters, though with different speeds. I'm not sure what to conclude of that. Something is being corrupted in bucket index? On Mon, 10 Aug 2026, 09:31 Eugen Block via ceph-users, <ceph-users(a)ceph.io> wrote: > Do you have loadbalancers in front of the client traffic? Maybe > something changed there? > > Zitat von Moti <motaharesdq(a)gmail.com>: > > > Hello Eugen, > > I agree that a performance issue this major would've gone noticed by now, > > but I did a load of investigation and still can't find the root cause, it > > might be a non-Ceph issue, but also possible to be a corner case? > > No there haven't been any OS upgrade, or another specific change. Did a > > ceph config dump (including defaults) before/after upgrade, checked all > the > > changed ones, reverted the fishy ones, nothing changed. The previous 3 > > clusters that I was testing on were all bare-metal - all had multi-site > > between 2 zones; separation between ec and replica pool is handled with a > > lua script, no RBD only object store. I also checked on a VM-based > cluster > > (again multi-site, but no hybrid build this time). Same issue there. > > However, when tested on a small single site cluster, that wasn't the case > > (so no visible latency increase there), but it just made things more > > confusing since in all of the other multi-site clusters, the RGWs that > run > > sync threads are separate from the ones that clients connect too, and > it's > > the client latency that I'm monitoring. I did notice this issue: > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… > > but that's not much similar to our case since we don't see any jump in > CPU > > or memory usage. > > Also checked pg ops (including scrubs), some clusters were running scrubs > > and stuff for a while after upgrade but that cleared soon enough, though > > the latency jump seems persistent still after 2 months. > > One correction about my observation on NVMe iops: that started dropping a > > little while "before" upgrade so it's most likely a separate issue, but > > latency jump case stays the same. > > As for other parameters, LC ops do not seem to have changed, there is a > > jump in GC max age but that's minor, cross site replication looks fine, > no > > jump in osd compression. > > Any ideas what else might worth checking/testing? > > > > On Tue, 14 Jul 2026, 12:53 Eugen Block via ceph-users, < > ceph-users(a)ceph.io> > > wrote: > > > >> Hi, > >> > >> I'm not sure if it could be related to summer vacation, but I feel > >> like there should have been several reports similar to yours by now. I > >> don't have upgraded any production cluster to 20 yet, so I can't > >> really deny or confirm anything. > >> Did you upgrade the OS as well? Maybe there were some relevant changes > >> there. Or some non-persistent configs were overwritten, either in the > >> OS or Ceph. > >> Was there more deep-scrubbing after the upgrade which could have > >> impacted the IO? > >> > >> Regards, > >> Eugen > >> > >> Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > >> > >> > Hello everyone! > >> > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid > thing > >> as > >> > well as Tentacle) and latency went ~3-4x more on PUT, even worse on > GET > >> and > >> > HEAD requests. > >> > > >> > There is no resource cap or significant increase in usage is visible. > >> > Neither fast ec nor any other new features are enabled yet, just did > the > >> > upgrade and left it as is. For measuring latency I have a services > that > >> > probes the cluster with consistent load so it's not a difference in > load > >> > either. PG status history does not show any pg operation being > heavily in > >> > process. > >> > > >> > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db > >> block. > >> > Both ec and replicated pools exist. > >> > The strange hint was: NVMe disks' iops went from something around 5-6K > >> > constant io per second to almost zero (less than a hundred per > second, on > >> > all host/OSDs, with some non-frequent high peaks), while the HDD iops > is > >> > not much changed. db blocks on NVMe are still attached to OSDs with no > >> > issue. > >> > > >> > I can't find a clue to explain this. Got the same results on 3 > separate > >> > clusters (all went version 18 --> 20, same hybrid build and configs). > >> > Nothing in config dump changes (changed defaults) catches my > attention to > >> > have caused this. > >> > > >> > Anyone had similar issues? Any tips and ideas on what to check? > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users(a)ceph.io > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: umbrella 21.1.1 RC2 QE validation status
by Brad Hubbard 13 Aug '26

13 Aug '26

On Mon, Aug 10, 2026 at 11:32 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/78413 > > Release Notes - email version for RCs > https://github.com/ceph/ceph/pull/70240#issuecomment-5232852937 @Neha Ojha > <nojha(a)redhat.com> @Patrick Donnelly <pdonnell(a)redhat.com> > LRC upgrade - TBD > > PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Radek, Nitzan, Afreen (dashboard), Travis (rook) > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane, Guillaume > rbd: Ilya > krbd: Ilya > > nvmeof: PASSED > > upgrade: Kamoltat > > upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: PASSED > upgrade-clients:client-upgrade-squid-umbrella-umbrella: PASSED > > perf-basic: PASSED > > crimson-rados: Matan > > powercycle: Brad (same as before) > Approved, waiting on known issue https://tracker.ceph.com/issues/79075 > TIA > -- Cheers, Brad

1 0

Re: RGW: cannot PUT data into S3-like bucket
by Janne Johansson 12 Aug '26

12 Aug '26

> > > after upgrade from ceph 19.2.5 to 20.2.2 (and now I'm on 20.2.3), > I have big problem to write any bigger blocks of data to S3 > buckets. > After upgrade to 20.2.2, everythink PUT requests gets 502 HTTP > code. > Today I've tried this: > > 1) I've created new bucket > 2) I set this bucket for rustic (rustic init) with user, which is > Owner of this buckets. 'rustic init' was fine. > 3) I've tried to do backup: 'rustic backup'. No one block of > backup data was written. I would certainly try a manual S3 client and validate if you can or not put objects into buckets, also reading the RGW logs from when this is erroring out seems like a very good idea. -- May the most significant bit of your life be positive.

2 1

RGW: cannot PUT data into S3-like bucket
by Jan Marek 12 Aug '26

12 Aug '26

Hello, after upgrade from ceph 19.2.5 to 20.2.2 (and now I'm on 20.2.3), I have big problem to write any bigger blocks of data to S3 buckets. My config: CEPH on HDD, everythink works fine. We are using not only S3 RGW, but RADOS for images of virtual machines - without problem. I have 9 host with total 108 HDD. On 4 fast machines I have RGW containers. CEPH cluster is maintained by cephadm (podman). I have haproxy, which do gateway to RGW and maintain Let's Encrypt certificates. After upgrade to 20.2.2, everythink PUT requests gets 502 HTTP code. I've manage Veeam backup of M365 and some rustic backups. Today I've tried this: 1) I've created new bucket 2) I set this bucket for rustic (rustic init) with user, which is Owner of this buckets. 'rustic init' was fine. 3) I've tried to do backup: 'rustic backup'. No one block of backup data was written. Please, can you help me? Many thanks. Sincerely Jan Marek -- Ing. Jan Marek správce IT Jihočeská univerzita v Českých Budějovicích Centum informačních technologií Branišovská 1645/31a, 370 05 České Budějovice T/ +420 38 903 2080 M/ +420 604 660 121 E/ jmarek(a)jcu.cz www.jcu.cz

1 0

Re: Performance issue after tentacle upgrade
by Oguzhan Ozmen (BLOOMBERG/ 120 PARK) 11 Aug '26

11 Aug '26

Hey Moti, Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > I recently upgraded from 18.2.4 to 20.2.1 > > I did notice this issue: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… > but that's not much similar to our case since we don't see any jump in CPU or memory usage. Although the symptoms don't look the same, I'd recommend upgrading to v20.2.3 to get the fix for this issue and the other fixes that went in: https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle After v20.2.3 upgrade, if you still see the same issue, then I'd suggest collecting "perf" output per process per component (rgw, osd mainly) to further analyze the bottleneck. You can also quickly check rgw logs to see what RGW reports as the latency for those PUT/GET/HEAD requests whether it matches the latency you see on your end (your wkld generator); especially for HEAD where no data streaming happens. RGW logs "start" and "end" of a request. This is to rule out whether bottleneck is outside of ceph (e.g., load-balancers). Cheers, Oguzhan

1 0

Announcing go-ceph v0.41.0
by John Mulligan 11 Aug '26

11 Aug '26

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.41.0 Enhancements include additions to cephfs and rgw admin subpackages. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project.

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Stefan Möding 11 Aug '26

11 Aug '26

Hi, thanks for your feedback! Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > that sounds a bit impractical to create a new pool for each tenant > regarding PG and pool management (adding and possibly removing data > pools from the CephFS), depending on how many tenants you have or plan > to have. In our setup each K8S cluster would be on tenant. So the pools would be on the subvolumegroup level and we expect to go ahead with less than ten pools for the foreseeable future. > What I just tested was to manually encrypt a directory in a test > cluster with fscrypt, that seems to work as intended. But I don't have > anything with k8s to test it at the moment. From what I have tested, the problem seems to be located in ceph-csi where the encryption is initially set up. The code tries to allocates an exclusive lock on an object representing the volume to prevent a race condition with multiple clients sharing the volume. The relevant code looks like this: ioctx, err := volOptions.GetConnection().GetIoctx(volOptions.MetadataPool) if err != nil { log.ErrorLog(ctx, "Failed to create ioctx: %s", err) return err } lock := iolock.NewLock(ioctx, objectUUID, lockName, lockCookie, lockDesc, lockDuration) err = lock.LockExclusive(ctx) if err != nil { log.ErrorLog(ctx, "failed to create lock for volume ID %s: %v", volID, err) return err } The error message on the ceph/csi pods clearly shows that the LockExclusive call fails with “Operation not permitted”. I failed to find the place in the relevant code, where the namespace for this operation is set, so it looks like the default namespace is used. But the capabilities only allow a specific namespace and so this could be the root cause. I tried to use the following code to create an object in the metadata pool and lock it. I used the same user with the same capabilities: % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put foobar /dev/null % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock get foobar lock1 --lock-cookie=cookie --lock-duration=60 --lock-type=exclusive % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock info foobar lock1 {"name":"lock1","type":"exclusive","tag":"","lockers":[{"name":"client.2690590","cookie":"cookie","description":"","expiration":"2026-07-31T08:21:36.190835+0200","addr":"172.21.70.10:0/2292729948"}]} So the capabilities seem to be sufficient for the lock operation. But leaving out the namespace when creating the lock will show the exact same error message that the ceph-csi shows: % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put foobar /dev/null % rados --id=k8s-test --pool=cephfs.meta lock get foobar lock1 --lock-cookie=cookie --lock-duration=60 --lock-type=exclusive ERROR: failed locking: (1) Operation not permitted error 1: (1) Operation not permitted -- Stefan

1 0

Re: Survey: Is anyone using 32-bit builds of the CephFS kernel client?
by Xiubo Li 11 Aug '26

11 Aug '26

Hi everyone, Thanks for the feedback. So far, I have received several replies regarding the 32-bit CephFS kernel client usage. Based on the responses, there do not appear to be any active users of 32-bit builds, and I have not received any objections to moving forward with a 64-bit-only implementation. Therefore, I will continue working on the patch to remove 32-bit support and proceed with the 64-bit-only approach. I will make sure the change is properly documented, including the deprecation/removal of 32-bit support in the relevant release notes. Thanks again for the feedback and discussion. Best regards, Xiubo On Thu, 16 Jul 2026 at 16:53, Xiubo Li <xiubo.li(a)clyso.com> wrote: > > Hi everyone, > > We are considering a performance improvement for the CephFS kernel > client (kclient) that would be significantly simpler and cleaner to > implement as 64-bit-only. Supporting 32-bit in this case adds > non-trivial complexity and ongoing maintenance burden, so we'd like to > understand whether continuing 32-bit support is still necessary. > > Before making any decision, we want to hear from the community: > > - Are you currently using, or planning to use, 32-bit (arm/armhf, > i386/i686) builds of the CephFS kernel client? > - If yes, is this for production workloads? Or legacy/testing purposes? > - What is your platform (distribution, kernel version, architecture)? > > If there are no active 32-bit users, we would proceed with a > 64-bit-only implementation and clearly document the deprecation in the > release notes with appropriate warnings. > > If 32-bit support is still needed, we will explore alternative > approaches — for example, keeping the current code path for 32-bit > while the new fast path is 64-bit-only, or finding a way to cleanly > abstract the difference. > > Please reply by August 6, 2026 so we can factor your feedback into the plan. > > Thanks! > Xiubo Li

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Eugen Block 10 Aug '26

10 Aug '26

Hi, that sounds a bit impractical to create a new pool for each tenant regarding PG and pool management (adding and possibly removing data pools from the CephFS), depending on how many tenants you have or plan to have. I would recommend to use this client authorization as a template: ceph fs subvolume authorize cephfs subvol-ebl user2 --group_name=subvolgroup1 --access_level=rw This results in these caps: ceph auth get client.user2 [client.user2] key = XXX caps mds = "allow rw path=/volumes/subvolgroup1/subvol-ebl/2596be92-9168-439f-b39f-61eb092e207b" caps mon = "allow r" caps osd = "allow rw pool=cephfs.cephfs.data namespace=fsvolumens_subvol-ebl" I created the subvolume in the dashboard with isolated namespace (which can be done via CLI as well, of course). And then you also need the "rw" caps (which you already have) for the mgr for k8s to be able to utilize the volume module. But I'm not sure if this will suffice to make it work. What I just tested was to manually encrypt a directory in a test cluster with fscrypt, that seems to work as intended. But I don't have anything with k8s to test it at the moment. IIRC, the caps you specified wrt metadata would indeed allow a client to access other tenant's metadata, but I'm not entirely sure. Hopefully someone else can chime in and has better answers for you. Regards, Eugen Zitat von Stefan Möding via ceph-users <ceph-users(a)ceph.io>: > Hi, > > Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > >> I haven't tried it myself yet so I don't really have an answer. But >> I noticed that your OSD caps may be incomplete wrt x permissions: > >>> osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, >>> allow rw namespace=k8s-test tag cephfs data=root' > >> The second allow statement doesn't have an "x". Maybe that's the issue >> why the workaround fails? > > We tried that, but the error remains. We now consider to use the > following capabilities in combination to creating a dedicated data pool > for each tenant (K8s cluster): > > mon_caps: 'allow r fsname=root' > mgr_caps: 'allow rw' > osd_caps: 'allow rwx pool=cephfs.meta, allow rw pool=cephfs.data, > allow rw pool=cephfs.k8s-test' > mds_caps: 'allow r fsname=root path=/volumes, allow rws fsname=root > path=/volumes/k8s-test' > > CephFS is created with cephfs.meta as metadata pool and cephfs.data as > default data pool while cephfs.k8s-test is the pool for this tenant. > > What are the implications to share the metadata/default data pools and > have no namespace? Is the MDS capability sufficient to separate tenants > or can everyone access the metadata objects of everyone else? > > -- > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Performance issue after tentacle upgrade
by Moti 10 Aug '26

10 Aug '26

Hello Eugen, I agree that a performance issue this major would've gone noticed by now, but I did a load of investigation and still can't find the root cause, it might be a non-Ceph issue, but also possible to be a corner case? No there haven't been any OS upgrade, or another specific change. Did a ceph config dump (including defaults) before/after upgrade, checked all the changed ones, reverted the fishy ones, nothing changed. The previous 3 clusters that I was testing on were all bare-metal - all had multi-site between 2 zones; separation between ec and replica pool is handled with a lua script, no RBD only object store. I also checked on a VM-based cluster (again multi-site, but no hybrid build this time). Same issue there. However, when tested on a small single site cluster, that wasn't the case (so no visible latency increase there), but it just made things more confusing since in all of the other multi-site clusters, the RGWs that run sync threads are separate from the ones that clients connect too, and it's the client latency that I'm monitoring. I did notice this issue: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… but that's not much similar to our case since we don't see any jump in CPU or memory usage. Also checked pg ops (including scrubs), some clusters were running scrubs and stuff for a while after upgrade but that cleared soon enough, though the latency jump seems persistent still after 2 months. One correction about my observation on NVMe iops: that started dropping a little while "before" upgrade so it's most likely a separate issue, but latency jump case stays the same. As for other parameters, LC ops do not seem to have changed, there is a jump in GC max age but that's minor, cross site replication looks fine, no jump in osd compression. Any ideas what else might worth checking/testing? On Tue, 14 Jul 2026, 12:53 Eugen Block via ceph-users, <ceph-users(a)ceph.io> wrote: > Hi, > > I'm not sure if it could be related to summer vacation, but I feel > like there should have been several reports similar to yours by now. I > don't have upgraded any production cluster to 20 yet, so I can't > really deny or confirm anything. > Did you upgrade the OS as well? Maybe there were some relevant changes > there. Or some non-persistent configs were overwritten, either in the > OS or Ceph. > Was there more deep-scrubbing after the upgrade which could have > impacted the IO? > > Regards, > Eugen > > Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > > > Hello everyone! > > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing > as > > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET > and > > HEAD requests. > > > > There is no resource cap or significant increase in usage is visible. > > Neither fast ec nor any other new features are enabled yet, just did the > > upgrade and left it as is. For measuring latency I have a services that > > probes the cluster with consistent load so it's not a difference in load > > either. PG status history does not show any pg operation being heavily in > > process. > > > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db > block. > > Both ec and replicated pools exist. > > The strange hint was: NVMe disks' iops went from something around 5-6K > > constant io per second to almost zero (less than a hundred per second, on > > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > > not much changed. db blocks on NVMe are still attached to OSDs with no > > issue. > > > > I can't find a clue to explain this. Got the same results on 3 separate > > clusters (all went version 18 --> 20, same hybrid build and configs). > > Nothing in config dump changes (changed defaults) catches my attention to > > have caused this. > > > > Anyone had similar issues? Any tips and ideas on what to check? > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1