Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually
This is a really interesting observation. - Are you using a "versioned" bucket by any chance? - Do you have load-balancer like HAProxy in front of rgws? - Can you also provide more details on the nature of the HEAD requests: - HEAD /bucket or - HEAD /bucket/key or - both are degrading. - Some questions on the workload, in case it's relevant: - Are the HEAD'd keys a small hot set, or spread evenly across the bucket? - Are those same keys also being overwritten repeatedly, or mostly read-only? - Explicit head_object() calls, or indirect (transfer manager, existence checks, waiters)? - Any lifecycle/expiration rules or other delete/reshard activity on this bucket? - Index shard count and rough object/version count for this bucket? Oguzhan From: ceph-users@ceph.io At: 09/14/26 10:06:07 UTC-4:00To: Oguzhan Ozmen (BLOOMBERG/ 120 PARK ) Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Performance issue after tentacle upgrade Hi Oguzhan, Thanks for your reply I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening? Regards On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users@ceph.io> wrote:
Hey Moti,
Zitat von Moti via ceph-users <ceph-users@ceph.io>:
I recently upgraded from 18.2.4 to 20.2.1
I did notice this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3M... SSRVRPMDTTOMTT2K6/
but that's not much similar to our case since we don't see any jump in CPU or memory usage.
Although the symptoms don't look the same, I'd recommend upgrading to v20.2.3 to get the fix for this issue and the other fixes that went in: https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle
After v20.2.3 upgrade, if you still see the same issue, then I'd suggest collecting "perf" output per process per component (rgw, osd mainly) to further analyze the bottleneck.
You can also quickly check rgw logs to see what RGW reports as the latency for those PUT/GET/HEAD requests whether it matches the latency you see on your end (your wkld generator); especially for HEAD where no data streaming happens. RGW logs "start" and "end" of a request. This is to rule out whether bottleneck is outside of ceph (e.g., load-balancers).
Cheers, Oguzhan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io