Hi Oguzhan, Thanks for your reply I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening? Regards On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users@ceph.io> wrote:
Hey Moti,
Zitat von Moti via ceph-users <ceph-users@ceph.io>:
I recently upgraded from 18.2.4 to 20.2.1
I did notice this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3M...
but that's not much similar to our case since we don't see any jump in CPU or memory usage.
Although the symptoms don't look the same, I'd recommend upgrading to v20.2.3 to get the fix for this issue and the other fixes that went in: https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle
After v20.2.3 upgrade, if you still see the same issue, then I'd suggest collecting "perf" output per process per component (rgw, osd mainly) to further analyze the bottleneck.
You can also quickly check rgw logs to see what RGW reports as the latency for those PUT/GET/HEAD requests whether it matches the latency you see on your end (your wkld generator); especially for HEAD where no data streaming happens. RGW logs "start" and "end" of a request. This is to rule out whether bottleneck is outside of ceph (e.g., load-balancers).
Cheers, Oguzhan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io