Hi again, - No, buckets are not versioned Let me describe the logic of my ceph prober service, it would clarify multiple of those questions: every minute: for endpoint in endpoints: for object_size in sizes: key="<bucket>/prefix_{object_size}_{endpoint}_{timestamp}" putObject(key) headObject(key) getObject(key) deleteObject(key) so: - HEAD is only for specific keys /bucket/key - There is load balancer in front of RGWs, but that's only one of the endpoints in above iterations; rest of the iterations are directly calling rgw endpoints and they still experience the same increase in HEAD latency (actually increase is the same for all object sizes as well; some show more fluctuations but overall a huge leap even per size/endpoint) - They're not targeting a small hot set since object is created with new key every iteration and removed shortly after - same key won't be overwritten but there are a lot of put+deletes - headObject(key) function is currently a minio client doing a lazy GetObject (not actually fetching anything) then calling .Stats() on the result (which calls head) - There is a lifecycle to clear up any leftovers (in case deleteObject failed) after 3 days, but that should be minimal, however manual delete is frequently happening - shard count is 11, bucket is not filling up so it always has zero to endpoints*sizes objects, 30 objects tops. Note1: Alongside this prober there are other users, but they don't have continuous load, it's usually sparse, burst-like and occasional, so I didn't study any of them individually, however, fearing that the issue might be prober and not ceph, I excluded prober beast logs, then among remaining beast logs unwrapped latency=*, this also showed the same peak in head latency immediately after upgrade. Other users' buckets are mostly unversioned, size however varies, there are tiny buckets and there are buckets with millions of objects. (The re-indexing that I mentioned was only done on the target bucket of prober though) Note2: prober latency was observed both from service's own exported metrics and also from beast logs, same both Note3: As I mentioned before I'm seeing this leap on 9 tentacle clusters, I highly doubt there's an external independent issue as those 9 were upgraded on different months, and they have different network/build/load/size, but it's possible that I set the same bad setting on all of them, or had a prior corruption on all of them, or my users are doing ops in a specific way that corrupts the index in new version Note4: just a recall: re-indexing fixes HEAD latency on all those clusters for a while, but it takes each cluster a different delay to have the latency increased again, some take half an hour, some take days (rate of prober CRUD is similar on all) Note5: again a reminder: The only method that consistently and massively worsened in latency is HEAD. Anywhere I'm talking about latency jump I mean the head one. Note6: to confuse you more :) let me note that among all those 9 clusters with the same result, one of them was slightly different: remember when I said latency leap is there on all endpoints, haproxy and direct rgw both? on one cluster it wasn't; on one of the clusters, latency leap was only visible on load balancer endpoint (while for the other 8 it was everywhere, and same prober is probing all 9). So if we assume lb issue in that one is external and just coincides with upgrade, that cluster would be the only cluster without latency being effected. I checked config dump and all setting diff between this 1 and the other 8, the only consistent difference is that this one is single-site while all the other 8 have active-active multisite enabled. I also managed to find a colleague with a recently upgraded tentacle cluster, whom cluster did not show this latency jump at all, and the only difference there was also multisite. I don't see how enabling multisite would effect head latency, but maybe this observation gives you a new clue. Apologies for the long message, I wanted to provide a full context. Regards On Fri, 18 Sept 2026, 17:36 Oguzhan Ozmen (BLOOMBERG/ 120 PARK), < ceph-users@ceph.io> wrote:
Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually
This is a really interesting observation.
- Are you using a "versioned" bucket by any chance? - Do you have load-balancer like HAProxy in front of rgws? - Can you also provide more details on the nature of the HEAD requests: - HEAD /bucket or - HEAD /bucket/key or - both are degrading. - Some questions on the workload, in case it's relevant: - Are the HEAD'd keys a small hot set, or spread evenly across the bucket? - Are those same keys also being overwritten repeatedly, or mostly read-only? - Explicit head_object() calls, or indirect (transfer manager, existence checks, waiters)? - Any lifecycle/expiration rules or other delete/reshard activity on this bucket? - Index shard count and rough object/version count for this bucket?
Oguzhan
From: ceph-users@ceph.io At: 09/14/26 10:06:07 UTC-4:00To: Oguzhan Ozmen (BLOOMBERG/ 120 PARK <https://www.google.com/maps/search/120+PARK?entry=gmail&source=g> ) Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Performance issue after tentacle upgrade
Hi Oguzhan, Thanks for your reply
I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening?
Regards
On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users@ceph.io> wrote:
Hey Moti,
Zitat von Moti via ceph-users <ceph-users@ceph.io>:
I recently upgraded from 18.2.4 to 20.2.1
I did notice this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3M... SSRVRPMDTTOMTT2K6/ <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3MREOESSRVRPMDTTOMTT2K6/>
but that's not much similar to our case since we don't see any jump in CPU or memory usage.
Although the symptoms don't look the same, I'd recommend upgrading to v20.2.3 to get the fix for this issue and the other fixes that went in: https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle
After v20.2.3 upgrade, if you still see the same issue, then I'd suggest collecting "perf" output per process per component (rgw, osd mainly) to further analyze the bottleneck.
You can also quickly check rgw logs to see what RGW reports as the latency for those PUT/GET/HEAD requests whether it matches the latency you see on your end (your wkld generator); especially for HEAD where no data streaming happens. RGW logs "start" and "end" of a request. This is to rule out whether bottleneck is outside of ceph (e.g., load-balancers).
Cheers, Oguzhan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io