Re: Performance issue after tentacle upgrade
Hi Oguzhan, Thanks for your reply I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening? Regards On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users@ceph.io> wrote:
Hey Moti,
Zitat von Moti via ceph-users <ceph-users@ceph.io>:
I recently upgraded from 18.2.4 to 20.2.1
I did notice this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3M...
but that's not much similar to our case since we don't see any jump in CPU or memory usage.
Although the symptoms don't look the same, I'd recommend upgrading to v20.2.3 to get the fix for this issue and the other fixes that went in: https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle
After v20.2.3 upgrade, if you still see the same issue, then I'd suggest collecting "perf" output per process per component (rgw, osd mainly) to further analyze the bottleneck.
You can also quickly check rgw logs to see what RGW reports as the latency for those PUT/GET/HEAD requests whether it matches the latency you see on your end (your wkld generator); especially for HEAD where no data streaming happens. RGW logs "start" and "end" of a request. This is to rule out whether bottleneck is outside of ceph (e.g., load-balancers).
Cheers, Oguzhan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually
This is a really interesting observation. - Are you using a "versioned" bucket by any chance? - Do you have load-balancer like HAProxy in front of rgws? - Can you also provide more details on the nature of the HEAD requests: - HEAD /bucket or - HEAD /bucket/key or - both are degrading. - Some questions on the workload, in case it's relevant: - Are the HEAD'd keys a small hot set, or spread evenly across the bucket? - Are those same keys also being overwritten repeatedly, or mostly read-only? - Explicit head_object() calls, or indirect (transfer manager, existence checks, waiters)? - Any lifecycle/expiration rules or other delete/reshard activity on this bucket? - Index shard count and rough object/version count for this bucket? Oguzhan From: ceph-users@ceph.io At: 09/14/26 10:06:07 UTC-4:00To: Oguzhan Ozmen (BLOOMBERG/ 120 PARK ) Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Performance issue after tentacle upgrade Hi Oguzhan, Thanks for your reply I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening? Regards On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users@ceph.io> wrote:
Hey Moti,
Zitat von Moti via ceph-users <ceph-users@ceph.io>:
I recently upgraded from 18.2.4 to 20.2.1
I did notice this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3M... SSRVRPMDTTOMTT2K6/
but that's not much similar to our case since we don't see any jump in CPU or memory usage.
Although the symptoms don't look the same, I'd recommend upgrading to v20.2.3 to get the fix for this issue and the other fixes that went in: https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle
After v20.2.3 upgrade, if you still see the same issue, then I'd suggest collecting "perf" output per process per component (rgw, osd mainly) to further analyze the bottleneck.
You can also quickly check rgw logs to see what RGW reports as the latency for those PUT/GET/HEAD requests whether it matches the latency you see on your end (your wkld generator); especially for HEAD where no data streaming happens. RGW logs "start" and "end" of a request. This is to rule out whether bottleneck is outside of ceph (e.g., load-balancers).
Cheers, Oguzhan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi again, - No, buckets are not versioned Let me describe the logic of my ceph prober service, it would clarify multiple of those questions: every minute: for endpoint in endpoints: for object_size in sizes: key="<bucket>/prefix_{object_size}_{endpoint}_{timestamp}" putObject(key) headObject(key) getObject(key) deleteObject(key) so: - HEAD is only for specific keys /bucket/key - There is load balancer in front of RGWs, but that's only one of the endpoints in above iterations; rest of the iterations are directly calling rgw endpoints and they still experience the same increase in HEAD latency (actually increase is the same for all object sizes as well; some show more fluctuations but overall a huge leap even per size/endpoint) - They're not targeting a small hot set since object is created with new key every iteration and removed shortly after - same key won't be overwritten but there are a lot of put+deletes - headObject(key) function is currently a minio client doing a lazy GetObject (not actually fetching anything) then calling .Stats() on the result (which calls head) - There is a lifecycle to clear up any leftovers (in case deleteObject failed) after 3 days, but that should be minimal, however manual delete is frequently happening - shard count is 11, bucket is not filling up so it always has zero to endpoints*sizes objects, 30 objects tops. Note1: Alongside this prober there are other users, but they don't have continuous load, it's usually sparse, burst-like and occasional, so I didn't study any of them individually, however, fearing that the issue might be prober and not ceph, I excluded prober beast logs, then among remaining beast logs unwrapped latency=*, this also showed the same peak in head latency immediately after upgrade. Other users' buckets are mostly unversioned, size however varies, there are tiny buckets and there are buckets with millions of objects. (The re-indexing that I mentioned was only done on the target bucket of prober though) Note2: prober latency was observed both from service's own exported metrics and also from beast logs, same both Note3: As I mentioned before I'm seeing this leap on 9 tentacle clusters, I highly doubt there's an external independent issue as those 9 were upgraded on different months, and they have different network/build/load/size, but it's possible that I set the same bad setting on all of them, or had a prior corruption on all of them, or my users are doing ops in a specific way that corrupts the index in new version Note4: just a recall: re-indexing fixes HEAD latency on all those clusters for a while, but it takes each cluster a different delay to have the latency increased again, some take half an hour, some take days (rate of prober CRUD is similar on all) Note5: again a reminder: The only method that consistently and massively worsened in latency is HEAD. Anywhere I'm talking about latency jump I mean the head one. Note6: to confuse you more :) let me note that among all those 9 clusters with the same result, one of them was slightly different: remember when I said latency leap is there on all endpoints, haproxy and direct rgw both? on one cluster it wasn't; on one of the clusters, latency leap was only visible on load balancer endpoint (while for the other 8 it was everywhere, and same prober is probing all 9). So if we assume lb issue in that one is external and just coincides with upgrade, that cluster would be the only cluster without latency being effected. I checked config dump and all setting diff between this 1 and the other 8, the only consistent difference is that this one is single-site while all the other 8 have active-active multisite enabled. I also managed to find a colleague with a recently upgraded tentacle cluster, whom cluster did not show this latency jump at all, and the only difference there was also multisite. I don't see how enabling multisite would effect head latency, but maybe this observation gives you a new clue. Apologies for the long message, I wanted to provide a full context. Regards On Fri, 18 Sept 2026, 17:36 Oguzhan Ozmen (BLOOMBERG/ 120 PARK), < ceph-users@ceph.io> wrote:
Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually
This is a really interesting observation.
- Are you using a "versioned" bucket by any chance? - Do you have load-balancer like HAProxy in front of rgws? - Can you also provide more details on the nature of the HEAD requests: - HEAD /bucket or - HEAD /bucket/key or - both are degrading. - Some questions on the workload, in case it's relevant: - Are the HEAD'd keys a small hot set, or spread evenly across the bucket? - Are those same keys also being overwritten repeatedly, or mostly read-only? - Explicit head_object() calls, or indirect (transfer manager, existence checks, waiters)? - Any lifecycle/expiration rules or other delete/reshard activity on this bucket? - Index shard count and rough object/version count for this bucket?
Oguzhan
From: ceph-users@ceph.io At: 09/14/26 10:06:07 UTC-4:00To: Oguzhan Ozmen (BLOOMBERG/ 120 PARK <https://www.google.com/maps/search/120+PARK?entry=gmail&source=g> ) Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Performance issue after tentacle upgrade
Hi Oguzhan, Thanks for your reply
I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening?
Regards
On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users@ceph.io> wrote:
Hey Moti,
Zitat von Moti via ceph-users <ceph-users@ceph.io>:
I recently upgraded from 18.2.4 to 20.2.1
I did notice this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3M... SSRVRPMDTTOMTT2K6/ <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3MREOESSRVRPMDTTOMTT2K6/>
but that's not much similar to our case since we don't see any jump in CPU or memory usage.
Although the symptoms don't look the same, I'd recommend upgrading to v20.2.3 to get the fix for this issue and the other fixes that went in: https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle
After v20.2.3 upgrade, if you still see the same issue, then I'd suggest collecting "perf" output per process per component (rgw, osd mainly) to further analyze the bottleneck.
You can also quickly check rgw logs to see what RGW reports as the latency for those PUT/GET/HEAD requests whether it matches the latency you see on your end (your wkld generator); especially for HEAD where no data streaming happens. RGW logs "start" and "end" of a request. This is to rule out whether bottleneck is outside of ceph (e.g., load-balancers).
Cheers, Oguzhan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Great details. I'm 99% sure this is due to the "RGW_ATTR_OBJ_REPLICATION_STATUS" work added in Squid and onwards: see https://tracker.ceph.com/issues/64436 TLDR; on the GET/HEAD path, when the object's RGW_ATTR_OBJ_REPLICATION_STATUS attr is "PENDING", rgw calls is_sync_completed() to see whether the object has been replicated to the other site yet and this one looks for the oldest un-trimmed bilog entry in that key's shard: so a backend (RADOS) call to look into the bucket index. This is essentially a bilog OMAP (bucket's index shard is held as OMAP in RocksDB) read seeking to the front of that object's bilog shard (the 1 shard of 11 that owns the key). As you continuously put/delete, that shard's bilog churns constantly: entries are appended and auto-trim (may be) removing consumed ones from the front. Two things might be happening 1] bilog entries are accumulating over time. Ideally, auto-trim should remove processed bilog entries but on a large cluster auto-trim can lag and leave old/stale processed entries accumulating. So, the amount of data is_sync_completed() needs to go thru increases. 2] The put/delete churn leaves RocksDB tombstones and SST bloat at the front of the range. That grows until RocksDB auto-compacts those tombstones. To confirm without a reshard/recreate: when HEAD latency becomes high again over time, 1] run ceph tell osd.N compact on the index-pool OSDs for that bucket: if latency drops right away, it points at RocksDB tombstone/SST bloat from your deletes. 2] Also check radosgw-admin bilog list --bucket=<> | wc -l to see whether auto-trim is keeping up. Just for testing (do NOT run this on a client bucket but only in your test bucket; manual bilog trim drops entries the secondary may not have replicated yet, so only do it on a throwaway bucket you don't care about syncing): radosgw-admin bilog trim --bucket=<>. if latency drops right away, it's the accumulation of bilog entries (i.e., auto-trim pace is too slow for your cluster). Hope this helps. Oguzhan From: ceph-users@ceph.io At: 09/24/26 10:23:18 UTC-4:00To: Oguzhan Ozmen (BLOOMBERG/ 120 PARK ) Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Performance issue after tentacle upgrade Hi again, - No, buckets are not versioned Let me describe the logic of my ceph prober service, it would clarify multiple of those questions: every minute: for endpoint in endpoints: for object_size in sizes: key="<bucket>/prefix_{object_size}_{endpoint}_{timestamp}" putObject(key) headObject(key) getObject(key) deleteObject(key) so: - HEAD is only for specific keys /bucket/key - There is load balancer in front of RGWs, but that's only one of the endpoints in above iterations; rest of the iterations are directly calling rgw endpoints and they still experience the same increase in HEAD latency (actually increase is the same for all object sizes as well; some show more fluctuations but overall a huge leap even per size/endpoint) - They're not targeting a small hot set since object is created with new key every iteration and removed shortly after - same key won't be overwritten but there are a lot of put+deletes - headObject(key) function is currently a minio client doing a lazy GetObject (not actually fetching anything) then calling .Stats() on the result (which calls head) - There is a lifecycle to clear up any leftovers (in case deleteObject failed) after 3 days, but that should be minimal, however manual delete is frequently happening - shard count is 11, bucket is not filling up so it always has zero to endpoints*sizes objects, 30 objects tops. Note1: Alongside this prober there are other users, but they don't have continuous load, it's usually sparse, burst-like and occasional, so I didn't study any of them individually, however, fearing that the issue might be prober and not ceph, I excluded prober beast logs, then among remaining beast logs unwrapped latency=*, this also showed the same peak in head latency immediately after upgrade. Other users' buckets are mostly unversioned, size however varies, there are tiny buckets and there are buckets with millions of objects. (The re-indexing that I mentioned was only done on the target bucket of prober though) Note2: prober latency was observed both from service's own exported metrics and also from beast logs, same both Note3: As I mentioned before I'm seeing this leap on 9 tentacle clusters, I highly doubt there's an external independent issue as those 9 were upgraded on different months, and they have different network/build/load/size, but it's possible that I set the same bad setting on all of them, or had a prior corruption on all of them, or my users are doing ops in a specific way that corrupts the index in new version Note4: just a recall: re-indexing fixes HEAD latency on all those clusters for a while, but it takes each cluster a different delay to have the latency increased again, some take half an hour, some take days (rate of prober CRUD is similar on all) Note5: again a reminder: The only method that consistently and massively worsened in latency is HEAD. Anywhere I'm talking about latency jump I mean the head one. Note6: to confuse you more :) let me note that among all those 9 clusters with the same result, one of them was slightly different: remember when I said latency leap is there on all endpoints, haproxy and direct rgw both? on one cluster it wasn't; on one of the clusters, latency leap was only visible on load balancer endpoint (while for the other 8 it was everywhere, and same prober is probing all 9). So if we assume lb issue in that one is external and just coincides with upgrade, that cluster would be the only cluster without latency being effected. I checked config dump and all setting diff between this 1 and the other 8, the only consistent difference is that this one is single-site while all the other 8 have active-active multisite enabled. I also managed to find a colleague with a recently upgraded tentacle cluster, whom cluster did not show this latency jump at all, and the only difference there was also multisite. I don't see how enabling multisite would effect head latency, but maybe this observation gives you a new clue. Apologies for the long message, I wanted to provide a full context. Regards On Fri, 18 Sept 2026, 17:36 Oguzhan Ozmen (BLOOMBERG/ 120 PARK), < ceph-users@ceph.io> wrote:
Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually
This is a really interesting observation.
- Are you using a "versioned" bucket by any chance? - Do you have load-balancer like HAProxy in front of rgws? - Can you also provide more details on the nature of the HEAD requests: - HEAD /bucket or - HEAD /bucket/key or - both are degrading. - Some questions on the workload, in case it's relevant: - Are the HEAD'd keys a small hot set, or spread evenly across the bucket? - Are those same keys also being overwritten repeatedly, or mostly read-only? - Explicit head_object() calls, or indirect (transfer manager, existence checks, waiters)? - Any lifecycle/expiration rules or other delete/reshard activity on this bucket? - Index shard count and rough object/version count for this bucket?
Oguzhan
From: ceph-users@ceph.io At: 09/14/26 10:06:07 UTC-4:00To: Oguzhan Ozmen (BLOOMBERG/ 120 PARK <https://www.google.com/maps/search/120+PARK?entry=gmail&source=g> ) Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Performance issue after tentacle upgrade
Hi Oguzhan, Thanks for your reply
I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening?
Regards
On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users@ceph.io> wrote:
Hey Moti,
Zitat von Moti via ceph-users <ceph-users@ceph.io>:
I recently upgraded from 18.2.4 to 20.2.1
I did notice this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3M...
SSRVRPMDTTOMTT2K6/
<https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3MREOESSRVRPMDTTOMTT2K6/>
but that's not much similar to our case since we don't see any jump in CPU or memory usage.
Although the symptoms don't look the same, I'd recommend upgrading to v20.2.3 to get the fix for this issue and the other fixes that went in: https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle
After v20.2.3 upgrade, if you still see the same issue, then I'd suggest collecting "perf" output per process per component (rgw, osd mainly) to further analyze the bottleneck.
You can also quickly check rgw logs to see what RGW reports as the latency for those PUT/GET/HEAD requests whether it matches the latency you see on your end (your wkld generator); especially for HEAD where no data streaming happens. RGW logs "start" and "end" of a request. This is to rule out whether bottleneck is outside of ceph (e.g., load-balancers).
Cheers, Oguzhan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Moti
-
Oguzhan Ozmen (BLOOMBERG/ 120 PARK)