Hello Eugen, I agree that a performance issue this major would've gone noticed by now, but I did a load of investigation and still can't find the root cause, it might be a non-Ceph issue, but also possible to be a corner case? No there haven't been any OS upgrade, or another specific change. Did a ceph config dump (including defaults) before/after upgrade, checked all the changed ones, reverted the fishy ones, nothing changed. The previous 3 clusters that I was testing on were all bare-metal - all had multi-site between 2 zones; separation between ec and replica pool is handled with a lua script, no RBD only object store. I also checked on a VM-based cluster (again multi-site, but no hybrid build this time). Same issue there. However, when tested on a small single site cluster, that wasn't the case (so no visible latency increase there), but it just made things more confusing since in all of the other multi-site clusters, the RGWs that run sync threads are separate from the ones that clients connect too, and it's the client latency that I'm monitoring. I did notice this issue: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3M... but that's not much similar to our case since we don't see any jump in CPU or memory usage. Also checked pg ops (including scrubs), some clusters were running scrubs and stuff for a while after upgrade but that cleared soon enough, though the latency jump seems persistent still after 2 months. One correction about my observation on NVMe iops: that started dropping a little while "before" upgrade so it's most likely a separate issue, but latency jump case stays the same. As for other parameters, LC ops do not seem to have changed, there is a jump in GC max age but that's minor, cross site replication looks fine, no jump in osd compression. Any ideas what else might worth checking/testing? On Tue, 14 Jul 2026, 12:53 Eugen Block via ceph-users, <ceph-users@ceph.io> wrote:
Hi,
I'm not sure if it could be related to summer vacation, but I feel like there should have been several reports similar to yours by now. I don't have upgraded any production cluster to 20 yet, so I can't really deny or confirm anything. Did you upgrade the OS as well? Maybe there were some relevant changes there. Or some non-persistent configs were overwritten, either in the OS or Ceph. Was there more deep-scrubbing after the upgrade which could have impacted the IO?
Regards, Eugen
Zitat von Moti via ceph-users <ceph-users@ceph.io>:
Hello everyone! I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing as well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET and HEAD requests.
There is no resource cap or significant increase in usage is visible. Neither fast ec nor any other new features are enabled yet, just did the upgrade and left it as is. For measuring latency I have a services that probes the cluster with consistent load so it's not a difference in load either. PG status history does not show any pg operation being heavily in process.
Cluster has a hybrid build so OSDs have HDD for data and NVMe for db block. Both ec and replicated pools exist. The strange hint was: NVMe disks' iops went from something around 5-6K constant io per second to almost zero (less than a hundred per second, on all host/OSDs, with some non-frequent high peaks), while the HDD iops is not much changed. db blocks on NVMe are still attached to OSDs with no issue.
I can't find a clue to explain this. Got the same results on 3 separate clusters (all went version 18 --> 20, same hybrid build and configs). Nothing in config dump changes (changed defaults) catches my attention to have caused this.
Anyone had similar issues? Any tips and ideas on what to check? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io