Hello Frédéric Thanks for your response About iops; yes iops drop is good news, but this amount (far more than 50%) was a bit suspicious, and it shouldn't worsen latency. I was worried that slow disk might be doing what fast disk is supposed to do, but that's also not the case, so it looks like io just vanished to near zero. However this very well could be a separate problem/topic. About config defaults; thanks for that link, that's very handy. I checked the diff and tried to point out what might be problematic, these are the configs I tested changing, however did not get any better latency in response: - bluestore_elastic_shared_blob -> false - swapping osd_op_num_shards_hdd and osd_op_num_threads_per_shard that you mentioned - disabling ec partial reads - decreasing rgw thread pool size (I had this set to a higher value than default on reef) - decreasing rgw log trim interval I haven't tried changing envelope mode, I will, and thanks for the hint, but is it straight forward to upgrade wal back to envelope mode using bluestore tool if downgrade didn't seem to solve the issue? Also meanwhile since my original post I've been doing more tests, and the same case happened on other clusters as well (VMs with no separate fast/slow disks), the only time latency peak didn't happen, was when I tested on a single zone cluster, which is confusing since all my multi zone clusters use separate RGWs for running sync threads, and I'm seeing latency peak on client RGWs, not the sync ones. It might as well be another parameter that's different on the single zone one and I'm missing it, but that's my initial observation. Best regards Moti On Thu, 16 Jul 2026, 19:34 Frédéric Nass via ceph-users, <ceph-users@ceph.io> wrote:

Hi Moti,

Anything suspicious on 'ceph osd perf' output?

...

NVMe disks' iops went from something around 5-6K constant io per second to almost zero (less than a hundred per second, on all host/OSDs, with some non-frequent high peaks),

That would be pretty good news, actually. This would likely be the result of the new 'bluefs_wal_envelope_mode' [1]. Its own description says it "reduces by ~50% the amount of fdatasync syscalls" on the RocksDB WAL. :-) I don't expect this to explain the performance drop at this time, though.

I'd recommend that you check the new/old settings and default values between both versions using our online Config Diff tool here [2]. It might ring a bell.

The tool highlights that the default values for 'osd_op_num_shards_hdd' and 'osd_op_num_threads_per_shard_hdd' were swapped between both versions. I'm not sure what the impact could be on your cluster in terms of performance, but if you're using mClock, reverting this change or trying 'wpq' osd scheduler instead might help. You can run 'ceph config set osd osd_op_queue wpq' and restart all OSDs to figure out if mClock has anything to do with what you observe.

If swapping values or changing the osd scheduler to wpq doesn't help, you could try disabling 'bluefs_wal_envelope_mode' and running 'ceph-bluestore-tool --command downgrade-wal-to-v1' (make sure to test this out on a single OSD first).

Best regards,

Frédéric

[1]

https://github.com/ceph/ceph/blob/708737d3ec0669a8e01265d9699d6f939debec13/s... [2]

https://analyzer.clyso.com/#/analyzer/config-diff?source=v18.2.4&target=v20.2.1

Frédéric Nass

Senior Ceph Engineer

Ceph Ambassador, France

+49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751>

frederic.nass@clyso.com

www.clyso.com

Hohenzollernstr. 27, 80801 Muni <https://www.google.com/maps/search/Hohenzollernstr.+27,+80801+Muni?entry=gmail&source=g> ch

Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306

Le jeu. 9 juil. 2026 à 15:33, Moti via ceph-users <ceph-users@ceph.io> a écrit :

...

Hello everyone! I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing as well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET and HEAD requests.

There is no resource cap or significant increase in usage is visible. Neither fast ec nor any other new features are enabled yet, just did the upgrade and left it as is. For measuring latency I have a services that probes the cluster with consistent load so it's not a difference in load either. PG status history does not show any pg operation being heavily in process.

Cluster has a hybrid build so OSDs have HDD for data and NVMe for db block. Both ec and replicated pools exist. The strange hint was: NVMe disks' iops went from something around 5-6K constant io per second to almost zero (less than a hundred per second, on all host/OSDs, with some non-frequent high peaks), while the HDD iops is not much changed. db blocks on NVMe are still attached to OSDs with no issue.

I can't find a clue to explain this. Got the same results on 3 separate clusters (all went version 18 --> 20, same hybrid build and configs). Nothing in config dump changes (changed defaults) catches my attention to have caused this.

Anyone had similar issues? Any tips and ideas on what to check? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io