Hello Igor, On Tue, May 02, 2023 at 05:41:04PM +0300, Igor Fedotov wrote:
Hi Nikola,
I'd suggest to start monitoring perf counters for your osds. op_w_lat/subop_w_lat ones specifically. I presume they raise eventually, don't they? OK, starting collecting those for all OSDs..
currently values for avgtime are around 0.0003 for subop_w_lat and 0.001-0.002 for op_w_lat I guess it'll need some time to find some trend, so I'll check tomorrow
Does subop_w_lat grow for every OSD or just a subset of them? How large is the delta between the best and the worst OSDs after a one week period? How many "bad" OSDs are at this point?
I'll see and report
And some more questions:
How large are space utilization/fragmentation for your OSDs?
OSD usage is around 16-18%. fragmentation should not be very bad, this cluster is deployed for few months only
Is the same performance drop observed for artificial benchmarks, e.g. 4k random writes to a fresh RBD image using fio?
will check again when the slowdown occurs and report
Is there any RAM utilization growth for OSD processes over time? Or may be any suspicious growth in mempool stats?
nope, RAM usage seems to be pretty constant. hewever, probably worh noting, historically we're using following OSD options: ceph config set osd bluestore_rocksdb_options compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB ceph config set osd bluestore_cache_autotune 0 ceph config set osd bluestore_cache_size_ssd 2G ceph config set osd bluestore_cache_kv_ratio 0.2 ceph config set osd bluestore_cache_meta_ratio 0.8 ceph config set osd osd_min_pg_log_entries 10 ceph config set osd osd_max_pg_log_entries 10 ceph config set osd osd_pg_log_dups_tracked 10 ceph config set osd osd_pg_log_trim_min 10 so maybe I'll start resetting those to defaults (ie enabling cache autotune etc) as a first step..
As a blind and brute force approach you might also want to compact RocksDB through ceph-kvstore-tool and switch bluestore allocator to bitmap (presuming default hybrid one is effective right now). Please do one modification at a time to realize what action is actually helpful if any.
will do.. thanks again for your hints BR nik
Thanks,
Igor
On 5/2/2023 11:32 AM, Nikola Ciprich wrote:
Hello dear CEPH users and developers,
we're dealing with strange problems.. we're having 12 node alma linux 9 cluster, initially installed CEPH 15.2.16, then upgraded to 17.2.5. It's running bunch of KVM virtual machines accessing volumes using RBD.
everything is working well, but there is strange and for us quite serious issue - speed of write operations (both sequential and random) is constantly degrading drastically to almost unusable numbers (in ~1week it drops from ~70k 4k writes/s from 1 VM to ~7k writes/s)
When I restart all OSD daemons, numbers immediately return to normal..
volumes are stored on replicated pool of 4 replicas, on top of 7*12 = 84 INTEL SSDPE2KX080T8 NVMEs.
I've updated cluster to 17.2.6 some time ago, but the problem persists. This is especially annoying in connection with https://tracker.ceph.com/issues/56896 as restarting OSDs is quite painfull when half of them crash..
I don't see anything suspicious, nodes load is quite low, no logs errors, network latency and throughput is OK too
Anyone having simimar issue?
I'd like to ask for hints on what should I check further..
we're running lots of 14.2.x and 15.2.x clusters, none showing similar issue, so I'm suspecting this is something related to quincy
thanks a lot in advance
with best regards
nikola ciprich
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
-- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@linuxbox.cz -------------------------------------