Hi all, I am experiencing a critical memory issue with OSD daemons during sequential read tests. I would appreciate your advice on whether this is a known issue or if there is a flaw in my methodology. Environment Ceph Version: Tentacle v20.2.0Deployment: cephadmOSD Nodes: 8 nodes (128GB RAM each)OSD Configuration: 10 OSDs per node (Total 80 OSDs)Pool Type: Erasure Coding 6+2 (allow_ec_optimizations: true, allow_ec_overwrites: true)Workload: fio Sequential Read (12 clients, 40 jobs/client, BS=4M, QD=128, direct=1)Key Observations Random OSD Spikes: In every test run, a small number of OSD daemons exhibit a massive memory spike. However, these few daemons consume enough memory to exhaust the entire host’s RAM and swap space.Lack of Memory Reclamation: The memory is never reclaimed automatically after the workload stops. It remains occupied until I manually intervene (e.g., by restarting daemons or clearing heap free list of tcmalloc).Production Concern: My primary concern is that if this behavior occurs in a live production environment, it will inevitably lead to severe system-wide thrashing and service instability.Configuration Efforts: I have set osd_memory_target to 10G (with autotune off) and bluestore_cache_size to 7G, but the OSDs continue to ignore these targets during the read workload.Attempted Workarounds TCMALLOC_RELEASE_RATE=1000: No effect.TCMALLOC_HEAP_LIMIT_MB / Container Memory Limits: Problematic OSDs are repeatedly killed by OOM.TCMALLOC_AGGRESSIVE_DECOMMIT=true: Keeps memory stable but causes a 20% performance drop.Bluestore Cache Autotune: Controls memory but results in a 40% performance drop.Questions Is this behavior normal, or is there a flaw in my experimental methodology? Given that the memory usage seems abnormal, I am wondering if my fio configuration or OSD setup is fundamentally misconfigured for this version.Why does this spike occur on random OSDs during reads? Could this be related to how specific OSDs handle EC primary shards or is it a known TCMalloc fragmentation issue in v20.2.0?Is there a way to strictly enforce memory limits without such significant performance degradation?Any insights or suggestions for further debugging would be greatly appreciated. Best regards, Chanu Yu