Environment: Ceph Nautilus 14.2.8 Object Storage Data nodes: 12 * HDD OSDs drives each with a 12TB capacity + 2 * SSD OSDs drives for rgw bucket index pool & rgw meta pool. Custom configs (since we dealing with a majority smaller sized objects) bluestore_min_alloc_size_ssd 4096 bluestore_min_alloc_size_hdd 4096 Observations from cosbench performance tests Stage Op-Type Op-Count Byte-Count Avg-ResTime s7-read1KB 48W read 2004202 2004202000 43.11 s13-read2KB 48W read 2013906 4027812000 42.9 s19-read4KB 48W read 2014701 8058804000 42.88 s25-read8KB 48W read 2002337 16018696000 43.15 s31-read16KB 48W read 1987785 31804560000 43.46 s37-read32KB 48W read 1976190 63238080000 43.7 s43-read64KB 48W read 1929183 123467712000 44.78 s49-read128KB 48W read 9965032 1275524096000 8.67 s55-read256KB 48W read 6505554 1665421824000 13.28 The response time improves drastically when the object size is greater than 64KB. What could be the reason? Thanks, Ronnie [https://d1dejaj6dcqv24.cloudfront.net/asset/image/email-banner-384-2x.png]<https://www.qualys.com/email-banner> This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Qualys email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails ("spam"). If you have any concerns about this process, please contact us.