I'm tryting to debug low performance on nvme-based cluster. I have 24 NVME in 4 servers, plenty of CPU, cluster perfectly balanced, no scrubbing or replication atm, 1024 pg for 24 OSD/17TB. I expect to see some performance (~250k IOPS total, 50% r/w, few hundred volumes with capped by IOPS, pre-warmed). I see about half of it. I looked at drive utilization, it's about 70% (per atop), but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). I looked at osd (they loaded uniformely, so any of them shows the same results). I see "op_wip": 10-27, but in-flight value is about 0-2, mostly around 1. I can't get away from the feeling that somehow osd is doing operations (almost) sequential. I already played with osd_op_num_threads_per_shard (4), osd_op_num_shards (8), set mclock profile to high_client_ops, ms_async_op_threads 24, and it can't get more inflight ios. I feel I miss something. How to make Ceph to send more requests to underlaying NVME?