[ceph-users] If you know your cluster is performing as expected？

My ceph cluster was made of all-flash nvme, like this: 1. 10*nodes , every node has 21 nvme device was used to rgw data pool, and other 1 nvme was used to index pool ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -26 1467.11609 root bd-ssd03-data -43 146.71161 host bd-ssd03-node01-f6fe3bdc-a1b6-496f-8fff-adb240afaedf 0 nvme 6.98627 osd.0 up 1.00000 1.00000 4 nvme 6.98627 osd.4 up 1.00000 1.00000 11 nvme 6.98627 osd.11 up 1.00000 1.00000 21 nvme 6.98627 osd.21 up 1.00000 1.00000 31 nvme 6.98627 osd.31 up 1.00000 1.00000 46 nvme 6.98627 osd.46 up 1.00000 1.00000 56 nvme 6.98627 osd.56 up 1.00000 1.00000 67 nvme 6.98627 osd.67 up 1.00000 1.00000 77 nvme 6.98627 osd.77 up 1.00000 1.00000 87 nvme 6.98627 osd.87 up 1.00000 1.00000 97 nvme 6.98627 osd.97 up 1.00000 1.00000 107 nvme 6.98627 osd.107 up 1.00000 1.00000 117 nvme 6.98627 osd.117 up 1.00000 1.00000 127 nvme 6.98627 osd.127 up 1.00000 1.00000 137 nvme 6.98627 osd.137 up 1.00000 1.00000 147 nvme 6.98627 osd.147 up 1.00000 1.00000 157 nvme 6.98627 osd.157 up 1.00000 1.00000 168 nvme 6.98627 osd.168 up 1.00000 1.00000 178 nvme 6.98627 osd.178 up 1.00000 1.00000 188 nvme 6.98627 osd.188 up 1.00000 1.00000 198 nvme 6.98627 osd.198 up 1.00000 1.00000 -31 146.71161 host bd-ssd03-node02-ffedbbd9-3794-4b10-ac1d-bf7595395892 1 nvme 6.98627 osd.1 up 1.00000 1.00000 6 nvme 6.98627 osd.6 up 1.00000 1.00000 14 nvme 6.98627 osd.14 up 1.00000 1.00000 24 nvme 6.98627 osd.24 up 1.00000 1.00000 34 nvme 6.98627 osd.34 up 1.00000 1.00000 45 nvme 6.98627 osd.45 up 1.00000 1.00000 55 nvme 6.98627 osd.55 up 1.00000 1.00000 66 nvme 6.98627 osd.66 up 1.00000 1.00000 76 nvme 6.98627 osd.76 up 1.00000 1.00000 86 nvme 6.98627 osd.86 up 1.00000 1.00000 96 nvme 6.98627 osd.96 up 1.00000 1.00000 106 nvme 6.98627 osd.106 up 1.00000 1.00000 116 nvme 6.98627 osd.116 up 1.00000 1.00000 130 nvme 6.98627 osd.130 up 1.00000 1.00000 141 nvme 6.98627 osd.141 up 1.00000 1.00000 151 nvme 6.98627 osd.151 up 1.00000 1.00000 161 nvme 6.98627 osd.161 up 1.00000 1.00000 171 nvme 6.98627 osd.171 up 1.00000 1.00000 181 nvme 6.98627 osd.181 up 1.00000 1.00000 191 nvme 6.98627 osd.191 up 1.00000 1.00000 201 nvme 6.98627 osd.201 up 1.00000 1.00000 -29 146.71161 host bd-ssd03-node03-637c96a1-6b93-42cc-9b35-44b729bd7a5c 2 nvme 6.98627 osd.2 up 1.00000 1.00000 8 nvme 6.98627 osd.8 up 1.00000 1.00000 17 nvme 6.98627 osd.17 up 1.00000 1.00000 27 nvme 6.98627 osd.27 up 1.00000 1.00000 37 nvme 6.98627 osd.37 up 1.00000 1.00000 48 nvme 6.98627 osd.48 up 1.00000 1.00000 58 nvme 6.98627 osd.58 up 1.00000 1.00000 68 nvme 6.98627 osd.68 up 1.00000 1.00000 79 nvme 6.98627 osd.79 up 1.00000 1.00000 89 nvme 6.98627 osd.89 up 1.00000 1.00000 99 nvme 6.98627 osd.99 up 1.00000 1.00000 109 nvme 6.98627 osd.109 up 1.00000 1.00000 119 nvme 6.98627 osd.119 up 1.00000 1.00000 129 nvme 6.98627 osd.129 up 1.00000 1.00000 140 nvme 6.98627 osd.140 up 1.00000 1.00000 150 nvme 6.98627 osd.150 up 1.00000 1.00000 166 nvme 6.98627 osd.166 up 1.00000 1.00000 176 nvme 6.98627 osd.176 up 1.00000 1.00000 186 nvme 6.98627 osd.186 up 1.00000 1.00000 196 nvme 6.98627 osd.196 up 1.00000 1.00000 206 nvme 6.98627 osd.206 up 1.00000 1.00000 -41 146.71161 host bd-ssd03-node04-87186f77-3df0-4b71-8103-fdc2a104eb17 3 nvme 6.98627 osd.3 up 1.00000 1.00000 10 nvme 6.98627 osd.10 up 1.00000 1.00000 20 nvme 6.98627 osd.20 up 1.00000 1.00000 30 nvme 6.98627 osd.30 up 1.00000 1.00000 40 nvme 6.98627 osd.40 up 1.00000 1.00000 50 nvme 6.98627 osd.50 up 1.00000 1.00000 60 nvme 6.98627 osd.60 up 1.00000 1.00000 70 nvme 6.98627 osd.70 up 1.00000 1.00000 81 nvme 6.98627 osd.81 up 1.00000 1.00000 90 nvme 6.98627 osd.90 up 1.00000 1.00000 100 nvme 6.98627 osd.100 up 1.00000 1.00000 110 nvme 6.98627 osd.110 up 1.00000 1.00000 120 nvme 6.98627 osd.120 up 1.00000 1.00000 132 nvme 6.98627 osd.132 up 1.00000 1.00000 143 nvme 6.98627 osd.143 up 1.00000 1.00000 153 nvme 6.98627 osd.153 up 1.00000 1.00000 162 nvme 6.98627 osd.162 up 1.00000 1.00000 172 nvme 6.98627 osd.172 up 1.00000 1.00000 182 nvme 6.98627 osd.182 up 1.00000 1.00000 192 nvme 6.98627 osd.192 up 1.00000 1.00000 202 nvme 6.98627 osd.202 up 1.00000 1.00000 -25 146.71161 host bd-ssd03-node05-a6f31379-cb27-4bc3-bbda-726023d9ff96 5 nvme 6.98627 osd.5 up 1.00000 1.00000 13 nvme 6.98627 osd.13 up 1.00000 1.00000 23 nvme 6.98627 osd.23 up 1.00000 1.00000 33 nvme 6.98627 osd.33 up 1.00000 1.00000 43 nvme 6.98627 osd.43 up 1.00000 1.00000 53 nvme 6.98627 osd.53 up 1.00000 1.00000 64 nvme 6.98627 osd.64 up 1.00000 1.00000 78 nvme 6.98627 osd.78 up 1.00000 1.00000 88 nvme 6.98627 osd.88 up 1.00000 1.00000 98 nvme 6.98627 osd.98 up 1.00000 1.00000 108 nvme 6.98627 osd.108 up 1.00000 1.00000 118 nvme 6.98627 osd.118 up 1.00000 1.00000 128 nvme 6.98627 osd.128 up 1.00000 1.00000 139 nvme 6.98627 osd.139 up 1.00000 1.00000 149 nvme 6.98627 osd.149 up 1.00000 1.00000 159 nvme 6.98627 osd.159 up 1.00000 1.00000 169 nvme 6.98627 osd.169 up 1.00000 1.00000 179 nvme 6.98627 osd.179 up 1.00000 1.00000 189 nvme 6.98627 osd.189 up 1.00000 1.00000 199 nvme 6.98627 osd.199 up 1.00000 1.00000 209 nvme 6.98627 osd.209 up 1.00000 1.00000 -33 146.71161 host bd-ssd03-node06-e4288b8f-7d64-40c1-a3f1-c60757e1fbb1 7 nvme 6.98627 osd.7 up 1.00000 1.00000 16 nvme 6.98627 osd.16 up 1.00000 1.00000 26 nvme 6.98627 osd.26 up 1.00000 1.00000 36 nvme 6.98627 osd.36 up 1.00000 1.00000 51 nvme 6.98627 osd.51 up 1.00000 1.00000 62 nvme 6.98627 osd.62 up 1.00000 1.00000 71 nvme 6.98627 osd.71 up 1.00000 1.00000 82 nvme 6.98627 osd.82 up 1.00000 1.00000 93 nvme 6.98627 osd.93 up 1.00000 1.00000 103 nvme 6.98627 osd.103 up 1.00000 1.00000 113 nvme 6.98627 osd.113 up 1.00000 1.00000 123 nvme 6.98627 osd.123 up 1.00000 1.00000 134 nvme 6.98627 osd.134 up 1.00000 1.00000 145 nvme 6.98627 osd.145 up 1.00000 1.00000 155 nvme 6.98627 osd.155 up 1.00000 1.00000 165 nvme 6.98627 osd.165 up 1.00000 1.00000 175 nvme 6.98627 osd.175 up 1.00000 1.00000 185 nvme 6.98627 osd.185 up 1.00000 1.00000 195 nvme 6.98627 osd.195 up 1.00000 1.00000 208 nvme 6.98627 osd.208 up 1.00000 1.00000 215 nvme 6.98627 osd.215 up 1.00000 1.00000 -37 146.71161 host bd-ssd03-node07-fabbd3f5-1466-493b-85f4-dac9254720c4 9 nvme 6.98627 osd.9 up 1.00000 1.00000 18 nvme 6.98627 osd.18 up 1.00000 1.00000 28 nvme 6.98627 osd.28 up 1.00000 1.00000 44 nvme 6.98627 osd.44 up 1.00000 1.00000 54 nvme 6.98627 osd.54 up 1.00000 1.00000 65 nvme 6.98627 osd.65 up 1.00000 1.00000 75 nvme 6.98627 osd.75 up 1.00000 1.00000 85 nvme 6.98627 osd.85 up 1.00000 1.00000 95 nvme 6.98627 osd.95 up 1.00000 1.00000 105 nvme 6.98627 osd.105 up 1.00000 1.00000 115 nvme 6.98627 osd.115 up 1.00000 1.00000 125 nvme 6.98627 osd.125 up 1.00000 1.00000 136 nvme 6.98627 osd.136 up 1.00000 1.00000 146 nvme 6.98627 osd.146 up 1.00000 1.00000 156 nvme 6.98627 osd.156 up 1.00000 1.00000 167 nvme 6.98627 osd.167 up 1.00000 1.00000 177 nvme 6.98627 osd.177 up 1.00000 1.00000 187 nvme 6.98627 osd.187 up 1.00000 1.00000 197 nvme 6.98627 osd.197 up 1.00000 1.00000 207 nvme 6.98627 osd.207 up 1.00000 1.00000 214 nvme 6.98627 osd.214 up 1.00000 1.00000 -35 146.71161 host bd-ssd03-node08-65ab0995-22d0-4b4c-b661-4cc53ceb5b61 12 nvme 6.98627 osd.12 up 1.00000 1.00000 22 nvme 6.98627 osd.22 up 1.00000 1.00000 32 nvme 6.98627 osd.32 up 1.00000 1.00000 42 nvme 6.98627 osd.42 up 1.00000 1.00000 52 nvme 6.98627 osd.52 up 1.00000 1.00000 63 nvme 6.98627 osd.63 up 1.00000 1.00000 73 nvme 6.98627 osd.73 up 1.00000 1.00000 84 nvme 6.98627 osd.84 up 1.00000 1.00000 94 nvme 6.98627 osd.94 up 1.00000 1.00000 104 nvme 6.98627 osd.104 up 1.00000 1.00000 114 nvme 6.98627 osd.114 up 1.00000 1.00000 124 nvme 6.98627 osd.124 up 1.00000 1.00000 138 nvme 6.98627 osd.138 up 1.00000 1.00000 148 nvme 6.98627 osd.148 up 1.00000 1.00000 158 nvme 6.98627 osd.158 up 1.00000 1.00000 170 nvme 6.98627 osd.170 up 1.00000 1.00000 180 nvme 6.98627 osd.180 up 1.00000 1.00000 190 nvme 6.98627 osd.190 up 1.00000 1.00000 200 nvme 6.98627 osd.200 up 1.00000 1.00000 210 nvme 6.98627 osd.210 up 1.00000 1.00000 216 nvme 6.98627 osd.216 up 1.00000 1.00000 -39 146.71161 host bd-ssd03-node09-4833dda0-f95f-4116-97fa-635ee8685ba2 15 nvme 6.98627 osd.15 up 1.00000 1.00000 25 nvme 6.98627 osd.25 up 1.00000 1.00000 35 nvme 6.98627 osd.35 up 1.00000 1.00000 47 nvme 6.98627 osd.47 up 1.00000 1.00000 61 nvme 6.98627 osd.61 up 1.00000 1.00000 72 nvme 6.98627 osd.72 up 1.00000 1.00000 83 nvme 6.98627 osd.83 up 1.00000 1.00000 92 nvme 6.98627 osd.92 up 1.00000 1.00000 102 nvme 6.98627 osd.102 up 1.00000 1.00000 112 nvme 6.98627 osd.112 up 1.00000 1.00000 122 nvme 6.98627 osd.122 up 1.00000 1.00000 133 nvme 6.98627 osd.133 up 1.00000 1.00000 144 nvme 6.98627 osd.144 up 1.00000 1.00000 154 nvme 6.98627 osd.154 up 1.00000 1.00000 164 nvme 6.98627 osd.164 up 1.00000 1.00000 174 nvme 6.98627 osd.174 up 1.00000 1.00000 184 nvme 6.98627 osd.184 up 1.00000 1.00000 194 nvme 6.98627 osd.194 up 1.00000 1.00000 204 nvme 6.98627 osd.204 up 1.00000 1.00000 213 nvme 6.98627 osd.213 up 1.00000 1.00000 218 nvme 6.98627 osd.218 up 1.00000 1.00000 -45 146.71161 host bd-ssd03-node10-9cfe177c-2e0f-43d3-a1a9-6524dac0009a 19 nvme 6.98627 osd.19 up 1.00000 1.00000 29 nvme 6.98627 osd.29 up 1.00000 1.00000 39 nvme 6.98627 osd.39 up 1.00000 1.00000 49 nvme 6.98627 osd.49 up 1.00000 1.00000 59 nvme 6.98627 osd.59 up 1.00000 1.00000 69 nvme 6.98627 osd.69 up 1.00000 1.00000 80 nvme 6.98627 osd.80 up 1.00000 1.00000 91 nvme 6.98627 osd.91 up 1.00000 1.00000 101 nvme 6.98627 osd.101 up 1.00000 1.00000 111 nvme 6.98627 osd.111 up 1.00000 1.00000 121 nvme 6.98627 osd.121 up 1.00000 1.00000 131 nvme 6.98627 osd.131 up 1.00000 1.00000 142 nvme 6.98627 osd.142 up 1.00000 1.00000 152 nvme 6.98627 osd.152 up 1.00000 1.00000 163 nvme 6.98627 osd.163 up 1.00000 1.00000 173 nvme 6.98627 osd.173 up 1.00000 1.00000 183 nvme 6.98627 osd.183 up 1.00000 1.00000 193 nvme 6.98627 osd.193 up 1.00000 1.00000 203 nvme 6.98627 osd.203 up 1.00000 1.00000 212 nvme 6.98627 osd.212 up 1.00000 1.00000 217 nvme 6.98627 osd.217 up 1.00000 1.00000 -4 14.55475 root bd-ssd03-index -11 1.45547 host bd-ssd03-node01-0581ef65-8df6-4cfc-8992-08ffb7959914 41 nvme 1.45547 osd.41 up 1.00000 1.00000 -13 1.45547 host bd-ssd03-node02-0025ee25-77d9-4731-bc94-1c541e1fa2e7 126 nvme 1.45547 osd.126 up 1.00000 1.00000 -3 1.45547 host bd-ssd03-node03-a2629c55-9ddc-4c16-ba85-187af70eef23 160 nvme 1.45547 osd.160 up 1.00000 1.00000 -9 1.45547 host bd-ssd03-node04-9e02d968-28b5-43a8-b93a-418cb61af7ee 211 nvme 1.45547 osd.211 up 1.00000 1.00000 -15 1.45547 host bd-ssd03-node05-536f9f5e-cf15-4bb5-abaa-afd313473ab1 74 nvme 1.45547 osd.74 up 1.00000 1.00000 -23 1.45547 host bd-ssd03-node06-43949775-80b1-4297-b23f-d36ffb784d76 205 nvme 1.45547 osd.205 up 1.00000 1.00000 -21 1.45547 host bd-ssd03-node07-11ca9165-47b2-4b1d-84d7-c6b4f62f51eb 38 nvme 1.45547 osd.38 up 1.00000 1.00000 -7 1.45547 host bd-ssd03-node08-d6739f0c-a356-415c-ae4d-323cb4197efe 135 nvme 1.45547 osd.135 up 1.00000 1.00000 -17 1.45547 host bd-ssd03-node09-ece27fca-7711-45f1-b899-6d6f61aad2cf 57 nvme 1.45547 osd.57 up 1.00000 1.00000 -19 1.45547 host bd-ssd03-node10-c53d891c-b393-4663-88bf-f2ec4e9e0a9a 219 nvme 1.45547 osd.219 up 1.00000 1.00000 2. memory 512G [root@bd-ssd03-node01 ~]# free -g total used free shared buff/cache available Mem: 503 105 134 0 262 393 Swap: 0 0 0 3. cpu 128 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-255 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 2 NUMA node(s): 2 Vendor ID: AuthenticAMD BIOS Vendor ID: Advanced Micro Devices, Inc. CPU family: 25 Model: 1 Model name: AMD EPYC 7763 64-Core Processor BIOS Model name: AMD EPYC 7763 64-Core Processor Stepping: 1 CPU MHz: 3243.566 CPU max MHz: 3529.0520 CPU min MHz: 1500.0000 BogoMIPS: 4890.70 Virtualization: AMD-V L1d cache: 32K L1i cache: 32K L2 cache: 512K L3 cache: 32768K 4. Network 50Gb/s Settings for bond4: Supported ports: [ ] Supported link modes: Not reported Supported pause frame use: No Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 50000Mb/s Duplex: Full Auto-negotiation: off Port: Other PHYAD: 0 Transceiver: internal Link detected: yes I'm using cosbench to test the rgw, the xml config of the cosbench is 10 driver to 10 rgw and every driver with 16 works: <?xml version="1.0" encoding="UTF-8" ?> <workload name="test" description="sample benchmark for s3"> <workflow> <workstage name="create bucket" closuredelay="0" config=""> <auth type="none" config=""/> <work name="rgw1-create" type="init" workers="1" interval="5" division="container" totalOps="2560000" rampup="0" rampdown="0" afr="0" totalBytes="0" config="cprefix=bdssd03-bucket;containers=r(1,10)"> <auth type="none" config=""/> <storage type="s3" config="accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.143:80;path_style_access=true"/> <operation type="init" ratio="100" division="container" config="containers=r(1,1);objects=r(0,0);sizes=c(0)B" id="none"/> </work> </workstage> <workstage name="write"> <work name="w1" workers="16" totalOps="2560000" driver="driver1"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(1);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.144:80;server_side_encryption=false; "/> </work> <work name="w2" workers="16" totalOps="2560000" driver="driver2"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(2);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.145:80;server_side_encryption=false;" /> </work> <work name="w3" workers="16" totalOps="2560000" driver="driver3"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(3);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.146:80;server_side_encryption=false;" /> </work> <work name="w4" workers="16" totalOps="2560000" driver="driver4"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(4);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.147:80;server_side_encryption=false; "/> </work> <work name="w5" workers="16" totalOps="2560000" driver="driver5"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(5);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.148:80;server_side_encryption=false;" /> </work> <work name="w6" workers="16" totalOps="2560000" driver="driver6"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(6);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.149:80;server_side_encryption=false;" /> </work> <work name="w7" workers="16" totalOps="2560000" driver="driver7"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(7);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.150:80;server_side_encryption=false; "/> </work> <work name="w8" workers="16" totalOps="2560000" driver="driver8"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(8);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.151:80;server_side_encryption=false;" /> </work> <work name="w9" workers="16" totalOps="2560000" driver="driver9"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(9);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.152:80;server_side_encryption=false;" /> </work> <work name="w10" workers="16" totalOps="2560000" driver="driver10"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(10);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.143:80;server_side_encryption=false; "/> </work> </workstage> <workstage name="cleanup"> <work type="cleanup" workers="16" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=r(1,10);objects=r(1,2000000)" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.143:80;server_side_encryption=false;" /> </workstage> <workstage name="dispose"> <work type="dispose" workers="10" config="cprefix=bdssd03-bucket;containers=r(1,10)" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.143:80;server_side_encryption=false;" /> </workstage> </workflow> </workload> the performance result is: cluster: id: 075b70fa-6a72-4ce2-8956-d249c7cf10ce health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 17h) mgr: a(active, since 17h), standbys: b osd: 220 osds: 220 up (since 17h), 220 in (since 17h) rgw: 10 daemons active (10 hosts, 1 zones) data: pools: 9 pools, 4273 pgs objects: 17.76M objects, 65 TiB usage: 82 TiB used, 1.4 PiB / 1.4 PiB avail pgs: 4273 active+clean io: client: 11 MiB/s rd, 9.2 GiB/s wr, 15.44k op/s rd, 28.70k op/s wr only can reach to 9.2 GiB with 4MB write, I don't know that whether the performance meet expectations?