Hi, I would like to understand why the OSD HDDs on node2 of my three identical ceph hosts claim to have processed 10 times more reads/writes than the other two nodes. OSD weights are all the similar, disk usage in space also, same disk sizes, same reported disk usage hours etc, etc. All data (QEMU VMs) is in one large 3/2 pool, called "ceph-storage". This is on ceph version 12.2.10. The spreadsheet with all the stats: https://docs.google.com/spreadsheets/d/1n8aOC1tpPPMi2iALhxfHzSQTRmfIz6wCERla... I hope the information is now both complete and readable, let me know if anything else is needed. Curious for any insights. The previously requested outputs below. root@node1:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 0 hdd 3.64000 1.00000 3.64TiB 2.01TiB 1.62TiB 55.35 0.98 137 1 hdd 3.64000 1.00000 3.64TiB 2.09TiB 1.54TiB 57.56 1.02 141 2 hdd 3.63689 1.00000 3.64TiB 1.92TiB 1.72TiB 52.80 0.94 128 3 hdd 3.64000 1.00000 3.64TiB 2.07TiB 1.57TiB 56.91 1.01 143 12 hdd 3.64000 1.00000 3.64TiB 2.15TiB 1.48TiB 59.19 1.05 138 13 hdd 3.64000 1.00000 3.64TiB 1.99TiB 1.64TiB 54.81 0.97 131 14 hdd 3.64000 1.00000 3.64TiB 1.93TiB 1.70TiB 53.14 0.94 127 15 hdd 3.64000 1.00000 3.64TiB 2.19TiB 1.45TiB 60.11 1.07 143 4 hdd 3.64000 1.00000 3.64TiB 2.11TiB 1.53TiB 57.98 1.03 142 5 hdd 3.64000 1.00000 3.64TiB 1.97TiB 1.67TiB 54.11 0.96 134 6 hdd 3.64000 1.00000 3.64TiB 2.12TiB 1.51TiB 58.41 1.04 142 7 hdd 3.64000 1.00000 3.64TiB 1.97TiB 1.66TiB 54.29 0.97 128 16 hdd 3.64000 1.00000 3.64TiB 2.00TiB 1.64TiB 54.90 0.98 133 17 hdd 3.64000 1.00000 3.64TiB 2.33TiB 1.30TiB 64.15 1.14 153 18 hdd 3.64000 1.00000 3.64TiB 1.97TiB 1.67TiB 54.08 0.96 132 19 hdd 3.64000 1.00000 3.64TiB 1.89TiB 1.75TiB 51.94 0.92 124 8 hdd 3.64000 1.00000 3.64TiB 1.79TiB 1.85TiB 49.25 0.88 123 9 hdd 3.64000 1.00000 3.64TiB 2.17TiB 1.46TiB 59.73 1.06 144 10 hdd 3.64000 1.00000 3.64TiB 2.40TiB 1.24TiB 65.89 1.17 157 11 hdd 3.64000 1.00000 3.64TiB 2.06TiB 1.58TiB 56.65 1.01 133 20 hdd 3.64000 1.00000 3.64TiB 2.19TiB 1.45TiB 60.24 1.07 148 21 hdd 3.64000 1.00000 3.64TiB 1.74TiB 1.90TiB 47.80 0.85 115 22 hdd 3.64000 1.00000 3.64TiB 2.05TiB 1.59TiB 56.28 1.00 138 23 hdd 3.63689 1.00000 3.64TiB 1.96TiB 1.67TiB 54.02 0.96 130 TOTAL 87.3TiB 49.1TiB 38.2TiB 56.23 MIN/MAX VAR: 0.85/1.17 STDDEV: 4.08 and root@node1:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 87.35376 root default -2 29.11688 host node1 0 hdd 3.64000 osd.0 up 1.00000 1.00000 1 hdd 3.64000 osd.1 up 1.00000 1.00000 2 hdd 3.63689 osd.2 up 1.00000 1.00000 3 hdd 3.64000 osd.3 up 1.00000 1.00000 12 hdd 3.64000 osd.12 up 1.00000 1.00000 13 hdd 3.64000 osd.13 up 1.00000 1.00000 14 hdd 3.64000 osd.14 up 1.00000 1.00000 15 hdd 3.64000 osd.15 up 1.00000 1.00000 -3 29.12000 host node2 4 hdd 3.64000 osd.4 up 1.00000 1.00000 5 hdd 3.64000 osd.5 up 1.00000 1.00000 6 hdd 3.64000 osd.6 up 1.00000 1.00000 7 hdd 3.64000 osd.7 up 1.00000 1.00000 16 hdd 3.64000 osd.16 up 1.00000 1.00000 17 hdd 3.64000 osd.17 up 1.00000 1.00000 18 hdd 3.64000 osd.18 up 1.00000 1.00000 19 hdd 3.64000 osd.19 up 1.00000 1.00000 -4 29.11688 host node3 8 hdd 3.64000 osd.8 up 1.00000 1.00000 9 hdd 3.64000 osd.9 up 1.00000 1.00000 10 hdd 3.64000 osd.10 up 1.00000 1.00000 11 hdd 3.64000 osd.11 up 1.00000 1.00000 20 hdd 3.64000 osd.20 up 1.00000 1.00000 21 hdd 3.64000 osd.21 up 1.00000 1.00000 22 hdd 3.64000 osd.22 up 1.00000 1.00000 23 hdd 3.63689 osd.23 up 1.00000 1.00000 MJ