squid 19.2.2 - added disk does not reflect in available capacity on pools
Hi, I have added 42 x 18TB HDD disks ( 6 on each of the 7 servers ) My expectation was that the pools configured to use "hdd_class" will have their capacity increased ( e.g. default.rgw.buckets.data which is uses an EC 4+2 pool for data ) It seems it is not happening ...yet ?! Is it because the peering is still going ? Many thanks Steven Below are outputs from ceph -s ceph df ceph osd df tree cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore 1 failed cephadm daemon(s) 569 pgs not deep-scrubbed in time 2609 pgs not scrubbed in time 39 daemons have recently crashed services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 2m) mgr: ceph-host-1.lqlece(active, since 18h), standbys: ceph-host-2.suiuxi mds: 19/19 daemons up, 7 standby osd: 181 osds: 181 up (since 4d), 181 in (since 14h); 2770 remapped pgs flags noautoscale data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 256.82M objects, 484 TiB usage: 742 TiB used, 1.5 PiB / 2.2 PiB avail pgs: 575889786/1468742421 objects misplaced (39.210%) 4247 active+clean 2763 active+remapped+backfill_wait 77 active+clean+scrubbing 43 active+clean+scrubbing+deep 7 active+remapped+backfilling io: client: 84 B/s rd, 2.8 KiB/s wr, 0 op/s rd, 1 op/s wr recovery: 384 MiB/s, 128 objects/s --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54 nvme_class 293 TiB 293 TiB 216 GiB 216 GiB 0.07 ssd_class 587 TiB 424 TiB 163 TiB 163 TiB 27.80 TOTAL 2.2 PiB 1.5 PiB 742 TiB 742 TiB 32.64 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB metadata_fs_nvme 13 32 260 MiB 177 780 MiB 0 93 TiB metadata_fs_ssd_rep 14 32 17 MiB 103 51 MiB 0 93 TiB ssd_rep_projects 15 1024 132 B 1 12 KiB 0 130 TiB nvme_rep_projects 16 512 3.5 KiB 30 336 KiB 0 93 TiB ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2272.93311 - 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 1.00 - root default -7 254.54175 - 255 TiB 104 TiB 102 TiB 9.6 GiB 455 GiB 151 TiB 40.78 1.25 - host ceph-host-1 107 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 4 KiB 48 GiB 5.2 TiB 71.71 2.20 422 up osd.107 110 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 4 KiB 45 GiB 5.3 TiB 71.30 2.18 421 up osd.110 116 hdd_class 18.43300 0.95001 18 TiB 13 TiB 13 TiB 4 KiB 43 GiB 5.3 TiB 71.50 2.19 424 up osd.116 123 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 14 KiB 44 GiB 5.5 TiB 70.20 2.15 414 up osd.123 130 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 56 GiB 4.2 TiB 76.99 2.36 451 up osd.130 137 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 50 GiB 4.3 TiB 76.47 2.34 449 up osd.137 152 hdd_class 18.19040 1.00000 18 TiB 241 GiB 239 GiB 10 KiB 1.8 GiB 18 TiB 1.29 0.04 7 up osd.152 2 nvme_class 13.97260 1.00000 14 TiB 14 GiB 906 MiB 3.9 GiB 8.9 GiB 14 TiB 0.10 0.00 300 up osd.2 8 nvme_class 13.97260 1.00000 14 TiB 7.2 GiB 846 MiB 2.5 GiB 3.9 GiB 14 TiB 0.05 0.00 293 up osd.8 15 nvme_class 13.97260 1.00000 14 TiB 11 GiB 887 MiB 3.3 GiB 6.9 GiB 14 TiB 0.08 0.00 288 up osd.15 23 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 6 KiB 14 GiB 5.2 TiB 26.28 0.81 106 up osd.23 30 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.4 GiB 5.0 TiB 27.73 0.85 114 up osd.30 37 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 6 KiB 15 GiB 5.2 TiB 26.28 0.80 104 up osd.37 44 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.3 GiB 5.1 TiB 27.34 0.84 111 up osd.44 52 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 4 KiB 15 GiB 5.1 TiB 26.35 0.81 104 up osd.52 59 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.0 TiB 27.79 0.85 107 up osd.59 66 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 4 KiB 14 GiB 5.1 TiB 26.29 0.81 104 up osd.66 72 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.8 TiB 4 KiB 15 GiB 5.1 TiB 26.68 0.82 107 up osd.72 79 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 15 GiB 5.1 TiB 27.05 0.83 111 up osd.79 86 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 4 KiB 10 GiB 5.0 TiB 28.49 0.87 116 up osd.86 93 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 5.5 GiB 5.1 TiB 27.29 0.84 112 up osd.93 100 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 15 GiB 5.0 TiB 28.19 0.86 114 up osd.100 -15 345.49374 - 345 TiB 106 TiB 104 TiB 8.7 GiB 523 GiB 239 TiB 30.69 0.94 - host ceph-host-2 115 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 7 KiB 43 GiB 5.9 TiB 68.17 2.09 402 up osd.115 122 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 2 KiB 45 GiB 5.4 TiB 70.72 2.17 420 up osd.122 134 hdd_class 18.43300 1.00000 18 TiB 14 TiB 13 TiB 12 KiB 52 GiB 4.7 TiB 74.54 2.28 440 up osd.134 141 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 54 GiB 4.3 TiB 76.58 2.35 447 up osd.141 144 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 4 KiB 56 GiB 4.5 TiB 75.77 2.32 443 up osd.144 146 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 7 KiB 52 GiB 4.4 TiB 76.13 2.33 449 up osd.146 159 hdd_class 18.19040 1.00000 18 TiB 203 GiB 200 GiB 10 KiB 2.4 GiB 18 TiB 1.09 0.03 6 up osd.159 176 hdd_class 18.19040 1.00000 18 TiB 135 GiB 133 GiB 6 KiB 1.4 GiB 18 TiB 0.72 0.02 4 up osd.176 177 hdd_class 18.19040 1.00000 18 TiB 74 GiB 73 GiB 6 KiB 1.7 GiB 18 TiB 0.40 0.01 2 up osd.177 178 hdd_class 18.19040 1.00000 18 TiB 238 GiB 236 GiB 7 KiB 2.3 GiB 18 TiB 1.28 0.04 5 up osd.178 179 hdd_class 18.19040 1.00000 18 TiB 76 GiB 75 GiB 6 KiB 828 MiB 18 TiB 0.41 0.01 2 up osd.179 180 hdd_class 18.19040 1.00000 18 TiB 198 GiB 197 GiB 10 KiB 1.7 GiB 18 TiB 1.07 0.03 5 up osd.180 6 nvme_class 13.97260 1.00000 14 TiB 11 GiB 903 MiB 2.9 GiB 7.0 GiB 14 TiB 0.08 0.00 282 up osd.6 13 nvme_class 13.97260 1.00000 14 TiB 9.4 GiB 1.1 GiB 2.7 GiB 5.6 GiB 14 TiB 0.07 0.00 300 up osd.13 20 nvme_class 13.97260 1.00000 14 TiB 11 GiB 862 MiB 3.1 GiB 7.2 GiB 14 TiB 0.08 0.00 288 up osd.20 26 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.58 0.88 108 up osd.26 32 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.21 0.86 107 up osd.32 41 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.23 0.86 113 up osd.41 48 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 11 KiB 16 GiB 5.0 TiB 28.19 0.86 114 up osd.48 53 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 15 GiB 5.0 TiB 28.21 0.86 107 up osd.53 61 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.22 0.86 112 up osd.61 68 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.19 0.86 107 up osd.68 74 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.56 0.88 114 up osd.74 82 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 9 KiB 16 GiB 5.0 TiB 27.79 0.85 106 up osd.82 89 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 9 KiB 16 GiB 5.0 TiB 27.85 0.85 108 up osd.89 95 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 11 KiB 16 GiB 5.0 TiB 28.58 0.88 112 up osd.95 104 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 9 KiB 16 GiB 5.0 TiB 27.83 0.85 108 up osd.104 -13 345.49374 - 345 TiB 108 TiB 106 TiB 9.5 GiB 423 GiB 238 TiB 31.16 0.95 - host ceph-host-3 114 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 4 KiB 42 GiB 6.3 TiB 66.07 2.02 389 up osd.114 121 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 9 KiB 44 GiB 5.9 TiB 67.77 2.08 400 up osd.121 127 hdd_class 18.43300 1.00000 18 TiB 14 TiB 13 TiB 14 KiB 48 GiB 4.7 TiB 74.32 2.28 437 up osd.127 133 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 54 GiB 4.0 TiB 78.33 2.40 461 up osd.133 140 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 2 KiB 50 GiB 4.4 TiB 76.20 2.33 447 up osd.140 143 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 11 KiB 50 GiB 4.4 TiB 76.39 2.34 446 up osd.143 154 hdd_class 18.19040 1.00000 18 TiB 281 GiB 279 GiB 9 KiB 2.3 GiB 18 TiB 1.51 0.05 8 up osd.154 158 hdd_class 18.19040 1.00000 18 TiB 725 GiB 722 GiB 20 KiB 3.0 GiB 17 TiB 3.89 0.12 22 up osd.158 163 hdd_class 18.19040 1.00000 18 TiB 696 GiB 692 GiB 22 KiB 3.2 GiB 18 TiB 3.73 0.11 22 up osd.163 167 hdd_class 18.19040 1.00000 18 TiB 458 GiB 455 GiB 15 KiB 2.4 GiB 18 TiB 2.46 0.08 11 up osd.167 171 hdd_class 18.19040 1.00000 18 TiB 788 GiB 785 GiB 25 KiB 3.1 GiB 17 TiB 4.23 0.13 23 up osd.171 175 hdd_class 18.19040 1.00000 18 TiB 518 GiB 515 GiB 22 KiB 3.0 GiB 18 TiB 2.78 0.09 15 up osd.175 4 nvme_class 13.97260 1.00000 14 TiB 9.1 GiB 1.1 GiB 2.8 GiB 5.2 GiB 14 TiB 0.06 0.00 283 up osd.4 11 nvme_class 13.97260 1.00000 14 TiB 10 GiB 848 MiB 3.0 GiB 6.4 GiB 14 TiB 0.07 0.00 289 up osd.11 19 nvme_class 13.97260 1.00000 14 TiB 9.1 GiB 922 MiB 3.6 GiB 4.6 GiB 14 TiB 0.06 0.00 289 up osd.19 25 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.8 GiB 5.0 TiB 28.42 0.87 113 up osd.25 31 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.05 0.86 110 up osd.31 38 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.1 GiB 5.1 TiB 27.34 0.84 110 up osd.38 45 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.1 GiB 5.0 TiB 27.74 0.85 108 up osd.45 51 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 14 GiB 5.0 TiB 28.16 0.86 111 up osd.51 58 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 12 GiB 5.1 TiB 27.03 0.83 105 up osd.58 65 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 14 GiB 5.1 TiB 27.04 0.83 109 up osd.65 73 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.1 TiB 27.03 0.83 109 up osd.73 80 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.9 GiB 5.0 TiB 28.41 0.87 110 up osd.80 87 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.8 GiB 5.0 TiB 28.42 0.87 114 up osd.87 94 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.10 0.86 114 up osd.94 101 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.2 GiB 5.0 TiB 28.07 0.86 113 up osd.101 -11 345.49374 - 345 TiB 107 TiB 105 TiB 8.2 GiB 439 GiB 239 TiB 30.95 0.95 - host ceph-host-4 108 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 4 KiB 43 GiB 6.1 TiB 67.05 2.05 394 up osd.108 113 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 7 KiB 47 GiB 6.3 TiB 65.92 2.02 387 up osd.113 120 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 2 KiB 40 GiB 6.2 TiB 66.60 2.04 389 up osd.120 126 hdd_class 18.43300 0.95001 18 TiB 15 TiB 15 TiB 7 KiB 58 GiB 3.6 TiB 80.37 2.46 473 up osd.126 132 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 52 GiB 4.1 TiB 78.00 2.39 457 up osd.132 139 hdd_class 18.43300 1.00000 18 TiB 15 TiB 14 TiB 14 KiB 53 GiB 3.9 TiB 78.87 2.42 463 up osd.139 151 hdd_class 18.19040 1.00000 18 TiB 38 GiB 37 GiB 6 KiB 1.1 GiB 18 TiB 0.20 0.01 1 up osd.151 156 hdd_class 18.19040 1.00000 18 TiB 205 GiB 203 GiB 12 KiB 1.9 GiB 18 TiB 1.10 0.03 6 up osd.156 161 hdd_class 18.19040 1.00000 18 TiB 853 GiB 849 GiB 4 KiB 3.9 GiB 17 TiB 4.58 0.14 26 up osd.161 165 hdd_class 18.19040 1.00000 18 TiB 569 GiB 566 GiB 22 KiB 2.7 GiB 18 TiB 3.05 0.09 15 up osd.165 169 hdd_class 18.19040 1.00000 18 TiB 891 GiB 888 GiB 24 KiB 3.3 GiB 17 TiB 4.78 0.15 25 up osd.169 173 hdd_class 18.19040 1.00000 18 TiB 591 GiB 589 GiB 24 KiB 2.6 GiB 18 TiB 3.17 0.10 19 up osd.173 3 nvme_class 13.97260 1.00000 14 TiB 10 GiB 929 MiB 2.8 GiB 6.3 GiB 14 TiB 0.07 0.00 283 up osd.3 10 nvme_class 13.97260 1.00000 14 TiB 8.7 GiB 872 MiB 2.6 GiB 5.3 GiB 14 TiB 0.06 0.00 271 up osd.10 17 nvme_class 13.97260 1.00000 14 TiB 9.5 GiB 904 MiB 2.8 GiB 5.8 GiB 14 TiB 0.07 0.00 298 up osd.17 24 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.3 GiB 5.0 TiB 28.11 0.86 112 up osd.24 33 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.41 0.87 113 up osd.33 40 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.8 GiB 5.0 TiB 28.44 0.87 115 up osd.40 47 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 14 GiB 5.1 TiB 27.03 0.83 106 up osd.47 55 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 15 GiB 5.0 TiB 28.22 0.86 108 up osd.55 60 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 14 GiB 5.1 TiB 27.06 0.83 111 up osd.60 67 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.3 GiB 5.1 TiB 27.28 0.84 110 up osd.67 75 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 109 up osd.75 83 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 16 GiB 5.1 TiB 27.05 0.83 105 up osd.83 88 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 5.0 GiB 5.0 TiB 28.42 0.87 109 up osd.88 98 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.7 GiB 5.0 TiB 28.41 0.87 114 up osd.98 103 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.7 GiB 5.0 TiB 28.07 0.86 107 up osd.103 -3 290.92255 - 291 TiB 104 TiB 103 TiB 8.7 GiB 452 GiB 187 TiB 35.81 1.10 - host ceph-host-5 105 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 11 KiB 53 GiB 5.3 TiB 71.10 2.18 417 up osd.105 109 hdd_class 18.43300 0.95001 18 TiB 13 TiB 13 TiB 7 KiB 47 GiB 5.1 TiB 72.56 2.22 428 up osd.109 117 hdd_class 18.43300 0.95001 18 TiB 13 TiB 13 TiB 6 KiB 47 GiB 5.3 TiB 71.49 2.19 422 up osd.117 124 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 11 KiB 52 GiB 5.2 TiB 71.87 2.20 424 up osd.124 129 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 2 KiB 50 GiB 4.4 TiB 75.91 2.33 449 up osd.129 136 hdd_class 18.43300 1.00000 18 TiB 14 TiB 13 TiB 11 KiB 49 GiB 4.9 TiB 73.30 2.25 432 up osd.136 181 hdd_class 18.19040 1.00000 18 TiB 109 GiB 108 GiB 7 KiB 846 MiB 18 TiB 0.59 0.02 3 up osd.181 182 hdd_class 18.19040 1.00000 18 TiB 356 GiB 354 GiB 12 KiB 2.6 GiB 18 TiB 1.91 0.06 11 up osd.182 183 hdd_class 18.19040 1.00000 18 TiB 155 GiB 154 GiB 10 KiB 1.8 GiB 18 TiB 0.83 0.03 5 up osd.183 1 nvme_class 13.97260 1.00000 14 TiB 12 GiB 875 MiB 3.2 GiB 7.7 GiB 14 TiB 0.08 0.00 290 up osd.1 7 nvme_class 13.97260 1.00000 14 TiB 7.4 GiB 1.2 GiB 2.6 GiB 3.7 GiB 14 TiB 0.05 0.00 285 up osd.7 14 nvme_class 13.97260 1.00000 14 TiB 11 GiB 837 MiB 3.0 GiB 7.6 GiB 14 TiB 0.08 0.00 290 up osd.14 21 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.2 GiB 5.1 TiB 27.24 0.83 106 up osd.21 28 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 15 GiB 5.1 TiB 27.08 0.83 105 up osd.28 35 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.05 0.83 104 up osd.35 42 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.9 GiB 5.0 TiB 27.72 0.85 107 up osd.42 49 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 15 GiB 5.0 TiB 28.19 0.86 112 up osd.49 56 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.2 GiB 5.0 TiB 28.10 0.86 107 up osd.56 63 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.4 GiB 5.0 TiB 28.10 0.86 111 up osd.63 70 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.4 GiB 5.0 TiB 28.01 0.86 112 up osd.70 77 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 109 up osd.77 84 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 104 up osd.84 90 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.9 GiB 5.0 TiB 28.42 0.87 114 up osd.90 96 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 104 up osd.96 -9 345.49374 - 345 TiB 107 TiB 106 TiB 9.7 GiB 446 GiB 238 TiB 31.02 0.95 - host ceph-host-6 111 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 6 KiB 45 GiB 6.4 TiB 65.30 2.00 384 up osd.111 118 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 11 KiB 52 GiB 6.0 TiB 67.51 2.07 397 up osd.118 128 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 12 KiB 50 GiB 4.4 TiB 75.96 2.33 445 up osd.128 135 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 11 KiB 48 GiB 4.7 TiB 74.68 2.29 440 up osd.135 142 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 56 GiB 4.1 TiB 77.54 2.38 454 up osd.142 145 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 55 GiB 4.4 TiB 75.98 2.33 445 up osd.145 153 hdd_class 18.19040 1.00000 18 TiB 169 GiB 167 GiB 7 KiB 1.7 GiB 18 TiB 0.90 0.03 5 up osd.153 157 hdd_class 18.19040 1.00000 18 TiB 488 GiB 486 GiB 14 KiB 2.7 GiB 18 TiB 2.62 0.08 15 up osd.157 162 hdd_class 18.19040 1.00000 18 TiB 440 GiB 437 GiB 17 KiB 2.6 GiB 18 TiB 2.36 0.07 11 up osd.162 166 hdd_class 18.19040 1.00000 18 TiB 852 GiB 849 GiB 24 KiB 3.3 GiB 17 TiB 4.57 0.14 25 up osd.166 170 hdd_class 18.19040 1.00000 18 TiB 473 GiB 470 GiB 22 KiB 2.5 GiB 18 TiB 2.54 0.08 14 up osd.170 174 hdd_class 18.19040 1.00000 18 TiB 673 GiB 670 GiB 20 KiB 3.0 GiB 18 TiB 3.61 0.11 21 up osd.174 5 nvme_class 13.97260 1.00000 14 TiB 9.1 GiB 846 MiB 2.9 GiB 5.3 GiB 14 TiB 0.06 0.00 276 up osd.5 12 nvme_class 13.97260 1.00000 14 TiB 12 GiB 873 MiB 3.5 GiB 7.8 GiB 14 TiB 0.08 0.00 296 up osd.12 18 nvme_class 13.97260 1.00000 14 TiB 15 GiB 857 MiB 3.3 GiB 11 GiB 14 TiB 0.10 0.00 288 up osd.18 27 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.5 GiB 5.0 TiB 28.44 0.87 114 up osd.27 34 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.11 0.86 113 up osd.34 39 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.4 GiB 5.0 TiB 28.09 0.86 108 up osd.39 46 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.5 GiB 5.1 TiB 27.65 0.85 107 up osd.46 54 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.0 GiB 5.1 TiB 27.70 0.85 107 up osd.54 62 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.5 GiB 5.0 TiB 28.49 0.87 109 up osd.62 69 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.2 GiB 5.0 TiB 28.08 0.86 113 up osd.69 76 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.0 TiB 27.78 0.85 111 up osd.76 81 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 5.1 GiB 5.0 TiB 28.46 0.87 115 up osd.81 91 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 15 GiB 5.0 TiB 27.82 0.85 108 up osd.91 97 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 5.0 GiB 5.0 TiB 28.47 0.87 109 up osd.97 102 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.11 0.86 108 up osd.102 -5 345.49374 - 345 TiB 106 TiB 105 TiB 8.8 GiB 435 GiB 239 TiB 30.73 0.94 - host ceph-host-7 106 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 2 KiB 47 GiB 5.8 TiB 68.70 2.10 406 up osd.106 112 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 6 KiB 51 GiB 5.8 TiB 68.47 2.10 403 up osd.112 119 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 11 KiB 54 GiB 5.7 TiB 69.31 2.12 407 up osd.119 125 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 4 KiB 45 GiB 5.7 TiB 68.98 2.11 410 up osd.125 131 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 11 KiB 57 GiB 4.2 TiB 77.12 2.36 454 up osd.131 138 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 58 GiB 4.1 TiB 77.85 2.38 456 up osd.138 150 hdd_class 18.19040 1.00000 18 TiB 133 GiB 131 GiB 7 KiB 1.5 GiB 18 TiB 0.71 0.02 4 up osd.150 155 hdd_class 18.19040 1.00000 18 TiB 48 GiB 46 GiB 9 KiB 2.0 GiB 18 TiB 0.26 0.01 2 up osd.155 160 hdd_class 18.19040 1.00000 18 TiB 873 GiB 869 GiB 4 KiB 3.6 GiB 17 TiB 4.69 0.14 27 up osd.160 164 hdd_class 18.19040 1.00000 18 TiB 850 GiB 847 GiB 4 KiB 3.2 GiB 17 TiB 4.57 0.14 25 up osd.164 168 hdd_class 18.19040 1.00000 18 TiB 696 GiB 693 GiB 19 KiB 3.2 GiB 18 TiB 3.74 0.11 17 up osd.168 172 hdd_class 18.19040 1.00000 18 TiB 852 GiB 848 GiB 2 KiB 3.7 GiB 17 TiB 4.57 0.14 25 up osd.172 0 nvme_class 13.97260 1.00000 14 TiB 10 GiB 858 MiB 2.8 GiB 6.3 GiB 14 TiB 0.07 0.00 293 up osd.0 9 nvme_class 13.97260 1.00000 14 TiB 9.3 GiB 847 MiB 2.9 GiB 5.6 GiB 14 TiB 0.07 0.00 281 up osd.9 16 nvme_class 13.97260 1.00000 14 TiB 10 GiB 897 MiB 3.1 GiB 6.3 GiB 14 TiB 0.07 0.00 288 up osd.16 22 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.2 GiB 5.1 TiB 27.28 0.84 107 up osd.22 29 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.9 GiB 5.0 TiB 28.42 0.87 112 up osd.29 36 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.08 0.86 109 up osd.36 43 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.42 0.87 115 up osd.43 50 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 5.6 GiB 5.1 TiB 27.28 0.84 111 up osd.50 57 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.3 GiB 5.1 TiB 27.32 0.84 112 up osd.57 64 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.9 GiB 5.0 TiB 28.45 0.87 113 up osd.64 71 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.4 GiB 5.0 TiB 28.46 0.87 109 up osd.71 78 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 14 GiB 5.1 TiB 27.08 0.83 105 up osd.78 85 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.9 GiB 5.0 TiB 28.50 0.87 114 up osd.85 92 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.41 0.87 109 up osd.92 99 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.0 TiB 27.77 0.85 112 up osd.99 TOTAL 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 MIN/MAX VAR: 0.00/2.46 STDDEV: 26.17
On Aug 31, 2025, at 4:15 AM, Steven Vacaroaia <stef97@gmail.com> wrote:
Hi,
I have added 42 x 18TB HDD disks ( 6 on each of the 7 servers )
The ultimate answer to Ceph, the cluster, and everything!
My expectation was that the pools configured to use "hdd_class" will have their capacity increased ( e.g. default.rgw.buckets.data which is uses an EC 4+2 pool for data )
First, did the raw capacity increase when you added these drives?
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
Was the number of OSDs previously 139?
It seems it is not happening ...yet ?! Is it because the peering is still going ?
Ceph nomenclature can be mystifying at first. And sometimes at thirteenth. Peering is daemons checking in with each other to ensure they’re in agreement. I think you mean backfill / balancing. The available space reported by “ceph df” for a *pool* is a function of: * Raw space available in the associated CRUSH rule’s device class (or if the rule isn’t ideal, all device classes) * The cluster’s three full ratios # ceph osd dump | grep ratio * The fullness of the single most-full OSD in the device class BTW I learned only yesterday that you can restrict `ceph osd df` by specifying a device class, so try running `ceph osd df hdd_class | tail -10` Notably, this will show you the min/max variance among OSDs of just that device class, and the standard deviation. When you have multiple OSD sizes, these figures are much less useful when calculated across the whole cluster by “ceph osd df” # ceph osd df hdd ... 318 hdd 18.53969 1.00000 19 TiB 15 TiB 15 TiB 15 KiB 67 GiB 3.6 TiB 80.79 1.04 127 up 319 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 936 KiB 60 GiB 3.7 TiB 79.87 1.03 129 up 320 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 33 KiB 72 GiB 3.7 TiB 79.99 1.03 129 up 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 11 GiB 15 TiB 17.55 0.23 26 up TOTAL 5.4 PiB 4.2 PiB 4.1 PiB 186 MiB 17 TiB 1.2 PiB 77.81 MIN/MAX VAR: 0.23/1.09 STDDEV: 4.39 You can even run this for a specific OSD so you don’t have to get creative with an egrep regex or exercise your pattern-matching skills, though the summary values naturally aren’t useful. # ceph osd df osd.30 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 12 GiB 15 TiB 17.55 1.00 26 up TOTAL 19 TiB 3.3 TiB 2.9 TiB 130 KiB 12 GiB 15 TiB 17.55 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 Here there’s a wide variation among the hdd OSDs because osd.30 had been down for a while and was recently restarted due to a host reboot, so it’s slowly filling with data.
ssd_class 6.98630
That seems like an unusual size, what are these? Are they SAN LUNs?
Below are outputs from ceph -s ceph df ceph osd df tree
Thanks for providing the needful up front.
cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
This warning state by default persists for a long time after it clears, I’m not sure why but I like to set this lower: # ceph config dump | grep blue global advanced bluestore_slow_ops_warn_lifetime 300
1 failed cephadm daemon(s) 39 daemons have recently crashed
That’s a bit worrisome, what happened? `ceph crash ls`
569 pgs not deep-scrubbed in time 2609 pgs not scrubbed in time
Scrubs don’t happen during recovery, when complete these should catch up.
services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 2m) mgr: ceph-host-1.lqlece(active, since 18h), standbys: ceph-host-2.suiuxi
I’m paranoid and would suggest deploying at least one more mgr.
mds: 19/19 daemons up, 7 standby
Yikes why so many?
osd: 181 osds: 181 up (since 4d), 181 in (since 14h)
What happened 14 hours ago? It seems unusual for these durations to vary so much.
2770 remapped pgs
That’s an indication of balancing or backfill in progress.
flags noautoscale
data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 256.82M objects, 484 TiB usage: 742 TiB used, 1.5 PiB / 2.2 PiB avail pgs: 575889786/1468742421 objects misplaced (39.210%)
39% is a lot of misplaced objects, this would be consistent with you having successfully added those OSDs. Here is where the factor of the most-full OSD comes in. Technically backfill is a subset of recovery, but in practice people usually think in terms: Recovery: PGs healing from OSDs having failed or been down Backfill: Rebalancing of data due to topology changes, including adjusted CRUSH rules, expansion, etc.
4247 active+clean 2763 active+remapped+backfill_wait 77 active+clean+scrubbing 43 active+clean+scrubbing+deep 7 active+remapped+backfilling
Configuration options throttle how much backfill goes on in parallel to keep the cluster from DoSing itself. Here I suspect that you’re running a recent release with the notorious mclock op scheduling shortcomings, which is a tangent. I suggest checking out these two resources re upmap-remapped.py : https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering Ceph Operations with Upmap.pdf <https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering%20Ceph%20Operations%20with%20Upmap.pdf> https://community.ibm.com/community/user/blogs/anthony-datri/2025/07/30/grac... This tool, in conjunction with the balancer module, will do the backfill more elegantly with various benefits.
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
I hope the formatting below comes through, makes it a lot easier to read a table.
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB metadata_fs_nvme 13 32 260 MiB 177 780 MiB 0 93 TiB metadata_fs_ssd_rep 14 32 17 MiB 103 51 MiB 0 93 TiB ssd_rep_projects 15 1024 132 B 1 12 KiB 0 130 TiB nvme_rep_projects 16 512 3.5 KiB 30 336 KiB 093 TiB
Do you have multiple EC RBD pools and/or multiple CephFSes?
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2272.93311 - 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 1.00 - root default -7 254.54175 - 255 TiB 104 TiB 102 TiB 9.6 GiB 455 GiB 151 TiB 40.78 1.25 - host ceph-host-1
...
137 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 50 GiB 4.3 TiB 76.47 2.34 449 up osd.137 152 hdd_class 18.19040 1.00000 18 TiB 241 GiB 239 GiB 10 KiB 1.8 GiB 18 TiB 1.29 0.04 7 up osd.152 3.1 TiB 1.5 PiB 32.64 MIN/MAX VAR: 0.00/2.46 STDDEV: 26.17
There ya go. osd.152 must be one of the new OSDs. Note that only 7 PGs are currently resident and that it holds just 4% of the average amount of data on the entire set of OSDs. Run the focused `osd df` above and that number will change slightly. Here is your least full hdd_class OSD: 151 hdd_class 18.19040 1.00000 18 TiB 38 GiB 37 GiB 6 KiB 1.1 GiB 18 TiB 0.20 0.01 1 up osd.151 And the most full: 180 hdd_class 18.19040 1.00000 18 TiB 198 GiB 197 GiB 10 KiB 1.7 GiB 18 TiB 1.07 0.03 5 up osd.180 I suspect that the most-full is at 107% of average due to the bolus of backfill and/or the balancer not being active. Using upmap-remapped as described above can help avoid this kind of overload. In a nutshell, the available space will gradually increase as data is backfilled, especially if you have the balancer enabled.
Hi, Thanks Anthony - as always a very useful and comprehensive response yes, there were only 139 OSD and , indeed, raw capacity increased I also noticed that the "max available " column from "cepf df" is getting higher (177 TB) so, it seems the capacity is being added The drives used in the SSD_class are 7TB Micron_5400 The failed daemon is the manager - will follow your advice and add one more Blue_store warning was set to 600 - will change it to 300 Too many MDS daemons are due to the fact that I have over 150 CEPHFS clients and thought that deploying a daemon on each host for each filesystem is going to provide better performance - was I wrong ? High number of remapped PGs are due to improperly using osd.all-available-devices unmaged command so adding the new drives triggered automatically detecting and adding them I am not sure what I did wrong though - see below output from "ceph orch ls" before adding the drive Shouldn't setting it like that prevent automatic discovery ? NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 7m ago 5w count:1 ceph-exporter 7/7 10m ago 5w * crash 7/7 10m ago 5w * grafana ?:3001 1/1 7m ago 4w count:1 ingress.rgw.default.default 192.168.122.239:1969,8080 4/4 10m ago 4w ceph-host-4;ceph-host-5 loki ?:3100 1/1 7m ago 4w ceph-host-2;count:1 mds.hdd_ec_archive 5/5 10m ago 5w ceph-host-1;ceph-host-5;ceph-host-4;ceph-host-6;ceph-host-7;count:5 mds.project_ec_ssd 7/7 10m ago 4w ceph-host-1;ceph-host-2;ceph-host-3;ceph-host-4;ceph-host-5;ceph-host-6;ceph-host-7;count:7 mds.projects_rep_nvme 7/7 10m ago 4w ceph-host-1;ceph-host-2;ceph-host-3;ceph-host-4;ceph-host-5;ceph-host-6;ceph-host-7;count:7 mds.projects_rep_ssd 7/7 10m ago 4w ceph-host-1;ceph-host-2;ceph-host-3;ceph-host-4;ceph-host-5;ceph-host-6;ceph-host-7;count:7 mgr 2/2 7m ago 5w count:2 mon 5/5 8m ago 5w count:5 node-exporter ?:9100 7/7 10m ago 5w * osd.all-available-devices 0 - 4w <unmanaged> osd.hdd_osds 72 10m ago 5w * osd.nvme_osds 25 10m ago 5w * osd.ssd_osds 84 10m ago 3w ceph-host-1 prometheus ?:9095 1/1 7m ago 5w count:1 promtail ?:9080 7/7 10m ago 4w ceph-host-1;ceph-host-2;ceph-host-3;ceph-host-4;ceph-host-5;ceph-host-6;ceph-host-7;count:7 rgw.default.default ?:80 0/2 10m ago 4w count-per-host:1;label:rgw 0.090000 { "active": true, Resources mentioned are very useful running upmap-remapped.py did bring the number of PGs to be remapped close to zero Just wanted to clarify the steps needed because , doing it like below, I am , eventually, ending up again with lots of unmapped PGs and with an "unhappy" balancer 1. # ceph osd set norebalance # ceph balancer off 2. upmap-remapped.py | sh 3. change target_max_misplaced_ratio to a higher number than default 0.005 (since we want to rebalance faster and client performance is not a huge issue ) 4. enable balancer 5.wait Doing it like this will, eventually, increase the number misplaced PGs until it is higher than the ratio when , I guess, the balancer stops ( "optimize_result": "Too many objects (0.115742 > 0.090000) are misplaced; try again later ) Should I repeat the process when the number of objects misplaced is higher than the ratio or what is the proper way of doing it ? Steven On Sun, 31 Aug 2025 at 11:08, Anthony D'Atri <aad@dreamsnake.net> wrote:
On Aug 31, 2025, at 4:15 AM, Steven Vacaroaia <stef97@gmail.com> wrote:
Hi,
I have added 42 x 18TB HDD disks ( 6 on each of the 7 servers )
The ultimate answer to Ceph, the cluster, and everything!
My expectation was that the pools configured to use "hdd_class" will have their capacity increased ( e.g. default.rgw.buckets.data which is uses an EC 4+2 pool for data )
First, did the raw capacity increase when you added these drives?
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
Was the number of OSDs previously 139?
It seems it is not happening ...yet ?! Is it because the peering is still going ?
Ceph nomenclature can be mystifying at first. And sometimes at thirteenth.
Peering is daemons checking in with each other to ensure they’re in agreement.
I think you mean backfill / balancing.
The available space reported by “ceph df” for a *pool* is a function of:
* Raw space available in the associated CRUSH rule’s device class (or if the rule isn’t ideal, all device classes) * The cluster’s three full ratios # ceph osd dump | grep ratio * The fullness of the single most-full OSD in the device class
BTW I learned only yesterday that you can restrict `ceph osd df` by specifying a device class, so try running
`ceph osd df hdd_class | tail -10`
Notably, this will show you the min/max variance among OSDs of just that device class, and the standard deviation. When you have multiple OSD sizes, these figures are much less useful when calculated across the whole cluster by “ceph osd df”
# ceph osd df hdd ... 318 hdd 18.53969 1.00000 19 TiB 15 TiB 15 TiB 15 KiB 67 GiB 3.6 TiB 80.79 1.04 127 up 319 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 936 KiB 60 GiB 3.7 TiB 79.87 1.03 129 up 320 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 33 KiB 72 GiB 3.7 TiB 79.99 1.03 129 up 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 11 GiB 15 TiB 17.55 0.23 26 up TOTAL 5.4 PiB 4.2 PiB 4.1 PiB 186 MiB 17 TiB 1.2 PiB 77.81 MIN/MAX VAR: 0.23/1.09 STDDEV: 4.39
You can even run this for a specific OSD so you don’t have to get creative with an egrep regex or exercise your pattern-matching skills, though the summary values naturally aren’t useful.
# ceph osd df osd.30 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 12 GiB 15 TiB 17.55 1.00 26 up TOTAL 19 TiB 3.3 TiB 2.9 TiB 130 KiB 12 GiB 15 TiB 17.55 MIN/MAX VAR: 1.00/1.00 STDDEV: 0
Here there’s a wide variation among the hdd OSDs because osd.30 had been down for a while and was recently restarted due to a host reboot, so it’s slowly filling with data.
ssd_class 6.98630
That seems like an unusual size, what are these? Are they SAN LUNs?
Below are outputs from ceph -s ceph df ceph osd df tree
Thanks for providing the needful up front.
cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
This warning state by default persists for a long time after it clears, I’m not sure why but I like to set this lower:
# ceph config dump | grep blue global advanced bluestore_slow_ops_warn_lifetime 300
1 failed cephadm daemon(s)
39 daemons have recently crashed
That’s a bit worrisome, what happened?
`ceph crash ls`
569 pgs not deep-scrubbed in time 2609 pgs not scrubbed in time
Scrubs don’t happen during recovery, when complete these should catch up.
services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 2m) mgr: ceph-host-1.lqlece(active, since 18h), standbys: ceph-host-2.suiuxi
I’m paranoid and would suggest deploying at least one more mgr.
mds: 19/19 daemons up, 7 standby
Yikes why so many?
osd: 181 osds: 181 up (since 4d), 181 in (since 14h)
What happened 14 hours ago? It seems unusual for these durations to vary so much.
2770 remapped pgs
That’s an indication of balancing or backfill in progress.
flags noautoscale
data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 256.82M objects, 484 TiB usage: 742 TiB used, 1.5 PiB / 2.2 PiB avail pgs: 575889786/1468742421 objects misplaced (39.210%)
39% is a lot of misplaced objects, this would be consistent with you having successfully added those OSDs. Here is where the factor of the most-full OSD comes in.
Technically backfill is a subset of recovery, but in practice people usually think in terms:
Recovery: PGs healing from OSDs having failed or been down Backfill: Rebalancing of data due to topology changes, including adjusted CRUSH rules, expansion, etc.
4247 active+clean 2763 active+remapped+backfill_wait 77 active+clean+scrubbing 43 active+clean+scrubbing+deep 7 active+remapped+backfilling
Configuration options throttle how much backfill goes on in parallel to keep the cluster from DoSing itself. Here I suspect that you’re running a recent release with the notorious mclock op scheduling shortcomings, which is a tangent.
I suggest checking out these two resources re upmap-remapped.py :
https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering Ceph Operations with Upmap.pdf
https://community.ibm.com/community/user/blogs/anthony-datri/2025/07/30/grac...
This tool, in conjunction with the balancer module, will do the backfill more elegantly with various benefits.
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
I hope the formatting below comes through, makes it a lot easier to read a table.
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB metadata_fs_nvme 13 32 260 MiB 177 780 MiB 0 93 TiB metadata_fs_ssd_rep 14 32 17 MiB 103 51 MiB 0 93 TiB ssd_rep_projects 15 1024 132 B 1 12 KiB 0 130 TiB nvme_rep_projects 16 512 3.5 KiB 30 336 KiB 093 TiB
Do you have multiple EC RBD pools and/or multiple CephFSes?
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2272.93311 - 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 1.00 - root default -7 254.54175 - 255 TiB 104 TiB 102 TiB 9.6 GiB 455 GiB 151 TiB 40.78 1.25 - host ceph-host-1
...
137 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 50 GiB 4.3 TiB 76.47 2.34 449 up osd.137 152 hdd_class 18.19040 1.00000 18 TiB 241 GiB 239 GiB 10 KiB 1.8 GiB 18 TiB 1.29 0.04 7 up osd.152 3.1 TiB 1.5 PiB 32.64 MIN/MAX VAR: 0.00/2.46 STDDEV: 26.17
There ya go. osd.152 must be one of the new OSDs. Note that only 7 PGs are currently resident and that it holds just 4% of the average amount of data on the entire set of OSDs. Run the focused `osd df` above and that number will change slightly.
Here is your least full hdd_class OSD:
151 hdd_class 18.19040 1.00000 18 TiB 38 GiB 37 GiB 6 KiB 1.1 GiB 18 TiB 0.20 0.01 1 up osd.151
And the most full:
180 hdd_class 18.19040 1.00000 18 TiB 198 GiB 197 GiB 10 KiB 1.7 GiB 18 TiB 1.07 0.03 5 up osd.180
I suspect that the most-full is at 107% of average due to the bolus of backfill and/or the balancer not being active. Using upmap-remapped as described above can help avoid this kind of overload.
In a nutshell, the available space will gradually increase as data is backfilled, especially if you have the balancer enabled.
Hi, Thanks Anthony - as always a very useful and comprehensive response
Most welcome.
yes, there were only 139 OSD and , indeed, raw capacity increased I also noticed that the "max available " column from "cepf df" is getting higher (177 TB) so, it seems the capacity is being added
Aye, as expected.
Too many MDS daemons are due to the fact that I have over 150 CEPHFS clients and thought that deploying a daemon on each host for each filesystem is going to provide better performance - was I wrong ?
MDS strategy is nuanced, but I think deploying them redundantly on each back-end node might not give you much. The workload AIUI is more a function of the number and size of files.
High number of remapped PGs are due to improperly using osd.all-available-devices unmaged command so adding the new drives triggered automatically detecting and adding them
Yep, I typically suggest setting OSD services to unmanaged when not actively using them.
I am not sure what I did wrong though - see below output from "ceph orch ls" before adding the drive Shouldn't setting it like that prevent automatic discovery ?
You do have three other OSD rules, at least one of them likely matched and took action.
osd.all-available-devices 0 - 4w <unmanaged> osd.hdd_osds 72 10m ago 5w * osd.nvme_osds 25 10m ago 5w * osd.ssd_osds 84 10m ago 3w ceph-host-1
`ceph orch ls --export`
Resources mentioned are very useful running upmap-remapped.py did bring the number of PGs to be remapped close to zero
It's a super, super useful tool. Letting the balancer do the work incrementally helps deter unexpected OSD fullness issues and if a problem arises, you're a lot closer to HEALTH_OK.
2. upmap-remapped.py | sh
Sometimes 2-3 runs are needed for full effect.
3. change target_max_misplaced_ratio to a higher number than default 0.005 (since we want to rebalance faster and client performance is not a huge issue )
4. enable balancer
5.wait
Doing it like this will, eventually, increase the number misplaced PGs until it is higher than the ratio when , I guess, the balancer stops ( "optimize_result": "Too many objects (0.115742 > 0.090000) are misplaced; try again later )
Exactly.
Should I repeat the process when the number of objects misplaced is higher than the ratio or what is the proper way of doing it ?
As the backfill progresses and the misplaced percentage drops, the balancer will kick in with another increment.
Steven
On Sun, 31 Aug 2025 at 11:08, Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>> wrote:
On Aug 31, 2025, at 4:15 AM, Steven Vacaroaia <stef97@gmail.com <mailto:stef97@gmail.com>> wrote:
Hi,
I have added 42 x 18TB HDD disks ( 6 on each of the 7 servers )
The ultimate answer to Ceph, the cluster, and everything!
My expectation was that the pools configured to use "hdd_class" will have their capacity increased ( e.g. default.rgw.buckets.data which is uses an EC 4+2 pool for data )
First, did the raw capacity increase when you added these drives?
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
Was the number of OSDs previously 139?
It seems it is not happening ...yet ?! Is it because the peering is still going ?
Ceph nomenclature can be mystifying at first. And sometimes at thirteenth.
Peering is daemons checking in with each other to ensure they’re in agreement.
I think you mean backfill / balancing.
The available space reported by “ceph df” for a *pool* is a function of:
* Raw space available in the associated CRUSH rule’s device class (or if the rule isn’t ideal, all device classes) * The cluster’s three full ratios # ceph osd dump | grep ratio * The fullness of the single most-full OSD in the device class
BTW I learned only yesterday that you can restrict `ceph osd df` by specifying a device class, so try running
`ceph osd df hdd_class | tail -10`
Notably, this will show you the min/max variance among OSDs of just that device class, and the standard deviation. When you have multiple OSD sizes, these figures are much less useful when calculated across the whole cluster by “ceph osd df”
# ceph osd df hdd ... 318 hdd 18.53969 1.00000 19 TiB 15 TiB 15 TiB 15 KiB 67 GiB 3.6 TiB 80.79 1.04 127 up 319 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 936 KiB 60 GiB 3.7 TiB 79.87 1.03 129 up 320 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 33 KiB 72 GiB 3.7 TiB 79.99 1.03 129 up 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 11 GiB 15 TiB 17.55 0.23 26 up TOTAL 5.4 PiB 4.2 PiB 4.1 PiB 186 MiB 17 TiB 1.2 PiB 77.81 MIN/MAX VAR: 0.23/1.09 STDDEV: 4.39
You can even run this for a specific OSD so you don’t have to get creative with an egrep regex or exercise your pattern-matching skills, though the summary values naturally aren’t useful.
# ceph osd df osd.30 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 12 GiB 15 TiB 17.55 1.00 26 up TOTAL 19 TiB 3.3 TiB 2.9 TiB 130 KiB 12 GiB 15 TiB 17.55 MIN/MAX VAR: 1.00/1.00 STDDEV: 0
Here there’s a wide variation among the hdd OSDs because osd.30 had been down for a while and was recently restarted due to a host reboot, so it’s slowly filling with data.
ssd_class 6.98630
That seems like an unusual size, what are these? Are they SAN LUNs?
Below are outputs from ceph -s ceph df ceph osd df tree
Thanks for providing the needful up front.
cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
This warning state by default persists for a long time after it clears, I’m not sure why but I like to set this lower:
# ceph config dump | grep blue global advanced bluestore_slow_ops_warn_lifetime 300
1 failed cephadm daemon(s) 39 daemons have recently crashed
That’s a bit worrisome, what happened?
`ceph crash ls`
569 pgs not deep-scrubbed in time 2609 pgs not scrubbed in time
Scrubs don’t happen during recovery, when complete these should catch up.
services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 2m) mgr: ceph-host-1.lqlece(active, since 18h), standbys: ceph-host-2.suiuxi
I’m paranoid and would suggest deploying at least one more mgr.
mds: 19/19 daemons up, 7 standby
Yikes why so many?
osd: 181 osds: 181 up (since 4d), 181 in (since 14h)
What happened 14 hours ago? It seems unusual for these durations to vary so much.
2770 remapped pgs
That’s an indication of balancing or backfill in progress.
flags noautoscale
data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 256.82M objects, 484 TiB usage: 742 TiB used, 1.5 PiB / 2.2 PiB avail pgs: 575889786/1468742421 objects misplaced (39.210%)
39% is a lot of misplaced objects, this would be consistent with you having successfully added those OSDs. Here is where the factor of the most-full OSD comes in.
Technically backfill is a subset of recovery, but in practice people usually think in terms:
Recovery: PGs healing from OSDs having failed or been down Backfill: Rebalancing of data due to topology changes, including adjusted CRUSH rules, expansion, etc.
4247 active+clean 2763 active+remapped+backfill_wait 77 active+clean+scrubbing 43 active+clean+scrubbing+deep 7 active+remapped+backfilling
Configuration options throttle how much backfill goes on in parallel to keep the cluster from DoSing itself. Here I suspect that you’re running a recent release with the notorious mclock op scheduling shortcomings, which is a tangent.
I suggest checking out these two resources re upmap-remapped.py :
https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering Ceph Operations with Upmap.pdf <https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering%20Ceph%20Operations%20with%20Upmap.pdf> https://community.ibm.com/community/user/blogs/anthony-datri/2025/07/30/grac...
This tool, in conjunction with the balancer module, will do the backfill more elegantly with various benefits.
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
I hope the formatting below comes through, makes it a lot easier to read a table.
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB metadata_fs_nvme 13 32 260 MiB 177 780 MiB 0 93 TiB metadata_fs_ssd_rep 14 32 17 MiB 103 51 MiB 0 93 TiB ssd_rep_projects 15 1024 132 B 1 12 KiB 0 130 TiB nvme_rep_projects 16 512 3.5 KiB 30 336 KiB 093 TiB
Do you have multiple EC RBD pools and/or multiple CephFSes?
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2272.93311 - 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 1.00 - root default -7 254.54175 - 255 TiB 104 TiB 102 TiB 9.6 GiB 455 GiB 151 TiB 40.78 1.25 - host ceph-host-1
...
137 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 50 GiB 4.3 TiB 76.47 2.34 449 up osd.137 152 hdd_class 18.19040 1.00000 18 TiB 241 GiB 239 GiB 10 KiB 1.8 GiB 18 TiB 1.29 0.04 7 up osd.152 3.1 TiB 1.5 PiB 32.64 MIN/MAX VAR: 0.00/2.46 STDDEV: 26.17
There ya go. osd.152 must be one of the new OSDs. Note that only 7 PGs are currently resident and that it holds just 4% of the average amount of data on the entire set of OSDs. Run the focused `osd df` above and that number will change slightly.
Here is your least full hdd_class OSD:
151 hdd_class 18.19040 1.00000 18 TiB 38 GiB 37 GiB 6 KiB 1.1 GiB 18 TiB 0.20 0.01 1 up osd.151
And the most full:
180 hdd_class 18.19040 1.00000 18 TiB 198 GiB 197 GiB 10 KiB 1.7 GiB 18 TiB 1.07 0.03 5 up osd.180
I suspect that the most-full is at 107% of average due to the bolus of backfill and/or the balancer not being active. Using upmap-remapped as described above can help avoid this kind of overload.
In a nutshell, the available space will gradually increase as data is backfilled, especially if you have the balancer enabled.
Thanks again Anthony I guess I assumed wrong that "osd.all-available-devices" includes all available devices It seems I should have targeted specifical ones (like osd.hdd_osds) Going back to the capacity increase I am trying to estimate / calculate the final capacity Looking at the output of "ceph df" below it would be greatly appreciated if you could please confirm if this is the correct interpretation for hdd_class : "Out of a total capacity of 1.4 PiB you currently use 579 TiB . The 579 TiB used are the sum of the data used by pools configured to use hdd_class e.g osd.all-available-devices (434) , hdd_ec_archive (135)...etc Therefore, when the backfilling / balancing is done (and assuming no more data is added to the pools) the amount of pools "MAX Available" should be 814 TB " Is the above correct ? Note The "MAX available " is increasing - 202 TB now Steven --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54 nvme_class 293 TiB 293 TiB 216 GiB 216 GiB 0.07 ssd_class 587 TiB 424 TiB 163 TiB 163 TiB 27.80 TOTAL 2.2 PiB 1.5 PiB 742 TiB 742 TiB 32.64 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB On Mon, 1 Sept 2025 at 17:07, Anthony D'Atri <aad@dreamsnake.net> wrote:
Hi, Thanks Anthony - as always a very useful and comprehensive response
Most welcome.
yes, there were only 139 OSD and , indeed, raw capacity increased I also noticed that the "max available " column from "cepf df" is getting higher (177 TB) so, it seems the capacity is being added
Aye, as expected.
Too many MDS daemons are due to the fact that I have over 150 CEPHFS clients and thought that deploying a daemon on each host for each filesystem is going to provide better performance - was I wrong ?
MDS strategy is nuanced, but I think deploying them redundantly on each back-end node might not give you much. The workload AIUI is more a function of the number and size of files.
High number of remapped PGs are due to improperly using osd.all-available-devices unmaged command so adding the new drives triggered automatically detecting and adding them
Yep, I typically suggest setting OSD services to unmanaged when not actively using them.
I am not sure what I did wrong though - see below output from "ceph orch ls" before adding the drive Shouldn't setting it like that prevent automatic discovery ?
You do have three other OSD rules, at least one of them likely matched and took action.
osd.all-available-devices 0 - 4w <unmanaged>
osd.hdd_osds 72 10m ago 5w *
osd.nvme_osds 25 10m ago 5w *
osd.ssd_osds 84 10m ago 3w ceph-host-1
`ceph orch ls --export`
Resources mentioned are very useful running upmap-remapped.py did bring the number of PGs to be remapped close to zero
It's a super, super useful tool. Letting the balancer do the work incrementally helps deter unexpected OSD fullness issues and if a problem arises, you're a lot closer to HEALTH_OK.
2. upmap-remapped.py | sh
Sometimes 2-3 runs are needed for full effect.
3. change target_max_misplaced_ratio to a higher number than default 0.005 (since we want to rebalance faster and client performance is not a huge issue )
4. enable balancer
5.wait
Doing it like this will, eventually, increase the number misplaced PGs until it is higher than the ratio when , I guess, the balancer stops ( "optimize_result": "Too many objects (0.115742 > 0.090000) are misplaced; try again later )
Exactly.
Should I repeat the process when the number of objects misplaced is higher than the ratio or what is the proper way of doing it ?
As the backfill progresses and the misplaced percentage drops, the balancer will kick in with another increment.
Steven
On Sun, 31 Aug 2025 at 11:08, Anthony D'Atri <aad@dreamsnake.net> wrote:
On Aug 31, 2025, at 4:15 AM, Steven Vacaroaia <stef97@gmail.com> wrote:
Hi,
I have added 42 x 18TB HDD disks ( 6 on each of the 7 servers )
The ultimate answer to Ceph, the cluster, and everything!
My expectation was that the pools configured to use "hdd_class" will have their capacity increased ( e.g. default.rgw.buckets.data which is uses an EC 4+2 pool for data )
First, did the raw capacity increase when you added these drives?
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
Was the number of OSDs previously 139?
It seems it is not happening ...yet ?! Is it because the peering is still going ?
Ceph nomenclature can be mystifying at first. And sometimes at thirteenth.
Peering is daemons checking in with each other to ensure they’re in agreement.
I think you mean backfill / balancing.
The available space reported by “ceph df” for a *pool* is a function of:
* Raw space available in the associated CRUSH rule’s device class (or if the rule isn’t ideal, all device classes) * The cluster’s three full ratios # ceph osd dump | grep ratio * The fullness of the single most-full OSD in the device class
BTW I learned only yesterday that you can restrict `ceph osd df` by specifying a device class, so try running
`ceph osd df hdd_class | tail -10`
Notably, this will show you the min/max variance among OSDs of just that device class, and the standard deviation. When you have multiple OSD sizes, these figures are much less useful when calculated across the whole cluster by “ceph osd df”
# ceph osd df hdd ... 318 hdd 18.53969 1.00000 19 TiB 15 TiB 15 TiB 15 KiB 67 GiB 3.6 TiB 80.79 1.04 127 up 319 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 936 KiB 60 GiB 3.7 TiB 79.87 1.03 129 up 320 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 33 KiB 72 GiB 3.7 TiB 79.99 1.03 129 up 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 11 GiB 15 TiB 17.55 0.23 26 up TOTAL 5.4 PiB 4.2 PiB 4.1 PiB 186 MiB 17 TiB 1.2 PiB 77.81 MIN/MAX VAR: 0.23/1.09 STDDEV: 4.39
You can even run this for a specific OSD so you don’t have to get creative with an egrep regex or exercise your pattern-matching skills, though the summary values naturally aren’t useful.
# ceph osd df osd.30 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 12 GiB 15 TiB 17.55 1.00 26 up TOTAL 19 TiB 3.3 TiB 2.9 TiB 130 KiB 12 GiB 15 TiB 17.55 MIN/MAX VAR: 1.00/1.00 STDDEV: 0
Here there’s a wide variation among the hdd OSDs because osd.30 had been down for a while and was recently restarted due to a host reboot, so it’s slowly filling with data.
ssd_class 6.98630
That seems like an unusual size, what are these? Are they SAN LUNs?
Below are outputs from ceph -s ceph df ceph osd df tree
Thanks for providing the needful up front.
cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
This warning state by default persists for a long time after it clears, I’m not sure why but I like to set this lower:
# ceph config dump | grep blue global advanced bluestore_slow_ops_warn_lifetime 300
1 failed cephadm daemon(s)
39 daemons have recently crashed
That’s a bit worrisome, what happened?
`ceph crash ls`
569 pgs not deep-scrubbed in time 2609 pgs not scrubbed in time
Scrubs don’t happen during recovery, when complete these should catch up.
services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 2m) mgr: ceph-host-1.lqlece(active, since 18h), standbys: ceph-host-2.suiuxi
I’m paranoid and would suggest deploying at least one more mgr.
mds: 19/19 daemons up, 7 standby
Yikes why so many?
osd: 181 osds: 181 up (since 4d), 181 in (since 14h)
What happened 14 hours ago? It seems unusual for these durations to vary so much.
2770 remapped pgs
That’s an indication of balancing or backfill in progress.
flags noautoscale
data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 256.82M objects, 484 TiB usage: 742 TiB used, 1.5 PiB / 2.2 PiB avail pgs: 575889786/1468742421 objects misplaced (39.210%)
39% is a lot of misplaced objects, this would be consistent with you having successfully added those OSDs. Here is where the factor of the most-full OSD comes in.
Technically backfill is a subset of recovery, but in practice people usually think in terms:
Recovery: PGs healing from OSDs having failed or been down Backfill: Rebalancing of data due to topology changes, including adjusted CRUSH rules, expansion, etc.
4247 active+clean 2763 active+remapped+backfill_wait 77 active+clean+scrubbing 43 active+clean+scrubbing+deep 7 active+remapped+backfilling
Configuration options throttle how much backfill goes on in parallel to keep the cluster from DoSing itself. Here I suspect that you’re running a recent release with the notorious mclock op scheduling shortcomings, which is a tangent.
I suggest checking out these two resources re upmap-remapped.py :
https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering Ceph Operations with Upmap.pdf
https://community.ibm.com/community/user/blogs/anthony-datri/2025/07/30/grac...
This tool, in conjunction with the balancer module, will do the backfill more elegantly with various benefits.
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
I hope the formatting below comes through, makes it a lot easier to read a table.
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB metadata_fs_nvme 13 32 260 MiB 177 780 MiB 0 93 TiB metadata_fs_ssd_rep 14 32 17 MiB 103 51 MiB 0 93 TiB ssd_rep_projects 15 1024 132 B 1 12 KiB 0 130 TiB nvme_rep_projects 16 512 3.5 KiB 30 336 KiB 093 TiB
Do you have multiple EC RBD pools and/or multiple CephFSes?
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2272.93311 - 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 1.00 - root default -7 254.54175 - 255 TiB 104 TiB 102 TiB 9.6 GiB 455 GiB 151 TiB 40.78 1.25 - host ceph-host-1
...
137 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 50 GiB 4.3 TiB 76.47 2.34 449 up osd.137 152 hdd_class 18.19040 1.00000 18 TiB 241 GiB 239 GiB 10 KiB 1.8 GiB 18 TiB 1.29 0.04 7 up osd.152 3.1 TiB 1.5 PiB 32.64 MIN/MAX VAR: 0.00/2.46 STDDEV: 26.17
There ya go. osd.152 must be one of the new OSDs. Note that only 7 PGs are currently resident and that it holds just 4% of the average amount of data on the entire set of OSDs. Run the focused `osd df` above and that number will change slightly.
Here is your least full hdd_class OSD:
151 hdd_class 18.19040 1.00000 18 TiB 38 GiB 37 GiB 6 KiB 1.1 GiB 18 TiB 0.20 0.01 1 up osd.151
And the most full:
180 hdd_class 18.19040 1.00000 18 TiB 198 GiB 197 GiB 10 KiB 1.7 GiB 18 TiB 1.07 0.03 5 up osd.180
I suspect that the most-full is at 107% of average due to the bolus of backfill and/or the balancer not being active. Using upmap-remapped as described above can help avoid this kind of overload.
In a nutshell, the available space will gradually increase as data is backfilled, especially if you have the balancer enabled.
On Sep 2, 2025, at 6:36 AM, Steven Vacaroaia <stef97@gmail.com> wrote:
Thanks again Anthony
I guess I assumed wrong that "osd.all-available-devices" includes all available devices
I *think* the last match wins when there are multiple OSD specs. Please excerpt them from `ceph orch ls --export` so we can see them in detail.
It seems I should have targeted specifical ones (like osd.hdd_osds)
Going back to the capacity increase I am trying to estimate / calculate the final capacity
Looking at the output of "ceph df" below it would be greatly appreciated if you could please confirm if this is the correct interpretation for hdd_class :
"Out of a total capacity of 1.4 PiB you currently use 579 TiB .
Yes.
The 579 TiB used are the sum of the data used by pools configured to use hdd_class e.g osd.all-available-devices (434) , hdd_ec_archive (135)...etc
Yes. Note that if you have any pools using a CRUSH rule that does not specify a device class, they will use *all* OSDs and hilarity will ensue.
Therefore, when the backfilling / balancing is done (and assuming no more data is added to the pools) the amount of pools "MAX Available" should be 814 TB
Ah. Not so fast. 814 TiB is aggregate raw capacity. Max avail is nuanced. This can be slippery to wrap one's mind around. If your mind is pre-warped, that helps. Max avail is *per pool*. It indicates the projected amount of user data that may be written to that pool *if* no other pools mapped to the same OSDs gain or lose data. It's a zero sum game. Note for example that .mgr and default.rgw.buckets.index (and a slew of others) all report 93 TiB max avail. They're all (presumably) replicated size=3 and on the same OSDs. So that *set* of pools can accept ~~ 93 TiB of new data split any way across them. This is a function of the delta between the 90% full_ratio and that the most-full current OSD is 28.58% full, divided by 3 for replication. It becomes a bit more complicated for the hdd_class, where it would seem that EC is in effect. It's also a function of the CRUSH topology and rules. I see that for example ceph-host-1 offers ~ 98 CRUSH weight of hdd_class, where ceph-host-1 offers ~ 226 CRUSH weight of hdd_class. Your latest message doesn't show the CRUSH rules and pool attributes but I suspect multiple EC profiles are in use, which complicates the calculation a bit. You have 7 hosts and thus likely 7 CRUSH failure domains. If you have, say, a 5+2 EC pool, that pool's max avail will be limited by the lowest CRUSH weight of the failure domains, since each PG will need to hit one OSD on every host. A 4+2 pool however would have more flexibility in placement and be less constrained by the unequal distribution. Why does ceph-host-1 have fewer HDDs than the other hosts?
"
Is the above correct ?
Note The "MAX available " is increasing - 202 TB now
That's a function of the balancer reducing how full the most-full OSD of that device class is
Steven
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54 nvme_class 293 TiB 293 TiB 216 GiB 216 GiB 0.07 ssd_class 587 TiB 424 TiB 163 TiB 163 TiB 27.80 TOTAL 2.2 PiB 1.5 PiB 742 TiB 742 TiB 32.64
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB
On Mon, 1 Sept 2025 at 17:07, Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>> wrote:
Hi, Thanks Anthony - as always a very useful and comprehensive response
Most welcome.
yes, there were only 139 OSD and , indeed, raw capacity increased I also noticed that the "max available " column from "cepf df" is getting higher (177 TB) so, it seems the capacity is being added
Aye, as expected.
Too many MDS daemons are due to the fact that I have over 150 CEPHFS clients and thought that deploying a daemon on each host for each filesystem is going to provide better performance - was I wrong ?
MDS strategy is nuanced, but I think deploying them redundantly on each back-end node might not give you much. The workload AIUI is more a function of the number and size of files.
High number of remapped PGs are due to improperly using osd.all-available-devices unmaged command so adding the new drives triggered automatically detecting and adding them
Yep, I typically suggest setting OSD services to unmanaged when not actively using them.
I am not sure what I did wrong though - see below output from "ceph orch ls" before adding the drive Shouldn't setting it like that prevent automatic discovery ?
You do have three other OSD rules, at least one of them likely matched and took action.
osd.all-available-devices 0 - 4w <unmanaged> osd.hdd_osds 72 10m ago 5w * osd.nvme_osds 25 10m ago 5w * osd.ssd_osds 84 10m ago 3w ceph-host-1
`ceph orch ls --export`
Resources mentioned are very useful running upmap-remapped.py did bring the number of PGs to be remapped close to zero
It's a super, super useful tool. Letting the balancer do the work incrementally helps deter unexpected OSD fullness issues and if a problem arises, you're a lot closer to HEALTH_OK.
2. upmap-remapped.py | sh
Sometimes 2-3 runs are needed for full effect.
3. change target_max_misplaced_ratio to a higher number than default 0.005 (since we want to rebalance faster and client performance is not a huge issue )
4. enable balancer
5.wait
Doing it like this will, eventually, increase the number misplaced PGs until it is higher than the ratio when , I guess, the balancer stops ( "optimize_result": "Too many objects (0.115742 > 0.090000) are misplaced; try again later )
Exactly.
Should I repeat the process when the number of objects misplaced is higher than the ratio or what is the proper way of doing it ?
As the backfill progresses and the misplaced percentage drops, the balancer will kick in with another increment.
Steven
On Sun, 31 Aug 2025 at 11:08, Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>> wrote:
On Aug 31, 2025, at 4:15 AM, Steven Vacaroaia <stef97@gmail.com <mailto:stef97@gmail.com>> wrote:
Hi,
I have added 42 x 18TB HDD disks ( 6 on each of the 7 servers )
The ultimate answer to Ceph, the cluster, and everything!
My expectation was that the pools configured to use "hdd_class" will have their capacity increased ( e.g. default.rgw.buckets.data which is uses an EC 4+2 pool for data )
First, did the raw capacity increase when you added these drives?
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
Was the number of OSDs previously 139?
It seems it is not happening ...yet ?! Is it because the peering is still going ?
Ceph nomenclature can be mystifying at first. And sometimes at thirteenth.
Peering is daemons checking in with each other to ensure they’re in agreement.
I think you mean backfill / balancing.
The available space reported by “ceph df” for a *pool* is a function of:
* Raw space available in the associated CRUSH rule’s device class (or if the rule isn’t ideal, all device classes) * The cluster’s three full ratios # ceph osd dump | grep ratio * The fullness of the single most-full OSD in the device class
BTW I learned only yesterday that you can restrict `ceph osd df` by specifying a device class, so try running
`ceph osd df hdd_class | tail -10`
Notably, this will show you the min/max variance among OSDs of just that device class, and the standard deviation. When you have multiple OSD sizes, these figures are much less useful when calculated across the whole cluster by “ceph osd df”
# ceph osd df hdd ... 318 hdd 18.53969 1.00000 19 TiB 15 TiB 15 TiB 15 KiB 67 GiB 3.6 TiB 80.79 1.04 127 up 319 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 936 KiB 60 GiB 3.7 TiB 79.87 1.03 129 up 320 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 33 KiB 72 GiB 3.7 TiB 79.99 1.03 129 up 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 11 GiB 15 TiB 17.55 0.23 26 up TOTAL 5.4 PiB 4.2 PiB 4.1 PiB 186 MiB 17 TiB 1.2 PiB 77.81 MIN/MAX VAR: 0.23/1.09 STDDEV: 4.39
You can even run this for a specific OSD so you don’t have to get creative with an egrep regex or exercise your pattern-matching skills, though the summary values naturally aren’t useful.
# ceph osd df osd.30 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 12 GiB 15 TiB 17.55 1.00 26 up TOTAL 19 TiB 3.3 TiB 2.9 TiB 130 KiB 12 GiB 15 TiB 17.55 MIN/MAX VAR: 1.00/1.00 STDDEV: 0
Here there’s a wide variation among the hdd OSDs because osd.30 had been down for a while and was recently restarted due to a host reboot, so it’s slowly filling with data.
ssd_class 6.98630
That seems like an unusual size, what are these? Are they SAN LUNs?
Below are outputs from ceph -s ceph df ceph osd df tree
Thanks for providing the needful up front.
cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
This warning state by default persists for a long time after it clears, I’m not sure why but I like to set this lower:
# ceph config dump | grep blue global advanced bluestore_slow_ops_warn_lifetime 300
1 failed cephadm daemon(s) 39 daemons have recently crashed
That’s a bit worrisome, what happened?
`ceph crash ls`
569 pgs not deep-scrubbed in time 2609 pgs not scrubbed in time
Scrubs don’t happen during recovery, when complete these should catch up.
services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 2m) mgr: ceph-host-1.lqlece(active, since 18h), standbys: ceph-host-2.suiuxi
I’m paranoid and would suggest deploying at least one more mgr.
mds: 19/19 daemons up, 7 standby
Yikes why so many?
osd: 181 osds: 181 up (since 4d), 181 in (since 14h)
What happened 14 hours ago? It seems unusual for these durations to vary so much.
2770 remapped pgs
That’s an indication of balancing or backfill in progress.
flags noautoscale
data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 256.82M objects, 484 TiB usage: 742 TiB used, 1.5 PiB / 2.2 PiB avail pgs: 575889786/1468742421 objects misplaced (39.210%)
39% is a lot of misplaced objects, this would be consistent with you having successfully added those OSDs. Here is where the factor of the most-full OSD comes in.
Technically backfill is a subset of recovery, but in practice people usually think in terms:
Recovery: PGs healing from OSDs having failed or been down Backfill: Rebalancing of data due to topology changes, including adjusted CRUSH rules, expansion, etc.
4247 active+clean 2763 active+remapped+backfill_wait 77 active+clean+scrubbing 43 active+clean+scrubbing+deep 7 active+remapped+backfilling
Configuration options throttle how much backfill goes on in parallel to keep the cluster from DoSing itself. Here I suspect that you’re running a recent release with the notorious mclock op scheduling shortcomings, which is a tangent.
I suggest checking out these two resources re upmap-remapped.py :
https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering Ceph Operations with Upmap.pdf <https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering%20Ceph%20Operations%20with%20Upmap.pdf> https://community.ibm.com/community/user/blogs/anthony-datri/2025/07/30/grac...
This tool, in conjunction with the balancer module, will do the backfill more elegantly with various benefits.
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
I hope the formatting below comes through, makes it a lot easier to read a table.
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB metadata_fs_nvme 13 32 260 MiB 177 780 MiB 0 93 TiB metadata_fs_ssd_rep 14 32 17 MiB 103 51 MiB 0 93 TiB ssd_rep_projects 15 1024 132 B 1 12 KiB 0 130 TiB nvme_rep_projects 16 512 3.5 KiB 30 336 KiB 093 TiB
Do you have multiple EC RBD pools and/or multiple CephFSes?
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2272.93311 - 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 1.00 - root default -7 254.54175 - 255 TiB 104 TiB 102 TiB 9.6 GiB 455 GiB 151 TiB 40.78 1.25 - host ceph-host-1
...
137 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 50 GiB 4.3 TiB 76.47 2.34 449 up osd.137 152 hdd_class 18.19040 1.00000 18 TiB 241 GiB 239 GiB 10 KiB 1.8 GiB 18 TiB 1.29 0.04 7 up osd.152 3.1 TiB 1.5 PiB 32.64 MIN/MAX VAR: 0.00/2.46 STDDEV: 26.17
There ya go. osd.152 must be one of the new OSDs. Note that only 7 PGs are currently resident and that it holds just 4% of the average amount of data on the entire set of OSDs. Run the focused `osd df` above and that number will change slightly.
Here is your least full hdd_class OSD:
151 hdd_class 18.19040 1.00000 18 TiB 38 GiB 37 GiB 6 KiB 1.1 GiB 18 TiB 0.20 0.01 1 up osd.151
And the most full:
180 hdd_class 18.19040 1.00000 18 TiB 198 GiB 197 GiB 10 KiB 1.7 GiB 18 TiB 1.07 0.03 5 up osd.180
I suspect that the most-full is at 107% of average due to the bolus of backfill and/or the balancer not being active. Using upmap-remapped as described above can help avoid this kind of overload.
In a nutshell, the available space will gradually increase as data is backfilled, especially if you have the balancer enabled.
Hi, Appreciate your advice and clear explanations ceph-host-1 has the same number of HDD but I am trying to reconfigure the newest 6 to use NVME for WAL?DB so I that is CEPH does not "see" them Will send another email about this as I am running into strange issue whereas CEPH is seeing the disks as "available" but configure them as OSD failes because "device is in use" lsblk show and LVM on them BUT lvs,vgs,ceph-volume does NOT Will send details on another email Many thanks Steven On Tue, 2 Sept 2025 at 20:10, Anthony D'Atri <aad@dreamsnake.net> wrote:
On Sep 2, 2025, at 6:36 AM, Steven Vacaroaia <stef97@gmail.com> wrote:
Thanks again Anthony
I guess I assumed wrong that "osd.all-available-devices" includes all available devices
I *think* the last match wins when there are multiple OSD specs. Please excerpt them from `ceph orch ls --export` so we can see them in detail.
It seems I should have targeted specifical ones (like osd.hdd_osds)
Going back to the capacity increase I am trying to estimate / calculate the final capacity
Looking at the output of "ceph df" below it would be greatly appreciated if you could please confirm if this is the correct interpretation for hdd_class :
"Out of a total capacity of 1.4 PiB you currently use 579 TiB .
Yes.
The 579 TiB used are the sum of the data used by pools configured to use hdd_class e.g osd.all-available-devices (434) , hdd_ec_archive (135)...etc
Yes. Note that if you have any pools using a CRUSH rule that does not specify a device class, they will use *all* OSDs and hilarity will ensue.
Therefore, when the backfilling / balancing is done (and assuming no more data is added to the pools) the amount of pools "MAX Available" should be 814 TB
Ah. Not so fast. 814 TiB is aggregate raw capacity. Max avail is nuanced. This can be slippery to wrap one's mind around. If your mind is pre-warped, that helps.
Max avail is *per pool*. It indicates the projected amount of user data that may be written to that pool *if* no other pools mapped to the same OSDs gain or lose data. It's a zero sum game.
Note for example that .mgr and default.rgw.buckets.index (and a slew of others) all report 93 TiB max avail. They're all (presumably) replicated size=3 and on the same OSDs. So that *set* of pools can accept ~~ 93 TiB of new data split any way across them.
This is a function of the delta between the 90% full_ratio and that the most-full current OSD is 28.58% full, divided by 3 for replication.
It becomes a bit more complicated for the hdd_class, where it would seem that EC is in effect. It's also a function of the CRUSH topology and rules. I see that for example ceph-host-1 offers ~ 98 CRUSH weight of hdd_class, where ceph-host-1 offers ~ 226 CRUSH weight of hdd_class. Your latest message doesn't show the CRUSH rules and pool attributes but I suspect multiple EC profiles are in use, which complicates the calculation a bit. You have 7 hosts and thus likely 7 CRUSH failure domains. If you have, say, a 5+2 EC pool, that pool's max avail will be limited by the lowest CRUSH weight of the failure domains, since each PG will need to hit one OSD on every host. A 4+2 pool however would have more flexibility in placement and be less constrained by the unequal distribution. Why does ceph-host-1 have fewer HDDs than the other hosts?
"
Is the above correct ?
Note The "MAX available " is increasing - 202 TB now
That's a function of the balancer reducing how full the most-full OSD of that device class is
Steven
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54 nvme_class 293 TiB 293 TiB 216 GiB 216 GiB 0.07 ssd_class 587 TiB 424 TiB 163 TiB 163 TiB 27.80 TOTAL 2.2 PiB 1.5 PiB 742 TiB 742 TiB 32.64
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB
On Mon, 1 Sept 2025 at 17:07, Anthony D'Atri <aad@dreamsnake.net> wrote:
Hi, Thanks Anthony - as always a very useful and comprehensive response
Most welcome.
yes, there were only 139 OSD and , indeed, raw capacity increased I also noticed that the "max available " column from "cepf df" is getting higher (177 TB) so, it seems the capacity is being added
Aye, as expected.
Too many MDS daemons are due to the fact that I have over 150 CEPHFS clients and thought that deploying a daemon on each host for each filesystem is going to provide better performance - was I wrong ?
MDS strategy is nuanced, but I think deploying them redundantly on each back-end node might not give you much. The workload AIUI is more a function of the number and size of files.
High number of remapped PGs are due to improperly using osd.all-available-devices unmaged command so adding the new drives triggered automatically detecting and adding them
Yep, I typically suggest setting OSD services to unmanaged when not actively using them.
I am not sure what I did wrong though - see below output from "ceph orch ls" before adding the drive Shouldn't setting it like that prevent automatic discovery ?
You do have three other OSD rules, at least one of them likely matched and took action.
osd.all-available-devices 0 - 4w <unmanaged>
osd.hdd_osds 72 10m ago 5w *
osd.nvme_osds 25 10m ago 5w *
osd.ssd_osds 84 10m ago 3w ceph-host-1
`ceph orch ls --export`
Resources mentioned are very useful running upmap-remapped.py did bring the number of PGs to be remapped close to zero
It's a super, super useful tool. Letting the balancer do the work incrementally helps deter unexpected OSD fullness issues and if a problem arises, you're a lot closer to HEALTH_OK.
2. upmap-remapped.py | sh
Sometimes 2-3 runs are needed for full effect.
3. change target_max_misplaced_ratio to a higher number than default 0.005 (since we want to rebalance faster and client performance is not a huge issue )
4. enable balancer
5.wait
Doing it like this will, eventually, increase the number misplaced PGs until it is higher than the ratio when , I guess, the balancer stops ( "optimize_result": "Too many objects (0.115742 > 0.090000) are misplaced; try again later )
Exactly.
Should I repeat the process when the number of objects misplaced is higher than the ratio or what is the proper way of doing it ?
As the backfill progresses and the misplaced percentage drops, the balancer will kick in with another increment.
Steven
On Sun, 31 Aug 2025 at 11:08, Anthony D'Atri <aad@dreamsnake.net> wrote:
On Aug 31, 2025, at 4:15 AM, Steven Vacaroaia <stef97@gmail.com> wrote:
Hi,
I have added 42 x 18TB HDD disks ( 6 on each of the 7 servers )
The ultimate answer to Ceph, the cluster, and everything!
My expectation was that the pools configured to use "hdd_class" will have their capacity increased ( e.g. default.rgw.buckets.data which is uses an EC 4+2 pool for data )
First, did the raw capacity increase when you added these drives?
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
Was the number of OSDs previously 139?
It seems it is not happening ...yet ?! Is it because the peering is still going ?
Ceph nomenclature can be mystifying at first. And sometimes at thirteenth.
Peering is daemons checking in with each other to ensure they’re in agreement.
I think you mean backfill / balancing.
The available space reported by “ceph df” for a *pool* is a function of:
* Raw space available in the associated CRUSH rule’s device class (or if the rule isn’t ideal, all device classes) * The cluster’s three full ratios # ceph osd dump | grep ratio * The fullness of the single most-full OSD in the device class
BTW I learned only yesterday that you can restrict `ceph osd df` by specifying a device class, so try running
`ceph osd df hdd_class | tail -10`
Notably, this will show you the min/max variance among OSDs of just that device class, and the standard deviation. When you have multiple OSD sizes, these figures are much less useful when calculated across the whole cluster by “ceph osd df”
# ceph osd df hdd ... 318 hdd 18.53969 1.00000 19 TiB 15 TiB 15 TiB 15 KiB 67 GiB 3.6 TiB 80.79 1.04 127 up 319 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 936 KiB 60 GiB 3.7 TiB 79.87 1.03 129 up 320 hdd 18.53969 1.00000 19 TiB 15 TiB 14 TiB 33 KiB 72 GiB 3.7 TiB 79.99 1.03 129 up 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 11 GiB 15 TiB 17.55 0.23 26 up TOTAL 5.4 PiB 4.2 PiB 4.1 PiB 186 MiB 17 TiB 1.2 PiB 77.81 MIN/MAX VAR: 0.23/1.09 STDDEV: 4.39
You can even run this for a specific OSD so you don’t have to get creative with an egrep regex or exercise your pattern-matching skills, though the summary values naturally aren’t useful.
# ceph osd df osd.30 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 30 hdd 18.53969 1.00000 19 TiB 3.3 TiB 2.9 TiB 129 KiB 12 GiB 15 TiB 17.55 1.00 26 up TOTAL 19 TiB 3.3 TiB 2.9 TiB 130 KiB 12 GiB 15 TiB 17.55 MIN/MAX VAR: 1.00/1.00 STDDEV: 0
Here there’s a wide variation among the hdd OSDs because osd.30 had been down for a while and was recently restarted due to a host reboot, so it’s slowly filling with data.
ssd_class 6.98630
That seems like an unusual size, what are these? Are they SAN LUNs?
Below are outputs from ceph -s ceph df ceph osd df tree
Thanks for providing the needful up front.
cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
This warning state by default persists for a long time after it clears, I’m not sure why but I like to set this lower:
# ceph config dump | grep blue global advanced bluestore_slow_ops_warn_lifetime 300
1 failed cephadm daemon(s)
39 daemons have recently crashed
That’s a bit worrisome, what happened?
`ceph crash ls`
569 pgs not deep-scrubbed in time 2609 pgs not scrubbed in time
Scrubs don’t happen during recovery, when complete these should catch up.
services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 2m) mgr: ceph-host-1.lqlece(active, since 18h), standbys: ceph-host-2.suiuxi
I’m paranoid and would suggest deploying at least one more mgr.
mds: 19/19 daemons up, 7 standby
Yikes why so many?
osd: 181 osds: 181 up (since 4d), 181 in (since 14h)
What happened 14 hours ago? It seems unusual for these durations to vary so much.
2770 remapped pgs
That’s an indication of balancing or backfill in progress.
flags noautoscale
data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 256.82M objects, 484 TiB usage: 742 TiB used, 1.5 PiB / 2.2 PiB avail pgs: 575889786/1468742421 objects misplaced (39.210%)
39% is a lot of misplaced objects, this would be consistent with you having successfully added those OSDs. Here is where the factor of the most-full OSD comes in.
Technically backfill is a subset of recovery, but in practice people usually think in terms:
Recovery: PGs healing from OSDs having failed or been down Backfill: Rebalancing of data due to topology changes, including adjusted CRUSH rules, expansion, etc.
4247 active+clean 2763 active+remapped+backfill_wait 77 active+clean+scrubbing 43 active+clean+scrubbing+deep 7 active+remapped+backfilling
Configuration options throttle how much backfill goes on in parallel to keep the cluster from DoSing itself. Here I suspect that you’re running a recent release with the notorious mclock op scheduling shortcomings, which is a tangent.
I suggest checking out these two resources re upmap-remapped.py :
https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering Ceph Operations with Upmap.pdf
https://community.ibm.com/community/user/blogs/anthony-datri/2025/07/30/grac...
This tool, in conjunction with the balancer module, will do the backfill more elegantly with various benefits.
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54
I hope the formatting below comes through, makes it a lot easier to read a table.
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB metadata_fs_nvme 13 32 260 MiB 177 780 MiB 0 93 TiB metadata_fs_ssd_rep 14 32 17 MiB 103 51 MiB 0 93 TiB ssd_rep_projects 15 1024 132 B 1 12 KiB 0 130 TiB nvme_rep_projects 16 512 3.5 KiB 30 336 KiB 093 TiB
Do you have multiple EC RBD pools and/or multiple CephFSes?
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2272.93311 - 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 1.00 - root default -7 254.54175 - 255 TiB 104 TiB 102 TiB 9.6 GiB 455 GiB 151 TiB 40.78 1.25 - host ceph-host-1
...
137 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 50 GiB 4.3 TiB 76.47 2.34 449 up osd.137 152 hdd_class 18.19040 1.00000 18 TiB 241 GiB 239 GiB 10 KiB 1.8 GiB 18 TiB 1.29 0.04 7 up osd.152 3.1 TiB 1.5 PiB 32.64 MIN/MAX VAR: 0.00/2.46 STDDEV: 26.17
There ya go. osd.152 must be one of the new OSDs. Note that only 7 PGs are currently resident and that it holds just 4% of the average amount of data on the entire set of OSDs. Run the focused `osd df` above and that number will change slightly.
Here is your least full hdd_class OSD:
151 hdd_class 18.19040 1.00000 18 TiB 38 GiB 37 GiB 6 KiB 1.1 GiB 18 TiB 0.20 0.01 1 up osd.151
And the most full:
180 hdd_class 18.19040 1.00000 18 TiB 198 GiB 197 GiB 10 KiB 1.7 GiB 18 TiB 1.07 0.03 5 up osd.180
I suspect that the most-full is at 107% of average due to the bolus of backfill and/or the balancer not being active. Using upmap-remapped as described above can help avoid this kind of overload.
In a nutshell, the available space will gradually increase as data is backfilled, especially if you have the balancer enabled.
participants (2)
-
Anthony D'Atri
-
Steven Vacaroaia