Ceph Health error right after starting balancer
Hi, after enabling ceph balancer (with command ceph balancer on) the health status changed to error. This is the current output of ceph health detail: root@ld3955:~# ceph health detail HEALTH_ERR 1438 slow requests are blocked > 32 sec; 861 stuck requests are blocked > 4096 sec; mon ld5505 is low on available space REQUEST_SLOW 1438 slow requests are blocked > 32 sec 683 ops are blocked > 2097.15 sec 436 ops are blocked > 1048.58 sec 191 ops are blocked > 524.288 sec 78 ops are blocked > 262.144 sec 35 ops are blocked > 131.072 sec 11 ops are blocked > 65.536 sec 4 ops are blocked > 32.768 sec osd.62 has blocked requests > 65.536 sec osds 39,72 have blocked requests > 262.144 sec osds 6,19,67,173,174,187,188,269,434 have blocked requests > 524.288 sec osds 8,16,35,36,37,61,63,64,68,73,75,178,186,271,369,420,429,431,433,436 have blocked requests > 1048.58 sec osds 3,5,7,24,34,38,40,41,59,66,69,74,180,270,370,421,432,435 have blocked requests > 2097.15 sec REQUEST_STUCK 861 stuck requests are blocked > 4096 sec 25 ops are blocked > 8388.61 sec 836 ops are blocked > 4194.3 sec osds 2,28,29,32,60,65,181,185,268,368,423,424,426 have stuck requests > 4194.3 sec osds 0,30,70,71,184 have stuck requests > 8388.61 sec I understand that when balancer starts shifting PGs to other OSDs that this caused IO load on the cluster. However I don't understand why this is affecting OSD so heavily. And I don't understand why OSD of specific type (SSD, NVME) suffer although there's no balancing occuring on them. Regards Thomas
Requests stuck for > 2 hours cannot be attributed to "IO load on the cluster". Looks like some OSDs really are stuck, things to try: * run "ceph daemon osd.X dump_blocked_ops" on one of the affected OSDs to see what is stuck * try restarting OSDs to see if it clears up automatically Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Oct 31, 2019 at 2:27 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
after enabling ceph balancer (with command ceph balancer on) the health status changed to error. This is the current output of ceph health detail: root@ld3955:~# ceph health detail HEALTH_ERR 1438 slow requests are blocked > 32 sec; 861 stuck requests are blocked > 4096 sec; mon ld5505 is low on available space REQUEST_SLOW 1438 slow requests are blocked > 32 sec 683 ops are blocked > 2097.15 sec 436 ops are blocked > 1048.58 sec 191 ops are blocked > 524.288 sec 78 ops are blocked > 262.144 sec 35 ops are blocked > 131.072 sec 11 ops are blocked > 65.536 sec 4 ops are blocked > 32.768 sec osd.62 has blocked requests > 65.536 sec osds 39,72 have blocked requests > 262.144 sec osds 6,19,67,173,174,187,188,269,434 have blocked requests > 524.288 sec osds 8,16,35,36,37,61,63,64,68,73,75,178,186,271,369,420,429,431,433,436 have blocked requests > 1048.58 sec osds 3,5,7,24,34,38,40,41,59,66,69,74,180,270,370,421,432,435 have blocked requests > 2097.15 sec REQUEST_STUCK 861 stuck requests are blocked > 4096 sec 25 ops are blocked > 8388.61 sec 836 ops are blocked > 4194.3 sec osds 2,28,29,32,60,65,181,185,268,368,423,424,426 have stuck requests > 4194.3 sec osds 0,30,70,71,184 have stuck requests > 8388.61 sec
I understand that when balancer starts shifting PGs to other OSDs that this caused IO load on the cluster. However I don't understand why this is affecting OSD so heavily. And I don't understand why OSD of specific type (SSD, NVME) suffer although there's no balancing occuring on them.
Regards Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This is the output of OSD.270 that remains with slow requests blocked even after restarting. What's the interpretation of it? root@ld5507:~# ceph daemon osd.270 dump_blocked_ops [330/1857] { "ops": [ { "description": "osd_pg_create(e293649 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:22:13.563017", "age": 2785.2698560419999, "duration": 2785.2699056299998, "type_data": { "flag_point": "started", "events": [ { "time": "2019-10-31 19:22:13.563017", "event": "initiated" }, { "time": "2019-10-31 19:22:13.563017", "event": "header_read" }, { "time": "2019-10-31 19:22:13.563011", "event": "throttled" }, { "time": "2019-10-31 19:22:13.563024", "event": "all_read" }, { "time": "2019-10-31 20:07:43.881441", "event": "dispatched" }, [300/1857] { "time": "2019-10-31 20:07:43.881472", "event": "wait for new map" }, { "time": "2019-10-31 20:07:44.665714", "event": "started" } ] } }, { "description": "osd_pg_create(e293650 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:23:16.150040", "age": 2722.6828331659999, "duration": 2722.683007228, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:23:16.150040", "event": "initiated" }, { "time": "2019-10-31 19:23:16.150040", "event": "header_read" }, { "time": "2019-10-31 19:23:16.150035", "event": "throttled" }, [269/1857] { "time": "2019-10-31 19:23:16.150055", "event": "all_read" }, { "time": "2019-10-31 20:07:43.882197", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.882198", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293651 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:23:17.779034", "age": 2721.0538393319998, "duration": 2721.0541152350002, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:23:17.779034", "event": "initiated" }, { "time": "2019-10-31 19:23:17.779034", "event": "header_read" }, [238/1857] { "time": "2019-10-31 19:23:17.779027", "event": "throttled" }, { "time": "2019-10-31 19:23:17.779044", "event": "all_read" }, { "time": "2019-10-31 20:07:43.882326", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.882328", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293652 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:23:22.178466", "age": 2716.6544071140002, "duration": 2716.654784073, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:23:22.178466", "event": "initiated" }, [207/1857] { "time": "2019-10-31 19:23:22.178466", "event": "header_read" }, { "time": "2019-10-31 19:23:22.178463", "event": "throttled" }, { "time": "2019-10-31 19:23:22.178471", "event": "all_read" }, { "time": "2019-10-31 20:07:43.882355", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.882356", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293653 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:23:28.227563", "age": 2710.6053095460002, "duration": 2710.6057835810002, "type_data": { "flag_point": "delayed", "events": [ [176/1857] { "time": "2019-10-31 19:23:28.227563", "event": "initiated" }, { "time": "2019-10-31 19:23:28.227563", "event": "header_read" }, { "time": "2019-10-31 19:23:28.227559", "event": "throttled" }, { "time": "2019-10-31 19:23:28.227570", "event": "all_read" }, { "time": "2019-10-31 20:07:43.882383", "event": "dispatched" }, "age": 2574.9672041270001, [145/1857] "duration": 2574.9677752090001, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:25:43.865669", "event": "initiated" }, { "time": "2019-10-31 19:25:43.865669", "event": "header_read" }, { "time": "2019-10-31 19:25:43.865663", "event": "throttled" }, { "time": "2019-10-31 19:25:43.865682", "event": "all_read" }, { "time": "2019-10-31 20:07:43.883843", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.883846", "event": "wait for new map" } [117/1857] ] } }, { "description": "osd_pg_create(e293655 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:26:16.129187", "age": 2542.703685858, "duration": 2542.7043550879998, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:26:16.129187", "event": "initiated" }, { "time": "2019-10-31 19:26:16.129187", "event": "header_read" }, { "time": "2019-10-31 19:26:16.129181", "event": "throttled" }, { "time": "2019-10-31 19:26:16.129197", "event": "all_read" }, { "time": "2019-10-31 20:07:43.883928", "event": "dispatched" }, [86/1857] { "time": "2019-10-31 20:07:43.883929", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293656 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:30:47.890777", "age": 2270.9420963450002, "duration": 2270.9428696479999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:30:47.890777", "event": "initiated" }, { "time": "2019-10-31 19:30:47.890777", "event": "header_read" }, { "time": "2019-10-31 19:30:47.890772", "event": "throttled" }, { "time": "2019-10-31 19:30:47.890788", "event": "all_read" }, [55/1857] { "time": "2019-10-31 20:07:43.913783", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.913786", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293657 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:35:32.323775", "age": 1986.5090976690001, "duration": 1986.509968458, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:35:32.323775", "event": "initiated" }, { "time": "2019-10-31 19:35:32.323775", "event": "header_read" }, { "time": "2019-10-31 19:35:32.323769", "event": "throttled" }, [24/1857] { "time": "2019-10-31 19:35:32.323802", "event": "all_read" }, { "time": "2019-10-31 20:07:43.917500", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.917503", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293658 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:36:32.783376", "age": 1926.0494967300001, "duration": 1926.050464765, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:36:32.783376", "event": "initiated" }, { "time": "2019-10-31 19:36:32.783376", "event": "header_read" }, { "time": "2019-10-31 19:36:32.783373", "event": "throttled" }, { "time": "2019-10-31 19:36:32.783388", "event": "all_read" }, { "time": "2019-10-31 20:07:43.919280", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.919282", "event": "wait for new map" } ] } } ], "complaint_time": 30, "num_blocked_ops": 10 } { "time": "2019-10-31 20:07:43.882384", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293654 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:25:43.865669", Am 31.10.2019 um 18:27 schrieb Paul Emmerich:
ceph daemon osd.X dump_blocked_ops
Looks like you didn't tell the whole story, please post the *full* output of ceph -s and ceph osd df tree. Wild guess: you need to increase "mon max pg per osd" Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Oct 31, 2019 at 8:17 PM Thomas <74cmonty@gmail.com> wrote:
This is the output of OSD.270 that remains with slow requests blocked even after restarting. What's the interpretation of it?
root@ld5507:~# ceph daemon osd.270 dump_blocked_ops [330/1857] { "ops": [ { "description": "osd_pg_create(e293649 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:22:13.563017", "age": 2785.2698560419999, "duration": 2785.2699056299998, "type_data": { "flag_point": "started", "events": [ { "time": "2019-10-31 19:22:13.563017", "event": "initiated" }, { "time": "2019-10-31 19:22:13.563017", "event": "header_read" }, { "time": "2019-10-31 19:22:13.563011", "event": "throttled" }, { "time": "2019-10-31 19:22:13.563024", "event": "all_read" }, { "time": "2019-10-31 20:07:43.881441", "event": "dispatched" }, [300/1857] { "time": "2019-10-31 20:07:43.881472", "event": "wait for new map" }, { "time": "2019-10-31 20:07:44.665714", "event": "started" } ] } }, { "description": "osd_pg_create(e293650 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:23:16.150040", "age": 2722.6828331659999, "duration": 2722.683007228, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:23:16.150040", "event": "initiated" }, { "time": "2019-10-31 19:23:16.150040", "event": "header_read" }, { "time": "2019-10-31 19:23:16.150035", "event": "throttled" }, [269/1857] { "time": "2019-10-31 19:23:16.150055", "event": "all_read" }, { "time": "2019-10-31 20:07:43.882197", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.882198", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293651 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:23:17.779034", "age": 2721.0538393319998, "duration": 2721.0541152350002, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:23:17.779034", "event": "initiated" }, { "time": "2019-10-31 19:23:17.779034", "event": "header_read" }, [238/1857] { "time": "2019-10-31 19:23:17.779027", "event": "throttled" }, { "time": "2019-10-31 19:23:17.779044", "event": "all_read" }, { "time": "2019-10-31 20:07:43.882326", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.882328", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293652 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:23:22.178466", "age": 2716.6544071140002, "duration": 2716.654784073, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:23:22.178466", "event": "initiated" }, [207/1857] { "time": "2019-10-31 19:23:22.178466", "event": "header_read" }, { "time": "2019-10-31 19:23:22.178463", "event": "throttled" }, { "time": "2019-10-31 19:23:22.178471", "event": "all_read" }, { "time": "2019-10-31 20:07:43.882355", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.882356", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293653 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:23:28.227563", "age": 2710.6053095460002, "duration": 2710.6057835810002, "type_data": { "flag_point": "delayed", "events": [ [176/1857] { "time": "2019-10-31 19:23:28.227563", "event": "initiated" }, { "time": "2019-10-31 19:23:28.227563", "event": "header_read" }, { "time": "2019-10-31 19:23:28.227559", "event": "throttled" }, { "time": "2019-10-31 19:23:28.227570", "event": "all_read" }, { "time": "2019-10-31 20:07:43.882383", "event": "dispatched" }, "age": 2574.9672041270001, [145/1857] "duration": 2574.9677752090001, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:25:43.865669", "event": "initiated" }, { "time": "2019-10-31 19:25:43.865669", "event": "header_read" }, { "time": "2019-10-31 19:25:43.865663", "event": "throttled" }, { "time": "2019-10-31 19:25:43.865682", "event": "all_read" }, { "time": "2019-10-31 20:07:43.883843", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.883846", "event": "wait for new map" } [117/1857] ] } }, { "description": "osd_pg_create(e293655 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:26:16.129187", "age": 2542.703685858, "duration": 2542.7043550879998, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:26:16.129187", "event": "initiated" }, { "time": "2019-10-31 19:26:16.129187", "event": "header_read" }, { "time": "2019-10-31 19:26:16.129181", "event": "throttled" }, { "time": "2019-10-31 19:26:16.129197", "event": "all_read" }, { "time": "2019-10-31 20:07:43.883928", "event": "dispatched" }, [86/1857] { "time": "2019-10-31 20:07:43.883929", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293656 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:30:47.890777", "age": 2270.9420963450002, "duration": 2270.9428696479999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:30:47.890777", "event": "initiated" }, { "time": "2019-10-31 19:30:47.890777", "event": "header_read" }, { "time": "2019-10-31 19:30:47.890772", "event": "throttled" }, { "time": "2019-10-31 19:30:47.890788", "event": "all_read" }, [55/1857] { "time": "2019-10-31 20:07:43.913783", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.913786", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293657 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:35:32.323775", "age": 1986.5090976690001, "duration": 1986.509968458, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:35:32.323775", "event": "initiated" }, { "time": "2019-10-31 19:35:32.323775", "event": "header_read" }, { "time": "2019-10-31 19:35:32.323769", "event": "throttled" }, [24/1857] { "time": "2019-10-31 19:35:32.323802", "event": "all_read" }, { "time": "2019-10-31 20:07:43.917500", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.917503", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293658 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:36:32.783376", "age": 1926.0494967300001, "duration": 1926.050464765, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-10-31 19:36:32.783376", "event": "initiated" }, { "time": "2019-10-31 19:36:32.783376", "event": "header_read" }, { "time": "2019-10-31 19:36:32.783373", "event": "throttled" }, { "time": "2019-10-31 19:36:32.783388", "event": "all_read" }, { "time": "2019-10-31 20:07:43.919280", "event": "dispatched" }, { "time": "2019-10-31 20:07:43.919282", "event": "wait for new map" } ] } } ], "complaint_time": 30, "num_blocked_ops": 10 }
{ "time": "2019-10-31 20:07:43.882384", "event": "wait for new map" } ] } }, { "description": "osd_pg_create(e293654 59.b:267033 59.2c:267033)", "initiated_at": "2019-10-31 19:25:43.865669",
Am 31.10.2019 um 18:27 schrieb Paul Emmerich:
ceph daemon osd.X dump_blocked_ops
Hi Paul, the situation has changed in the meantime. However, I can reproduce a similar behaviour. This means, - I disable balancer (ceph balancer off) - and then start reweighting of a specific OSD (ceph osd reweight 134 1.0) The cluster immediatelly reports slow requests. root@ld3955:~# ceph health detail HEALTH_WARN 434 slow requests are blocked > 32 sec; mon ld5505 is low on available space REQUEST_SLOW 434 slow requests are blocked > 32 sec 131 ops are blocked > 131.072 sec 270 ops are blocked > 65.536 sec 33 ops are blocked > 32.768 sec osd.66 has blocked requests > 32.768 sec osds 19,65,67,426 have blocked requests > 65.536 sec osds 0,2,3,5,6,7,8,16,24,28,29,30,32,34,35,36,37,38,39,40,41,59,60,61,62,63,64,68,69,70,71,72,73,74,75,173,174,178,180,181,184,185 ,186,187,188,268,269,270,271,368,369,370,420,421,423,424,429,431,432,433,434,435,436 have blocked requests > 131.072 sec MON_DISK_LOW mon ld5505 is low on available space mon.ld5505 has 24% avail root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 453 slow requests are blocked > 32 sec mon ld5505 is low on available space services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 23h) mgr: ld5505(active, since 22h), standbys: ld5506, ld5507, ld5508 mds: cephfs:1 {0=ld4465=up:active} 1 up:standby osd: 442 osds: 442 up, 442 in; 5 remapped pgs data: pools: 6 pools, 8312 pgs objects: 63.92M objects, 244 TiB usage: 731 TiB used, 800 TiB / 1.5 PiB avail pgs: 37702/191053577 objects misplaced (0.020%) 8249 active+clean 29 active+clean+scrubbing+deep 29 active+clean+scrubbing 3 active+remapped+backfill_wait 2 active+remapped+backfilling io: client: 1.4 KiB/s rd, 87 MiB/s wr, 1 op/s rd, 22 op/s wr recovery: 34 MiB/s, 8 objects/s In this example I have reweight an HDD, but many OSDs that have blocked requests (0,2,3,5,6,7,8) are SSDs. root@ld3955:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -17 1363.19983 - 1.3 PiB 725 TiB 724 TiB 31 MiB 1.4 TiB 637 TiB 53.25 1.12 - root hdd_strgbox -43 349.43994 - 349 TiB 171 TiB 171 TiB 6.3 MiB 300 GiB 178 TiB 48.91 1.02 - host ld4257-hdd_strgbox 371 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 180 KiB 5.9 GiB 3.8 TiB 47.84 1.00 118 up osd.371 372 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 128 KiB 6.2 GiB 3.7 TiB 48.65 1.02 120 up osd.372 373 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 8 KiB 6.1 GiB 3.7 TiB 48.71 1.02 120 up osd.373 374 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 68 KiB 6.0 GiB 3.8 TiB 47.34 0.99 117 up osd.374 375 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 208 KiB 6.7 GiB 3.4 TiB 53.49 1.12 132 up osd.375 376 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 72 KiB 5.7 GiB 4.2 TiB 42.11 0.88 104 up osd.376 377 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 120 KiB 5.7 GiB 4.2 TiB 42.13 0.88 104 up osd.377 378 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 176 KiB 6.0 GiB 3.8 TiB 47.37 0.99 117 up osd.378 379 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 32 KiB 6.2 GiB 3.7 TiB 49.47 1.04 122 up osd.379 380 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 168 KiB 5.8 GiB 4.0 TiB 45.33 0.95 112 up osd.380 381 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 284 KiB 6.4 GiB 3.6 TiB 50.30 1.05 124 up osd.381 382 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 12 KiB 5.8 GiB 3.9 TiB 46.92 0.98 116 up osd.382 383 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 172 KiB 6.2 GiB 3.7 TiB 49.75 1.04 123 up osd.383 384 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 60 KiB 7.3 GiB 3.5 TiB 51.88 1.09 128 up osd.384 385 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 76 KiB 6.5 GiB 3.8 TiB 47.10 0.99 116 up osd.385 386 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 84 KiB 7.2 GiB 3.4 TiB 53.83 1.13 133 up osd.386 387 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 200 KiB 6.3 GiB 3.7 TiB 49.36 1.03 122 up osd.387 388 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 72 KiB 5.8 GiB 3.9 TiB 46.62 0.98 115 up osd.388 389 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 276 KiB 6.6 GiB 3.5 TiB 52.24 1.09 128 up osd.389 390 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 72 KiB 5.3 GiB 4.2 TiB 42.24 0.88 104 up osd.390 391 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 148 KiB 5.8 GiB 3.9 TiB 46.57 0.98 115 up osd.391 392 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 120 KiB 6.8 GiB 3.4 TiB 53.86 1.13 133 up osd.392 393 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 12 KiB 7.6 GiB 3.4 TiB 53.92 1.13 133 up osd.393 394 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 80 KiB 5.8 GiB 3.9 TiB 46.60 0.98 115 up osd.394 395 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 64 KiB 7.4 GiB 3.4 TiB 53.92 1.13 133 up osd.395 396 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 244 KiB 6.4 GiB 3.5 TiB 51.46 1.08 127 up osd.396 397 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 240 KiB 6.3 GiB 3.7 TiB 48.95 1.02 121 up osd.397 398 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 124 KiB 6.0 GiB 3.8 TiB 47.84 1.00 118 up osd.398 399 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 8 KiB 5.9 GiB 3.9 TiB 47.03 0.98 116 up osd.399 400 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 248 KiB 6.4 GiB 3.6 TiB 50.69 1.06 125 up osd.400 401 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 116 KiB 5.6 GiB 3.9 TiB 46.18 0.97 114 up osd.401 402 hdd 7.28000 1.00000 7.3 TiB 3.0 TiB 3.0 TiB 144 KiB 5.2 GiB 4.2 TiB 41.75 0.87 103 up osd.402 403 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 180 KiB 5.9 GiB 3.8 TiB 47.79 1.00 118 up osd.403 404 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 56 KiB 6.0 GiB 4.1 TiB 44.13 0.92 109 up osd.404 405 hdd 7.28000 1.00000 7.3 TiB 3.0 TiB 3.0 TiB 52 KiB 5.3 GiB 4.2 TiB 41.72 0.87 103 up osd.405 406 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 300 KiB 6.7 GiB 3.4 TiB 53.51 1.12 132 up osd.406 407 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 284 KiB 6.5 GiB 3.5 TiB 51.42 1.08 127 up osd.407 408 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 60 KiB 6.9 GiB 3.4 TiB 53.61 1.12 132 up osd.408 409 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 188 KiB 6.4 GiB 3.5 TiB 52.53 1.10 130 up osd.409 410 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 28 KiB 7.2 GiB 3.4 TiB 53.49 1.12 132 up osd.410 411 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 272 KiB 6.6 GiB 3.4 TiB 53.08 1.11 131 up osd.411 412 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 8 KiB 7.4 GiB 3.4 TiB 53.89 1.13 133 up osd.412 413 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 24 KiB 5.6 GiB 4.0 TiB 44.96 0.94 111 up osd.413 414 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 128 KiB 5.7 GiB 3.9 TiB 45.76 0.96 113 up osd.414 415 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 300 KiB 6.3 GiB 3.5 TiB 51.48 1.08 127 up osd.415 416 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.6 TiB 264 KiB 6.3 GiB 3.6 TiB 50.21 1.05 124 up osd.416 417 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 72 KiB 6.0 GiB 3.8 TiB 48.24 1.01 119 up osd.417 418 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.6 TiB 208 KiB 6.3 GiB 3.6 TiB 50.19 1.05 124 up osd.418 -31 349.44000 - 349 TiB 172 TiB 171 TiB 5.0 MiB 342 GiB 178 TiB 49.11 1.03 - host ld4464-hdd_strgbox 272 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.1 TiB 212 KiB 6.3 GiB 4.1 TiB 43.30 0.91 107 up osd.272 273 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 64 KiB 7.9 GiB 3.4 TiB 53.09 1.11 131 up osd.273 274 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 76 KiB 7.3 GiB 3.5 TiB 52.34 1.10 129 up osd.274 275 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 60 KiB 6.1 GiB 4.1 TiB 43.42 0.91 107 up osd.275 276 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 80 KiB 7.2 GiB 3.8 TiB 48.15 1.01 119 up osd.276 277 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 72 KiB 7.2 GiB 3.6 TiB 50.62 1.06 125 up osd.277 278 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 20 KiB 7.3 GiB 3.7 TiB 49.43 1.03 122 up osd.278 279 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 92 KiB 7.0 GiB 3.8 TiB 47.12 0.99 116 up osd.279 280 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 48 KiB 7.9 GiB 3.6 TiB 51.14 1.07 126 up osd.280 281 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 60 KiB 6.5 GiB 4.1 TiB 43.45 0.91 107 up osd.281 282 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 120 KiB 7.4 GiB 3.4 TiB 53.09 1.11 131 up osd.282 283 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 68 KiB 7.6 GiB 3.4 TiB 53.92 1.13 133 up osd.283 284 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 108 KiB 7.3 GiB 3.6 TiB 50.65 1.06 125 up osd.284 285 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 60 KiB 8.1 GiB 3.4 TiB 53.83 1.13 133 up osd.285 286 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 120 KiB 7.2 GiB 3.9 TiB 46.24 0.97 114 up osd.286 287 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.1 TiB 180 KiB 6.4 GiB 4.1 TiB 43.31 0.91 107 up osd.287 288 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.1 TiB 60 KiB 6.7 GiB 4.1 TiB 43.36 0.91 107 up osd.288 289 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 128 KiB 7.0 GiB 3.9 TiB 46.18 0.97 114 up osd.289 290 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 196 KiB 6.5 GiB 4.1 TiB 43.40 0.91 107 up osd.290 291 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 100 KiB 7.3 GiB 3.7 TiB 49.39 1.03 122 up osd.291 292 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 80 KiB 6.8 GiB 3.8 TiB 48.19 1.01 119 up osd.292 293 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 72 KiB 7.8 GiB 3.4 TiB 53.06 1.11 131 up osd.293 294 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 211 KiB 6.7 GiB 4.1 TiB 43.03 0.90 106 up osd.294 295 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 52 KiB 7.9 GiB 3.4 TiB 53.90 1.13 133 up osd.295 296 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 228 KiB 6.8 GiB 3.8 TiB 47.40 0.99 117 up osd.296 297 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.4 TiB 12 KiB 6.8 GiB 3.8 TiB 47.42 0.99 117 up osd.297 298 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 132 KiB 7.3 GiB 3.5 TiB 51.87 1.09 128 up osd.298 299 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 12 KiB 7.3 GiB 3.7 TiB 49.79 1.04 123 up osd.299 300 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 180 KiB 7.3 GiB 3.4 TiB 53.05 1.11 131 up osd.300 301 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 180 KiB 7.3 GiB 3.4 TiB 53.86 1.13 133 up osd.301 302 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 72 KiB 7.0 GiB 3.7 TiB 49.41 1.03 122 up osd.302 303 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 64 KiB 6.8 GiB 3.9 TiB 46.32 0.97 114 up osd.303 304 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 40 KiB 6.9 GiB 3.7 TiB 48.59 1.02 120 up osd.304 305 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 68 KiB 7.1 GiB 3.8 TiB 48.23 1.01 119 up osd.305 306 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 120 KiB 7.5 GiB 3.5 TiB 51.87 1.09 128 up osd.306 307 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 116 KiB 6.6 GiB 4.1 TiB 43.42 0.91 107 up osd.307 308 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 128 KiB 6.0 GiB 4.2 TiB 42.92 0.90 106 up osd.308 309 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 116 KiB 7.6 GiB 3.4 TiB 53.57 1.12 132 up osd.309 310 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 64 KiB 6.6 GiB 3.9 TiB 46.63 0.98 115 up osd.310 311 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 60 KiB 7.5 GiB 3.6 TiB 50.68 1.06 125 up osd.311 312 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 164 KiB 7.5 GiB 3.4 TiB 53.43 1.12 132 up osd.312 313 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 244 KiB 6.6 GiB 3.9 TiB 45.79 0.96 113 up osd.313 314 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 84 KiB 7.6 GiB 3.4 TiB 53.78 1.13 133 up osd.314 315 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 272 KiB 7.4 GiB 3.4 TiB 53.04 1.11 131 up osd.315 316 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 180 KiB 7.4 GiB 3.5 TiB 51.81 1.08 128 up osd.316 317 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 120 KiB 7.6 GiB 3.5 TiB 51.90 1.09 128 up osd.317 318 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 12 KiB 6.9 GiB 3.9 TiB 46.27 0.97 114 up osd.318 319 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 92 KiB 7.3 GiB 3.4 TiB 52.62 1.10 130 up osd.319 -40 349.43994 - 349 TiB 172 TiB 172 TiB 4.5 MiB 345 GiB 177 TiB 49.33 1.03 - host ld4465-hdd_strgbox 320 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 44 KiB 6.9 GiB 3.9 TiB 46.20 0.97 114 up osd.320 321 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 72 KiB 7.5 GiB 3.4 TiB 53.53 1.12 132 up osd.321 322 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 64 KiB 7.1 GiB 3.7 TiB 48.59 1.02 120 up osd.322 323 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 64 KiB 6.9 GiB 3.8 TiB 47.76 1.00 118 up osd.323 324 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 16 KiB 7.4 GiB 3.9 TiB 47.08 0.99 116 up osd.324 325 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 64 KiB 7.8 GiB 3.5 TiB 52.28 1.09 129 up osd.325 326 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 108 KiB 7.7 GiB 3.6 TiB 50.16 1.05 124 up osd.326 327 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 16 KiB 7.3 GiB 3.8 TiB 48.13 1.01 119 up osd.327 328 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 16 KiB 7.5 GiB 3.4 TiB 53.50 1.12 132 up osd.328 329 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 220 KiB 6.8 GiB 3.8 TiB 47.79 1.00 118 up osd.329 330 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 72 KiB 7.7 GiB 3.4 TiB 53.52 1.12 132 up osd.330 331 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 60 KiB 7.2 GiB 3.8 TiB 47.37 0.99 117 up osd.331 332 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 176 KiB 7.5 GiB 3.4 TiB 53.45 1.12 132 up osd.332 333 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 72 KiB 7.0 GiB 3.9 TiB 45.80 0.96 113 up osd.333 334 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 60 KiB 7.4 GiB 3.4 TiB 52.59 1.10 130 up osd.334 335 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 32 KiB 6.6 GiB 4.0 TiB 45.38 0.95 112 up osd.335 336 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 164 KiB 7.4 GiB 3.4 TiB 53.42 1.12 132 up osd.336 337 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.6 TiB 72 KiB 7.4 GiB 3.6 TiB 50.22 1.05 124 up osd.337 338 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 160 KiB 7.3 GiB 3.4 TiB 53.55 1.12 132 up osd.338 339 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.6 TiB 68 KiB 7.7 GiB 3.6 TiB 50.21 1.05 124 up osd.339 340 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 120 KiB 7.0 GiB 3.9 TiB 45.79 0.96 113 up osd.340 341 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 76 KiB 7.0 GiB 3.9 TiB 47.07 0.99 116 up osd.341 342 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 120 KiB 7.0 GiB 3.9 TiB 45.95 0.96 113 up osd.342 343 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 64 KiB 7.2 GiB 3.7 TiB 48.64 1.02 120 up osd.343 344 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 192 KiB 7.4 GiB 3.4 TiB 53.54 1.12 132 up osd.344 345 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 88 KiB 7.3 GiB 3.6 TiB 50.70 1.06 125 up osd.345 346 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 64 KiB 7.2 GiB 3.9 TiB 47.03 0.98 116 up osd.346 347 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 184 KiB 7.4 GiB 3.4 TiB 53.12 1.11 131 up osd.347 348 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 4 KiB 7.1 GiB 3.9 TiB 45.77 0.96 113 up osd.348 349 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 68 KiB 7.6 GiB 3.4 TiB 53.04 1.11 131 up osd.349 350 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 132 KiB 7.2 GiB 3.5 TiB 52.27 1.09 129 up osd.350 351 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 304 KiB 6.5 GiB 3.9 TiB 45.76 0.96 113 up osd.351 352 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 64 KiB 7.5 GiB 3.5 TiB 52.21 1.09 129 up osd.352 353 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 40 KiB 7.4 GiB 3.6 TiB 49.90 1.04 123 up osd.353 354 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 108 KiB 7.5 GiB 3.4 TiB 53.50 1.12 132 up osd.354 355 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 244 KiB 6.5 GiB 4.0 TiB 45.37 0.95 112 up osd.355 356 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 120 KiB 6.9 GiB 3.7 TiB 49.37 1.03 122 up osd.356 357 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 44 KiB 7.1 GiB 4.0 TiB 45.37 0.95 112 up osd.357 358 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 72 KiB 7.1 GiB 3.7 TiB 49.40 1.03 122 up osd.358 359 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 16 KiB 6.8 GiB 3.7 TiB 49.53 1.04 122 up osd.359 360 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 72 KiB 6.7 GiB 4.0 TiB 45.47 0.95 112 up osd.360 361 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 64 KiB 6.9 GiB 4.0 TiB 45.37 0.95 112 up osd.361 362 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 100 KiB 6.7 GiB 3.9 TiB 45.89 0.96 113 up osd.362 363 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 244 KiB 6.6 GiB 4.0 TiB 45.36 0.95 112 up osd.363 364 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 120 KiB 7.1 GiB 3.7 TiB 49.37 1.03 122 up osd.364 365 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 88 KiB 7.4 GiB 3.6 TiB 50.69 1.06 125 up osd.365 366 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 32 KiB 7.2 GiB 3.8 TiB 48.24 1.01 119 up osd.366 367 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 124 KiB 7.5 GiB 3.4 TiB 53.43 1.12 132 up osd.367 -16 78.71997 - 79 TiB 53 TiB 53 TiB 3.8 MiB 107 GiB 25 TiB 67.93 1.42 - host ld5505-hdd_strgbox 76 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 4 KiB 2.3 GiB 436 GiB 73.98 1.55 41 up osd.76 77 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 96 KiB 2.1 GiB 587 GiB 64.98 1.36 36 up osd.77 78 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.0 GiB 584 GiB 65.16 1.36 36 up osd.78 79 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 80 KiB 2.4 GiB 466 GiB 72.18 1.51 40 up osd.79 80 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 92 KiB 2.1 GiB 586 GiB 65.02 1.36 36 up osd.80 81 hdd 1.64000 1.00000 1.6 TiB 879 GiB 877 GiB 120 KiB 2.0 GiB 797 GiB 52.45 1.10 29 up osd.81 82 hdd 1.64000 1.00000 1.6 TiB 974 GiB 972 GiB 64 KiB 1.9 GiB 702 GiB 58.13 1.22 32 up osd.82 83 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 16 KiB 2.3 GiB 492 GiB 70.64 1.48 39 up osd.83 84 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 60 KiB 2.0 GiB 614 GiB 63.36 1.33 35 up osd.84 85 hdd 1.64000 1.00000 1.6 TiB 880 GiB 878 GiB 64 KiB 1.7 GiB 796 GiB 52.48 1.10 29 up osd.85 86 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 80 KiB 2.4 GiB 434 GiB 74.13 1.55 41 up osd.86 87 hdd 1.64000 1.00000 1.6 TiB 971 GiB 969 GiB 60 KiB 1.9 GiB 705 GiB 57.95 1.21 32 up osd.87 88 hdd 1.64000 1.00000 1.6 TiB 882 GiB 880 GiB 120 KiB 1.8 GiB 794 GiB 52.63 1.10 29 up osd.88 89 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 116 KiB 2.4 GiB 463 GiB 72.38 1.52 40 up osd.89 90 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.5 GiB 404 GiB 75.88 1.59 42 up osd.90 91 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 132 KiB 2.3 GiB 494 GiB 70.54 1.48 39 up osd.91 92 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 164 KiB 2.1 GiB 587 GiB 64.96 1.36 36 up osd.92 93 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 96 KiB 2.1 GiB 612 GiB 63.51 1.33 35 up osd.93 94 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 88 KiB 2.4 GiB 463 GiB 72.38 1.52 40 up osd.94 95 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.4 GiB 407 GiB 75.74 1.59 42 up osd.95 96 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 120 KiB 2.0 GiB 645 GiB 61.53 1.29 34 up osd.96 97 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 44 KiB 2.5 GiB 346 GiB 79.33 1.66 44 up osd.97 98 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 132 KiB 2.3 GiB 400 GiB 76.12 1.59 42 up osd.98 99 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.3 GiB 491 GiB 70.68 1.48 39 up osd.99 100 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 124 KiB 2.4 GiB 343 GiB 79.51 1.66 44 up osd.100 101 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.2 GiB 525 GiB 68.66 1.44 38 up osd.101 102 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 116 KiB 2.3 GiB 434 GiB 74.09 1.55 41 up osd.102 103 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.3 GiB 436 GiB 74.00 1.55 41 up osd.103 104 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 76 KiB 2.0 GiB 616 GiB 63.23 1.32 35 up osd.104 105 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.5 GiB 404 GiB 75.88 1.59 42 up osd.105 106 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.1 GiB 558 GiB 66.72 1.40 37 up osd.106 107 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 12 KiB 2.6 GiB 462 GiB 72.45 1.52 40 up osd.107 108 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 124 KiB 2.0 GiB 647 GiB 61.41 1.29 34 up osd.108 109 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.1 GiB 587 GiB 64.97 1.36 36 up osd.109 110 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.1 GiB 584 GiB 65.16 1.36 36 up osd.110 111 hdd 1.64000 1.00000 1.6 TiB 880 GiB 878 GiB 88 KiB 1.7 GiB 796 GiB 52.48 1.10 29 up osd.111 112 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 160 KiB 2.1 GiB 616 GiB 63.23 1.32 35 up osd.112 113 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 12 KiB 3.1 GiB 431 GiB 74.28 1.56 41 up osd.113 114 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 40 KiB 2.2 GiB 617 GiB 63.20 1.32 35 up osd.114 115 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 85 KiB 2.3 GiB 466 GiB 72.19 1.51 40 up osd.115 116 hdd 1.64000 1.00000 1.6 TiB 908 GiB 906 GiB 64 KiB 1.8 GiB 768 GiB 54.18 1.13 30 up osd.116 117 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 60 KiB 2.4 GiB 376 GiB 77.58 1.62 43 up osd.117 118 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.4 GiB 558 GiB 66.70 1.40 37 up osd.118 119 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 12 KiB 2.3 GiB 466 GiB 72.18 1.51 40 up osd.119 120 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 12 KiB 2.5 GiB 403 GiB 75.94 1.59 42 up osd.120 121 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 84 KiB 2.3 GiB 467 GiB 72.16 1.51 40 up osd.121 122 hdd 1.64000 0.70000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.2 GiB 463 GiB 72.40 1.52 40 up osd.122 123 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 96 KiB 2.4 GiB 469 GiB 72.04 1.51 40 up osd.123 -22 78.72000 - 79 TiB 53 TiB 53 TiB 3.2 MiB 102 GiB 26 TiB 67.11 1.41 - host ld5506-hdd_strgbox 124 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.0 GiB 555 GiB 66.88 1.40 37 up osd.124 125 hdd 1.64000 1.00000 1.6 TiB 972 GiB 970 GiB 12 KiB 1.9 GiB 704 GiB 57.99 1.21 32 up osd.125 126 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.1 GiB 558 GiB 66.71 1.40 37 up osd.126 127 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.2 GiB 521 GiB 68.92 1.44 38 up osd.127 128 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB 408 GiB 75.68 1.58 42 up osd.128 129 hdd 1.64000 1.00000 1.6 TiB 940 GiB 939 GiB 60 KiB 1.7 GiB 736 GiB 56.12 1.17 31 up osd.129 130 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 64 KiB 2.1 GiB 643 GiB 61.66 1.29 34 up osd.130 131 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 160 KiB 2.4 GiB 403 GiB 75.93 1.59 42 up osd.131 132 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 60 KiB 2.0 GiB 615 GiB 63.32 1.33 35 up osd.132 133 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 28 KiB 2.6 GiB 375 GiB 77.62 1.63 43 up osd.133 134 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 4 KiB 2.1 GiB 545 GiB 67.48 1.41 37 up osd.134 135 hdd 1.64000 1.00000 1.6 TiB 997 GiB 995 GiB 64 KiB 2.0 GiB 679 GiB 59.49 1.25 33 up osd.135 136 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 64 KiB 2.4 GiB 343 GiB 79.56 1.67 44 up osd.136 137 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.3 GiB 495 GiB 70.48 1.48 39 up osd.137 138 hdd 1.64000 1.00000 1.6 TiB 967 GiB 965 GiB 64 KiB 1.8 GiB 709 GiB 57.70 1.21 32 up osd.138 139 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.2 GiB 464 GiB 72.29 1.51 40 up osd.139 140 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 64 KiB 2.2 GiB 374 GiB 77.66 1.63 43 up osd.140 141 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.4 GiB 464 GiB 72.34 1.51 40 up osd.141 142 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 4 KiB 2.0 GiB 585 GiB 65.09 1.36 36 up osd.142 143 hdd 1.64000 1.00000 1.6 TiB 1000 GiB 998 GiB 64 KiB 2.0 GiB 676 GiB 59.64 1.25 33 up osd.143 144 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 96 KiB 2.4 GiB 402 GiB 76.00 1.59 42 up osd.144 145 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 36 KiB 2.4 GiB 344 GiB 79.48 1.66 44 up osd.145 146 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 16 KiB 2.2 GiB 524 GiB 68.76 1.44 38 up osd.146 147 hdd 1.64000 1.00000 1.6 TiB 968 GiB 966 GiB 92 KiB 1.7 GiB 708 GiB 57.77 1.21 32 up osd.147 148 hdd 1.64000 1.00000 1.6 TiB 971 GiB 970 GiB 60 KiB 1.8 GiB 705 GiB 57.96 1.21 32 up osd.148 149 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 76 KiB 2.5 GiB 379 GiB 77.39 1.62 43 up osd.149 150 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 8 KiB 2.5 GiB 376 GiB 77.57 1.62 43 up osd.150 151 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.0 GiB 553 GiB 67.03 1.40 37 up osd.151 152 hdd 1.64000 1.00000 1.6 TiB 1001 GiB 999 GiB 120 KiB 1.8 GiB 675 GiB 59.73 1.25 33 up osd.152 153 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.6 GiB 438 GiB 73.87 1.55 41 up osd.153 154 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.2 GiB 553 GiB 67.00 1.40 37 up osd.154 155 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.0 GiB 585 GiB 65.12 1.36 36 up osd.155 156 hdd 1.64000 0.70000 1.6 TiB 1.2 TiB 1.2 TiB 108 KiB 2.4 GiB 466 GiB 72.19 1.51 40 up osd.156 157 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.1 GiB 557 GiB 66.78 1.40 37 up osd.157 158 hdd 1.64000 1.00000 1.6 TiB 910 GiB 908 GiB 120 KiB 1.7 GiB 766 GiB 54.27 1.14 30 up osd.158 159 hdd 1.64000 1.00000 1.6 TiB 907 GiB 905 GiB 60 KiB 1.7 GiB 769 GiB 54.11 1.13 30 up osd.159 160 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 4 KiB 2.2 GiB 528 GiB 68.49 1.43 38 up osd.160 161 hdd 1.64000 1.00000 1.6 TiB 940 GiB 938 GiB 64 KiB 1.8 GiB 736 GiB 56.07 1.17 31 up osd.161 162 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 124 KiB 2.0 GiB 614 GiB 63.37 1.33 35 up osd.162 163 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.4 GiB 403 GiB 75.95 1.59 42 up osd.163 164 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.2 GiB 585 GiB 65.08 1.36 36 up osd.164 165 hdd 1.64000 1.00000 1.6 TiB 972 GiB 971 GiB 60 KiB 1.9 GiB 704 GiB 58.02 1.21 32 up osd.165 166 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.4 GiB 436 GiB 74.00 1.55 41 up osd.166 167 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 64 KiB 2.1 GiB 613 GiB 63.43 1.33 35 up osd.167 168 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.3 GiB 434 GiB 74.12 1.55 41 up osd.168 169 hdd 1.64000 1.00000 1.6 TiB 910 GiB 908 GiB 60 KiB 1.7 GiB 766 GiB 54.27 1.14 30 up osd.169 170 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.2 GiB 555 GiB 66.88 1.40 37 up osd.170 171 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 96 KiB 2.4 GiB 435 GiB 74.03 1.55 41 up osd.171 -25 78.72000 - 79 TiB 53 TiB 53 TiB 4.0 MiB 104 GiB 25 TiB 67.93 1.42 - host ld5507-hdd_strgbox 189 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 132 KiB 2.3 GiB 408 GiB 75.68 1.58 42 up osd.189 190 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 120 KiB 2.4 GiB 376 GiB 77.54 1.62 43 up osd.190 191 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 60 KiB 2.0 GiB 613 GiB 63.44 1.33 35 up osd.191 192 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.2 GiB 558 GiB 66.73 1.40 37 up osd.192 193 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.1 GiB 561 GiB 66.53 1.39 37 up osd.193 194 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.3 GiB 496 GiB 70.43 1.47 39 up osd.194 195 hdd 1.64000 1.00000 1.6 TiB 909 GiB 907 GiB 120 KiB 1.7 GiB 767 GiB 54.22 1.14 30 up osd.195 196 hdd 1.64000 1.00000 1.6 TiB 998 GiB 996 GiB 64 KiB 1.9 GiB 678 GiB 59.53 1.25 33 up osd.196 197 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB 434 GiB 74.08 1.55 41 up osd.197 198 hdd 1.64000 1.00000 1.6 TiB 971 GiB 969 GiB 132 KiB 1.9 GiB 705 GiB 57.92 1.21 32 up osd.198 199 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 124 KiB 2.1 GiB 585 GiB 65.08 1.36 36 up osd.199 200 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.4 GiB 434 GiB 74.08 1.55 41 up osd.200 201 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 76 KiB 2.3 GiB 498 GiB 70.30 1.47 39 up osd.201 202 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.3 GiB 496 GiB 70.40 1.47 39 up osd.202 203 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.2 GiB 495 GiB 70.44 1.47 39 up osd.203 204 hdd 1.64000 1.00000 1.6 TiB 882 GiB 880 GiB 16 KiB 2.0 GiB 794 GiB 52.63 1.10 29 up osd.204 205 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 132 KiB 2.1 GiB 586 GiB 65.05 1.36 36 up osd.205 206 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 64 KiB 1.9 GiB 645 GiB 61.53 1.29 34 up osd.206 207 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.3 GiB 460 GiB 72.52 1.52 40 up osd.207 208 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 144 KiB 2.3 GiB 497 GiB 70.32 1.47 39 up osd.208 209 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 108 KiB 2.2 GiB 584 GiB 65.14 1.36 36 up osd.209 210 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 180 KiB 2.2 GiB 492 GiB 70.67 1.48 39 up osd.210 211 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.2 GiB 557 GiB 66.75 1.40 37 up osd.211 212 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.2 GiB 585 GiB 65.09 1.36 36 up osd.212 213 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 5 KiB 2.7 GiB 432 GiB 74.24 1.55 41 up osd.213 214 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.2 GiB 554 GiB 66.97 1.40 37 up osd.214 215 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 28 KiB 2.4 GiB 405 GiB 75.81 1.59 42 up osd.215 216 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.2 GiB 527 GiB 68.58 1.44 38 up osd.216 217 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.2 GiB 493 GiB 70.61 1.48 39 up osd.217 218 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.4 GiB 461 GiB 72.46 1.52 40 up osd.218 219 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.0 GiB 588 GiB 64.89 1.36 36 up osd.219 251 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.3 GiB 463 GiB 72.35 1.51 40 up osd.251 252 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 28 KiB 2.4 GiB 312 GiB 81.40 1.70 45 up osd.252 253 hdd 1.64000 1.00000 1.6 TiB 999 GiB 997 GiB 60 KiB 1.8 GiB 677 GiB 59.59 1.25 33 up osd.253 254 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 108 KiB 2.2 GiB 562 GiB 66.49 1.39 37 up osd.254 255 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 108 KiB 2.2 GiB 555 GiB 66.88 1.40 37 up osd.255 256 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.2 GiB 402 GiB 76.03 1.59 42 up osd.256 257 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 64 KiB 2.0 GiB 613 GiB 63.44 1.33 35 up osd.257 258 hdd 1.64000 1.00000 1.6 TiB 1002 GiB 1000 GiB 60 KiB 1.9 GiB 674 GiB 59.76 1.25 33 up osd.258 259 hdd 1.64000 1.00000 1.6 TiB 881 GiB 879 GiB 124 KiB 1.7 GiB 795 GiB 52.55 1.10 29 up osd.259 260 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.2 GiB 464 GiB 72.34 1.51 40 up osd.260 261 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.3 GiB 463 GiB 72.36 1.52 40 up osd.261 262 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 136 KiB 2.4 GiB 372 GiB 77.78 1.63 43 up osd.262 263 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.2 GiB 466 GiB 72.17 1.51 40 up osd.263 264 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB 407 GiB 75.74 1.59 42 up osd.264 265 hdd 1.64000 1.00000 1.6 TiB 878 GiB 876 GiB 120 KiB 1.7 GiB 798 GiB 52.37 1.10 29 up osd.265 266 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 132 KiB 2.4 GiB 431 GiB 74.29 1.56 41 up osd.266 267 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.1 GiB 582 GiB 65.27 1.37 36 up osd.267 -28 78.71997 - 79 TiB 51 TiB 51 TiB 3.8 MiB 101 GiB 27 TiB 65.26 1.37 - host ld5508-hdd_strgbox 42 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 60 KiB 2.1 GiB 646 GiB 61.43 1.29 34 up osd.42 43 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.1 GiB 553 GiB 66.99 1.40 37 up osd.43 44 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB 437 GiB 73.92 1.55 41 up osd.44 45 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0 GiB 614 GiB 63.35 1.33 35 up osd.45 46 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.4 GiB 406 GiB 75.78 1.59 42 up osd.46 47 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 32 KiB 2.1 GiB 586 GiB 65.02 1.36 36 up osd.47 48 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 16 KiB 2.2 GiB 522 GiB 68.83 1.44 38 up osd.48 49 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.2 GiB 530 GiB 68.35 1.43 38 up osd.49 50 hdd 1.64000 1.00000 1.6 TiB 1001 GiB 999 GiB 72 KiB 1.9 GiB 675 GiB 59.71 1.25 33 up osd.50 51 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0 GiB 645 GiB 61.50 1.29 34 up osd.51 52 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.1 GiB 553 GiB 67.01 1.40 37 up osd.52 53 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB 467 GiB 72.15 1.51 40 up osd.53 54 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.1 TiB 84 KiB 2.2 GiB 498 GiB 70.29 1.47 39 up osd.54 55 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 72 KiB 2.3 GiB 496 GiB 70.39 1.47 39 up osd.55 56 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 120 KiB 2.0 GiB 645 GiB 61.50 1.29 34 up osd.56 57 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.2 GiB 525 GiB 68.65 1.44 38 up osd.57 58 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.5 GiB 402 GiB 75.99 1.59 42 up osd.58 220 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.4 GiB 461 GiB 72.47 1.52 40 up osd.220 221 hdd 1.64000 1.00000 1.6 TiB 972 GiB 970 GiB 60 KiB 1.8 GiB 704 GiB 57.97 1.21 32 up osd.221 222 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.3 GiB 492 GiB 70.63 1.48 39 up osd.222 223 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.3 GiB 468 GiB 72.08 1.51 40 up osd.223 224 hdd 1.64000 1.00000 1.6 TiB 940 GiB 938 GiB 60 KiB 1.9 GiB 736 GiB 56.06 1.17 31 up osd.224 225 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB 405 GiB 75.82 1.59 42 up osd.225 226 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.2 GiB 555 GiB 66.88 1.40 37 up osd.226 227 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.3 GiB 466 GiB 72.22 1.51 40 up osd.227 228 hdd 1.64000 1.00000 1.6 TiB 938 GiB 936 GiB 60 KiB 1.8 GiB 738 GiB 55.96 1.17 31 up osd.228 229 hdd 1.64000 1.00000 1.6 TiB 910 GiB 909 GiB 72 KiB 1.8 GiB 766 GiB 54.32 1.14 30 up osd.229 230 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB 404 GiB 75.88 1.59 42 up osd.230 231 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0 GiB 644 GiB 61.58 1.29 34 up osd.231 232 hdd 1.64000 1.00000 1.6 TiB 938 GiB 936 GiB 64 KiB 1.8 GiB 738 GiB 55.96 1.17 31 up osd.232 233 hdd 1.64000 1.00000 1.6 TiB 1001 GiB 1000 GiB 60 KiB 1.8 GiB 675 GiB 59.75 1.25 33 up osd.233 234 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 8 KiB 2.2 GiB 584 GiB 65.16 1.36 36 up osd.234 235 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0 GiB 617 GiB 63.17 1.32 35 up osd.235 236 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 72 KiB 2.2 GiB 497 GiB 70.34 1.47 39 up osd.236 237 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.1 GiB 555 GiB 66.88 1.40 37 up osd.237 238 hdd 1.64000 1.00000 1.6 TiB 880 GiB 878 GiB 76 KiB 1.7 GiB 796 GiB 52.48 1.10 29 up osd.238 239 hdd 1.64000 0.70000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.4 GiB 490 GiB 70.77 1.48 39 up osd.239 240 hdd 1.64000 1.00000 1.6 TiB 970 GiB 968 GiB 180 KiB 1.8 GiB 706 GiB 57.89 1.21 32 up osd.240 241 hdd 1.64000 1.00000 1.6 TiB 878 GiB 876 GiB 132 KiB 1.7 GiB 798 GiB 52.38 1.10 29 up osd.241 242 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 4 KiB 1.9 GiB 648 GiB 61.32 1.28 34 up osd.242 243 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 76 KiB 2.0 GiB 616 GiB 63.24 1.32 35 up osd.243 244 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 132 KiB 2.1 GiB 581 GiB 65.34 1.37 36 up osd.244 245 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 124 KiB 2.1 GiB 583 GiB 65.21 1.37 36 up osd.245 246 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.1 GiB 581 GiB 65.31 1.37 36 up osd.246 247 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 136 KiB 2.2 GiB 530 GiB 68.39 1.43 38 up osd.247 248 hdd 1.64000 1.00000 1.6 TiB 880 GiB 878 GiB 132 KiB 1.8 GiB 796 GiB 52.48 1.10 29 up osd.248 249 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 104 KiB 2.3 GiB 497 GiB 70.35 1.47 39 up osd.249 250 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 180 KiB 2.1 GiB 617 GiB 63.18 1.32 35 up osd.250 -1 169.68462 - 169 TiB 6.0 TiB 5.9 TiB 8.6 MiB 106 GiB 163 TiB 3.56 0.07 - root default -46 4.38295 - 4.4 TiB 212 GiB 206 GiB 452 KiB 6.0 GiB 4.1 TiB 4.76 0.10 - host ld4257 435 hdd 1.09000 1.00000 1.1 TiB 61 GiB 60 GiB 176 KiB 1024 MiB 1.0 TiB 5.46 0.11 2 up osd.435 436 hdd 1.09000 1.00000 1.1 TiB 51 GiB 50 GiB 20 KiB 1024 MiB 1.0 TiB 4.56 0.10 3 up osd.436 437 hdd 1.09000 1.00000 1.1 TiB 34 GiB 33 GiB 64 KiB 1024 MiB 1.1 TiB 3.03 0.06 1 up osd.437 20 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 64 KiB 1024 MiB 349 GiB 6.05 0.13 1 up osd.20 21 ssd 0.37099 1.00000 371 GiB 6.4 GiB 5.4 GiB 0 B 1 GiB 365 GiB 1.73 0.04 0 up osd.21 22 ssd 0.37099 1.00000 371 GiB 38 GiB 37 GiB 128 KiB 1024 MiB 333 GiB 10.26 0.21 2 up osd.22 -34 6.78294 - 6.8 TiB 407 GiB 401 GiB 680 KiB 6.0 GiB 6.4 TiB 5.89 0.12 - host ld4464 438 hdd 1.09000 1.00000 1.1 TiB 132 GiB 131 GiB 144 KiB 1024 MiB 984 GiB 11.82 0.25 4 up osd.438 439 hdd 1.09000 1.00000 1.1 TiB 78 GiB 77 GiB 128 KiB 1024 MiB 1.0 TiB 6.96 0.15 2 up osd.439 23 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 64 KiB 1024 MiB 349 GiB 5.93 0.12 1 up osd.23 24 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 96 KiB 1024 MiB 349 GiB 5.97 0.13 1 up osd.24 25 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 128 KiB 1024 MiB 349 GiB 6.01 0.13 1 up osd.25 26 ssd 3.48999 1.00000 3.5 TiB 131 GiB 130 GiB 120 KiB 1024 MiB 3.4 TiB 3.67 0.08 8 up osd.26 -37 6.78294 - 6.8 TiB 343 GiB 337 GiB 284 KiB 6.0 GiB 6.4 TiB 4.96 0.10 - host ld4465 440 hdd 1.09000 1.00000 1.1 TiB 51 GiB 50 GiB 0 B 1 GiB 1.0 TiB 4.55 0.10 1 up osd.440 441 hdd 1.09000 1.00000 1.1 TiB 77 GiB 76 GiB 24 KiB 1024 MiB 1.0 TiB 6.89 0.14 3 up osd.441 27 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 4 KiB 1024 MiB 349 GiB 6.02 0.13 1 up osd.27 28 ssd 0.37099 1.00000 371 GiB 38 GiB 37 GiB 172 KiB 1024 MiB 333 GiB 10.26 0.21 2 up osd.28 29 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 52 KiB 1024 MiB 349 GiB 5.91 0.12 1 up osd.29 30 ssd 3.48999 1.00000 3.5 TiB 133 GiB 132 GiB 32 KiB 1024 MiB 3.4 TiB 3.72 0.08 8 up osd.30 -3 37.93394 - 38 TiB 1.5 TiB 1.5 TiB 2.1 MiB 22 GiB 36 TiB 4.05 0.08 - host ld5505 31 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 72 KiB 1024 MiB 1.6 TiB 2.00 0.04 2 up osd.31 32 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 128 KiB 1024 MiB 1.6 TiB 4.66 0.10 3 up osd.32 33 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 32 KiB 1024 MiB 1.6 TiB 1.99 0.04 1 up osd.33 34 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 96 KiB 1024 MiB 1.6 TiB 4.65 0.10 2 up osd.34 35 hdd 1.64000 1.00000 1.6 TiB 131 GiB 130 GiB 192 KiB 1024 MiB 1.5 TiB 7.81 0.16 4 up osd.35 36 hdd 1.64000 1.00000 1.6 TiB 131 GiB 130 GiB 208 KiB 1024 MiB 1.5 TiB 7.84 0.16 5 up osd.36 37 hdd 1.64000 1.00000 1.6 TiB 77 GiB 76 GiB 200 KiB 1024 MiB 1.6 TiB 4.60 0.10 4 up osd.37 38 hdd 1.64000 1.00000 1.6 TiB 87 GiB 86 GiB 128 KiB 1024 MiB 1.6 TiB 5.19 0.11 3 up osd.38 39 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 80 KiB 1024 MiB 1.6 TiB 5.23 0.11 3 up osd.39 40 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 228 KiB 1024 MiB 1.6 TiB 3.60 0.08 2 up osd.40 41 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 120 KiB 1024 MiB 1.6 TiB 5.27 0.11 3 up osd.41 59 hdd 1.64000 1.00000 1.6 TiB 87 GiB 86 GiB 164 KiB 1024 MiB 1.6 TiB 5.22 0.11 4 up osd.59 60 hdd 1.64000 1.00000 1.6 TiB 132 GiB 131 GiB 216 KiB 1024 MiB 1.5 TiB 7.86 0.16 4 up osd.60 61 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 48 KiB 1024 MiB 1.6 TiB 1.98 0.04 3 up osd.61 62 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 80 KiB 1024 MiB 1.6 TiB 5.23 0.11 3 up osd.62 63 hdd 1.64000 1.00000 1.6 TiB 87 GiB 86 GiB 48 KiB 1024 MiB 1.6 TiB 5.19 0.11 3 up osd.63 64 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 48 KiB 1024 MiB 1.6 TiB 4.66 0.10 2 up osd.64 0 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 8 KiB 1024 MiB 2.9 TiB 0.17 0.00 1 up osd.0 1 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1 GiB 2.9 TiB 0.17 0.00 1 up osd.1 8 ssd 3.48999 1.00000 3.5 TiB 129 GiB 128 GiB 4 KiB 1024 MiB 3.4 TiB 3.60 0.08 8 up osd.8 9 ssd 0.37199 1.00000 372 GiB 36 GiB 35 GiB 16 KiB 1024 MiB 336 GiB 9.71 0.20 2 up osd.9 10 ssd 0.37199 1.00000 372 GiB 5.0 GiB 4.0 GiB 0 B 1 GiB 367 GiB 1.35 0.03 0 up osd.10 -7 37.93394 - 38 TiB 1.1 TiB 1.1 TiB 1.4 MiB 22 GiB 37 TiB 2.97 0.06 - host ld5506 65 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 16 KiB 1024 MiB 1.6 TiB 1.97 0.04 1 up osd.65 66 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 64 KiB 1024 MiB 1.6 TiB 4.65 0.10 2 up osd.66 67 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 168 KiB 1024 MiB 1.6 TiB 1.99 0.04 2 up osd.67 68 hdd 1.64000 1.00000 1.6 TiB 89 GiB 88 GiB 80 KiB 1024 MiB 1.6 TiB 5.29 0.11 3 up osd.68 69 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 124 KiB 1024 MiB 1.6 TiB 3.60 0.08 2 up osd.69 70 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 96 KiB 1024 MiB 1.6 TiB 2.00 0.04 1 up osd.70 71 hdd 1.64000 1.00000 1.6 TiB 79 GiB 78 GiB 48 KiB 1024 MiB 1.6 TiB 4.68 0.10 4 up osd.71 72 hdd 1.64000 1.00000 1.6 TiB 87 GiB 86 GiB 136 KiB 1024 MiB 1.6 TiB 5.19 0.11 4 up osd.72 73 hdd 1.64000 1.00000 1.6 TiB 105 GiB 104 GiB 16 KiB 1024 MiB 1.5 TiB 6.27 0.13 3 up osd.73 74 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 116 KiB 1024 MiB 1.6 TiB 3.57 0.07 3 up osd.74 75 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 16 KiB 1024 MiB 1.6 TiB 3.58 0.08 2 up osd.75 172 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 32 KiB 1024 MiB 1.6 TiB 3.58 0.08 2 up osd.172 173 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 12 KiB 1024 MiB 1.6 TiB 3.59 0.08 3 up osd.173 174 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 172 KiB 1024 MiB 1.6 TiB 4.67 0.10 2 up osd.174 175 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 16 KiB 1024 MiB 1.6 TiB 2.00 0.04 1 up osd.175 176 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 88 KiB 1024 MiB 1.6 TiB 1.98 0.04 1 up osd.176 177 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 216 KiB 1024 MiB 1.6 TiB 2.03 0.04 1 up osd.177 2 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 8 KiB 1024 MiB 2.9 TiB 0.17 0.00 2 up osd.2 3 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1 GiB 2.9 TiB 0.17 0.00 3 up osd.3 11 ssd 0.37199 1.00000 372 GiB 5.0 GiB 4.0 GiB 0 B 1 GiB 367 GiB 1.35 0.03 0 up osd.11 12 ssd 3.48999 1.00000 3.5 TiB 99 GiB 98 GiB 4 KiB 1024 MiB 3.4 TiB 2.76 0.06 6 up osd.12 13 ssd 0.37199 1.00000 372 GiB 21 GiB 20 GiB 4 KiB 1024 MiB 351 GiB 5.53 0.12 1 up osd.13 -10 37.93394 - 38 TiB 1.1 TiB 1.1 TiB 1.8 MiB 22 GiB 37 TiB 3.00 0.06 - host ld5507 178 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 148 KiB 1024 MiB 1.6 TiB 1.99 0.04 1 up osd.178 179 hdd 1.64000 1.00000 1.6 TiB 61 GiB 60 GiB 84 KiB 1024 MiB 1.6 TiB 3.61 0.08 3 up osd.179 180 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 136 KiB 1024 MiB 1.6 TiB 5.26 0.11 3 up osd.180 181 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 144 KiB 1024 MiB 1.6 TiB 2.00 0.04 1 up osd.181 182 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 212 KiB 1024 MiB 1.6 TiB 1.99 0.04 1 up osd.182 183 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 24 KiB 1024 MiB 1.6 TiB 1.98 0.04 1 up osd.183 184 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 96 KiB 1024 MiB 1.6 TiB 1.97 0.04 1 up osd.184 185 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 112 KiB 1024 MiB 1.6 TiB 1.99 0.04 1 up osd.185 186 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 64 KiB 1024 MiB 1.6 TiB 3.61 0.08 2 up osd.186 187 hdd 1.64000 1.00000 1.6 TiB 105 GiB 104 GiB 52 KiB 1024 MiB 1.5 TiB 6.25 0.13 3 up osd.187 188 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 32 KiB 1024 MiB 1.6 TiB 2.00 0.04 1 up osd.188 268 hdd 1.64000 1.00000 1.6 TiB 131 GiB 130 GiB 64 KiB 1024 MiB 1.5 TiB 7.80 0.16 5 up osd.268 269 hdd 1.64000 1.00000 1.6 TiB 61 GiB 60 GiB 112 KiB 1024 MiB 1.6 TiB 3.63 0.08 3 up osd.269 270 hdd 1.64000 1.00000 1.6 TiB 61 GiB 60 GiB 56 KiB 1024 MiB 1.6 TiB 3.63 0.08 2 up osd.270 271 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 208 KiB 1024 MiB 1.6 TiB 3.59 0.08 2 up osd.271 368 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 140 KiB 1024 MiB 1.6 TiB 1.99 0.04 1 up osd.368 369 hdd 1.64000 1.00000 1.6 TiB 105 GiB 104 GiB 116 KiB 1024 MiB 1.5 TiB 6.24 0.13 4 up osd.369 4 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1 GiB 2.9 TiB 0.17 0.00 1 up osd.4 5 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1 GiB 2.9 TiB 0.17 0.00 2 up osd.5 14 ssd 0.37199 1.00000 372 GiB 5.0 GiB 4.0 GiB 0 B 1 GiB 367 GiB 1.35 0.03 0 up osd.14 15 ssd 0.37199 1.00000 372 GiB 21 GiB 20 GiB 16 KiB 1024 MiB 351 GiB 5.64 0.12 1 up osd.15 16 ssd 3.48999 1.00000 3.5 TiB 129 GiB 128 GiB 40 KiB 1024 MiB 3.4 TiB 3.61 0.08 8 up osd.16 -13 37.93394 - 38 TiB 1.3 TiB 1.3 TiB 2.0 MiB 22 GiB 37 TiB 3.42 0.07 - host ld5508 370 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 144 KiB 1024 MiB 1.6 TiB 2.00 0.04 1 up osd.370 419 hdd 1.64000 1.00000 1.6 TiB 77 GiB 76 GiB 48 KiB 1024 MiB 1.6 TiB 4.62 0.10 2 up osd.419 420 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 80 KiB 1024 MiB 1.6 TiB 3.56 0.07 2 up osd.420 421 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 28 KiB 1024 MiB 1.6 TiB 1.96 0.04 1 up osd.421 422 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 72 KiB 1024 MiB 1.6 TiB 5.26 0.11 4 up osd.422 423 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 64 KiB 1024 MiB 1.6 TiB 5.22 0.11 4 up osd.423 424 hdd 1.64000 1.00000 1.6 TiB 176 GiB 175 GiB 224 KiB 1024 MiB 1.5 TiB 10.47 0.22 5 up osd.424 425 hdd 1.64000 1.00000 1.6 TiB 106 GiB 105 GiB 196 KiB 1024 MiB 1.5 TiB 6.33 0.13 3 up osd.425 426 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 112 KiB 1024 MiB 1.6 TiB 3.56 0.07 2 up osd.426 427 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 48 KiB 1024 MiB 1.6 TiB 1.97 0.04 1 up osd.427 428 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 32 KiB 1024 MiB 1.6 TiB 2.01 0.04 1 up osd.428 429 hdd 1.64000 1.00000 1.6 TiB 106 GiB 105 GiB 132 KiB 1024 MiB 1.5 TiB 6.30 0.13 3 up osd.429 430 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 120 KiB 1024 MiB 1.6 TiB 2.02 0.04 1 up osd.430 431 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 112 KiB 1024 MiB 1.6 TiB 3.58 0.08 2 up osd.431 432 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 208 KiB 1024 MiB 1.6 TiB 1.99 0.04 2 up osd.432 433 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 164 KiB 1024 MiB 1.6 TiB 3.56 0.07 2 up osd.433 434 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 156 KiB 1024 MiB 1.6 TiB 2.00 0.04 1 up osd.434 6 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1 GiB 2.9 TiB 0.17 0.00 3 up osd.6 7 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 8 KiB 1024 MiB 2.9 TiB 0.17 0.00 3 up osd.7 17 ssd 0.37199 1.00000 372 GiB 5.0 GiB 4.0 GiB 8 KiB 1024 MiB 367 GiB 1.35 0.03 0 up osd.17 18 ssd 0.37199 1.00000 372 GiB 36 GiB 35 GiB 12 KiB 1024 MiB 336 GiB 9.80 0.21 2 up osd.18 19 ssd 3.48999 1.00000 3.5 TiB 162 GiB 161 GiB 40 KiB 1024 MiB 3.3 TiB 4.54 0.10 10 up osd.19 TOTAL 1.5 PiB 731 TiB 730 TiB 39 MiB 1.5 TiB 800 TiB 47.76 MIN/MAX VAR: 0.00/1.70 STDDEV: 25.46 Am 01.11.2019 um 11:24 schrieb Paul Emmerich:
Looks like you didn't tell the whole story, please post the *full* output of ceph -s and ceph osd df tree.
Wild guess: you need to increase "mon max pg per osd"
Paul
participants (3)
-
Paul Emmerich
-
Thomas
-
Thomas Schneider