31 Oct
2019
31 Oct
'19
1:26 p.m.
Hi,
after enabling ceph balancer (with command ceph balancer on) the health
status changed to error.
This is the current output of ceph health detail:
root@ld3955:~# ceph health detail
HEALTH_ERR 1438 slow requests are blocked > 32 sec; 861 stuck requests
are blocked > 4096 sec; mon ld5505 is low on available space
REQUEST_SLOW 1438 slow requests are blocked > 32 sec
683 ops are blocked > 2097.15 sec
436 ops are blocked > 1048.58 sec
191 ops are blocked > 524.288 sec
78 ops are blocked > 262.144 sec
35 ops are blocked > 131.072 sec
11 ops are blocked > 65.536 sec
4 ops are blocked > 32.768 sec
osd.62 has blocked requests > 65.536 sec
osds 39,72 have blocked requests > 262.144 sec
osds 6,19,67,173,174,187,188,269,434 have blocked requests > 524.288 sec
osds
8,16,35,36,37,61,63,64,68,73,75,178,186,271,369,420,429,431,433,436 have
blocked requests > 1048.58 sec
osds 3,5,7,24,34,38,40,41,59,66,69,74,180,270,370,421,432,435 have
blocked requests > 2097.15 sec
REQUEST_STUCK 861 stuck requests are blocked > 4096 sec
25 ops are blocked > 8388.61 sec
836 ops are blocked > 4194.3 sec
osds 2,28,29,32,60,65,181,185,268,368,423,424,426 have stuck
requests > 4194.3 sec
osds 0,30,70,71,184 have stuck requests > 8388.61 sec
I understand that when balancer starts shifting PGs to other OSDs that
this caused IO load on the cluster.
However I don't understand why this is affecting OSD so heavily.
And I don't understand why OSD of specific type (SSD, NVME) suffer
although there's no balancing occuring on them.
Regards
Thomas
31 Oct
31 Oct
5:27 p.m.
Requests stuck for > 2 hours cannot be attributed to "IO load on the
cluster".
Looks like some OSDs really are stuck, things to try:
* run "ceph daemon osd.X dump_blocked_ops" on one of the affected OSDs
to see what is stuck
* try restarting OSDs to see if it clears up automatically
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Thu, Oct 31, 2019 at 2:27 PM Thomas Schneider <74cmonty(a)gmail.com> wrote:
>
> Hi,
>
> after enabling ceph balancer (with command ceph balancer on) the health
> status changed to error.
> This is the current output of ceph health detail:
> root@ld3955:~# ceph health detail
> HEALTH_ERR 1438 slow requests are blocked > 32 sec; 861 stuck requests
> are blocked > 4096 sec; mon ld5505 is low on available space
> REQUEST_SLOW 1438 slow requests are blocked > 32 sec
> 683 ops are blocked > 2097.15 sec
> 436 ops are blocked > 1048.58 sec
> 191 ops are blocked > 524.288 sec
> 78 ops are blocked > 262.144 sec
> 35 ops are blocked > 131.072 sec
> 11 ops are blocked > 65.536 sec
> 4 ops are blocked > 32.768 sec
> osd.62 has blocked requests > 65.536 sec
> osds 39,72 have blocked requests > 262.144 sec
> osds 6,19,67,173,174,187,188,269,434 have blocked requests > 524.288 sec
> osds
> 8,16,35,36,37,61,63,64,68,73,75,178,186,271,369,420,429,431,433,436 have
> blocked requests > 1048.58 sec
> osds 3,5,7,24,34,38,40,41,59,66,69,74,180,270,370,421,432,435 have
> blocked requests > 2097.15 sec
> REQUEST_STUCK 861 stuck requests are blocked > 4096 sec
> 25 ops are blocked > 8388.61 sec
> 836 ops are blocked > 4194.3 sec
> osds 2,28,29,32,60,65,181,185,268,368,423,424,426 have stuck
> requests > 4194.3 sec
> osds 0,30,70,71,184 have stuck requests > 8388.61 sec
>
> I understand that when balancer starts shifting PGs to other OSDs that
> this caused IO load on the cluster.
> However I don't understand why this is affecting OSD so heavily.
> And I don't understand why OSD of specific type (SSD, NVME) suffer
> although there's no balancing occuring on them.
>
> Regards
> Thomas
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:17 p.m.
This is the output of OSD.270 that remains with slow requests blocked
even after restarting.
What's the interpretation of it?
root@ld5507:~# ceph daemon osd.270 dump_blocked_ops [330/1857]
{
"ops": [
{
"description": "osd_pg_create(e293649 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:22:13.563017",
"age": 2785.2698560419999,
"duration": 2785.2699056299998,
"type_data": {
"flag_point": "started",
"events": [
{
"time": "2019-10-31 19:22:13.563017",
"event": "initiated"
},
{
"time": "2019-10-31 19:22:13.563017",
"event": "header_read"
},
{
"time": "2019-10-31 19:22:13.563011",
"event": "throttled"
},
{
"time": "2019-10-31 19:22:13.563024",
"event": "all_read"
},
{
"time": "2019-10-31 20:07:43.881441",
"event": "dispatched"
}, [300/1857]
{
"time": "2019-10-31 20:07:43.881472",
"event": "wait for new map"
},
{
"time": "2019-10-31 20:07:44.665714",
"event": "started"
}
]
}
},
{
"description": "osd_pg_create(e293650 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:23:16.150040",
"age": 2722.6828331659999,
"duration": 2722.683007228,
"type_data": {
"flag_point": "delayed",
"events": [
{
"time": "2019-10-31 19:23:16.150040",
"event": "initiated"
},
{
"time": "2019-10-31 19:23:16.150040",
"event": "header_read"
},
{
"time": "2019-10-31 19:23:16.150035",
"event": "throttled"
}, [269/1857]
{
"time": "2019-10-31 19:23:16.150055",
"event": "all_read"
},
{
"time": "2019-10-31 20:07:43.882197",
"event": "dispatched"
},
{
"time": "2019-10-31 20:07:43.882198",
"event": "wait for new map"
}
]
}
},
{
"description": "osd_pg_create(e293651 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:23:17.779034",
"age": 2721.0538393319998,
"duration": 2721.0541152350002,
"type_data": {
"flag_point": "delayed",
"events": [
{
"time": "2019-10-31 19:23:17.779034",
"event": "initiated"
},
{
"time": "2019-10-31 19:23:17.779034",
"event": "header_read"
}, [238/1857]
{
"time": "2019-10-31 19:23:17.779027",
"event": "throttled"
},
{
"time": "2019-10-31 19:23:17.779044",
"event": "all_read"
},
{
"time": "2019-10-31 20:07:43.882326",
"event": "dispatched"
},
{
"time": "2019-10-31 20:07:43.882328",
"event": "wait for new map"
}
]
}
},
{
"description": "osd_pg_create(e293652 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:23:22.178466",
"age": 2716.6544071140002,
"duration": 2716.654784073,
"type_data": {
"flag_point": "delayed",
"events": [
{
"time": "2019-10-31 19:23:22.178466",
"event": "initiated"
}, [207/1857]
{
"time": "2019-10-31 19:23:22.178466",
"event": "header_read"
},
{
"time": "2019-10-31 19:23:22.178463",
"event": "throttled"
},
{
"time": "2019-10-31 19:23:22.178471",
"event": "all_read"
},
{
"time": "2019-10-31 20:07:43.882355",
"event": "dispatched"
},
{
"time": "2019-10-31 20:07:43.882356",
"event": "wait for new map"
}
]
}
},
{
"description": "osd_pg_create(e293653 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:23:28.227563",
"age": 2710.6053095460002,
"duration": 2710.6057835810002,
"type_data": {
"flag_point": "delayed",
"events": [ [176/1857]
{
"time": "2019-10-31 19:23:28.227563",
"event": "initiated"
},
{
"time": "2019-10-31 19:23:28.227563",
"event": "header_read"
},
{
"time": "2019-10-31 19:23:28.227559",
"event": "throttled"
},
{
"time": "2019-10-31 19:23:28.227570",
"event": "all_read"
},
{
"time": "2019-10-31 20:07:43.882383",
"event": "dispatched"
}, "age": 2574.9672041270001, [145/1857]
"duration": 2574.9677752090001,
"type_data": {
"flag_point": "delayed",
"events": [
{
"time": "2019-10-31 19:25:43.865669",
"event": "initiated"
},
{
"time": "2019-10-31 19:25:43.865669",
"event": "header_read"
},
{
"time": "2019-10-31 19:25:43.865663",
"event": "throttled"
},
{
"time": "2019-10-31 19:25:43.865682",
"event": "all_read"
},
{
"time": "2019-10-31 20:07:43.883843",
"event": "dispatched"
},
{
"time": "2019-10-31 20:07:43.883846",
"event": "wait for new map"
} [117/1857]
]
}
},
{
"description": "osd_pg_create(e293655 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:26:16.129187",
"age": 2542.703685858,
"duration": 2542.7043550879998,
"type_data": {
"flag_point": "delayed",
"events": [
{
"time": "2019-10-31 19:26:16.129187",
"event": "initiated"
},
{
"time": "2019-10-31 19:26:16.129187",
"event": "header_read"
},
{
"time": "2019-10-31 19:26:16.129181",
"event": "throttled"
},
{
"time": "2019-10-31 19:26:16.129197",
"event": "all_read"
},
{
"time": "2019-10-31 20:07:43.883928",
"event": "dispatched"
}, [86/1857]
{
"time": "2019-10-31 20:07:43.883929",
"event": "wait for new map"
}
]
}
},
{
"description": "osd_pg_create(e293656 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:30:47.890777",
"age": 2270.9420963450002,
"duration": 2270.9428696479999,
"type_data": {
"flag_point": "delayed",
"events": [
{
"time": "2019-10-31 19:30:47.890777",
"event": "initiated"
},
{
"time": "2019-10-31 19:30:47.890777",
"event": "header_read"
},
{
"time": "2019-10-31 19:30:47.890772",
"event": "throttled"
},
{
"time": "2019-10-31 19:30:47.890788",
"event": "all_read"
}, [55/1857]
{
"time": "2019-10-31 20:07:43.913783",
"event": "dispatched"
},
{
"time": "2019-10-31 20:07:43.913786",
"event": "wait for new map"
}
]
}
},
{
"description": "osd_pg_create(e293657 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:35:32.323775",
"age": 1986.5090976690001,
"duration": 1986.509968458,
"type_data": {
"flag_point": "delayed",
"events": [
{
"time": "2019-10-31 19:35:32.323775",
"event": "initiated"
},
{
"time": "2019-10-31 19:35:32.323775",
"event": "header_read"
},
{
"time": "2019-10-31 19:35:32.323769",
"event": "throttled"
}, [24/1857]
{
"time": "2019-10-31 19:35:32.323802",
"event": "all_read"
},
{
"time": "2019-10-31 20:07:43.917500",
"event": "dispatched"
},
{
"time": "2019-10-31 20:07:43.917503",
"event": "wait for new map"
}
]
}
},
{
"description": "osd_pg_create(e293658 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:36:32.783376",
"age": 1926.0494967300001,
"duration": 1926.050464765,
"type_data": {
"flag_point": "delayed",
"events": [
{
"time": "2019-10-31 19:36:32.783376",
"event": "initiated"
},
{
"time": "2019-10-31 19:36:32.783376",
"event": "header_read"
},
{
"time": "2019-10-31 19:36:32.783373",
"event": "throttled"
},
{
"time": "2019-10-31 19:36:32.783388",
"event": "all_read"
},
{
"time": "2019-10-31 20:07:43.919280",
"event": "dispatched"
},
{
"time": "2019-10-31 20:07:43.919282",
"event": "wait for new map"
}
]
}
}
],
"complaint_time": 30,
"num_blocked_ops": 10
}
{
"time": "2019-10-31 20:07:43.882384",
"event": "wait for new map"
}
]
}
},
{
"description": "osd_pg_create(e293654 59.b:267033
59.2c:267033)",
"initiated_at": "2019-10-31 19:25:43.865669",
Am 31.10.2019 um 18:27 schrieb Paul Emmerich:
ceph daemon osd.X dump_blocked_ops
1 Nov
1 Nov
10:24 a.m.
Looks like you didn't tell the whole story, please post the *full*
output of ceph -s and ceph osd df tree.
Wild guess: you need to increase "mon max pg per osd"
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Thu, Oct 31, 2019 at 8:17 PM Thomas <74cmonty(a)gmail.com> wrote:
>
> This is the output of OSD.270 that remains with slow requests blocked
> even after restarting.
> What's the interpretation of it?
>
> root@ld5507:~# ceph daemon osd.270 dump_blocked_ops [330/1857]
> {
> "ops": [
> {
> "description": "osd_pg_create(e293649 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:22:13.563017",
> "age": 2785.2698560419999,
> "duration": 2785.2699056299998,
> "type_data": {
> "flag_point": "started",
> "events": [
> {
> "time": "2019-10-31 19:22:13.563017",
> "event": "initiated"
> },
> {
> "time": "2019-10-31 19:22:13.563017",
> "event": "header_read"
> },
> {
> "time": "2019-10-31 19:22:13.563011",
> "event": "throttled"
> },
> {
> "time": "2019-10-31 19:22:13.563024",
> "event": "all_read"
> },
> {
> "time": "2019-10-31 20:07:43.881441",
> "event": "dispatched"
> }, [300/1857]
> {
> "time": "2019-10-31 20:07:43.881472",
> "event": "wait for new map"
> },
> {
> "time": "2019-10-31 20:07:44.665714",
> "event": "started"
> }
> ]
> }
> },
> {
> "description": "osd_pg_create(e293650 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:23:16.150040",
> "age": 2722.6828331659999,
> "duration": 2722.683007228,
> "type_data": {
> "flag_point": "delayed",
> "events": [
> {
> "time": "2019-10-31 19:23:16.150040",
> "event": "initiated"
> },
> {
> "time": "2019-10-31 19:23:16.150040",
> "event": "header_read"
> },
> {
> "time": "2019-10-31 19:23:16.150035",
> "event": "throttled"
> }, [269/1857]
> {
> "time": "2019-10-31 19:23:16.150055",
> "event": "all_read"
> },
> {
> "time": "2019-10-31 20:07:43.882197",
> "event": "dispatched"
> },
> {
> "time": "2019-10-31 20:07:43.882198",
> "event": "wait for new map"
> }
> ]
> }
> },
> {
> "description": "osd_pg_create(e293651 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:23:17.779034",
> "age": 2721.0538393319998,
> "duration": 2721.0541152350002,
> "type_data": {
> "flag_point": "delayed",
> "events": [
> {
> "time": "2019-10-31 19:23:17.779034",
> "event": "initiated"
> },
> {
> "time": "2019-10-31 19:23:17.779034",
> "event": "header_read"
> }, [238/1857]
> {
> "time": "2019-10-31 19:23:17.779027",
> "event": "throttled"
> },
> {
> "time": "2019-10-31 19:23:17.779044",
> "event": "all_read"
> },
> {
> "time": "2019-10-31 20:07:43.882326",
> "event": "dispatched"
> },
> {
> "time": "2019-10-31 20:07:43.882328",
> "event": "wait for new map"
> }
> ]
> }
> },
> {
> "description": "osd_pg_create(e293652 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:23:22.178466",
> "age": 2716.6544071140002,
> "duration": 2716.654784073,
> "type_data": {
> "flag_point": "delayed",
> "events": [
> {
> "time": "2019-10-31 19:23:22.178466",
> "event": "initiated"
> }, [207/1857]
> {
> "time": "2019-10-31 19:23:22.178466",
> "event": "header_read"
> },
> {
> "time": "2019-10-31 19:23:22.178463",
> "event": "throttled"
> },
> {
> "time": "2019-10-31 19:23:22.178471",
> "event": "all_read"
> },
> {
> "time": "2019-10-31 20:07:43.882355",
> "event": "dispatched"
> },
> {
> "time": "2019-10-31 20:07:43.882356",
> "event": "wait for new map"
> }
> ]
> }
> },
> {
> "description": "osd_pg_create(e293653 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:23:28.227563",
> "age": 2710.6053095460002,
> "duration": 2710.6057835810002,
> "type_data": {
> "flag_point": "delayed",
> "events": [ [176/1857]
> {
> "time": "2019-10-31 19:23:28.227563",
> "event": "initiated"
> },
> {
> "time": "2019-10-31 19:23:28.227563",
> "event": "header_read"
> },
> {
> "time": "2019-10-31 19:23:28.227559",
> "event": "throttled"
> },
> {
> "time": "2019-10-31 19:23:28.227570",
> "event": "all_read"
> },
> {
> "time": "2019-10-31 20:07:43.882383",
> "event": "dispatched"
> }, "age": 2574.9672041270001, [145/1857]
> "duration": 2574.9677752090001,
> "type_data": {
> "flag_point": "delayed",
> "events": [
> {
> "time": "2019-10-31 19:25:43.865669",
> "event": "initiated"
> },
> {
> "time": "2019-10-31 19:25:43.865669",
> "event": "header_read"
> },
> {
> "time": "2019-10-31 19:25:43.865663",
> "event": "throttled"
> },
> {
> "time": "2019-10-31 19:25:43.865682",
> "event": "all_read"
> },
> {
> "time": "2019-10-31 20:07:43.883843",
> "event": "dispatched"
> },
> {
> "time": "2019-10-31 20:07:43.883846",
> "event": "wait for new map"
> } [117/1857]
> ]
> }
> },
> {
> "description": "osd_pg_create(e293655 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:26:16.129187",
> "age": 2542.703685858,
> "duration": 2542.7043550879998,
> "type_data": {
> "flag_point": "delayed",
> "events": [
> {
> "time": "2019-10-31 19:26:16.129187",
> "event": "initiated"
> },
> {
> "time": "2019-10-31 19:26:16.129187",
> "event": "header_read"
> },
> {
> "time": "2019-10-31 19:26:16.129181",
> "event": "throttled"
> },
> {
> "time": "2019-10-31 19:26:16.129197",
> "event": "all_read"
> },
> {
> "time": "2019-10-31 20:07:43.883928",
> "event": "dispatched"
> }, [86/1857]
> {
> "time": "2019-10-31 20:07:43.883929",
> "event": "wait for new map"
> }
> ]
> }
> },
> {
> "description": "osd_pg_create(e293656 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:30:47.890777",
> "age": 2270.9420963450002,
> "duration": 2270.9428696479999,
> "type_data": {
> "flag_point": "delayed",
> "events": [
> {
> "time": "2019-10-31 19:30:47.890777",
> "event": "initiated"
> },
> {
> "time": "2019-10-31 19:30:47.890777",
> "event": "header_read"
> },
> {
> "time": "2019-10-31 19:30:47.890772",
> "event": "throttled"
> },
> {
> "time": "2019-10-31 19:30:47.890788",
> "event": "all_read"
> }, [55/1857]
> {
> "time": "2019-10-31 20:07:43.913783",
> "event": "dispatched"
> },
> {
> "time": "2019-10-31 20:07:43.913786",
> "event": "wait for new map"
> }
> ]
> }
> },
> {
> "description": "osd_pg_create(e293657 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:35:32.323775",
> "age": 1986.5090976690001,
> "duration": 1986.509968458,
> "type_data": {
> "flag_point": "delayed",
> "events": [
> {
> "time": "2019-10-31 19:35:32.323775",
> "event": "initiated"
> },
> {
> "time": "2019-10-31 19:35:32.323775",
> "event": "header_read"
> },
> {
> "time": "2019-10-31 19:35:32.323769",
> "event": "throttled"
> }, [24/1857]
> {
> "time": "2019-10-31 19:35:32.323802",
> "event": "all_read"
> },
> {
> "time": "2019-10-31 20:07:43.917500",
> "event": "dispatched"
> },
> {
> "time": "2019-10-31 20:07:43.917503",
> "event": "wait for new map"
> }
> ]
> }
> },
> {
> "description": "osd_pg_create(e293658 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:36:32.783376",
> "age": 1926.0494967300001,
> "duration": 1926.050464765,
> "type_data": {
> "flag_point": "delayed",
> "events": [
> {
> "time": "2019-10-31 19:36:32.783376",
> "event": "initiated"
> },
> {
> "time": "2019-10-31 19:36:32.783376",
> "event": "header_read"
> },
> {
> "time": "2019-10-31 19:36:32.783373",
> "event": "throttled"
> },
> {
> "time": "2019-10-31 19:36:32.783388",
> "event": "all_read"
> },
> {
> "time": "2019-10-31 20:07:43.919280",
> "event": "dispatched"
> },
> {
> "time": "2019-10-31 20:07:43.919282",
> "event": "wait for new map"
> }
> ]
> }
> }
> ],
> "complaint_time": 30,
> "num_blocked_ops": 10
> }
>
> {
> "time": "2019-10-31 20:07:43.882384",
> "event": "wait for new map"
> }
> ]
> }
> },
> {
> "description": "osd_pg_create(e293654 59.b:267033
> 59.2c:267033)",
> "initiated_at": "2019-10-31 19:25:43.865669",
>
>
> Am 31.10.2019 um 18:27 schrieb Paul Emmerich:
> > ceph daemon osd.X dump_blocked_ops
>
6:19 p.m.
Hi Paul,
the situation has changed in the meantime.
However, I can reproduce a similar behaviour.
This means,
- I disable balancer (ceph balancer off)
- and then start reweighting of a specific OSD (ceph osd reweight 134 1.0)
The cluster immediatelly reports slow requests.
root@ld3955:~# ceph health detail
HEALTH_WARN 434 slow requests are blocked > 32 sec; mon ld5505 is low on
available space
REQUEST_SLOW 434 slow requests are blocked > 32 sec
131 ops are blocked > 131.072 sec
270 ops are blocked > 65.536 sec
33 ops are blocked > 32.768 sec
osd.66 has blocked requests > 32.768 sec
osds 19,65,67,426 have blocked requests > 65.536 sec
osds
0,2,3,5,6,7,8,16,24,28,29,30,32,34,35,36,37,38,39,40,41,59,60,61,62,63,64,68,69,70,71,72,73,74,75,173,174,178,180,181,184,185
,186,187,188,268,269,270,271,368,369,370,420,421,423,424,429,431,432,433,434,435,436
have blocked requests > 131.072 sec
MON_DISK_LOW mon ld5505 is low on available space
mon.ld5505 has 24% avail
root@ld3955:~# ceph -s
cluster:
id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
health: HEALTH_WARN
453 slow requests are blocked > 32 sec
mon ld5505 is low on available space
services:
mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 23h)
mgr: ld5505(active, since 22h), standbys: ld5506, ld5507, ld5508
mds: cephfs:1 {0=ld4465=up:active} 1 up:standby
osd: 442 osds: 442 up, 442 in; 5 remapped pgs
data:
pools: 6 pools, 8312 pgs
objects: 63.92M objects, 244 TiB
usage: 731 TiB used, 800 TiB / 1.5 PiB avail
pgs: 37702/191053577 objects misplaced (0.020%)
8249 active+clean
29 active+clean+scrubbing+deep
29 active+clean+scrubbing
3 active+remapped+backfill_wait
2 active+remapped+backfilling
io:
client: 1.4 KiB/s rd, 87 MiB/s wr, 1 op/s rd, 22 op/s wr
recovery: 34 MiB/s, 8 objects/s
In this example I have reweight an HDD, but many OSDs that have blocked
requests (0,2,3,5,6,7,8) are SSDs.
root@ld3955:~# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-17 1363.19983 - 1.3 PiB 725 TiB 724 TiB 31 MiB 1.4 TiB
637 TiB 53.25 1.12 - root hdd_strgbox
-43 349.43994 - 349 TiB 171 TiB 171 TiB 6.3 MiB 300 GiB
178 TiB 48.91 1.02 - host ld4257-hdd_strgbox
371 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 180 KiB 5.9 GiB
3.8 TiB 47.84 1.00 118 up osd.371
372 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 128 KiB 6.2 GiB
3.7 TiB 48.65 1.02 120 up osd.372
373 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 8 KiB 6.1 GiB
3.7 TiB 48.71 1.02 120 up osd.373
374 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 68 KiB 6.0 GiB
3.8 TiB 47.34 0.99 117 up osd.374
375 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 208 KiB 6.7 GiB
3.4 TiB 53.49 1.12 132 up osd.375
376 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 72 KiB 5.7 GiB
4.2 TiB 42.11 0.88 104 up osd.376
377 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 120 KiB 5.7 GiB
4.2 TiB 42.13 0.88 104 up osd.377
378 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 176 KiB 6.0 GiB
3.8 TiB 47.37 0.99 117 up osd.378
379 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 32 KiB 6.2 GiB
3.7 TiB 49.47 1.04 122 up osd.379
380 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 168 KiB 5.8 GiB
4.0 TiB 45.33 0.95 112 up osd.380
381 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 284 KiB 6.4 GiB
3.6 TiB 50.30 1.05 124 up osd.381
382 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 12 KiB 5.8 GiB
3.9 TiB 46.92 0.98 116 up osd.382
383 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 172 KiB 6.2 GiB
3.7 TiB 49.75 1.04 123 up osd.383
384 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 60 KiB 7.3 GiB
3.5 TiB 51.88 1.09 128 up osd.384
385 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 76 KiB 6.5 GiB
3.8 TiB 47.10 0.99 116 up osd.385
386 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 84 KiB 7.2 GiB
3.4 TiB 53.83 1.13 133 up osd.386
387 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 200 KiB 6.3 GiB
3.7 TiB 49.36 1.03 122 up osd.387
388 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 72 KiB 5.8 GiB
3.9 TiB 46.62 0.98 115 up osd.388
389 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 276 KiB 6.6 GiB
3.5 TiB 52.24 1.09 128 up osd.389
390 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 72 KiB 5.3 GiB
4.2 TiB 42.24 0.88 104 up osd.390
391 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 148 KiB 5.8 GiB
3.9 TiB 46.57 0.98 115 up osd.391
392 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 120 KiB 6.8 GiB
3.4 TiB 53.86 1.13 133 up osd.392
393 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 12 KiB 7.6 GiB
3.4 TiB 53.92 1.13 133 up osd.393
394 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 80 KiB 5.8 GiB
3.9 TiB 46.60 0.98 115 up osd.394
395 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 64 KiB 7.4 GiB
3.4 TiB 53.92 1.13 133 up osd.395
396 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 244 KiB 6.4 GiB
3.5 TiB 51.46 1.08 127 up osd.396
397 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 240 KiB 6.3 GiB
3.7 TiB 48.95 1.02 121 up osd.397
398 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 124 KiB 6.0 GiB
3.8 TiB 47.84 1.00 118 up osd.398
399 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 8 KiB 5.9 GiB
3.9 TiB 47.03 0.98 116 up osd.399
400 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 248 KiB 6.4 GiB
3.6 TiB 50.69 1.06 125 up osd.400
401 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 116 KiB 5.6 GiB
3.9 TiB 46.18 0.97 114 up osd.401
402 hdd 7.28000 1.00000 7.3 TiB 3.0 TiB 3.0 TiB 144 KiB 5.2 GiB
4.2 TiB 41.75 0.87 103 up osd.402
403 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 180 KiB 5.9 GiB
3.8 TiB 47.79 1.00 118 up osd.403
404 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 56 KiB 6.0 GiB
4.1 TiB 44.13 0.92 109 up osd.404
405 hdd 7.28000 1.00000 7.3 TiB 3.0 TiB 3.0 TiB 52 KiB 5.3 GiB
4.2 TiB 41.72 0.87 103 up osd.405
406 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 300 KiB 6.7 GiB
3.4 TiB 53.51 1.12 132 up osd.406
407 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 284 KiB 6.5 GiB
3.5 TiB 51.42 1.08 127 up osd.407
408 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 60 KiB 6.9 GiB
3.4 TiB 53.61 1.12 132 up osd.408
409 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 188 KiB 6.4 GiB
3.5 TiB 52.53 1.10 130 up osd.409
410 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 28 KiB 7.2 GiB
3.4 TiB 53.49 1.12 132 up osd.410
411 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 272 KiB 6.6 GiB
3.4 TiB 53.08 1.11 131 up osd.411
412 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 8 KiB 7.4 GiB
3.4 TiB 53.89 1.13 133 up osd.412
413 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 24 KiB 5.6 GiB
4.0 TiB 44.96 0.94 111 up osd.413
414 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 128 KiB 5.7 GiB
3.9 TiB 45.76 0.96 113 up osd.414
415 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 300 KiB 6.3 GiB
3.5 TiB 51.48 1.08 127 up osd.415
416 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.6 TiB 264 KiB 6.3 GiB
3.6 TiB 50.21 1.05 124 up osd.416
417 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 72 KiB 6.0 GiB
3.8 TiB 48.24 1.01 119 up osd.417
418 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.6 TiB 208 KiB 6.3 GiB
3.6 TiB 50.19 1.05 124 up osd.418
-31 349.44000 - 349 TiB 172 TiB 171 TiB 5.0 MiB 342 GiB
178 TiB 49.11 1.03 - host ld4464-hdd_strgbox
272 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.1 TiB 212 KiB 6.3 GiB
4.1 TiB 43.30 0.91 107 up osd.272
273 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 64 KiB 7.9 GiB
3.4 TiB 53.09 1.11 131 up osd.273
274 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 76 KiB 7.3 GiB
3.5 TiB 52.34 1.10 129 up osd.274
275 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 60 KiB 6.1 GiB
4.1 TiB 43.42 0.91 107 up osd.275
276 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 80 KiB 7.2 GiB
3.8 TiB 48.15 1.01 119 up osd.276
277 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 72 KiB 7.2 GiB
3.6 TiB 50.62 1.06 125 up osd.277
278 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 20 KiB 7.3 GiB
3.7 TiB 49.43 1.03 122 up osd.278
279 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 92 KiB 7.0 GiB
3.8 TiB 47.12 0.99 116 up osd.279
280 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 48 KiB 7.9 GiB
3.6 TiB 51.14 1.07 126 up osd.280
281 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 60 KiB 6.5 GiB
4.1 TiB 43.45 0.91 107 up osd.281
282 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 120 KiB 7.4 GiB
3.4 TiB 53.09 1.11 131 up osd.282
283 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 68 KiB 7.6 GiB
3.4 TiB 53.92 1.13 133 up osd.283
284 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 108 KiB 7.3 GiB
3.6 TiB 50.65 1.06 125 up osd.284
285 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 60 KiB 8.1 GiB
3.4 TiB 53.83 1.13 133 up osd.285
286 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 120 KiB 7.2 GiB
3.9 TiB 46.24 0.97 114 up osd.286
287 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.1 TiB 180 KiB 6.4 GiB
4.1 TiB 43.31 0.91 107 up osd.287
288 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.1 TiB 60 KiB 6.7 GiB
4.1 TiB 43.36 0.91 107 up osd.288
289 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 128 KiB 7.0 GiB
3.9 TiB 46.18 0.97 114 up osd.289
290 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 196 KiB 6.5 GiB
4.1 TiB 43.40 0.91 107 up osd.290
291 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 100 KiB 7.3 GiB
3.7 TiB 49.39 1.03 122 up osd.291
292 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 80 KiB 6.8 GiB
3.8 TiB 48.19 1.01 119 up osd.292
293 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 72 KiB 7.8 GiB
3.4 TiB 53.06 1.11 131 up osd.293
294 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 211 KiB 6.7 GiB
4.1 TiB 43.03 0.90 106 up osd.294
295 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 52 KiB 7.9 GiB
3.4 TiB 53.90 1.13 133 up osd.295
296 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 228 KiB 6.8 GiB
3.8 TiB 47.40 0.99 117 up osd.296
297 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.4 TiB 12 KiB 6.8 GiB
3.8 TiB 47.42 0.99 117 up osd.297
298 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 132 KiB 7.3 GiB
3.5 TiB 51.87 1.09 128 up osd.298
299 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 12 KiB 7.3 GiB
3.7 TiB 49.79 1.04 123 up osd.299
300 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 180 KiB 7.3 GiB
3.4 TiB 53.05 1.11 131 up osd.300
301 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 180 KiB 7.3 GiB
3.4 TiB 53.86 1.13 133 up osd.301
302 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 72 KiB 7.0 GiB
3.7 TiB 49.41 1.03 122 up osd.302
303 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 64 KiB 6.8 GiB
3.9 TiB 46.32 0.97 114 up osd.303
304 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 40 KiB 6.9 GiB
3.7 TiB 48.59 1.02 120 up osd.304
305 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 68 KiB 7.1 GiB
3.8 TiB 48.23 1.01 119 up osd.305
306 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 120 KiB 7.5 GiB
3.5 TiB 51.87 1.09 128 up osd.306
307 hdd 7.28000 1.00000 7.3 TiB 3.2 TiB 3.2 TiB 116 KiB 6.6 GiB
4.1 TiB 43.42 0.91 107 up osd.307
308 hdd 7.28000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 128 KiB 6.0 GiB
4.2 TiB 42.92 0.90 106 up osd.308
309 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 116 KiB 7.6 GiB
3.4 TiB 53.57 1.12 132 up osd.309
310 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 64 KiB 6.6 GiB
3.9 TiB 46.63 0.98 115 up osd.310
311 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 60 KiB 7.5 GiB
3.6 TiB 50.68 1.06 125 up osd.311
312 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 164 KiB 7.5 GiB
3.4 TiB 53.43 1.12 132 up osd.312
313 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 244 KiB 6.6 GiB
3.9 TiB 45.79 0.96 113 up osd.313
314 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 84 KiB 7.6 GiB
3.4 TiB 53.78 1.13 133 up osd.314
315 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 272 KiB 7.4 GiB
3.4 TiB 53.04 1.11 131 up osd.315
316 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 180 KiB 7.4 GiB
3.5 TiB 51.81 1.08 128 up osd.316
317 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 120 KiB 7.6 GiB
3.5 TiB 51.90 1.09 128 up osd.317
318 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 12 KiB 6.9 GiB
3.9 TiB 46.27 0.97 114 up osd.318
319 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 92 KiB 7.3 GiB
3.4 TiB 52.62 1.10 130 up osd.319
-40 349.43994 - 349 TiB 172 TiB 172 TiB 4.5 MiB 345 GiB
177 TiB 49.33 1.03 - host ld4465-hdd_strgbox
320 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 44 KiB 6.9 GiB
3.9 TiB 46.20 0.97 114 up osd.320
321 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 72 KiB 7.5 GiB
3.4 TiB 53.53 1.12 132 up osd.321
322 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 64 KiB 7.1 GiB
3.7 TiB 48.59 1.02 120 up osd.322
323 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 64 KiB 6.9 GiB
3.8 TiB 47.76 1.00 118 up osd.323
324 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 16 KiB 7.4 GiB
3.9 TiB 47.08 0.99 116 up osd.324
325 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 64 KiB 7.8 GiB
3.5 TiB 52.28 1.09 129 up osd.325
326 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 108 KiB 7.7 GiB
3.6 TiB 50.16 1.05 124 up osd.326
327 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 16 KiB 7.3 GiB
3.8 TiB 48.13 1.01 119 up osd.327
328 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 16 KiB 7.5 GiB
3.4 TiB 53.50 1.12 132 up osd.328
329 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 220 KiB 6.8 GiB
3.8 TiB 47.79 1.00 118 up osd.329
330 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 72 KiB 7.7 GiB
3.4 TiB 53.52 1.12 132 up osd.330
331 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 60 KiB 7.2 GiB
3.8 TiB 47.37 0.99 117 up osd.331
332 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 176 KiB 7.5 GiB
3.4 TiB 53.45 1.12 132 up osd.332
333 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 72 KiB 7.0 GiB
3.9 TiB 45.80 0.96 113 up osd.333
334 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 60 KiB 7.4 GiB
3.4 TiB 52.59 1.10 130 up osd.334
335 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 32 KiB 6.6 GiB
4.0 TiB 45.38 0.95 112 up osd.335
336 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 164 KiB 7.4 GiB
3.4 TiB 53.42 1.12 132 up osd.336
337 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.6 TiB 72 KiB 7.4 GiB
3.6 TiB 50.22 1.05 124 up osd.337
338 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 160 KiB 7.3 GiB
3.4 TiB 53.55 1.12 132 up osd.338
339 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.6 TiB 68 KiB 7.7 GiB
3.6 TiB 50.21 1.05 124 up osd.339
340 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 120 KiB 7.0 GiB
3.9 TiB 45.79 0.96 113 up osd.340
341 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 76 KiB 7.0 GiB
3.9 TiB 47.07 0.99 116 up osd.341
342 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 120 KiB 7.0 GiB
3.9 TiB 45.95 0.96 113 up osd.342
343 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 64 KiB 7.2 GiB
3.7 TiB 48.64 1.02 120 up osd.343
344 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 192 KiB 7.4 GiB
3.4 TiB 53.54 1.12 132 up osd.344
345 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 88 KiB 7.3 GiB
3.6 TiB 50.70 1.06 125 up osd.345
346 hdd 7.28000 1.00000 7.3 TiB 3.4 TiB 3.4 TiB 64 KiB 7.2 GiB
3.9 TiB 47.03 0.98 116 up osd.346
347 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 184 KiB 7.4 GiB
3.4 TiB 53.12 1.11 131 up osd.347
348 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 4 KiB 7.1 GiB
3.9 TiB 45.77 0.96 113 up osd.348
349 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 68 KiB 7.6 GiB
3.4 TiB 53.04 1.11 131 up osd.349
350 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 132 KiB 7.2 GiB
3.5 TiB 52.27 1.09 129 up osd.350
351 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 304 KiB 6.5 GiB
3.9 TiB 45.76 0.96 113 up osd.351
352 hdd 7.28000 1.00000 7.3 TiB 3.8 TiB 3.8 TiB 64 KiB 7.5 GiB
3.5 TiB 52.21 1.09 129 up osd.352
353 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 40 KiB 7.4 GiB
3.6 TiB 49.90 1.04 123 up osd.353
354 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 108 KiB 7.5 GiB
3.4 TiB 53.50 1.12 132 up osd.354
355 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 244 KiB 6.5 GiB
4.0 TiB 45.37 0.95 112 up osd.355
356 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 120 KiB 6.9 GiB
3.7 TiB 49.37 1.03 122 up osd.356
357 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 44 KiB 7.1 GiB
4.0 TiB 45.37 0.95 112 up osd.357
358 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 72 KiB 7.1 GiB
3.7 TiB 49.40 1.03 122 up osd.358
359 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 16 KiB 6.8 GiB
3.7 TiB 49.53 1.04 122 up osd.359
360 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 72 KiB 6.7 GiB
4.0 TiB 45.47 0.95 112 up osd.360
361 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 64 KiB 6.9 GiB
4.0 TiB 45.37 0.95 112 up osd.361
362 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 100 KiB 6.7 GiB
3.9 TiB 45.89 0.96 113 up osd.362
363 hdd 7.28000 1.00000 7.3 TiB 3.3 TiB 3.3 TiB 244 KiB 6.6 GiB
4.0 TiB 45.36 0.95 112 up osd.363
364 hdd 7.28000 1.00000 7.3 TiB 3.6 TiB 3.6 TiB 120 KiB 7.1 GiB
3.7 TiB 49.37 1.03 122 up osd.364
365 hdd 7.28000 1.00000 7.3 TiB 3.7 TiB 3.7 TiB 88 KiB 7.4 GiB
3.6 TiB 50.69 1.06 125 up osd.365
366 hdd 7.28000 1.00000 7.3 TiB 3.5 TiB 3.5 TiB 32 KiB 7.2 GiB
3.8 TiB 48.24 1.01 119 up osd.366
367 hdd 7.28000 1.00000 7.3 TiB 3.9 TiB 3.9 TiB 124 KiB 7.5 GiB
3.4 TiB 53.43 1.12 132 up osd.367
-16 78.71997 - 79 TiB 53 TiB 53 TiB 3.8 MiB 107
GiB 25 TiB 67.93 1.42 - host ld5505-hdd_strgbox
76 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 4 KiB 2.3
GiB 436 GiB 73.98 1.55 41 up osd.76
77 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 96 KiB 2.1
GiB 587 GiB 64.98 1.36 36 up osd.77
78 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.0
GiB 584 GiB 65.16 1.36 36 up osd.78
79 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 80 KiB 2.4
GiB 466 GiB 72.18 1.51 40 up osd.79
80 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 92 KiB 2.1
GiB 586 GiB 65.02 1.36 36 up osd.80
81 hdd 1.64000 1.00000 1.6 TiB 879 GiB 877 GiB 120 KiB 2.0
GiB 797 GiB 52.45 1.10 29 up osd.81
82 hdd 1.64000 1.00000 1.6 TiB 974 GiB 972 GiB 64 KiB 1.9
GiB 702 GiB 58.13 1.22 32 up osd.82
83 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 16 KiB 2.3
GiB 492 GiB 70.64 1.48 39 up osd.83
84 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 60 KiB 2.0
GiB 614 GiB 63.36 1.33 35 up osd.84
85 hdd 1.64000 1.00000 1.6 TiB 880 GiB 878 GiB 64 KiB 1.7
GiB 796 GiB 52.48 1.10 29 up osd.85
86 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 80 KiB 2.4
GiB 434 GiB 74.13 1.55 41 up osd.86
87 hdd 1.64000 1.00000 1.6 TiB 971 GiB 969 GiB 60 KiB 1.9
GiB 705 GiB 57.95 1.21 32 up osd.87
88 hdd 1.64000 1.00000 1.6 TiB 882 GiB 880 GiB 120 KiB 1.8
GiB 794 GiB 52.63 1.10 29 up osd.88
89 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 116 KiB 2.4
GiB 463 GiB 72.38 1.52 40 up osd.89
90 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.5
GiB 404 GiB 75.88 1.59 42 up osd.90
91 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 132 KiB 2.3
GiB 494 GiB 70.54 1.48 39 up osd.91
92 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 164 KiB 2.1
GiB 587 GiB 64.96 1.36 36 up osd.92
93 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 96 KiB 2.1
GiB 612 GiB 63.51 1.33 35 up osd.93
94 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 88 KiB 2.4
GiB 463 GiB 72.38 1.52 40 up osd.94
95 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.4
GiB 407 GiB 75.74 1.59 42 up osd.95
96 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 120 KiB 2.0
GiB 645 GiB 61.53 1.29 34 up osd.96
97 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 44 KiB 2.5
GiB 346 GiB 79.33 1.66 44 up osd.97
98 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 132 KiB 2.3
GiB 400 GiB 76.12 1.59 42 up osd.98
99 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.3
GiB 491 GiB 70.68 1.48 39 up osd.99
100 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 124 KiB 2.4 GiB
343 GiB 79.51 1.66 44 up osd.100
101 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.2 GiB
525 GiB 68.66 1.44 38 up osd.101
102 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 116 KiB 2.3 GiB
434 GiB 74.09 1.55 41 up osd.102
103 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.3 GiB
436 GiB 74.00 1.55 41 up osd.103
104 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 76 KiB 2.0 GiB
616 GiB 63.23 1.32 35 up osd.104
105 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.5 GiB
404 GiB 75.88 1.59 42 up osd.105
106 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.1 GiB
558 GiB 66.72 1.40 37 up osd.106
107 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 12 KiB 2.6 GiB
462 GiB 72.45 1.52 40 up osd.107
108 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 124 KiB 2.0 GiB
647 GiB 61.41 1.29 34 up osd.108
109 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.1 GiB
587 GiB 64.97 1.36 36 up osd.109
110 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.1 GiB
584 GiB 65.16 1.36 36 up osd.110
111 hdd 1.64000 1.00000 1.6 TiB 880 GiB 878 GiB 88 KiB 1.7 GiB
796 GiB 52.48 1.10 29 up osd.111
112 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 160 KiB 2.1 GiB
616 GiB 63.23 1.32 35 up osd.112
113 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 12 KiB 3.1 GiB
431 GiB 74.28 1.56 41 up osd.113
114 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 40 KiB 2.2 GiB
617 GiB 63.20 1.32 35 up osd.114
115 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 85 KiB 2.3 GiB
466 GiB 72.19 1.51 40 up osd.115
116 hdd 1.64000 1.00000 1.6 TiB 908 GiB 906 GiB 64 KiB 1.8 GiB
768 GiB 54.18 1.13 30 up osd.116
117 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 60 KiB 2.4 GiB
376 GiB 77.58 1.62 43 up osd.117
118 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.4 GiB
558 GiB 66.70 1.40 37 up osd.118
119 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 12 KiB 2.3 GiB
466 GiB 72.18 1.51 40 up osd.119
120 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 12 KiB 2.5 GiB
403 GiB 75.94 1.59 42 up osd.120
121 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 84 KiB 2.3 GiB
467 GiB 72.16 1.51 40 up osd.121
122 hdd 1.64000 0.70000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.2 GiB
463 GiB 72.40 1.52 40 up osd.122
123 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 96 KiB 2.4 GiB
469 GiB 72.04 1.51 40 up osd.123
-22 78.72000 - 79 TiB 53 TiB 53 TiB 3.2 MiB 102
GiB 26 TiB 67.11 1.41 - host ld5506-hdd_strgbox
124 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.0 GiB
555 GiB 66.88 1.40 37 up osd.124
125 hdd 1.64000 1.00000 1.6 TiB 972 GiB 970 GiB 12 KiB 1.9 GiB
704 GiB 57.99 1.21 32 up osd.125
126 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.1 GiB
558 GiB 66.71 1.40 37 up osd.126
127 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.2 GiB
521 GiB 68.92 1.44 38 up osd.127
128 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB
408 GiB 75.68 1.58 42 up osd.128
129 hdd 1.64000 1.00000 1.6 TiB 940 GiB 939 GiB 60 KiB 1.7 GiB
736 GiB 56.12 1.17 31 up osd.129
130 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 64 KiB 2.1 GiB
643 GiB 61.66 1.29 34 up osd.130
131 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 160 KiB 2.4 GiB
403 GiB 75.93 1.59 42 up osd.131
132 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 60 KiB 2.0 GiB
615 GiB 63.32 1.33 35 up osd.132
133 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 28 KiB 2.6 GiB
375 GiB 77.62 1.63 43 up osd.133
134 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 4 KiB 2.1 GiB
545 GiB 67.48 1.41 37 up osd.134
135 hdd 1.64000 1.00000 1.6 TiB 997 GiB 995 GiB 64 KiB 2.0 GiB
679 GiB 59.49 1.25 33 up osd.135
136 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 64 KiB 2.4 GiB
343 GiB 79.56 1.67 44 up osd.136
137 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.3 GiB
495 GiB 70.48 1.48 39 up osd.137
138 hdd 1.64000 1.00000 1.6 TiB 967 GiB 965 GiB 64 KiB 1.8 GiB
709 GiB 57.70 1.21 32 up osd.138
139 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.2 GiB
464 GiB 72.29 1.51 40 up osd.139
140 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 64 KiB 2.2 GiB
374 GiB 77.66 1.63 43 up osd.140
141 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.4 GiB
464 GiB 72.34 1.51 40 up osd.141
142 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 4 KiB 2.0 GiB
585 GiB 65.09 1.36 36 up osd.142
143 hdd 1.64000 1.00000 1.6 TiB 1000 GiB 998 GiB 64 KiB 2.0 GiB
676 GiB 59.64 1.25 33 up osd.143
144 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 96 KiB 2.4 GiB
402 GiB 76.00 1.59 42 up osd.144
145 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 36 KiB 2.4 GiB
344 GiB 79.48 1.66 44 up osd.145
146 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 16 KiB 2.2 GiB
524 GiB 68.76 1.44 38 up osd.146
147 hdd 1.64000 1.00000 1.6 TiB 968 GiB 966 GiB 92 KiB 1.7 GiB
708 GiB 57.77 1.21 32 up osd.147
148 hdd 1.64000 1.00000 1.6 TiB 971 GiB 970 GiB 60 KiB 1.8 GiB
705 GiB 57.96 1.21 32 up osd.148
149 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 76 KiB 2.5 GiB
379 GiB 77.39 1.62 43 up osd.149
150 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 8 KiB 2.5 GiB
376 GiB 77.57 1.62 43 up osd.150
151 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.0 GiB
553 GiB 67.03 1.40 37 up osd.151
152 hdd 1.64000 1.00000 1.6 TiB 1001 GiB 999 GiB 120 KiB 1.8 GiB
675 GiB 59.73 1.25 33 up osd.152
153 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.6 GiB
438 GiB 73.87 1.55 41 up osd.153
154 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.2 GiB
553 GiB 67.00 1.40 37 up osd.154
155 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.0 GiB
585 GiB 65.12 1.36 36 up osd.155
156 hdd 1.64000 0.70000 1.6 TiB 1.2 TiB 1.2 TiB 108 KiB 2.4 GiB
466 GiB 72.19 1.51 40 up osd.156
157 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.1 GiB
557 GiB 66.78 1.40 37 up osd.157
158 hdd 1.64000 1.00000 1.6 TiB 910 GiB 908 GiB 120 KiB 1.7 GiB
766 GiB 54.27 1.14 30 up osd.158
159 hdd 1.64000 1.00000 1.6 TiB 907 GiB 905 GiB 60 KiB 1.7 GiB
769 GiB 54.11 1.13 30 up osd.159
160 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 4 KiB 2.2 GiB
528 GiB 68.49 1.43 38 up osd.160
161 hdd 1.64000 1.00000 1.6 TiB 940 GiB 938 GiB 64 KiB 1.8 GiB
736 GiB 56.07 1.17 31 up osd.161
162 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 124 KiB 2.0 GiB
614 GiB 63.37 1.33 35 up osd.162
163 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.4 GiB
403 GiB 75.95 1.59 42 up osd.163
164 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.2 GiB
585 GiB 65.08 1.36 36 up osd.164
165 hdd 1.64000 1.00000 1.6 TiB 972 GiB 971 GiB 60 KiB 1.9 GiB
704 GiB 58.02 1.21 32 up osd.165
166 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.4 GiB
436 GiB 74.00 1.55 41 up osd.166
167 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 64 KiB 2.1 GiB
613 GiB 63.43 1.33 35 up osd.167
168 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.3 GiB
434 GiB 74.12 1.55 41 up osd.168
169 hdd 1.64000 1.00000 1.6 TiB 910 GiB 908 GiB 60 KiB 1.7 GiB
766 GiB 54.27 1.14 30 up osd.169
170 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.2 GiB
555 GiB 66.88 1.40 37 up osd.170
171 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 96 KiB 2.4 GiB
435 GiB 74.03 1.55 41 up osd.171
-25 78.72000 - 79 TiB 53 TiB 53 TiB 4.0 MiB 104
GiB 25 TiB 67.93 1.42 - host ld5507-hdd_strgbox
189 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 132 KiB 2.3 GiB
408 GiB 75.68 1.58 42 up osd.189
190 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 120 KiB 2.4 GiB
376 GiB 77.54 1.62 43 up osd.190
191 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 60 KiB 2.0 GiB
613 GiB 63.44 1.33 35 up osd.191
192 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.2 GiB
558 GiB 66.73 1.40 37 up osd.192
193 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.1 GiB
561 GiB 66.53 1.39 37 up osd.193
194 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.3 GiB
496 GiB 70.43 1.47 39 up osd.194
195 hdd 1.64000 1.00000 1.6 TiB 909 GiB 907 GiB 120 KiB 1.7 GiB
767 GiB 54.22 1.14 30 up osd.195
196 hdd 1.64000 1.00000 1.6 TiB 998 GiB 996 GiB 64 KiB 1.9 GiB
678 GiB 59.53 1.25 33 up osd.196
197 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB
434 GiB 74.08 1.55 41 up osd.197
198 hdd 1.64000 1.00000 1.6 TiB 971 GiB 969 GiB 132 KiB 1.9 GiB
705 GiB 57.92 1.21 32 up osd.198
199 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 124 KiB 2.1 GiB
585 GiB 65.08 1.36 36 up osd.199
200 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.4 GiB
434 GiB 74.08 1.55 41 up osd.200
201 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 76 KiB 2.3 GiB
498 GiB 70.30 1.47 39 up osd.201
202 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.3 GiB
496 GiB 70.40 1.47 39 up osd.202
203 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.2 GiB
495 GiB 70.44 1.47 39 up osd.203
204 hdd 1.64000 1.00000 1.6 TiB 882 GiB 880 GiB 16 KiB 2.0 GiB
794 GiB 52.63 1.10 29 up osd.204
205 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 132 KiB 2.1 GiB
586 GiB 65.05 1.36 36 up osd.205
206 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 64 KiB 1.9 GiB
645 GiB 61.53 1.29 34 up osd.206
207 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.3 GiB
460 GiB 72.52 1.52 40 up osd.207
208 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 144 KiB 2.3 GiB
497 GiB 70.32 1.47 39 up osd.208
209 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 108 KiB 2.2 GiB
584 GiB 65.14 1.36 36 up osd.209
210 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 180 KiB 2.2 GiB
492 GiB 70.67 1.48 39 up osd.210
211 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.2 GiB
557 GiB 66.75 1.40 37 up osd.211
212 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.2 GiB
585 GiB 65.09 1.36 36 up osd.212
213 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 5 KiB 2.7 GiB
432 GiB 74.24 1.55 41 up osd.213
214 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.2 GiB
554 GiB 66.97 1.40 37 up osd.214
215 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 28 KiB 2.4 GiB
405 GiB 75.81 1.59 42 up osd.215
216 hdd 1.64000 0.89999 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.2 GiB
527 GiB 68.58 1.44 38 up osd.216
217 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.2 GiB
493 GiB 70.61 1.48 39 up osd.217
218 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.4 GiB
461 GiB 72.46 1.52 40 up osd.218
219 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.0 GiB
588 GiB 64.89 1.36 36 up osd.219
251 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.3 GiB
463 GiB 72.35 1.51 40 up osd.251
252 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 28 KiB 2.4 GiB
312 GiB 81.40 1.70 45 up osd.252
253 hdd 1.64000 1.00000 1.6 TiB 999 GiB 997 GiB 60 KiB 1.8 GiB
677 GiB 59.59 1.25 33 up osd.253
254 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 108 KiB 2.2 GiB
562 GiB 66.49 1.39 37 up osd.254
255 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 108 KiB 2.2 GiB
555 GiB 66.88 1.40 37 up osd.255
256 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.2 GiB
402 GiB 76.03 1.59 42 up osd.256
257 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 64 KiB 2.0 GiB
613 GiB 63.44 1.33 35 up osd.257
258 hdd 1.64000 1.00000 1.6 TiB 1002 GiB 1000 GiB 60 KiB 1.9 GiB
674 GiB 59.76 1.25 33 up osd.258
259 hdd 1.64000 1.00000 1.6 TiB 881 GiB 879 GiB 124 KiB 1.7 GiB
795 GiB 52.55 1.10 29 up osd.259
260 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.2 GiB
464 GiB 72.34 1.51 40 up osd.260
261 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.3 GiB
463 GiB 72.36 1.52 40 up osd.261
262 hdd 1.64000 0.89999 1.6 TiB 1.3 TiB 1.3 TiB 136 KiB 2.4 GiB
372 GiB 77.78 1.63 43 up osd.262
263 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.2 GiB
466 GiB 72.17 1.51 40 up osd.263
264 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB
407 GiB 75.74 1.59 42 up osd.264
265 hdd 1.64000 1.00000 1.6 TiB 878 GiB 876 GiB 120 KiB 1.7 GiB
798 GiB 52.37 1.10 29 up osd.265
266 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 132 KiB 2.4 GiB
431 GiB 74.29 1.56 41 up osd.266
267 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.1 GiB
582 GiB 65.27 1.37 36 up osd.267
-28 78.71997 - 79 TiB 51 TiB 51 TiB 3.8 MiB 101
GiB 27 TiB 65.26 1.37 - host ld5508-hdd_strgbox
42 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 60 KiB 2.1
GiB 646 GiB 61.43 1.29 34 up osd.42
43 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 64 KiB 2.1
GiB 553 GiB 66.99 1.40 37 up osd.43
44 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4
GiB 437 GiB 73.92 1.55 41 up osd.44
45 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0
GiB 614 GiB 63.35 1.33 35 up osd.45
46 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.4
GiB 406 GiB 75.78 1.59 42 up osd.46
47 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 32 KiB 2.1
GiB 586 GiB 65.02 1.36 36 up osd.47
48 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 16 KiB 2.2
GiB 522 GiB 68.83 1.44 38 up osd.48
49 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.2
GiB 530 GiB 68.35 1.43 38 up osd.49
50 hdd 1.64000 1.00000 1.6 TiB 1001 GiB 999 GiB 72 KiB 1.9
GiB 675 GiB 59.71 1.25 33 up osd.50
51 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0
GiB 645 GiB 61.50 1.29 34 up osd.51
52 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.1
GiB 553 GiB 67.01 1.40 37 up osd.52
53 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4
GiB 467 GiB 72.15 1.51 40 up osd.53
54 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.1 TiB 84 KiB 2.2
GiB 498 GiB 70.29 1.47 39 up osd.54
55 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 72 KiB 2.3
GiB 496 GiB 70.39 1.47 39 up osd.55
56 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 120 KiB 2.0
GiB 645 GiB 61.50 1.29 34 up osd.56
57 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 120 KiB 2.2
GiB 525 GiB 68.65 1.44 38 up osd.57
58 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.5
GiB 402 GiB 75.99 1.59 42 up osd.58
220 hdd 1.64000 0.89999 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.4 GiB
461 GiB 72.47 1.52 40 up osd.220
221 hdd 1.64000 1.00000 1.6 TiB 972 GiB 970 GiB 60 KiB 1.8 GiB
704 GiB 57.97 1.21 32 up osd.221
222 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.3 GiB
492 GiB 70.63 1.48 39 up osd.222
223 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.3 GiB
468 GiB 72.08 1.51 40 up osd.223
224 hdd 1.64000 1.00000 1.6 TiB 940 GiB 938 GiB 60 KiB 1.9 GiB
736 GiB 56.06 1.17 31 up osd.224
225 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB
405 GiB 75.82 1.59 42 up osd.225
226 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 60 KiB 2.2 GiB
555 GiB 66.88 1.40 37 up osd.226
227 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.3 GiB
466 GiB 72.22 1.51 40 up osd.227
228 hdd 1.64000 1.00000 1.6 TiB 938 GiB 936 GiB 60 KiB 1.8 GiB
738 GiB 55.96 1.17 31 up osd.228
229 hdd 1.64000 1.00000 1.6 TiB 910 GiB 909 GiB 72 KiB 1.8 GiB
766 GiB 54.32 1.14 30 up osd.229
230 hdd 1.64000 0.79999 1.6 TiB 1.2 TiB 1.2 TiB 60 KiB 2.4 GiB
404 GiB 75.88 1.59 42 up osd.230
231 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0 GiB
644 GiB 61.58 1.29 34 up osd.231
232 hdd 1.64000 1.00000 1.6 TiB 938 GiB 936 GiB 64 KiB 1.8 GiB
738 GiB 55.96 1.17 31 up osd.232
233 hdd 1.64000 1.00000 1.6 TiB 1001 GiB 1000 GiB 60 KiB 1.8 GiB
675 GiB 59.75 1.25 33 up osd.233
234 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 8 KiB 2.2 GiB
584 GiB 65.16 1.36 36 up osd.234
235 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0 GiB
617 GiB 63.17 1.32 35 up osd.235
236 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 72 KiB 2.2 GiB
497 GiB 70.34 1.47 39 up osd.236
237 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 72 KiB 2.1 GiB
555 GiB 66.88 1.40 37 up osd.237
238 hdd 1.64000 1.00000 1.6 TiB 880 GiB 878 GiB 76 KiB 1.7 GiB
796 GiB 52.48 1.10 29 up osd.238
239 hdd 1.64000 0.70000 1.6 TiB 1.2 TiB 1.2 TiB 72 KiB 2.4 GiB
490 GiB 70.77 1.48 39 up osd.239
240 hdd 1.64000 1.00000 1.6 TiB 970 GiB 968 GiB 180 KiB 1.8 GiB
706 GiB 57.89 1.21 32 up osd.240
241 hdd 1.64000 1.00000 1.6 TiB 878 GiB 876 GiB 132 KiB 1.7 GiB
798 GiB 52.38 1.10 29 up osd.241
242 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 4 KiB 1.9 GiB
648 GiB 61.32 1.28 34 up osd.242
243 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 76 KiB 2.0 GiB
616 GiB 63.24 1.32 35 up osd.243
244 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 132 KiB 2.1 GiB
581 GiB 65.34 1.37 36 up osd.244
245 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 124 KiB 2.1 GiB
583 GiB 65.21 1.37 36 up osd.245
246 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 76 KiB 2.1 GiB
581 GiB 65.31 1.37 36 up osd.246
247 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 136 KiB 2.2 GiB
530 GiB 68.39 1.43 38 up osd.247
248 hdd 1.64000 1.00000 1.6 TiB 880 GiB 878 GiB 132 KiB 1.8 GiB
796 GiB 52.48 1.10 29 up osd.248
249 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.1 TiB 104 KiB 2.3 GiB
497 GiB 70.35 1.47 39 up osd.249
250 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 180 KiB 2.1 GiB
617 GiB 63.18 1.32 35 up osd.250
-1 169.68462 - 169 TiB 6.0 TiB 5.9 TiB 8.6 MiB 106
GiB 163 TiB 3.56 0.07 - root default
-46 4.38295 - 4.4 TiB 212 GiB 206 GiB 452 KiB 6.0 GiB
4.1 TiB 4.76 0.10 - host ld4257
435 hdd 1.09000 1.00000 1.1 TiB 61 GiB 60 GiB 176 KiB 1024 MiB
1.0 TiB 5.46 0.11 2 up osd.435
436 hdd 1.09000 1.00000 1.1 TiB 51 GiB 50 GiB 20 KiB 1024 MiB
1.0 TiB 4.56 0.10 3 up osd.436
437 hdd 1.09000 1.00000 1.1 TiB 34 GiB 33 GiB 64 KiB 1024 MiB
1.1 TiB 3.03 0.06 1 up osd.437
20 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 64 KiB 1024
MiB 349 GiB 6.05 0.13 1 up osd.20
21 ssd 0.37099 1.00000 371 GiB 6.4 GiB 5.4 GiB 0 B 1
GiB 365 GiB 1.73 0.04 0 up osd.21
22 ssd 0.37099 1.00000 371 GiB 38 GiB 37 GiB 128 KiB 1024
MiB 333 GiB 10.26 0.21 2 up osd.22
-34 6.78294 - 6.8 TiB 407 GiB 401 GiB 680 KiB 6.0 GiB
6.4 TiB 5.89 0.12 - host ld4464
438 hdd 1.09000 1.00000 1.1 TiB 132 GiB 131 GiB 144 KiB 1024 MiB
984 GiB 11.82 0.25 4 up osd.438
439 hdd 1.09000 1.00000 1.1 TiB 78 GiB 77 GiB 128 KiB 1024 MiB
1.0 TiB 6.96 0.15 2 up osd.439
23 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 64 KiB 1024
MiB 349 GiB 5.93 0.12 1 up osd.23
24 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 96 KiB 1024
MiB 349 GiB 5.97 0.13 1 up osd.24
25 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 128 KiB 1024
MiB 349 GiB 6.01 0.13 1 up osd.25
26 ssd 3.48999 1.00000 3.5 TiB 131 GiB 130 GiB 120 KiB 1024
MiB 3.4 TiB 3.67 0.08 8 up osd.26
-37 6.78294 - 6.8 TiB 343 GiB 337 GiB 284 KiB 6.0 GiB
6.4 TiB 4.96 0.10 - host ld4465
440 hdd 1.09000 1.00000 1.1 TiB 51 GiB 50 GiB 0 B 1 GiB
1.0 TiB 4.55 0.10 1 up osd.440
441 hdd 1.09000 1.00000 1.1 TiB 77 GiB 76 GiB 24 KiB 1024 MiB
1.0 TiB 6.89 0.14 3 up osd.441
27 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 4 KiB 1024
MiB 349 GiB 6.02 0.13 1 up osd.27
28 ssd 0.37099 1.00000 371 GiB 38 GiB 37 GiB 172 KiB 1024
MiB 333 GiB 10.26 0.21 2 up osd.28
29 ssd 0.37099 1.00000 371 GiB 22 GiB 21 GiB 52 KiB 1024
MiB 349 GiB 5.91 0.12 1 up osd.29
30 ssd 3.48999 1.00000 3.5 TiB 133 GiB 132 GiB 32 KiB 1024
MiB 3.4 TiB 3.72 0.08 8 up osd.30
-3 37.93394 - 38 TiB 1.5 TiB 1.5 TiB 2.1 MiB 22
GiB 36 TiB 4.05 0.08 - host ld5505
31 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 72 KiB 1024
MiB 1.6 TiB 2.00 0.04 2 up osd.31
32 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 128 KiB 1024
MiB 1.6 TiB 4.66 0.10 3 up osd.32
33 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 32 KiB 1024
MiB 1.6 TiB 1.99 0.04 1 up osd.33
34 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 96 KiB 1024
MiB 1.6 TiB 4.65 0.10 2 up osd.34
35 hdd 1.64000 1.00000 1.6 TiB 131 GiB 130 GiB 192 KiB 1024
MiB 1.5 TiB 7.81 0.16 4 up osd.35
36 hdd 1.64000 1.00000 1.6 TiB 131 GiB 130 GiB 208 KiB 1024
MiB 1.5 TiB 7.84 0.16 5 up osd.36
37 hdd 1.64000 1.00000 1.6 TiB 77 GiB 76 GiB 200 KiB 1024
MiB 1.6 TiB 4.60 0.10 4 up osd.37
38 hdd 1.64000 1.00000 1.6 TiB 87 GiB 86 GiB 128 KiB 1024
MiB 1.6 TiB 5.19 0.11 3 up osd.38
39 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 80 KiB 1024
MiB 1.6 TiB 5.23 0.11 3 up osd.39
40 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 228 KiB 1024
MiB 1.6 TiB 3.60 0.08 2 up osd.40
41 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 120 KiB 1024
MiB 1.6 TiB 5.27 0.11 3 up osd.41
59 hdd 1.64000 1.00000 1.6 TiB 87 GiB 86 GiB 164 KiB 1024
MiB 1.6 TiB 5.22 0.11 4 up osd.59
60 hdd 1.64000 1.00000 1.6 TiB 132 GiB 131 GiB 216 KiB 1024
MiB 1.5 TiB 7.86 0.16 4 up osd.60
61 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 48 KiB 1024
MiB 1.6 TiB 1.98 0.04 3 up osd.61
62 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 80 KiB 1024
MiB 1.6 TiB 5.23 0.11 3 up osd.62
63 hdd 1.64000 1.00000 1.6 TiB 87 GiB 86 GiB 48 KiB 1024
MiB 1.6 TiB 5.19 0.11 3 up osd.63
64 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 48 KiB 1024
MiB 1.6 TiB 4.66 0.10 2 up osd.64
0 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 8 KiB 1024
MiB 2.9 TiB 0.17 0.00 1 up osd.0
1 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1
GiB 2.9 TiB 0.17 0.00 1 up osd.1
8 ssd 3.48999 1.00000 3.5 TiB 129 GiB 128 GiB 4 KiB 1024
MiB 3.4 TiB 3.60 0.08 8 up osd.8
9 ssd 0.37199 1.00000 372 GiB 36 GiB 35 GiB 16 KiB 1024
MiB 336 GiB 9.71 0.20 2 up osd.9
10 ssd 0.37199 1.00000 372 GiB 5.0 GiB 4.0 GiB 0 B 1
GiB 367 GiB 1.35 0.03 0 up osd.10
-7 37.93394 - 38 TiB 1.1 TiB 1.1 TiB 1.4 MiB 22
GiB 37 TiB 2.97 0.06 - host ld5506
65 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 16 KiB 1024
MiB 1.6 TiB 1.97 0.04 1 up osd.65
66 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 64 KiB 1024
MiB 1.6 TiB 4.65 0.10 2 up osd.66
67 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 168 KiB 1024
MiB 1.6 TiB 1.99 0.04 2 up osd.67
68 hdd 1.64000 1.00000 1.6 TiB 89 GiB 88 GiB 80 KiB 1024
MiB 1.6 TiB 5.29 0.11 3 up osd.68
69 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 124 KiB 1024
MiB 1.6 TiB 3.60 0.08 2 up osd.69
70 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 96 KiB 1024
MiB 1.6 TiB 2.00 0.04 1 up osd.70
71 hdd 1.64000 1.00000 1.6 TiB 79 GiB 78 GiB 48 KiB 1024
MiB 1.6 TiB 4.68 0.10 4 up osd.71
72 hdd 1.64000 1.00000 1.6 TiB 87 GiB 86 GiB 136 KiB 1024
MiB 1.6 TiB 5.19 0.11 4 up osd.72
73 hdd 1.64000 1.00000 1.6 TiB 105 GiB 104 GiB 16 KiB 1024
MiB 1.5 TiB 6.27 0.13 3 up osd.73
74 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 116 KiB 1024
MiB 1.6 TiB 3.57 0.07 3 up osd.74
75 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 16 KiB 1024
MiB 1.6 TiB 3.58 0.08 2 up osd.75
172 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 32 KiB 1024 MiB
1.6 TiB 3.58 0.08 2 up osd.172
173 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 12 KiB 1024 MiB
1.6 TiB 3.59 0.08 3 up osd.173
174 hdd 1.64000 1.00000 1.6 TiB 78 GiB 77 GiB 172 KiB 1024 MiB
1.6 TiB 4.67 0.10 2 up osd.174
175 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 16 KiB 1024 MiB
1.6 TiB 2.00 0.04 1 up osd.175
176 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 88 KiB 1024 MiB
1.6 TiB 1.98 0.04 1 up osd.176
177 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 216 KiB 1024 MiB
1.6 TiB 2.03 0.04 1 up osd.177
2 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 8 KiB 1024
MiB 2.9 TiB 0.17 0.00 2 up osd.2
3 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1
GiB 2.9 TiB 0.17 0.00 3 up osd.3
11 ssd 0.37199 1.00000 372 GiB 5.0 GiB 4.0 GiB 0 B 1
GiB 367 GiB 1.35 0.03 0 up osd.11
12 ssd 3.48999 1.00000 3.5 TiB 99 GiB 98 GiB 4 KiB 1024
MiB 3.4 TiB 2.76 0.06 6 up osd.12
13 ssd 0.37199 1.00000 372 GiB 21 GiB 20 GiB 4 KiB 1024
MiB 351 GiB 5.53 0.12 1 up osd.13
-10 37.93394 - 38 TiB 1.1 TiB 1.1 TiB 1.8 MiB 22
GiB 37 TiB 3.00 0.06 - host ld5507
178 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 148 KiB 1024 MiB
1.6 TiB 1.99 0.04 1 up osd.178
179 hdd 1.64000 1.00000 1.6 TiB 61 GiB 60 GiB 84 KiB 1024 MiB
1.6 TiB 3.61 0.08 3 up osd.179
180 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 136 KiB 1024 MiB
1.6 TiB 5.26 0.11 3 up osd.180
181 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 144 KiB 1024 MiB
1.6 TiB 2.00 0.04 1 up osd.181
182 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 212 KiB 1024 MiB
1.6 TiB 1.99 0.04 1 up osd.182
183 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 24 KiB 1024 MiB
1.6 TiB 1.98 0.04 1 up osd.183
184 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 96 KiB 1024 MiB
1.6 TiB 1.97 0.04 1 up osd.184
185 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 112 KiB 1024 MiB
1.6 TiB 1.99 0.04 1 up osd.185
186 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 64 KiB 1024 MiB
1.6 TiB 3.61 0.08 2 up osd.186
187 hdd 1.64000 1.00000 1.6 TiB 105 GiB 104 GiB 52 KiB 1024 MiB
1.5 TiB 6.25 0.13 3 up osd.187
188 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 32 KiB 1024 MiB
1.6 TiB 2.00 0.04 1 up osd.188
268 hdd 1.64000 1.00000 1.6 TiB 131 GiB 130 GiB 64 KiB 1024 MiB
1.5 TiB 7.80 0.16 5 up osd.268
269 hdd 1.64000 1.00000 1.6 TiB 61 GiB 60 GiB 112 KiB 1024 MiB
1.6 TiB 3.63 0.08 3 up osd.269
270 hdd 1.64000 1.00000 1.6 TiB 61 GiB 60 GiB 56 KiB 1024 MiB
1.6 TiB 3.63 0.08 2 up osd.270
271 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 208 KiB 1024 MiB
1.6 TiB 3.59 0.08 2 up osd.271
368 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 140 KiB 1024 MiB
1.6 TiB 1.99 0.04 1 up osd.368
369 hdd 1.64000 1.00000 1.6 TiB 105 GiB 104 GiB 116 KiB 1024 MiB
1.5 TiB 6.24 0.13 4 up osd.369
4 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1
GiB 2.9 TiB 0.17 0.00 1 up osd.4
5 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1
GiB 2.9 TiB 0.17 0.00 2 up osd.5
14 ssd 0.37199 1.00000 372 GiB 5.0 GiB 4.0 GiB 0 B 1
GiB 367 GiB 1.35 0.03 0 up osd.14
15 ssd 0.37199 1.00000 372 GiB 21 GiB 20 GiB 16 KiB 1024
MiB 351 GiB 5.64 0.12 1 up osd.15
16 ssd 3.48999 1.00000 3.5 TiB 129 GiB 128 GiB 40 KiB 1024
MiB 3.4 TiB 3.61 0.08 8 up osd.16
-13 37.93394 - 38 TiB 1.3 TiB 1.3 TiB 2.0 MiB 22
GiB 37 TiB 3.42 0.07 - host ld5508
370 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 144 KiB 1024 MiB
1.6 TiB 2.00 0.04 1 up osd.370
419 hdd 1.64000 1.00000 1.6 TiB 77 GiB 76 GiB 48 KiB 1024 MiB
1.6 TiB 4.62 0.10 2 up osd.419
420 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 80 KiB 1024 MiB
1.6 TiB 3.56 0.07 2 up osd.420
421 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 28 KiB 1024 MiB
1.6 TiB 1.96 0.04 1 up osd.421
422 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 72 KiB 1024 MiB
1.6 TiB 5.26 0.11 4 up osd.422
423 hdd 1.64000 1.00000 1.6 TiB 88 GiB 87 GiB 64 KiB 1024 MiB
1.6 TiB 5.22 0.11 4 up osd.423
424 hdd 1.64000 1.00000 1.6 TiB 176 GiB 175 GiB 224 KiB 1024 MiB
1.5 TiB 10.47 0.22 5 up osd.424
425 hdd 1.64000 1.00000 1.6 TiB 106 GiB 105 GiB 196 KiB 1024 MiB
1.5 TiB 6.33 0.13 3 up osd.425
426 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 112 KiB 1024 MiB
1.6 TiB 3.56 0.07 2 up osd.426
427 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 48 KiB 1024 MiB
1.6 TiB 1.97 0.04 1 up osd.427
428 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 32 KiB 1024 MiB
1.6 TiB 2.01 0.04 1 up osd.428
429 hdd 1.64000 1.00000 1.6 TiB 106 GiB 105 GiB 132 KiB 1024 MiB
1.5 TiB 6.30 0.13 3 up osd.429
430 hdd 1.64000 1.00000 1.6 TiB 34 GiB 33 GiB 120 KiB 1024 MiB
1.6 TiB 2.02 0.04 1 up osd.430
431 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 112 KiB 1024 MiB
1.6 TiB 3.58 0.08 2 up osd.431
432 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 208 KiB 1024 MiB
1.6 TiB 1.99 0.04 2 up osd.432
433 hdd 1.64000 1.00000 1.6 TiB 60 GiB 59 GiB 164 KiB 1024 MiB
1.6 TiB 3.56 0.07 2 up osd.433
434 hdd 1.64000 1.00000 1.6 TiB 33 GiB 32 GiB 156 KiB 1024 MiB
1.6 TiB 2.00 0.04 1 up osd.434
6 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 0 B 1
GiB 2.9 TiB 0.17 0.00 3 up osd.6
7 nvme 2.90999 1.00000 2.9 TiB 5.0 GiB 4.0 GiB 8 KiB 1024
MiB 2.9 TiB 0.17 0.00 3 up osd.7
17 ssd 0.37199 1.00000 372 GiB 5.0 GiB 4.0 GiB 8 KiB 1024
MiB 367 GiB 1.35 0.03 0 up osd.17
18 ssd 0.37199 1.00000 372 GiB 36 GiB 35 GiB 12 KiB 1024
MiB 336 GiB 9.80 0.21 2 up osd.18
19 ssd 3.48999 1.00000 3.5 TiB 162 GiB 161 GiB 40 KiB 1024
MiB 3.3 TiB 4.54 0.10 10 up osd.19
TOTAL 1.5 PiB 731 TiB 730 TiB 39 MiB 1.5
TiB 800 TiB 47.76
MIN/MAX VAR: 0.00/1.70 STDDEV: 25.46
Am 01.11.2019 um 11:24 schrieb Paul Emmerich:
Looks like you didn't tell the whole story, please
post the *full*
output of ceph -s and ceph osd df tree.
Wild guess: you need to increase "mon max pg per osd"
Paul
1645
days inactive
1646
days old
4 comments
3 participants
participants (3)
-
Paul Emmerich -
Thomas -
Thomas Schneider