Size and capacity calculations questions
Hi! We have a ceph cluster with 42 OSD in production as a server providing mainly home-directories of users. Ceph is 14.2.4 nautilus. We have 3 pools. One images (for rbd images) a cephfs_metadata and a cephfs_data pool. Our raw data is about 5.6T. All pools have replica size 3 and there are only very little snapshots in the rbd images pool, the cephfspool doesnt use snapshots. How is it possible that the status tells us, that 21T/46T is used, because thats much more than 3 times the raw size. Also, to make that more confusing, there as at least half of the cluster free, and we get pg backfill_toofull after we added some OSDs lately. The ceph dashboard tells aus the pool ist 82 % full and has only 4.5 T free. The autoscale module seems to calculate the 20T times 3 for the space needed and thus has wrong numbers (see below). Status of the cluster is added below too. how can these size/capacity numbers be explained? and, would be there a recommendation to change something? Thank you in advance! best Jochen # ceph -s cluster: id: 2b16167f-3f33-4580-a0e9-7a71978f403d health: HEALTH_ERR Degraded data redundancy (low space): 1 pg backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 2 pools have too many placement groups services: mon: 4 daemons, quorum jade,assam,matcha,jasmine (age 2d) mgr: earl(active, since 24h), standbys: assam mds: cephfs:1 {0=assam=up:active} 1 up:standby osd: 42 osds: 42 up (since 106m), 42 in (since 115m); 30 remapped pgs data: pools: 3 pools, 2048 pgs objects: 29.80M objects, 5.6 TiB usage: 21 TiB used, 25 TiB / 46 TiB avail pgs: 1164396/89411013 objects misplaced (1.302%) 2018 active+clean 22 active+remapped+backfill_wait 7 active+remapped+backfilling 1 active+remapped+backfill_wait+backfill_toofull io: client: 1.7 KiB/s rd, 516 KiB/s wr, 0 op/s rd, 28 op/s wr recovery: 9.2 MiB/s, 41 objects/s # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 354.2G 3.0 46100G 0.0231 1.0 1024 32 warn cephfs_metadata 13260M 3.0 595.7G 0.0652 1.0 512 8 warn cephfs_data 20802G 3.0 46100G 1.3537 1.0 512 warn
On 6.12.19 13:29, Jochen Schulz wrote:
Hi!
We have a ceph cluster with 42 OSD in production as a server providing mainly home-directories of users. Ceph is 14.2.4 nautilus.
We have 3 pools. One images (for rbd images) a cephfs_metadata and a cephfs_data pool.
Our raw data is about 5.6T. All pools have replica size 3 and there are only very little snapshots in the rbd images pool, the cephfspool doesnt use snapshots.
How is it possible that the status tells us, that 21T/46T is used, because thats much more than 3 times the raw size.
Also, to make that more confusing, there as at least half of the cluster free, and we get pg backfill_toofull after we added some OSDs lately. The ceph dashboard tells aus the pool ist 82 % full and has only 4.5 T free.
The autoscale module seems to calculate the 20T times 3 for the space needed and thus has wrong numbers (see below).
Status of the cluster is added below too.
how can these size/capacity numbers be explained? and, would be there a recommendation to change something?
Thank you in advance!
best Jochen
# ceph -s
cluster: id: 2b16167f-3f33-4580-a0e9-7a71978f403d health: HEALTH_ERR Degraded data redundancy (low space): 1 pg backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 2 pools have too many placement groups
services: mon: 4 daemons, quorum jade,assam,matcha,jasmine (age 2d) mgr: earl(active, since 24h), standbys: assam mds: cephfs:1 {0=assam=up:active} 1 up:standby osd: 42 osds: 42 up (since 106m), 42 in (since 115m); 30 remapped pgs
data: pools: 3 pools, 2048 pgs objects: 29.80M objects, 5.6 TiB usage: 21 TiB used, 25 TiB / 46 TiB avail pgs: 1164396/89411013 objects misplaced (1.302%) 2018 active+clean 22 active+remapped+backfill_wait 7 active+remapped+backfilling 1 active+remapped+backfill_wait+backfill_toofull
io: client: 1.7 KiB/s rd, 516 KiB/s wr, 0 op/s rd, 28 op/s wr recovery: 9.2 MiB/s, 41 objects/s
# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 354.2G 3.0 46100G 0.0231 1.0 1024 32 warn cephfs_metadata 13260M 3.0 595.7G 0.0652 1.0 512 8 warn cephfs_data 20802G 3.0 46100G 1.3537 1.0 512 warn
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Please, provide output of ceph df and ceph osd df - that should explain both questions (21T and 82%).
Hi! Thank you! The output of both commands are below. I still dont understand why there are 21T used data (because 5.5T*3 = 16.5T != 21T) and why there seems to be only 4.5 T MAX AVAIL, but the osd output tells we have 25T free space. $ sudo ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 45 TiB 24 TiB 21 TiB 21 TiB 46.33 ssd 596 GiB 524 GiB 1.7 GiB 72 GiB 12.09 TOTAL 46 TiB 25 TiB 21 TiB 21 TiB 45.89 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL images 8 149 GiB 38.30k 354 GiB 2.52 4.5 TiB cephfs_data 9 5.5 TiB 26.61M 20 TiB 60.36 4.5 TiB cephfs_metadata 10 12 GiB 3.17M 13 GiB 2.57 164 GiB $ sudo ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.89000 1.00000 931 GiB 456 GiB 453 GiB 136 MiB 3.5 GiB 475 GiB 49.01 1.07 103 up 1 hdd 0.89000 1.00000 931 GiB 495 GiB 491 GiB 100 MiB 3.9 GiB 436 GiB 53.14 1.16 89 up 4 hdd 0.89000 1.00000 931 GiB 345 GiB 342 GiB 108 MiB 2.9 GiB 586 GiB 37.05 0.81 87 up 5 hdd 0.89000 1.00000 931 GiB 521 GiB 517 GiB 108 MiB 4.1 GiB 410 GiB 55.96 1.22 98 up 6 hdd 0.89000 1.00000 931 GiB 367 GiB 364 GiB 95 MiB 3.2 GiB 564 GiB 39.44 0.86 95 up 7 hdd 0.89000 1.00000 931 GiB 540 GiB 536 GiB 77 MiB 3.7 GiB 392 GiB 57.96 1.26 111 up 20 hdd 0.89000 1.00000 931 GiB 382 GiB 378 GiB 60 MiB 3.3 GiB 550 GiB 40.96 0.89 85 up 23 hdd 1.81929 1.00000 1.8 TiB 706 GiB 701 GiB 113 MiB 4.9 GiB 1.1 TiB 37.92 0.83 182 up 44 hdd 0.89000 1.00000 931 GiB 468 GiB 465 GiB 34 MiB 3.3 GiB 463 GiB 50.29 1.10 93 up 45 hdd 1.78999 1.00000 1.8 TiB 882 GiB 875 GiB 138 MiB 6.3 GiB 981 GiB 47.33 1.03 179 up 46 hdd 1.78999 1.00000 1.8 TiB 910 GiB 903 GiB 127 MiB 6.4 GiB 953 GiB 48.83 1.06 192 up 22 ssd 0.11639 1.00000 119 GiB 15 GiB 357 MiB 12 GiB 2.8 GiB 104 GiB 12.61 0.27 315 up 12 hdd 0.89000 1.00000 931 GiB 499 GiB 494 GiB 64 MiB 4.5 GiB 432 GiB 53.57 1.17 116 up 13 hdd 0.89000 1.00000 931 GiB 536 GiB 532 GiB 48 MiB 4.4 GiB 395 GiB 57.59 1.26 109 up 30 hdd 0.89000 1.00000 931 GiB 510 GiB 506 GiB 33 MiB 3.9 GiB 421 GiB 54.80 1.19 100 up 32 hdd 0.89000 1.00000 931 GiB 495 GiB 491 GiB 56 MiB 4.1 GiB 436 GiB 53.17 1.16 101 up 33 hdd 0.89000 1.00000 931 GiB 333 GiB 330 GiB 56 MiB 3.1 GiB 598 GiB 35.80 0.78 82 up 15 ssd 0.11639 1.00000 119 GiB 14 GiB 336 MiB 11 GiB 2.9 GiB 105 GiB 12.13 0.26 305 up 17 hdd 0.89000 1.00000 931 GiB 577 GiB 573 GiB 77 MiB 4.4 GiB 354 GiB 61.99 1.35 97 up 18 hdd 0.89000 1.00000 931 GiB 413 GiB 409 GiB 70 MiB 4.0 GiB 518 GiB 44.34 0.97 95 up 19 hdd 1.81879 1.00000 1.8 TiB 895 GiB 889 GiB 144 MiB 5.6 GiB 967 GiB 48.06 1.05 184 up 21 hdd 0.89000 1.00000 931 GiB 360 GiB 357 GiB 60 MiB 3.4 GiB 570 GiB 38.72 0.84 100 up 31 hdd 0.90909 1.00000 931 GiB 508 GiB 505 GiB 80 MiB 3.5 GiB 423 GiB 54.58 1.19 102 up 25 ssd 0.11639 1.00000 119 GiB 14 GiB 339 MiB 11 GiB 2.7 GiB 105 GiB 11.86 0.26 310 up 8 hdd 0.89000 1.00000 931 GiB 359 GiB 356 GiB 72 MiB 3.1 GiB 572 GiB 38.55 0.84 80 up 9 hdd 0.89000 1.00000 931 GiB 376 GiB 373 GiB 42 MiB 3.0 GiB 555 GiB 40.39 0.88 87 up 24 hdd 0.89000 1.00000 931 GiB 342 GiB 339 GiB 70 MiB 2.8 GiB 590 GiB 36.69 0.80 78 up 26 hdd 1.78999 1.00000 1.8 TiB 921 GiB 915 GiB 129 MiB 6.1 GiB 942 GiB 49.45 1.08 177 up 27 hdd 1.78999 1.00000 1.8 TiB 891 GiB 885 GiB 125 MiB 5.7 GiB 972 GiB 47.82 1.04 208 up 35 hdd 1.81929 1.00000 1.8 TiB 819 GiB 814 GiB 110 MiB 5.3 GiB 1.0 TiB 43.99 0.96 184 up 29 ssd 0.11638 1.00000 119 GiB 15 GiB 339 MiB 11 GiB 2.9 GiB 105 GiB 12.25 0.27 311 up 14 hdd 1.81929 1.00000 1.8 TiB 889 GiB 884 GiB 1.9 MiB 4.3 GiB 974 GiB 47.70 1.04 162 up 28 hdd 1.81929 1.00000 1.8 TiB 609 GiB 606 GiB 3.4 MiB 2.9 GiB 1.2 TiB 32.67 0.71 169 up 36 hdd 1.36429 1.00000 1.4 TiB 591 GiB 589 GiB 2.0 MiB 2.9 GiB 806 GiB 42.34 0.92 135 up 37 hdd 1.81929 1.00000 1.8 TiB 840 GiB 836 GiB 2.5 MiB 4.1 GiB 1023 GiB 45.12 0.98 177 up 38 hdd 1.81929 1.00000 1.8 TiB 914 GiB 909 GiB 2.2 MiB 4.4 GiB 949 GiB 49.05 1.07 182 up 2 hdd 0.89000 1.00000 931 GiB 449 GiB 445 GiB 60 MiB 3.4 GiB 482 GiB 48.19 1.05 89 up 3 hdd 0.89000 1.00000 931 GiB 333 GiB 329 GiB 75 MiB 3.0 GiB 598 GiB 35.73 0.78 86 up 10 hdd 0.89000 1.00000 931 GiB 416 GiB 412 GiB 54 MiB 3.8 GiB 515 GiB 44.68 0.97 100 up 11 hdd 1.81879 1.00000 1.8 TiB 803 GiB 798 GiB 70 MiB 5.3 GiB 1.0 TiB 43.14 0.94 188 up 34 hdd 0.90909 1.00000 931 GiB 605 GiB 600 GiB 64 MiB 4.2 GiB 326 GiB 64.95 1.42 106 up 16 ssd 0.11639 1.00000 119 GiB 14 GiB 322 MiB 10 GiB 3.1 GiB 105 GiB 11.59 0.25 295 up TOTAL 46 TiB 21 TiB 21 TiB 59 GiB 165 GiB 25 TiB 45.89 MIN/MAX VAR: 0.25/1.42 STDDEV: 13.76 On 06.12.19 12:27, Aleksey Gutikov wrote:
On 6.12.19 13:29, Jochen Schulz wrote:
Hi!
We have a ceph cluster with 42 OSD in production as a server providing mainly home-directories of users. Ceph is 14.2.4 nautilus.
We have 3 pools. One images (for rbd images) a cephfs_metadata and a cephfs_data pool.
Our raw data is about 5.6T. All pools have replica size 3 and there are only very little snapshots in the rbd images pool, the cephfspool doesnt use snapshots.
How is it possible that the status tells us, that 21T/46T is used, because thats much more than 3 times the raw size.
Also, to make that more confusing, there as at least half of the cluster free, and we get pg backfill_toofull after we added some OSDs lately. The ceph dashboard tells aus the pool ist 82 % full and has only 4.5 T free.
The autoscale module seems to calculate the 20T times 3 for the space needed and thus has wrong numbers (see below).
Status of the cluster is added below too.
how can these size/capacity numbers be explained? and, would be there a recommendation to change something?
Thank you in advance!
best Jochen
# ceph -s
cluster: id: 2b16167f-3f33-4580-a0e9-7a71978f403d health: HEALTH_ERR Degraded data redundancy (low space): 1 pg backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 2 pools have too many placement groups
services: mon: 4 daemons, quorum jade,assam,matcha,jasmine (age 2d) mgr: earl(active, since 24h), standbys: assam mds: cephfs:1 {0=assam=up:active} 1 up:standby osd: 42 osds: 42 up (since 106m), 42 in (since 115m); 30 remapped pgs
data: pools: 3 pools, 2048 pgs objects: 29.80M objects, 5.6 TiB usage: 21 TiB used, 25 TiB / 46 TiB avail pgs: 1164396/89411013 objects misplaced (1.302%) 2018 active+clean 22 active+remapped+backfill_wait 7 active+remapped+backfilling 1 active+remapped+backfill_wait+backfill_toofull
io: client: 1.7 KiB/s rd, 516 KiB/s wr, 0 op/s rd, 28 op/s wr recovery: 9.2 MiB/s, 41 objects/s
# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 354.2G 3.0 46100G 0.0231 1.0 1024 32 warn cephfs_metadata 13260M 3.0 595.7G 0.0652 1.0 512 8 warn cephfs_data 20802G 3.0 46100G 1.3537 1.0 512 warn
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Please, provide output of ceph df and ceph osd df - that should explain both questions (21T and 82%).
Home directories probably means lots of small objects. Default minimum allocation size of BlueStore on HDD is 64 kiB, so there's a lot of overhead for everything smaller; Details: google bluestore min alloc size, can only be changed during OSD creation Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Dec 6, 2019 at 12:57 PM Jochen Schulz <schulz@math.uni-goettingen.de> wrote:
Hi!
Thank you! The output of both commands are below. I still dont understand why there are 21T used data (because 5.5T*3 = 16.5T != 21T) and why there seems to be only 4.5 T MAX AVAIL, but the osd output tells we have 25T free space.
$ sudo ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 45 TiB 24 TiB 21 TiB 21 TiB 46.33 ssd 596 GiB 524 GiB 1.7 GiB 72 GiB 12.09 TOTAL 46 TiB 25 TiB 21 TiB 21 TiB 45.89
POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL images 8 149 GiB 38.30k 354 GiB 2.52 4.5 TiB cephfs_data 9 5.5 TiB 26.61M 20 TiB 60.36 4.5 TiB cephfs_metadata 10 12 GiB 3.17M 13 GiB 2.57 164 GiB
$ sudo ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.89000 1.00000 931 GiB 456 GiB 453 GiB 136 MiB 3.5 GiB 475 GiB 49.01 1.07 103 up 1 hdd 0.89000 1.00000 931 GiB 495 GiB 491 GiB 100 MiB 3.9 GiB 436 GiB 53.14 1.16 89 up 4 hdd 0.89000 1.00000 931 GiB 345 GiB 342 GiB 108 MiB 2.9 GiB 586 GiB 37.05 0.81 87 up 5 hdd 0.89000 1.00000 931 GiB 521 GiB 517 GiB 108 MiB 4.1 GiB 410 GiB 55.96 1.22 98 up 6 hdd 0.89000 1.00000 931 GiB 367 GiB 364 GiB 95 MiB 3.2 GiB 564 GiB 39.44 0.86 95 up 7 hdd 0.89000 1.00000 931 GiB 540 GiB 536 GiB 77 MiB 3.7 GiB 392 GiB 57.96 1.26 111 up 20 hdd 0.89000 1.00000 931 GiB 382 GiB 378 GiB 60 MiB 3.3 GiB 550 GiB 40.96 0.89 85 up 23 hdd 1.81929 1.00000 1.8 TiB 706 GiB 701 GiB 113 MiB 4.9 GiB 1.1 TiB 37.92 0.83 182 up 44 hdd 0.89000 1.00000 931 GiB 468 GiB 465 GiB 34 MiB 3.3 GiB 463 GiB 50.29 1.10 93 up 45 hdd 1.78999 1.00000 1.8 TiB 882 GiB 875 GiB 138 MiB 6.3 GiB 981 GiB 47.33 1.03 179 up 46 hdd 1.78999 1.00000 1.8 TiB 910 GiB 903 GiB 127 MiB 6.4 GiB 953 GiB 48.83 1.06 192 up 22 ssd 0.11639 1.00000 119 GiB 15 GiB 357 MiB 12 GiB 2.8 GiB 104 GiB 12.61 0.27 315 up 12 hdd 0.89000 1.00000 931 GiB 499 GiB 494 GiB 64 MiB 4.5 GiB 432 GiB 53.57 1.17 116 up 13 hdd 0.89000 1.00000 931 GiB 536 GiB 532 GiB 48 MiB 4.4 GiB 395 GiB 57.59 1.26 109 up 30 hdd 0.89000 1.00000 931 GiB 510 GiB 506 GiB 33 MiB 3.9 GiB 421 GiB 54.80 1.19 100 up 32 hdd 0.89000 1.00000 931 GiB 495 GiB 491 GiB 56 MiB 4.1 GiB 436 GiB 53.17 1.16 101 up 33 hdd 0.89000 1.00000 931 GiB 333 GiB 330 GiB 56 MiB 3.1 GiB 598 GiB 35.80 0.78 82 up 15 ssd 0.11639 1.00000 119 GiB 14 GiB 336 MiB 11 GiB 2.9 GiB 105 GiB 12.13 0.26 305 up 17 hdd 0.89000 1.00000 931 GiB 577 GiB 573 GiB 77 MiB 4.4 GiB 354 GiB 61.99 1.35 97 up 18 hdd 0.89000 1.00000 931 GiB 413 GiB 409 GiB 70 MiB 4.0 GiB 518 GiB 44.34 0.97 95 up 19 hdd 1.81879 1.00000 1.8 TiB 895 GiB 889 GiB 144 MiB 5.6 GiB 967 GiB 48.06 1.05 184 up 21 hdd 0.89000 1.00000 931 GiB 360 GiB 357 GiB 60 MiB 3.4 GiB 570 GiB 38.72 0.84 100 up 31 hdd 0.90909 1.00000 931 GiB 508 GiB 505 GiB 80 MiB 3.5 GiB 423 GiB 54.58 1.19 102 up 25 ssd 0.11639 1.00000 119 GiB 14 GiB 339 MiB 11 GiB 2.7 GiB 105 GiB 11.86 0.26 310 up 8 hdd 0.89000 1.00000 931 GiB 359 GiB 356 GiB 72 MiB 3.1 GiB 572 GiB 38.55 0.84 80 up 9 hdd 0.89000 1.00000 931 GiB 376 GiB 373 GiB 42 MiB 3.0 GiB 555 GiB 40.39 0.88 87 up 24 hdd 0.89000 1.00000 931 GiB 342 GiB 339 GiB 70 MiB 2.8 GiB 590 GiB 36.69 0.80 78 up 26 hdd 1.78999 1.00000 1.8 TiB 921 GiB 915 GiB 129 MiB 6.1 GiB 942 GiB 49.45 1.08 177 up 27 hdd 1.78999 1.00000 1.8 TiB 891 GiB 885 GiB 125 MiB 5.7 GiB 972 GiB 47.82 1.04 208 up 35 hdd 1.81929 1.00000 1.8 TiB 819 GiB 814 GiB 110 MiB 5.3 GiB 1.0 TiB 43.99 0.96 184 up 29 ssd 0.11638 1.00000 119 GiB 15 GiB 339 MiB 11 GiB 2.9 GiB 105 GiB 12.25 0.27 311 up 14 hdd 1.81929 1.00000 1.8 TiB 889 GiB 884 GiB 1.9 MiB 4.3 GiB 974 GiB 47.70 1.04 162 up 28 hdd 1.81929 1.00000 1.8 TiB 609 GiB 606 GiB 3.4 MiB 2.9 GiB 1.2 TiB 32.67 0.71 169 up 36 hdd 1.36429 1.00000 1.4 TiB 591 GiB 589 GiB 2.0 MiB 2.9 GiB 806 GiB 42.34 0.92 135 up 37 hdd 1.81929 1.00000 1.8 TiB 840 GiB 836 GiB 2.5 MiB 4.1 GiB 1023 GiB 45.12 0.98 177 up 38 hdd 1.81929 1.00000 1.8 TiB 914 GiB 909 GiB 2.2 MiB 4.4 GiB 949 GiB 49.05 1.07 182 up 2 hdd 0.89000 1.00000 931 GiB 449 GiB 445 GiB 60 MiB 3.4 GiB 482 GiB 48.19 1.05 89 up 3 hdd 0.89000 1.00000 931 GiB 333 GiB 329 GiB 75 MiB 3.0 GiB 598 GiB 35.73 0.78 86 up 10 hdd 0.89000 1.00000 931 GiB 416 GiB 412 GiB 54 MiB 3.8 GiB 515 GiB 44.68 0.97 100 up 11 hdd 1.81879 1.00000 1.8 TiB 803 GiB 798 GiB 70 MiB 5.3 GiB 1.0 TiB 43.14 0.94 188 up 34 hdd 0.90909 1.00000 931 GiB 605 GiB 600 GiB 64 MiB 4.2 GiB 326 GiB 64.95 1.42 106 up 16 ssd 0.11639 1.00000 119 GiB 14 GiB 322 MiB 10 GiB 3.1 GiB 105 GiB 11.59 0.25 295 up TOTAL 46 TiB 21 TiB 21 TiB 59 GiB 165 GiB 25 TiB 45.89 MIN/MAX VAR: 0.25/1.42 STDDEV: 13.76
On 06.12.19 12:27, Aleksey Gutikov wrote:
On 6.12.19 13:29, Jochen Schulz wrote:
Hi!
We have a ceph cluster with 42 OSD in production as a server providing mainly home-directories of users. Ceph is 14.2.4 nautilus.
We have 3 pools. One images (for rbd images) a cephfs_metadata and a cephfs_data pool.
Our raw data is about 5.6T. All pools have replica size 3 and there are only very little snapshots in the rbd images pool, the cephfspool doesnt use snapshots.
How is it possible that the status tells us, that 21T/46T is used, because thats much more than 3 times the raw size.
Also, to make that more confusing, there as at least half of the cluster free, and we get pg backfill_toofull after we added some OSDs lately. The ceph dashboard tells aus the pool ist 82 % full and has only 4.5 T free.
The autoscale module seems to calculate the 20T times 3 for the space needed and thus has wrong numbers (see below).
Status of the cluster is added below too.
how can these size/capacity numbers be explained? and, would be there a recommendation to change something?
Thank you in advance!
best Jochen
# ceph -s
cluster: id: 2b16167f-3f33-4580-a0e9-7a71978f403d health: HEALTH_ERR Degraded data redundancy (low space): 1 pg backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 2 pools have too many placement groups
services: mon: 4 daemons, quorum jade,assam,matcha,jasmine (age 2d) mgr: earl(active, since 24h), standbys: assam mds: cephfs:1 {0=assam=up:active} 1 up:standby osd: 42 osds: 42 up (since 106m), 42 in (since 115m); 30 remapped pgs
data: pools: 3 pools, 2048 pgs objects: 29.80M objects, 5.6 TiB usage: 21 TiB used, 25 TiB / 46 TiB avail pgs: 1164396/89411013 objects misplaced (1.302%) 2018 active+clean 22 active+remapped+backfill_wait 7 active+remapped+backfilling 1 active+remapped+backfill_wait+backfill_toofull
io: client: 1.7 KiB/s rd, 516 KiB/s wr, 0 op/s rd, 28 op/s wr recovery: 9.2 MiB/s, 41 objects/s
# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 354.2G 3.0 46100G 0.0231 1.0 1024 32 warn cephfs_metadata 13260M 3.0 595.7G 0.0652 1.0 512 8 warn cephfs_data 20802G 3.0 46100G 1.3537 1.0 512 warn
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Please, provide output of ceph df and ceph osd df - that should explain both questions (21T and 82%).
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6.12.19 14:57, Jochen Schulz wrote:
Hi!
Thank you! The output of both commands are below. I still dont understand why there are 21T used data (because 5.5T*3 = 16.5T != 21T) and why there seems to be only 4.5 T MAX AVAIL, but the osd output tells we have 25T free space.
As I know MAX AVAIL is calculated with respect to mon_osd_full_ratio, max OSD %USE, WEIGHT, total weight of set of OSDs in pool and replication factor of pool. Max used hdd OSD is 34. So (0.95 - 0.65) * 931 / 3 * 1.3 ~= 4.5T 1.3 here state for avg osd size / 931 (you have 13*1.8T osd and other 24*931G) The logic behind is to estimate amount of data you can write into pool before max used OSD became full. Sorry, can't say anything precise about 82% and backfill_toofull pg while max used OSD is 65% used.
On 6.12.19 17:01, Aleksey Gutikov wrote:
On 6.12.19 14:57, Jochen Schulz wrote:
Hi!
Thank you! The output of both commands are below. I still dont understand why there are 21T used data (because 5.5T*3 = 16.5T != 21T) and why there seems to be only 4.5 T MAX AVAIL, but the osd output tells we have 25T free space.
As I know MAX AVAIL is calculated with respect to mon_osd_full_ratio, max OSD %USE, WEIGHT, total weight of set of OSDs in pool and replication factor of pool. Max used hdd OSD is 34. So (0.95 - 0.65) * 931 / 3 * 1.3 ~= 4.5T
(0.95 - 0.65) * 931 / 3 * 1.3 * 37 ~= 4.5T
1.3 here state for avg osd size / 931 (you have 13*1.8T osd and other 24*931G) The logic behind is to estimate amount of data you can write into pool before max used OSD became full.
Sorry, can't say anything precise about 82% and backfill_toofull pg while max used OSD is 65% used.
Hi!
Thank you! The output of both commands are below. I still dont understand why there are 21T used data (because 5.5T*3 = 16.5T != 21T) and why there seems to be only 4.5 T MAX AVAIL, but the osd output tells we have 25T free space.
As I know MAX AVAIL is calculated with respect to mon_osd_full_ratio, max OSD %USE, WEIGHT, total weight of set of OSDs in pool and replication factor of pool. Max used hdd OSD is 34. So (0.95 - 0.65) * 931 / 3 * 1.3 ~= 4.5T
(0.95 - 0.65) * 931 / 3 * 1.3 * 37 ~= 4.5T
1.3 here state for avg osd size / 931 (you have 13*1.8T osd and other 24*931G) The logic behind is to estimate amount of data you can write into pool before max used OSD became full.
Thanks for helping understanding these numbers! probably we then recreate the bluestore osds one by one with reduced min alloc size.
Glad to see I am not the only one with unexpected increased disk usage. I do have a case for a few months now where the reported size on disk is 10 times higher than it should be. Unfortunately no solution so far. Therefore, I am very curious whether the min alloc size will solve your problem I do not expect it to be the solution in my case. This is how it looks in my cluster: I've moved a 1TiB pool (3TiB raw use) from hdd osds (7) to newly added nvme osds (14). The hdd osds should be almost empty by now as just small pools reside on them. The pools on the hdd osds in sum store about 25GiB, which should use about 75GiB with a pool size of 3. Wal and db are on separate devices. However the outputs of ceph df and ceph osd df tell a different story: # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 19 TiB 18 TiB 775 GiB 782 GiB 3.98 # ceph osd df | egrep "(ID|hdd)" ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 8 hdd 2.72392 1.00000 2.8 TiB 111 GiB 10 GiB 111 KiB 1024 MiB 2.7 TiB 3.85 0.60 65 up 6 hdd 2.17914 1.00000 2.3 TiB 112 GiB 11 GiB 83 KiB 1024 MiB 2.2 TiB 4.82 0.75 58 up 3 hdd 2.72392 1.00000 2.8 TiB 114 GiB 13 GiB 71 KiB 1024 MiB 2.7 TiB 3.94 0.62 76 up 5 hdd 2.72392 1.00000 2.8 TiB 109 GiB 7.6 GiB 83 KiB 1024 MiB 2.7 TiB 3.76 0.59 63 up 4 hdd 2.72392 1.00000 2.8 TiB 112 GiB 11 GiB 55 KiB 1024 MiB 2.7 TiB 3.87 0.60 59 up 7 hdd 2.72392 1.00000 2.8 TiB 114 GiB 13 GiB 8 KiB 1024 MiB 2.7 TiB 3.93 0.61 66 up 2 hdd 2.72392 1.00000 2.8 TiB 111 GiB 9.9 GiB 78 KiB 1024 MiB 2.7 TiB 3.84 0.60 69 up The sum of "DATA" is 75,5GiB which is what I am expecting to be used by the pools. How come the sum of "RAW USE" is 783GiB? More than 10x the size of the stored data. On my nvme osds the "RAW USE" to "DATA" overhead is <1%: ceph osd df|egrep "(ID|nvme)" ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 nvme 2.61989 1.00000 2.6 TiB 181 GiB 180 GiB 31 KiB 1.0 GiB 2.4 TiB 6.74 1.05 12 up 1 nvme 2.61989 1.00000 2.6 TiB 151 GiB 150 GiB 39 KiB 1024 MiB 2.5 TiB 5.62 0.88 10 up 13 nvme 2.61989 1.00000 2.6 TiB 239 GiB 238 GiB 55 KiB 1.0 GiB 2.4 TiB 8.89 1.39 16 up -- truncated -- I am running ceph version 14.2.3 (0f776cf838a1ae3130b2b73dc26be9c95c6ccc39) nautilus (stable) which was upgraded recently from 13.2.1. Best Georg
Hi Georg, I suspect your db device size is around 100GiB size? And actual total hdd class size is rather 700 GiB (100 GiB * 7 osds) less than reported 19 TiB. Is the above correct? If so then high raw size(s) is caused by osd stats reporting design - it unconditionally includes full db volume size into reported total/used space. Hence one gets both "ceph df's" RAW STORAGE SIZE and "ceph osd df's" SIZE/RAW USE numbers increased. You might want to inspect per-pool usage sizes via "ceph df detail" command and highly likely they will show expected numbers. Thanks, Igor On 12/10/2019 11:39 AM, Georg F wrote:
Glad to see I am not the only one with unexpected increased disk usage. I do have a case for a few months now where the reported size on disk is 10 times higher than it should be. Unfortunately no solution so far. Therefore, I am very curious whether the min alloc size will solve your problem I do not expect it to be the solution in my case.
This is how it looks in my cluster:
I've moved a 1TiB pool (3TiB raw use) from hdd osds (7) to newly added nvme osds (14). The hdd osds should be almost empty by now as just small pools reside on them. The pools on the hdd osds in sum store about 25GiB, which should use about 75GiB with a pool size of 3. Wal and db are on separate devices.
However the outputs of ceph df and ceph osd df tell a different story:
# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 19 TiB 18 TiB 775 GiB 782 GiB 3.98
# ceph osd df | egrep "(ID|hdd)" ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 8 hdd 2.72392 1.00000 2.8 TiB 111 GiB 10 GiB 111 KiB 1024 MiB 2.7 TiB 3.85 0.60 65 up 6 hdd 2.17914 1.00000 2.3 TiB 112 GiB 11 GiB 83 KiB 1024 MiB 2.2 TiB 4.82 0.75 58 up 3 hdd 2.72392 1.00000 2.8 TiB 114 GiB 13 GiB 71 KiB 1024 MiB 2.7 TiB 3.94 0.62 76 up 5 hdd 2.72392 1.00000 2.8 TiB 109 GiB 7.6 GiB 83 KiB 1024 MiB 2.7 TiB 3.76 0.59 63 up 4 hdd 2.72392 1.00000 2.8 TiB 112 GiB 11 GiB 55 KiB 1024 MiB 2.7 TiB 3.87 0.60 59 up 7 hdd 2.72392 1.00000 2.8 TiB 114 GiB 13 GiB 8 KiB 1024 MiB 2.7 TiB 3.93 0.61 66 up 2 hdd 2.72392 1.00000 2.8 TiB 111 GiB 9.9 GiB 78 KiB 1024 MiB 2.7 TiB 3.84 0.60 69 up
The sum of "DATA" is 75,5GiB which is what I am expecting to be used by the pools. How come the sum of "RAW USE" is 783GiB? More than 10x the size of the stored data. On my nvme osds the "RAW USE" to "DATA" overhead is <1%:
ceph osd df|egrep "(ID|nvme)" ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 nvme 2.61989 1.00000 2.6 TiB 181 GiB 180 GiB 31 KiB 1.0 GiB 2.4 TiB 6.74 1.05 12 up 1 nvme 2.61989 1.00000 2.6 TiB 151 GiB 150 GiB 39 KiB 1024 MiB 2.5 TiB 5.62 0.88 10 up 13 nvme 2.61989 1.00000 2.6 TiB 239 GiB 238 GiB 55 KiB 1.0 GiB 2.4 TiB 8.89 1.39 16 up -- truncated --
I am running ceph version 14.2.3 (0f776cf838a1ae3130b2b73dc26be9c95c6ccc39) nautilus (stable) which was upgraded recently from 13.2.1.
Best Georg _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Igor, you're right indeed, the db volume is 100G in size for the hdd osds. Knowing this, the actual raw use is 783G - 7x100G = 83G which is pretty close to the sum of the files in the HDD pools times the pool size which is roughly 25x3=75G. Thanks a lot for your explanation of this tiny but highly confusing detail. Best Georg Igor Fedotov wrote:
Hi Georg,
I suspect your db device size is around 100GiB size? And actual total hdd class size is rather 700 GiB (100 GiB * 7 osds) less than reported 19 TiB.
Is the above correct? If so then high raw size(s) is caused by osd stats reporting design - it unconditionally includes full db volume size into reported total/used space.
Hence one gets both "ceph df's" RAW STORAGE SIZE and "ceph osd df's" SIZE/RAW USE numbers increased.
You might want to inspect per-pool usage sizes via "ceph df detail" command and highly likely they will show expected numbers.
Thanks,
Igor
participants (5)
-
Aleksey Gutikov
-
Georg F
-
Igor Fedotov
-
Jochen Schulz
-
Paul Emmerich