Your pg_num is fine, there's no reason to change it if you don't encounter any issues. One could argue that your smaller OSDs have too few PGs but the larger OSDs have reasonable values. I would probably leave it as it is. Regarding the inactive PGs, how are your pools configured? Can you share ceph osd pool ls detail It could be an issue with min_size (is it also set to 3?). Zitat von Lindsay Mathieson <lindsay.mathieson@gmail.com>:

Nautilus 14.2.9, setup using Proxmox.

* 5 Hosts * 18 OSDs with a mix of disk sizes (3TB, 1TB, 500GB), all bluestore * Pool size = 3, pg_num = 512

According to:

https://docs.ceph.com/docs/nautilus/rados/operations/placement-groups/#prese...

With 18 OSD's I should be using pg_num=1024, but I actually have it set to 512.

However autoscale is recommending pg_num=128

Additionally, I accidentally set autoscale to on, rather than warn, so it started the process. I rapidly got a "Reduced data availability: 2 pgs inactive" warning and io on the pool stopped. I cleared the warning by restarting the effected OSD's for the pg id, but then more cropped up. I only made it stop and restored access to the pool by turning off autoscale and setting pg_num back to 512.

Autoscale warning:

POOL   SIZE TARGET SIZE RATE RAW CAPACITY  RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE ceph  3239G              3.0       31205G 0.3114               1.0    512        128 warn

osd tree

ID  CLASS WEIGHT   REWEIGHT SIZE    RAW USE DATA    OMAP META     AVAIL   %USE  VAR  PGS STATUS TYPE NAME  -1       30.47374        -  30 TiB 7.4 TiB 7.4 TiB  28 MiB   23 GiB  23 TiB 24.40 1.00   -        root default  -5        5.45798        - 5.5 TiB 1.4 TiB 1.3 TiB 4.4 MiB  3.4 GiB 4.1 TiB 24.78 1.02   -            host loc   1   hdd  2.72899  0.95001 2.7 TiB 679 GiB 677 GiB 3.1 MiB  1.9 GiB 2.1 TiB 24.31 1.00 136     up         osd.1   8   hdd  2.72899  1.00000 2.7 TiB 706 GiB 704 GiB 1.3 MiB  1.5 GiB 2.0 TiB 25.26 1.04 143     up         osd.8  -9        1.36449        - 1.4 TiB 431 GiB 429 GiB 1.4 MiB  2.1 GiB 966 GiB 30.84 1.26   -            host lod   5   hdd  0.90970  1.00000 932 GiB 291 GiB 290 GiB 1.1 MiB  1.1 GiB 641 GiB 31.20 1.28  59     up         osd.5  12   hdd  0.45479  1.00000 466 GiB 140 GiB 139 GiB 293 KiB 1024 MiB 325 GiB 30.12 1.23  28     up         osd.12 -11       10.91595        -  11 TiB 2.4 TiB 2.4 TiB 6.9 MiB  5.8 GiB 8.5 TiB 21.94 0.90   -            host vnb   6   hdd  2.72899  1.00000 2.7 TiB 613 GiB 612 GiB 2.5 MiB  1.5 GiB 2.1 TiB 21.95 0.90 124     up         osd.6   7   hdd  2.72899  1.00000 2.7 TiB 614 GiB 613 GiB 1.7 MiB  1.4 GiB 2.1 TiB 21.98 0.90 124     up         osd.7   9   hdd  2.72899  1.00000 2.7 TiB 617 GiB 615 GiB 1.5 MiB  1.4 GiB 2.1 TiB 22.06 0.90 124     up         osd.9  17   hdd  2.72899  1.00000 2.7 TiB 608 GiB 607 GiB 1.3 MiB  1.5 GiB 2.1 TiB 21.76 0.89 124     up         osd.17  -3        4.54836        - 4.5 TiB 1.3 TiB 1.3 TiB  11 MiB  7.0 GiB 3.3 TiB 27.76 1.14   -            host vnh   0   hdd  0.90970  0.95001 932 GiB 220 GiB 219 GiB 3.0 MiB 1021 MiB 711 GiB 23.64 0.97  44     up         osd.0   2   hdd  0.90970  0.95001 932 GiB 252 GiB 251 GiB 536 KiB 1023 MiB 679 GiB 27.06 1.11  51     up         osd.2  10   hdd  0.54579  1.00000 559 GiB 158 GiB 157 GiB 1.6 MiB 1022 MiB 401 GiB 28.29 1.16  32     up         osd.10  11   hdd  0.54579  1.00000 559 GiB 157 GiB 156 GiB 332 KiB 1024 MiB 402 GiB 28.10 1.15  32     up         osd.11  14   hdd  0.54579  1.00000 559 GiB 187 GiB 186 GiB 1.1 MiB  1.0 GiB 372 GiB 33.45 1.37  38     up         osd.14  15   hdd  0.54579  1.00000 559 GiB 159 GiB 158 GiB 2.0 MiB 1022 MiB 400 GiB 28.51 1.17  32     up         osd.15  16   hdd  0.54579  1.00000 559 GiB 159 GiB 158 GiB 2.8 MiB 1021 MiB 400 GiB 28.46 1.17  32     up         osd.16  -7        8.18697        - 8.2 TiB 2.0 TiB 2.0 TiB 4.6 MiB  4.5 GiB 6.2 TiB 24.50 1.00   -            host vni   3   hdd  2.72899  1.00000 2.7 TiB 670 GiB 669 GiB 1.2 MiB  1.4 GiB 2.1 TiB 23.99 0.98 134     up         osd.3   4   hdd  2.72899  1.00000 2.7 TiB 681 GiB 679 GiB 1.6 MiB  1.6 GiB 2.1 TiB 24.36 1.00 136     up         osd.4  13   hdd  2.72899  1.00000 2.7 TiB 703 GiB 701 GiB 1.8 MiB  1.5 GiB 2.0 TiB 25.14 1.03 143     up         osd.13                       TOTAL  30 TiB 7.4 TiB 7.4 TiB  28 MiB   23 GiB  23 TiB 24.40

Should I be reducing the pg_num? is there a way to do it safely?

Thanks.

-- Lindsay

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io