+ceph-users On Mon, 6 Jan 2025 at 15:50, Bruno Gomes Pessanha <bruno.pessanha@gmail.com> wrote:

`ceph osd df`, look at the PGS column. Could be you’re hitting the limit

...

on some OSDs. It’s odd to stop at a non power of 2.

# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 8 ssd 13.97069 1.00000 14 TiB 9.4 TiB 9.4 TiB 9.8 MiB 21 GiB 4.6 TiB 67.09 1.07 63 up 18 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 8.5 MiB 19 GiB 5.7 TiB 59.02 0.94 54 up 28 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 1.5 GiB 19 GiB 5.9 TiB 58.02 0.93 54 up 37 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 84 MiB 31 GiB 1.2 TiB 91.14 1.45 135 up 45 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 542 MiB 22 GiB 4.8 TiB 65.79 1.05 66 up 56 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 5.0 MiB 19 GiB 5.7 TiB 59.19 0.94 56 up 66 ssd 13.97069 1.00000 14 TiB 8.9 TiB 8.8 TiB 358 MiB 20 GiB 5.1 TiB 63.39 1.01 54 up 77 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 3.7 MiB 19 GiB 5.3 TiB 62.18 0.99 53 up 6 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 355 MiB 18 GiB 5.6 TiB 59.67 0.95 52 up 16 ssd 13.97069 1.00000 14 TiB 7.3 TiB 7.3 TiB 572 MiB 18 GiB 6.7 TiB 52.05 0.83 47 up 26 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 921 MiB 20 GiB 5.4 TiB 61.55 0.98 57 up 36 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1.2 GiB 31 GiB 1.2 TiB 91.45 1.46 141 up 46 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 419 MiB 19 GiB 5.9 TiB 57.77 0.92 57 up 54 ssd 13.97069 1.00000 14 TiB 9.0 TiB 8.9 TiB 1011 MiB 20 GiB 5.0 TiB 64.10 1.02 65 up 64 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.9 TiB 962 MiB 19 GiB 6.0 TiB 56.87 0.91 59 up 74 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 879 MiB 17 GiB 6.8 TiB 51.23 0.82 49 up 3 ssd 13.97069 1.00000 14 TiB 9.1 TiB 9.1 TiB 2.0 GiB 20 GiB 4.8 TiB 65.45 1.04 71 up 12 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 433 MiB 19 GiB 5.6 TiB 59.92 0.96 61 up 22 ssd 13.97069 1.00000 14 TiB 8.8 TiB 8.8 TiB 501 MiB 20 GiB 5.2 TiB 63.01 1.01 66 up 31 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 932 MiB 18 GiB 5.8 TiB 58.20 0.93 53 up 41 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.6 TiB 627 MiB 20 GiB 5.3 TiB 61.96 0.99 54 up 51 ssd 13.97069 1.00000 14 TiB 6.9 TiB 6.9 TiB 1.4 GiB 16 GiB 7.1 TiB 49.18 0.78 43 up 61 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1000 MiB 31 GiB 1.4 TiB 89.99 1.44 134 up 71 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.0 TiB 553 MiB 19 GiB 5.9 TiB 57.63 0.92 51 up 4 ssd 13.97069 1.00000 14 TiB 7.1 TiB 7.1 TiB 525 MiB 15 GiB 6.9 TiB 50.87 0.81 47 up 15 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 606 MiB 19 GiB 5.2 TiB 62.55 1.00 55 up 23 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.8 TiB 423 MiB 18 GiB 6.1 TiB 56.32 0.90 49 up 34 ssd 13.97069 1.00000 14 TiB 10 TiB 10 TiB 551 MiB 24 GiB 3.5 TiB 74.79 1.19 72 up 44 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 77 MiB 28 GiB 1.4 TiB 89.90 1.43 132 up 59 ssd 13.97069 1.00000 14 TiB 9.1 TiB 9.0 TiB 510 MiB 20 GiB 4.9 TiB 64.91 1.04 61 up 69 ssd 13.97069 1.00000 14 TiB 9.7 TiB 9.7 TiB 1.0 GiB 22 GiB 4.2 TiB 69.62 1.11 71 up 79 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 1.0 GiB 17 GiB 6.8 TiB 51.26 0.82 47 up 9 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 1.1 MiB 18 GiB 5.8 TiB 58.81 0.94 49 up 19 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 421 MiB 21 GiB 4.8 TiB 65.68 1.05 63 up 29 ssd 13.97069 1.00000 14 TiB 8.8 TiB 8.7 TiB 511 MiB 21 GiB 5.2 TiB 62.68 1.00 61 up 39 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 556 MiB 20 GiB 5.6 TiB 60.15 0.96 58 up 49 ssd 13.97069 1.00000 14 TiB 7.0 TiB 7.0 TiB 2.9 MiB 15 GiB 7.0 TiB 49.92 0.80 45 up 58 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 432 MiB 16 GiB 6.8 TiB 51.22 0.82 49 up 67 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 1.1 GiB 17 GiB 6.6 TiB 52.80 0.84 47 up 78 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 574 MiB 19 GiB 6.3 TiB 54.94 0.88 55 up 0 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 455 MiB 19 GiB 5.8 TiB 58.66 0.94 58 up 10 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 439 MiB 19 GiB 5.4 TiB 61.07 0.97 48 up 21 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 646 MiB 30 GiB 1.3 TiB 90.87 1.45 135 up 32 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 3.2 MiB 19 GiB 5.4 TiB 61.60 0.98 56 up 42 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 582 MiB 33 GiB 1.3 TiB 90.78 1.45 133 up 52 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 1.4 GiB 17 GiB 6.6 TiB 52.74 0.84 47 up 62 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.3 TiB 1.5 GiB 16 GiB 6.6 TiB 52.68 0.84 49 up 72 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 1.2 GiB 17 GiB 6.3 TiB 55.20 0.88 53 up 1 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.0 TiB 1.5 GiB 19 GiB 5.9 TiB 57.73 0.92 64 up 11 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 422 MiB 19 GiB 5.5 TiB 60.37 0.96 62 up 20 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 405 MiB 17 GiB 6.3 TiB 55.02 0.88 52 up 30 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 494 MiB 19 GiB 5.9 TiB 57.84 0.92 49 up 40 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 376 MiB 22 GiB 4.7 TiB 66.17 1.06 74 up 50 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 1.0 GiB 20 GiB 5.7 TiB 59.43 0.95 54 up 60 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 2.1 GiB 32 GiB 1.2 TiB 91.27 1.46 136 up 70 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 498 MiB 19 GiB 5.6 TiB 59.64 0.95 56 up 2 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1.1 GiB 31 GiB 1.2 TiB 91.62 1.46 133 up 14 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 9.8 MiB 22 GiB 4.7 TiB 66.05 1.05 60 up 24 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 426 MiB 17 GiB 6.5 TiB 53.13 0.85 50 up 33 ssd 13.97069 1.00000 14 TiB 8.9 TiB 8.9 TiB 413 MiB 20 GiB 5.1 TiB 63.54 1.01 56 up 43 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 529 MiB 18 GiB 5.8 TiB 58.58 0.93 58 up 53 ssd 13.97069 1.00000 14 TiB 7.6 TiB 7.6 TiB 533 MiB 17 GiB 6.4 TiB 54.29 0.87 52 up 63 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 569 MiB 19 GiB 5.4 TiB 61.20 0.98 58 up 73 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 536 MiB 20 GiB 5.3 TiB 62.14 0.99 63 up 5 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 1.3 GiB 19 GiB 5.7 TiB 59.46 0.95 53 up 13 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 923 MiB 21 GiB 5.4 TiB 61.60 0.98 49 up 25 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 4.3 MiB 16 GiB 6.6 TiB 53.03 0.85 46 up 35 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.3 TiB 397 MiB 17 GiB 6.6 TiB 52.70 0.84 45 up 47 ssd 13.97069 1.00000 14 TiB 9.0 TiB 9.0 TiB 10 MiB 21 GiB 4.9 TiB 64.74 1.03 60 up 57 ssd 13.97069 1.00000 14 TiB 7.1 TiB 7.1 TiB 418 MiB 17 GiB 6.8 TiB 51.13 0.82 50 up 68 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 590 MiB 30 GiB 1.2 TiB 91.28 1.46 141 up 76 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.2 TiB 976 MiB 16 GiB 6.7 TiB 51.70 0.82 46 up 7 ssd 13.97069 1.00000 14 TiB 7.8 TiB 7.8 TiB 2.6 MiB 18 GiB 6.1 TiB 56.03 0.89 49 up 17 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.2 TiB 443 MiB 16 GiB 6.8 TiB 51.34 0.82 43 up 27 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.4 TiB 8.6 MiB 21 GiB 5.5 TiB 60.51 0.97 54 up 38 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.9 TiB 1005 MiB 18 GiB 6.0 TiB 56.88 0.91 52 up 48 ssd 13.97069 1.00000 14 TiB 7.6 TiB 7.5 TiB 2.0 MiB 17 GiB 6.4 TiB 54.08 0.86 45 up 55 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 509 MiB 30 GiB 1.2 TiB 91.50 1.46 137 up 65 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 901 MiB 17 GiB 6.3 TiB 55.07 0.88 51 up 75 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 6.0 MiB 19 GiB 5.5 TiB 60.66 0.97 58 up TOTAL 1.1 PiB 701 TiB 699 TiB 49 GiB 1.6 TiB 417 TiB 62.70 MIN/MAX VAR: 0.78/1.46 STDDEV: 11.76

We’re a community! That curve is way gentler than it used to be.

...

🙌👍

On Sun, 5 Jan 2025 at 14:59, Anthony D'Atri <anthony.datri@gmail.com> wrote:

...

...

...

Do you use the autoscalar or did you trigger a manual PG increment of the pool?

The pool had autoscale enabled until 2 days ago when I thought it was better to change things manually in order to have a more deterministic result. Yes, I wanted to increase from "1" to something like "1024" but it looks like it was capped to the 144 no matter what I do:

`ceph osd df`, look at the PGS column. Could be you’re hitting the limit on some OSDs. It’s odd to stop at a non power of 2.

...

Is it correct to say that every PG/OSD change can potentially cause data misplacements, unbalanced osd's and long backfills? I'll be way more careful before tuning it if that's the case.

The autoscaler will usually only bump pg_num for a pool when the value is <half what it thinks it should be.

I suggest setting the ‘bulk’ flag one pool at a time to effectively pre-split PGs as if the pool were full of data already. Ignore .mgr. Start with the pools with the fewest , let the cluster settle between each adjustment. That way the autoscaler will only make changes if cluster topology changes.

...

Thank you both so much! It definitely helped me to understand Ceph better. It is kind of a steep curve :).

We’re a community! That curve is way gentler than it used to be.

-- Bruno Gomes Pessanha

-- Bruno Gomes Pessanha