Hi, with EC you need to take into account the total number of chunks for PG calculation. Given your 16348 target, devide that number by 12 (if I'm reading your crush rule correctly) results in 1362, this would be either 1024 or 2048 PGs for that pool. By the way, I wouldn't recommend to use all 12 hosts for your EC profile since that would mean a degraded cluster as soon as one host fails. You should consider one or two spare nodes so the PGs can be recovered in case one node fails. Regards, Eugen Zitat von Gunnar Bandelow <gunnar.bandelow@uni-greifswald.de>:

Hello Everyone,

i've run into problems with placement groups.

We have a 12 Host Ceph-Cluster with 408 OSDs (hdd and ssd).

I create a replicated pool with a large pg_num (16384). No problems everything works.

If I do this with an erasure pool, i get a warning which is fixable by extending the max_pgs_per_osd and afterwards i get a health_warn in the ceph status due too few PGs.

Checking the pg_num after pool creation, it is capped at 1024.

I'm stuck at this point. Maybe i did something fundamentally wrong?

To illustrate my steps I tried to summarize everything in a small example:

# ceph -v ceph version 14.2.7 (fb8e34a687d76cd3bd45c2a0fb445432ab69b4ff) nautilus (stable)

# ceph osd erasure-code-profile get myerasurehdd crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=7 m=5 plugin=jerasure technique=reed_sol_van w=8

# ceph osd crush rule dump sas_rule {     "rule_id": 0,     "rule_name": "sas_rule",     "ruleset": 0,     "type": 3,     "min_size": 1,     "max_size": 12,     "steps": [         {             "op": "take",             "item": -2,             "item_name": "default~hdd"         },         {             "op": "chooseleaf_firstn",             "num": 0,             "type": "rack"         },         {             "op": "emit"         }     ] }

# ceph osd pool create sas-pool 16384 16384 erasure myerasurehdd sas_rule Error ERANGE:  pg_num 16384 size 12 would mean 196704 total pgs, which exceeds max 102000 (mon_max_pg_per_osd 250 * num_in_osds 408)

# ceph tell mon.\* injectargs '--mon-max-pg-per-osd=500' mon.ceph-fs01: injectargs:mon_max_pg_per_osd = '500' (not observed, change may require restart) mon.ceph-fs05: injectargs:mon_max_pg_per_osd = '500' (not observed, change may require restart) mon.ceph-fs09: injectargs:mon_max_pg_per_osd = '500' (not observed, change may require restart) ceph-fs01:/opt/ceph-setup# ceph osd pool create sas-pool 16384 16384 erasure myerasurehdd sas_rule pool 'sas-pool' created

# ceph -s

cluster:     id:     b9471b57-95a2-4e58-8f69-b5e6048bea7c     health: HEALTH_WARN             Reduced data availability: 1024 pgs incomplete             too few PGs per OSD (7 < min 30)

# ceph osd pool get sas-pool pg_num pg_num: 1024

Best regards,

Gunnar