Hi, We have use case where we had like to restrict some pools to a subset of the OSDs located in a particular section of the crush map hierarchy (OSDs backed up by UPS/Diesel). We tried to define for these (replica 3) pools a specific crush rule with the root paramater defined to a specific row (which contains 3 OSD servers with #10 OSD each). At the beginning it worked but after some time (probably after doing a reweight on the OSDs in this row to reduce the number of PGs from other pools), a few PGs are active+clean+remapped and 1 is undersized. 'ceph osd pg dump|grep remapped' gives an output similar to the following one for each remapped PG: 20.1ae 648 0 0 648 0 374416846 0 0 438 1404 438 active+clean+remapped 2025-04-16T07:19:40.507778+0000 43117'1018433 48443:1131738 [70,58] 70 [70,58,45] 70 43117'1018433 2025-04-16T07:19:40.507443+0000 43117'1018433 2025-04-16T07:19:40.507443+0000 0 15 periodic scrub scheduled @ 2025-04-17T19:18:23.470846+0000 648 0 We can see that we currently have 3 replica but that Ceph would like to move to 2... (the undersized PG has currently only 2 replica for an unknown reason, probably the same). Is it wrong trying to do what we did, i.e. using a row for the crush rule root parameter? If not, where could we find more information about the cause? Thanks in advance for any help. Best regards, Michel --------------------- Crush rule used ----------------- { "rule_id": 2, "rule_name": "ha-replicated_ruleset", "type": 1, "steps": [ { "op": "take", "item": -22, "item_name": "row-01~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ------------------- Beginning of the CRUSH tree ------------------- ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 843.57141 root default -19 843.57141 datacenter bat.206 -21 283.81818 row row-01 -15 87.32867 host cephdevel-76079 1 hdd 7.27739 osd.1 up 0.50000 1.00000 2 hdd 7.27739 osd.2 up 0.50000 1.00000 14 hdd 7.27739 osd.14 up 0.50000 1.00000 39 hdd 7.27739 osd.39 up 0.50000 1.00000 40 hdd 7.27739 osd.40 up 0.50000 1.00000 41 hdd 7.27739 osd.41 up 0.50000 1.00000 42 hdd 7.27739 osd.42 up 0.50000 1.00000 43 hdd 7.27739 osd.43 up 0.50000 1.00000 44 hdd 7.27739 osd.44 up 0.50000 1.00000 45 hdd 7.27739 osd.45 up 0.50000 1.00000 46 hdd 7.27739 osd.46 up 0.50000 1.00000 47 hdd 7.27739 osd.47 up 0.50000 1.00000 -3 94.60606 host cephdevel-76154 49 hdd 7.27739 osd.49 up 0.50000 1.00000 50 hdd 7.27739 osd.50 up 0.50000 1.00000 51 hdd 7.27739 osd.51 up 0.50000 1.00000 66 hdd 7.27739 osd.66 up 0.50000 1.00000 67 hdd 7.27739 osd.67 up 0.50000 1.00000 68 hdd 7.27739 osd.68 up 0.50000 1.00000 69 hdd 7.27739 osd.69 up 0.50000 1.00000 70 hdd 7.27739 osd.70 up 0.50000 1.00000 71 hdd 7.27739 osd.71 up 0.50000 1.00000 72 hdd 7.27739 osd.72 up 0.50000 1.00000 73 hdd 7.27739 osd.73 up 0.50000 1.00000 74 hdd 7.27739 osd.74 up 0.50000 1.00000 75 hdd 7.27739 osd.75 up 0.50000 1.00000 -4 101.88345 host cephdevel-76204 48 hdd 7.27739 osd.48 up 0.50000 1.00000 52 hdd 7.27739 osd.52 up 0.50000 1.00000 53 hdd 7.27739 osd.53 up 0.50000 1.00000 54 hdd 7.27739 osd.54 up 0.50000 1.00000 56 hdd 7.27739 osd.56 up 0.50000 1.00000 57 hdd 7.27739 osd.57 up 0.50000 1.00000 58 hdd 7.27739 osd.58 up 0.50000 1.00000 59 hdd 7.27739 osd.59 up 0.50000 1.00000 60 hdd 7.27739 osd.60 up 0.50000 1.00000 61 hdd 7.27739 osd.61 up 0.50000 1.00000 62 hdd 7.27739 osd.62 up 0.50000 1.00000 63 hdd 7.27739 osd.63 up 0.50000 1.00000 64 hdd 7.27739 osd.64 up 0.50000 1.00000 65 hdd 7.27739 osd.65 up 0.50000 1.00000 -23 203.16110 row row-02 -13 87.32867 host cephdevel-76213 27 hdd 7.27739 osd.27 up 1.00000 1.00000 28 hdd 7.27739 osd.28 up 1.00000 1.00000 29 hdd 7.27739 osd.29 up 1.00000 1.00000 30 hdd 7.27739 osd.30 up 1.00000 1.00000 31 hdd 7.27739 osd.31 up 1.00000 1.00000 32 hdd 7.27739 osd.32 up 1.00000 1.00000 33 hdd 7.27739 osd.33 up 1.00000 1.00000 34 hdd 7.27739 osd.34 up 1.00000 1.00000 35 hdd 7.27739 osd.35 up 1.00000 1.00000 36 hdd 7.27739 osd.36 up 1.00000 1.00000 37 hdd 7.27739 osd.37 up 1.00000 1.00000 38 hdd 7.27739 osd.38 up 1.00000 1.00000 ......