Hi Anthony, No we use the same mons (that are also backed up by UPS/Diesel). The idea doing this was to allow sharing the OSD between the pool using this "critical area" (thus OSD located only in this row) and the other normal pools, to avoid dedicated potentially a large storage volume to this critical area that doesn't require much. Thus also the choice to reweight the OSDs in this row so that they are less used than other OSDs by normal pools to avoid exploding the number of PGs on these OSDs. I am not sure that we can use a custom device class to achieve what we had in mind as this will not allow to share an OSD between critical and non critical pools. But it may be a better way in fact, dedicated only a fraction of the OSDs on each server in the "critical row" to these pools and using other OSDs on these servers for normal pools without any reweighting. Thanks for the idea. We may also try to bump the number of retries to see if it has an effect. Best regards, Michel Le 16/04/2025 à 13:16, Anthony D'Atri a écrit :
First time I recall anyone trying this. Thoughts:
* Manually edit the crush map and bump retries from 50 to 100 * Better yet, give those OSDs a custom device class and change the CRUSH rule to use that and the default root.
Do you also constrain mons to those systems ?
On Apr 16, 2025, at 6:41 AM, Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:
Hi,
We have use case where we had like to restrict some pools to a subset of the OSDs located in a particular section of the crush map hierarchy (OSDs backed up by UPS/Diesel). We tried to define for these (replica 3) pools a specific crush rule with the root paramater defined to a specific row (which contains 3 OSD servers with #10 OSD each). At the beginning it worked but after some time (probably after doing a reweight on the OSDs in this row to reduce the number of PGs from other pools), a few PGs are active+clean+remapped and 1 is undersized.
'ceph osd pg dump|grep remapped' gives an output similar to the following one for each remapped PG:
20.1ae 648 0 0 648 0 374416846 0 0 438 1404 438 active+clean+remapped 2025-04-16T07:19:40.507778+0000 43117'1018433 48443:1131738 [70,58] 70 [70,58,45] 70 43117'1018433 2025-04-16T07:19:40.507443+0000 43117'1018433 2025-04-16T07:19:40.507443+0000 0 15 periodic scrub scheduled @ 2025-04-17T19:18:23.470846+0000 648 0
We can see that we currently have 3 replica but that Ceph would like to move to 2... (the undersized PG has currently only 2 replica for an unknown reason, probably the same).
Is it wrong trying to do what we did, i.e. using a row for the crush rule root parameter? If not, where could we find more information about the cause?
Thanks in advance for any help. Best regards,
Michel
--------------------- Crush rule used -----------------
{ "rule_id": 2, "rule_name": "ha-replicated_ruleset", "type": 1, "steps": [ { "op": "take", "item": -22, "item_name": "row-01~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }
------------------- Beginning of the CRUSH tree -------------------
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 843.57141 root default -19 843.57141 datacenter bat.206 -21 283.81818 row row-01 -15 87.32867 host cephdevel-76079 1 hdd 7.27739 osd.1 up 0.50000 1.00000 2 hdd 7.27739 osd.2 up 0.50000 1.00000 14 hdd 7.27739 osd.14 up 0.50000 1.00000 39 hdd 7.27739 osd.39 up 0.50000 1.00000 40 hdd 7.27739 osd.40 up 0.50000 1.00000 41 hdd 7.27739 osd.41 up 0.50000 1.00000 42 hdd 7.27739 osd.42 up 0.50000 1.00000 43 hdd 7.27739 osd.43 up 0.50000 1.00000 44 hdd 7.27739 osd.44 up 0.50000 1.00000 45 hdd 7.27739 osd.45 up 0.50000 1.00000 46 hdd 7.27739 osd.46 up 0.50000 1.00000 47 hdd 7.27739 osd.47 up 0.50000 1.00000 -3 94.60606 host cephdevel-76154 49 hdd 7.27739 osd.49 up 0.50000 1.00000 50 hdd 7.27739 osd.50 up 0.50000 1.00000 51 hdd 7.27739 osd.51 up 0.50000 1.00000 66 hdd 7.27739 osd.66 up 0.50000 1.00000 67 hdd 7.27739 osd.67 up 0.50000 1.00000 68 hdd 7.27739 osd.68 up 0.50000 1.00000 69 hdd 7.27739 osd.69 up 0.50000 1.00000 70 hdd 7.27739 osd.70 up 0.50000 1.00000 71 hdd 7.27739 osd.71 up 0.50000 1.00000 72 hdd 7.27739 osd.72 up 0.50000 1.00000 73 hdd 7.27739 osd.73 up 0.50000 1.00000 74 hdd 7.27739 osd.74 up 0.50000 1.00000 75 hdd 7.27739 osd.75 up 0.50000 1.00000 -4 101.88345 host cephdevel-76204 48 hdd 7.27739 osd.48 up 0.50000 1.00000 52 hdd 7.27739 osd.52 up 0.50000 1.00000 53 hdd 7.27739 osd.53 up 0.50000 1.00000 54 hdd 7.27739 osd.54 up 0.50000 1.00000 56 hdd 7.27739 osd.56 up 0.50000 1.00000 57 hdd 7.27739 osd.57 up 0.50000 1.00000 58 hdd 7.27739 osd.58 up 0.50000 1.00000 59 hdd 7.27739 osd.59 up 0.50000 1.00000 60 hdd 7.27739 osd.60 up 0.50000 1.00000 61 hdd 7.27739 osd.61 up 0.50000 1.00000 62 hdd 7.27739 osd.62 up 0.50000 1.00000 63 hdd 7.27739 osd.63 up 0.50000 1.00000 64 hdd 7.27739 osd.64 up 0.50000 1.00000 65 hdd 7.27739 osd.65 up 0.50000 1.00000 -23 203.16110 row row-02 -13 87.32867 host cephdevel-76213 27 hdd 7.27739 osd.27 up 1.00000 1.00000 28 hdd 7.27739 osd.28 up 1.00000 1.00000 29 hdd 7.27739 osd.29 up 1.00000 1.00000 30 hdd 7.27739 osd.30 up 1.00000 1.00000 31 hdd 7.27739 osd.31 up 1.00000 1.00000 32 hdd 7.27739 osd.32 up 1.00000 1.00000 33 hdd 7.27739 osd.33 up 1.00000 1.00000 34 hdd 7.27739 osd.34 up 1.00000 1.00000 35 hdd 7.27739 osd.35 up 1.00000 1.00000 36 hdd 7.27739 osd.36 up 1.00000 1.00000 37 hdd 7.27739 osd.37 up 1.00000 1.00000 38 hdd 7.27739 osd.38 up 1.00000 1.00000 ...... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io