So clearly some modification to the crush map is needed or some post processing of the crush mappings are in order to account for OSDs marked out.
Someone can correct me if I am wrong, but after a quick look at the code, the crushmap is not touched when an OSD is marked out. The new weight is stored in the OSDMap. When the new epoch of this map is sent to the OSDs, CRUSH calculates the new PG mappings, skipping the OSD with weight 0 (passed down from the new map). The new mappings differ from the old ones in this case, which triggers peering.
I.e. how to simulate a reweight (to 0 for being out, for any other number other than 1.0 - same problem). I.e. I have a crush map, I'd like to understand how to determine which PGs get remapped to what OSDs when I reweight an OSD from 1.0 to something lower.
I think you can use osdmaptool with --test-map-pgs-dump" and "--mark-out" flag to simulate an out OSD. ( https://docs.ceph.com/en/latest/man/8/osdmaptool/#cmdoption-osdmaptool-mark-... ) I don't think just the crushmap alone can achieve this. I'll let someone else confirm this. You can also simulate changing the crush weights using the "--adjust-crush-weight" flag. Thanks, Sam On Mon, 24 Aug 2026 at 15:33, Andras Pataki <ceph-users@ceph.io> wrote:
I'm trying to understand how 'reweighting' the OSD affects the placement of the PGs. I.e. how to simulate a reweight (to 0 for being out, for any other number other than 1.0 - same problem). I.e. I have a crush map, I'd like to understand how to determine which PGs get remapped to what OSDs when I reweight an OSD from 1.0 to something lower.
Andras
On 8/24/26 5:47 PM, Joshua Blanch wrote:
The "reweight" field that is set from 1.0 to 0.0 Have you tried reweighting to a non-zero, near-zero crush weight, e.g., 0.0001? That should leave it in the crush map and simulate it being 'out'.
- Joshua
On Mon, 24 Aug 2026 at 14:12, Andras Pataki <ceph-users@ceph.io> wrote:
Hi friends of ceph,
I'd like to better understand how ceph moves PGs off of OSDs that are marked out for the purposes of simulating potential setup changes. When all OSDs are in (and no upmaps are present), I can use crushtool to find the PG to OSDs mappings of all PGs in the cluster given a crush map (for example: "crushtool -i crushmap.compiled --test --show-mappings --num-rep 3 --min-x 0 --max-x 511 --rule 3 --pool-id 3" would do this for one of our pools). If I change any weights in the crush map, I can recompile it, rerun crushtool to get an updated mapping, and I have found this to perfectly match what ceph does when the crush map is loaded into the cluster.
What I am a bit unsure about is how to get the PG to OSD mappings when an OSD is marked out. The "reweight" field that is set from 1.0 to 0.0 when an OSD is marked out is not in the crush map, it is in the osdmap only. So clearly some modification to the crush map is needed or some post processing of the crush mappings are in order to account for OSDs marked out. I originally thought that changing the OSD weight to 0.0 in the crush map without modifying the parent bucket weights would do the trick. While this approach does remap the correct PGs (the ones that originally contain the OSD that is marked out), the new mappings are not the right ones (i.e. the new mappings do not match what live ceph does).
So my question if someone can enlighten me or point me to the right direction: starting with a crush map, what do I need to do to simulate the mappings for an OSD that is marked out? I.e. what changes are needed beyond what crushtool --show-mappings would give and how to calculate those offline (without touching a live cluster)? I've tried looking at the code (OSDMap.cc) - but couldn't quite understand where the adjustments are made for OSDs marked out. Any pointers/hints/insights would be appreciated.
Thanks,
Andras _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io