Hi community Please help me understand what is going on. I have a ceph (Reef) test cluster with the following crushmap ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 12.00000 root default -7 3.00000 host ksr-ceph-osd1 0 hdd 1.00000 osd.0 6 hdd 1.00000 osd.6 10 hdd 1.00000 osd.10 -9 3.00000 host ksr-ceph-osd2 3 hdd 1.00000 osd.3 7 hdd 1.00000 osd.7 11 hdd 1.00000 osd.11 -5 3.00000 host ksr-ceph-osd3 2 hdd 1.00000 osd.2 5 hdd 1.00000 osd.5 9 hdd 1.00000 osd.9 -3 3.00000 host ksr-ceph-osd4 1 hdd 1.00000 osd.1 4 hdd 1.00000 osd.4 8 hdd 1.00000 osd.8 -11 0 rack rack1 -13 0 rack rack2 -15 0 rack rack3 -17 0 rack rack4 Ceph status is like: cluster: id: 8a174287-42f8-43b6-9973-f174110b508b health: HEALTH_OK services: mon: 5 daemons, quorum ksr-ceph-mon2,ksr-ceph-mon3,ksr-ceph-mon1,ksr-ceph-mon5,ksr-ceph-mon4 (age 3h) mgr: ksr-ceph-mon1(active, since 3h), standbys: ksr-ceph-mon2, ksr-ceph-mon3 mds: 2/2 daemons up, 3 standby osd: 12 osds: 12 up (since 3h), 12 in (since 6d) data: volumes: 2/2 healthy pools: 5 pools, 129 pgs objects: 578 objects, 154 MiB usage: 1.0 GiB used, 599 GiB / 600 GiB avail pgs: 129 active+clean I then run: ceph osd set norecover ceph osd set nobackfill ceph osd set norebalance ceph osd crush move ksr-ceph-osd1 rack=rack1 ceph osd crush move ksr-ceph-osd2 rack=rack2 ceph osd crush move ksr-ceph-osd3 rack=rack3 ceph osd crush move ksr-ceph-osd4 rack=rack4 resulting in the following crush tree ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 12.00000 root default -11 3.00000 rack rack1 -7 3.00000 host ksr-ceph-osd1 0 hdd 1.00000 osd.0 6 hdd 1.00000 osd.6 10 hdd 1.00000 osd.10 -13 3.00000 rack rack2 -9 3.00000 host ksr-ceph-osd2 3 hdd 1.00000 osd.3 7 hdd 1.00000 osd.7 11 hdd 1.00000 osd.11 -15 3.00000 rack rack3 -5 3.00000 host ksr-ceph-osd3 2 hdd 1.00000 osd.2 5 hdd 1.00000 osd.5 9 hdd 1.00000 osd.9 -17 3.00000 rack rack4 -3 3.00000 host ksr-ceph-osd4 1 hdd 1.00000 osd.1 4 hdd 1.00000 osd.4 8 hdd 1.00000 osd.8 And ceph status is like: cluster: id: 8a174287-42f8-43b6-9973-f174110b508b health: HEALTH_WARN nobackfill,norebalance,norecover flag(s) set Degraded data redundancy: 2701/1734 objects degraded (155.767%), 55 pgs degraded services: mon: 5 daemons, quorum ksr-ceph-mon2,ksr-ceph-mon3,ksr-ceph-mon1,ksr-ceph-mon5,ksr-ceph-mon4 (age 3h) mgr: ksr-ceph-mon1(active, since 3h), standbys: ksr-ceph-mon2, ksr-ceph-mon3 mds: 2/2 daemons up, 3 standby osd: 12 osds: 12 up (since 3h), 12 in (since 6d); 22 remapped pgs flags nobackfill,norebalance,norecover data: volumes: 2/2 healthy pools: 5 pools, 129 pgs objects: 578 objects, 154 MiB usage: 1.0 GiB used, 599 GiB / 600 GiB avail pgs: 2701/1734 objects degraded (155.767%) 479/1734 objects misplaced (27.624%) 70 active+clean 34 active+recovery_wait+degraded 20 active+recovery_wait+undersized+degraded+remapped 3 active+recovering 1 active+recovery_wait+remapped 1 active+recovery_wait+degraded+remapped No crush rules have been changed on the pools. All pools has the default replicated_rule ceph osd crush rule dump replicated_rule { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } Questions 1 - Why do this result in such a high - objects degraded - percentage? 2 - Why do PGs get undersized? All in all this behavior does not make sense to me - I expect nothing to happen basically since I've only implemented some buckets in the map, no rules have changed - so I reach out in hope that someone can explain me the logic.