Hi, TL;DR: Selecting a different CRUSH rule (stretch_rule, no device class) for pool SSD results in degraded objects (unexpected) and misplaced objects (expected). Why would Ceph drop up to two healthy copies? Consider this two data center cluster: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.78400 root default -10 0.39200 datacenter DC1 -3 0.39200 host pve1 0 hdd 0.09799 osd.0 up 1.00000 1.00000 1 hdd 0.09799 osd.1 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -11 0.39200 datacenter DC2 -5 0.39200 host pve2 2 hdd 0.09799 osd.2 up 1.00000 1.00000 3 hdd 0.09799 osd.3 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000 Pools available: device_health_metrics HDD SSD Let's focus on SSD for now. Crush rule in use for SSD pool: rule SSD { id 2 type replicated min_size 1 max_size 10 step take default class ssd step choose firstn 0 type host step chooseleaf firstn 0 type osd step emit } SSD pool replication settings: min_size=2, size=4 The new stretch rule to use: rule stretch_rule { id 3 type replicated min_size 1 max_size 10 step take default step take DC1 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take default step take DC2 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } ceph pg ls for pool 3 (SSD):
3.0 184 0 0 0 738070528 0 0 7458 active+clean 9m 926'7458 951:18695 [7,6,5,4]p7 [7,6,5,4]p7 2024-06-05T08:52:51.951936+0200 2024-06-05T08:52:51.951936+0200 3.1 169 0 0 0 667156655 0 0 3242 active+clean 18m 926'3242 951:14151 [4,5,6,7]p4 [4,5,6,7]p4 2024-06-05T08:54:42.948682+0200 2024-06-05T08:54:42.948682+0200 3.2 221 0 0 0 885641968 0 0 6989 active+clean 9m 926'6989 951:17645 [4,5,7,6]p4 [4,5,7,6]p4 2024-06-05T08:53:27.981787+0200 2024-06-05T08:53:27.981787+0200 3.3 180 0 0 0 716509184 0 0 3194 active+clean 9m 926'3194 951:14191 [4,5,6,7]p4 [4,5,6,7]p4 2024-06-05T08:54:29.584216+0200 2024-06-05T08:54:29.584216+0200 3.4 189 0 0 0 754417698 137 8 3616 active+clean 9m 926'3616 951:19245 [6,7,4,5]p6 [6,7,4,5]p6 2024-06-05T08:54:02.307323+0200 2024-06-05T08:54:02.307323+0200 3.5 188 0 0 0 742543377 0 0 5992 active+clean 9m 926'5992 951:18862 [6,7,5,4]p6 [6,7,5,4]p6 2024-06-05T08:53:09.483136+0200 2024-06-05T08:53:09.483136+0200 3.6 191 0 0 0 769482752 150 16 6810 active+clean 9m 926'6810 951:30043 [7,6,4,5]p7 [7,6,4,5]p7 2024-06-05T08:53:46.646517+0200 2024-06-05T08:53:46.646517+0200 3.7 170 0 0 0 681587379 0 0 10081 active+clean 9m 926'21581 951:28473 [4,5,7,6]p4 [4,5,7,6]p4 2024-06-05T08:54:16.047967+0200 2024-06-05T08:54:16.047967+0200
ceph pg ls when the new crush rule is selected for pool SSD:
3.0 184 372 186 0 738070528 0 0 7458 active+recovery_wait+undersized+degraded+remapped 14s 926'7458 955:18688 [1,0,3,7]p1 [4,7]p4 2024-06-05T08:52:51.951936+0200 2024-06-05T08:52:51.951936+0200 3.1 169 0 0 0 667156655 0 0 3242 active+clean 19m 926'3242 954:14154 [4,5,6,7]p4 [4,5,6,7]p4 2024-06-05T08:54:42.948682+0200 2024-06-05T08:54:42.948682+0200 3.2 221 444 0 0 885641968 0 0 6989 active+recovery_wait+undersized+degraded+remapped 14s 926'6989 955:17657 [4,5,3,2]p4 [4,5]p4 2024-06-05T08:53:27.981787+0200 2024-06-05T08:53:27.981787+0200 3.3 180 0 540 0 716509184 0 0 3194 active+recovering+undersized+degraded+remapped 15s 926'3194 955:14204 [4,0,2,3]p4 [4,0]p4 2024-06-05T08:54:29.584216+0200 2024-06-05T08:54:29.584216+0200 3.4 189 378 0 0 754417698 0 0 3616 active+recovery_wait+undersized+degraded+remapped 15s 926'3616 955:19220 [1,4,6,2]p1 [4,6]p4 2024-06-05T08:54:02.307323+0200 2024-06-05T08:54:02.307323+0200 3.5 188 189 0 0 742543377 0 0 5992 active+recovery_wait+undersized+degraded+remapped 15s 926'5992 955:18845 [5,4,6,3]p5 [5,4,6]p5 2024-06-05T08:53:09.483136+0200 2024-06-05T08:53:09.483136+0200 3.6 191 390 195 0 769482752 0 0 6810 active+recovery_wait+undersized+degraded+remapped 14s 926'6810 955:30016 [0,1,2,7]p0 [4,7]p4 2024-06-05T08:53:46.646517+0200 2024-06-05T08:53:46.646517+0200 3.7 170 0 170 0 681587379 0 0 10081 active+remapped+backfill_wait 14s 926'21581 955:28486 [4,5,3,6]p4 [4,5,6,7]p4 2024-06-05T08:54:16.047967+0200 2024-06-05T08:54:16.047967+0200
So CRUSH is able to find a suitable mapping just fine, but somehow Ceph decides to drop up to two healthy copies from its acting se,t and I do not understand why. I would expect only misplaced objects at this point. Ceph version is 15.2.17. Latest CRUSH tunables (ceph osd crush tunables optimal). Do I miss something obvious here? If so, would you please point it out to me? :D. Gr. Stefan