Oh, but of cause everything smooths out after while. My main concern is just, if I do this on a large cluster, it will send it spinning... ________________________________ From: Kasper Rasmussen <kasper_steengaard@hotmail.com> Sent: Tuesday, January 21, 2025 18:35 To: Dan van der Ster <dan.vanderster@clyso.com>; Anthony D'Atri <aad@dreamsnake.net> Cc: ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Changing crush map result in > 100% objects degraded Hi Dan "Also, in the process of moving the hosts one by one, each step creates a new topology which can change the ordering of hosts, incrementally putting things out of whack." RESPONSE: Will it be better to edit the crushmap as a file, and load the new with ceph osd setcrushmap -i <file> ? Kaspar: I assume the cluster was idle during your tests? RESPONSE: Yes, the cluster was indeed idle Also -- can you reproduce it without norecover/nobackfill set ? RESPONSE: Yes, I reproduced with no flags set - Starting point HEALTH_OK - Result is as seen here below - (The output from the ceph pg "$pgid" query is 8548 lines - I don't know how to send that - Just as an attachment?): ubuntu@ksr-ceph-deploy:~$ ceph osd crush move ksr-ceph-osd1 rack=rack1; moved item id -7 name 'ksr-ceph-osd1' to location {rack=rack1} in crush map ubuntu@ksr-ceph-deploy:~$ sleep 10 ubuntu@ksr-ceph-deploy:~$ ceph pg ls undersized; PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 1.0 2 2 0 0 459280 0 0 117 0 active+recovery_wait+undersized+degraded+remapped 7s 732'117 963:994 [11,2,8]p11 [11,2]p11 2025-01-21T05:25:36.075052+0000 2025-01-17T14:16:37.883834+0000 1 periodic scrub scheduled @ 2025-01-22T15:03:39.660268+0000 6.0 22 60 0 0 2252800 0 0 124 0 active+recovery_wait+undersized+degraded+remapped 7s 732'124 962:2333 [7,6,1]p7 [7,1]p7 2025-01-21T09:06:38.302061+0000 2025-01-20T03:35:48.722520+0000 1 periodic scrub scheduled @ 2025-01-22T18:15:49.710411+0000 6.1 12 50 0 0 1228800 0 0 110 0 active+recovery_wait+undersized+degraded+remapped 6s 732'110 963:2298 [6,3,1]p6 [6,3]p6 2025-01-21T09:04:29.912825+0000 2025-01-20T03:11:56.962281+0000 1 periodic scrub scheduled @ 2025-01-22T14:09:59.472013+0000 6.2 13 52 52 0 5423104 0 0 107 0 active+recovery_wait+undersized+degraded+remapped 6s 732'107 963:2273 [10,11,4]p10 [0,4]p0 2025-01-21T06:46:11.657543+0000 2025-01-17T14:16:41.932263+0000 0 periodic scrub scheduled @ 2025-01-22T13:52:17.796513+0000 6.5 18 100 0 0 10027008 0 0 113 0 active+recovery_wait+undersized+degraded+remapped 6s 732'113 963:726 [0,9,3]p0 [0,3]p0 2025-01-21T05:24:25.006369+0000 2025-01-21T05:24:25.006369+0000 0 periodic scrub scheduled @ 2025-01-22T12:29:24.576297+0000 6.9 16 55 0 0 5730304 0 0 104 0 active+recovery_wait+undersized+degraded+remapped 6s 732'104 963:996 [10,9,1]p10 [10,1]p10 2025-01-21T09:02:25.957504+0000 2025-01-17T14:16:50.479422+0000 0 periodic scrub scheduled @ 2025-01-22T19:49:15.092705+0000 6.b 18 60 0 0 1843200 0 0 114 0 active+recovery_wait+undersized+degraded+remapped 6s 732'114 962:3860 [1,2,6]p1 [1,2]p1 2025-01-21T06:57:22.832565+0000 2025-01-17T14:16:57.820141+0000 0 periodic scrub scheduled @ 2025-01-22T09:03:26.636583+0000 6.f 17 60 0 0 5832704 0 0 117 0 active+recovery_wait+undersized+degraded+remapped 6s 732'117 963:1437 [4,3,2]p4 [4,3]p4 2025-01-21T07:59:55.049488+0000 2025-01-17T14:17:05.581667+0000 0 periodic scrub scheduled @ 2025-01-22T14:09:31.906176+0000 6.11 20 59 0 0 7888896 0 0 123 0 active+recovery_wait+undersized+degraded+remapped 7s 732'123 963:3329 [11,0,8]p11 [11,8]p11 2025-01-21T01:32:23.458956+0000 2025-01-17T14:17:09.774195+0000 0 periodic scrub scheduled @ 2025-01-22T06:25:32.462414+0000 6.12 21 4 42 0 10334208 0 0 141 0 active+recovering+undersized+degraded+remapped 1.11259s 732'141 964:613 [9,8,11]p9 [2,11]p2 2025-01-21T05:01:52.629884+0000 2025-01-21T05:01:52.629884+0000 0 periodic scrub scheduled @ 2025-01-22T07:57:03.899309+0000 6.13 22 138 0 0 8093696 0 0 156 0 active+recovery_wait+undersized+degraded+remapped 6s 732'156 963:1312 [3,6,9]p3 [3,6]p3 2025-01-21T04:47:23.091543+0000 2025-01-18T19:41:37.702881+0000 0 periodic scrub scheduled @ 2025-01-22T13:44:27.566620+0000 6.14 14 57 0 0 5525504 0 0 116 0 active+recovery_wait+undersized+degraded+remapped 6s 732'116 963:804 [11,9,8]p11 [11,9]p11 2025-01-21T04:58:22.800659+0000 2025-01-18T18:51:30.797784+0000 0 periodic scrub scheduled @ 2025-01-22T14:10:05.285157+0000 6.17 15 58 0 0 3284992 0 0 117 0 active+recovery_wait+undersized+degraded+remapped 7s 732'117 963:1758 [11,8,5]p11 [11,8]p11 2025-01-21T17:02:51.283098+0000 2025-01-17T14:17:24.300985+0000 3 periodic scrub scheduled @ 2025-01-23T02:23:35.432833+0000 6.1a 16 2 0 0 3387392 0 0 118 0 active+recovering+undersized+remapped 0.802018s 732'118 964:1263 [6,7,1]p6 [1,6]p1 2025-01-21T09:04:47.728667+0000 2025-01-18T21:05:47.129277+0000 1 periodic scrub scheduled @ 2025-01-22T16:41:28.828878+0000 6.1d 18 118 0 0 11776000 0 0 111 0 active+recovery_wait+undersized+degraded+remapped 6s 732'111 963:1239 [3,5,10]p3 [3,5]p3 2025-01-21T06:57:54.358465+0000 2025-01-17T14:17:34.523095+0000 0 periodic scrub scheduled @ 2025-01-22T10:05:29.250471+0000 * NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. ubuntu@ksr-ceph-deploy:~$ sleep 5; ubuntu@ksr-ceph-deploy:~$ ceph pg ls degraded; PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 1.0 2 2 0 0 459280 0 0 117 0 active+recovery_wait+undersized+degraded+remapped 12s 732'117 965:996 [11,2,8]p11 [11,2]p11 2025-01-21T05:25:36.075052+0000 2025-01-17T14:16:37.883834+0000 1 periodic scrub scheduled @ 2025-01-22T15:03:39.660268+0000 6.2 13 52 52 0 5423104 0 0 107 0 active+recovery_wait+undersized+degraded+remapped 12s 732'107 965:2275 [10,11,4]p10 [0,4]p0 2025-01-21T06:46:11.657543+0000 2025-01-17T14:16:41.932263+0000 0 periodic scrub scheduled @ 2025-01-22T13:52:17.796513+0000 6.5 18 100 0 0 10027008 0 0 113 0 active+recovery_wait+undersized+degraded+remapped 11s 732'113 965:728 [0,9,3]p0 [0,3]p0 2025-01-21T05:24:25.006369+0000 2025-01-21T05:24:25.006369+0000 0 periodic scrub scheduled @ 2025-01-22T12:29:24.576297+0000 6.7 19 2 0 0 9535488 0 0 100 0 active+recovering+degraded 2s 732'100 966:1371 [11,10,1]p11 [11,10,1]p11 2025-01-21T11:56:25.453326+0000 2025-01-17T14:16:46.382792+0000 0 periodic scrub scheduled @ 2025-01-22T21:24:43.245561+0000 6.9 16 55 0 0 5730304 0 0 104 0 active+recovery_wait+undersized+degraded+remapped 11s 732'104 965:998 [10,9,1]p10 [10,1]p10 2025-01-21T09:02:25.957504+0000 2025-01-17T14:16:50.479422+0000 0 periodic scrub scheduled @ 2025-01-22T19:49:15.092705+0000 6.b 18 60 0 0 1843200 0 0 114 0 active+recovery_wait+undersized+degraded+remapped 12s 732'114 965:3862 [1,2,6]p1 [1,2]p1 2025-01-21T06:57:22.832565+0000 2025-01-17T14:16:57.820141+0000 0 periodic scrub scheduled @ 2025-01-22T09:03:26.636583+0000 6.f 17 60 0 0 5832704 0 0 117 0 active+recovery_wait+undersized+degraded+remapped 12s 732'117 965:1439 [4,3,2]p4 [4,3]p4 2025-01-21T07:59:55.049488+0000 2025-01-17T14:17:05.581667+0000 0 periodic scrub scheduled @ 2025-01-22T14:09:31.906176+0000 6.11 20 59 0 0 7888896 0 0 123 0 active+recovery_wait+undersized+degraded+remapped 12s 732'123 965:3331 [11,0,8]p11 [11,8]p11 2025-01-21T01:32:23.458956+0000 2025-01-17T14:17:09.774195+0000 0 periodic scrub scheduled @ 2025-01-22T06:25:32.462414+0000 6.13 22 138 0 0 8093696 0 0 156 0 active+recovery_wait+undersized+degraded+remapped 11s 732'156 963:1312 [3,6,9]p3 [3,6]p3 2025-01-21T04:47:23.091543+0000 2025-01-18T19:41:37.702881+0000 0 periodic scrub scheduled @ 2025-01-22T13:44:27.566620+0000 6.14 14 57 0 0 5525504 0 0 116 0 active+recovery_wait+undersized+degraded+remapped 12s 732'116 965:806 [11,9,8]p11 [11,9]p11 2025-01-21T04:58:22.800659+0000 2025-01-18T18:51:30.797784+0000 0 periodic scrub scheduled @ 2025-01-22T14:10:05.285157+0000 6.17 15 58 0 0 3284992 0 0 117 0 active+recovery_wait+undersized+degraded+remapped 12s 732'117 965:1760 [11,8,5]p11 [11,8]p11 2025-01-21T17:02:51.283098+0000 2025-01-17T14:17:24.300985+0000 3 periodic scrub scheduled @ 2025-01-23T02:23:35.432833+0000 6.18 10 41 0 0 5115904 0 0 95 0 active+recovery_wait+degraded 12s 732'95 965:888 [4,10,5]p4 [4,10,5]p4 2025-01-21T07:22:43.040812+0000 2025-01-17T14:17:26.898595+0000 0 periodic scrub scheduled @ 2025-01-22T17:24:55.003668+0000 6.1c 9 86 0 0 5013504 0 0 88 0 active+recovery_wait+degraded 12s 732'88 963:24 [3,0,8]p3 [3,0,8]p3 2025-01-21T09:08:33.536101+0000 2025-01-17T14:17:37.489220+0000 1 periodic scrub scheduled @ 2025-01-22T11:35:58.254737+0000 6.1d 18 118 0 0 11776000 0 0 111 0 active+recovery_wait+undersized+degraded+remapped 11s 732'111 963:1239 [3,5,10]p3 [3,5]p3 2025-01-21T06:57:54.358465+0000 2025-01-17T14:17:34.523095+0000 0 periodic scrub scheduled @ 2025-01-22T10:05:29.250471+0000 * NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. ________________________________ From: Dan van der Ster <dan.vanderster@clyso.com> Sent: Tuesday, January 21, 2025 16:51 To: Anthony D'Atri <aad@dreamsnake.net>; Kasper Rasmussen <kasper_steengaard@hotmail.com> Cc: ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Changing crush map result in > 100% objects degraded On Tue, Jan 21, 2025 at 7:12 AM Anthony D'Atri <aad@dreamsnake.net> wrote:
On Jan 21, 2025, at 7:59 AM, Kasper Rasmussen <kasper_steengaard@hotmail.com> wrote:
1 - Why do this result in such a high - objects degraded - percentage?
I suspect that’s a function of the new topology having changed the mappings of multiple OSDs for given PGs. It’s subtle, but when you move hosts into rack CRUSH buckets, that’s a different set of inputs into the CRUSH hash function, so the mappings that come out are different, even though you haven’t changed the rules and would think that hosts are hosts.
Also, in the process of moving the hosts one by one, each step creates a new topology which can change the ordering of hosts, incrementally putting things out of whack.
2 - Why do PGs get undersized?
That often means that CRUSH can’t find a complete set of placements. In your situation maybe those would resolve themselves when you unleash the recovery hounds.
We started noticing this kind of issue around pacific, but haven't fully tracked down what broke yet. See https://tracker.ceph.com/issues/56046 for similar. Undersized or degraded should only happen -- by design -- if objects were modified while the PG did not have 3 OSDs up and acting. Kaspar: I assume the cluster was idle during your tests? Also -- can you reproduce it without norecover/nobackfill set ? Could you simplify your reproducer down to:
HEALTH_OK ceph osd crush move ksr-ceph-osd1 rack=rack1 ceph pg ls undersized / degraded # get a pgid of a degraded PG ceph pg $pgid query
Cheers, dan -- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com