Any idea why misplace recovery wont finish?
Hello List, i have this 3 node Setup with 17 hdds (new - spinning rust). After putting it all together and after resetting the manual Rack-Placements back to default it seems to recover forever. I also changed the pg placements at some time. The disks where very busy until this morning when i disabled scrubbing to see if this would speed up the recovery. Here is my current status: ---------------------------------------- root@ceph12:~# ceph -s cluster: id: 2ed9f8fb-1316-4ef1-996d-a3223a3dd594 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set services: mon: 3 daemons, quorum ceph10,ceph11,ceph12 (age 2h) mgr: ceph11(active, since 3h), standbys: ceph10, ceph12 osd: 17 osds: 17 up (since 3h), 17 in (since 11d); 18 remapped pgs flags noscrub,nodeep-scrub data: pools: 2 pools, 257 pgs objects: 9.62M objects, 37 TiB usage: 110 TiB used, 134 TiB / 244 TiB avail pgs: 1534565/28857846 objects misplaced (5.318%) 239 active+clean 16 active+remapped+backfill_wait 2 active+remapped+backfilling io: recovery: 25 MiB/s, 6 objects/s I set: ceph tell 'osd.*' injectargs --osd-max-backfills=6 --osd-recovery-max-active=9 My hourly recovery process seems not to really proceed (it decreeses and increases again): 5.410 5.036 5.269 5.008 5.373 5.769 5.555 5.103 5.067 5.135 5.409 5.417 5.373 5.197 5.090 5.458 5.204 5.339 5.164 5.425 5.692 5.383 5.726 5.492 6.694 6.576 6.362 6.243 6.011 5.880 5.589 5.433 5.846 5.378 5.184 5.647 5.374 5.513 root@ceph12:~# ceph osd perf (this is okay for spinning rust i guess) osd commit_latency(ms) apply_latency(ms) 17 0 0 13 0 0 11 57 57 9 4 4 0 0 0 1 79 79 14 43 43 2 0 0 3 0 0 16 43 43 4 33 33 5 0 0 6 0 0 7 0 0 10 0 0 12 0 0 8 48 48 iostat -xt 3 shows that they are busy but its not overloaded as i can tell. Any idea why it will only backfill 2? How could i speed this up? My other cluster with 7 nodes and very old hdds and 45 OSDs can make easily 76 objects/s Cheers, Michael
Hi Michael, Are you using mclock or wpq? Mclock should handle backfills by itself allowing a bigger amount, but is not favoured by many admins due to its limitations for recovery operations. You can crosscheck what profile you are using with osd_mclock_profile and set it to high_recovery_ops at the expense of client traffic. With wpq your have the ability to set osd_max_backfills more precisely - check what value you have currently, it looks like its set to 1. Try increasing if so. Best, Laimis J. laimis.juzeliunas@oxylabs.io
On 14 Jan 2025, at 17:03, Ml Ml <mliebherr99@googlemail.com> wrote:
Hello List,
i have this 3 node Setup with 17 hdds (new - spinning rust).
After putting it all together and after resetting the manual Rack-Placements back to default it seems to recover forever. I also changed the pg placements at some time.
The disks where very busy until this morning when i disabled scrubbing to see if this would speed up the recovery.
Here is my current status: ---------------------------------------- root@ceph12:~# ceph -s cluster: id: 2ed9f8fb-1316-4ef1-996d-a3223a3dd594 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set
services: mon: 3 daemons, quorum ceph10,ceph11,ceph12 (age 2h) mgr: ceph11(active, since 3h), standbys: ceph10, ceph12 osd: 17 osds: 17 up (since 3h), 17 in (since 11d); 18 remapped pgs flags noscrub,nodeep-scrub
data: pools: 2 pools, 257 pgs objects: 9.62M objects, 37 TiB usage: 110 TiB used, 134 TiB / 244 TiB avail pgs: 1534565/28857846 objects misplaced (5.318%) 239 active+clean 16 active+remapped+backfill_wait 2 active+remapped+backfilling
io: recovery: 25 MiB/s, 6 objects/s
I set: ceph tell 'osd.*' injectargs --osd-max-backfills=6 --osd-recovery-max-active=9
My hourly recovery process seems not to really proceed (it decreeses and increases again): 5.410 5.036 5.269 5.008 5.373 5.769 5.555 5.103 5.067 5.135 5.409 5.417 5.373 5.197 5.090 5.458 5.204 5.339 5.164 5.425 5.692 5.383 5.726 5.492 6.694 6.576 6.362 6.243 6.011 5.880 5.589 5.433 5.846 5.378 5.184 5.647 5.374 5.513
root@ceph12:~# ceph osd perf (this is okay for spinning rust i guess) osd commit_latency(ms) apply_latency(ms) 17 0 0 13 0 0 11 57 57 9 4 4 0 0 0 1 79 79 14 43 43 2 0 0 3 0 0 16 43 43 4 33 33 5 0 0 6 0 0 7 0 0 10 0 0 12 0 0 8 48 48
iostat -xt 3 shows that they are busy but its not overloaded as i can tell.
Any idea why it will only backfill 2? How could i speed this up?
My other cluster with 7 nodes and very old hdds and 45 OSDs can make easily 76 objects/s
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Try, in order: 1) You don’t say which release you’re running. You might try reverting to the wpq scheduler 2) ceph health detail - look at the lead OSDs of each stuck PG. Pick one and issue `ceph osd down` on it, see if that gooses any recovery 3) Edit your crushmap and increase the number of tries Also note that if your OSDs are all the same size, you have capacity you can’t use. ceph osd tree | grep host When the number of failure domains is not larger than your replication factor (presumably 3) you want the failure domains to be of equal size. I suspect you have 6 drives per node and one is dead?
On Jan 14, 2025, at 10:03 AM, Ml Ml <mliebherr99@googlemail.com> wrote:
Hello List,
i have this 3 node Setup with 17 hdds (new - spinning rust).
After putting it all together and after resetting the manual Rack-Placements back to default it seems to recover forever. I also changed the pg placements at some time.
The disks where very busy until this morning when i disabled scrubbing to see if this would speed up the recovery.
Here is my current status: ---------------------------------------- root@ceph12:~# ceph -s cluster: id: 2ed9f8fb-1316-4ef1-996d-a3223a3dd594 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set
services: mon: 3 daemons, quorum ceph10,ceph11,ceph12 (age 2h) mgr: ceph11(active, since 3h), standbys: ceph10, ceph12 osd: 17 osds: 17 up (since 3h), 17 in (since 11d); 18 remapped pgs flags noscrub,nodeep-scrub
data: pools: 2 pools, 257 pgs objects: 9.62M objects, 37 TiB usage: 110 TiB used, 134 TiB / 244 TiB avail pgs: 1534565/28857846 objects misplaced (5.318%) 239 active+clean 16 active+remapped+backfill_wait 2 active+remapped+backfilling
io: recovery: 25 MiB/s, 6 objects/s
I set: ceph tell 'osd.*' injectargs --osd-max-backfills=6 --osd-recovery-max-active=9
My hourly recovery process seems not to really proceed (it decreeses and increases again): 5.410 5.036 5.269 5.008 5.373 5.769 5.555 5.103 5.067 5.135 5.409 5.417 5.373 5.197 5.090 5.458 5.204 5.339 5.164 5.425 5.692 5.383 5.726 5.492 6.694 6.576 6.362 6.243 6.011 5.880 5.589 5.433 5.846 5.378 5.184 5.647 5.374 5.513
root@ceph12:~# ceph osd perf (this is okay for spinning rust i guess) osd commit_latency(ms) apply_latency(ms) 17 0 0 13 0 0 11 57 57 9 4 4 0 0 0 1 79 79 14 43 43 2 0 0 3 0 0 16 43 43 4 33 33 5 0 0 6 0 0 7 0 0 10 0 0 12 0 0 8 48 48
iostat -xt 3 shows that they are busy but its not overloaded as i can tell.
Any idea why it will only backfill 2? How could i speed this up?
My other cluster with 7 nodes and very old hdds and 45 OSDs can make easily 76 objects/s
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The balancer is on, that is what triggers new misplaced object whenever the ratio goes near/below 5%. You may want to disable it, or by all means let it eventually finish. Den 2025-01-14 kl. 16:03, skrev Ml Ml:
Hello List,
i have this 3 node Setup with 17 hdds (new - spinning rust).
After putting it all together and after resetting the manual Rack-Placements back to default it seems to recover forever. I also changed the pg placements at some time.
The disks where very busy until this morning when i disabled scrubbing to see if this would speed up the recovery.
Here is my current status: ---------------------------------------- root@ceph12:~# ceph -s cluster: id: 2ed9f8fb-1316-4ef1-996d-a3223a3dd594 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set
services: mon: 3 daemons, quorum ceph10,ceph11,ceph12 (age 2h) mgr: ceph11(active, since 3h), standbys: ceph10, ceph12 osd: 17 osds: 17 up (since 3h), 17 in (since 11d); 18 remapped pgs flags noscrub,nodeep-scrub
data: pools: 2 pools, 257 pgs objects: 9.62M objects, 37 TiB usage: 110 TiB used, 134 TiB / 244 TiB avail pgs: 1534565/28857846 objects misplaced (5.318%) 239 active+clean 16 active+remapped+backfill_wait 2 active+remapped+backfilling
io: recovery: 25 MiB/s, 6 objects/s
I set: ceph tell 'osd.*' injectargs --osd-max-backfills=6 --osd-recovery-max-active=9
My hourly recovery process seems not to really proceed (it decreeses and increases again): 5.410 5.036 5.269 5.008 5.373 5.769 5.555 5.103 5.067 5.135 5.409 5.417 5.373 5.197 5.090 5.458 5.204 5.339 5.164 5.425 5.692 5.383 5.726 5.492 6.694 6.576 6.362 6.243 6.011 5.880 5.589 5.433 5.846 5.378 5.184 5.647 5.374 5.513
root@ceph12:~# ceph osd perf (this is okay for spinning rust i guess) osd commit_latency(ms) apply_latency(ms) 17 0 0 13 0 0 11 57 57 9 4 4 0 0 0 1 79 79 14 43 43 2 0 0 3 0 0 16 43 43 4 33 33 5 0 0 6 0 0 7 0 0 10 0 0 12 0 0 8 48 48
iostat -xt 3 shows that they are busy but its not overloaded as i can tell.
Any idea why it will only backfill 2? How could i speed this up?
My other cluster with 7 nodes and very old hdds and 45 OSDs can make easily 76 objects/s
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Michael, if you have mClock scheduler active you need first enable mclock overwrite ➜ ~ # ceph config set osd osd_mclock_override_recovery_settings true and then change osd settings e.g. ➜ ~ # ceph config set osd osd_max_backfills 5 Cheers Stephan Am Di., 14. Jan. 2025 um 16:04 Uhr schrieb Ml Ml <mliebherr99@googlemail.com
:
Hello List,
i have this 3 node Setup with 17 hdds (new - spinning rust).
After putting it all together and after resetting the manual Rack-Placements back to default it seems to recover forever. I also changed the pg placements at some time.
The disks where very busy until this morning when i disabled scrubbing to see if this would speed up the recovery.
Here is my current status: ---------------------------------------- root@ceph12:~# ceph -s cluster: id: 2ed9f8fb-1316-4ef1-996d-a3223a3dd594 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set
services: mon: 3 daemons, quorum ceph10,ceph11,ceph12 (age 2h) mgr: ceph11(active, since 3h), standbys: ceph10, ceph12 osd: 17 osds: 17 up (since 3h), 17 in (since 11d); 18 remapped pgs flags noscrub,nodeep-scrub
data: pools: 2 pools, 257 pgs objects: 9.62M objects, 37 TiB usage: 110 TiB used, 134 TiB / 244 TiB avail pgs: 1534565/28857846 objects misplaced (5.318%) 239 active+clean 16 active+remapped+backfill_wait 2 active+remapped+backfilling
io: recovery: 25 MiB/s, 6 objects/s
I set: ceph tell 'osd.*' injectargs --osd-max-backfills=6 --osd-recovery-max-active=9
My hourly recovery process seems not to really proceed (it decreeses and increases again): 5.410 5.036 5.269 5.008 5.373 5.769 5.555 5.103 5.067 5.135 5.409 5.417 5.373 5.197 5.090 5.458 5.204 5.339 5.164 5.425 5.692 5.383 5.726 5.492 6.694 6.576 6.362 6.243 6.011 5.880 5.589 5.433 5.846 5.378 5.184 5.647 5.374 5.513
root@ceph12:~# ceph osd perf (this is okay for spinning rust i guess) osd commit_latency(ms) apply_latency(ms) 17 0 0 13 0 0 11 57 57 9 4 4 0 0 0 1 79 79 14 43 43 2 0 0 3 0 0 16 43 43 4 33 33 5 0 0 6 0 0 7 0 0 10 0 0 12 0 0 8 48 48
iostat -xt 3 shows that they are busy but its not overloaded as i can tell.
Any idea why it will only backfill 2? How could i speed this up?
My other cluster with 7 nodes and very old hdds and 45 OSDs can make easily 76 objects/s
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Laimis Juzeliūnas
-
Ml Ml
-
peter.linder＠fiberdirekt.se
-
Stephan Hohn