Rebuilding data resiliency after adding new OSD's stuck for so long at 5%
We have a user-provisioned instance( Bare Metal Installation) of OpenShift cluster running on version 4.12 and we are using OpenShift Data Foundation as the Storage System. Earlier we had 3 disks attached to the storage system and 3 OSDs were available in the cluster. Today, while adding additional disks to the storage cluster, we increased the number of disks from 3 to 9, that is 3 per node. The addition of storage capacity was successful, resulting in 6 new OSDs in the cluster. But, after this operation, we noticed that Rebuilding Data Resiliency is stuck at 5% and not moving forward. At the same time, ceph status shows 65% of objects are misplaced and PGs are not in active+clean state. Here is more information about the ceph cluster: sh-4.4$ ceph status cluster: id: 18bf836d-4937-4925-b964-7a026c1d548d health: HEALTH_OK services: mon: 3 daemons, quorum b,u,v (age 2w) mgr: a(active, since 7w) mds: 1/1 daemons up, 1 hot standby osd: 9 osds: 9 up (since 5h), 9 in (since 5h); 191 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 12 pools, 305 pgs objects: 2.69M objects, 2.9 TiB usage: 8.8 TiB used, 27 TiB / 36 TiB avail pgs: 4723077/8079717 objects misplaced (58.456%) 188 active+remapped+backfill_wait 114 active+clean 3 active+remapped+backfilling io: client: 679 KiB/s rd, 11 MiB/s wr, 13 op/s rd, 622 op/s wr recovery: 20 MiB/s, 89 keys/s, 22 objects/s sh-4.4$ ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000276", "last_optimize_started": "Tue Sep 12 17:36:03 2023", "mode": "upmap", "optimize_result": "Too many objects (0.581933 > 0.050000) are misplaced; try again later", "plans": [] } One more thing we observed is that the number of misplaced objects is decreasing and also there is a drop in the percentage. What might be the reason behind Rebuilding Data Resiliency is not moving forward? Any inputs would be appreciated. Thanks
Hi, As long as you see changes and "recovery" it will make progress so i guess you have just to wait... What kind of disks did you add? Hth Mehmet Am 12. September 2023 20:37:56 MESZ schrieb sharathvuthpala@gmail.com:
We have a user-provisioned instance( Bare Metal Installation) of OpenShift cluster running on version 4.12 and we are using OpenShift Data Foundation as the Storage System. Earlier we had 3 disks attached to the storage system and 3 OSDs were available in the cluster. Today, while adding additional disks to the storage cluster, we increased the number of disks from 3 to 9, that is 3 per node. The addition of storage capacity was successful, resulting in 6 new OSDs in the cluster.
But, after this operation, we noticed that Rebuilding Data Resiliency is stuck at 5% and not moving forward. At the same time, ceph status shows 65% of objects are misplaced and PGs are not in active+clean state.
Here is more information about the ceph cluster:
sh-4.4$ ceph status cluster: id: 18bf836d-4937-4925-b964-7a026c1d548d health: HEALTH_OK
services: mon: 3 daemons, quorum b,u,v (age 2w) mgr: a(active, since 7w) mds: 1/1 daemons up, 1 hot standby osd: 9 osds: 9 up (since 5h), 9 in (since 5h); 191 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 12 pools, 305 pgs objects: 2.69M objects, 2.9 TiB usage: 8.8 TiB used, 27 TiB / 36 TiB avail pgs: 4723077/8079717 objects misplaced (58.456%) 188 active+remapped+backfill_wait 114 active+clean 3 active+remapped+backfilling
io: client: 679 KiB/s rd, 11 MiB/s wr, 13 op/s rd, 622 op/s wr recovery: 20 MiB/s, 89 keys/s, 22 objects/s
sh-4.4$ ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000276", "last_optimize_started": "Tue Sep 12 17:36:03 2023", "mode": "upmap", "optimize_result": "Too many objects (0.581933 > 0.050000) are misplaced; try again later", "plans": [] }
One more thing we observed is that the number of misplaced objects is decreasing and also there is a drop in the percentage. What might be the reason behind Rebuilding Data Resiliency is not moving forward?
Any inputs would be appreciated.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, We have HDD disks. Today, after almost 36 hours, Rebuilding Data Resiliency is 58% and still going on. The good thing is it is not stuck at 5%. Does it take this long to complete rebuilding resiliency process whenever there is a maintenance in the cluster?
We are using ceph version 16.2.10-172.el8cp (00a157ecd158911ece116ae43095de793ed9f389) pacific (stable).
Hi Guys, Thanks for your responses. The issue has been resolved. We increased the number of backfill threads to 5 from default value(1) and noticed some increase in the speed of rebalancing. Anyhow, it took almost 3 and a half days for the entire rebalancing process, which we believe would not be the case from next time. Thanks!!
participants (3)
-
ceph＠elchaka.de
-
Sake
-
sharathvuthpala＠gmail.com