Why a lot of pgs are degraded after host(+osd) restarted?
Hi all, While I am testing host failover, there are a lot of degraded pg after host(+osd) is up. In spite that it takes a short time to restart, I don't understand why pg should check all objects related to the failed host(+osd). I'd like to know how to prevent to become degraded pg when osd restart. FYI. degraded pg means "active+undersized+degraded+remapped+backfilling" pg state
Hi Jaemin, It is normal for PGs to become degraded during a host reboot, since a copy of the data was taken offline and needs to be resynchronized after the host comes back. Normally this is quick, as the recovery mechanism only needs to modify those objects that have changed while the host is down. However, if you have backfills ongoing and reboot a host that contains OSDs involved in those backfills, then those backfills become degraded, and you will need to wait for them to complete for degradation to clear. Do you know if you had backfills at the time the host was rebooted? If so, the way to avoid this is to wait for backfill to complete before taking any OSDs/hosts down for maintenance. Josh On Wed, Mar 20, 2024 at 1:50 AM Jaemin Joo <jm7.joo@gmail.com> wrote:
Hi all,
While I am testing host failover, there are a lot of degraded pg after host(+osd) is up. In spite that it takes a short time to restart, I don't understand why pg should check all objects related to the failed host(+osd). I'd like to know how to prevent to become degraded pg when osd restart.
FYI. degraded pg means "active+undersized+degraded+remapped+backfilling" pg state _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I understood the mechanism more through your answer. I'm using erasure coding and backfilling step took quite a long time :( If there was just a lot of pg peering. I think it's reasonable. but I was curious why there was a lot of backfill_wait instead of peering. (e.g. pg 9.5a is stuck undersized for 39h, current state active+undersized+degraded+remapped+backfill_wait ) let me know if you have the tips to increase the performance of backfill or prevent unnecessary backfill. Thank you for your answer. Joshua Baergen wrote:
Hi Jaemin,
It is normal for PGs to become degraded during a host reboot, since a copy of the data was taken offline and needs to be resynchronized after the host comes back. Normally this is quick, as the recovery mechanism only needs to modify those objects that have changed while the host is down.
However, if you have backfills ongoing and reboot a host that contains OSDs involved in those backfills, then those backfills become degraded, and you will need to wait for them to complete for degradation to clear. Do you know if you had backfills at the time the host was rebooted? If so, the way to avoid this is to wait for backfill to complete before taking any OSDs/hosts down for maintenance.
Josh
participants (3)
-
jaemin joo
-
Jaemin Joo
-
Joshua Baergen