EC Backfill Observations
Hey all, I wanted to confirm my understanding of some of the mechanics of backfill in EC pools. I've yet to find a document that outlines this in detail; if there is one, please send it my way. :) Some of what I write below is likely in the "well, duh" category, but I tended towards completeness. First off, I understand that backfill reservations work the same way between replicated pools and EC pools. A local reservation is taken on the primary OSD, then a remote reservation on the backfill target(s), before the backfill is allowed to begin. Until this point, the backfill is in the backfill_wait state. When the backfill begins, though, is when the differences begin. Let's say we have an EC 3:2 PG that's backfilling from OSD 2 to OSD 5 (formatted here like pgs_brief): 1.1 active+remapped+backfilling [0,1,5,3,4] 0 [0,1,2,3,4] 0 The question in my mind was: Where is the data for this backfill coming from? In replicated pools, all reads come from the primary. However, in this case, the primary does not have the data in question; the primary has to either read the EC chunk from OSD 2, or it has to reconstruct it by reading from 3 of the OSDs in the acting set. Based on observation, I _think_ this is what happens: 1. As long as the PG is not degraded, the backfill read is simply forwarded by the primary to OSD 2. 2. Once the PG becomes degraded, the backfill read needs to use the reconstructing path, and begins reading from 3 of the OSDs in the acting set. Questions: 1. Can anyone confirm or correct my description of how EC backfill operates? In particular, in case 2 above, does it matter whether OSD 2 is the cause of degradation, for example? Does the read still get forwarded to a single OSD when it's parity chunks that are being moved via backfill? 2. I'm curious as to why a 3rd reservation, for the source OSD, wasn't introduced as a part of EC in Ceph. We've occasionally seen an OSD become overloaded because several backfills were reading from it simultaneously, and there's no way to control this via the normal osd_max_backfills mechanism. Is anyone aware of discussions to this effect? Thanks! Josh
Hey Josh, adding the dev list where you may get more input. Generally I think your analysis is correct about the current behavior. In particular if another copy of a shard is available, backfill or recovery will read from just that copy, not the rest of the OSDs. Otherwise, k shards must be read to reconstruct the data (for reed- solomon family erasure codes). IIRC it doesn't matter whether it's a data or parity shard, the path is the same. With respect to reservations, it seems like an oversight that we don't reserve other shards for backfilling. We reserve all shards for recovery [0]. On the other hand, overload from recovery is handled better in pacific and beyond with mclock-based QoS, which provides much more effective control of recovery traffic [1][2]. In prior versions, the osd_recovery_sleep option was the best way to get more fine-grained control of recovery and backfill traffic, but this was not dynamic at all. osd_max_backfills allowed a maximum limit to parallelism. mclock supercedes these both when it's enabled, since it can handle bursting and throttling itself. Josh [0] https://github.com/ceph/ceph/blob/v16.2.1/src/osd/PeeringState.cc#L5914-L592... [1] https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#dmclock-... [2] https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/ On 4/19/21 12:24 PM, Josh Baergen wrote:
Hey all,
I wanted to confirm my understanding of some of the mechanics of backfill in EC pools. I've yet to find a document that outlines this in detail; if there is one, please send it my way. :) Some of what I write below is likely in the "well, duh" category, but I tended towards completeness.
First off, I understand that backfill reservations work the same way between replicated pools and EC pools. A local reservation is taken on the primary OSD, then a remote reservation on the backfill target(s), before the backfill is allowed to begin. Until this point, the backfill is in the backfill_wait state.
When the backfill begins, though, is when the differences begin. Let's say we have an EC 3:2 PG that's backfilling from OSD 2 to OSD 5 (formatted here like pgs_brief):
1.1 active+remapped+backfilling [0,1,5,3,4] 0 [0,1,2,3,4] 0
The question in my mind was: Where is the data for this backfill coming from? In replicated pools, all reads come from the primary. However, in this case, the primary does not have the data in question; the primary has to either read the EC chunk from OSD 2, or it has to reconstruct it by reading from 3 of the OSDs in the acting set.
Based on observation, I _think_ this is what happens: 1. As long as the PG is not degraded, the backfill read is simply forwarded by the primary to OSD 2. 2. Once the PG becomes degraded, the backfill read needs to use the reconstructing path, and begins reading from 3 of the OSDs in the acting set.
Questions: 1. Can anyone confirm or correct my description of how EC backfill operates? In particular, in case 2 above, does it matter whether OSD 2 is the cause of degradation, for example? Does the read still get forwarded to a single OSD when it's parity chunks that are being moved via backfill? 2. I'm curious as to why a 3rd reservation, for the source OSD, wasn't introduced as a part of EC in Ceph. We've occasionally seen an OSD become overloaded because several backfills were reading from it simultaneously, and there's no way to control this via the normal osd_max_backfills mechanism. Is anyone aware of discussions to this effect?
Thanks! Josh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Josh, Thanks for the info!
With respect to reservations, it seems like an oversight that we don't reserve other shards for backfilling. We reserve all shards for recovery [0].
Very interesting that there is a reservation difference between backfill and recovery.
On the other hand, overload from recovery is handled better in pacific and beyond with mclock-based QoS, which provides much more effective control of recovery traffic [1][2].
Indeed, I was wondering if mclock was ultimately the answer here, though I wonder how mclock acts in the case where a source OSD gets overloaded in the way that I described. Will it throttle backfill too aggressively, for example, compared to if the reservation was in place, preventing overload in the first place? One more question in this space: Has there ever been discussion about a back-off mechanism when one of the remote reservations is blocked? Another issue that we've commonly seen is that a backfill that should be able to make progress can't because of a backfill_wait that holds some of its reservations but is waiting for others. Example (with simplified up/acting sets): 1.1 active+remapped+backfilling [0,2] 0 [0,1] 0 1.2 active+remapped+backfill_wait [3,2] 3 [3,1] 3 1.3 active+remapped+backfill_wait [3,5] 3 [3,4] 3 1.3's backfill could make progress independent of 1.1, but is blocked behind 1.2 because the latter is holding the local reservation on osd.3 and is waiting for the remote reservation on osd.2. Josh
On 4/21/21 9:29 AM, Josh Baergen wrote:
Hey Josh,
Thanks for the info!
With respect to reservations, it seems like an oversight that we don't reserve other shards for backfilling. We reserve all shards for recovery [0].
Very interesting that there is a reservation difference between backfill and recovery.
On the other hand, overload from recovery is handled better in pacific and beyond with mclock-based QoS, which provides much more effective control of recovery traffic [1][2].
Indeed, I was wondering if mclock was ultimately the answer here, though I wonder how mclock acts in the case where a source OSD gets overloaded in the way that I described. Will it throttle backfill too aggressively, for example, compared to if the reservation was in place, preventing overload in the first place?
I expect you'd see more backfills proceeding, each at a slower pace, than if you had the reservations on all replicas. The total backfill throughput would be about the same, but completing a particular backfill would take longer.
One more question in this space: Has there ever been discussion about a back-off mechanism when one of the remote reservations is blocked? Another issue that we've commonly seen is that a backfill that should be able to make progress can't because of a backfill_wait that holds some of its reservations but is waiting for others. Example (with simplified up/acting sets):
1.1 active+remapped+backfilling [0,2] 0 [0,1] 0 1.2 active+remapped+backfill_wait [3,2] 3 [3,1] 3 1.3 active+remapped+backfill_wait [3,5] 3 [3,4] 3
1.3's backfill could make progress independent of 1.1, but is blocked behind 1.2 because the latter is holding the local reservation on osd.3 and is waiting for the remote reservation on osd.2.
Yes, the reservation mechanism is rather complex and intertwined with the recovery state machine. There was some discussion about this (including the idea of backoffs) before: https://marc.info/?t=152095454200002&r=1&w=2 and summarized in this card: https://trello.com/c/ppJKaJeT/331-osd-refactor-reserver Josh
participants (2)
-
Josh Baergen
-
Josh Durgin