You have only 8 PGs in total? They take up several hundreds of GB, making one PG quite large.

Agreed. We don't have information about whether the autoscaler is enabled, but the docs do have guidance about pg_num for small clusters. Fewer than 5 OSDs: set pg_num to 128. Between 5 and 10 OSDs: set pg_num to 512. Between 10 and 50 OSDs: set pg_num to 1024. That simplistically describes a single pool but you get the idea. A contributing factor is the very small OSDs. OSDs smaller than, say, half a GiB have a disproportionate fraction devoted to metadata and tend to not fill as one would like.

...

1 pg backfill_toofull;

This is subtle: an OSD being over the backfill_toofull ratio is different from a PG being backfill_toofull, another case where we might have picked more distinct names but it's too late :-/

That's why it's backfill_toofull, it can't store another 200 GB (or so) on that OSD because it would breach the (default) limit of 90%. You need to increase pg_num drastically to decrease your PG sizes.

For reasons that I can't explain, this calculation in some circumstances effectively double-counts the data in a PG: once for (stale) data already on the destination, and once for the newer data on the source. PGs can be undersized because CRUSH can't place all shards/replicas because of the space / granularity dynamic. This will be especially pronounced if you're trying to do EC here. I'll guess that this is a sandbox cobbled together from what was laying around? If so I suggest sticking to replicated pools with size=2 to partly mitigate the limitations.

As a first shot I would aim for 100 PGs per OSD or so.

For these small and non-uniform OSDs, I might go ever smaller