On my new ceph cluster consisting of 3 servers (45 22TB disks on each) I have created an EC8+4 pool for rgw application and forgot to add BULK flag on before uploading 270TB worth of data on it. Now the data is there and everything is fine except that pool is with 128 PG which should probably be 1024 and cluster is stuck backfilling all the time and seems to reset progress whenever it reaches ~5% misplaced object threshold, probably because of ceph balancer.
Yes, the balancer is aiming for a larger number of PGs than you currently have, and since doubling the number of PGs basically means 50% of the data must move in some way, it does a small amount at a time, 5% in your care. This can be tuned in the mgr balancer settings.
The actual question is what should I do - wait indefinitely until cluster recovers and add that BULK flag to the pool or add it right now and allow ceph to scale to some more reasonable amount of PG for this pool and let it then continue with backfill? So far I have temporarily disabled deepscrub as in combination with backfill that grinds recovery speed to a halt. Ceph health detail reports only warnings about not deep-scrubbed PGs, everything else looks in order, pg stat is as follows: 450 pgs: 2 active+remapped+backfill_wait, 12 active+remapped+backfilling, 436 active+clean; 276 TiB data, 442 TiB used, 2.6 PiB / 3.0 PiB avail; 93827081/1377582381 objects misplaced (6.811%); 33 MiB/s, 13 objects/s recovering Let me know if more information about the cluster might be helpful.
Most admins would probably just let it work. The scrubbing warnings are a bit boring to see, but you are reading and rewriting the data as the PGs splits, so the overall effect that scrubs make sure you don't have a lot of undetected errors is still accomplished, even if the warnings says otherwise. You could tune the 5% down to 1% or so, but you don't have a large queue of "backfill_wait", so it seems quite ok as is now. The "problem" here is that 8+4 means 12 OSDs get "busy" per PG backfilling, and you have some 12 of those PGs doing backfills now, so it ends up keeping upto 144 OSDs (in the absolute worst case) from doing deep-scrubs since the backfills hog the IO. In order to allow deepscrubs to actually run, you'd have to tune down the balancer a lot, and the backfills will be lots fewer, but also take quite a longer time before they actually do finish. -- May the most significant bit of your life be positive.