Hi Eugene!
I have a case, where PG with millions of objects, like this
```
root@host# ./show_osd_pool_pg_usage.sh <pool> | less | head
id used_mbytes used_objects omap_used_mbytes omap_used_keys
-- ----------- ------------ ---------------- --------------
17.c91 1213.2482748031616 2539152 0 0
17.9ae 1213.3145303726196 2539025 0 0
17.1a4 1213.432228088379 2539752 0 0
17.8f4 1213.4958791732788 2539831 0 0
17.f9 1213.5339193344116 2539837 0 0
17.c9d 1213.564414024353 2540014 0 0
17.89 1213.6339054107666 2540183 0 0
17.412 1213.6393299102783 2539797 0 0
```
And OSD was very small, like 1TB with RocksDB ~150-200GB. Actually currently you see
splitted PG. So one OSD was serve 64PG * 4M = 256,000,000 of objects...
Main problem was - to remove something, you need to move something. While the move is in
progress, nothing is deleted
Also, deleting is slower than writing. So one task for all operations was impossible. I do
it manually for a 9 moths. After the splitting of the some PG was completed, I took other
PG away from the most crowded (from the operator’s point of view, problematic) OSD. The
pgremapper [1] helped me with this. As far as I remember, in this way I got from 2048 to
3000 PG, then I was able to set 4096 PG, after which it became possible to move to 4TV
NVME
Your case doesn't look that scary. Firstly, your 85% means that you have hundreds of
free gigabytes (8TB's). If new data does not arrive, the reservation mechanism is
sufficient and after some time the process will end. On the other hand, I had a replica,
so compared to the EC - my case is a simpler
In any case, it’s worth trying and using the maximum capabilities of the upmap
Good luck,
k
[1]
https://github.com/digitalocean/pgremapper
On 9 Apr 2024, at 11:39, Eugen Block
<eblock(a)nde.ag> wrote:
I'm trying to estimate the possible impact when large PGs are splitted. Here's
one example of such a PG:
PG_STAT OBJECTS BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG UP
86.3ff 277708 414403098409 0 0 3092 3092
[187,166,122,226,171,234,177,163,155,34,81,239,101,13,117,8,57,111]
Their main application is RGW on EC (currently 1024 PGs on 240 OSDs), 8TB HDDs backed by
SSDs. There are 6 RGWs running behind HAProxies. It took me a while to convince them to do
a PG split and now they're trying to assess how big the impact could be. The fullest
OSD is already at 85% usage, the least filled one at 59%, so there is definitely room for
a better balancing which, will be necessary until the new hardware arrives. The current
distribution is around 100 PGs per OSD which usually would be fine, but since the PGs are
that large only a few PGs difference have a huge impact on the OSD utilization.
I'm targeting 2048 PGs for that pool for now, probably do another split when the new
hardware has been integrated.
Any comments are appreciated!