On Fri, Apr 21, 2023 at 7:26 AM Kilian Ries <mail@kilian-ries.de> wrote:
Still didn't find out what will happen when the pool is full - but tried a little bit in our testing environment and i were not able to get the pool full before an OSD got full. So in first place one OSD reached the full ratio (pool not quite full, about 98%) and IO stopped (like expected when an OSD reaches full ratio).
I *think* pool full doesn't actually matter if you haven't set quotas, but those properties have seen some code changes recently. CCing RADOS people. We do have a proposed fix but it seems to have languished. :( -Greg
I were able to re-balance the OSDs by manually doing reweights. Now, the cluster is much more balanced and even the pool shows more free space (about 75% used).
Also the pg-autoscaler does not really play well with the stretch crush rule ... had to increase / adjust the PGs manually to get a better distribution.
Regards, Kilian ________________________________ Von: Kilian Ries <mail@kilian-ries.de> Gesendet: Mittwoch, 19. April 2023 12:18:06 An: ceph-users Betreff: [ceph-users] Ceph stretch mode / POOL_BACKFILLFULL
Hi,
we run a ceph cluster in stretch mode with one pool. We know about this bug:
https://tracker.ceph.com/issues/56650
https://github.com/ceph/ceph/pull/47189
Can anyone tell me what happens when a pool gets to 100% full? At the moment raw OSD usage is about 54% but ceph throws me a "POOL_BACKFILLFULL" error:
$ ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 63 TiB 29 TiB 34 TiB 34 TiB 54.19
TOTAL 63 TiB 29 TiB 34 TiB 34 TiB 54.19
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.mgr 1 1 415 MiB 105 1.2 GiB 0.04 1.1 TiB
vm_stretch_live 2 64 15 TiB 4.02M 34 TiB 95.53 406 GiB
So the pool warning / calculation is just a bug, because it thinks its 50% of the total size. I know ceph will stop IO / set OSDs to read only if the hit a "backfillfull_ratio" ... but what will happen if the pool gets to 100% full ?
Will IO still be possible?
No limits / quotas are set on the pool ...
Thanks
Regards,
Kilian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io