Changing allocation size
Hi all, A while back, I indicated we had an issue with our cluster filling up too fast. After checking everything, we've concluded this was because we had a lot of small files and the allocation size on the bluestore was too high (64kb). We are now recreating the OSD's (2 disk at the same time) but, this will take a very long time as we're dealing with 130 OSDs. The current process we're following is removing 2 osd's and recreating them. We're using erasure coding (6 + 3). Has anyone some advice on how we can move forward with this? We've already increased some parameters to speed up recovery, but even then, it would still cost us too much time. If we could recreate them faster, that would be great... Or adapt the allocation on the fly? Any suggestions are welcome... Thank you, Kristof.
Hi Kristof, just some thoughts/insight on the issue. First of all it's not clear if you're going to migrate to EC 6+3 only or downsizing allocation size as well. Anyway I'd suggest to postpone these modifications for a while if possible. Ceph core team is aware of both space overhead caused by tons of small objects and in some scenarios when using EC pools. For the latter see https://tracker.ceph.com/issues/44213 The general solution for both seems to be downsizing min alloc size for spinners to 4K but currently this results in a significant performance drop. Some benchmarks for standalone BlueStore can be found here: https://docs.google.com/spreadsheets/d/1DzTEumM2hCnUu9Ceib7GfM035zFmPgPHmXke... A couple of PRs to avoid performance penalty are on the way: https://github.com/ceph/ceph/pull/33365 https://github.com/ceph/ceph/pull/33434 Once they get into master we might consider backporting these changes to Nautilus. So my general point is that currently the work is in progress and IMO it's better to wait until its completion. Thanks, Igor On 2/24/2020 11:48 PM, Kristof Coucke wrote:
Hi all,
A while back, I indicated we had an issue with our cluster filling up too fast. After checking everything, we've concluded this was because we had a lot of small files and the allocation size on the bluestore was too high (64kb). We are now recreating the OSD's (2 disk at the same time) but, this will take a very long time as we're dealing with 130 OSDs. The current process we're following is removing 2 osd's and recreating them. We're using erasure coding (6 + 3).
Has anyone some advice on how we can move forward with this? We've already increased some parameters to speed up recovery, but even then, it would still cost us too much time.
If we could recreate them faster, that would be great... Or adapt the allocation on the fly?
Any suggestions are welcome...
Thank you,
Kristof. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Igor Fedotov
-
Kristof Coucke