Where has my capacity gone?
Hi, I have a ceph nautilus (14.2.9) cluster with 10 nodes. Each node has 19x16TB disks attached. I created radosgw pools. secondaryzone.rgw.buckets.data pool is configured as EC 8+2 (jerasure). ceph df shows 2.1PiB MAX AVAIL space. Then I configured radosgw as a secondary zone and 100TiB of S3 data is replicated. But weirdly enough ceph df shows 1.8PiB MAX AVAIL for the same pool. But there is only 100TiB of written data. ceph df also confirms it. I can not figure out where 200TiB capacity is gone. Would someone please tell me what I am missing? Thanks.
I created radosgw pools. secondaryzone.rgw.buckets.data pool is configured as EC 8+2 (jerasure).
Did you override the default bluestore_min_alloc_size_hdd (64k in that version IIRC) when creating your hdd OSDs? If not, all of the small objects produced by that EC configuration will be leading to significant on-disk allocation overhead. Josh
I did not. Honestly I was not aware of such a thing. Thanks for the notification. And I hope this is not bad news. May I ask if it can be dynamically changed and any disadvantages should be expected?
On 27 Jan 2021, at 01:33, Josh Baergen <jbaergen@digitalocean.com> wrote:
I created radosgw pools. secondaryzone.rgw.buckets.data pool is configured as EC 8+2 (jerasure).
Did you override the default bluestore_min_alloc_size_hdd (64k in that version IIRC) when creating your hdd OSDs? If not, all of the small objects produced by that EC configuration will be leading to significant on-disk allocation overhead.
Josh
On Wed, Jan 27, 2021 at 12:24 AM George Yil <georgeyil75@gmail.com> wrote:
May I ask if it can be dynamically changed and any disadvantages should be expected?
Unless there's some magic I'm unaware of, there is no way to dynamically change this. Each OSD must be recreated with the new min_alloc_size setting. In production systems this can be quite the chore, since the safest way to accomplish this is to drain the OSD (set it 'out', use CRUSH map changes, or use upmaps), recreate it, and then repopulate it. With automation this can run in the background. Given how much room you have currently you may be able to do this host-at-a-time by storing a host's data on the other hosts in a given rack (though I don't remember what your CRUSH tree looks like so maybe you can't do this and maintain host independence). The downside is potentially more tracking metadata at the OSD level, though I understand that Nautilus has made improvements here. I'm not up to speed on the latest state in this area, though. Josh
Thank you. This helps a lot.
Josh Baergen <jbaergen@digitalocean.com> şunları yazdı (27 Oca 2021 17:08):
On Wed, Jan 27, 2021 at 12:24 AM George Yil <georgeyil75@gmail.com> wrote:
May I ask if it can be dynamically changed and any disadvantages should be expected?
Unless there's some magic I'm unaware of, there is no way to dynamically change this. Each OSD must be recreated with the new min_alloc_size setting. In production systems this can be quite the chore, since the safest way to accomplish this is to drain the OSD (set it 'out', use CRUSH map changes, or use upmaps), recreate it, and then repopulate it. With automation this can run in the background. Given how much room you have currently you may be able to do this host-at-a-time by storing a host's data on the other hosts in a given rack (though I don't remember what your CRUSH tree looks like so maybe you can't do this and maintain host independence).
The downside is potentially more tracking metadata at the OSD level, though I understand that Nautilus has made improvements here. I'm not up to speed on the latest state in this area, though.
Josh
May I ask if enabling pool compression helps for the future space amplification?
George Yil <georgeyil75@gmail.com> şunları yazdı (27 Oca 2021 18:57):
Thank you. This helps a lot.
Josh Baergen <jbaergen@digitalocean.com> şunları yazdı (27 Oca 2021 17:08):
On Wed, Jan 27, 2021 at 12:24 AM George Yil <georgeyil75@gmail.com> wrote:
May I ask if it can be dynamically changed and any disadvantages should be expected?
Unless there's some magic I'm unaware of, there is no way to dynamically change this. Each OSD must be recreated with the new min_alloc_size setting. In production systems this can be quite the chore, since the safest way to accomplish this is to drain the OSD (set it 'out', use CRUSH map changes, or use upmaps), recreate it, and then repopulate it. With automation this can run in the background. Given how much room you have currently you may be able to do this host-at-a-time by storing a host's data on the other hosts in a given rack (though I don't remember what your CRUSH tree looks like so maybe you can't do this and maintain host independence).
The downside is potentially more tracking metadata at the OSD level, though I understand that Nautilus has made improvements here. I'm not up to speed on the latest state in this area, though.
Josh
Hi George,
May I ask if enabling pool compression helps for the future space amplification?
If the amplification is indeed due to min_alloc_size, then I don't think that compression will help. My understanding is that compression is applied post-EC (and thus probably won't even activate due to the small chunks), and that the compressed bits will still be stored on disk in the same way as before (min_alloc_size still applies). More info here: https://www.suse.com/support/kb/doc/?id=000019629 It's possible, though, that turning on compression and tuning its settings could reduce the overall number of blocks allocated, which would compensate slightly for the amplification. To confirm that you'd have to analyze the object sizes of your data set. There are also pathological cases where perhaps most of your EC chunks are slightly over 64K and by forcing them to compress (they won't by default) you actually cut allocated blocks in half. Again, that would take analysis to determine. Josh
participants (2)
-
George Yil
-
Josh Baergen