EC pool used space high
Hi, I have a rook-provisioned cluster to be used for RBDs only. I have 2 pools named replicated-metadata-pool and ec-data-pool. EC parameters are 6+3. I've been writing some data to this cluster for some time and noticed that the reported usage is not what I was expecting. # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.4 PiB 4.3 PiB 1.2 PiB 1.2 PiB 21.77 TOTAL 5.4 PiB 4.3 PiB 1.2 PiB 1.2 PiB 21.77 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL replicated-metadata-pool 1 90 KiB 408 38 MiB 0 1.2 PiB ec-data-pool 2 722 TiB 191.64M 1.2 PiB 25.04 2.4 PiB Since these numbers are rounded a bit too much, I generally use prometheus metrics on mgr, which are as follows: ceph_pool_stored : 793,746 G for ec-data-pool and 92323 for replicated-metadata-pool ceph_pool_stored_raw: 1,190,865 G for ec-data-pool and 99213 for replicated-metadata-pool ceph_cluster_total_used_bytes: 1,329,374 G ceph_cluster_total_used_raw_bytes: 1,333,013 G sum(ceph_bluefs_db_used_bytes) : 3,638 G So ceph_pool_stored for the EC pool is a bit higher than the total used space of the formatted RBDs. I think that's because of the sparse nature and deleted blocks not being fstrimmed yet. That's OK. ceph_pool_stored_raw is almost exactly 1.5x ceph_pool_stored which is what I'd expect considering EC parameters of 6+3. What I can't find is the 138,509 G difference between the ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case? We also have a ceph_cluster_total_used_raw_bytes metric, I believe to be close to data+metadata. Which is why I tried to show with sum(ceph_bluefs_db_used_bytes). Is that correct? Best, -- erdem agaoglu
On 11/25/19 6:05 PM, Erdem Agaoglu wrote:
What I can't find is the 138,509 G difference between the ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case?
May be min_size related? Because you are right, 6+3 is a 1.50, but 6+3 (+1) is a your calculated 1.67. k
I thought of that but it doesn't make much sense. AFAICT min_size should block IO when i lose 3 osds, but it shouldn't effect the amount of the stored data. Am i missing something? On Tue, Nov 26, 2019 at 6:04 AM Konstantin Shalygin <k0ste@k0ste.ru> wrote:
On 11/25/19 6:05 PM, Erdem Agaoglu wrote:
What I can't find is the 138,509 G difference between the ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case?
May be min_size related? Because you are right, 6+3 is a 1.50, but 6+3 (+1) is a your calculated 1.67.
k
-- erdem agaoglu
Maybe following link helps... https://www.spinics.net/lists/dev-ceph/msg00795.html On Tue, Nov 26, 2019 at 6:17 PM Erdem Agaoglu <erdem.agaoglu@gmail.com> wrote:
I thought of that but it doesn't make much sense. AFAICT min_size should block IO when i lose 3 osds, but it shouldn't effect the amount of the stored data. Am i missing something?
On Tue, Nov 26, 2019 at 6:04 AM Konstantin Shalygin <k0ste@k0ste.ru> wrote:
On 11/25/19 6:05 PM, Erdem Agaoglu wrote:
What I can't find is the 138,509 G difference between the ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case?
May be min_size related? Because you are right, 6+3 is a 1.50, but 6+3 (+1) is a your calculated 1.67.
k
-- erdem agaoglu _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That seems like it. Thanks a lot Serkan! On 26 Nov 2019 Tue at 20:08 Serkan Çoban <cobanserkan@gmail.com> wrote:
Maybe following link helps... https://www.spinics.net/lists/dev-ceph/msg00795.html
On Tue, Nov 26, 2019 at 6:17 PM Erdem Agaoglu <erdem.agaoglu@gmail.com> wrote:
I thought of that but it doesn't make much sense. AFAICT min_size should
block IO when i lose 3 osds, but it shouldn't effect the amount of the stored data. Am i missing something?
On Tue, Nov 26, 2019 at 6:04 AM Konstantin Shalygin <k0ste@k0ste.ru>
wrote:
On 11/25/19 6:05 PM, Erdem Agaoglu wrote:
What I can't find is the 138,509 G difference between the
ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case?
May be min_size related? Because you are right, 6+3 is a 1.50, but 6+3
(+1) is a your calculated 1.67.
k
-- erdem agaoglu _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- erdem agaoglu
Hi again, Even with this, our 6+3 EC pool with default bluestore_min_alloc_size 64KiB filled with 4MiB RBD objects should not take 1.67x space. It should be around 1.55x. There still is a 12% un-accounted overhead. Could there be something else too? Best, On Tue, Nov 26, 2019 at 8:08 PM Serkan Çoban <cobanserkan@gmail.com> wrote:
Maybe following link helps... https://www.spinics.net/lists/dev-ceph/msg00795.html
On Tue, Nov 26, 2019 at 6:17 PM Erdem Agaoglu <erdem.agaoglu@gmail.com> wrote:
I thought of that but it doesn't make much sense. AFAICT min_size should
block IO when i lose 3 osds, but it shouldn't effect the amount of the stored data. Am i missing something?
On Tue, Nov 26, 2019 at 6:04 AM Konstantin Shalygin <k0ste@k0ste.ru>
wrote:
On 11/25/19 6:05 PM, Erdem Agaoglu wrote:
What I can't find is the 138,509 G difference between the
ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case?
May be min_size related? Because you are right, 6+3 is a 1.50, but 6+3
(+1) is a your calculated 1.67.
k
-- erdem agaoglu _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- erdem agaoglu
participants (3)
-
Erdem Agaoglu
-
Konstantin Shalygin
-
Serkan Çoban