Hi,

I have a rook-provisioned cluster to be used for RBDs only. I have 2 pools named replicated-metadata-pool and ec-data-pool. EC parameters are 6+3. I've been writing some data to this cluster for some time and noticed that the reported usage is not what I was expecting.

# ceph df      
RAW STORAGE:
    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED
    hdd       5.4 PiB     4.3 PiB     1.2 PiB      1.2 PiB         21.77
    TOTAL     5.4 PiB     4.3 PiB     1.2 PiB      1.2 PiB         21.77
 
POOLS:
    POOL                         ID     STORED      OBJECTS     USED        %USED     MAX AVAIL
    replicated-metadata-pool      1      90 KiB         408      38 MiB         0       1.2 PiB
    ec-data-pool                  2     722 TiB     191.64M     1.2 PiB     25.04       2.4 PiB

Since these numbers are rounded a bit too much, I generally use prometheus metrics on mgr, which are as follows:

ceph_pool_stored : 793,746 G for ec-data-pool and 92323 for replicated-metadata-pool

ceph_pool_stored_raw: 1,190,865 G for ec-data-pool and 99213 for replicated-metadata-pool

ceph_cluster_total_used_bytes: 1,329,374 G

ceph_cluster_total_used_raw_bytes: 1,333,013 G

sum(ceph_bluefs_db_used_bytes) : 3,638 G

So ceph_pool_stored for the EC pool is a bit higher than the total used space of the formatted RBDs. I think that's because of the sparse nature and deleted blocks not being fstrimmed yet. That's OK.

ceph_pool_stored_raw is almost exactly 1.5x ceph_pool_stored which is what I'd expect considering EC parameters of 6+3.

What I can't find is the 138,509 G difference between the ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case?

We also have a ceph_cluster_total_used_raw_bytes metric, I believe to be close to data+metadata. Which is why I tried to show with sum(ceph_bluefs_db_used_bytes). Is that correct?

Best,

--