Hi Wido,
'b' prefix relates to free list manager which keeps all the free extents
for main device in a bitmap. Its records have fixed size hence you can
easily estimate the overall size for these type of data.
But I doubt it takes that much. I presume that DB just lacks the proper
compaction. Which could happen eventually but looks like you interrupted
the process by going offline.
May be try manual compaction with ceph-kvstore-tool?
Thanks,
Igor
On 8/31/2020 10:57 AM, Wido den Hollander wrote:
> Hello,
>
> On a Nautilus 14.2.8 cluster I am seeing large RocksDB database with
> many slow DB bytes in use.
>
> To investigate this further I marked one OSD as out and waited for the
> all the backfilling to complete.
>
> Once the backfilling was completed I exported BlueFS and investigated
> the RocksDB using 'ceph-kvstore-tool'. This resulted in 22GB of data.
>
> Listing all the keys in the RocksDB shows me there are 747.000 keys in
> the DB. A small portion are osdmaps, but the biggest amount are keys
> prefixed with 'b'.
>
> I dumped the stats of the RocksDB and this shows me:
>
> L1: 1/0: 439.32 KB
> L2: 1/0: 2.65 MB
> L3: 5/0: 14.36 MB
> L4: 127/0: 7.22 GB
> L5: 217/0: 13.73 GB
> Sum: 351/0: 20.98 GB
>
> So there is almost 21GB of data in this RocksDB database. Why? Where
> is this coming from?
>
> Throughout this cluster OSDs are suffering from many slow bytes used
> and I can't figure out why.
>
> Has anybody seen this or has a clue on what is going on?
>
> I have an external copy of this RocksDB database to do investigations on.
>
> Thank you,
>
> Wido
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io