You may be facing this "BlueFS files take too much space" bug [1]. Have a look at the figures in the PR [2]. Fix hasn't been merged to Reef yet. Regards, Frédéric. [1] [ https://tracker.ceph.com/issues/68385 | https://tracker.ceph.com/issues/68385 ] [2] [ https://github.com/ceph/ceph/pull/60158 | https://github.com/ceph/ceph/pull/60158 ] ----- Le 8 Nov 24, à 5:51, Jaemin Joo <jm7.joo@gmail.com> a écrit :
Thank you for your response. I will test it and left the result.
How about the bluestore_rocksdb_options ? I don't understand why rocksdb has a lot of log data in sst file permanently except file metadata. I think some of log data needs to be rotated and left recent log data. I found rocksdb options which is related to log file like log_file_time_to_roll, keep_log_file_num, log_file_time_to_roll. Is it possible to change rocksdb options for META data reduction?
2024년 11월 8일 (금) 오전 12:53, Frédéric Nass < [ mailto:frederic.nass@univ-lorraine.fr | frederic.nass@univ-lorraine.fr ] >님이 작성:
Hi,
You could give rocksdb compression a try. It's safe to use since Pacific and it's now enabled by default in Squid:
$ ceph config set osd bluestore_rocksdb_options_annex 'compression=kLZ4Compression'
Restart all OSDs and compact them twice. You can check db_used_bytes before and after enabling compression with:
$ ceph tell osd.x perf dump bluefs | grep -E "db_"
Regards, Frédéric.
----- Le 7 Nov 24, à 16:07, Jaemin Joo [ mailto:jm7.joo@gmail.com | jm7.joo@gmail.com ] a écrit :
Hi All,
I have RADOS gw(rgw) ceph cluster (v18.2.1) applied Erasure Coding and 10009 shards to save a lot of objects in a bucket root:/home/ceph# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.00490 1.00000 5.0 GiB 318 MiB 58 MiB 1 KiB 260 MiB 4.7 GiB 6.21 0.94 143 up 1 hdd 0.00490 1.00000 5.0 GiB 330 MiB 56 MiB 2 KiB 274 MiB 4.7 GiB 6.45 0.98 155 up 2 hdd 0.00490 1.00000 5.0 GiB 372 MiB 60 MiB 1 KiB 313 MiB 4.6 GiB 7.28 1.10 148 up
(The osds are just data pool, it's not index pool for rgw..I think a lot of shards are not impact data pool META size) the OSD META occupied same as data size. I know that OSD META is rocksdb. I think oversized even though rocksdb META is needed. I can use osd compact to decrease META size. but I want to know why OSD META size is big than I expect, Is there way to decrease the META size by changing parameter instead of osd compact? I guess that some of "bluestore_rocksdb_options" can impact the META size.
And let me know if you need the other parameter of ceph cluster to figure out the symptom. _______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ]