BlueFS spillover detected (Nautilus 14.2.16)
Hello. I'm trying to fix a wrong cluster deployment (Nautilus 14.2.16) Cluster usage is %40 EC pool with RGW Every node has: 20 x OSD = TOSHIBA MG08SCA16TEY 16.0TB 2 x DB = NVME PM1725b 1.6TB (linux mdadm raid1) NVME usage always goes around %90-99. With "iostat -xdh 1" r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util Device 168.00 3619.00 7.2M 367.7M 0.00 90510.00 0.0% 96.2% 1.10 9.21 22.86 43.8k 104.0k 0.25 96.0% nvme0c0n1 19.00 3670.00 1.7M 373.5M 0.00 90510.00 0.0% 96.1% 0.26 29.61 95.99 89.7k 104.2k 0.27 98.0% nvme1c1n1 The problem is: BLUEFS_SPILLOVER BlueFS spillover detected on 120 OSD(s) osd.194 spilled over 42 GiB metadata from 'db' device (39 GiB used of 50 GiB) to slow device osd.195 spilled over 34 GiB metadata from 'db' device (40 GiB used of 50 GiB) to slow device osd.196 spilled over 28 GiB metadata from 'db' device (40 GiB used of 50 GiB) to slow device osd.197 spilled over 25 GiB metadata from 'db' device (41 GiB used of 50 GiB) to slow device osd.198 spilled over 30 GiB metadata from 'db' device (41 GiB used of 50 GiB) to slow device Block and wal size: bluestore_block_db_size = 53687091200 bluestore_block_wal_size = 0 nvme0n1 259:2 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md +n └─md0p20 259:22 0 50G 0 md How can I change the level up to 500MB --> 5GB --> 50GB ?
You need to adjust max_bytes_for_level_base rocksdb option from bluestore_rocksdb_options to 536870912 * get you release options `ceph config help bluestore_rocksdb_options` * append `bluestore_rocksdb_options=536870912` to this list * set `ceph config set osd bluestore_rocksdb_options <options_list>` Restart your OSD's. k
On 19 Apr 2021, at 17:34, by morphin <morphinwithyou@gmail.com> wrote:
How can I change the level up to 500MB --> 5GB --> 50GB ?
Thanks for the answer. It seems very easy. I've never played with rocksdb options before. I always used default and I think I need to play more with it but I couldn't find a good config reference to understand at ceph side. Can I use this guide instead? https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide This is the default options: "ceph config help bluestore_rocksdb_options" bluestore_rocksdb_options - Rocksdb options (str, advanced) Default: compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2 Can update at runtime: false
* get you release options `ceph config help bluestore_rocksdb_options` * append `bluestore_rocksdb_options=536870912` to this list * set `ceph config set osd bluestore_rocksdb_options <options_list>`
Are you tried to say add these (below) options to the config? - options.max_bytes_for_level_base = 536870912; // 512MB - options.max_bytes_for_level_multiplier = 10; At the link below I've found a tune but its for All-Flash cluster. https://ceph.io/community/bluestore-default-vs-tuned-performance-comparison/ bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB Also I've 2 type OSD's. SSD's for RGW index(no external db) and HDD's for EC pool with NVME cache. Do I need different options? Konstantin Shalygin <k0ste@k0ste.ru>, 19 Nis 2021 Pzt, 20:16 tarihinde şunu yazdı:
You need to adjust max_bytes_for_level_base rocksdb option from bluestore_rocksdb_options to 536870912
* get you release options `ceph config help bluestore_rocksdb_options` * append `bluestore_rocksdb_options=536870912` to this list * set `ceph config set osd bluestore_rocksdb_options <options_list>`
Restart your OSD's.
k
On 19 Apr 2021, at 17:34, by morphin <morphinwithyou@gmail.com> wrote:
How can I change the level up to 500MB --> 5GB --> 50GB ?
Multiplier is already == 10, it's not need to change, just a base. Or only multiplier. I better suggest upgrade to 14.2.19 and use new bluestore policy to use extra space for rocksdb levels (will be activated by default) k
On 19 Apr 2021, at 21:09, by morphin <morphinwithyou@gmail.com> wrote:
Are you tried to say add these (below) options to the config?
- options.max_bytes_for_level_base = 536870912; // 512MB - options.max_bytes_for_level_multiplier = 10;
There is a lot of bug-fix on RGW between 14.2.16 --> 19 and this is a prod environment. I always follow few versions behind to minimize the risk. Only for OSD improvement I'll not take the risk at RGW side. It's better to play with rocksdb options. Thanks for the advice. Konstantin Shalygin <k0ste@k0ste.ru>, 19 Nis 2021 Pzt, 22:57 tarihinde şunu yazdı:
Multiplier is already == 10, it's not need to change, just a base. Or only multiplier. I better suggest upgrade to 14.2.19 and use new bluestore policy to use extra space for rocksdb levels (will be activated by default)
k
On 19 Apr 2021, at 21:09, by morphin <morphinwithyou@gmail.com> wrote:
Are you tried to say add these (below) options to the config?
- options.max_bytes_for_level_base = 536870912; // 512MB - options.max_bytes_for_level_multiplier = 10;
participants (2)
-
by morphin
-
Konstantin Shalygin