Hi Igor, Thank you for your fast reply. I'll look into the provided URL, thanks. Please see below: for i in osd.17 osd.37 osd.89 osd.91 osd.95 osd.106; do ceph tell $i bluefs stats; done 1 : device size 0xc7fffe000 : using 0x423c00000(17 GiB) 2 : device size 0x9187fc00000 : using 0x5bb0bab0000(5.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 18 MiB 0 B 0 B 0 B 15 MiB 1 WAL 0 B 36 MiB 0 B 0 B 0 B 28 MiB 2 DB 0 B 17 GiB 0 B 0 B 0 B 13 GiB 211 SLOW 0 B 0 B 70 MiB 0 B 0 B 62 MiB 1 TOTAL 0 B 17 GiB 70 MiB 0 B 0 B 0 B 215 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 126 MiB 0 B 0 B 0 B 92 MiB DB 0 B 47 GiB 986 MiB 0 B 0 B 17 GiB SLOW 0 B 3.0 GiB 352 MiB 0 B 0 B 2.4 GiB TOTAL 0 B 50 GiB 1.3 GiB 0 B 0 B 0 B
SIZE << 0 B 48 GiB 8.6 TiB 1 : device size 0xc7fffe000 : using 0x434200000(17 GiB) 2 : device size 0x9187fc00000 : using 0x5fd47880000(6.0 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 14 MiB 0 B 0 B 0 B 9.4 MiB 1 WAL 0 B 18 MiB 0 B 0 B 0 B 11 MiB 1 DB 0 B 17 GiB 0 B 0 B 0 B 13 GiB 216 SLOW 0 B 0 B 70 MiB 0 B 0 B 53 MiB 1 TOTAL 0 B 17 GiB 70 MiB 0 B 0 B 0 B 219 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 126 MiB 0 B 0 B 0 B 93 MiB DB 0 B 48 GiB 0 B 0 B 0 B 16 GiB SLOW 0 B 1.9 GiB 141 MiB 0 B 0 B 1.5 GiB TOTAL 0 B 49 GiB 141 MiB 0 B 0 B 0 B SIZE << 0 B 48 GiB 8.6 TiB 1 : device size 0xc7fffe000 : using 0x458500000(17 GiB) 2 : device size 0x9187fc00000 : using 0x6b27b860000(6.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 14 MiB 0 B 0 B 0 B 12 MiB 1 WAL 0 B 72 MiB 0 B 0 B 0 B 51 MiB 4 DB 0 B 17 GiB 0 B 0 B 0 B 14 GiB 231 SLOW 0 B 0 B 70 MiB 0 B 0 B 60 MiB 1 TOTAL 0 B 17 GiB 70 MiB 0 B 0 B 0 B 237 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 198 MiB 0 B 0 B 0 B 172 MiB DB 0 B 48 GiB 0 B 0 B 0 B 20 GiB SLOW 0 B 1.7 GiB 423 MiB 0 B 0 B 1.1 GiB TOTAL 0 B 49 GiB 423 MiB 0 B 0 B 0 B SIZE << 0 B 48 GiB 8.6 TiB 1 : device size 0xc7fffe000 : using 0x476800000(18 GiB) 2 : device size 0x9187fc00000 : using 0x6a6d46f0000(6.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 14 MiB 0 B 0 B 0 B 9.6 MiB 1 WAL 0 B 54 MiB 0 B 0 B 0 B 34 MiB 3 DB 0 B 18 GiB 0 B 0 B 0 B 14 GiB 238 SLOW 0 B 71 MiB 141 MiB 0 B 0 B 5.9 MiB 3 TOTAL 0 B 18 GiB 141 MiB 0 B 0 B 0 B 245 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 180 MiB 0 B 0 B 0 B 155 MiB DB 0 B 41 GiB 0 B 0 B 0 B 19 GiB SLOW 0 B 6.5 GiB 2.5 GiB 0 B 0 B 6.8 GiB TOTAL 0 B 43 GiB 2.5 GiB 0 B 0 B 0 B SIZE << 0 B 48 GiB 8.6 TiB 1 : device size 0xc7fffe000 : using 0x405100000(16 GiB) 2 : device size 0x9187fc00000 : using 0x5bb806e0000(5.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 18 MiB 0 B 0 B 0 B 13 MiB 1 WAL 0 B 36 MiB 0 B 0 B 0 B 15 MiB 2 DB 0 B 16 GiB 0 B 0 B 0 B 13 GiB 210 SLOW 0 B 0 B 141 MiB 0 B 0 B 71 MiB 2 TOTAL 0 B 16 GiB 141 MiB 0 B 0 B 0 B 215 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 126 MiB 0 B 0 B 0 B 93 MiB DB 0 B 48 GiB 0 B 0 B 0 B 16 GiB SLOW 0 B 2.0 GiB 141 MiB 0 B 0 B 1.8 GiB TOTAL 0 B 50 GiB 141 MiB 0 B 0 B 0 B SIZE << 0 B 48 GiB 8.6 TiB 1 : device size 0xc7fffe000 : using 0x3cdd00000(15 GiB) 2 : device size 0x9187fc00000 : using 0x5bb9b2e0000(5.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 6 MiB 0 B 0 B 0 B 3.4 MiB 1 WAL 0 B 108 MiB 0 B 0 B 0 B 78 MiB 6 DB 0 B 15 GiB 0 B 0 B 0 B 12 GiB 202 SLOW 0 B 142 MiB 70 MiB 0 B 0 B 34 MiB 3 TOTAL 0 B 15 GiB 70 MiB 0 B 0 B 0 B 212 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 126 MiB 0 B 0 B 0 B 93 MiB DB 0 B 49 GiB 563 MiB 0 B 0 B 16 GiB SLOW 0 B 1014 MiB 323 MiB 0 B 0 B 895 MiB TOTAL 0 B 50 GiB 886 MiB 0 B 0 B 0 B SIZE << 0 B 48 GiB 8.6 TiB
On Mon, Aug 26, 2024 at 11:01 AM Igor Fedotov <igor.fedotov@croit.io> wrote:
Hi Ruben,
it could be nice if you share 'ceph tell osd.N bluefs stats' command output for these OSDs.
Also you might want to read the following thread https://www.spinics.net/lists/ceph-users/msg79062.html
which describes using 'ceph-volume lvm migrate' (or its counterpart in ceph-bluestore-tool) to migrate BlueFS data from slow to DB volume.
The latter might have temporary or permanent impact depending on the spillover root cause though.
Thanks,
Igor
On 8/26/2024 10:08 AM, Ruben Bosch wrote:
Hi all,
ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)
We are working on marking out OSDs on a host with EC4+2. The OSDs are HDDs. The OSDs have a separate DB on an NVMe disk. All the operations take ages. After some time we see BLUEFS_SPILLOVER. Telling the mentioned OSDs to compact sometimes helps, but not always. The OSDs have plenty space remaining in the db but the spillover does not disappear.
[WRN] BLUEFS_SPILLOVER: 2 OSD(s) experiencing BlueFS spillover osd.91 spilled over 141 MiB metadata from 'db' device (15 GiB used of 50 GiB) to slow device osd.106 spilled over 70 MiB metadata from 'db' device (12 GiB used of 50 GiB) to slow device
Has anyone seen similar behavior before and have they found a workaround or solution?
Kind regards,
Ruben Bosch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx