On Fri, Oct 06, 2023 at 02:55:22PM +1100, Chris Dunlop wrote:
Hi,
tl;dr why are my osds still spilling?
I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few):
osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device
Sigh. After battling this for days, I found the solution shortly after sending this plea for help. Via the hint from: https://tracker.ceph.com/issues/44509 using ceph-bluestore-tool bluefs-bdev-new-db results in 128k or 64k leftover on slow device The answer being, after expanding the db LV, and running the "bluefs-bdev-expand", you need to run "bluefs-bdev-migrate", e.g. all together: { lvextend -l ${size} ${lv}/${db} ${ssd_dev} cephadm unit --fsid $cid --name osd.${osd} stop cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool \ bluefs-bdev-expand \ --path /var/lib/ceph/osd/ceph-${osd} cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool \ bluefs-bdev-migrate \ --path /var/lib/ceph/osd/ceph-${osd} \ --devs-source /var/lib/ceph/osd/ceph-${osd}/block \ --dev-target /var/lib/ceph/osd/ceph-${osd}/block.db \ cephadm unit --fsid $cid --name osd.${osd} start } After the "bluefs-bdev-migrate", the "128 KiB" that was previously in the DB/SLOW location has reduced to zero: # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xe2e500000(57 GiB) 2 : device size 0xe8d7fc00000 : using 0x65598084000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 6 MiB 0 B 0 B 0 B 2.0 MiB 1 WAL 0 B 282 MiB 0 B 0 B 0 B 1.3 MiB 1 DB 0 B 56 GiB 0 B 0 B 0 B 50 GiB 841 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 57 GiB 0 B 0 B 0 B 0 B 843 MAXIMUMS: LOG 0 B 6 MiB 0 B 0 B 0 B 2.0 MiB WAL 0 B 564 MiB 0 B 0 B 0 B 56 MiB DB 0 B 56 GiB 0 B 0 B 0 B 50 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 57 GiB 0 B 0 B 0 B 0 B
SIZE << 0 B 453 GiB 14 TiB
...and the "osd.xx spilled over" has disappeared from my "ceph health detail". Now to do the others... Cheers, Chris