One more manual compaction updated bluefs stats figures accordingly. So at the end, it is: 1/ ceph orch daemon stop osd.${osd} 2/ cephadm shell --fsid $(ceph fsid) --name osd.${osd} -- ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-${osd} --devs-source /var/lib/ceph/osd/ceph-${osd}/block --dev-target /var/lib/ceph/osd/ceph-${osd}/block.db 3/ ceph orch daemon start osd.${osd} 4/ ceph tell osd.${osd} compact Regards, Frédéric. ----- Le 8 Juil 24, à 17:39, Frédéric Nass frederic.nass@univ-lorraine.fr a écrit :
Hello,
I just wanted to share that the following command also helped us move slow used bytes back to the fast device (without using bluefs-bdev-expand), when several compactions couldn't:
$ cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-${osd} --devs-source /var/lib/ceph/osd/ceph-${osd}/block --dev-target /var/lib/ceph/osd/ceph-${osd}/block.db
slow_used_bytes is now back to 0 on perf dump and BLUEFS_SPILLOVER alert got cleared but 'bluefs stats' is not on par:
$ ceph tell osd.451 bluefs stats 1 : device size 0x1effbfe000 : using 0x309600000(12 GiB) 2 : device size 0x746dfc00000 : using 0x3abd77d2000(3.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 22 MiB 0 B 0 B 0 B 3.9 MiB 1 WAL 0 B 33 MiB 0 B 0 B 0 B 32 MiB 2 DB 0 B 12 GiB 0 B 0 B 0 B 12 GiB 196 SLOW 0 B 4 MiB 0 B 0 B 0 B 3.8 MiB 1 TOTAL 0 B 12 GiB 0 B 0 B 0 B 0 B 200 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 17 MiB WAL 0 B 33 MiB 0 B 0 B 0 B 32 MiB DB 0 B 24 GiB 0 B 0 B 0 B 24 GiB SLOW 0 B 4 MiB 0 B 0 B 0 B 3.8 MiB TOTAL 0 B 24 GiB 0 B 0 B 0 B 0 B
SIZE << 0 B 118 GiB 6.9 TiB
Any idea? Is this something to worry about?
Regards, Frédéric.
----- Le 16 Oct 23, à 14:46, Igor Fedotov igor.fedotov@croit.io a écrit :
Hi Chris,
for the first question (osd.76) you might want to try ceph-volume's "lvm migrate --from data --target <db lvm>" command. Looks like some persistent DB remnants are still kept at main device causing the alert.
W.r.t osd.86's question - the line "SLOW 0 B 3.0 GiB 59 GiB" means that RocksDB higher levels data (usually L3+) are spread over DB and main (aka slow) devices as 3 GB and 59 GB respectively.
In other words SLOW row refers to DB data which is originally supposed to be at SLOW device (due to RocksDB data mapping mechanics). But improved bluefs logic (introduced by https://github.com/ceph/ceph/pull/29687) permitted extra DB disk usage for a part of this data.
Resizing DB volume and following DB compaction should do the trick and move all the data to DB device. Alternatively ceph-volume's lvm migrate command should do the same but the result will be rather temporary without DB volume resizing.
Hope this helps.
Thanks,
Igor
On 06/10/2023 06:55, Chris Dunlop wrote:
Hi,
tl;dr why are my osds still spilling?
I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few):
osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device
This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size.
There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand.
However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact:
# ceph tell osd.76 compact
See attached perf-dump-76 for the perf dump output:
# cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs'
In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device:
"db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K
A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?):
# ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B
SIZE << 0 B 453 GiB 14 TiB
I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76.
I also tried "ceph-kvstore-tool bluestore-kv ... stats":
$ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start }
Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it.
So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage?
And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device:
$ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B
SIZE << 0 B 57 GiB 14 TiB
Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they?
Cheers,
Chris
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io