Fixing BlueFS spillover (pacific 16.2.14)
Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B
SIZE << 0 B 453 GiB 14 TiB
I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B
SIZE << 0 B 57 GiB 14 TiB
Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris
On Fri, Oct 06, 2023 at 02:55:22PM +1100, Chris Dunlop wrote:
Hi,
tl;dr why are my osds still spilling?
I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few):
osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device
Sigh. After battling this for days, I found the solution shortly after sending this plea for help. Via the hint from: https://tracker.ceph.com/issues/44509 using ceph-bluestore-tool bluefs-bdev-new-db results in 128k or 64k leftover on slow device The answer being, after expanding the db LV, and running the "bluefs-bdev-expand", you need to run "bluefs-bdev-migrate", e.g. all together: { lvextend -l ${size} ${lv}/${db} ${ssd_dev} cephadm unit --fsid $cid --name osd.${osd} stop cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool \ bluefs-bdev-expand \ --path /var/lib/ceph/osd/ceph-${osd} cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool \ bluefs-bdev-migrate \ --path /var/lib/ceph/osd/ceph-${osd} \ --devs-source /var/lib/ceph/osd/ceph-${osd}/block \ --dev-target /var/lib/ceph/osd/ceph-${osd}/block.db \ cephadm unit --fsid $cid --name osd.${osd} start } After the "bluefs-bdev-migrate", the "128 KiB" that was previously in the DB/SLOW location has reduced to zero: # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xe2e500000(57 GiB) 2 : device size 0xe8d7fc00000 : using 0x65598084000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 6 MiB 0 B 0 B 0 B 2.0 MiB 1 WAL 0 B 282 MiB 0 B 0 B 0 B 1.3 MiB 1 DB 0 B 56 GiB 0 B 0 B 0 B 50 GiB 841 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 57 GiB 0 B 0 B 0 B 0 B 843 MAXIMUMS: LOG 0 B 6 MiB 0 B 0 B 0 B 2.0 MiB WAL 0 B 564 MiB 0 B 0 B 0 B 56 MiB DB 0 B 56 GiB 0 B 0 B 0 B 50 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 57 GiB 0 B 0 B 0 B 0 B
SIZE << 0 B 453 GiB 14 TiB
...and the "osd.xx spilled over" has disappeared from my "ceph health detail". Now to do the others... Cheers, Chris
Hi Chris, for the first question (osd.76) you might want to try ceph-volume's "lvm migrate --from data --target <db lvm>" command. Looks like some persistent DB remnants are still kept at main device causing the alert. W.r.t osd.86's question - the line "SLOW 0 B 3.0 GiB 59 GiB" means that RocksDB higher levels data (usually L3+) are spread over DB and main (aka slow) devices as 3 GB and 59 GB respectively. In other words SLOW row refers to DB data which is originally supposed to be at SLOW device (due to RocksDB data mapping mechanics). But improved bluefs logic (introduced by https://github.com/ceph/ceph/pull/29687) permitted extra DB disk usage for a part of this data. Resizing DB volume and following DB compaction should do the trick and move all the data to DB device. Alternatively ceph-volume's lvm migrate command should do the same but the result will be rather temporary without DB volume resizing. Hope this helps. Thanks, Igor On 06/10/2023 06:55, Chris Dunlop wrote:
Hi,
tl;dr why are my osds still spilling?
I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few):
osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device
This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size.
There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand.
However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact:
# ceph tell osd.76 compact
See attached perf-dump-76 for the perf dump output:
# cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs'
In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device:
"db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K
A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?):
# ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B
SIZE << 0 B 453 GiB 14 TiB
I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76.
I also tried "ceph-kvstore-tool bluestore-kv ... stats":
$ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start }
Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it.
So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage?
And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device:
$ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B
SIZE << 0 B 57 GiB 14 TiB
Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they?
Cheers,
Chris
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Igor, Thanks for the suggestions. You may have already seen my followup message where the solution was to use "ceph-bluestore-tool bluefs-bdev-migrate" to get the lingering 128KiB of data moved from the slow to the fast device. I wonder if your suggested "ceph-volume lvm migrate" would do the same. Notably, DB compaction didn't help in my case. Cheers, Chris On Mon, Oct 16, 2023 at 03:46:17PM +0300, Igor Fedotov wrote:
Hi Chris,
for the first question (osd.76) you might want to try ceph-volume's "lvm migrate --from data --target <db lvm>" command. Looks like some persistent DB remnants are still kept at main device causing the alert.
W.r.t osd.86's question - the line "SLOW 0 B 3.0 GiB 59 GiB" means that RocksDB higher levels data (usually L3+) are spread over DB and main (aka slow) devices as 3 GB and 59 GB respectively.
In other words SLOW row refers to DB data which is originally supposed to be at SLOW device (due to RocksDB data mapping mechanics). But improved bluefs logic (introduced by https://github.com/ceph/ceph/pull/29687) permitted extra DB disk usage for a part of this data.
Resizing DB volume and following DB compaction should do the trick and move all the data to DB device. Alternatively ceph-volume's lvm migrate command should do the same but the result will be rather temporary without DB volume resizing.
Hope this helps.
Thanks,
Igor
Hello Chris, Igor, I came here to say two things. Firstly, thank you for this thread. I've not run perf dump or bluefs stats before and found it helpful in diagnosing the same problem you had. Secondly, yes 'ceph-volume lvm migrate' was effective (in Quincy 17.2.7) to finalise the migration of RocksDB data like this. We discovered an OSD without a flash RocksDB, and used 'ceph-volume lvm new-db' to set it up. That resulted in the spillover warning and led me here. Fixed now. Regards, Greg. On 18/10/23 08:20, Chris Dunlop wrote:
Hi Igor,
Thanks for the suggestions. You may have already seen my followup message where the solution was to use "ceph-bluestore-tool bluefs-bdev-migrate" to get the lingering 128KiB of data moved from the slow to the fast device. I wonder if your suggested "ceph-volume lvm migrate" would do the same.
Notably, DB compaction didn't help in my case.
Cheers,
Chris
On Mon, Oct 16, 2023 at 03:46:17PM +0300, Igor Fedotov wrote:
Hi Chris,
for the first question (osd.76) you might want to try ceph-volume's "lvm migrate --from data --target <db lvm>" command. Looks like some persistent DB remnants are still kept at main device causing the alert.
W.r.t osd.86's question - the line "SLOW 0 B 3.0 GiB 59 GiB" means that RocksDB higher levels data (usually L3+) are spread over DB and main (aka slow) devices as 3 GB and 59 GB respectively.
In other words SLOW row refers to DB data which is originally supposed to be at SLOW device (due to RocksDB data mapping mechanics). But improved bluefs logic (introduced by https://github.com/ceph/ceph/pull/29687) permitted extra DB disk usage for a part of this data.
Resizing DB volume and following DB compaction should do the trick and move all the data to DB device. Alternatively ceph-volume's lvm migrate command should do the same but the result will be rather temporary without DB volume resizing.
Hope this helps.
Thanks,
Igor
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Glad to be of help, and thanks for confirming the 'lvm migrate' works. Cheers, Chris On Sun, Jun 30, 2024 at 05:23:42PM +0800, Gregory Orange wrote:
Hello Chris, Igor,
I came here to say two things.
Firstly, thank you for this thread. I've not run perf dump or bluefs stats before and found it helpful in diagnosing the same problem you had.
Secondly, yes 'ceph-volume lvm migrate' was effective (in Quincy 17.2.7) to finalise the migration of RocksDB data like this. We discovered an OSD without a flash RocksDB, and used 'ceph-volume lvm new-db' to set it up. That resulted in the spillover warning and led me here. Fixed now.
Regards, Greg.
Hello, I just wanted to share that the following command also helped us move slow used bytes back to the fast device (without using bluefs-bdev-expand), when several compactions couldn't: $ cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-${osd} --devs-source /var/lib/ceph/osd/ceph-${osd}/block --dev-target /var/lib/ceph/osd/ceph-${osd}/block.db slow_used_bytes is now back to 0 on perf dump and BLUEFS_SPILLOVER alert got cleared but 'bluefs stats' is not on par: $ ceph tell osd.451 bluefs stats 1 : device size 0x1effbfe000 : using 0x309600000(12 GiB) 2 : device size 0x746dfc00000 : using 0x3abd77d2000(3.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 22 MiB 0 B 0 B 0 B 3.9 MiB 1 WAL 0 B 33 MiB 0 B 0 B 0 B 32 MiB 2 DB 0 B 12 GiB 0 B 0 B 0 B 12 GiB 196 SLOW 0 B 4 MiB 0 B 0 B 0 B 3.8 MiB 1 TOTAL 0 B 12 GiB 0 B 0 B 0 B 0 B 200 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 17 MiB WAL 0 B 33 MiB 0 B 0 B 0 B 32 MiB DB 0 B 24 GiB 0 B 0 B 0 B 24 GiB SLOW 0 B 4 MiB 0 B 0 B 0 B 3.8 MiB TOTAL 0 B 24 GiB 0 B 0 B 0 B 0 B
SIZE << 0 B 118 GiB 6.9 TiB
Any idea? Is this something to worry about? Regards, Frédéric. ----- Le 16 Oct 23, à 14:46, Igor Fedotov igor.fedotov@croit.io a écrit :
Hi Chris,
for the first question (osd.76) you might want to try ceph-volume's "lvm migrate --from data --target <db lvm>" command. Looks like some persistent DB remnants are still kept at main device causing the alert.
W.r.t osd.86's question - the line "SLOW 0 B 3.0 GiB 59 GiB" means that RocksDB higher levels data (usually L3+) are spread over DB and main (aka slow) devices as 3 GB and 59 GB respectively.
In other words SLOW row refers to DB data which is originally supposed to be at SLOW device (due to RocksDB data mapping mechanics). But improved bluefs logic (introduced by https://github.com/ceph/ceph/pull/29687) permitted extra DB disk usage for a part of this data.
Resizing DB volume and following DB compaction should do the trick and move all the data to DB device. Alternatively ceph-volume's lvm migrate command should do the same but the result will be rather temporary without DB volume resizing.
Hope this helps.
Thanks,
Igor
On 06/10/2023 06:55, Chris Dunlop wrote:
Hi,
tl;dr why are my osds still spilling?
I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few):
osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device
This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size.
There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand.
However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact:
# ceph tell osd.76 compact
See attached perf-dump-76 for the perf dump output:
# cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs'
In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device:
"db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K
A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?):
# ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B
SIZE << 0 B 453 GiB 14 TiB
I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76.
I also tried "ceph-kvstore-tool bluestore-kv ... stats":
$ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start }
Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it.
So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage?
And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device:
$ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B
SIZE << 0 B 57 GiB 14 TiB
Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they?
Cheers,
Chris
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
One more manual compaction updated bluefs stats figures accordingly. So at the end, it is: 1/ ceph orch daemon stop osd.${osd} 2/ cephadm shell --fsid $(ceph fsid) --name osd.${osd} -- ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-${osd} --devs-source /var/lib/ceph/osd/ceph-${osd}/block --dev-target /var/lib/ceph/osd/ceph-${osd}/block.db 3/ ceph orch daemon start osd.${osd} 4/ ceph tell osd.${osd} compact Regards, Frédéric. ----- Le 8 Juil 24, à 17:39, Frédéric Nass frederic.nass@univ-lorraine.fr a écrit :
Hello,
I just wanted to share that the following command also helped us move slow used bytes back to the fast device (without using bluefs-bdev-expand), when several compactions couldn't:
$ cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-${osd} --devs-source /var/lib/ceph/osd/ceph-${osd}/block --dev-target /var/lib/ceph/osd/ceph-${osd}/block.db
slow_used_bytes is now back to 0 on perf dump and BLUEFS_SPILLOVER alert got cleared but 'bluefs stats' is not on par:
$ ceph tell osd.451 bluefs stats 1 : device size 0x1effbfe000 : using 0x309600000(12 GiB) 2 : device size 0x746dfc00000 : using 0x3abd77d2000(3.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 22 MiB 0 B 0 B 0 B 3.9 MiB 1 WAL 0 B 33 MiB 0 B 0 B 0 B 32 MiB 2 DB 0 B 12 GiB 0 B 0 B 0 B 12 GiB 196 SLOW 0 B 4 MiB 0 B 0 B 0 B 3.8 MiB 1 TOTAL 0 B 12 GiB 0 B 0 B 0 B 0 B 200 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 17 MiB WAL 0 B 33 MiB 0 B 0 B 0 B 32 MiB DB 0 B 24 GiB 0 B 0 B 0 B 24 GiB SLOW 0 B 4 MiB 0 B 0 B 0 B 3.8 MiB TOTAL 0 B 24 GiB 0 B 0 B 0 B 0 B
SIZE << 0 B 118 GiB 6.9 TiB
Any idea? Is this something to worry about?
Regards, Frédéric.
----- Le 16 Oct 23, à 14:46, Igor Fedotov igor.fedotov@croit.io a écrit :
Hi Chris,
for the first question (osd.76) you might want to try ceph-volume's "lvm migrate --from data --target <db lvm>" command. Looks like some persistent DB remnants are still kept at main device causing the alert.
W.r.t osd.86's question - the line "SLOW 0 B 3.0 GiB 59 GiB" means that RocksDB higher levels data (usually L3+) are spread over DB and main (aka slow) devices as 3 GB and 59 GB respectively.
In other words SLOW row refers to DB data which is originally supposed to be at SLOW device (due to RocksDB data mapping mechanics). But improved bluefs logic (introduced by https://github.com/ceph/ceph/pull/29687) permitted extra DB disk usage for a part of this data.
Resizing DB volume and following DB compaction should do the trick and move all the data to DB device. Alternatively ceph-volume's lvm migrate command should do the same but the result will be rather temporary without DB volume resizing.
Hope this helps.
Thanks,
Igor
On 06/10/2023 06:55, Chris Dunlop wrote:
Hi,
tl;dr why are my osds still spilling?
I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few):
osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device
This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size.
There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand.
However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact:
# ceph tell osd.76 compact
See attached perf-dump-76 for the perf dump output:
# cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs'
In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device:
"db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K
A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?):
# ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B
SIZE << 0 B 453 GiB 14 TiB
I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76.
I also tried "ceph-kvstore-tool bluestore-kv ... stats":
$ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start }
Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it.
So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage?
And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device:
$ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B
SIZE << 0 B 57 GiB 14 TiB
Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they?
Cheers,
Chris
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Chris Dunlop
-
Frédéric Nass
-
Gregory Orange
-
Igor Fedotov