BlueFS spilling over to SLOW even when DB has a lot of free space
Good day We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions. However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left. We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why? Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again. We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs. Below are some of our servers and the DB used / SLOW used. Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa
Hi, I always liked this page for the explanation: https://github.com/facebook/rocksdb/wiki/Leveled-Compaction Regards, Joachim www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Fr., 9. Okt. 2026 um 11:04 Uhr schrieb Jeremi-Ernst Avenant < ceph-users@ceph.io>:
Good day
We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions.
However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left.
We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why?
Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again.
We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs.
Below are some of our servers and the DB used / SLOW used.
Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Jeremi-Ernst, Just to rule out a Tentacle regression ( https://tracker.ceph.com/issues/76256) could you run: for osd in 605 599 448; do echo "=== osd.$osd"; ceph tell osd.$osd bluefs stats ceph tell osd.$osd perf dump bluefs | grep -E '"(db|slow)_used_bytes"|"max_bytes_(db|slow)"|"alloc_slow_fallback"' done and, one OSD at a time, for osd.605 and then osd.599: ceph osd set noout ceph orch daemon stop osd.605 cephadm shell --name osd.605 -- ceph-bluestore-tool show-sharding --path /var/lib/ceph/osd/ceph-605 ceph orch daemon start osd.605 ceph osd unset noout Avoid any compaction before running the above commands. Best, Frédéric -- Frédéric Nass Ceph Ambassador France, Ceph Foundation Le ven. 9 oct. 2026 à 11:04, Jeremi-Ernst Avenant <ceph-users@ceph.io> a écrit :
Good day
We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions.
However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left.
We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why?
Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again.
We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs.
Below are some of our servers and the DB used / SLOW used.
Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frédéric, Anthony and Joachim, Thanks for the suggestions. I collected the requested BlueFS statistics and counters for osd.605, osd.599 and osd.448 before the offline inspection of osd.599. The performance counters were: Counter osd.605 osd.599 osd.448 db_total_bytes 88906661888 88906661888 480101007360 db_used_bytes 588251136 664797184 68065230848 slow_total_bytes 12000134430720 12000134430720 0 slow_used_bytes 24031789056 23413325824 0 max_bytes_db 1003487232 25630343168 127384027136 max_bytes_slow 24770379776 28683862016 0 alloc_slow_fallback 0 0 0 In rounded terms, osd.605 and osd.599 each have an 82.8 GiB DB device, with only 0.55–0.62 GiB currently used there, but approximately 22 GiB used on SLOW. I stopped osd.605 and osd.599 individually and ran ceph-bluestore-tool show-sharding Both returned: failed to retrieve sharding def The cluster predates Pacific, so these may be legacy unsharded databases. Does this output support the unsharded case described in issue #76256? Before proceeding, which resharding definition would you recommend for these OSDs on 20.2.3, and should we wait for a release containing the regression fix? Regards, On Sat, Oct 10, 2026 at 7:27 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Jeremi-Ernst,
Just to rule out a Tentacle regression ( https://tracker.ceph.com/issues/76256) could you run:
for osd in 605 599 448; do echo "=== osd.$osd"; ceph tell osd.$osd bluefs stats ceph tell osd.$osd perf dump bluefs | grep -E '"(db|slow)_used_bytes"|"max_bytes_(db|slow)"|"alloc_slow_fallback"' done
and, one OSD at a time, for osd.605 and then osd.599:
ceph osd set noout ceph orch daemon stop osd.605 cephadm shell --name osd.605 -- ceph-bluestore-tool show-sharding --path /var/lib/ceph/osd/ceph-605 ceph orch daemon start osd.605 ceph osd unset noout
Avoid any compaction before running the above commands.
Best, Frédéric
-- Frédéric Nass Ceph Ambassador France, Ceph Foundation
Le ven. 9 oct. 2026 à 11:04, Jeremi-Ernst Avenant <ceph-users@ceph.io> a écrit :
Good day
We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions.
However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left.
We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why?
Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again.
We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs.
Below are some of our servers and the DB used / SLOW used.
Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa
Hi Jeremi-Ernst, Thanks, that's very useful. Yes, "failed to retrieve sharding def" means these OSDs have never been resharded, which matches the case described in issue #76256. One more reason to get https://github.com/ceph/ceph/pull/70623 backported to every maintained Ceph release. :-) To check whether the root cause is the same, we need to compare the space BlueFS has allocated with the actual size of the RocksDB files. Could you run the following command for osd.605, osd.599, and osd.448? for o in 605 599 448; do echo "=== osd.$o" ceph tell osd.$o bluefs files list --format=json-pretty | \ awk -F'[:,]' '/"size"/{s+=$2} /"(wal|db|slow)"/{a+=$2} END{printf "real file size: %.1f GiB, allocated: %.1f GiB\n", s/2^30, a/2^30}' done This is the same check as in the tracker (adding up the size fields from bluefs files list), but it also sums the allocated space from the same output so you get both numbers at once. If the allocated space is significantly larger than the real file size, it confirms the #76256 leak. Regarding resharding: there is no need to wait for the fix. The fix is not included in any release yet (including 20.2.4), and resharding greatly reduces the leak in the meantime. You can use the Tentacle default definition: m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 Make sure to proceed one OSD at a time, compacting the OSD offline or online after resharding it. Cheers, Frédéric -- Frédéric Nass Ceph Ambassador France, Senior Ceph Engineer @ CLYSO Le lun. 12 oct. 2026 à 14:00, Jeremi-Ernst Avenant <jeremi@idia.ac.za> a écrit :
Hi Frédéric, Anthony and Joachim,
Thanks for the suggestions. I collected the requested BlueFS statistics and counters for osd.605, osd.599 and osd.448 before the offline inspection of osd.599.
The performance counters were:
Counter osd.605 osd.599 osd.448 db_total_bytes 88906661888 88906661888 480101007360 db_used_bytes 588251136 664797184 68065230848 slow_total_bytes 12000134430720 12000134430720 0 slow_used_bytes 24031789056 23413325824 0 max_bytes_db 1003487232 25630343168 127384027136 max_bytes_slow 24770379776 28683862016 0 alloc_slow_fallback 0 0 0
In rounded terms, osd.605 and osd.599 each have an 82.8 GiB DB device, with only 0.55–0.62 GiB currently used there, but approximately 22 GiB used on SLOW.
I stopped osd.605 and osd.599 individually and ran ceph-bluestore-tool show-sharding
Both returned:
failed to retrieve sharding def
The cluster predates Pacific, so these may be legacy unsharded databases. Does this output support the unsharded case described in issue #76256? Before proceeding, which resharding definition would you recommend for these OSDs on 20.2.3, and should we wait for a release containing the regression fix?
Regards,
On Sat, Oct 10, 2026 at 7:27 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Jeremi-Ernst,
Just to rule out a Tentacle regression ( https://tracker.ceph.com/issues/76256) could you run:
for osd in 605 599 448; do echo "=== osd.$osd"; ceph tell osd.$osd bluefs stats ceph tell osd.$osd perf dump bluefs | grep -E '"(db|slow)_used_bytes"|"max_bytes_(db|slow)"|"alloc_slow_fallback"' done
and, one OSD at a time, for osd.605 and then osd.599:
ceph osd set noout ceph orch daemon stop osd.605 cephadm shell --name osd.605 -- ceph-bluestore-tool show-sharding --path /var/lib/ceph/osd/ceph-605 ceph orch daemon start osd.605 ceph osd unset noout
Avoid any compaction before running the above commands.
Best, Frédéric
-- Frédéric Nass Ceph Ambassador France, Ceph Foundation
Le ven. 9 oct. 2026 à 11:04, Jeremi-Ernst Avenant <ceph-users@ceph.io> a écrit :
Good day
We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions.
However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left.
We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why?
Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again.
We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs.
Below are some of our servers and the DB used / SLOW used.
Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa
Hi Frederic Thanks for the e-mail. I kicked off a resharding project last week since we have about 581 OSDs that are affected, so roughly 50% of the cluster. It is just going to take one helluva long time to work through all of them one at a time, even when automated. But I've completed all of the SSDs that host the cephfs_metadata , so no chance (hopefully) of those running full due to spillovers. I fully agree that PR should be backported :) Have a good week further. Regards On Mon, Oct 12, 2026 at 3:56 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Jeremi-Ernst,
Thanks, that's very useful.
Yes, "failed to retrieve sharding def" means these OSDs have never been resharded, which matches the case described in issue #76256. One more reason to get https://github.com/ceph/ceph/pull/70623 backported to every maintained Ceph release. :-)
To check whether the root cause is the same, we need to compare the space BlueFS has allocated with the actual size of the RocksDB files. Could you run the following command for osd.605, osd.599, and osd.448?
for o in 605 599 448; do echo "=== osd.$o" ceph tell osd.$o bluefs files list --format=json-pretty | \ awk -F'[:,]' '/"size"/{s+=$2} /"(wal|db|slow)"/{a+=$2} END{printf "real file size: %.1f GiB, allocated: %.1f GiB\n", s/2^30, a/2^30}' done
This is the same check as in the tracker (adding up the size fields from bluefs files list), but it also sums the allocated space from the same output so you get both numbers at once. If the allocated space is significantly larger than the real file size, it confirms the #76256 leak.
Regarding resharding: there is no need to wait for the fix. The fix is not included in any release yet (including 20.2.4), and resharding greatly reduces the leak in the meantime. You can use the Tentacle default definition: m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32
Make sure to proceed one OSD at a time, compacting the OSD offline or online after resharding it.
Cheers, Frédéric
-- Frédéric Nass Ceph Ambassador France, Senior Ceph Engineer @ CLYSO
Le lun. 12 oct. 2026 à 14:00, Jeremi-Ernst Avenant <jeremi@idia.ac.za> a écrit :
Hi Frédéric, Anthony and Joachim,
Thanks for the suggestions. I collected the requested BlueFS statistics and counters for osd.605, osd.599 and osd.448 before the offline inspection of osd.599.
The performance counters were:
Counter osd.605 osd.599 osd.448 db_total_bytes 88906661888 88906661888 480101007360 db_used_bytes 588251136 664797184 68065230848 slow_total_bytes 12000134430720 12000134430720 0 slow_used_bytes 24031789056 23413325824 0 max_bytes_db 1003487232 25630343168 127384027136 max_bytes_slow 24770379776 28683862016 0 alloc_slow_fallback 0 0 0
In rounded terms, osd.605 and osd.599 each have an 82.8 GiB DB device, with only 0.55–0.62 GiB currently used there, but approximately 22 GiB used on SLOW.
I stopped osd.605 and osd.599 individually and ran ceph-bluestore-tool show-sharding
Both returned:
failed to retrieve sharding def
The cluster predates Pacific, so these may be legacy unsharded databases. Does this output support the unsharded case described in issue #76256? Before proceeding, which resharding definition would you recommend for these OSDs on 20.2.3, and should we wait for a release containing the regression fix?
Regards,
On Sat, Oct 10, 2026 at 7:27 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Jeremi-Ernst,
Just to rule out a Tentacle regression ( https://tracker.ceph.com/issues/76256) could you run:
for osd in 605 599 448; do echo "=== osd.$osd"; ceph tell osd.$osd bluefs stats ceph tell osd.$osd perf dump bluefs | grep -E '"(db|slow)_used_bytes"|"max_bytes_(db|slow)"|"alloc_slow_fallback"' done
and, one OSD at a time, for osd.605 and then osd.599:
ceph osd set noout ceph orch daemon stop osd.605 cephadm shell --name osd.605 -- ceph-bluestore-tool show-sharding --path /var/lib/ceph/osd/ceph-605 ceph orch daemon start osd.605 ceph osd unset noout
Avoid any compaction before running the above commands.
Best, Frédéric
-- Frédéric Nass Ceph Ambassador France, Ceph Foundation
Le ven. 9 oct. 2026 à 11:04, Jeremi-Ernst Avenant <ceph-users@ceph.io> a écrit :
Good day
We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions.
However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left.
We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why?
Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again.
We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs.
Below are some of our servers and the DB used / SLOW used.
Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa
-- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa
participants (3)
-
Frédéric Nass
-
Jeremi-Ernst Avenant
-
Joachim Kraftmayer