Hi Frédéric, Anthony and Joachim, Thanks for the suggestions. I collected the requested BlueFS statistics and counters for osd.605, osd.599 and osd.448 before the offline inspection of osd.599. The performance counters were: Counter osd.605 osd.599 osd.448 db_total_bytes 88906661888 88906661888 480101007360 db_used_bytes 588251136 664797184 68065230848 slow_total_bytes 12000134430720 12000134430720 0 slow_used_bytes 24031789056 23413325824 0 max_bytes_db 1003487232 25630343168 127384027136 max_bytes_slow 24770379776 28683862016 0 alloc_slow_fallback 0 0 0 In rounded terms, osd.605 and osd.599 each have an 82.8 GiB DB device, with only 0.55–0.62 GiB currently used there, but approximately 22 GiB used on SLOW. I stopped osd.605 and osd.599 individually and ran ceph-bluestore-tool show-sharding Both returned: failed to retrieve sharding def The cluster predates Pacific, so these may be legacy unsharded databases. Does this output support the unsharded case described in issue #76256? Before proceeding, which resharding definition would you recommend for these OSDs on 20.2.3, and should we wait for a release containing the regression fix? Regards, On Sat, Oct 10, 2026 at 7:27 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Jeremi-Ernst,
Just to rule out a Tentacle regression ( https://tracker.ceph.com/issues/76256) could you run:
for osd in 605 599 448; do echo "=== osd.$osd"; ceph tell osd.$osd bluefs stats ceph tell osd.$osd perf dump bluefs | grep -E '"(db|slow)_used_bytes"|"max_bytes_(db|slow)"|"alloc_slow_fallback"' done
and, one OSD at a time, for osd.605 and then osd.599:
ceph osd set noout ceph orch daemon stop osd.605 cephadm shell --name osd.605 -- ceph-bluestore-tool show-sharding --path /var/lib/ceph/osd/ceph-605 ceph orch daemon start osd.605 ceph osd unset noout
Avoid any compaction before running the above commands.
Best, Frédéric
-- Frédéric Nass Ceph Ambassador France, Ceph Foundation
Le ven. 9 oct. 2026 à 11:04, Jeremi-Ernst Avenant <ceph-users@ceph.io> a écrit :
Good day
We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions.
However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left.
We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why?
Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again.
We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs.
Below are some of our servers and the DB used / SLOW used.
Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa