Hello Adam, Can you also share output of 'ceph osd df tree' or 'ceph osd tree' and 'ceph osd crush rule dump'? Regards, On 5.05.2025 14:37, Adam Prycki wrote:

Hello,

I don't think bluestore OSDs are affected by fstrim. Running fstrim -va on our OSD nodes returns no output related to OSDs. Afaik it affects only linux kenrel filesystems.

Best regardsm Adam

W dniu 5.05.2025 o 14:27, Jörg Model pisze:

...

Hello Adam,

without checking any deeper your problem:  maybe the filesystem on your ssds need a fstrim?

See https://www.man7.org/linux/man-pages/man8/fstrim.8.html

I remember a similar issue on my test ceph cluster...

Regards Jörg

Am Montag, dem 05.05.2025 um 13:48 +0200 schrieb Adam Prycki:

...

Hello,

we have just discovered new issue with our cephfs cluster. All of our SSD osd have been flagged as nearful.

Ceph df on our cluster looks very wrong. Raw storage usage shows that SSDs are almost full, but SSD pools are hardly using any space.

ceph df --- RAW STORAGE --- CLASS    SIZE   AVAIL    USED  RAW USED  %RAW USED hdd    27 PiB  16 PiB  11 PiB    11 PiB      40.57 ssd    71 TiB  10 TiB  60 TiB    60 TiB      85.49 TOTAL  27 PiB  16 PiB  11 PiB    11 PiB      40.68

--- POOLS --- POOL                       ID    PGS   STORED  OBJECTS USED %USED MAX AVAIL .mgr                        1     16   47 GiB   12.12k  142 GiB 2.48    1.8 TiB cephfs.arm-vol.meta         2   4096   19 GiB    2.02M   57 GiB 1.01    1.8 TiB cephfs.arm-vol.data         4  16384  7.7 PiB    2.14G   10 PiB 58.29    5.5 PiB .rgw.root                  24     32   16 KiB       24  264 KiB 0    2.4 PiB bst-ro-backup.rgw.log      25     32  343 MiB    2.90k  1.0 GiB 0    2.4 PiB bst-ro-backup.rgw.control  26     32      0 B        8      0 B 0    2.4 PiB bst-ro-backup.rgw.meta     27     32  198 KiB      440  5.0 MiB 0    2.4 PiB .nfs                       32     32  8.2 KiB       10   99 KiB 0    2.4 PiB

cephfs.arm-vol.meta pool seems to store 19GiB of data. rados -p cephfs.arm-vol.meta | wc -l confirms that there are about 2.02M objects in that pool. Our SSD OSDs are almost full

ceph osd df ssd ID    CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA OMAP META       AVAIL   %USE   VAR   PGS  STATUS    807    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB 395 MiB 1.3 GiB  64 GiB  85.78  1.00   85      up    812    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB 364 MiB 2.0 GiB  66 GiB  85.23  1.00   74      up    569    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB 347 MiB 2.0 GiB  66 GiB  85.24  1.00   72      up    577    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB 334 MiB 1.5 GiB  67 GiB  85.12  1.00   72      up    571    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB 369 MiB 2.3 GiB  66 GiB  85.30  1.00   86      up    578    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB 329 MiB 1.8 GiB  66 GiB  85.19  1.00   68      up 1775    ssd  0.43660   1.00000  447 GiB  383 GiB  382 GiB  379 MiB 1.2 GiB  64 GiB  85.70  1.00   76      up 1838    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  358 MiB 2.3 GiB  66 GiB  85.29  1.00   74      up 1783    ssd  0.43660   1.00000  447 GiB  385 GiB  382 GiB  443 MiB 2.5 GiB  62 GiB  86.05  1.01   98      up 1846    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  382 MiB 1.7 GiB  66 GiB  85.18  1.00   85      up 1854    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  384 MiB 2.4 GiB  66 GiB  85.33  1.00   80      up 1855    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  364 MiB 2.6 GiB  65 GiB  85.37  1.00   78      up 1761    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  368 MiB 1.7 GiB  63 GiB  85.86  1.00   79      up 1826    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  331 MiB 2.2 GiB  66 GiB  85.29  1.00   73      up 1773    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  324 MiB 2.0 GiB  66 GiB  85.24  1.00   73      up 1836    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  390 MiB 2.3 GiB  66 GiB  85.30  1.00   82      up 1771    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  340 MiB 2.3 GiB  66 GiB  85.31  1.00   75      up 1834    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  376 MiB 2.1 GiB  66 GiB  85.27  1.00   79      up 2111    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  396 MiB 2.0 GiB  63 GiB  85.90  1.01   84      up 2112    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  338 MiB 2.2 GiB  66 GiB  85.27  1.00   77      up 2272    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  374 MiB 2.0 GiB  66 GiB  85.24  1.00   76      up 2279    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  332 MiB 2.5 GiB  66 GiB  85.34  1.00   77      up 2276    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  360 MiB 2.4 GiB  66 GiB  85.34  1.00   82      up 2285    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  371 MiB 2.2 GiB  66 GiB  85.27  1.00   79      up 2275    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  368 MiB 2.3 GiB  66 GiB  85.31  1.00   79      up 2282    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  336 MiB 2.0 GiB  66 GiB  85.23  1.00   75      up 2271    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  343 MiB 1.6 GiB  66 GiB  85.15  1.00   81      up 2278    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  356 MiB 1.7 GiB  63 GiB  85.85  1.00   77      up 2277    ssd  0.43660   1.00000  447 GiB  383 GiB  382 GiB  334 MiB 1.3 GiB  64 GiB  85.70  1.00   72      up 2286    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  351 MiB 2.3 GiB  66 GiB  85.30  1.00   74      up 2347    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  362 MiB 2.6 GiB  65 GiB  85.38  1.00   81      up 2348    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  339 MiB 2.0 GiB  66 GiB  85.23  1.00   76      up    573    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB 404 MiB 2.7 GiB  65 GiB  85.40  1.00   85      up    580    ssd  0.43660   1.00000  447 GiB  387 GiB  385 GiB 320 MiB 2.3 GiB  60 GiB  86.65  1.01   70      up    169    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB 415 MiB 2.0 GiB  66 GiB  85.24  1.00   88      up    171    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB 323 MiB 2.5 GiB  66 GiB  85.34  1.00   71      up 1216    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  364 MiB 2.5 GiB  66 GiB  85.35  1.00   76      up 1218    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  334 MiB 1.9 GiB  63 GiB  85.84  1.00   70      up    811    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB 377 MiB 2.0 GiB  66 GiB  85.24  1.00   80      up    815    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB 419 MiB 2.6 GiB  65 GiB  85.38  1.00   91      up    810    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB 312 MiB 2.2 GiB  66 GiB  85.27  1.00   67      up    813    ssd  0.43660   1.00000  447 GiB  384 GiB  381 GiB 340 MiB 2.4 GiB  63 GiB  85.93  1.01   72      up 1760    ssd  0.43660   1.00000  447 GiB  380 GiB  379 GiB  388 MiB 974 MiB  67 GiB  85.01  0.99   77      up 1823    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  366 MiB 2.5 GiB  66 GiB  85.35  1.00   73      up 1762    ssd  0.43660   1.00000  447 GiB  380 GiB  379 GiB  361 MiB 1.4 GiB  67 GiB  85.10  1.00   75      up 1824    ssd  0.43660   1.00000  447 GiB  385 GiB  382 GiB  278 MiB 2.6 GiB  62 GiB  86.06  1.01   65      up 1766    ssd  0.43660   1.00000  447 GiB  387 GiB  385 GiB  377 MiB 1.9 GiB  60 GiB  86.58  1.01   82      up 1830    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  339 MiB 2.6 GiB  65 GiB  85.36  1.00   74      up 1755    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  388 MiB 2.5 GiB  65 GiB  85.36  1.00   77      up 1818    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  340 MiB 2.3 GiB  66 GiB  85.31  1.00   75      up 1774    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  315 MiB 2.2 GiB  66 GiB  85.26  1.00   67      up 1837    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  311 MiB 2.5 GiB  66 GiB  85.33  1.00   68      up 1748    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  339 MiB 2.4 GiB  66 GiB  85.31  1.00   76      up 1812    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  336 MiB 2.2 GiB  66 GiB  85.28  1.00   77      up 1337    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  312 MiB 2.3 GiB  66 GiB  85.30  1.00   64      up 1338    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  331 MiB 1.9 GiB  63 GiB  85.89  1.00   76      up 1091    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  329 MiB 2.5 GiB  66 GiB  85.34  1.00   69      up 1096    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  373 MiB 1.6 GiB  63 GiB  85.85  1.00   82      up 1093    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  379 MiB 2.6 GiB  65 GiB  85.38  1.00   84      up 1098    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  332 MiB 2.3 GiB  66 GiB  85.30  1.00   79      up 1752    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  350 MiB 1.5 GiB  67 GiB  85.11  1.00   80      up 1815    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  349 MiB 2.3 GiB  66 GiB  85.30  1.00   70      up 1754    ssd  0.43660   1.00000  447 GiB  387 GiB  384 GiB  362 MiB 2.0 GiB  60 GiB  86.48  1.01   75      up 1816    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  346 MiB 2.5 GiB  66 GiB  85.35  1.00   72      up 1753    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  348 MiB 1.6 GiB  64 GiB  85.80  1.00   75      up 1817    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  356 MiB 2.4 GiB  66 GiB  85.33  1.00   75      up 1739    ssd  0.43660   1.00000  447 GiB  380 GiB  379 GiB  363 MiB 1.3 GiB  67 GiB  85.09  1.00   75      up 1804    ssd  0.43660   1.00000  447 GiB  385 GiB  382 GiB  365 MiB 2.6 GiB  62 GiB  86.03  1.01   78      up 1763    ssd  0.43660   1.00000  447 GiB  385 GiB  382 GiB  369 MiB 2.6 GiB  62 GiB  86.04  1.01   76      up 1825    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  379 MiB 2.2 GiB  63 GiB  85.91  1.01   83      up 1738    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  354 MiB 1.9 GiB  63 GiB  85.84  1.00   75      up 1802    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  339 MiB 1.6 GiB  63 GiB  85.83  1.00   77      up 1661    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  320 MiB 2.4 GiB  66 GiB  85.33  1.00   65      up 1666    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  367 MiB 2.3 GiB  66 GiB  85.31  1.00   80      up 1663    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  444 MiB 1.9 GiB  63 GiB  85.91  1.01   98      up 1669    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  375 MiB 2.2 GiB  66 GiB  85.29  1.00   76      up 1665    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  341 MiB 2.1 GiB  66 GiB  85.26  1.00   75      up 1671    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  363 MiB 2.4 GiB  66 GiB  85.33  1.00   76      up 1726    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  356 MiB 1.7 GiB  64 GiB  85.78  1.00   77      up 1790    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  349 MiB 2.3 GiB  66 GiB  85.29  1.00   80      up 1765    ssd  0.43660   1.00000  447 GiB  385 GiB  382 GiB  437 MiB 2.5 GiB  62 GiB  86.07  1.01   91      up 1828    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  356 MiB 2.3 GiB  66 GiB  85.31  1.00   78      up 1744    ssd  0.43660   1.00000  447 GiB  385 GiB  382 GiB  355 MiB 2.5 GiB  62 GiB  86.03  1.01   77      up 1807    ssd  0.43660   1.00000  447 GiB  387 GiB  385 GiB  327 MiB 1.9 GiB  60 GiB  86.51  1.01   74      up 1768    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  415 MiB 2.2 GiB  66 GiB  85.29  1.00   88      up 1831    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  332 MiB 2.0 GiB  66 GiB  85.23  1.00   71      up 1756    ssd  0.43660   1.00000  447 GiB  380 GiB  379 GiB  339 MiB 1.3 GiB  67 GiB  85.07  1.00   65      up 1819    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  323 MiB 2.3 GiB  66 GiB  85.28  1.00   67      up 1772    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  407 MiB 1.8 GiB  66 GiB  85.21  1.00   87      up 1835    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  340 MiB 1.5 GiB  67 GiB  85.11  1.00   74      up 1879    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  342 MiB 2.5 GiB  66 GiB  85.34  1.00   70      up 1882    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  310 MiB 2.2 GiB  66 GiB  85.28  1.00   66      up 1880    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  344 MiB 2.6 GiB  65 GiB  85.36  1.00   79      up 1883    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  319 MiB 2.6 GiB  65 GiB  85.36  1.00   65      up 1881    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  313 MiB 2.1 GiB  66 GiB  85.25  1.00   64      up 1884    ssd  0.43660   1.00000  447 GiB  390 GiB  388 GiB  387 MiB 2.1 GiB  57 GiB  87.27  1.02   87      up 1747    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  426 MiB 2.6 GiB  65 GiB  85.38  1.00   87      up 1809    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  359 MiB 1.6 GiB  66 GiB  85.14  1.00   77      up 1751    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  340 MiB 2.6 GiB  65 GiB  85.36  1.00   74      up 1813    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  367 MiB 2.2 GiB  63 GiB  85.93  1.01   81      up 1785    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  403 MiB 2.5 GiB  66 GiB  85.35  1.00   81      up 1848    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  370 MiB 2.5 GiB  66 GiB  85.34  1.00   74      up 1749    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  337 MiB 1.5 GiB  67 GiB  85.12  1.00   74      up 1811    ssd  0.43660   1.00000  447 GiB  388 GiB  385 GiB  335 MiB 2.5 GiB  59 GiB  86.70  1.01   74      up 1725    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  361 MiB 2.5 GiB  65 GiB  85.35  1.00   73      up 1788    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  352 MiB 2.6 GiB  65 GiB  85.36  1.00   74      up 1779    ssd  0.43660   1.00000  447 GiB  380 GiB  379 GiB  363 MiB 1.1 GiB  67 GiB  85.03  0.99   77      up 1843    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  387 MiB 2.4 GiB  66 GiB  85.34  1.00   78      up 1967    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  333 MiB 2.2 GiB  66 GiB  85.28  1.00   66      up 1970    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  382 MiB 1.9 GiB  66 GiB  85.21  1.00   78      up 1968    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  337 MiB 1.9 GiB  66 GiB  85.22  1.00   71      up 1971    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  354 MiB 2.1 GiB  66 GiB  85.26  1.00   79      up 1966    ssd  0.43660   1.00000  447 GiB  384 GiB  381 GiB  352 MiB 1.8 GiB  64 GiB  85.79  1.00   82      up 1969    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  315 MiB 2.3 GiB  63 GiB  85.99  1.01   73      up 1781    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  381 MiB 1.8 GiB  66 GiB  85.20  1.00   81      up 1844    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  330 MiB 2.1 GiB  66 GiB  85.25  1.00   71      up 1743    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  360 MiB 2.1 GiB  63 GiB  85.95  1.01   79      up 1806    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  385 MiB 2.4 GiB  66 GiB  85.34  1.00   78      up 1778    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  354 MiB 2.5 GiB  66 GiB  85.35  1.00   74      up 1839    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  348 MiB 1.6 GiB  66 GiB  85.15  1.00   79      up 1757    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  409 MiB 2.5 GiB  65 GiB  85.36  1.00   82      up 1820    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  335 MiB 2.1 GiB  66 GiB  85.26  1.00   73      up 1758    ssd  0.43660   1.00000  447 GiB  385 GiB  382 GiB  403 MiB 2.5 GiB  63 GiB  86.02  1.01   81      up 1821    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  364 MiB 2.1 GiB  66 GiB  85.27  1.00   80      up 1787    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  281 MiB 2.1 GiB  66 GiB  85.24  1.00   64      up 1850    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  260 MiB 2.6 GiB  65 GiB  85.36  1.00   59      up 2054    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  318 MiB 1.6 GiB  66 GiB  85.14  1.00   68      up 2057    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  364 MiB 2.3 GiB  66 GiB  85.31  1.00   78      up 2055    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  343 MiB 2.1 GiB  66 GiB  85.26  1.00   79      up 2058    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  289 MiB 2.5 GiB  63 GiB  85.99  1.01   65      up 2053    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  335 MiB 2.0 GiB  66 GiB  85.23  1.00   74      up 2056    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  362 MiB 2.3 GiB  63 GiB  85.93  1.01   77      up 1735    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  363 MiB 1.8 GiB  66 GiB  85.20  1.00   76      up 1799    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  360 MiB 2.1 GiB  66 GiB  85.27  1.00   74      up 1784    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  361 MiB 1.8 GiB  66 GiB  85.19  1.00   74      up 1847    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  370 MiB 2.4 GiB  66 GiB  85.33  1.00   74      up 1740    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  368 MiB 2.4 GiB  66 GiB  85.33  1.00   72      up 1801    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  362 MiB 2.1 GiB  66 GiB  85.26  1.00   74      up 1776    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  367 MiB 2.1 GiB  66 GiB  85.25  1.00   80      up 1841    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  393 MiB 2.1 GiB  66 GiB  85.26  1.00   86      up 1742    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  358 MiB 1.6 GiB  66 GiB  85.14  1.00   74      up 1805    ssd  0.43660   1.00000  447 GiB  385 GiB  382 GiB  330 MiB 2.6 GiB  62 GiB  86.05  1.01   75      up 1764    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  347 MiB 2.4 GiB  66 GiB  85.32  1.00   74      up 1827    ssd  0.43660   1.00000  447 GiB  385 GiB  382 GiB  434 MiB 2.3 GiB  63 GiB  86.00  1.01   88      up 2301    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  355 MiB 2.3 GiB  66 GiB  85.30  1.00   79      up 2309    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  315 MiB 1.7 GiB  66 GiB  85.17  1.00   69      up 2302    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  314 MiB 2.5 GiB  66 GiB  85.33  1.00   68      up 2315    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  333 MiB 2.3 GiB  66 GiB  85.30  1.00   70      up 2308    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  356 MiB 2.4 GiB  66 GiB  85.33  1.00   79      up 2316    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  366 MiB 2.3 GiB  66 GiB  85.29  1.00   76      up 1733    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  348 MiB 1.9 GiB  63 GiB  85.86  1.00   74      up 1796    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  352 MiB 2.6 GiB  65 GiB  85.38  1.00   74      up 1782    ssd  0.43660   1.00000  447 GiB  383 GiB  382 GiB  313 MiB 1.4 GiB  64 GiB  85.77  1.00   70      up 1845    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  379 MiB 2.2 GiB  66 GiB  85.29  1.00   81      up 1730    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  347 MiB 2.2 GiB  66 GiB  85.29  1.00   74      up 1794    ssd  0.43660   1.00000  447 GiB  384 GiB  382 GiB  377 MiB 2.1 GiB  63 GiB  85.94  1.01   80      up 1786    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  381 MiB 2.6 GiB  65 GiB  85.37  1.00   76      up 1849    ssd  0.43660   1.00000  447 GiB  383 GiB  382 GiB  337 MiB 963 MiB  64 GiB  85.63  1.00   76      up 1741    ssd  0.43660   1.00000  447 GiB  380 GiB  379 GiB  358 MiB 1.4 GiB  67 GiB  85.09  1.00   75      up 1803    ssd  0.43660   1.00000  447 GiB  382 GiB  379 GiB  383 MiB 2.7 GiB  65 GiB  85.40  1.00   78      up 1750    ssd  0.43660   1.00000  447 GiB  381 GiB  379 GiB  353 MiB 2.1 GiB  66 GiB  85.26  1.00   70      up 1814    ssd  0.43660   1.00000  447 GiB  380 GiB  379 GiB  351 MiB 772 MiB  67 GiB  84.95  0.99   81      up

Both ssd pools run with basic 3x replication and rack failure domain. cephfs.arm-vol.data pool is EC pool on HDD OSDs. I've checked few OSDs and pools are not overlapping over.

Cluster runs on 19.2.0 and OSD nodes are ARM64. In emergency I'm planning to just move these pool to HDDs OSD.

Do you have any idea what could have filled these SSDs and how to clean it up?

Today I've

Best regards Adam Prycki _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Sorry guys, I've forgotten to add attachments to my previous email. In attachments: ceph osd crush rule dump ceph osd df tree Best regards Adam Prycki

Hi Igor, It looks like you're right. ceph tell osd.1750 status { "cluster_fsid": "bec60cda-a306-11ed-abd9-75488d4e8f4a", "osd_fsid": "388906f8-1df8-45a2-9895-067ee2e0c055", "whoami": 1750, "state": "active", "maps": "[1502335~261370]", "oldest_map": "1502335", "newest_map": "1763704", "cluster_osdmap_trim_lower_bound": 1502335, "num_pgs": 0 } One of the OSDs I've checked has about 261K maps Could this cause bluestore to grow to ~380GiB? What settings could affect number of mapps stored on OSD. Only think that comes to mind is mon_min_osdmap_epochs which I configured to 2000 a while ago. We are also running long rebalancing on this cluster. At the beginning of the year we've expended this cluster and we are still waiting for HDD pool to finish rebalancing. We have only 2% of data left to rebalance. It's taking so long because I wanted to stay with mclock scheduler at first. Rebalancing shouldn't affect SSD pool directly, but it keeps hdd pools from being 100% active+clean Evacuating all PGs from OSD doesn't lower it's usage. For example 1814 ssd 0 1.00000 447 GiB 382 GiB 381 GiB 351 MiB 1.0 GiB 65 GiB 85.44 1.00 0 up I assume it's bluestore and not PG/Crush related issue. Best regards Adam W dniu 5.05.2025 o 16:20, Igor Fedotov pisze:

Hi Adam,

I recall a case where pretty high OSD utilization was caused by no trimming for osdmaps.

You might want to check the output of 'ceph tell osd.N status' command and compare oldest_map vs. newest_map values.

Having a delta in hundreds of thousands of epochs would confirm the case.

Thanks,

Igor

On 05.05.2025 16:41, Adam Prycki wrote:

...

Sorry guys, I've forgotten to add attachments to my previous email.

In attachments: ceph osd crush rule dump ceph osd df tree

Best regards Adam Prycki

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hello Igor

...

It looks like you're right.

ceph tell osd.1750 status { "cluster_fsid": "bec60cda-a306-11ed-abd9-75488d4e8f4a", "osd_fsid": "388906f8-1df8-45a2-9895-067ee2e0c055", "whoami": 1750, "state": "active", "maps": "[1502335~261370]", "oldest_map": "1502335", "newest_map": "1763704", "cluster_osdmap_trim_lower_bound": 1502335, "num_pgs": 0 }

One of the OSDs I've checked has about 261K maps Could this cause bluestore to grow to ~380GiB?

You better check objects size/count in meta pool using ceph-objectstore- tool and estimate the total numbers:

ceph-objectstore-tool --data-path <path-to-osd> --op meta-list > meta_list; cat meta_list | wc

ceph-objectstore-tool --data-path <path-to-osd> --pgid meta <oid> dump | grep size

The latter command can obtain onode size for a given object - just use a few oids corresponding to specific onode types (osdmap* and inc_osdmap* are of particular interest, other types to be checked if they are in bulky counts) from meta_list file.

There is a lot of objects on the OSD. ceph-objectstore-tool --data-path ./osd.1750 --op meta-list | wc -l 528023 (Almost all are osdmap or inc_osdmap) On smaller healthy cluster similar command gives me about 1400 objects I've checked few osdmap.* objects they are also 6-7 times bigger than osdmap.* objects on smaller healthy cluster. examples: ceph-objectstore-tool --data-path ./osd.1750 --pgid meta '{"oid":"osdmap.1612379","key":"","snapid":0,"hash":42339 99933,"max":0,"pool":-1,"namespace":"","max":0}' dump |grep size Error getting attr on : meta,#-1:bc6dba3f:::osdmap.1612379:0#, (61) No data available "size": 1622339, "blksize": 4096, "size": 1622339, "expected_object_size": 0, "expected_write_size": 0, ceph-objectstore-tool --data-path ./osd.1750 --pgid meta '{"oid":"osdmap.1524595","key":"","snapid":0,"hash":4208913981,"max":0,"pool":-1,"namespace":"","max":0}' dump |grep size Error getting attr on : meta,#-1:bc777b5f:::osdmap.1524595:0#, (61) No data available "size": 1922137, "blksize": 4096, "size": 1922137, "expected_object_size": 0, "expected_write_size": 0,

...

What settings could affect number of mapps stored on OSD. Only think that comes to mind is mon_min_osdmap_epochs which I configured to 2000 a while ago.

osdmap-s should be trimmed automatically in a healthy cluster. Perhaps an ongoing rebalancing prevents or some other issue from that. The first question would be how osdmap epochs evolve? Is oldest_map increasing? Is the delta decreasing?

oldest_map is not increasing. Delta has slightly increased since yesterday by about 2763. SSD OSD usage increased by about 1% yesterday. We are quite close to finishing backfill. I'm expecting to get active+clean cluster before OSD fill up. data: volumes: 1/1 healthy pools: 8 pools, 20641 pgs objects: 2.15G objects, 7.6 PiB usage: 11 PiB used, 16 PiB / 27 PiB avail pgs: 196301608/17155230197 objects misplaced (1.144%) 17576 active+clean 1723 active+clean+scrubbing 779 active+clean+scrubbing+deep 381 active+remapped+backfill_wait 182 active+remapped+backfilling io: client: 232 B/s rd, 0 op/s rd, 0 op/s wr recovery: 2.4 GiB/s, 639 objects/s I guess we will wait 3-4 days for cluster recovery to finish and I will update you if something happens once it's HEALTH_OK. Is there a way to force epoch/osdmap trimming on rebalancing cluster? Afaik there no way to do it. It would be nice if we had such ability. It's not the first time we have this class of issues (big mon DBs). I would consider it a major flaw of the ceph. It cannot get healthier until it's not in a perfectly healthy state. Best regards Adam Prycki

Thank you guys for help diagnosing this issue. Once rebalancing finished and everything was active+clean cluster_osdmap_trim_lower_bound increased on OSDs. OSDs are slowly cleaning up old maps and we are regaining free space on OSDs. It may take over a week to get rid off them all but we are no longer in danger. Best regards Adam Prycki W dniu 6.05.2025 o 20:10, Anthony D'Atri pisze:

...

I guess we will wait 3-4 days for cluster recovery to finish and I will update you if something happens once it's HEALTH_OK.

Is there a way to force epoch/osdmap trimming on rebalancing cluster?

I would use https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-remap...

A pass or two of that will enter upmaps to obviate the current backfill. If you also temporarily disable the balancer you should very quickly get to a point with no backfill_wait. Then after the trimming and the dust settles you should re-enable the balancer to finish.

...

Afaik there no way to do it. It would be nice if we had such ability. It's not the first time we have this class of issues (big mon DBs). I would consider it a major flaw of the ceph. It cannot get healthier until it's not in a perfectly healthy state.

Best regards Adam Prycki _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

One thing that jumps out at me is that these SSDs are all 480GB models. BlueStore has a certain constant amount of overhead that means

Hello, that very small OSDs can behave capacity-wise in unexpected ways. We didn't really had a choice when it came to purchase of these SSDs. Is it mentioned and explained somewhere in the documentation? I've heard that rocksdb was sensitive to device size but I've assumed this issue was fixed while ago.

I suggest checking your pool attributes and CRUSH rules to ensure that I've checked all pools are on devices they are supposed to be. Every crush rules has device class (and ceph osd pool autoscale-status works correctly)

I see RGW pools, but no index pool. Is RGW in use, or did you trim the `ceph df` output? If RGW isn’t actually being used, you might want to clean up those unused pools. RGW pools are just leftovers from some experiments. We do not have running RGW daemons on this cluster.

Has anyone *ever* run `rados bench` against any pool? I suggest looking at `rados ls` for each pool for RADOS objects matching a bench profile/naming. I’ve in the past encountered detritus from bench runs that did’t get cleaned up, e.g. if one ^Cs during the run. A bench cleanup or a very careful pipe through `rados rm` can address that. Probably not. We usually use randos bench on dedicated pool.s

Evacuating all PGs from OSD doesn't lower it's usage. For example 1814 ssd 0 1.00000 447 GiB 382 GiB 381 GiB 351 MiB 1.0 GiB 65 GiB 85.44 1.00 0 up I suspect its OSD/bluestore issue since data usage seems not to come from PGs. Best regards Adam W dniu 5.05.2025 o 16:31, Anthony D'Atri pisze:

...

we have just discovered new issue with our cephfs cluster. All of our SSD osd have been flagged as nearful.

Ceph df on our cluster looks very wrong. Raw storage usage shows that SSDs are almost full, but SSD pools are hardly using any space.

One thing that jumps out at me is that these SSDs are all 480GB models. BlueStore has a certain constant amount of overhead that means that very small OSDs can behave capacity-wise in unexpected ways.

I suggest checking your pool attributes and CRUSH rules to ensure that

* The pools each are using the intended CRUSH rule * Every CRUSH rule specifies a device class, either ssd or hdd. It is all too easy to have a rule in use that does not specify device class, which would mean that SSD capacity is being used by a pool that should only be using HDDs.

I see RGW pools, but no index pool. Is RGW in use, or did you trim the `ceph df` output? If RGW isn’t actually being used, you might want to clean up those unused pools.

Ah, I see an update with the CRUSH rules …

Please send `ceph osd dump | grep pool`

* rule 0 replicated_rule item name is `default`, so any pools using this rule will use *all* device classes * rule 1 selects the `nvme` device class. Do you have OSDs tagged with that device class vs `ssd`? * rules 2,3,5,6 select only hdd * rule 4 selects only ssd

Check against the pool details output and verify that you don’t have a pool using the wrong CRUSH rule, and especially that no pools are using rule 0.

...

rados -p cephfs.arm-vol.meta | wc -l confirms that there are about 2.02M objects in that pool.

Has anyone *ever* run `rados bench` against any pool? I suggest looking at `rados ls` for each pool for RADOS objects matching a bench profile/naming. I’ve in the past encountered detritus from bench runs that did’t get cleaned up, e.g. if one ^Cs during the run. A bench cleanup or a very careful pipe through `rados rm` can address that.

...

Our SSD OSDs are almost full

ceph osd df ssd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 807 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 395 MiB 1.3 GiB 64 GiB 85.78 1.00 85 up 812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.0 GiB 66 GiB 85.23 1.00 74 up 569 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.0 GiB 66 GiB 85.24 1.00 72 up

...

without checking any deeper your problem: maybe the filesystem on your ssds need a fstrim?

They’re presumably BlueStore and thus don’t have a filesystem on them. Fstrim is more of an issue with filesystems build on RBD volumes. I don’t *think* trimming is a thing for CephFS but I could be wrong.

...

577 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 334 MiB 1.5 GiB 67 GiB 85.12 1.00 72 up 571 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 369 MiB 2.3 GiB 66 GiB 85.30 1.00 86 up 578 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 329 MiB 1.8 GiB 66 GiB 85.19 1.00 68 up 1775 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 379 MiB 1.2 GiB 64 GiB 85.70 1.00 76 up 1838 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 2.3 GiB 66 GiB 85.29 1.00 74 up 1783 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 443 MiB 2.5 GiB 62 GiB 86.05 1.01 98 up 1846 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.7 GiB 66 GiB 85.18 1.00 85 up 1854 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 384 MiB 2.4 GiB 66 GiB 85.33 1.00 80 up 1855 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.6 GiB 65 GiB 85.37 1.00 78 up 1761 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 368 MiB 1.7 GiB 63 GiB 85.86 1.00 79 up 1826 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 331 MiB 2.2 GiB 66 GiB 85.29 1.00 73 up 1773 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 324 MiB 2.0 GiB 66 GiB 85.24 1.00 73 up 1836 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 390 MiB 2.3 GiB 66 GiB 85.30 1.00 82 up 1771 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB 66 GiB 85.31 1.00 75 up 1834 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 376 MiB 2.1 GiB 66 GiB 85.27 1.00 79 up 2111 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 396 MiB 2.0 GiB 63 GiB 85.90 1.01 84 up 2112 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 338 MiB 2.2 GiB 66 GiB 85.27 1.00 77 up 2272 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 374 MiB 2.0 GiB 66 GiB 85.24 1.00 76 up 2279 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 332 MiB 2.5 GiB 66 GiB 85.34 1.00 77 up 2276 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 360 MiB 2.4 GiB 66 GiB 85.34 1.00 82 up 2285 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 371 MiB 2.2 GiB 66 GiB 85.27 1.00 79 up 2275 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 368 MiB 2.3 GiB 66 GiB 85.31 1.00 79 up 2282 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.0 GiB 66 GiB 85.23 1.00 75 up 2271 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 1.6 GiB 66 GiB 85.15 1.00 81 up 2278 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB 63 GiB 85.85 1.00 77 up 2277 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 334 MiB 1.3 GiB 64 GiB 85.70 1.00 72 up 2286 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 351 MiB 2.3 GiB 66 GiB 85.30 1.00 74 up 2347 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 362 MiB 2.6 GiB 65 GiB 85.38 1.00 81 up 2348 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.0 GiB 66 GiB 85.23 1.00 76 up 573 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 404 MiB 2.7 GiB 65 GiB 85.40 1.00 85 up 580 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 320 MiB 2.3 GiB 60 GiB 86.65 1.01 70 up 169 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.0 GiB 66 GiB 85.24 1.00 88 up 171 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 323 MiB 2.5 GiB 66 GiB 85.34 1.00 71 up 1216 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.5 GiB 66 GiB 85.35 1.00 76 up 1218 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 334 MiB 1.9 GiB 63 GiB 85.84 1.00 70 up 811 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 377 MiB 2.0 GiB 66 GiB 85.24 1.00 80 up 815 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 419 MiB 2.6 GiB 65 GiB 85.38 1.00 91 up 810 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.2 GiB 66 GiB 85.27 1.00 67 up 813 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 340 MiB 2.4 GiB 63 GiB 85.93 1.01 72 up 1760 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 388 MiB 974 MiB 67 GiB 85.01 0.99 77 up 1823 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 366 MiB 2.5 GiB 66 GiB 85.35 1.00 73 up 1762 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 361 MiB 1.4 GiB 67 GiB 85.10 1.00 75 up 1824 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 278 MiB 2.6 GiB 62 GiB 86.06 1.01 65 up 1766 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 377 MiB 1.9 GiB 60 GiB 86.58 1.01 82 up 1830 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 339 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1755 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 388 MiB 2.5 GiB 65 GiB 85.36 1.00 77 up 1818 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB 66 GiB 85.31 1.00 75 up 1774 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 2.2 GiB 66 GiB 85.26 1.00 67 up 1837 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 311 MiB 2.5 GiB 66 GiB 85.33 1.00 68 up 1748 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.4 GiB 66 GiB 85.31 1.00 76 up 1812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.2 GiB 66 GiB 85.28 1.00 77 up 1337 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.3 GiB 66 GiB 85.30 1.00 64 up 1338 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 331 MiB 1.9 GiB 63 GiB 85.89 1.00 76 up 1091 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 329 MiB 2.5 GiB 66 GiB 85.34 1.00 69 up 1096 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 373 MiB 1.6 GiB 63 GiB 85.85 1.00 82 up 1093 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 379 MiB 2.6 GiB 65 GiB 85.38 1.00 84 up 1098 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.3 GiB 66 GiB 85.30 1.00 79 up 1752 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 350 MiB 1.5 GiB 67 GiB 85.11 1.00 80 up 1815 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB 66 GiB 85.30 1.00 70 up 1754 ssd 0.43660 1.00000 447 GiB 387 GiB 384 GiB 362 MiB 2.0 GiB 60 GiB 86.48 1.01 75 up 1816 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 346 MiB 2.5 GiB 66 GiB 85.35 1.00 72 up 1753 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.6 GiB 64 GiB 85.80 1.00 75 up 1817 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB 66 GiB 85.33 1.00 75 up 1739 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.3 GiB 67 GiB 85.09 1.00 75 up 1804 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 365 MiB 2.6 GiB 62 GiB 86.03 1.01 78 up 1763 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 369 MiB 2.6 GiB 62 GiB 86.04 1.01 76 up 1825 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 379 MiB 2.2 GiB 63 GiB 85.91 1.01 83 up 1738 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 354 MiB 1.9 GiB 63 GiB 85.84 1.00 75 up 1802 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 339 MiB 1.6 GiB 63 GiB 85.83 1.00 77 up 1661 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 320 MiB 2.4 GiB 66 GiB 85.33 1.00 65 up 1666 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.3 GiB 66 GiB 85.31 1.00 80 up 1663 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 444 MiB 1.9 GiB 63 GiB 85.91 1.01 98 up 1669 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 375 MiB 2.2 GiB 66 GiB 85.29 1.00 76 up 1665 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 341 MiB 2.1 GiB 66 GiB 85.26 1.00 75 up 1671 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 363 MiB 2.4 GiB 66 GiB 85.33 1.00 76 up 1726 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB 64 GiB 85.78 1.00 77 up 1790 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB 66 GiB 85.29 1.00 80 up 1765 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 437 MiB 2.5 GiB 62 GiB 86.07 1.01 91 up 1828 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 356 MiB 2.3 GiB 66 GiB 85.31 1.00 78 up 1744 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 355 MiB 2.5 GiB 62 GiB 86.03 1.01 77 up 1807 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 327 MiB 1.9 GiB 60 GiB 86.51 1.01 74 up 1768 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.2 GiB 66 GiB 85.29 1.00 88 up 1831 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.0 GiB 66 GiB 85.23 1.00 71 up 1756 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 339 MiB 1.3 GiB 67 GiB 85.07 1.00 65 up 1819 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 323 MiB 2.3 GiB 66 GiB 85.28 1.00 67 up 1772 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 407 MiB 1.8 GiB 66 GiB 85.21 1.00 87 up 1835 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 1.5 GiB 67 GiB 85.11 1.00 74 up 1879 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 342 MiB 2.5 GiB 66 GiB 85.34 1.00 70 up 1882 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 310 MiB 2.2 GiB 66 GiB 85.28 1.00 66 up 1880 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 344 MiB 2.6 GiB 65 GiB 85.36 1.00 79 up 1883 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 319 MiB 2.6 GiB 65 GiB 85.36 1.00 65 up 1881 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 313 MiB 2.1 GiB 66 GiB 85.25 1.00 64 up 1884 ssd 0.43660 1.00000 447 GiB 390 GiB 388 GiB 387 MiB 2.1 GiB 57 GiB 87.27 1.02 87 up 1747 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 426 MiB 2.6 GiB 65 GiB 85.38 1.00 87 up 1809 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 359 MiB 1.6 GiB 66 GiB 85.14 1.00 77 up 1751 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 340 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1813 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 367 MiB 2.2 GiB 63 GiB 85.93 1.01 81 up 1785 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 403 MiB 2.5 GiB 66 GiB 85.35 1.00 81 up 1848 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.5 GiB 66 GiB 85.34 1.00 74 up 1749 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.5 GiB 67 GiB 85.12 1.00 74 up 1811 ssd 0.43660 1.00000 447 GiB 388 GiB 385 GiB 335 MiB 2.5 GiB 59 GiB 86.70 1.01 74 up 1725 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 361 MiB 2.5 GiB 65 GiB 85.35 1.00 73 up 1788 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1779 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.1 GiB 67 GiB 85.03 0.99 77 up 1843 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 387 MiB 2.4 GiB 66 GiB 85.34 1.00 78 up 1967 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.2 GiB 66 GiB 85.28 1.00 66 up 1970 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.9 GiB 66 GiB 85.21 1.00 78 up 1968 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.9 GiB 66 GiB 85.22 1.00 71 up 1971 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 354 MiB 2.1 GiB 66 GiB 85.26 1.00 79 up 1966 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 352 MiB 1.8 GiB 64 GiB 85.79 1.00 82 up 1969 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 315 MiB 2.3 GiB 63 GiB 85.99 1.01 73 up 1781 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 381 MiB 1.8 GiB 66 GiB 85.20 1.00 81 up 1844 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 330 MiB 2.1 GiB 66 GiB 85.25 1.00 71 up 1743 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 360 MiB 2.1 GiB 63 GiB 85.95 1.01 79 up 1806 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 385 MiB 2.4 GiB 66 GiB 85.34 1.00 78 up 1778 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 354 MiB 2.5 GiB 66 GiB 85.35 1.00 74 up 1839 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 348 MiB 1.6 GiB 66 GiB 85.15 1.00 79 up 1757 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 409 MiB 2.5 GiB 65 GiB 85.36 1.00 82 up 1820 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.1 GiB 66 GiB 85.26 1.00 73 up 1758 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 403 MiB 2.5 GiB 63 GiB 86.02 1.01 81 up 1821 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.1 GiB 66 GiB 85.27 1.00 80 up 1787 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 281 MiB 2.1 GiB 66 GiB 85.24 1.00 64 up 1850 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 260 MiB 2.6 GiB 65 GiB 85.36 1.00 59 up 2054 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 318 MiB 1.6 GiB 66 GiB 85.14 1.00 68 up 2057 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.3 GiB 66 GiB 85.31 1.00 78 up 2055 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 2.1 GiB 66 GiB 85.26 1.00 79 up 2058 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 289 MiB 2.5 GiB 63 GiB 85.99 1.01 65 up 2053 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.0 GiB 66 GiB 85.23 1.00 74 up 2056 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 362 MiB 2.3 GiB 63 GiB 85.93 1.01 77 up 1735 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 363 MiB 1.8 GiB 66 GiB 85.20 1.00 76 up 1799 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 360 MiB 2.1 GiB 66 GiB 85.27 1.00 74 up 1784 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 361 MiB 1.8 GiB 66 GiB 85.19 1.00 74 up 1847 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.4 GiB 66 GiB 85.33 1.00 74 up 1740 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 368 MiB 2.4 GiB 66 GiB 85.33 1.00 72 up 1801 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 362 MiB 2.1 GiB 66 GiB 85.26 1.00 74 up 1776 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.1 GiB 66 GiB 85.25 1.00 80 up 1841 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 393 MiB 2.1 GiB 66 GiB 85.26 1.00 86 up 1742 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 1.6 GiB 66 GiB 85.14 1.00 74 up 1805 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 330 MiB 2.6 GiB 62 GiB 86.05 1.01 75 up 1764 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.4 GiB 66 GiB 85.32 1.00 74 up 1827 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 434 MiB 2.3 GiB 63 GiB 86.00 1.01 88 up 2301 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 355 MiB 2.3 GiB 66 GiB 85.30 1.00 79 up 2309 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 1.7 GiB 66 GiB 85.17 1.00 69 up 2302 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 314 MiB 2.5 GiB 66 GiB 85.33 1.00 68 up 2315 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.3 GiB 66 GiB 85.30 1.00 70 up 2308 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB 66 GiB 85.33 1.00 79 up 2316 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 366 MiB 2.3 GiB 66 GiB 85.29 1.00 76 up 1733 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.9 GiB 63 GiB 85.86 1.00 74 up 1796 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB 65 GiB 85.38 1.00 74 up 1782 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 313 MiB 1.4 GiB 64 GiB 85.77 1.00 70 up 1845 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 379 MiB 2.2 GiB 66 GiB 85.29 1.00 81 up 1730 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.2 GiB 66 GiB 85.29 1.00 74 up 1794 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 377 MiB 2.1 GiB 63 GiB 85.94 1.01 80 up 1786 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 381 MiB 2.6 GiB 65 GiB 85.37 1.00 76 up 1849 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 337 MiB 963 MiB 64 GiB 85.63 1.00 76 up 1741 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 358 MiB 1.4 GiB 67 GiB 85.09 1.00 75 up 1803 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 383 MiB 2.7 GiB 65 GiB 85.40 1.00 78 up 1750 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 353 MiB 2.1 GiB 66 GiB 85.26 1.00 70 up 1814 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 351 MiB 772 MiB 67 GiB 84.95 0.99 81 up

Both ssd pools run with basic 3x replication and rack failure domain. cephfs.arm-vol.data pool is EC pool on HDD OSDs. I've checked few OSDs and pools are not overlapping over.

Cluster runs on 19.2.0 and OSD nodes are ARM64. In emergency I'm planning to just move these pool to HDDs OSD.

Do you have any idea what could have filled these SSDs and how to clean it up?

Today I've

Best regards Adam Prycki _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, I've just tried to do it. Fresh OSD is down gets stuck in preboot It seems to slowly download maps from the cluster. It's probably the issue Igor mentioned. { "cluster_fsid": "bec60cda-a306-11ed-abd9-75488d4e8f4a", "osd_fsid": "fbfeba00-3daa-461e-9e02-5c257e677e8a", "whoami": 2349, "state": "preboot", "maps": "[1502335~121041]", "oldest_map": "1502335", "newest_map": "1623375", "cluster_osdmap_trim_lower_bound": 1502335, "num_pgs": 0 } What's the healthy amount of maps OSD should store? Best regards Adam Prycki W dniu 5.05.2025 o 17:11, Robert Sander pisze:

Am 5/5/25 um 16:58 schrieb Adam Prycki:

...

Evacuating all PGs from OSD doesn't lower it's usage. For example 1814    ssd        0   1.00000  447 GiB  382 GiB  381 GiB  351 MiB 1.0 GiB  65 GiB  85.44  1.00    0      up I suspect its OSD/bluestore issue since data usage seems not to come from PGs.

What happens when you recreate this OSD?

Regards

Hi, Our ceph cluster has been rebalancing for a long time after we added a lot of servers. (mclock backfilling speed was very bad) Ceph monitors DBs are really big (60-90GB). I expected DBs to grow and we have enough space for them. We didn't expect OSDs to fill up. Fresh OSD finally started after it finished downloading maps from monitors. New OSD grew to the same size as old OSDs. example: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1750 ssd 0 1.00000 447 GiB 384 GiB 381 GiB 245 MiB 2.4 GiB 63 GiB 85.83 1.00 0 up 2349 ssd 0.43660 1.00000 447 GiB 383 GiB 381 GiB 290 MiB 1.7 GiB 64 GiB 85.70 1.00 70 up osd.1750 is old OSD with PGs evacuated osd.2349 is new OSD Size is almost the same. Best regards Adam Prycki W dniu 5.05.2025 o 17:51, Robert Sander pisze:

Hi,

Am 5/5/25 um 17:26 schrieb Adam Prycki:

...

Fresh OSD is down gets stuck in preboot It seems to slowly download maps from the cluster. It's probably the issue Igor mentioned.

What is the health status of the cluster?

Regards

On May 5, 2025, at 10:58 AM, Adam Prycki <aprycki@man.poznan.pl> wrote:

Every crush rules has device class

No device class here. [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] },