What else is going on? (ceph -s). If there is a lot of data being shuffled around, it may just be because its waiting for some other actions to complete first. Thanks, Kevin ________________________________________ From: Torkil Svensgaard <torkil@drcmr.dk> Sent: Tuesday, January 10, 2023 2:36 AM To: ceph-users@ceph.io Cc: Ruben Vestergaard Subject: [ceph-users] 2 pgs backfill_toofull but plenty of space Check twice before you click! This email originated from outside PNNL. Hi Ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable) Looking at this: " Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull " " [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull pg 3.11f is active+remapped+backfill_wait+backfill_toofull, acting [98,51,39,100] pg 3.74c is active+remapped+backfill_wait+backfill_toofull, acting [96,120,58,48] " But the disks are noway near being full as far as I can determine, so why backfill_toofull? The PGs in question are in the rbd_data pool. " # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.4 PiB 730 TiB 686 TiB 686 TiB 48.46 ssd 1.3 TiB 1.2 TiB 162 GiB 162 GiB 12.11 TOTAL 1.4 PiB 731 TiB 686 TiB 686 TiB 48.42 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 1.1 GiB 273 545 MiB 0.05 549 GiB rbd_data 3 4096 294 TiB 78.56M 450 TiB 45.72 267 TiB rbd 4 32 4.1 MiB 26 3.5 MiB 0 549 GiB rbd_internal 5 32 54 KiB 16 172 KiB 0 549 GiB cephfs_data 6 2048 127 TiB 148.64M 229 TiB 29.99 267 TiB cephfs_metadata 7 128 71 GiB 2.84M 142 GiB 11.46 549 GiB libvirt 8 32 37 MiB 221 74 MiB 0 549 GiB nfs-ganesha 9 32 2.7 KiB 7 52 KiB 0 366 GiB .nfs 10 32 53 KiB 47 306 KiB 0 366 GiB " The top utilized disk is at 57% and the PGs in that pool are ~50GB. " TOP BOTTOM USE WEIGHT PGS ID |USE WEIGHT PGS ID --------------------------------+-------------------------------- 57.71% 1.00000 54 osd.68 |46.60% 1.00000 286 osd.17 57.08% 1.00000 53 osd.80 |46.55% 1.00000 286 osd.99 54.95% 1.00000 70 osd.86 |46.48% 1.00000 284 osd.106 54.86% 1.00000 52 osd.63 |45.88% 1.00000 187 osd.27 54.06% 1.00000 68 osd.88 |45.81% 1.00000 279 osd.5 53.89% 1.00000 51 osd.79 |44.95% 1.00000 272 osd.13 53.65% 1.00000 51 osd.67 |43.63% 1.00000 269 osd.16 53.59% 1.00000 52 osd.65 |43.30% 1.00000 261 osd.12 53.58% 1.00000 51 osd.82 |32.17% 1.00000 172 osd.4 53.52% 1.00000 50 osd.72 |0% 0 0 osd.49 --------------------------------+-------------------------------- " Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io