2 pgs backfill_toofull but plenty of space
Hi Ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable) Looking at this: " Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull " " [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull pg 3.11f is active+remapped+backfill_wait+backfill_toofull, acting [98,51,39,100] pg 3.74c is active+remapped+backfill_wait+backfill_toofull, acting [96,120,58,48] " But the disks are noway near being full as far as I can determine, so why backfill_toofull? The PGs in question are in the rbd_data pool. " # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.4 PiB 730 TiB 686 TiB 686 TiB 48.46 ssd 1.3 TiB 1.2 TiB 162 GiB 162 GiB 12.11 TOTAL 1.4 PiB 731 TiB 686 TiB 686 TiB 48.42 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 1.1 GiB 273 545 MiB 0.05 549 GiB rbd_data 3 4096 294 TiB 78.56M 450 TiB 45.72 267 TiB rbd 4 32 4.1 MiB 26 3.5 MiB 0 549 GiB rbd_internal 5 32 54 KiB 16 172 KiB 0 549 GiB cephfs_data 6 2048 127 TiB 148.64M 229 TiB 29.99 267 TiB cephfs_metadata 7 128 71 GiB 2.84M 142 GiB 11.46 549 GiB libvirt 8 32 37 MiB 221 74 MiB 0 549 GiB nfs-ganesha 9 32 2.7 KiB 7 52 KiB 0 366 GiB .nfs 10 32 53 KiB 47 306 KiB 0 366 GiB " The top utilized disk is at 57% and the PGs in that pool are ~50GB. " TOP BOTTOM USE WEIGHT PGS ID |USE WEIGHT PGS ID --------------------------------+-------------------------------- 57.71% 1.00000 54 osd.68 |46.60% 1.00000 286 osd.17 57.08% 1.00000 53 osd.80 |46.55% 1.00000 286 osd.99 54.95% 1.00000 70 osd.86 |46.48% 1.00000 284 osd.106 54.86% 1.00000 52 osd.63 |45.88% 1.00000 187 osd.27 54.06% 1.00000 68 osd.88 |45.81% 1.00000 279 osd.5 53.89% 1.00000 51 osd.79 |44.95% 1.00000 272 osd.13 53.65% 1.00000 51 osd.67 |43.63% 1.00000 269 osd.16 53.59% 1.00000 52 osd.65 |43.30% 1.00000 261 osd.12 53.58% 1.00000 51 osd.82 |32.17% 1.00000 172 osd.4 53.52% 1.00000 50 osd.72 |0% 0 0 osd.49 --------------------------------+-------------------------------- " Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark
What else is going on? (ceph -s). If there is a lot of data being shuffled around, it may just be because its waiting for some other actions to complete first. Thanks, Kevin ________________________________________ From: Torkil Svensgaard <torkil@drcmr.dk> Sent: Tuesday, January 10, 2023 2:36 AM To: ceph-users@ceph.io Cc: Ruben Vestergaard Subject: [ceph-users] 2 pgs backfill_toofull but plenty of space Check twice before you click! This email originated from outside PNNL. Hi Ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable) Looking at this: " Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull " " [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull pg 3.11f is active+remapped+backfill_wait+backfill_toofull, acting [98,51,39,100] pg 3.74c is active+remapped+backfill_wait+backfill_toofull, acting [96,120,58,48] " But the disks are noway near being full as far as I can determine, so why backfill_toofull? The PGs in question are in the rbd_data pool. " # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.4 PiB 730 TiB 686 TiB 686 TiB 48.46 ssd 1.3 TiB 1.2 TiB 162 GiB 162 GiB 12.11 TOTAL 1.4 PiB 731 TiB 686 TiB 686 TiB 48.42 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 1.1 GiB 273 545 MiB 0.05 549 GiB rbd_data 3 4096 294 TiB 78.56M 450 TiB 45.72 267 TiB rbd 4 32 4.1 MiB 26 3.5 MiB 0 549 GiB rbd_internal 5 32 54 KiB 16 172 KiB 0 549 GiB cephfs_data 6 2048 127 TiB 148.64M 229 TiB 29.99 267 TiB cephfs_metadata 7 128 71 GiB 2.84M 142 GiB 11.46 549 GiB libvirt 8 32 37 MiB 221 74 MiB 0 549 GiB nfs-ganesha 9 32 2.7 KiB 7 52 KiB 0 366 GiB .nfs 10 32 53 KiB 47 306 KiB 0 366 GiB " The top utilized disk is at 57% and the PGs in that pool are ~50GB. " TOP BOTTOM USE WEIGHT PGS ID |USE WEIGHT PGS ID --------------------------------+-------------------------------- 57.71% 1.00000 54 osd.68 |46.60% 1.00000 286 osd.17 57.08% 1.00000 53 osd.80 |46.55% 1.00000 286 osd.99 54.95% 1.00000 70 osd.86 |46.48% 1.00000 284 osd.106 54.86% 1.00000 52 osd.63 |45.88% 1.00000 187 osd.27 54.06% 1.00000 68 osd.88 |45.81% 1.00000 279 osd.5 53.89% 1.00000 51 osd.79 |44.95% 1.00000 272 osd.13 53.65% 1.00000 51 osd.67 |43.63% 1.00000 269 osd.16 53.59% 1.00000 52 osd.65 |43.30% 1.00000 261 osd.12 53.58% 1.00000 51 osd.82 |32.17% 1.00000 172 osd.4 53.52% 1.00000 50 osd.72 |0% 0 0 osd.49 --------------------------------+-------------------------------- " Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 10-01-2023 18:59, Fox, Kevin M wrote:
What else is going on? (ceph -s). If there is a lot of data being shuffled around, it may just be because its waiting for some other actions to complete first.
There's a bit going on but if it is waiting for something else it shouldn't be backfill_toofull but something else? There's plenty of space and no disk near anything full. " # ceph -s cluster: id: 3b7736c6-00e4-11ec-a3c5-3cecef467984 health: HEALTH_WARN 3 host(s) running different kernel versions 1 failed cephadm daemon(s) Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 223 pgs not deep-scrubbed in time 134 pgs not scrubbed in time 1 pools have too many placement groups services: mon: 5 daemons, quorum test-ceph-03,test-ceph-04,dcn-ceph-03,dcn-ceph-02,dcn-ceph-04 (age 2m) mgr: dcn-ceph-02.jahyzc(active, since 2d), standbys: dcn-ceph-03.lrhaxo mds: 1/1 daemons up, 1 standby osd: 122 osds: 121 up (since 6d), 121 in (since 8d); 98 remapped pgs rbd-mirror: 2 daemons active (2 hosts) data: volumes: 1/1 healthy pools: 9 pools, 6433 pgs objects: 230.41M objects, 317 TiB usage: 688 TiB used, 729 TiB / 1.4 PiB avail pgs: 4056139/915969556 objects misplaced (0.443%) 6316 active+clean 88 active+remapped+backfill_wait 18 active+clean+scrubbing+deep 8 active+remapped+backfilling 2 active+remapped+backfill_wait+backfill_toofull 1 active+clean+snaptrim io: client: 812 KiB/s rd, 178 MiB/s wr, 609 op/s rd, 339 op/s wr recovery: 96 MiB/s, 49 objects/s progress: Global Recovery Event (2d) [===========================.] (remaining: 59m) " Mvh. Torkil
Thanks, Kevin
________________________________________ From: Torkil Svensgaard <torkil-BAyU0AAS9Wk@public.gmane.org> Sent: Tuesday, January 10, 2023 2:36 AM To: ceph-users-a8pt6IJUokc@public.gmane.org Cc: Ruben Vestergaard Subject: [ceph-users] 2 pgs backfill_toofull but plenty of space
Check twice before you click! This email originated from outside PNNL.
Hi
Ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)
Looking at this:
" Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull "
" [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull pg 3.11f is active+remapped+backfill_wait+backfill_toofull, acting [98,51,39,100] pg 3.74c is active+remapped+backfill_wait+backfill_toofull, acting [96,120,58,48] "
But the disks are noway near being full as far as I can determine, so why backfill_toofull? The PGs in question are in the rbd_data pool.
" # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.4 PiB 730 TiB 686 TiB 686 TiB 48.46 ssd 1.3 TiB 1.2 TiB 162 GiB 162 GiB 12.11 TOTAL 1.4 PiB 731 TiB 686 TiB 686 TiB 48.42
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 1.1 GiB 273 545 MiB 0.05 549 GiB rbd_data 3 4096 294 TiB 78.56M 450 TiB 45.72 267 TiB rbd 4 32 4.1 MiB 26 3.5 MiB 0 549 GiB rbd_internal 5 32 54 KiB 16 172 KiB 0 549 GiB cephfs_data 6 2048 127 TiB 148.64M 229 TiB 29.99 267 TiB cephfs_metadata 7 128 71 GiB 2.84M 142 GiB 11.46 549 GiB libvirt 8 32 37 MiB 221 74 MiB 0 549 GiB nfs-ganesha 9 32 2.7 KiB 7 52 KiB 0 366 GiB .nfs 10 32 53 KiB 47 306 KiB 0 366 GiB "
The top utilized disk is at 57% and the PGs in that pool are ~50GB.
" TOP BOTTOM USE WEIGHT PGS ID |USE WEIGHT PGS ID --------------------------------+-------------------------------- 57.71% 1.00000 54 osd.68 |46.60% 1.00000 286 osd.17 57.08% 1.00000 53 osd.80 |46.55% 1.00000 286 osd.99 54.95% 1.00000 70 osd.86 |46.48% 1.00000 284 osd.106 54.86% 1.00000 52 osd.63 |45.88% 1.00000 187 osd.27 54.06% 1.00000 68 osd.88 |45.81% 1.00000 279 osd.5 53.89% 1.00000 51 osd.79 |44.95% 1.00000 272 osd.13 53.65% 1.00000 51 osd.67 |43.63% 1.00000 269 osd.16 53.59% 1.00000 52 osd.65 |43.30% 1.00000 261 osd.12 53.58% 1.00000 51 osd.82 |32.17% 1.00000 172 osd.4 53.52% 1.00000 50 osd.72 |0% 0 0 osd.49 --------------------------------+-------------------------------- "
Mvh.
Torkil
-- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark _______________________________________________ ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org _______________________________________________ ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org
participants (3)
-
Fox, Kevin M
-
Torkil Svensgaard
-
Torkil Svensgaard