Disks are filling up even if there is not a single placement group on them
Hi all, Our SSD disks are filling up, even if there is not a single placement group on them. How is that possible please? ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1997 ssd 1.15269 1.00000 1.2 TiB 943 GiB 942 GiB 6 KiB 1.5 GiB 237 GiB 79.90 1.71 0 up Cluster uses Pacific 16.2.7. Thank you Michal
Hi, we know nothing about the cluster status, so it’s really hard to know exactly. Is backfill going on? Did you delete a pool and the garbage collection is in progress? Zitat von Michal Strnad <michal.strnad@cesnet.cz>:
Hi all,
Our SSD disks are filling up, even if there is not a single placement group on them. How is that possible please?
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1997 ssd 1.15269 1.00000 1.2 TiB 943 GiB 942 GiB 6 KiB 1.5 GiB 237 GiB 79.90 1.71 0 up
Cluster uses Pacific 16.2.7.
Thank you
Michal
Hi Eugen. Sorry for my hasty and incomplete report. We did not remove any pool. Garbage collecion is not in progress. radosgw-admin gc list [] However, I have noticed that there are several placement groups that are in the backfill_wait state for some time. cluster: id: a12aa2d2-fae7-df35-ea2f-3de23100e345 health: HEALTH_WARN 3492 pgs not deep-scrubbed in time 3830 pgs not scrubbed in time 1 pool(s) do not have an application enabled services: mon: 3 daemons, quorum mon001-clx,mon002-clx,mon003-clx (age 10h) mgr: mon002-clx(active, since 10h), standbys: mon001-clx, mon003-clx mds: 1/1 daemons up, 1 standby osd: 2438 osds: 2433 up (since 12h), 2361 in (since 12h); 382 remapped pgs rgw: 31 daemons active (4 hosts, 2 zones) data: volumes: 1/1 healthy pools: 42 pools, 9521 pgs objects: 1.87G objects, 6.9 PiB usage: 14 PiB used, 16 PiB / 29 PiB avail pgs: 1656117639/32580808518 objects misplaced (5.083%) 9134 active+clean 375 active+remapped+backfill_wait 7 active+remapped+backfilling 4 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 754 KiB/s rd, 380 MiB/s wr, 131 op/s rd, 358 op/s wr recovery: 324 MiB/s, 81 objects/s ceph daemon mon.mon001-clx config show | grep full "mon_cache_target_full_warn_ratio": "0.660000", "mon_osd_backfillfull_ratio": "0.900000", "mon_osd_full_ratio": "0.950000", "mon_osd_nearfull_ratio": "0.850000", "mon_osdmap_full_prune_enabled": "true", "mon_osdmap_full_prune_interval": "10", "mon_osdmap_full_prune_min": "10000", "mon_osdmap_full_prune_txsize": "100", "osd_debug_skip_full_check_in_backfill_reservation": "false", "osd_debug_skip_full_check_in_recovery": "false", "osd_failsafe_full_ratio": "0.970000", "osd_pool_default_cache_target_full_ratio": "0.800000", "paxos_stash_full_interval": "25", ceph osd dump | grep ratio full_ratio 0.9 backfillfull_ratio 0.85 nearfull_ratio 0.8 Thank you Michal On 4/7/23 21:09, Eugen Block wrote:
Hi,
we know nothing about the cluster status, so it’s really hard to know exactly. Is backfill going on? Did you delete a pool and the garbage collection is in progress?
Zitat von Michal Strnad <michal.strnad@cesnet.cz>:
Hi all,
Our SSD disks are filling up, even if there is not a single placement group on them. How is that possible please?
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1997 ssd 1.15269 1.00000 1.2 TiB 943 GiB 942 GiB 6 KiB 1.5 GiB 237 GiB 79.90 1.71 0 up
Cluster uses Pacific 16.2.7.
Thank you
Michal
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Sat, Apr 8, 2023 at 2:26 PM Michal Strnad <michal.strnad@cesnet.cz> wrote:
cluster: id: a12aa2d2-fae7-df35-ea2f-3de23100e345 health: HEALTH_WARN ... pgs: 1656117639/32580808518 objects misplaced (5.083%)
That's why the space is eaten. The stuff that eats the disk space on MONs is osdmaps, and the MONs have to keep old osdmaps back to the moment in the past when the cluster was 100% healthy. Note that osdmaps are also copied to all OSDs and eat space there, which is what you have seen. The relevant (but dangerous) configuration parameter is "mon_osd_force_trim_to". Better don't use it, and let your ceph cluster recover. If you can't wait, try to use upmaps to say that all PGs are fine where they are now, i.e that they are not misplaced. There is a script somewhere on GitHub that does this, but unfortunately I can't find it right now. -- Alexander E. Patrakov
Hi. Thank you for the explanation. I get it now. Michal On 4/10/23 20:44, Alexander E. Patrakov wrote:
On Sat, Apr 8, 2023 at 2:26 PM Michal Strnad <michal.strnad@cesnet.cz> wrote:
cluster: id: a12aa2d2-fae7-df35-ea2f-3de23100e345 health: HEALTH_WARN
...
pgs: 1656117639/32580808518 objects misplaced (5.083%)
That's why the space is eaten. The stuff that eats the disk space on MONs is osdmaps, and the MONs have to keep old osdmaps back to the moment in the past when the cluster was 100% healthy. Note that osdmaps are also copied to all OSDs and eat space there, which is what you have seen.
The relevant (but dangerous) configuration parameter is "mon_osd_force_trim_to". Better don't use it, and let your ceph cluster recover. If you can't wait, try to use upmaps to say that all PGs are fine where they are now, i.e that they are not misplaced. There is a script somewhere on GitHub that does this, but unfortunately I can't find it right now.
-- Alexander E. Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Alexander E. Patrakov
-
Eugen Block
-
Michal Strnad