Hi, I can't seem to find the threads I was looking for, this has been discussed before. Anyway, IIRC it could be a MGR issue which fails to update the stats. Maybe a MGR failover clears things up? If that doesn't help I would try a compaction on one OSD and see if the stats are corrected (and if they are, do it for the rest as well). If not, someone else might have more ideas. :-) Regards, Eugen Zitat von Andras Pataki <apataki@flatironinstitute.org>:
Here is a strange problem that I don't seem to be able to figure out. Some of our OSDs that have zero weight, and no PGs have lots of allocated space: [root@cephosd0032 ~]# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2925 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.32 0.15 0 up 2926 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.33 0.15 0 up 2927 ssd 0 1.00000 1.1 TiB 125 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.31 0.15 0 up 2928 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.32 0.15 0 up
so 120GB+ allocated, but no PGs. The cluster is clean, all PGs are active+clean, no rebalancing/recovery/etc. happening. There has been no rebalancing/recovery for at least a couple of days (after having added some nodes to the cluster).
Doing a perf dump on the OSD confirms the space usage - that it has 130GB or so allocated:
[root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok perf dump .... "bluestore_allocated": 133277466624, "bluestore_stored": 133014427663, ...
Bringing the OSD down - ceph-objectstore-tool claims that there are no objects stored in the OSD:
[root@cephosd0032 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2925 --op list [root@cephosd0032 ~]#
No fragmentation on the OSD either: [root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok bluestore allocator fragmentation block { "fragmentation_rating": 1.2165147867554808e-08 }
BlueFS also claims not to use significant space: [root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok bluefs stats 1 : device size 0x11548000000 : own 0x[8518510000~b175d0000] = 0xb175d0000 : using 0xd1e0000(210 MiB) : bluestore has 0xeb3b630000(941 GiB) available wal_total:0, db_total:1131368205516, slow_total:0
So the puzzle is, what is stored in that 130GB allocated space?
The cluster is Octopus 15.2.17. I came to looking at this by a mismatch in free space on our OSDs that are in service. After adding some nodes and waiting for the rebalance to finish, the free space didn't increase by the amount of space that was added, although the total space in the cluster did increase correctly (and data in pools didn't increase by a significant amount to explain this). So there is a bit of a mystery regarding what a bunch of space is allocated for.
Any ideas/pointers would be appreciated.
Andras _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io