Hi Khodayar That rados purge command did not actually seem to delete anything. It is still using 90GiB like before. I have 4 pools on these OSD's. But the "fast" pool still has 90GB but should be empty. Looking at the PG's in `ceph pg ls` LOG is using very little space on the pool. Are there any other logs ? # ceph osd pool ls detail pool 42 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 186707 lfor 0/124366/185624 flags hashpspool stripe_width 0 application cephfs pool 51 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 186709 lfor 0/186698/186696 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive compression_required_ratio 0.9 application rbd removed_snaps [1~9] pool 53 'kube' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 186711 lfor 0/185831/185829 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~3] pool 54 'fast' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 186713 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive compression_required_ratio 0.9 application cephfs removed_snaps [52~2,55~2,59~2] # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 175 nvme 1.81898 1.00000 1.8 TiB 361 GiB 342 GiB 9.7 GiB 9.8 GiB 1.5 TiB 19.38 0.23 160 up osd.175 176 nvme 1.81898 1.00000 1.8 TiB 361 GiB 342 GiB 9.5 GiB 9.6 GiB 1.5 TiB 19.36 0.23 160 up osd.176 # ceph df POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR cephfs_metadata 42 320 MiB 2.82M 320 MiB 0.01 1.4 TiB N/A N/A 2.82M 0 B 0 B rbd 51 417 GiB 106.93k 417 GiB 12.32 1.4 TiB N/A N/A 106.93k 83 GiB 167 GiB kube 53 38 B 3 38 B 0 1.4 TiB N/A N/A 3 0 B 0 B fast 54 90 GiB 5.25M 90 GiB 2.93 1.4 TiB N/A N/A 5.25M 0 B 0 B # ceph pg ls PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP... ... 54.3a 81396 0 0 0 1512202599 0 0 3020 active+clean 9m 186704'339296 186722:615571 54.3b 82612 0 0 0 1556478139 0 0 3073 active+clean 9m 186706'340432 186722:609005 54.3c 82526 0 0 0 1553816829 0 0 3031 active+clean 9m 186704'806269 186722:1175811 54.3d 82144 0 0 0 1454278881 0 0 3016 active+clean 9m 186706'339167 186722:743880 54.3e 82740 0 0 0 1501081654 0 0 3056 active+clean 9m 186704'343212 186722:621323 54.3f 80846 0 0 0 1474357106 0 0 3048 active+clean 9m 186706'333741 186722:733149 On Sun, Apr 19, 2020 at 4:00 AM Khodayar Doustar <doustar@rayanexon.ir> wrote:
Hi Kári,
You are purging only 2.6M objects with your purge command, so I guess that 5.25M objects would be something else like logs. Have you checked the osd df detail and osd metadata? I have a case which osds bluestore logs are eating my whole space up, maybe you are facing a similar one.
Regards, Khodayar
On Sun, Apr 19, 2020 at 6:47 AM Kári Bertilsson <karibertils@gmail.com> wrote:
Hello
Running ceph v14.2.8 on everything. Pool is using replicated_rule with size/min_size 2 with 2 OSD's. I have scrubbed and deep scrubbed the OSDs.
This pool was attached as data pool to cephfs containing alot of small files. I have since removed all files and detached the pool from the fs. Somehow the pool is still using 90GB.
POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR fast 54 90 GiB 5.25M 90 GiB 2.93 1.4 TiB N/A N/A 5.25M 0 B 0 B
# rados -p fast ls|wc -l 1312030
# rados -p fast stat 100073acce3.00000000 error stat-ing fast/100073acce3.00000000: (2) No such file or directory
# rados -p fast rm 100073acce3.00000000 error removing fast>100073acce3.00000000: (2) No such file or directory
# rados -p fast get 100073acce3.00000000 test error getting fast/100073acce3.00000000: (2) No such file or directory
I get the same error for every single object in the pool
# rados purge fast --yes-i-really-really-mean-it Warning: using slow linear search Removed 2625749 objects successfully purged pool fast
There was 5.25M objects in the pool before and after running this command. No change in ceph df.
Any ideas how to reclaim the free space ? I can remove & recreate the pool, but i would like to know why and how to deal with this situation when i don't have that privilege
Best regards Kári Bertilsson _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io