Hi, did you check if your cluster has many "shadow" or "multipart" objects in the pool? Those are taken into account when calculating the total number of objects in a pool but are not in the user stats of radosgw. Here's an example of a small rgw setup: rados -p <pool> ls | grep -vE "shadow|multipart" | wc -l 455 and it matches the number of files reported by user stats: radosgw-admin user stats --uid <user> 2>/dev/null | jq -r '.stats | select(.num_objects > 0) | .num_objects' 455 Although the total number of objects is much higher because of the mentioned shadow objects: rados -p <pool> ls | grep -cE "shadow|multipart" 2248 And the total number (2248 + 455) matches the stats from ceph (2.70k): ceph df | grep <pool> <pool> 59 8 7.7 GiB 2.70k 21 GiB 0.57 1.4 TiB Regards, Eugen Zitat von Ramin Najjarbashi <ramin.najarbashi@gmail.com>:
On Thu, Feb 2, 2023 at 7:56 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I have a cluster with approximately one billion objects and when I run a PG query, it shows that I have 27,000 objects per PG.
which query is that, can you provide more details about that cluster and pool?
Thanks for your response
ceph df | grep mypoo
--- POOLS ---
POOL OBJECTS
mypool 1.11G
---------------
and from this, I got 8.8M objects :
for item in `radosgw-admin user list | jq -r ".[]" | head`; do B_OBJ=$(radosgw-admin user stats --uid $item 2>/dev/null | jq -r '.stats | select(.num_objects > 0) | .num_objects'); SUM=$((SUM + B_OBJ)); done
However, when I run the same command per pg, the results are much
less, with only 20 million objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls | wc -l" shows only three objects in the specified PG.
It seems like your PG and object distribution might not be balanced very well. Did you check each PG of that pool? The individual PG's numbers should add up to the total number of objects. Here's a quick example from an almost empty pool with 8 PGs:
storage01:~ # ceph pg ls-by-pool volumes | awk '{print $1" - "$2}' PG - OBJECTS 3.0 - 2 3.1 - 0 3.2 - 2 3.3 - 1 3.4 - 3 3.5 - 0 3.6 - 2 3.7 - 0
``` ~# ceph pg ls-by-pool mypool | awk '{print $1" - "$2} ... 9.ff2 - 271268 9.ff3 - 271046 9.ff4 - 271944 9.ff5 - 270864 9.ff6 - 272122 9.ff7 - 272244 9.ff8 - 271638 9.ff9 - 271702 9.ffa - 270906 9.ffb - 271114 9.ffc - 271986 9.ffd - 270766 9.ffe - 271702 9.fff - 271693 ... ```
and it sums up to 10 objects which matches the total stats:
storage01:~ # ceph df | grep -E "OBJECTS|volumes" POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL volumes 3 8 379 B 10 960 KiB 0 7.5 GiB
Regards, Eugen
Zitat von Ramin Najjarbashi <ramin.najarbashi@gmail.com>:
Hi I hope this email finds you well. I am reaching out to you because I have encountered an issue with my CEPH Bluestore cluster and I am seeking your assistance. I have a cluster with approximately one billion objects and when I run a PG query, it shows that I have 27,000 objects per PG. I have run the following command: "rados -p <mypool> ls | wc -l" which returns the correct number of one billion objects. However, when I run the same command per pg, the results are much less, with only 20 million objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls | wc -l" shows only three objects in the specified PG. This is a significant discrepancy and I am concerned about the integrity of my data. Do you have any idea about this discrepancy?
p.s: I have a total of 30 million objects in a single bucket and versioning has not been enabled for this particular bucket.
Thank you for your time and I look forward to your response.
Best regards, Ramin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io