1 Feb
2023
1 Feb
'23
7:57 a.m.
Hi
I hope this email finds you well. I am reaching out to you because I have
encountered an issue with my CEPH Bluestore cluster and I am seeking your
assistance.
I have a cluster with approximately one billion objects and when I run a PG
query, it shows that I have 27,000 objects per PG.
I have run the following command: "rados -p <mypool> ls | wc -l" which
returns the correct number of one billion objects. However, when I run the
same command per pg, the results are much less, with only 20 million
objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls | wc
-l" shows only three objects in the specified PG.
This is a significant discrepancy and I am concerned about the integrity of
my data.
Do you have any idea about this discrepancy?
p.s:
I have a total of 30 million objects in a single bucket and versioning has
not been enabled for this particular bucket.
Thank you for your time and I look forward to your response.
Best regards,
Ramin
2 Feb
2 Feb
7:48 a.m.
Hi,
I have a cluster with approximately one billion
objects and when I run a PG
query, it shows that I have 27,000 objects per PG.
which query is that, can you provide more details about that cluster and pool?
However, when I run the same command per pg, the
results are much
less, with only 20 million
objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls | wc
-l" shows only three objects in the specified PG.
It seems like your PG and object distribution might not be balanced
very well. Did you check each PG of that pool? The individual PG's
numbers should add up to the total number of objects. Here's a quick
example from an almost empty pool with 8 PGs:
storage01:~ # ceph pg ls-by-pool volumes | awk '{print $1" - "$2}'
PG - OBJECTS
3.0 - 2
3.1 - 0
3.2 - 2
3.3 - 1
3.4 - 3
3.5 - 0
3.6 - 2
3.7 - 0
and it sums up to 10 objects which matches the total stats:
storage01:~ # ceph df | grep -E "OBJECTS|volumes"
POOL ID PGS STORED OBJECTS USED %USED
MAX AVAIL
volumes 3 8 379 B 10 960 KiB 0 7.5 GiB
Regards,
Eugen
Zitat von Ramin Najjarbashi <ramin.najarbashi(a)gmail.com>om>:
> Hi
> I hope this email finds you well. I am reaching out to you because I have
> encountered an issue with my CEPH Bluestore cluster and I am seeking your
> assistance.
I have a cluster with approximately one billion
objects and when I run a PG
query, it shows that I have 27,000 objects per PG.
> I have run the following
command: "rados -p <mypool> ls | wc -l" which
> returns the correct number of one billion objects. However, when I run the
> same command per pg, the results are much less, with only 20 million
> objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls |
wc
> -l" shows only three objects in the specified PG.
> This is a significant discrepancy and I am concerned about the integrity of
> my data.
> Do you have any idea about this discrepancy?
>
> p.s:
> I have a total of 30 million objects in a single bucket and versioning has
> not been enabled for this particular bucket.
>
> Thank you for your time and I look forward to your response.
>
> Best regards,
> Ramin
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
4 Feb
4 Feb
11:54 a.m.
On Thu, Feb 2, 2023 at 7:56 PM Eugen Block <eblock(a)nde.ag> wrote:
Hi,
Thanks for your response
ceph df | grep mypoo
--- POOLS ---
POOL OBJECTS
mypool 1.11G
---------------
and from this, I got 8.8M objects :
for item in `radosgw-admin user list | jq -r ".[]" | head`; do
B_OBJ=$(radosgw-admin user stats --uid $item 2>/dev/null | jq -r '.stats |
select(.num_objects > 0) | .num_objects'); SUM=$((SUM + B_OBJ)); done
I have a cluster with approximately one billion
objects and when I run a
PG
query, it shows that I have 27,000 objects per
PG.
which query is that, can you provide more details about that cluster and
pool?
However, when I run the same command per pg, the
results are much
less, with only 20 million
objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls |
wc
-l" shows only three objects in the
specified PG.
It seems like your PG and object distribution might not be balanced
very well. Did you check each PG of that pool? The individual PG's
numbers should add up to the total number of objects. Here's a quick
example from an almost empty pool with 8 PGs:
storage01:~ # ceph pg ls-by-pool volumes | awk
'{print $1" - "$2}'
PG - OBJECTS
3.0 - 2
3.1 - 0
3.2 - 2
3.3 - 1
3.4 - 3
3.5 - 0
3.6 - 2
3.7 - 0
```
~# ceph pg ls-by-pool mypool | awk '{print $1" - "$2}
...
9.ff2 - 271268
9.ff3 - 271046
9.ff4 - 271944
9.ff5 - 270864
9.ff6 - 272122
9.ff7 - 272244
9.ff8 - 271638
9.ff9 - 271702
9.ffa - 270906
9.ffb - 271114
9.ffc - 271986
9.ffd - 270766
9.ffe - 271702
9.fff - 271693
...
```
and it sums up to 10 objects which matches the total
stats:
storage01:~ # ceph df | grep -E "OBJECTS|volumes"
POOL ID PGS STORED OBJECTS USED %USED
MAX AVAIL
volumes 3 8 379 B 10 960 KiB 0 7.5 GiB
Regards,
Eugen
Zitat von Ramin Najjarbashi <ramin.najarbashi(a)gmail.com>om>:
Hi
I hope this email finds you well. I am reaching out to you because I have
encountered an issue with my CEPH Bluestore cluster and I am seeking your
assistance.
I have a cluster with approximately one billion objects and when I run a
PG
query, it shows that I have 27,000 objects per
PG.
I have run the following command: "rados -p <mypool> ls | wc -l" which
returns the correct number of one billion objects. However, when I run
the
same command per pg, the results are much less,
with only 20 million
objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls |
wc
-l" shows only three objects in the
specified PG.
This is a significant discrepancy and I am concerned about the integrity
of
my data.
Do you have any idea about this discrepancy?
p.s:
I have a total of 30 million objects in a single bucket and versioning
has
not been enabled for this particular bucket.
Thank you for your time and I look forward to your response.
Best regards,
Ramin
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6 Feb
6 Feb
2:10 a.m.
Hi,
did you check if your cluster has many "shadow" or "multipart" objects
in the pool? Those are taken into account when calculating the total
number of objects in a pool but are not in the user stats of radosgw.
Here's an example of a small rgw setup:
rados -p <pool> ls | grep -vE "shadow|multipart" | wc -l
455
and it matches the number of files reported by user stats:
radosgw-admin user stats --uid <user> 2>/dev/null | jq -r '.stats |
select(.num_objects > 0) | .num_objects'
455
Although the total number of objects is much higher because of the
mentioned shadow objects:
rados -p <pool> ls | grep -cE "shadow|multipart"
2248
And the total number (2248 + 455) matches the stats from ceph (2.70k):
ceph df | grep <pool>
<pool> 59 8 7.7 GiB 2.70k 21 GiB 0.57 1.4 TiB
Regards,
Eugen
Zitat von Ramin Najjarbashi <ramin.najarbashi(a)gmail.com>om>:
On Thu, Feb 2, 2023 at 7:56 PM Eugen Block
<eblock(a)nde.ag> wrote:
Hi,
Thanks for your response
ceph df | grep mypoo
--- POOLS ---
POOL OBJECTS
mypool 1.11G
---------------
and from this, I got 8.8M objects :
for item in `radosgw-admin user list | jq -r ".[]" | head`; do
B_OBJ=$(radosgw-admin user stats --uid $item 2>/dev/null | jq -r '.stats |
select(.num_objects > 0) | .num_objects'); SUM=$((SUM + B_OBJ)); done
I have a cluster with approximately one billion
objects and when I run a
PG
query, it shows that I have 27,000 objects per
PG.
which query is that, can you provide more details about that cluster and
pool?
However, when I run the same command per pg, the
results are much
less, with only 20 million
objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls |
wc
-l" shows only three objects in the
specified PG.
It seems like your PG and object distribution might not be balanced
very well. Did you check each PG of that pool? The individual PG's
numbers should add up to the total number of objects. Here's a quick
example from an almost empty pool with 8 PGs:
storage01:~ # ceph pg ls-by-pool volumes | awk
'{print $1" - "$2}'
PG - OBJECTS
3.0 - 2
3.1 - 0
3.2 - 2
3.3 - 1
3.4 - 3
3.5 - 0
3.6 - 2
3.7 - 0
```
~# ceph pg ls-by-pool mypool | awk '{print $1" - "$2}
...
9.ff2 - 271268
9.ff3 - 271046
9.ff4 - 271944
9.ff5 - 270864
9.ff6 - 272122
9.ff7 - 272244
9.ff8 - 271638
9.ff9 - 271702
9.ffa - 270906
9.ffb - 271114
9.ffc - 271986
9.ffd - 270766
9.ffe - 271702
9.fff - 271693
...
```
and it sums up to 10 objects which matches the
total stats:
storage01:~ # ceph df | grep -E "OBJECTS|volumes"
POOL ID PGS STORED OBJECTS USED %USED
MAX AVAIL
volumes 3 8 379 B 10 960 KiB 0 7.5 GiB
Regards,
Eugen
Zitat von Ramin Najjarbashi <ramin.najarbashi(a)gmail.com>om>:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Hi
I hope this email finds you well. I am reaching out to you because I have
encountered an issue with my CEPH Bluestore cluster and I am seeking your
assistance.
I have a cluster with approximately one billion objects and when I run a
PG
query, it shows that I have 27,000 objects per
PG.
I have run the following command: "rados -p <mypool> ls | wc -l" which
returns the correct number of one billion objects. However, when I run
the
same command per pg, the results are much less,
with only 20 million
objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls |
wc
-l" shows only three objects in the
specified PG.
This is a significant discrepancy and I am concerned about the integrity
of
my data.
Do you have any idea about this discrepancy?
p.s:
I have a total of 30 million objects in a single bucket and versioning
has
not been enabled for this particular bucket.
Thank you for your time and I look forward to your response.
Best regards,
Ramin
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:52 a.m.
On 04.02.23 20:54, Ramin Najjarbashi wrote:
ceph df | grep mypoo
--- POOLS ---
POOL OBJECTS
mypool 1.11G
---------------
and from this, I got 8.8M objects :
for item in `radosgw-admin user list | jq -r ".[]" | head`; do
B_OBJ=$(radosgw-admin user stats --uid $item 2>/dev/null | jq -r '.stats |
select(.num_objects > 0) | .num_objects'); SUM=$((SUM + B_OBJ)); done
You have mixed RADOS objects and S3 objects.
These are two different layers. Only small (< 4MB) S3 objects are stored
in a single RADOS object. Larger S3 objects are split into multiple 4MB
sized RAOS objects by the rados-gateway.
This is why you see much more RADOS objects than S3 objects.
Regards
--
Robert Sander
Heinlein Support GmbH
Linux: Akademie - Support - Hosting
http://www.heinlein-support.de
Tel: 030-405051-43
Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
5:42 a.m.
Thank you for your email and for providing the solution to check for shadow
and multipart objects in CEPH. I have checked the objects in my CEPH
cluster and found the following results:
The command rados -p <pool> ls | grep --text -vE "shadow|multipart" | wc
-l
returns about 80 million objects.
The command radosgw-admin user stats --uid <user> 2>/dev/null | jq -r
'.stats | select(.num_objects > 0) | .num_objects' returns 889684340
objects, including all buckets for all users.
The data match and are sensible, but I still encounter an inconsistency
when trying to get objects per PG with the rados command. The sum of all
objects in all PGs without shadow and multipart objects is 18 million.
It appears that the S3 objects are stored in the RADOS layer as follows:
Small objects (less than 4 MB) are stored in RADOS without any prefix. If
an object is larger, it is split into multiple objects, each one 4 MB, and
the remaining part (less than 4 MB) is stored as a shadow file. However, in
any case, all S3 objects have a corresponding object in RADOS as a header
that holds the metadata of the object.
https://access.redhat.com/solutions/4177821
Please let me know if you have any further suggestions or if there is
anything else I can assist with.
On Mon, Feb 6, 2023 at 4:24 PM Robert Sander <r.sander(a)heinlein-support.de>
wrote:
On 04.02.23 20:54, Ramin Najjarbashi wrote:
ceph df | grep mypoo
--- POOLS ---
POOL OBJECTS
mypool 1.11G
---------------
and from this, I got 8.8M objects :
for item in `radosgw-admin user list | jq -r ".[]" | head`; do
B_OBJ=$(radosgw-admin user stats --uid $item 2>/dev/null | jq -r '.stats
|
select(.num_objects > 0) | .num_objects');
SUM=$((SUM + B_OBJ)); done
You have mixed RADOS objects and S3 objects.
These are two different layers. Only small (< 4MB) S3 objects are stored
in a single RADOS object. Larger S3 objects are split into multiple 4MB
sized RAOS objects by the rados-gateway.
This is why you see much more RADOS objects than S3 objects.
Regards
--
Robert Sander
Heinlein Support GmbH
Linux: Akademie - Support - Hosting
http://www.heinlein-support.de
Tel: 030-405051-43
Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
444
days inactive
449
days old
5 comments
3 participants
participants (3)
-
Eugen Block -
Ramin Najjarbashi -
Robert Sander