On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi Casey,
1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d...
2. Regarding multiparts, if it would have multipart thrash, it would be listed here: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... as a rgw.multimeta under the usage, right?
3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it?
some replication logs are in the log pool, but the per-object logs are stored in the bucket index objects. you can inspect these with `radosgw-admin bilog list --bucket=X`. by default, that will only list --max-entries=1000. you can add --shard-id=Y to look at specific 'large omap' objects even if your single-site bucket doesn't exist on the secondary zone, changes on the primary zone are probably still generating these bilog entries. you would need to do something like `radosgw-admin bucket sync disable --bucket=X` to make it stop. because you don't expect these changes to replicate, it's safe to delete any of this bucket's bilog entries with `radosgw-admin bilog trim --end-marker 9 --bucket=X`. depending on ceph version, you may need to run this trim command in a loop until the `bilog list` output is empty radosgw does eventually trim bilogs in the background after they're processed, but the secondary zone isn't processing them in this case
Thank you
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard:
1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions
2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup
separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings
On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io