Large omap in index pool even if properly sharded and not "OVER"
Hi, I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects. Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK" This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1" This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920 2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458 Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing? Thank you in advance for your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hi, the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, Yeah, still: the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005 and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378 And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588 This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc... I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this: pg26.427 osd.261 host8 osd.488 host13 osd.276 host4 pg26.606 osd.443 host12 osd.197 host8 osd.524 host14 pg26.78c osd.89 host7 osd.406 host11 osd.254 host6 If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕 Ty ________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER" Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Hi, the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Are those three different buckets? Could you share the stats for each of them? radosgw-admin bucket stats --bucket=<BUCKET> Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d ________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER" Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Are those three different buckets? Could you share the stats for each of them? radosgw-admin bucket stats --bucket=<BUCKET> Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard: 1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions 2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Casey, 1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... 2. Regarding multiparts, if it would have multipart thrash, it would be listed here: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... as a rgw.multimeta under the usage, right? 3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it? Thank you ________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER" Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard: 1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions 2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi Casey,
1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d...
2. Regarding multiparts, if it would have multipart thrash, it would be listed here: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... as a rgw.multimeta under the usage, right?
3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it?
some replication logs are in the log pool, but the per-object logs are stored in the bucket index objects. you can inspect these with `radosgw-admin bilog list --bucket=X`. by default, that will only list --max-entries=1000. you can add --shard-id=Y to look at specific 'large omap' objects even if your single-site bucket doesn't exist on the secondary zone, changes on the primary zone are probably still generating these bilog entries. you would need to do something like `radosgw-admin bucket sync disable --bucket=X` to make it stop. because you don't expect these changes to replicate, it's safe to delete any of this bucket's bilog entries with `radosgw-admin bilog trim --end-marker 9 --bucket=X`. depending on ceph version, you may need to run this trim command in a loop until the `bilog list` output is empty radosgw does eventually trim bilogs in the background after they're processed, but the secondary zone isn't processing them in this case
Thank you
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard:
1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions
2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup
separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings
On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Casey, Thanks for that info on the bilog. I'm in a similar situation with large omap objects and we have also had to reshard buckets on multisite losing the index on the secondary. We also now have a lot of buckets with sync disable so I wanted to check that it's always safe to trim the bilog on buckets with sync disabled? I can see some stale entries with "completed" state and a timestamp of a number of months ago but also some that say pending and have no timestamp. Istvan, I can also possibly help with your orphaned 40TB on the secondary zone. Each object has the bucket marker in its name. If you do a `rados -p {pool_name} ls` and find all the ones that start with the bucket marker (found with `radosgw-admin bucket stats --bucket={bucket_name}`) then you can do one of two things: 1, `rados rm` the object 2, restore the index with info from the object itself - create a dummy index template (use `radosgw-admin bi get` on a known good index to get the structure) - grab the etag from the object xattribs and use this and the name in the template (`rados -p {pool} getxattr {objname} user.rgw.etag`) - use ` radosgw-admin bi put` to create the index - use `radosgw-admin bucket check --check-objects --fix --bucket={bucket_name}` to fix up the bucket object count and object sizes at the end This process takes quite some time and I can't say if it's 100% perfect but it enabled us to get to a state where we could delete the buckets and clean up the objects. I hope this helps. Regards, Richard On Thu, 11 Jul 2024 at 01:25, Casey Bodley <cbodley@redhat.com> wrote:
On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi Casey,
1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d...
2. Regarding multiparts, if it would have multipart thrash, it would be listed here: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... as a rgw.multimeta under the usage, right?
3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it?
some replication logs are in the log pool, but the per-object logs are stored in the bucket index objects. you can inspect these with `radosgw-admin bilog list --bucket=X`. by default, that will only list --max-entries=1000. you can add --shard-id=Y to look at specific 'large omap' objects
even if your single-site bucket doesn't exist on the secondary zone, changes on the primary zone are probably still generating these bilog entries. you would need to do something like `radosgw-admin bucket sync disable --bucket=X` to make it stop. because you don't expect these changes to replicate, it's safe to delete any of this bucket's bilog entries with `radosgw-admin bilog trim --end-marker 9 --bucket=X`. depending on ceph version, you may need to run this trim command in a loop until the `bilog list` output is empty
radosgw does eventually trim bilogs in the background after they're processed, but the secondary zone isn't processing them in this case
Thank you
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard:
1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions
2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup
separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings
On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Jul 10, 2024 at 6:23 PM Richard Bade <hitrich@gmail.com> wrote:
Hi Casey, Thanks for that info on the bilog. I'm in a similar situation with large omap objects and we have also had to reshard buckets on multisite losing the index on the secondary. We also now have a lot of buckets with sync disable so I wanted to check that it's always safe to trim the bilog on buckets with sync disabled?
in general, the only risk of trimming bilogs is that they refer to changes you still want to replicate after `bucket sync disable`, it's fine to trim if it isn't happening automatically. it's fine even if you want to `bucket sync enable` later, because that restarts a 'full sync' and skips any bilog entries from before
I can see some stale entries with "completed" state and a timestamp of a number of months ago but also some that say pending and have no timestamp.
Istvan, I can also possibly help with your orphaned 40TB on the secondary zone. Each object has the bucket marker in its name. If you do a `rados -p {pool_name} ls` and find all the ones that start with the bucket marker (found with `radosgw-admin bucket stats --bucket={bucket_name}`) then you can do one of two things: 1, `rados rm` the object 2, restore the index with info from the object itself - create a dummy index template (use `radosgw-admin bi get` on a known good index to get the structure) - grab the etag from the object xattribs and use this and the name in the template (`rados -p {pool} getxattr {objname} user.rgw.etag`) - use ` radosgw-admin bi put` to create the index - use `radosgw-admin bucket check --check-objects --fix --bucket={bucket_name}` to fix up the bucket object count and object sizes at the end
This process takes quite some time and I can't say if it's 100% perfect but it enabled us to get to a state where we could delete the buckets and clean up the objects. I hope this helps.
Regards, Richard
On Thu, 11 Jul 2024 at 01:25, Casey Bodley <cbodley@redhat.com> wrote:
On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi Casey,
1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d...
2. Regarding multiparts, if it would have multipart thrash, it would be listed here: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... as a rgw.multimeta under the usage, right?
3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it?
some replication logs are in the log pool, but the per-object logs are stored in the bucket index objects. you can inspect these with `radosgw-admin bilog list --bucket=X`. by default, that will only list --max-entries=1000. you can add --shard-id=Y to look at specific 'large omap' objects
even if your single-site bucket doesn't exist on the secondary zone, changes on the primary zone are probably still generating these bilog entries. you would need to do something like `radosgw-admin bucket sync disable --bucket=X` to make it stop. because you don't expect these changes to replicate, it's safe to delete any of this bucket's bilog entries with `radosgw-admin bilog trim --end-marker 9 --bucket=X`. depending on ceph version, you may need to run this trim command in a loop until the `bilog list` output is empty
radosgw does eventually trim bilogs in the background after they're processed, but the secondary zone isn't processing them in this case
Thank you
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard:
1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions
2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup
separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings
On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
----- Le 11 Juil 24, à 0:23, Richard Bade hitrich@gmail.com a écrit :
Hi Casey, Thanks for that info on the bilog. I'm in a similar situation with large omap objects and we have also had to reshard buckets on multisite losing the index on the secondary. We also now have a lot of buckets with sync disable so I wanted to check that it's always safe to trim the bilog on buckets with sync disabled? I can see some stale entries with "completed" state and a timestamp of a number of months ago but also some that say pending and have no timestamp.
Istvan, I can also possibly help with your orphaned 40TB on the secondary zone. Each object has the bucket marker in its name. If you do a `rados -p {pool_name} ls` and find all the ones that start with the bucket marker (found with `radosgw-admin bucket stats --bucket={bucket_name}`) then you can do one of two things: 1, `rados rm` the object 2, restore the index with info from the object itself - create a dummy index template (use `radosgw-admin bi get` on a known good index to get the structure) - grab the etag from the object xattribs and use this and the name in the template (`rados -p {pool} getxattr {objname} user.rgw.etag`) - use ` radosgw-admin bi put` to create the index - use `radosgw-admin bucket check --check-objects --fix --bucket={bucket_name}` to fix up the bucket object count and object sizes at the end
One could also use `radosgw-admin object reindex --bucket {bucket_name}` to scan the data pool for objects that belong to a given bucket and add those objects back to the bucket index. Same logic as rgw-restore-bucket-index [1][2] script that has proven to be successful in recovering bucket indexes destroyed by resharding [3]. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/man/8/rgw-restore-bucket-index/ [2] https://github.com/ceph/ceph/blob/main/src/rgw/rgw-restore-bucket-index [3] https://github.com/ceph/ceph/pull/50329
This process takes quite some time and I can't say if it's 100% perfect but it enabled us to get to a state where we could delete the buckets and clean up the objects. I hope this helps.
Regards, Richard
On Thu, 11 Jul 2024 at 01:25, Casey Bodley <cbodley@redhat.com> wrote:
On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi Casey,
1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d...
2. Regarding multiparts, if it would have multipart thrash, it would be listed here: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... as a rgw.multimeta under the usage, right?
3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it?
some replication logs are in the log pool, but the per-object logs are stored in the bucket index objects. you can inspect these with `radosgw-admin bilog list --bucket=X`. by default, that will only list --max-entries=1000. you can add --shard-id=Y to look at specific 'large omap' objects
even if your single-site bucket doesn't exist on the secondary zone, changes on the primary zone are probably still generating these bilog entries. you would need to do something like `radosgw-admin bucket sync disable --bucket=X` to make it stop. because you don't expect these changes to replicate, it's safe to delete any of this bucket's bilog entries with `radosgw-admin bilog trim --end-marker 9 --bucket=X`. depending on ceph version, you may need to run this trim command in a loop until the `bilog list` output is empty
radosgw does eventually trim bilogs in the background after they're processed, but the secondary zone isn't processing them in this case
Thank you
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard:
1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions
2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup
separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings
On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, But this not cleaning right? Just restore if lost. Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Friday, July 12, 2024 6:52 PM To: Richard Bade <hitrich@gmail.com>; Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Casey Bodley <cbodley@redhat.com>; Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER" Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ ----- Le 11 Juil 24, à 0:23, Richard Bade hitrich@gmail.com a écrit :
Hi Casey, Thanks for that info on the bilog. I'm in a similar situation with large omap objects and we have also had to reshard buckets on multisite losing the index on the secondary. We also now have a lot of buckets with sync disable so I wanted to check that it's always safe to trim the bilog on buckets with sync disabled? I can see some stale entries with "completed" state and a timestamp of a number of months ago but also some that say pending and have no timestamp.
Istvan, I can also possibly help with your orphaned 40TB on the secondary zone. Each object has the bucket marker in its name. If you do a `rados -p {pool_name} ls` and find all the ones that start with the bucket marker (found with `radosgw-admin bucket stats --bucket={bucket_name}`) then you can do one of two things: 1, `rados rm` the object 2, restore the index with info from the object itself - create a dummy index template (use `radosgw-admin bi get` on a known good index to get the structure) - grab the etag from the object xattribs and use this and the name in the template (`rados -p {pool} getxattr {objname} user.rgw.etag`) - use ` radosgw-admin bi put` to create the index - use `radosgw-admin bucket check --check-objects --fix --bucket={bucket_name}` to fix up the bucket object count and object sizes at the end
One could also use `radosgw-admin object reindex --bucket {bucket_name}` to scan the data pool for objects that belong to a given bucket and add those objects back to the bucket index. Same logic as rgw-restore-bucket-index [1][2] script that has proven to be successful in recovering bucket indexes destroyed by resharding [3]. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/man/8/rgw-restore-bucket-index/ [2] https://github.com/ceph/ceph/blob/main/src/rgw/rgw-restore-bucket-index [3] https://github.com/ceph/ceph/pull/50329
This process takes quite some time and I can't say if it's 100% perfect but it enabled us to get to a state where we could delete the buckets and clean up the objects. I hope this helps.
Regards, Richard
On Thu, 11 Jul 2024 at 01:25, Casey Bodley <cbodley@redhat.com> wrote:
On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi Casey,
1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d...
2. Regarding multiparts, if it would have multipart thrash, it would be listed here: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... as a rgw.multimeta under the usage, right?
3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it?
some replication logs are in the log pool, but the per-object logs are stored in the bucket index objects. you can inspect these with `radosgw-admin bilog list --bucket=X`. by default, that will only list --max-entries=1000. you can add --shard-id=Y to look at specific 'large omap' objects
even if your single-site bucket doesn't exist on the secondary zone, changes on the primary zone are probably still generating these bilog entries. you would need to do something like `radosgw-admin bucket sync disable --bucket=X` to make it stop. because you don't expect these changes to replicate, it's safe to delete any of this bucket's bilog entries with `radosgw-admin bilog trim --end-marker 9 --bucket=X`. depending on ceph version, you may need to run this trim command in a loop until the `bilog list` output is empty
radosgw does eventually trim bilogs in the background after they're processed, but the secondary zone isn't processing them in this case
Thank you
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard:
1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions
2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup
separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings
On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Right. This procedure would recatalog any lost S3 objects (out of the 40TB) and would allow you to delete them using S3 afterwards if that's what you want. Note that I don't think it handles different versions of S3 objects, if any exist, so you might still end up with orphaned data in the RADOS pool. If you no longer have any interest in this bucket, you could simply purge the bucket with all its data, then use 'rados ls' to list any orphan objects whose names begin with the bucket prefix (make sure you saved this information before deleting the bucket), and finally use 'rados rm' to remove them. Regards, Frédéric. ----- Le 15 Juil 24, à 5:30, Istvan Szabo, Agoda <Istvan.Szabo@agoda.com> a écrit :
Hi,
But this not cleaning right? Just restore if lost.
Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: [ mailto:istvan.szabo@agoda.com | istvan.szabo@agoda.com ] ---------------------------------------------------
From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Friday, July 12, 2024 6:52 PM To: Richard Bade <hitrich@gmail.com>; Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Casey Bodley <cbodley@redhat.com>; Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER" Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
----- Le 11 Juil 24, à 0:23, Richard Bade hitrich@gmail.com a écrit :
Hi Casey, Thanks for that info on the bilog. I'm in a similar situation with large omap objects and we have also had to reshard buckets on multisite losing the index on the secondary. We also now have a lot of buckets with sync disable so I wanted to check that it's always safe to trim the bilog on buckets with sync disabled? I can see some stale entries with "completed" state and a timestamp of a number of months ago but also some that say pending and have no timestamp.
Istvan, I can also possibly help with your orphaned 40TB on the secondary zone. Each object has the bucket marker in its name. If you do a `rados -p {pool_name} ls` and find all the ones that start with the bucket marker (found with `radosgw-admin bucket stats --bucket={bucket_name}`) then you can do one of two things: 1, `rados rm` the object 2, restore the index with info from the object itself - create a dummy index template (use `radosgw-admin bi get` on a known good index to get the structure) - grab the etag from the object xattribs and use this and the name in the template (`rados -p {pool} getxattr {objname} user.rgw.etag`) - use ` radosgw-admin bi put` to create the index - use `radosgw-admin bucket check --check-objects --fix --bucket={bucket_name}` to fix up the bucket object count and object sizes at the end
One could also use `radosgw-admin object reindex --bucket {bucket_name}` to scan the data pool for objects that belong to a given bucket and add those objects back to the bucket index.
Same logic as rgw-restore-bucket-index [1][2] script that has proven to be successful in recovering bucket indexes destroyed by resharding [3].
Regards, Frédéric.
[1] [ https://docs.ceph.com/en/latest/man/8/rgw-restore-bucket-index/ | https://docs.ceph.com/en/latest/man/8/rgw-restore-bucket-index/ ] [2] [ https://github.com/ceph/ceph/blob/main/src/rgw/rgw-restore-bucket-index | https://github.com/ceph/ceph/blob/main/src/rgw/rgw-restore-bucket-index ] [3] [ https://github.com/ceph/ceph/pull/50329 | https://github.com/ceph/ceph/pull/50329 ]
This process takes quite some time and I can't say if it's 100% perfect but it enabled us to get to a state where we could delete the buckets and clean up the objects. I hope this helps.
Regards, Richard
On Thu, 11 Jul 2024 at 01:25, Casey Bodley <cbodley@redhat.com> wrote:
On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi Casey,
1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: [ https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... | https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... ]
2. Regarding multiparts, if it would have multipart thrash, it would be listed here: [ https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... | https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... ] as a rgw.multimeta under the usage, right?
3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it?
some replication logs are in the log pool, but the per-object logs are stored in the bucket index objects. you can inspect these with `radosgw-admin bilog list --bucket=X`. by default, that will only list --max-entries=1000. you can add --shard-id=Y to look at specific 'large omap' objects
even if your single-site bucket doesn't exist on the secondary zone, changes on the primary zone are probably still generating these bilog entries. you would need to do something like `radosgw-admin bucket sync disable --bucket=X` to make it stop. because you don't expect these changes to replicate, it's safe to delete any of this bucket's bilog entries with `radosgw-admin bilog trim --end-marker 9 --bucket=X`. depending on ceph version, you may need to run this trim command in a loop until the `bilog list` output is empty
radosgw does eventually trim bilogs in the background after they're processed, but the secondary zone isn't processing them in this case
Thank you
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard:
1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions
2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup
separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings
On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: [ https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d | https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d ]
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) [ https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc... | https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc... ]
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
> Hi,
> I have a pretty big bucket which sharded with 1999 shard so in > theory can hold close to 200m objects (199.900.000). > Currently it has 54m objects.
> Bucket limit check looks also good: > "bucket": ""xyz, > "tenant": "", > "num_objects": 53619489, > "num_shards": 1999, > "objects_per_shard": 26823, > "fill_status": "OK"
> This is the bucket id: > "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
> This is the log lines: > 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster > [WRN] Large omap object found. Object: > 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head > PG: 26.3a67cc27 (26.427) Key count: 236919 Size > (bytes): > 89969920
> 2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] > Large omap object found. Object: > 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head > PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size > (bytes): > 95560458
> Tried to deep scrub the affected pgs, tried to deep-scrub the > mentioned osds in the log but didn't help. > Why? What I'm missing?
> Thank you in advance for your help.
> ________________________________ > This message is confidential and is for the sole use of the intended > recipient(s). It may also be privileged or otherwise protected by > copyright or other legal rules. If you have received it by mistake > please let us know by reply email and delete it from your system. It > is prohibited to copy this message or disclose its content to > anyone. Any confidentiality or privilege is not waived or lost by > any mistaken delivery or unauthorized disclosure of the message. All > messages sent to and from Agoda may be monitored to ensure > compliance with company policies, to protect the company's interests > and to remove potential malware. Electronic messages may be > intercepted, amended, lost or deleted, or contain viruses. > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Finally this article was the solution: https://access.redhat.com/solutions/6450561 Main point is to trim the bilog: radosgw-admin bilog trim --bucket="bucket-name" --bucket-id="bucket-id" Then scrub, done! Happy day! ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Monday, July 15, 2024 8:22 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Richard Bade <hitrich@gmail.com>; Casey Bodley <cbodley@redhat.com>; Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER" Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Right. This procedure would recatalog any lost S3 objects (out of the 40TB) and would allow you to delete them using S3 afterwards if that's what you want. Note that I don't think it handles different versions of S3 objects, if any exist, so you might still end up with orphaned data in the RADOS pool. If you no longer have any interest in this bucket, you could simply purge the bucket with all its data, then use 'rados ls' to list any orphan objects whose names begin with the bucket prefix (make sure you saved this information before deleting the bucket), and finally use 'rados rm' to remove them. Regards, Frédéric. ----- Le 15 Juil 24, à 5:30, Istvan Szabo, Agoda <Istvan.Szabo@agoda.com> a écrit : Hi, But this not cleaning right? Just restore if lost. Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Friday, July 12, 2024 6:52 PM To: Richard Bade <hitrich@gmail.com>; Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Casey Bodley <cbodley@redhat.com>; Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER" Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ ----- Le 11 Juil 24, à 0:23, Richard Bade hitrich@gmail.com a écrit :
Hi Casey, Thanks for that info on the bilog. I'm in a similar situation with large omap objects and we have also had to reshard buckets on multisite losing the index on the secondary. We also now have a lot of buckets with sync disable so I wanted to check that it's always safe to trim the bilog on buckets with sync disabled? I can see some stale entries with "completed" state and a timestamp of a number of months ago but also some that say pending and have no timestamp.
Istvan, I can also possibly help with your orphaned 40TB on the secondary zone. Each object has the bucket marker in its name. If you do a `rados -p {pool_name} ls` and find all the ones that start with the bucket marker (found with `radosgw-admin bucket stats --bucket={bucket_name}`) then you can do one of two things: 1, `rados rm` the object 2, restore the index with info from the object itself - create a dummy index template (use `radosgw-admin bi get` on a known good index to get the structure) - grab the etag from the object xattribs and use this and the name in the template (`rados -p {pool} getxattr {objname} user.rgw.etag`) - use ` radosgw-admin bi put` to create the index - use `radosgw-admin bucket check --check-objects --fix --bucket={bucket_name}` to fix up the bucket object count and object sizes at the end
One could also use `radosgw-admin object reindex --bucket {bucket_name}` to scan the data pool for objects that belong to a given bucket and add those objects back to the bucket index. Same logic as rgw-restore-bucket-index [1][2] script that has proven to be successful in recovering bucket indexes destroyed by resharding [3]. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/man/8/rgw-restore-bucket-index/ [2] https://github.com/ceph/ceph/blob/main/src/rgw/rgw-restore-bucket-index [3] https://github.com/ceph/ceph/pull/50329
This process takes quite some time and I can't say if it's 100% perfect but it enabled us to get to a state where we could delete the buckets and clean up the objects. I hope this helps.
Regards, Richard
On Thu, 11 Jul 2024 at 01:25, Casey Bodley <cbodley@redhat.com> wrote:
On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi Casey,
1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d...
2. Regarding multiparts, if it would have multipart thrash, it would be listed here: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... as a rgw.multimeta under the usage, right?
3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it?
some replication logs are in the log pool, but the per-object logs are stored in the bucket index objects. you can inspect these with `radosgw-admin bilog list --bucket=X`. by default, that will only list --max-entries=1000. you can add --shard-id=Y to look at specific 'large omap' objects
even if your single-site bucket doesn't exist on the secondary zone, changes on the primary zone are probably still generating these bilog entries. you would need to do something like `radosgw-admin bucket sync disable --bucket=X` to make it stop. because you don't expect these changes to replicate, it's safe to delete any of this bucket's bilog entries with `radosgw-admin bilog trim --end-marker 9 --bucket=X`. depending on ceph version, you may need to run this trim command in a loop until the `bilog list` output is empty
radosgw does eventually trim bilogs in the background after they're processed, but the secondary zone isn't processing them in this case
Thank you
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard:
1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions
2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup
separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings
On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hi Casey, So the multisite thing when we resharded the bucket I've completely disabled and removed the bucket from the sync before like, disable, removed the pipe and everything step by step, finally period updated so this is not syncing I'm kind of sure so I think we can focus on the master zone only and after I check on the secondary site the bilogs, there I can try to trim. 2nd zone shouldn't cause large omap in the master zone if bucket sync already disabled I guess. The 2nd that you've suggested is more interesting, so when you say shard id, let's say from this entry ( .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151), the 151 is the shard that you mentioned, which means in cli I guess: radosgw-admin bilog list --bucket=bucketname --shard-id=151 (We use octopus) And this huge list we should identify with the app owner objects by objects which is not there anymore and they don't use? Not easy for sure 😄Not sure is there any proper way to do this? We've run bucket check with fix but that one didn't help. Thank you ________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Wednesday, July 10, 2024 8:24 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER" Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi Casey,
1. Regarding versioning, the user doesn't use verisoning it if I'm not mistaken: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d...
2. Regarding multiparts, if it would have multipart thrash, it would be listed here: https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d... as a rgw.multimeta under the usage, right?
3. Regarding the multisite idea, this bucket has been a multisite bucket last year but we had to reshard (accepting to loose the replica on the 2nd site and just keep it in the master site) and that time as expected it has disappeared completely from the 2nd site (I guess the 40TB thrash still there but can't really find it how to clean 🙁 ). Now it is a single site bucket. Also it is the index pool, multisite logs should go to the rgw.log pool shouldn't it?
some replication logs are in the log pool, but the per-object logs are stored in the bucket index objects. you can inspect these with `radosgw-admin bilog list --bucket=X`. by default, that will only list --max-entries=1000. you can add --shard-id=Y to look at specific 'large omap' objects even if your single-site bucket doesn't exist on the secondary zone, changes on the primary zone are probably still generating these bilog entries. you would need to do something like `radosgw-admin bucket sync disable --bucket=X` to make it stop. because you don't expect these changes to replicate, it's safe to delete any of this bucket's bilog entries with `radosgw-admin bilog trim --end-marker 9 --bucket=X`. depending on ceph version, you may need to run this trim command in a loop until the `bilog list` output is empty radosgw does eventually trim bilogs in the background after they're processed, but the secondary zone isn't processing them in this case
Thank you
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, July 9, 2024 10:39 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
in general, these omap entries should be evenly spread over the bucket's index shard objects. but there are two features that may cause entries to clump on a single shard:
1. for versioned buckets, multiple versions of the same object name map to the same index shard. this can become an issue if an application is repeatedly overwriting an object without cleaning up old versions. lifecycle rules can help to manage these noncurrent versions
2. during a multipart upload, all of the parts are tracked on the same index shard as the final object name. if applications are leaving a lot of incomplete multipart uploads behind (especially if they target the same object name) this can lead to similar clumping. the S3 api has operations to list and abort incomplete multipart uploads, along with lifecycle rules to automate their cleanup
separately, multisite clusters use these same index shards to store replication logs. if sync gets far enough behind, these log entries can also lead to large omap warnings
On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It's the same bucket: https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 8:03 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005
and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378
And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588
This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cc...
I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this:
pg26.427 osd.261 host8 osd.488 host13 osd.276 host4
pg26.606 osd.443 host12 osd.197 host8 osd.524 host14
pg26.78c osd.89 host7 osd.406 host11 osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕
Ty
________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Large omap in index pool even if properly sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects.
Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK"
This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing?
Thank you in advance for your help.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Casey Bodley
-
Eugen Block
-
Frédéric Nass
-
Richard Bade
-
Szabo, Istvan (Agoda)