Hi Istvan: Thanks for your reply. Does directional sync solve the problem? I tried to run `radosgw-admin sync init`, bit it still did not work. :( Thanks Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年2月26日周五 上午7:47写道：

Hi Istvan: Thank you so much. :) Also I open an issue on *tracker.ceph.com <http://tracker.ceph.com>*, hope we could get some responses soon. Issue: https://tracker.ceph.com/issues/49542 Thanks Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; 于2021年3月1日周一 上午10:01写道：

Yeah, doesn't work. Last week they fixed my problem ticket which caused the crashes, and due to the crashes stopped the replication I'll give a try this week again after the update if the daemon doesn't crash, maybe it will work, because if crash hasn't happened, the data was synced. Fingers crossed ;) Don't give up 😄 ________________________________ From: 特木勒 &lt;twl007(a)gmail.com&gt; Sent: Monday, March 22, 2021 1:38 PM To: Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; Cc: ceph-users(a)ceph.io &lt;ceph-users(a)ceph.io&gt; Subject: Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site does not sync olds data Hi Istvan: Do you have any update on directional sync? I am trying to upgrade cluster to 15.2.10 to see if the problem is solved. :( Thanks Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> 于2021年3月1日周一 上午10:01写道： So-so. I had some interruption so it failed on one site, but the other is kind of working. This is the first time when I saw data caught up in the radosgw-admin data sync status on 1 side. Today will finish the other problematic site, I’ll let you know the result is it working or not. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: 特木勒 <twl007@gmail.com<mailto:twl007@gmail.com>> Sent: Sunday, February 28, 2021 1:34 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site does not sync olds data Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________ Hi Istvan: Thanks for your reply. Does directional sync solve the problem? I tried to run `radosgw-admin sync init`, bit it still did not work. :( Thanks Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> 于2021年2月26日周五 上午7:47写道： Same for me, 15.2.8 also. I’m trying directional sync now, looks like symmetrical has issue. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com><mailto:istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>> --------------------------------------------------- On 2021. Feb 26., at 1:03, 特木勒 <twl007@gmail.com<mailto:twl007@gmail.com>> wrote: Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________ Hi all: ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) I have a strange question, I just create a multiple site for Ceph cluster. But I notice the old data of source cluster is not synced. Only new data will be synced into second zone cluster. Is there anything I need to do to enable full sync for bucket or this is a bug? Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. Internal

Hi Istvan: We just upgraded whole cluster to 15.2.10 and the multiple site still cannot sync whole objects to secondary cluster. 🙁 Do you have any suggestions on this? And I open another issues in ceph tracker site: https://tracker.ceph.com/issues/50474 Hope someone could go to check this issue. Thanks 特木勒 &lt;twl007(a)gmail.xn--com&gt;2021322-6x8vy47pnp0auje 周一下午9:08写道：

Hi Istvan: Thanks Amit's suggestion. I followed his suggestion to fix bucket index and re-do sync on buckets, but it still did not work for me. Then I tried to use bucket rewrite command to rewrite all the objects in buckets and it works for me. I think the reason is there's something wrong with bucket index and rewrite has rebuilt the index. Here's the command I use: `sudo radosgw-admin bucket rewrite -b BUCKET-NAME --min-rewrite-size 0` Maybe you can try this to fix the sync issues. @Amit Ghadge &lt;amitg.b14(a)gmail.com&gt; Thanks for your suggestions. Without your suggestions, I will not notice something wrong with index part. Thanks :) Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; 于2021年4月26日周一 上午9:57写道：

Hi, I hit the same errors when doing multisite sync between luminous and octopus, but what I founded is that my sync errors was mainly on old multipart and shadow objects, at the "rados level" if I might say. (leftovers from luminous bugs) So check at the "user level", using s3cmd/awscli and the objects md5, you will probably find that your pretty much in sync. Hopefully. Cheers! On 4/25/21 11:29 PM, 特木勒 wrote: > [Externe UL*] > > Another problem I notice for a new bucket, the first object in the bucket > will not be sync. the sync will start with the second object. I tried to > fix the index on the bucket and manually rerun bucket sync, but the first > object still does not sync with secondary cluster. > > Do you have any ideas for this issue? > > Thanks > > 特木勒 &lt;twl007(a)gmail.com&gt; 于2021年4月26日周一 上午11:16写道： > >> Hi Istvan: >> >> Thanks Amit's suggestion. >> >> I followed his suggestion to fix bucket index and re-do sync on buckets, >> but it still did not work for me. >> >> Then I tried to use bucket rewrite command to rewrite all the objects in >> buckets and it works for me. I think the reason is there's something wrong >> with bucket index and rewrite has rebuilt the index. >> >> Here's the command I use: >> `sudo radosgw-admin bucket rewrite -b BUCKET-NAME --min-rewrite-size 0` >> >> Maybe you can try this to fix the sync issues. >> >> @Amit Ghadge &lt;amitg.b14(a)gmail.com&gt; Thanks for your suggestions. Without >> your suggestions, I will not notice something wrong with index part. >> >> Thanks :) >> >> Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; 于2021年4月26日周一 上午9:57写道： >> >>> Hi, >>> >>> >>> >>> No, doesn’t work, now we will write our own sync app for ceph, I gave up. >>> >>> >>> >>> Istvan Szabo >>> Senior Infrastructure Engineer >>> --------------------------------------------------- >>> Agoda Services Co., Ltd. >>> e: istvan.szabo(a)agoda.com >>> --------------------------------------------------- >>> >>> >>> >>> *From:* 特木勒 &lt;twl007(a)gmail.com&gt; >>> *Sent:* Friday, April 23, 2021 7:50 PM >>> *To:* Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; >>> *Cc:* ceph-users(a)ceph.io >>> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >>> does not sync olds data >>> >>> >>> >>> Hi Istvan: >>> >>> >>> >>> We just upgraded whole cluster to 15.2.10 and the multiple site still >>> cannot sync whole objects to secondary cluster. 🙁 >>> >>> >>> >>> Do you have any suggestions on this? And I open another issues in ceph >>> tracker site: >>> >>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.c… >>> >>> >>> >>> Hope someone could go to check this issue. >>> >>> >>> >>> Thanks >>> >>> >>> >>> 特木勒 &lt;twl007(a)gmail.xn--com&gt;2021322-6x8vy47pnp0auje 周一下午9:08写道： >>> >>> Thank you~ >>> >>> >>> >>> I will try to upgrade cluster too. Seem like this is the only way for >>> now. 😭 >>> >>> >>> >>> I will let you know once I complete testing. :) >>> >>> >>> >>> Have a good day >>> >>> >>> >>> Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.xn--com&gt;2021322-6x8vy47pnp0auje 周一下午3:38写道： >>> >>> Yeah, doesn't work. Last week they fixed my problem ticket which caused >>> the crashes, and due to the crashes stopped the replication I'll give a try >>> this week again after the update if the daemon doesn't crash, maybe it will >>> work, because if crash hasn't happened, the data was synced. Fingers >>> crossed ;) Don't give up 😄 >>> ------------------------------ >>> >>> *From:* 特木勒 &lt;twl007(a)gmail.com&gt; >>> *Sent:* Monday, March 22, 2021 1:38 PM >>> *To:* Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; >>> *Cc:* ceph-users(a)ceph.io &lt;ceph-users(a)ceph.io&gt; >>> >>> >>> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >>> does not sync olds data >>> >>> >>> >>> Hi Istvan: >>> >>> >>> >>> Do you have any update on directional sync? >>> >>> >>> >>> I am trying to upgrade cluster to 15.2.10 to see if the problem is >>> solved. :( >>> >>> >>> >>> Thanks >>> >>> >>> >>> Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; 于2021年3月1日周一 上午10:01写道： >>> >>> So-so. I had some interruption so it failed on one site, but the other is >>> kind of working. This is the first time when I saw data caught up in the >>> radosgw-admin data sync status on 1 side. >>> >>> Today will finish the other problematic site, I’ll let you know the >>> result is it working or not. >>> >>> >>> >>> Istvan Szabo >>> Senior Infrastructure Engineer >>> --------------------------------------------------- >>> Agoda Services Co., Ltd. >>> e: istvan.szabo(a)agoda.com >>> --------------------------------------------------- >>> >>> >>> >>> *From:* 特木勒 &lt;twl007(a)gmail.com&gt; >>> *Sent:* Sunday, February 28, 2021 1:34 PM >>> *To:* Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; >>> *Cc:* ceph-users(a)ceph.io >>> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >>> does not sync olds data >>> >>> >>> >>> Email received from outside the company. If in doubt don't click links >>> nor open attachments! >>> ------------------------------ >>> >>> Hi Istvan: >>> >>> >>> >>> Thanks for your reply. >>> >>> >>> >>> Does directional sync solve the problem? I tried to run `radosgw-admin >>> sync init`, bit it still did not work. :( >>> >>> >>> >>> Thanks >>> >>> >>> >>> Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; 于2021年2月26日周五 上午7:47写道： >>> >>> Same for me, 15.2.8 also. >>> I’m trying directional sync now, looks like symmetrical has issue. >>> >>> Istvan Szabo >>> Senior Infrastructure Engineer >>> --------------------------------------------------- >>> Agoda Services Co., Ltd. >>> e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> >>> --------------------------------------------------- >>> >>> On 2021. Feb 26., at 1:03, 特木勒 &lt;twl007(a)gmail.com&gt; wrote: >>> >>> Email received from outside the company. If in doubt don't click links >>> nor open attachments! >>> ________________________________ >>> >>> Hi all: >>> >>> ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) >>> >>> I have a strange question, I just create a multiple site for Ceph cluster. >>> But I notice the old data of source cluster is not synced. Only new data >>> will be synced into second zone cluster. >>> >>> Is there anything I need to do to enable full sync for bucket or this is a >>> bug? >>> >>> Thanks >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> ________________________________ >>> This message is confidential and is for the sole use of the intended >>> recipient(s). It may also be privileged or otherwise protected by copyright >>> or other legal rules. If you have received it by mistake please let us know >>> by reply email and delete it from your system. It is prohibited to copy >>> this message or disclose its content to anyone. Any confidentiality or >>> privilege is not waived or lost by any mistaken delivery or unauthorized >>> disclosure of the message. All messages sent to and from Agoda may be >>> monitored to ensure compliance with company policies, to protect the >>> company's interests and to remove potential malware. Electronic messages >>> may be intercepted, amended, lost or deleted, or contain viruses. >>> >>> >>> >>> Internal >>> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > *ATTENTION : L’émetteur de ce courriel est externe à l’Université Laval. > Évitez de cliquer sur un hyperlien, d’ouvrir une pièce jointe ou de transmettre des informations si vous ne connaissez pas l’expéditeur du courriel. En cas de doute, contactez l’équipe de soutien informatique de votre unité ou hameconnage(a)ulaval.ca. > ________________________________ >

So how is your multisite things going at the moment? Seems like with this rewrite you’ve moved further than me 😊 Is it working properly now? If yes, what is the steps to make it work? Where is the magic 😊 ? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: 特木勒 &lt;twl007(a)gmail.com&gt; Sent: Thursday, May 6, 2021 11:27 AM To: Jean-Sebastien Landry &lt;Jean-Sebastien.Landry.6(a)ulaval.ca&gt; Cc: Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt;om>; ceph-users(a)ceph.io; Amit Ghadge &lt;amitg.b14(a)gmail.com&gt; Subject: Re: [ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data Hi Jean: Thanks for your info. Unfortunately I check the secondary cluster and non-objects had been synced. The only way I have is to force rewrite objects for whole buckets. I have tried to set up multiple site between Nautilus and octopus. It works pretty well. But after I upgrade primary cluster to octopus, we have this issue. :( Here is the issue: https://tracker.ceph.com/issues/49542#change-193975 Thanks Jean-Sebastien Landry <Jean-Sebastien.Landry.6@ulaval.ca<mailto:Jean-Sebastien.Landry.6@ulaval.ca>> 于2021年4月27日周二 下午7:52写道： Hi, I hit the same errors when doing multisite sync between luminous and octopus, but what I founded is that my sync errors was mainly on old multipart and shadow objects, at the "rados level" if I might say. (leftovers from luminous bugs) So check at the "user level", using s3cmd/awscli and the objects md5, you will probably find that your pretty much in sync. Hopefully. Cheers! On 4/25/21 11:29 PM, 特木勒 wrote: ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

Ok, will be challenging with an 800 millions object bucket 😃 But I might give a try. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: 特木勒 &lt;twl007(a)gmail.com&gt; Sent: Monday, May 10, 2021 6:53 PM To: Szabo, Istvan (Agoda) &lt;Istvan.Szabo(a)agoda.com&gt; Cc: Jean-Sebastien Landry &lt;Jean-Sebastien.Landry.6(a)ulaval.ca&gt;ca>; ceph-users(a)ceph.io; Amit Ghadge &lt;amitg.b14(a)gmail.com&gt; Subject: Re: [ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data Hi Istvan: Thanks for your help. After we rewrite all the objects that in buckets, the sync seems to work again. We are using this command to rewrite all the objects in specific bucket: `radosgw-admin bucket rewrite —bucket=BUCKET_NAME --min-rewrite-size 0` You can try to run this on 1 bucket and see if it could help you fix the problem. Thank you~ Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> 于2021年5月10日周一 下午12:16写道： So how is your multisite things going at the moment? Seems like with this rewrite you’ve moved further than me 😊 Is it working properly now? If yes, what is the steps to make it work? Where is the magic 😊 ? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: 特木勒 <twl007@gmail.com<mailto:twl007@gmail.com>> Sent: Thursday, May 6, 2021 11:27 AM To: Jean-Sebastien Landry <Jean-Sebastien.Landry.6@ulaval.ca<mailto:Jean-Sebastien.Landry.6@ulaval.ca>> Cc: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>>; ceph-users@ceph.io<mailto:ceph-users@ceph.io>; Amit Ghadge <amitg.b14@gmail.com<mailto:amitg.b14@gmail.com>> Subject: Re: [ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data Hi Jean: Thanks for your info. Unfortunately I check the secondary cluster and non-objects had been synced. The only way I have is to force rewrite objects for whole buckets. I have tried to set up multiple site between Nautilus and octopus. It works pretty well. But after I upgrade primary cluster to octopus, we have this issue. :( Here is the issue: https://tracker.ceph.com/issues/49542#change-193975 Thanks Jean-Sebastien Landry <Jean-Sebastien.Landry.6@ulaval.ca<mailto:Jean-Sebastien.Landry.6@ulaval.ca>> 于2021年4月27日周二 下午7:52写道： Hi, I hit the same errors when doing multisite sync between luminous and octopus, but what I founded is that my sync errors was mainly on old multipart and shadow objects, at the "rados level" if I might say. (leftovers from luminous bugs) So check at the "user level", using s3cmd/awscli and the objects md5, you will probably find that your pretty much in sync. Hopefully. Cheers! On 4/25/21 11:29 PM, 特木勒 wrote: ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.