Le jeudi 11 avril 2024, 23:44:05 CEST Gilles Mocellin a écrit : something else). use checksums, launch several time (sync not just copy). Ah, and of course, if you use versioning, lifecycle rules, bucket policies, or all other "metadata" other than objects in buckets, rclone will NOT copy them. You should re-create (or something like export / import) with awscli or s3cmd.

Hello, I used to use rclone for data synchronization between 2 ceph clusters and for a directional sync from AWS to Ceph. In general, rclone is a really good and reliable pice of software, but could be slow with large amount of syncing objects. Large - 10^6+ objects. As a disclaimer - my experience is 3 years old. Very likely rclone was improved since that time, and it definitely should be considered at least as POC. You can offload rclone's sync operations by skipping hashes and other costly comparisons if this is an appropriate approach in your project. Good luck! Sincerely, Vladimir. Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: Casey Bodley &lt;cbodley(a)redhat.com&gt; Sent: Thursday, April 11, 2024 5:29:30 PM To: James McClune &lt;mcclune.789(a)gmail.com&gt; Cc: ceph-users(a)ceph.io &lt;ceph-users(a)ceph.io&gt; Subject: [ceph-users] Re: Migrating from S3 to Ceph RGW (Cloud Sync Module) unfortunately, this cloud sync module only exports data from ceph to a remote s3 endpoint, not the other way around: "This module syncs zone data to a remote cloud service. The sync is unidirectional; data is not synced back from the remote zone." i believe that rclone supports copying from one s3 endpoint to another. does anyone have experience with that? On Thu, Apr 11, 2024 at 4:45 PM James McClune &lt;mcclune.789(a)gmail.com&gt; wrote: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

At Clyso we've been building a tool that can migrate S3 data around called Chorus. Normally I wouldn't promote it here, but it's open source and sounds like it might be useful in this case. I don't work on it myself, but thought I'd mention it: https://github.com/clyso/chorus One problem with rclone is that it doesn't support copying ACLs AFAIK. There was an incomplete PR for that a while back that never got merged. We did some work on that as well, though I'm not sure if it's been up streamed yet. Mark On 4/11/24 16:29, Casey Bodley wrote: > unfortunately, this cloud sync module only exports data from ceph to a > remote s3 endpoint, not the other way around: > > "This module syncs zone data to a remote cloud service. The sync is > unidirectional; data is not synced back from the remote zone." > > i believe that rclone supports copying from one s3 endpoint to > another. does anyone have experience with that? > > On Thu, Apr 11, 2024 at 4:45 PM James McClune &lt;mcclune.789(a)gmail.com&gt; wrote: >> >> Hello Ceph User Community, >> >> I currently have a large Amazon S3 environment with terabytes of data >> spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an >> on-site Ceph cluster using the RGW. I'm trying to figure out the >> most efficient way to achieve this. Looking through the documentation, I >> found articles related to the cloud sync module, released in Mimic ( >> https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched >> a video on the cloud sync module as well. It *sounds* like this is the >> functionality I'm looking for. >> >> Given I'm moving away from Amazon S3, I'm really just looking for a one-way >> replication between the buckets (i.e. Provide an Amazon S3 access >> key/secret which is associated to the buckets and the same for the Ceph >> environment, so object data can be replicated one-to-one, without creating >> ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on >> modifying my boto connection objects to use the new Ceph environment. Is >> what I'm describing feasible with the cloud sync module? Just looking for >> some affirmation, given I'm not well versed in Ceph's RGW, especially >> around multi-site configurations. >> >> Thanks, >> Jimmy >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io