At Clyso we've been building a tool that can migrate S3 data around
called Chorus. Normally I wouldn't promote it here, but it's open
source and sounds like it might be useful in this case. I don't work on
it myself, but thought I'd mention it:
https://github.com/clyso/chorus
One problem with rclone is that it doesn't support copying ACLs AFAIK.
There was an incomplete PR for that a while back that never got merged.
We did some work on that as well, though I'm not sure if it's been up
streamed yet.
Mark
On 4/11/24 16:29, Casey Bodley wrote:
> unfortunately, this cloud sync module only exports data from ceph to a
> remote s3 endpoint, not the other way around:
>
> "This module syncs zone data to a remote cloud service. The sync is
> unidirectional; data is not synced back from the remote zone."
>
> i believe that rclone supports copying from one s3 endpoint to
> another. does anyone have experience with that?
>
> On Thu, Apr 11, 2024 at 4:45 PM James McClune <mcclune.789(a)gmail.com> wrote:
>>
>> Hello Ceph User Community,
>>
>> I currently have a large Amazon S3 environment with terabytes of data
>> spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an
>> on-site Ceph cluster using the RGW. I'm trying to figure out the
>> most efficient way to achieve this. Looking through the documentation, I
>> found articles related to the cloud sync module, released in Mimic (
>>
https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched
>> a video on the cloud sync module as well. It *sounds* like this is the
>> functionality I'm looking for.
>>
>> Given I'm moving away from Amazon S3, I'm really just looking for a
one-way
>> replication between the buckets (i.e. Provide an Amazon S3 access
>> key/secret which is associated to the buckets and the same for the Ceph
>> environment, so object data can be replicated one-to-one, without creating
>> ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on
>> modifying my boto connection objects to use the new Ceph environment. Is
>> what I'm describing feasible with the cloud sync module? Just looking for
>> some affirmation, given I'm not well versed in Ceph's RGW, especially
>> around multi-site configurations.
>>
>> Thanks,
>> Jimmy
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io