Migrating from S3 to Ceph RGW (Cloud Sync Module)
Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy
unfortunately, this cloud sync module only exports data from ceph to a remote s3 endpoint, not the other way around: "This module syncs zone data to a remote cloud service. The sync is unidirectional; data is not synced back from the remote zone." i believe that rclone supports copying from one s3 endpoint to another. does anyone have experience with that? On Thu, Apr 11, 2024 at 4:45 PM James McClune <mcclune.789@gmail.com> wrote:
Hello Ceph User Community,
I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for.
Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations.
Thanks, Jimmy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le jeudi 11 avril 2024, 23:29:30 CEST Casey Bodley a écrit :
unfortunately, this cloud sync module only exports data from ceph to a remote s3 endpoint, not the other way around:
"This module syncs zone data to a remote cloud service. The sync is unidirectional; data is not synced back from the remote zone."
i believe that rclone supports copying from one s3 endpoint to another. does anyone have experience with that?
Hello, Yes, rclone can do that, and seems too have good performances (I didn't try something else). You can (should) parallelize (--transfers), use many options, like rsync... use checksums, launch several time (sync not just copy). Of course, the machine where you launch the copy/sync, should be well connected (good bandwidth) with either the source and the destination. You can also launch several rclone, for specific buckets, from different machines, in parallel...
Le jeudi 11 avril 2024, 23:44:05 CEST Gilles Mocellin a écrit :
Le jeudi 11 avril 2024, 23:29:30 CEST Casey Bodley a écrit :
unfortunately, this cloud sync module only exports data from ceph to a remote s3 endpoint, not the other way around:
"This module syncs zone data to a remote cloud service. The sync is unidirectional; data is not synced back from the remote zone."
i believe that rclone supports copying from one s3 endpoint to another. does anyone have experience with that?
Hello,
Yes, rclone can do that, and seems too have good performances (I didn't try
something else).
You can (should) parallelize (--transfers), use many options, like rsync...
use checksums, launch several time (sync not just copy).
Of course, the machine where you launch the copy/sync, should be well connected (good bandwidth) with either the source and the destination.
You can also launch several rclone, for specific buckets, from different machines, in parallel...
Ah, and of course, if you use versioning, lifecycle rules, bucket policies, or all other "metadata" other than objects in buckets, rclone will NOT copy them. You should re-create (or something like export / import) with awscli or s3cmd.
Thanks cbodley for the clarification. I'll definitely look more into rclone Gilles. I was actually putting together a POC with that too, in case my understanding of the cloud sync module was wrong. Thanks for the heads-up on the other stuff too :) Best Regards, Jimmy On April 11, 2024 5:29:30 PM EDT, Casey Bodley <cbodley@redhat.com> wrote:
unfortunately, this cloud sync module only exports data from ceph to a remote s3 endpoint, not the other way around:
"This module syncs zone data to a remote cloud service. The sync is unidirectional; data is not synced back from the remote zone."
i believe that rclone supports copying from one s3 endpoint to another. does anyone have experience with that?
On Thu, Apr 11, 2024 at 4:45 PM James McClune <mcclune.789@gmail.com> wrote:
Hello Ceph User Community,
I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for.
Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations.
Thanks, Jimmy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, I used to use rclone for data synchronization between 2 ceph clusters and for a directional sync from AWS to Ceph. In general, rclone is a really good and reliable pice of software, but could be slow with large amount of syncing objects. Large - 10^6+ objects. As a disclaimer - my experience is 3 years old. Very likely rclone was improved since that time, and it definitely should be considered at least as POC. You can offload rclone's sync operations by skipping hashes and other costly comparisons if this is an appropriate approach in your project. Good luck! Sincerely, Vladimir. Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Thursday, April 11, 2024 5:29:30 PM To: James McClune <mcclune.789@gmail.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Migrating from S3 to Ceph RGW (Cloud Sync Module) unfortunately, this cloud sync module only exports data from ceph to a remote s3 endpoint, not the other way around: "This module syncs zone data to a remote cloud service. The sync is unidirectional; data is not synced back from the remote zone." i believe that rclone supports copying from one s3 endpoint to another. does anyone have experience with that? On Thu, Apr 11, 2024 at 4:45 PM James McClune <mcclune.789@gmail.com> wrote:
Hello Ceph User Community,
I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for.
Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations.
Thanks, Jimmy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, If you have a quite large amount of data you can maybe try the Chorus from CLYSO. https://docs.clyso.com/blog/2024/01/24/opensourcing-chorus-project Opensourcing Chorus project | Clyso GmbH docs.clyso.com https://github.com/clyso/chorus clyso/chorus: s3 multi provider data lifecycle management github.com It's using rclone for the copying, but it has some tricks that can be handy with large amounts of data. Ondrej
On 12. 4. 2024, at 4:00, Vladimir Sigunov <vladimir.sigunov@gmail.com> wrote:
Hello, I used to use rclone for data synchronization between 2 ceph clusters and for a directional sync from AWS to Ceph. In general, rclone is a really good and reliable pice of software, but could be slow with large amount of syncing objects. Large - 10^6+ objects. As a disclaimer - my experience is 3 years old. Very likely rclone was improved since that time, and it definitely should be considered at least as POC. You can offload rclone's sync operations by skipping hashes and other costly comparisons if this is an appropriate approach in your project. Good luck! Sincerely, Vladimir.
Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Thursday, April 11, 2024 5:29:30 PM To: James McClune <mcclune.789@gmail.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Migrating from S3 to Ceph RGW (Cloud Sync Module)
unfortunately, this cloud sync module only exports data from ceph to a remote s3 endpoint, not the other way around:
"This module syncs zone data to a remote cloud service. The sync is unidirectional; data is not synced back from the remote zone."
i believe that rclone supports copying from one s3 endpoint to another. does anyone have experience with that?
On Thu, Apr 11, 2024 at 4:45 PM James McClune <mcclune.789@gmail.com> wrote:
Hello Ceph User Community,
I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for.
Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations.
Thanks, Jimmy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 15/4/24 19:58, Ondřej Kukla wrote:
If you have a quite large amount of data you can maybe try the Chorus from CLYSO.
In March we completed a migration of 17PB of data between two local Ceph clusters using Chorus. It took some work to prepare network configurations and test it and increase our comfort with it to the point of hitting go on the actual migration, but the data migration part was effectively flawless.
At Clyso we've been building a tool that can migrate S3 data around called Chorus. Normally I wouldn't promote it here, but it's open source and sounds like it might be useful in this case. I don't work on it myself, but thought I'd mention it: https://github.com/clyso/chorus One problem with rclone is that it doesn't support copying ACLs AFAIK. There was an incomplete PR for that a while back that never got merged. We did some work on that as well, though I'm not sure if it's been up streamed yet. Mark On 4/11/24 16:29, Casey Bodley wrote:
unfortunately, this cloud sync module only exports data from ceph to a remote s3 endpoint, not the other way around:
"This module syncs zone data to a remote cloud service. The sync is unidirectional; data is not synced back from the remote zone."
i believe that rclone supports copying from one s3 endpoint to another. does anyone have experience with that?
On Thu, Apr 11, 2024 at 4:45 PM James McClune <mcclune.789@gmail.com> wrote:
Hello Ceph User Community,
I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for.
Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations.
Thanks, Jimmy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Casey Bodley
-
Gilles Mocellin
-
Gregory Orange
-
James McClune
-
Mark Nelson
-
Ondřej Kukla
-
Vladimir Sigunov