A pleasure. Hope it helps :) Happy to share if you need any more information Zac. Cheers, Tom On Wed, 26 Apr 2023 at 18:14, Dan van der Ster <dan.vanderster@clyso.com> wrote:
Thanks Tom, this is a very useful post! I've added our docs guy Zac in cc: IMHO this would be useful in a "Tips & Tricks" section of the docs.
-- dan
______________________________ Clyso GmbH | https://www.clyso.com
On Wed, Apr 26, 2023 at 7:46 AM Thomas Bennett <thomas@tsolo.io> wrote:
I would second Joachim's suggestion - this is exactly what we're in the process of doing for a client, i.e migrating from Luminous to Quincy. However below would also work if you're moving to Nautilus.
The only catch with this plan would be if you plan to reuse any hardware
i.e the hosts running rados gateways and mons, etc. If you have enough hardware to spare this is a good plan.
My process:
1. Stand a new Quincy cluster and tune the cluster. 2. Migrate user information, secrets and access keys (using radosg-admin in a script). 3. Using a combination of rclone and parallel to push data across from the old cluster to the new cluster.
Below is a bash script I used to capture all the user information on the old cluster and I ran it on the new cluster to create users and keep
secrets and keys the same.
# for i in $(radosgw-admin user list | jq -r .[]); do USER_INFO=$(radosgw-admin user info --uid=$i) USER_ID=$(echo $USER_INFO | jq -r '.user_id') DISPLAY_NAME=$(echo $USER_INFO | jq '.display_name') EMAIL=$(echo $USER_INFO | jq '.email') MAX_BUCKETS=$(echo $USER_INFO | jq -r '(.max_buckets|tostring)') ACCESS=$(echo $USER_INFO | jq -r '.keys[].access_key') SECRET=$(echo $USER_INFO | jq -r '.keys[].secret_key') echo "radosgw-admin user create --uid=$USER_ID --display-name=$DISPLAY_NAME --email=$EMAIL --max-buckets=$MAX_BUCKETS --access-key=$ACCESS --secret-key=$SECRET" | tee -a generated.radosgw-admin-user-create.sh done #
Rclone is a really powerful tool! I lazily set up a backends for each user, by appending below to the for loop in the above script. Below script is not pretty but it does the job: # echo "" >> generated.rclone.conf echo [old-cluster-$USER_ID] >> generated.rclone.conf echo type = s3 >> generated.rclone.conf echo provider = Ceph >> generated.rclone.conf echo env_auth = false >> generated.rclone.conf echo access_key_id = $ACCESS >> generated.rclone.conf echo secret_access_key = $SECRET >> generated.rclone.conf echo endpoint = http://xx.xx.xx.xx:xxxx >> generated.rclone.conf echo acl = public-read >> generated.rclone.conf echo "" >> generated.rclone.conf echo [new-cluster-$USER_ID] >> generated.rclone.conf echo type = s3 >> generated.rclone.conf echo provider = Ceph >> generated.rclone.conf echo env_auth = false >> generated.rclone.conf echo access_key_id = $ACCESS >> generated.rclone.conf echo secret_access_key = $SECRET >> generated.rclone.conf echo endpoint = http://yy.yy.yy.yy:yyyy >> generated.rclone.conf echo acl = public-read >> generated.rclone.conf echo "" >> generated.rclone.conf #
Copy the generated.rclone.conf to the node that is going to act as the transfer node (I just used the new rados gateway node) into ~/.config/rclone/rclone.conf
Now if you run rclone lsd old-cluser-{user}: (it even tab completes!) you'll get a list of all the buckets for that user.
You could even simply rclone sync old-cluser-{user}: new-cluser-{user}: and it should sync all buckets for a user.
Catches:
- Use the scripts carefully - our buckets for this one user are set public-read - you might want to check each line of the script if you use it. - Quincy bucket naming convention is stricter than Luminous. I've had to catch some '_' and upper cases and fix them in the command line I generate for copying each bucket. - Using rclone will take a long time.Feed a script into parallel sped things up for me: - # parallel -j 10 < sync-script - Watch out for lifecycling! Not sure how to handle this to make sure it's captured correctly.
Cheers, Tom
On Tue, 25 Apr 2023 at 22:36, Marc <Marc@f1-outsourcing.eu> wrote:
Maybe he is limited by the supported OS
I would create a new cluster with Quincy and would migrate the data
from
the old to the new cluster bucket by bucket. Nautilus is out of support and I would recommend at least to use a ceph version that is receiving Backports.
huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> schrieb am Di., 25. Apr. 2023, 18:30:
Dear Ceph folks,
I would like to listen to your advice on the following topic: We have a 6-node Ceph cluster (for RGW usage only ) running on Luminous 12.2.12, and now will add 10 new nodes. Our plan is to phase out the old 6 nodes, and run RGW Ceph cluster with the new 10 nodes on Nautilus version。
I can think of two ways to achieve the above goal. The first method would be: 1) Upgrade the current 6-node cluster from Luminous 12.2.12 to Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and then re-balance; 3) After rebalance completes, remove the 6 old nodes from the cluster
The second method would get rid of the procedure to upgrade the
- their old 6-
node
from Luminous to Nautilus, because those 6 nodes will be phased out anyway, but then we have to deal with a hybrid cluster with 6-node on Luminous 12.2.12, and 10-node on Nautilus, and after re-balancing, we can remove the 6 old nodes from the cluster.
Any suggestions, advice, or best practice would be highly appreciated.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io