Thanks Tom, this is a very useful post! I've added our docs guy Zac in cc: IMHO this would be useful in a "Tips & Tricks" section of the docs. -- dan ______________________________ Clyso GmbH | https://www.clyso.com On Wed, Apr 26, 2023 at 7:46 AM Thomas Bennett <thomas@tsolo.io> wrote:
I would second Joachim's suggestion - this is exactly what we're in the process of doing for a client, i.e migrating from Luminous to Quincy. However below would also work if you're moving to Nautilus.
The only catch with this plan would be if you plan to reuse any hardware - i.e the hosts running rados gateways and mons, etc. If you have enough hardware to spare this is a good plan.
My process:
1. Stand a new Quincy cluster and tune the cluster. 2. Migrate user information, secrets and access keys (using radosg-admin in a script). 3. Using a combination of rclone and parallel to push data across from the old cluster to the new cluster.
Below is a bash script I used to capture all the user information on the old cluster and I ran it on the new cluster to create users and keep their secrets and keys the same.
# for i in $(radosgw-admin user list | jq -r .[]); do USER_INFO=$(radosgw-admin user info --uid=$i) USER_ID=$(echo $USER_INFO | jq -r '.user_id') DISPLAY_NAME=$(echo $USER_INFO | jq '.display_name') EMAIL=$(echo $USER_INFO | jq '.email') MAX_BUCKETS=$(echo $USER_INFO | jq -r '(.max_buckets|tostring)') ACCESS=$(echo $USER_INFO | jq -r '.keys[].access_key') SECRET=$(echo $USER_INFO | jq -r '.keys[].secret_key') echo "radosgw-admin user create --uid=$USER_ID --display-name=$DISPLAY_NAME --email=$EMAIL --max-buckets=$MAX_BUCKETS --access-key=$ACCESS --secret-key=$SECRET" | tee -a generated.radosgw-admin-user-create.sh done #
Rclone is a really powerful tool! I lazily set up a backends for each user, by appending below to the for loop in the above script. Below script is not pretty but it does the job: # echo "" >> generated.rclone.conf echo [old-cluster-$USER_ID] >> generated.rclone.conf echo type = s3 >> generated.rclone.conf echo provider = Ceph >> generated.rclone.conf echo env_auth = false >> generated.rclone.conf echo access_key_id = $ACCESS >> generated.rclone.conf echo secret_access_key = $SECRET >> generated.rclone.conf echo endpoint = http://xx.xx.xx.xx:xxxx >> generated.rclone.conf echo acl = public-read >> generated.rclone.conf echo "" >> generated.rclone.conf echo [new-cluster-$USER_ID] >> generated.rclone.conf echo type = s3 >> generated.rclone.conf echo provider = Ceph >> generated.rclone.conf echo env_auth = false >> generated.rclone.conf echo access_key_id = $ACCESS >> generated.rclone.conf echo secret_access_key = $SECRET >> generated.rclone.conf echo endpoint = http://yy.yy.yy.yy:yyyy >> generated.rclone.conf echo acl = public-read >> generated.rclone.conf echo "" >> generated.rclone.conf #
Copy the generated.rclone.conf to the node that is going to act as the transfer node (I just used the new rados gateway node) into ~/.config/rclone/rclone.conf
Now if you run rclone lsd old-cluser-{user}: (it even tab completes!) you'll get a list of all the buckets for that user.
You could even simply rclone sync old-cluser-{user}: new-cluser-{user}: and it should sync all buckets for a user.
Catches:
- Use the scripts carefully - our buckets for this one user are set public-read - you might want to check each line of the script if you use it. - Quincy bucket naming convention is stricter than Luminous. I've had to catch some '_' and upper cases and fix them in the command line I generate for copying each bucket. - Using rclone will take a long time.Feed a script into parallel sped things up for me: - # parallel -j 10 < sync-script - Watch out for lifecycling! Not sure how to handle this to make sure it's captured correctly.
Cheers, Tom
On Tue, 25 Apr 2023 at 22:36, Marc <Marc@f1-outsourcing.eu> wrote:
Maybe he is limited by the supported OS
I would create a new cluster with Quincy and would migrate the data from the old to the new cluster bucket by bucket. Nautilus is out of support and I would recommend at least to use a ceph version that is receiving Backports.
huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> schrieb am Di., 25. Apr. 2023, 18:30:
Dear Ceph folks,
I would like to listen to your advice on the following topic: We have a 6-node Ceph cluster (for RGW usage only ) running on Luminous 12.2.12, and now will add 10 new nodes. Our plan is to phase out the old 6 nodes, and run RGW Ceph cluster with the new 10 nodes on Nautilus version。
I can think of two ways to achieve the above goal. The first method would be: 1) Upgrade the current 6-node cluster from Luminous 12.2.12 to Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and then re-balance; 3) After rebalance completes, remove the 6 old nodes from the cluster
The second method would get rid of the procedure to upgrade the old 6- node from Luminous to Nautilus, because those 6 nodes will be phased out anyway, but then we have to deal with a hybrid cluster with 6-node on Luminous 12.2.12, and 10-node on Nautilus, and after re-balancing, we can remove the 6 old nodes from the cluster.
Any suggestions, advice, or best practice would be highly appreciated.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io