25 Apr
2023
25 Apr
'23
7:30 p.m.
Dear Ceph folks,
I would like to listen to your advice on the following topic: We have a 6-node Ceph
cluster (for RGW usage only ) running on Luminous 12.2.12, and now will add 10 new nodes.
Our plan is to phase out the old 6 nodes, and run RGW Ceph cluster with the new 10 nodes
on Nautilus version。
I can think of two ways to achieve the above goal. The first method would be: 1) Upgrade
the current 6-node cluster from Luminous 12.2.12 to Nautilus 14.2.22; 2) Expand the
cluster with the 10 new nodes, and then re-balance; 3) After rebalance completes, remove
the 6 old nodes from the cluster
The second method would get rid of the procedure to upgrade the old 6-node from Luminous
to Nautilus, because those 6 nodes will be phased out anyway, but then we have to deal
with a hybrid cluster with 6-node on Luminous 12.2.12, and 10-node on Nautilus, and after
re-balancing, we can remove the 6 old nodes from the cluster.
Any suggestions, advice, or best practice would be highly appreciated.
best regards,
Samuel
huxiaoyu(a)horebdata.cn
25 Apr
25 Apr
7:44 p.m.
Hi Samuel,
While the second method would probably work fine in the happy path, if
something goes wrong I think you'll be happier having a uniform
release installed. In general, we've found the backfill experience to
be better on Nautilus than Luminous, so my vote would be for the first
method. Given that your usage is RGW, just note that the OMAP format
change that happens between Luminous and Nautilus can sometimes take a
while.
Josh
On Tue, Apr 25, 2023 at 10:31 AM huxiaoyu(a)horebdata.cn
<huxiaoyu(a)horebdata.cn> wrote:
>
> Dear Ceph folks,
>
> I would like to listen to your advice on the following topic: We have a 6-node Ceph
cluster (for RGW usage only ) running on Luminous 12.2.12, and now will add 10 new nodes.
Our plan is to phase out the old 6 nodes, and run RGW Ceph cluster with the new 10 nodes
on Nautilus version。
>
> I can think of two ways to achieve the above goal. The first method would be: 1)
Upgrade the current 6-node cluster from Luminous 12.2.12 to Nautilus 14.2.22; 2) Expand
the cluster with the 10 new nodes, and then re-balance; 3) After rebalance completes,
remove the 6 old nodes from the cluster
>
> The second method would get rid of the procedure to upgrade the old 6-node from
Luminous to Nautilus, because those 6 nodes will be phased out anyway, but then we have to
deal with a hybrid cluster with 6-node on Luminous 12.2.12, and 10-node on Nautilus, and
after re-balancing, we can remove the 6 old nodes from the cluster.
>
> Any suggestions, advice, or best practice would be highly appreciated.
>
> best regards,
>
>
> Samuel
>
>
>
> huxiaoyu(a)horebdata.cn
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:55 p.m.
Get on nautilus first and (perhaps even go to pacific) before expansion.
Primarily for the reason that starting in nautilus degraded data recovery
will be prioritized over remapped data recovery. As you phase out old
hardware and phase in new hardware you will have a very large amount of
backfill happening and if you get into a degraded state in the middle of
this backfill it will take a much longer time for the degraded data to
become clean again.
Additionally, you will want to follow the best practice of updating your
cluster in order. In short monitors then managers then osds then MDS and
RGW then other clients. More details here:
https://docs.ceph.com/en/latest/releases/nautilus/#upgrading-from-mimic-or-…
You dont want to run with a mixed software version cluster longer than a
well coordinated upgrade takes.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Apr 25, 2023 at 12:31 PM huxiaoyu(a)horebdata.cn <
huxiaoyu(a)horebdata.cn> wrote:
Dear Ceph folks,
I would like to listen to your advice on the following topic: We have a
6-node Ceph cluster (for RGW usage only ) running on Luminous 12.2.12, and
now will add 10 new nodes. Our plan is to phase out the old 6 nodes, and
run RGW Ceph cluster with the new 10 nodes on Nautilus version。
I can think of two ways to achieve the above goal. The first method would
be: 1) Upgrade the current 6-node cluster from Luminous 12.2.12 to
Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and then
re-balance; 3) After rebalance completes, remove the 6 old nodes from the
cluster
The second method would get rid of the procedure to upgrade the old 6-node
from Luminous to Nautilus, because those 6 nodes will be phased out anyway,
but then we have to deal with a hybrid cluster with 6-node on Luminous
12.2.12, and 10-node on Nautilus, and after re-balancing, we can remove the
6 old nodes from the cluster.
Any suggestions, advice, or best practice would be highly appreciated.
best regards,
Samuel
huxiaoyu(a)horebdata.cn
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:42 p.m.
Thanks a lot for the valuable input, Wesley, Josh, and Anthony.
It seems the best practice would be upgrade first, and then expand, remove old nodes
afterwards.
best regards,
Samuel
huxiaoyu(a)horebdata.cn
From: Wesley Dillingham
Date: 2023-04-25 19:55
To: huxiaoyu(a)horebdata.cn
CC: ceph-users
Subject: Re: [ceph-users] For suggestions and best practices on expanding Ceph cluster and
removing old nodes
Get on nautilus first and (perhaps even go to pacific) before expansion. Primarily for the
reason that starting in nautilus degraded data recovery will be prioritized over remapped
data recovery. As you phase out old hardware and phase in new hardware you will have a
very large amount of backfill happening and if you get into a degraded state in the middle
of this backfill it will take a much longer time for the degraded data to become clean
again.
Additionally, you will want to follow the best practice of updating your cluster in order.
In short monitors then managers then osds then MDS and RGW then other clients. More
details here:
https://docs.ceph.com/en/latest/releases/nautilus/#upgrading-from-mimic-or-…
You dont want to run with a mixed software version cluster longer than a well coordinated
upgrade takes.
Respectfully,
Wes Dillingham
wes(a)wesdillingham.com
LinkedIn
On Tue, Apr 25, 2023 at 12:31 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn>
wrote:
Dear Ceph folks,
I would like to listen to your advice on the following topic: We have a 6-node Ceph
cluster (for RGW usage only ) running on Luminous 12.2.12, and now will add 10 new nodes.
Our plan is to phase out the old 6 nodes, and run RGW Ceph cluster with the new 10 nodes
on Nautilus version。
I can think of two ways to achieve the above goal. The first method would be: 1) Upgrade
the current 6-node cluster from Luminous 12.2.12 to Nautilus 14.2.22; 2) Expand the
cluster with the 10 new nodes, and then re-balance; 3) After rebalance completes, remove
the 6 old nodes from the cluster
The second method would get rid of the procedure to upgrade the old 6-node from Luminous
to Nautilus, because those 6 nodes will be phased out anyway, but then we have to deal
with a hybrid cluster with 6-node on Luminous 12.2.12, and 10-node on Nautilus, and after
re-balancing, we can remove the 6 old nodes from the cluster.
Any suggestions, advice, or best practice would be highly appreciated.
best regards,
Samuel
huxiaoyu(a)horebdata.cn
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:47 p.m.
I would create a new cluster with Quincy and would migrate the data from
the old to the new cluster bucket by bucket. Nautilus is out of support and
I would recommend at least to use a ceph version that is receiving
Backports.
huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> schrieb am Di., 25. Apr.
2023, 18:30:
Dear Ceph folks,
I would like to listen to your advice on the following topic: We have a
6-node Ceph cluster (for RGW usage only ) running on Luminous 12.2.12, and
now will add 10 new nodes. Our plan is to phase out the old 6 nodes, and
run RGW Ceph cluster with the new 10 nodes on Nautilus version。
I can think of two ways to achieve the above goal. The first method would
be: 1) Upgrade the current 6-node cluster from Luminous 12.2.12 to
Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and then
re-balance; 3) After rebalance completes, remove the 6 old nodes from the
cluster
The second method would get rid of the procedure to upgrade the old 6-node
from Luminous to Nautilus, because those 6 nodes will be phased out anyway,
but then we have to deal with a hybrid cluster with 6-node on Luminous
12.2.12, and 10-node on Nautilus, and after re-balancing, we can remove the
6 old nodes from the cluster.
Any suggestions, advice, or best practice would be highly appreciated.
best regards,
Samuel
huxiaoyu(a)horebdata.cn
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:34 p.m.
Maybe he is limited by the supported OS
I would create a new cluster with Quincy and would migrate the data from
the old to the new cluster bucket by bucket. Nautilus is out of support
and
I would recommend at least to use a ceph version that is receiving
Backports.
huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> schrieb am Di., 25. Apr.
2023, 18:30:
Dear Ceph folks,
I would like to listen to your advice on the following topic: We have
a
6-node Ceph cluster (for RGW usage only ) running
on Luminous 12.2.12,
and
now will add 10 new nodes. Our plan is to phase
out the old 6 nodes,
and
run RGW Ceph cluster with the new 10 nodes on
Nautilus version。
I can think of two ways to achieve the above goal. The first method
would
be: 1) Upgrade the current 6-node cluster from
Luminous 12.2.12 to
Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and
then
re-balance; 3) After rebalance completes, remove
the 6 old nodes from
the
cluster
The second method would get rid of the procedure to upgrade the old 6-
node
from Luminous to Nautilus, because those 6 nodes
will be phased out
anyway,
but then we have to deal with a hybrid cluster
with 6-node on Luminous
12.2.12, and 10-node on Nautilus, and after re-balancing, we can
remove the
> 6 old nodes from the cluster.
>
> Any suggestions, advice, or best practice would be highly appreciated.
>
26 Apr
26 Apr
5:46 p.m.
I would second Joachim's suggestion - this is exactly what we're in the
process of doing for a client, i.e migrating from Luminous to Quincy.
However below would also work if you're moving to Nautilus.
The only catch with this plan would be if you plan to reuse any hardware -
i.e the hosts running rados gateways and mons, etc. If you have enough
hardware to spare this is a good plan.
My process:
1. Stand a new Quincy cluster and tune the cluster.
2. Migrate user information, secrets and access keys (using
radosg-admin in a script).
3. Using a combination of rclone and parallel to push data across from
the old cluster to the new cluster.
Below is a bash script I used to capture all the user information on the
old cluster and I ran it on the new cluster to create users and keep their
secrets and keys the same.
#
for i in $(radosgw-admin user list | jq -r .[]); do
USER_INFO=$(radosgw-admin user info --uid=$i)
USER_ID=$(echo $USER_INFO | jq -r '.user_id')
DISPLAY_NAME=$(echo $USER_INFO | jq '.display_name')
EMAIL=$(echo $USER_INFO | jq '.email')
MAX_BUCKETS=$(echo $USER_INFO | jq -r '(.max_buckets|tostring)')
ACCESS=$(echo $USER_INFO | jq -r '.keys[].access_key')
SECRET=$(echo $USER_INFO | jq -r '.keys[].secret_key')
echo "radosgw-admin user create --uid=$USER_ID
--display-name=$DISPLAY_NAME --email=$EMAIL --max-buckets=$MAX_BUCKETS
--access-key=$ACCESS --secret-key=$SECRET" | tee -a
generated.radosgw-admin-user-create.sh
done
#
Rclone is a really powerful tool! I lazily set up a backends for each user,
by appending below to the for loop in the above script. Below script is not
pretty but it does the job:
#
echo "" >> generated.rclone.conf
echo [old-cluster-$USER_ID] >> generated.rclone.conf
echo type = s3 >> generated.rclone.conf
echo provider = Ceph >> generated.rclone.conf
echo env_auth = false >> generated.rclone.conf
echo access_key_id = $ACCESS >> generated.rclone.conf
echo secret_access_key = $SECRET >> generated.rclone.conf
echo endpoint = http://xx.xx.xx.xx:xxxx >> generated.rclone.conf
echo acl = public-read >> generated.rclone.conf
echo "" >> generated.rclone.conf
echo [new-cluster-$USER_ID] >> generated.rclone.conf
echo type = s3 >> generated.rclone.conf
echo provider = Ceph >> generated.rclone.conf
echo env_auth = false >> generated.rclone.conf
echo access_key_id = $ACCESS >> generated.rclone.conf
echo secret_access_key = $SECRET >> generated.rclone.conf
echo endpoint = http://yy.yy.yy.yy:yyyy >> generated.rclone.conf
echo acl = public-read >> generated.rclone.conf
echo "" >> generated.rclone.conf
#
Copy the generated.rclone.conf to the node that is going to act as the
transfer node (I just used the new rados gateway node) into
~/.config/rclone/rclone.conf
Now if you run rclone lsd old-cluser-{user}: (it even tab completes!)
you'll get a list of all the buckets for that user.
You could even simply rclone sync old-cluser-{user}: new-cluser-{user}: and
it should sync all buckets for a user.
Catches:
- Use the scripts carefully - our buckets for this one user are set
public-read - you might want to check each line of the script if you use it.
- Quincy bucket naming convention is stricter than Luminous. I've had to
catch some '_' and upper cases and fix them in the command line I generate
for copying each bucket.
- Using rclone will take a long time.Feed a script into parallel sped
things up for me:
- # parallel -j 10 < sync-script
- Watch out for lifecycling! Not sure how to handle this to make sure
it's captured correctly.
Cheers,
Tom
On Tue, 25 Apr 2023 at 22:36, Marc <Marc(a)f1-outsourcing.eu> wrote:
Maybe he is limited by the supported OS
I would create a new cluster with Quincy and would migrate the data from
the old to the new cluster bucket by bucket. Nautilus is out of support
and
I would recommend at least to use a ceph version that is receiving
Backports.
huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> schrieb am Di., 25. Apr.
2023, 18:30:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Dear Ceph folks,
I would like to listen to your advice on the following topic: We have
a
6-node Ceph cluster (for RGW usage only ) running
on Luminous 12.2.12,
and
now will add 10 new nodes. Our plan is to phase
out the old 6 nodes,
and
run RGW Ceph cluster with the new 10 nodes on
Nautilus version。
I can think of two ways to achieve the above goal. The first method
would
be: 1) Upgrade the current 6-node cluster from
Luminous 12.2.12 to
Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and
then
re-balance; 3) After rebalance completes, remove
the 6 old nodes from
the
cluster
The second method would get rid of the procedure to upgrade the old 6-
node
from Luminous to Nautilus, because those 6 nodes
will be phased out
anyway,
but then we have to deal with a hybrid cluster
with 6-node on Luminous
12.2.12, and 10-node on Nautilus, and after re-balancing, we can
remove the
> 6 old nodes from the cluster.
>
> Any suggestions, advice, or best practice would be highly appreciated.
> 
7:13 p.m.
Thanks Tom, this is a very useful post!
I've added our docs guy Zac in cc: IMHO this would be useful in a
"Tips & Tricks" section of the docs.
-- dan
______________________________
Clyso GmbH | https://www.clyso.com
On Wed, Apr 26, 2023 at 7:46 AM Thomas Bennett <thomas(a)tsolo.io> wrote:
>
> I would second Joachim's suggestion - this is exactly what we're in the
> process of doing for a client, i.e migrating from Luminous to Quincy.
> However below would also work if you're moving to Nautilus.
>
> The only catch with this plan would be if you plan to reuse any hardware -
> i.e the hosts running rados gateways and mons, etc. If you have enough
> hardware to spare this is a good plan.
>
> My process:
>
> 1. Stand a new Quincy cluster and tune the cluster.
> 2. Migrate user information, secrets and access keys (using
> radosg-admin in a script).
> 3. Using a combination of rclone and parallel to push data across from
> the old cluster to the new cluster.
>
>
> Below is a bash script I used to capture all the user information on the
> old cluster and I ran it on the new cluster to create users and keep their
> secrets and keys the same.
>
> #
> for i in $(radosgw-admin user list | jq -r .[]); do
> USER_INFO=$(radosgw-admin user info --uid=$i)
> USER_ID=$(echo $USER_INFO | jq -r '.user_id')
> DISPLAY_NAME=$(echo $USER_INFO | jq '.display_name')
> EMAIL=$(echo $USER_INFO | jq '.email')
> MAX_BUCKETS=$(echo $USER_INFO | jq -r '(.max_buckets|tostring)')
> ACCESS=$(echo $USER_INFO | jq -r '.keys[].access_key')
> SECRET=$(echo $USER_INFO | jq -r '.keys[].secret_key')
> echo "radosgw-admin user create --uid=$USER_ID
> --display-name=$DISPLAY_NAME --email=$EMAIL --max-buckets=$MAX_BUCKETS
> --access-key=$ACCESS --secret-key=$SECRET" | tee -a
> generated.radosgw-admin-user-create.sh
> done
> #
>
> Rclone is a really powerful tool! I lazily set up a backends for each user,
> by appending below to the for loop in the above script. Below script is not
> pretty but it does the job:
> #
> echo "" >> generated.rclone.conf
> echo [old-cluster-$USER_ID] >> generated.rclone.conf
> echo type = s3 >> generated.rclone.conf
> echo provider = Ceph >> generated.rclone.conf
> echo env_auth = false >> generated.rclone.conf
> echo access_key_id = $ACCESS >> generated.rclone.conf
> echo secret_access_key = $SECRET >> generated.rclone.conf
> echo endpoint = http://xx.xx.xx.xx:xxxx >> generated.rclone.conf
> echo acl = public-read >> generated.rclone.conf
> echo "" >> generated.rclone.conf
> echo [new-cluster-$USER_ID] >> generated.rclone.conf
> echo type = s3 >> generated.rclone.conf
> echo provider = Ceph >> generated.rclone.conf
> echo env_auth = false >> generated.rclone.conf
> echo access_key_id = $ACCESS >> generated.rclone.conf
> echo secret_access_key = $SECRET >> generated.rclone.conf
> echo endpoint = http://yy.yy.yy.yy:yyyy >> generated.rclone.conf
> echo acl = public-read >> generated.rclone.conf
> echo "" >> generated.rclone.conf
> #
>
> Copy the generated.rclone.conf to the node that is going to act as the
> transfer node (I just used the new rados gateway node) into
> ~/.config/rclone/rclone.conf
>
> Now if you run rclone lsd old-cluser-{user}: (it even tab completes!)
> you'll get a list of all the buckets for that user.
>
> You could even simply rclone sync old-cluser-{user}: new-cluser-{user}: and
> it should sync all buckets for a user.
>
> Catches:
>
> - Use the scripts carefully - our buckets for this one user are set
> public-read - you might want to check each line of the script if you use it.
> - Quincy bucket naming convention is stricter than Luminous. I've had to
> catch some '_' and upper cases and fix them in the command line I
generate
> for copying each bucket.
> - Using rclone will take a long time.Feed a script into parallel sped
> things up for me:
> - # parallel -j 10 < sync-script
> - Watch out for lifecycling! Not sure how to handle this to make sure
> it's captured correctly.
>
> Cheers,
> Tom
>
> On Tue, 25 Apr 2023 at 22:36, Marc <Marc(a)f1-outsourcing.eu> wrote:
>
> >
> > Maybe he is limited by the supported OS
> >
> >
> > >
> > > I would create a new cluster with Quincy and would migrate the data from
> > > the old to the new cluster bucket by bucket. Nautilus is out of support
> > > and
> > > I would recommend at least to use a ceph version that is receiving
> > > Backports.
> > >
> > > huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> schrieb am Di., 25.
Apr.
> > > 2023, 18:30:
> > >
> > > > Dear Ceph folks,
> > > >
> > > > I would like to listen to your advice on the following topic: We have
> > > a
> > > > 6-node Ceph cluster (for RGW usage only ) running on Luminous
12.2.12,
> > > and
> > > > now will add 10 new nodes. Our plan is to phase out the old 6 nodes,
> > > and
> > > > run RGW Ceph cluster with the new 10 nodes on Nautilus version。
> > > >
> > > > I can think of two ways to achieve the above goal. The first method
> > > would
> > > > be: 1) Upgrade the current 6-node cluster from Luminous 12.2.12 to
> > > > Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and
> > > then
> > > > re-balance; 3) After rebalance completes, remove the 6 old nodes
from
> > > the
> > > > cluster
> > > >
> > > > The second method would get rid of the procedure to upgrade the old
6-
> > > node
> > > > from Luminous to Nautilus, because those 6 nodes will be phased out
> > > anyway,
> > > > but then we have to deal with a hybrid cluster with 6-node on
Luminous
> > > > 12.2.12, and 10-node on Nautilus, and after re-balancing, we can
> > > remove the
> > > > 6 old nodes from the cluster.
> > > >
> > > > Any suggestions, advice, or best practice would be highly
appreciated.
> > > >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
28 Apr
28 Apr
4:15 p.m.
A pleasure. Hope it helps :)
Happy to share if you need any more information Zac.
Cheers,
Tom
On Wed, 26 Apr 2023 at 18:14, Dan van der Ster <dan.vanderster(a)clyso.com>
wrote:
Thanks Tom, this is a very useful post!
I've added our docs guy Zac in cc: IMHO this would be useful in a
"Tips & Tricks" section of the docs.
-- dan
______________________________
Clyso GmbH | https://www.clyso.com
On Wed, Apr 26, 2023 at 7:46 AM Thomas Bennett <thomas(a)tsolo.io> wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I would second Joachim's suggestion - this is exactly what we're in the
process of doing for a client, i.e migrating from Luminous to Quincy.
However below would also work if you're moving to Nautilus.
The only catch with this plan would be if you plan to reuse any hardware
-
i.e the hosts running rados gateways and mons,
etc. If you have enough
hardware to spare this is a good plan.
My process:
1. Stand a new Quincy cluster and tune the cluster.
2. Migrate user information, secrets and access keys (using
radosg-admin in a script).
3. Using a combination of rclone and parallel to push data across from
the old cluster to the new cluster.
Below is a bash script I used to capture all the user information on the
old cluster and I ran it on the new cluster to create users and keep
their
secrets and keys the same.
#
for i in $(radosgw-admin user list | jq -r .[]); do
USER_INFO=$(radosgw-admin user info --uid=$i)
USER_ID=$(echo $USER_INFO | jq -r '.user_id')
DISPLAY_NAME=$(echo $USER_INFO | jq '.display_name')
EMAIL=$(echo $USER_INFO | jq '.email')
MAX_BUCKETS=$(echo $USER_INFO | jq -r '(.max_buckets|tostring)')
ACCESS=$(echo $USER_INFO | jq -r '.keys[].access_key')
SECRET=$(echo $USER_INFO | jq -r '.keys[].secret_key')
echo "radosgw-admin user create --uid=$USER_ID
--display-name=$DISPLAY_NAME --email=$EMAIL --max-buckets=$MAX_BUCKETS
--access-key=$ACCESS --secret-key=$SECRET" | tee -a
generated.radosgw-admin-user-create.sh
done
#
Rclone is a really powerful tool! I lazily set up a backends for each
user,
by appending below to the for loop in the above
script. Below script is
not
pretty but it does the job:
#
echo "" >> generated.rclone.conf
echo [old-cluster-$USER_ID] >> generated.rclone.conf
echo type = s3 >> generated.rclone.conf
echo provider = Ceph >> generated.rclone.conf
echo env_auth = false >> generated.rclone.conf
echo access_key_id = $ACCESS >> generated.rclone.conf
echo secret_access_key = $SECRET >> generated.rclone.conf
echo endpoint = http://xx.xx.xx.xx:xxxx >> generated.rclone.conf
echo acl = public-read >> generated.rclone.conf
echo "" >> generated.rclone.conf
echo [new-cluster-$USER_ID] >> generated.rclone.conf
echo type = s3 >> generated.rclone.conf
echo provider = Ceph >> generated.rclone.conf
echo env_auth = false >> generated.rclone.conf
echo access_key_id = $ACCESS >> generated.rclone.conf
echo secret_access_key = $SECRET >> generated.rclone.conf
echo endpoint = http://yy.yy.yy.yy:yyyy >> generated.rclone.conf
echo acl = public-read >> generated.rclone.conf
echo "" >> generated.rclone.conf
#
Copy the generated.rclone.conf to the node that is going to act as the
transfer node (I just used the new rados gateway node) into
~/.config/rclone/rclone.conf
Now if you run rclone lsd old-cluser-{user}: (it even tab completes!)
you'll get a list of all the buckets for that user.
You could even simply rclone sync old-cluser-{user}: new-cluser-{user}:
and
it should sync all buckets for a user.
Catches:
- Use the scripts carefully - our buckets for this one user are set
public-read - you might want to check each line of the script if you
use it.
- Quincy bucket naming convention is stricter
than Luminous. I've had
to
catch some '_' and upper cases and fix
them in the command line I
generate
for copying each bucket.
- Using rclone will take a long time.Feed a script into parallel sped
things up for me:
- # parallel -j 10 < sync-script
- Watch out for lifecycling! Not sure how to handle this to make sure
it's captured correctly.
Cheers,
Tom
On Tue, 25 Apr 2023 at 22:36, Marc <Marc(a)f1-outsourcing.eu> wrote:
>
> Maybe he is limited by the supported OS
>
>
> >
> > I would create a new cluster with Quincy and would migrate the data
from
> > the old to the new cluster bucket by
bucket. Nautilus is out of
support
> > and
> > I would recommend at least to use a ceph version that is receiving
> > Backports.
> >
> > huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> schrieb am Di., 25.
Apr.
> > 2023, 18:30:
> >
> > > Dear Ceph folks,
> > >
> > > I would like to listen to your advice on the following topic: We
have
> > a
> > > 6-node Ceph cluster (for RGW usage only ) running on Luminous
12.2.12,
> > and
> > > now will add 10 new nodes. Our plan is to phase out the old 6
nodes,
> > and
> > > run RGW Ceph cluster with the new 10 nodes on Nautilus version。
> > >
> > > I can think of two ways to achieve the above goal. The first method
> > would
> > > be: 1) Upgrade the current 6-node cluster from Luminous 12.2.12
to
> > > Nautilus 14.2.22; 2) Expand the
cluster with the 10 new nodes, and
> > then
> > > re-balance; 3) After rebalance completes, remove the 6 old nodes
from
> > the
> > > cluster
> > >
> > > The second method would get rid of the procedure to upgrade the
old
6-
> > node
> > > from Luminous to Nautilus, because those 6 nodes will be phased out
> > anyway,
> > > but then we have to deal with a hybrid cluster with 6-node on
Luminous
> > > 12.2.12, and 10-node on Nautilus,
and after re-balancing, we can
> > remove the
> > > 6 old nodes from the cluster.
> > >
> > > Any suggestions, advice, or best practice would be highly
appreciated.
>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
387
days inactive
390
days old
8 comments
7 participants
participants (7)
-
Dan van der Ster -
huxiaoyu＠horebdata.cn -
Joachim Kraftmayer -
Josh Baergen -
Marc -
Thomas Bennett -
Wesley Dillingham