What is the proper way to setup Rados Gateway (RGW) under Ceph?
I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here: https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-s.... The installation steps were fairly easy and I was able to get the environment up and running in about 15 minutes under VMware ESXi 7. I have buckets and pools already setup. However, the ceph.io site is confusing on how to setup the Rados Gateway (radosgw) with Multi-site -- https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy also needed for handling the front-end load balancing or is it implied that Ceph sets it up? Command-line scripting I was planning on using for setting up the RGW: ``` radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin zonegroup create --rgw-zonegroup=sandbox --master --default radosgw-admin zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1 ceph-mon2" --port=8000 ``` What other steps are needed to get the RGW up and running so that it can be presented to something like Veeam for doing performance and I/O testing concepts? -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.
Hi, do you really need multi-site since you mentioned that you have one cluster? Maybe start with single-site RGW [1] since there's no replication target anyway. If you deploy multiple rgw daemons you might need an ingress service [2] as well and point your zone endpoints to your virtual IP. The ingress service deploys a haproxy and a keepalive daemon on each node, one of them has the virtual IP assigned. I just added an ingress service to my existing rgws in my lab and it seems to work, although I don't have any clients connected to the cluster. But the logs don't contain any errors so I think I'm good. Regards, Eugen [1] https://docs.ceph.com/en/latest/cephadm/services/rgw/#deploy-rgws [2] https://docs.ceph.com/en/latest/cephadm/services/rgw/#high-availability-serv... Zitat von Michael Worsham <mworsham@datadimensions.com>:
I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here: https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-s....
The installation steps were fairly easy and I was able to get the environment up and running in about 15 minutes under VMware ESXi 7. I have buckets and pools already setup. However, the ceph.io site is confusing on how to setup the Rados Gateway (radosgw) with Multi-site -- https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy also needed for handling the front-end load balancing or is it implied that Ceph sets it up?
Command-line scripting I was planning on using for setting up the RGW: ``` radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin zonegroup create --rgw-zonegroup=sandbox --master --default radosgw-admin zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1 ceph-mon2" --port=8000 ```
What other steps are needed to get the RGW up and running so that it can be presented to something like Veeam for doing performance and I/O testing concepts?
-- Michael
This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Can anyone help me on this? I can't be that hard to do. -- Michael -----Original Message----- From: Michael Worsham <mworsham@datadimensions.com> Sent: Thursday, February 8, 2024 3:03 PM To: ceph-users@ceph.io Subject: [ceph-users] What is the proper way to setup Rados Gateway (RGW) under Ceph? I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here: https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-s.... The installation steps were fairly easy and I was able to get the environment up and running in about 15 minutes under VMware ESXi 7. I have buckets and pools already setup. However, the ceph.io site is confusing on how to setup the Rados Gateway (radosgw) with Multi-site -- https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy also needed for handling the front-end load balancing or is it implied that Ceph sets it up? Command-line scripting I was planning on using for setting up the RGW: ``` radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin zonegroup create --rgw-zonegroup=sandbox --master --default radosgw-admin zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1 ceph-mon2" --port=8000 ``` What other steps are needed to get the RGW up and running so that it can be presented to something like Veeam for doing performance and I/O testing concepts? -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.
Hi, recommended methods of deploying rgw are imho overly complicated. You can get service up manually also with something simple like: [root@mon1 bin]# cat /etc/ceph/ceph.conf [global] fsid = 12345678-XXXx ... mon initial members = mon1,mon3 mon host = ip-mon1,ip-mon2 auth cluster required = none auth service required = none auth client required = none ms_mon_client_mode = crc [client.radosgw.mon1] host = mon1 log_file = /var/log/ceph/client.radosgw.mon1.log rgw_dns_name = mon1 rgw_frontends = "civetweb port=80 num_threads=500" # this is different in ceph versions 17, 18. rgw_crypt_require_ssl = false ---- [root@mon1 bin]# cat start-rgw.sh radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n client.radosgw.mon1 & --- This configuration has nginx in front of rgw .... all traffic goes from nginx 443 -> rgw 80 and it assumes you "own the network" and you are aware of "drawbacks". Rok On Mon, Feb 12, 2024 at 2:15 PM Michael Worsham <mworsham@datadimensions.com> wrote:
Can anyone help me on this? I can't be that hard to do.
-- Michael
-----Original Message----- From: Michael Worsham <mworsham@datadimensions.com> Sent: Thursday, February 8, 2024 3:03 PM To: ceph-users@ceph.io Subject: [ceph-users] What is the proper way to setup Rados Gateway (RGW) under Ceph?
I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here: https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-s... .
The installation steps were fairly easy and I was able to get the environment up and running in about 15 minutes under VMware ESXi 7. I have buckets and pools already setup. However, the ceph.io site is confusing on how to setup the Rados Gateway (radosgw) with Multi-site -- https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy also needed for handling the front-end load balancing or is it implied that Ceph sets it up?
Command-line scripting I was planning on using for setting up the RGW: ``` radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin zonegroup create --rgw-zonegroup=sandbox --master --default radosgw-admin zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1 ceph-mon2" --port=8000 ```
What other steps are needed to get the RGW up and running so that it can be presented to something like Veeam for doing performance and I/O testing concepts?
-- Michael
This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So, just so I am clear – in addition to the steps below, will I also need to also install NGINX or HAProxy on the server to act as the front end? -- M From: Rok Jaklič <rjaklic@gmail.com> Sent: Monday, February 12, 2024 12:30 PM To: Michael Worsham <mworsham@datadimensions.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: What is the proper way to setup Rados Gateway (RGW) under Ceph? This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. Hi, recommended methods of deploying rgw are imho overly complicated. You can get service up manually also with something simple like: [root@mon1 bin]# cat /etc/ceph/ceph.conf [global] fsid = 12345678-XXXx ... mon initial members = mon1,mon3 mon host = ip-mon1,ip-mon2 auth cluster required = none auth service required = none auth client required = none ms_mon_client_mode = crc [client.radosgw.mon1] host = mon1 log_file = /var/log/ceph/client.radosgw.mon1.log rgw_dns_name = mon1 rgw_frontends = "civetweb port=80 num_threads=500" # this is different in ceph versions 17, 18. rgw_crypt_require_ssl = false ---- [root@mon1 bin]# cat start-rgw.sh radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n client.radosgw.mon1 & --- This configuration has nginx in front of rgw .... all traffic goes from nginx 443 -> rgw 80 and it assumes you "own the network" and you are aware of "drawbacks". Rok On Mon, Feb 12, 2024 at 2:15 PM Michael Worsham <mworsham@datadimensions.com<mailto:mworsham@datadimensions.com>> wrote: Can anyone help me on this? I can't be that hard to do. -- Michael -----Original Message----- From: Michael Worsham <mworsham@datadimensions.com<mailto:mworsham@datadimensions.com>> Sent: Thursday, February 8, 2024 3:03 PM To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] What is the proper way to setup Rados Gateway (RGW) under Ceph? I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here: https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-s.... The installation steps were fairly easy and I was able to get the environment up and running in about 15 minutes under VMware ESXi 7. I have buckets and pools already setup. However, the ceph.io<http://ceph.io> site is confusing on how to setup the Rados Gateway (radosgw) with Multi-site -- https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy also needed for handling the front-end load balancing or is it implied that Ceph sets it up? Command-line scripting I was planning on using for setting up the RGW: ``` radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin zonegroup create --rgw-zonegroup=sandbox --master --default radosgw-admin zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1 ceph-mon2" --port=8000 ``` What other steps are needed to get the RGW up and running so that it can be presented to something like Veeam for doing performance and I/O testing concepts? -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.
You don't have to. You can serve rgw on the front end directly. You: 1. set certificate with smth like: rgw_frontends = " ... ssl_certificate=/etc/pki/ceph/cert.pem". We are using nginx on front end to act as a proxy and for some other stuff. 2. delete line with rgw_crypt_require_ssl ... you should be ready to go. :) Rok On Mon, Feb 12, 2024 at 6:43 PM Michael Worsham <mworsham@datadimensions.com> wrote:
So, just so I am clear – in addition to the steps below, will I also need to also install NGINX or HAProxy on the server to act as the front end?
-- M
*From:* Rok Jaklič <rjaklic@gmail.com> *Sent:* Monday, February 12, 2024 12:30 PM *To:* Michael Worsham <mworsham@datadimensions.com> *Cc:* ceph-users@ceph.io *Subject:* Re: [ceph-users] Re: What is the proper way to setup Rados Gateway (RGW) under Ceph?
This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security.
Hi,
recommended methods of deploying rgw are imho overly complicated. You can get service up manually also with something simple like:
[root@mon1 bin]# cat /etc/ceph/ceph.conf
[global] fsid = 12345678-XXXx ... mon initial members = mon1,mon3 mon host = ip-mon1,ip-mon2 auth cluster required = none auth service required = none auth client required = none ms_mon_client_mode = crc
[client.radosgw.mon1] host = mon1 log_file = /var/log/ceph/client.radosgw.mon1.log rgw_dns_name = mon1 rgw_frontends = "civetweb port=80 num_threads=500" # this is different in ceph versions 17, 18. rgw_crypt_require_ssl = false
----
[root@mon1 bin]# cat start-rgw.sh radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n client.radosgw.mon1 &
---
This configuration has nginx in front of rgw .... all traffic goes from nginx 443 -> rgw 80 and it assumes you "own the network" and you are aware of "drawbacks".
Rok
On Mon, Feb 12, 2024 at 2:15 PM Michael Worsham < mworsham@datadimensions.com> wrote:
Can anyone help me on this? I can't be that hard to do.
-- Michael
-----Original Message----- From: Michael Worsham <mworsham@datadimensions.com> Sent: Thursday, February 8, 2024 3:03 PM To: ceph-users@ceph.io Subject: [ceph-users] What is the proper way to setup Rados Gateway (RGW) under Ceph?
I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here: https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-s... .
The installation steps were fairly easy and I was able to get the environment up and running in about 15 minutes under VMware ESXi 7. I have buckets and pools already setup. However, the ceph.io site is confusing on how to setup the Rados Gateway (radosgw) with Multi-site -- https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy also needed for handling the front-end load balancing or is it implied that Ceph sets it up?
Command-line scripting I was planning on using for setting up the RGW: ``` radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin zonegroup create --rgw-zonegroup=sandbox --master --default radosgw-admin zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1 ceph-mon2" --port=8000 ```
What other steps are needed to get the RGW up and running so that it can be presented to something like Veeam for doing performance and I/O testing concepts?
-- Michael
This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.
I tried the follow the steps I did originally listed, but it seems I am not doing something right. Our Storage/backup admin cannot connect to port 8000 on the server instance. Do I need to setup a HAProxy as a front-end on a different port to communicate with the RGW listening ports between the two ceph monitors? -- Michael -----Original Message----- From: Michael Worsham <mworsham@datadimensions.com> Sent: Thursday, February 8, 2024 3:03 PM To: ceph-users@ceph.io Subject: [ceph-users] What is the proper way to setup Rados Gateway (RGW) under Ceph? I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here: https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-s.... The installation steps were fairly easy and I was able to get the environment up and running in about 15 minutes under VMware ESXi 7. I have buckets and pools already setup. However, the ceph.io site is confusing on how to setup the Rados Gateway (radosgw) with Multi-site -- https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy also needed for handling the front-end load balancing or is it implied that Ceph sets it up? Command-line scripting I was planning on using for setting up the RGW: ``` radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin zonegroup create --rgw-zonegroup=sandbox --master --default radosgw-admin zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1 ceph-mon2" --port=8000 ``` What other steps are needed to get the RGW up and running so that it can be presented to something like Veeam for doing performance and I/O testing concepts? -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.
participants (3)
-
Eugen Block
-
Michael Worsham
-
Rok Jaklič