Hello, Our company has a five node Proxmox cluster that recently implemented Ceph. We utilize multiple open-source applications that prefer (not require) S3 storage. Now that we are running Ceph, are we able to simply create a Ceph Object Gateway with the S3 API? I have read that not all S3 buckets share the same compatibility. Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>
On Jan 30, 2025, at 7:09 PM, Anthony Fecarotta <anthony@linehaul.ai> wrote:
Hello,
Our company has a five node Proxmox cluster that recently implemented Ceph.
Groovy. Welcome to Cephland.
We utilize multiple open-source applications that prefer (not require) S3 storage.
This is increasingly common.
Now that we are running Ceph, are we able to simply create a Ceph Object Gateway with the S3 API?
Yes, I believe so, though my Proxmox experience is limited.
I have read that not all S3 buckets share the same compatibility.
By buckets do you mean implementations? That would be true. Ceph RGW is compatible with the majority of AWS S3 behaviors, most clients and libraries slot right in by specifying the endpoint. What storage drives do you have? If you have mutliple, like SSDs and HDDs, TLC and QLC SSDs you’ll want to assign pools specifically.
Regards,
* Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, Anthony, I appreciate the inline responses. I'm not sure what the etiquette is here, if I am to reply all or not, so please let me know for future reference. - Yes, by buckets I was referring to implementations. - As for the compatibility, I have a great example to reference. There is an open-source alternative to Notion, named Outline, and the screenshot below is directly from their self-hosting documentation ( https://docs.getoutline.com/s/hosting/doc/file-storage-N4M0T6Ypu7): [image: image.png] I look at this and see that Ceph isn't even mentioned, so I decided this will not be the app that I test creating Rados Gateway. - Proxmox has a great UI for Ceph, with the exception that there is absolutely no reference to RGW, other than the documentation perhaps. - My setup is all Dell PowerEdge R740s running SAS SSDs 12Gbps, and each of them has a large Enterprise NVME. I have each SAS SSD configured as an OSD with the NVMe designated as the block.db. My performance is a mess now, that might be a topic for another thread. I would love to utilize the Rados Gateway for new software I am developing for our business. Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> On Thu, Jan 30, 2025 at 6:26 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
On Jan 30, 2025, at 7:09 PM, Anthony Fecarotta <anthony@linehaul.ai> wrote:
Hello,
Our company has a five node Proxmox cluster that recently implemented Ceph.
Groovy. Welcome to Cephland.
We utilize multiple open-source applications that prefer (not require) S3 storage.
This is increasingly common.
Now that we are running Ceph, are we able to simply create a Ceph Object Gateway with the S3 API?
Yes, I believe so, though my Proxmox experience is limited.
I have read that not all S3 buckets share the same compatibility.
By buckets do you mean implementations? That would be true. Ceph RGW is compatible with the majority of AWS S3 behaviors, most clients and libraries slot right in by specifying the endpoint.
What storage drives do you have? If you have mutliple, like SSDs and HDDs, TLC and QLC SSDs you’ll want to assign pools specifically.
Regards,
* Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, RGW in ProxMox (PM) has been answered somewhere in the PM community forum; basically, PM has no use for RGW so it’s not included. There are people who have varying success in setting up the ceph dashboard in PM and trying to get RGW in that way. You’d be better of creating another ceph cluster and use that for your S3 needs. It’s a great learning experience and you can start with vm’s and eventually move out of those. Venlig hilsen - Mit freundlichen Grüßen - Kind Regards, Jens Galsgaard Gitservice.dk Mob: +45 28864340 Fra: Anthony Fecarotta <anthony@linehaul.ai> Sendt: Friday, 31 January 2025 05.24 Til: Anthony D'Atri <aad@dreamsnake.net> Cc: ceph-users@ceph.io Emne: [ceph-users] Re: RGW S3 Compatibility Thanks, Anthony, I appreciate the inline responses. I'm not sure what the etiquette is here, if I am to reply all or not, so please let me know for future reference. * Yes, by buckets I was referring to implementations. * As for the compatibility, I have a great example to reference. There is an open-source alternative to Notion, named Outline, and the screenshot below is directly from their self-hosting documentation (https://docs.getoutline.com/s/hosting/doc/file-storage-N4M0T6Ypu7): [cid:image001.png@01DB73B8.3C149480] I look at this and see that Ceph isn't even mentioned, so I decided this will not be the app that I test creating Rados Gateway. * Proxmox has a great UI for Ceph, with the exception that there is absolutely no reference to RGW, other than the documentation perhaps. * My setup is all Dell PowerEdge R740s running SAS SSDs 12Gbps, and each of them has a large Enterprise NVME. I have each SAS SSD configured as an OSD with the NVMe designated as the block.db. My performance is a mess now, that might be a topic for another thread. I would love to utilize the Rados Gateway for new software I am developing for our business. Regards, [https://cdn.bulksignature.com/photo/5523_101429355110709750719.jpg?v=1] Anthony Fecarotta Founder & President [phone-icon] anthony@linehaul.ai<mailto:anthony@linehaul.ai> [phone-icon] 224-339-1182 [phone-icon] (855) 625-0300 [phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [phone-icon] www.linehaul.ai<http://www.linehaul.ai/> [linehaul.ai logo]<http://www.linehaul.ai/> [linkedin icon]<https://www.linkedin.com/in/anthony-fec/> On Thu, Jan 30, 2025 at 6:26 PM Anthony D'Atri <aad@dreamsnake.net<mailto:aad@dreamsnake.net>> wrote:
On Jan 30, 2025, at 7:09 PM, Anthony Fecarotta <anthony@linehaul.ai<mailto:anthony@linehaul.ai>> wrote:
Hello,
Our company has a five node Proxmox cluster that recently implemented Ceph.
Groovy. Welcome to Cephland.
We utilize multiple open-source applications that prefer (not require) S3 storage.
This is increasingly common.
Now that we are running Ceph, are we able to simply create a Ceph Object Gateway with the S3 API?
Yes, I believe so, though my Proxmox experience is limited.
I have read that not all S3 buckets share the same compatibility.
By buckets do you mean implementations? That would be true. Ceph RGW is compatible with the majority of AWS S3 behaviors, most clients and libraries slot right in by specifying the endpoint. What storage drives do you have? If you have mutliple, like SSDs and HDDs, TLC and QLC SSDs you’ll want to assign pools specifically.
Regards,
* Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai<mailto:anthony@linehaul.ai> [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai<http://www.linehaul.ai> [image: linehaul.ai<http://linehaul.ai> logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Thanks for the insight. Managing one Ceph Cluster has been a handful among all the other initiatives we're taking, I couldn't imagine running two right now. My other alternatives are getting the Ceph Dashboard up in Proxmox or managing RGW via CLI in Terminal? If the latter is an option, this would be a perfect opportunity to develop a TUI application for Ceph. Something similar to Lazydocker, Lazygit, CTOP, etc... thoughts? Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> On Fri, Jan 31, 2025 at 1:16 AM Jens Galsgaard <jens@gitservice.dk> wrote:
Hi Anthony,
RGW in ProxMox (PM) has been answered somewhere in the PM community forum; basically, PM has no use for RGW so it’s not included. There are people who have varying success in setting up the ceph dashboard in PM and trying to get RGW in that way.
You’d be better of creating another ceph cluster and use that for your S3 needs.
It’s a great learning experience and you can start with vm’s and eventually move out of those.
Venlig hilsen - Mit freundlichen Grüßen - Kind Regards, Jens Galsgaard
*Gitservice.dk *Mob: +45 28864340
*Fra:* Anthony Fecarotta <anthony@linehaul.ai> *Sendt:* Friday, 31 January 2025 05.24 *Til:* Anthony D'Atri <aad@dreamsnake.net> *Cc:* ceph-users@ceph.io *Emne:* [ceph-users] Re: RGW S3 Compatibility
Thanks, Anthony, I appreciate the inline responses. I'm not sure what the etiquette is here, if I am to reply all or not, so please let me know for future reference.
- Yes, by buckets I was referring to implementations. - As for the compatibility, I have a great example to reference. There is an open-source alternative to Notion, named Outline, and the screenshot below is directly from their self-hosting documentation ( https://docs.getoutline.com/s/hosting/doc/file-storage-N4M0T6Ypu7):
I look at this and see that Ceph isn't even mentioned, so I decided this will not be the app that I test creating Rados Gateway. - Proxmox has a great UI for Ceph, with the exception that there is absolutely no reference to RGW, other than the documentation perhaps. - My setup is all Dell PowerEdge R740s running SAS SSDs 12Gbps, and each of them has a large Enterprise NVME. I have each SAS SSD configured as an OSD with the NVMe designated as the block.db. My performance is a mess now, that might be a topic for another thread. I would love to utilize the Rados Gateway for new software I am developing for our business.
Regards,
* Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai
[image: linehaul.ai logo] <http://www.linehaul.ai/>
[image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>
On Thu, Jan 30, 2025 at 6:26 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
On Jan 30, 2025, at 7:09 PM, Anthony Fecarotta <anthony@linehaul.ai> wrote:
Hello,
Our company has a five node Proxmox cluster that recently implemented Ceph.
Groovy. Welcome to Cephland.
We utilize multiple open-source applications that prefer (not require) S3 storage.
This is increasingly common.
Now that we are running Ceph, are we able to simply create a Ceph Object Gateway with the S3 API?
Yes, I believe so, though my Proxmox experience is limited.
I have read that not all S3 buckets share the same compatibility.
By buckets do you mean implementations? That would be true. Ceph RGW is compatible with the majority of AWS S3 behaviors, most clients and libraries slot right in by specifying the endpoint.
What storage drives do you have? If you have mutliple, like SSDs and HDDs, TLC and QLC SSDs you’ll want to assign pools specifically.
Regards,
* Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This may work in order to add RGW to a proxmox-ceph cluster: https://pve.proxmox.com/wiki/User:Grin/Ceph_Object_Gateway I havent tried it myself yet, but I will when I get some spare time. There will be no dashboard or anything so be prepared to manage everything through cli only. It would make sense to me for Proxmox to add in RGW support themselves I think. Many customers will need a small scale S3 service for their applications. /Peter Den 2025-01-31 kl. 08:15, skrev Jens Galsgaard:
Hi Anthony,
RGW in ProxMox (PM) has been answered somewhere in the PM community forum; basically, PM has no use for RGW so it’s not included. There are people who have varying success in setting up the ceph dashboard in PM and trying to get RGW in that way.
You’d be better of creating another ceph cluster and use that for your S3 needs.
It’s a great learning experience and you can start with vm’s and eventually move out of those.
Venlig hilsen - Mit freundlichen Grüßen - Kind Regards, Jens Galsgaard
*Gitservice.dk *Mob: +45 28864340
*Fra:*Anthony Fecarotta <anthony@linehaul.ai> *Sendt:* Friday, 31 January 2025 05.24 *Til:* Anthony D'Atri <aad@dreamsnake.net> *Cc:* ceph-users@ceph.io *Emne:* [ceph-users] Re: RGW S3 Compatibility
Thanks, Anthony, I appreciate the inline responses. I'm not sure what the etiquette is here, if I am to reply all or not, so please let me know for future reference.
* Yes, by buckets I was referring to implementations. * As for the compatibility, I have a great example to reference. There is an open-source alternative to Notion, named Outline, and the screenshot below is directly from their self-hosting documentation (https://docs.getoutline.com/s/hosting/doc/file-storage-N4M0T6Ypu7):
I look at this and see that Ceph isn't even mentioned, so I decided this will not be the app that I test creating Rados Gateway. * Proxmox has a great UI for Ceph, with the exception that there is absolutely no reference to RGW, other than the documentation perhaps. * My setup is all Dell PowerEdge R740s running SAS SSDs 12Gbps, and each of them has a large Enterprise NVME. I have each SAS SSD configured as an OSD with the NVMe designated as the block.db. My performance is a mess now, that might be a topic for another thread. I would love to utilize the Rados Gateway for new software I am developing for our business.
Regards,
* Anthony Fecarotta* Founder & President phone-iconanthony@linehaul.ai <mailto:anthony@linehaul.ai> phone-icon224-339-1182 phone-icon(855) 625-0300 phone-icon 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 phone-icon www.linehaul.ai <http://www.linehaul.ai/>
linehaul.ai logo <http://www.linehaul.ai/>
linkedin icon <https://www.linkedin.com/in/anthony-fec/>
On Thu, Jan 30, 2025 at 6:26 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
> On Jan 30, 2025, at 7:09 PM, Anthony Fecarotta <anthony@linehaul.ai> wrote: > > Hello, > > Our company has a five node Proxmox cluster that recently implemented Ceph.
Groovy. Welcome to Cephland.
> We utilize multiple open-source applications that prefer (not require) S3 > storage.
This is increasingly common.
> Now that we are running Ceph, are we able to simply create a Ceph > Object Gateway with the S3 API?
Yes, I believe so, though my Proxmox experience is limited.
> I have read that not all S3 buckets share the same compatibility.
By buckets do you mean implementations? That would be true. Ceph RGW is compatible with the majority of AWS S3 behaviors, most clients and libraries slot right in by specifying the endpoint.
What storage drives do you have? If you have mutliple, like SSDs and HDDs, TLC and QLC SSDs you’ll want to assign pools specifically.
> > Regards, > > > * Anthony Fecarotta* > Founder & President > [image: phone-icon] anthony@linehaul.ai > [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 > [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 > [image: phone-icon] www.linehaul.ai <http://www.linehaul.ai> > [image: linehaul.ai <http://linehaul.ai> logo] <http://www.linehaul.ai/> > [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
I agree, but I don't see that coming anytime soon, unless they allow outside contributions. They seem pretty dug in on Proxmox Datacenter. Minio is becoming optional or a dependency in most of the open source CRUD applications I see. It would be a giant win ifor a proxmox user to have the ease to one click deploy a Rados Gateway. Running Minio on top of Ceph seems silly. The user made a comment: "*There is also mention of using ceph-deploy, but I knew that Proxmox uses
it's own pveceph tools"*
Correct me if I'm wrong, but that makes me believe that even if Ceph Dashboard was deployed on Proxmox, the logic behind the UI would be ineffective. I will try this method you sent and hopefully don't make a mess of things. Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> On Fri, Jan 31, 2025 at 2:52 AM Peter Linder <peter.linder@fiberdirekt.se> wrote:
This may work in order to add RGW to a proxmox-ceph cluster:
https://pve.proxmox.com/wiki/User:Grin/Ceph_Object_Gateway
I havent tried it myself yet, but I will when I get some spare time. There will be no dashboard or anything so be prepared to manage everything through cli only.
It would make sense to me for Proxmox to add in RGW support themselves I think. Many customers will need a small scale S3 service for their applications.
/Peter
Den 2025-01-31 kl. 08:15, skrev Jens Galsgaard:
Hi Anthony,
RGW in ProxMox (PM) has been answered somewhere in the PM community forum; basically, PM has no use for RGW so it’s not included. There are people who have varying success in setting up the ceph dashboard in PM and trying to get RGW in that way.
You’d be better of creating another ceph cluster and use that for your S3 needs.
It’s a great learning experience and you can start with vm’s and eventually move out of those.
Venlig hilsen - Mit freundlichen Grüßen - Kind Regards, Jens Galsgaard
*Gitservice.dk *Mob: +45 28864340
*Fra:*Anthony Fecarotta <anthony@linehaul.ai> *Sendt:* Friday, 31 January 2025 05.24 *Til:* Anthony D'Atri <aad@dreamsnake.net> *Cc:* ceph-users@ceph.io *Emne:* [ceph-users] Re: RGW S3 Compatibility
Thanks, Anthony, I appreciate the inline responses. I'm not sure what the etiquette is here, if I am to reply all or not, so please let me know for future reference.
* Yes, by buckets I was referring to implementations. * As for the compatibility, I have a great example to reference. There is an open-source alternative to Notion, named Outline, and the screenshot below is directly from their self-hosting documentation (https://docs.getoutline.com/s/hosting/doc/file-storage-N4M0T6Ypu7):
I look at this and see that Ceph isn't even mentioned, so I decided this will not be the app that I test creating Rados Gateway. * Proxmox has a great UI for Ceph, with the exception that there is absolutely no reference to RGW, other than the documentation perhaps. * My setup is all Dell PowerEdge R740s running SAS SSDs 12Gbps, and each of them has a large Enterprise NVME. I have each SAS SSD configured as an OSD with the NVMe designated as the block.db. My performance is a mess now, that might be a topic for another thread. I would love to utilize the Rados Gateway for new software I am developing for our business.
Regards,
* Anthony Fecarotta* Founder & President phone-iconanthony@linehaul.ai <mailto:anthony@linehaul.ai> phone-icon224-339-1182 phone-icon(855) 625-0300 phone-icon 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 phone-icon www.linehaul.ai <http://www.linehaul.ai/>
linehaul.ai logo <http://www.linehaul.ai/>
linkedin icon <https://www.linkedin.com/in/anthony-fec/>
On Thu, Jan 30, 2025 at 6:26 PM Anthony D'Atri <aad@dreamsnake.net>
wrote:
> On Jan 30, 2025, at 7:09 PM, Anthony Fecarotta <anthony@linehaul.ai> wrote: > > Hello, > > Our company has a five node Proxmox cluster that recently implemented Ceph.
Groovy. Welcome to Cephland.
> We utilize multiple open-source applications that prefer (not require) S3 > storage.
This is increasingly common.
> Now that we are running Ceph, are we able to simply create a Ceph > Object Gateway with the S3 API?
Yes, I believe so, though my Proxmox experience is limited.
> I have read that not all S3 buckets share the same compatibility.
By buckets do you mean implementations? That would be true. Ceph RGW is compatible with the majority of AWS S3 behaviors, most clients and libraries slot right in by specifying the endpoint.
What storage drives do you have? If you have mutliple, like SSDs and HDDs, TLC and QLC SSDs you’ll want to assign pools specifically.
> > Regards, > > > * Anthony Fecarotta* > Founder & President > [image: phone-icon] anthony@linehaul.ai > [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 > [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 > [image: phone-icon] www.linehaul.ai <http://www.linehaul.ai> > [image: linehaul.ai <http://linehaul.ai> logo] <http://www.linehaul.ai/> > [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Anthony Fecarotta
-
Jens Galsgaard
-
Peter Linder