HI Anthony, Based on your inputs and further digging into Ceph documentation, I am now thinking to go for 6 OSD nodes to have k=4 and m=2 EC set-up. As I mentioned., we need maximum usable space and we are more concerned about data safety and best read performance from the cluster. Writes operation will be done on a separate storage solution via NFS. So, with each OSD node having 22X4T Enterprise SSD , we will have 88X6 = 528T Raw Space. With 4X2 EC , it will hopefully provide us with 390T usable space. So, that will be enough for us to start with. So, I need to know what will be data safely level with the above set-up ( i.e. 6 OSDs with 4X2 EC ). How many OSDs ( disks ) and nodes failure , above set-up can withstand. Also, if , later, we need to add more OSD modes to get more usable space, will we need to add same size disks ( 4T ) or can we add nodes with bigger size disks ( 8T or 15T ) ? Beside OSDs server , going to have three Dell servers with 8 Core and 64G RAM to run 3 monitor daemons one on each server. One 4 core and 64G RAM with high core freq ( 4800 MHz ) server to run MDS daemon. Please advise Thanks, Gagan On Tue, Apr 15, 2025 at 8:14 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
It’s a function of your use-case.
On Apr 14, 2025, at 8:41 AM, Anthony Fecarotta <anthony@linehaul.ai> wrote:
MDS (if you’re going to CephFS vs using S3 object storage or RBD block) Hi Anthony,
Can you elaborate on this remark?
Should one choose between using CephFS vs S3 Storage (as it pertains to best practices)?
On Proxmox, I am. using both CephFS and RBD.
Regards, [image] Anthony Fecarotta Founder & President [image] anthony@linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>
On Sun Apr 13, 2025, 04:28 PM GMT, Anthony D'Atri <mailto: anthony.datri@gmail.com> wrote:
On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <brendon@netcal.com>
On Apr 11, 2025, at 10:13, gagan tiwari <
gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway).
Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can
wrote: lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference.
Other factors include:
* Enough hosts and OSDs. Three hosts with one OSD each aren’t going to
* At least 6GB of available physmem per NVMe OSD * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload.
Each step on the journey to enterprise SSDs made things faster. The
deliver a great experience problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant.
Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM
per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads.
So from my experience, I would say ceph is very slow in general
compared to DAS. You need all the help you can get.
If you want to use the consumer stuff, I would recommend to make a
slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster.
How much drives in each?
Good luck!
_BB
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io