ceph deployment best practice
Hi Guys, We have an HPC environment which currently has a single master host that stores the entire data around 100T and exports that data via NFS to the clients. We are using OpenZFS on the single master host. But now we need to store much more data around 500T and we are facing scalability issues with ZFS as it can't be scaled out. So, we are considering replacing OpenZFS with Ceph. I have gone the online docs on https://docs.ceph.com and I have come up with the following ceph deployment plan :- We would start deploying Ceph with 4 hosts ( HP Proliant servers ) each running RockyLinux 9. One of the hosts called ceph-adm will be smaller one and will have following hardware :- 2x4T SSD with raid 1 to install OS on. 8 Core with 3600MHz freq. 64G RAM We are planning to run all Ceph daemons except OSD daemon like monitor , metadata ,etc on this host. We will have 3 hosts to run OSD which will store actual data. Each OSD host will have following hardware 2x4T SSD with raid 1 to install OS on. 22X8T SSG to store data ( OSDs ) ( without partition ). We will use entire disk without partitions Each OSD host will have 128G RAM ( No swap space ) Each OSD host will have 16 cores. All 4 hosts will connect to each via 10G nic. The 500T data will be accessed by the clients. We need to have read performance as fast as possible. We can't afford data loss and downtime. So, we want to have a Ceph deployment which serves our purpose. So, please advise me if the plan that I have designed will serve our purpose. Or is there a better way , please advise that. Thanks, Gagan We have a HP storage server with 12 SDD of 5T each and have set-up hardware RAID6 on these disks. HP storage server has 64G RAM and 18 cores. So, please advise how I should go about setting up Ceph on it to have best read performance. We need fastest read performance. Thanks, Gagan
We would start deploying Ceph with 4 hosts ( HP Proliant servers ) each running RockyLinux 9.
One of the hosts called ceph-adm will be smaller one and will have following hardware :-
2x4T SSD with raid 1 to install OS on.
8 Core with 3600MHz freq.
64G RAM
We are planning to run all Ceph daemons except OSD daemon like monitor , metadata ,etc on this host.
8 core == 16 threads? Are you provisioning this node because you have it laying around idle? Note that you will want *at least* 3 Monitor (monitors) daemons, which must be on different nodes. 5 is better, but at least 3. You’ll also have Grafana, Prometheus, MDS (if you’re going to CephFS vs using S3 object storage or RBD block) 8c is likely on the light side for all of that. You would also benefit from not having that node be a single point of failure. I would suggest if you can raising this node to the spec of the planned 3x OSD nodes so you have 4x equivalent nodes, and spread that non-OSD daemons across them. Note also that your OSD nodes will also have node_exporter, crash, and other boilerplate daemons.
We will have 3 hosts to run OSD which will store actual data.
Each OSD host will have following hardware
2x4T SSD with raid 1 to install OS on.
22X8T SSD to store data ( OSDs ) ( without partition ). We will use entire disk without partitions
SAS, SATA, or NVMe SSDs? Which specific model? You really want to avoid client (desktop) models for Ceph, but you likely do not need to pay for higher endurance mixed-use SKUs.
Each OSD host will have 128G RAM ( No swap space )
Thank you for skipping swap. Some people are really stuck in the past in that regard.
Each OSD host will have 16 cores.
So 32 threads total? That is very light for 22 OSDs + other daemons. For HDD OSDs a common rule of thumb is at minimum 2x threads per, for SAS/SATA SSDs, 4, for NVMe SSDs 6. Plus margin for the OS and other processes.
All 4 hosts will connect to each via 10G nic.
Two ports with bonding? Redundant switches?
The 500T data
The specs you list above include 528 TB of *raw* space. Be advised that with three OSD nodes, you will necessarily be doing replication. For safety replication with size=3. Taking into consideration TB vs TiB and headroom, you’re looking at 133TiB of usable space. You could go with size=2 to get 300TB of usable space, but at increased risk of data unavailability or loss when drives/hosts fail or reboot. With at least 4 OSD nodes - even if they aren’t fully populated with capacity drives — you could do EC for a more favorable raw:usable ratio, at the expense of slower writes and recovery. With 4 nodes you could in theory do 2,2 EC for 200 TiB of usable space, with 5 you could do 3,2 for 240 TiB usable, etc.
will be accessed by the clients. We need to have read performance as fast as possible.
Hope your SSDs are enterprise NVMe.
We can't afford data loss and downtime.
Then no size=2 for you.
So, we want to have a Ceph deployment which serves our purpose.
So, please advise me if the plan that I have designed will serve our purpose. Or is there a better way , please advise that.
Thanks, Gagan
We have a HP storage server with 12 SDD of 5T each and have set-up hardware RAID6 on these disks.
HP storage server has 64G RAM and 18 cores.
So, please advise how I should go about setting up Ceph on it to have best read performance. We need fastest read performance.
Thanks, Gagan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, Thanks for the reply! We will be using CephFS to access Ceph Storage from clients. So, this will need MDS daemon also. So, based on your advice, I am thinking of having 4 Dell PowerEdge servers . 3 of them will run 3 Monitor daemons and one of them will run MDS daemon. These Dell Servers will have following hardware :- 1. 4 cores ( 8 threads ) ( Can go for 8 core and 16 threads ) 2. 64G RAM 3. 2x4T Samsung SSD with RA!D 1 to install OS and run monitor and metadata services. OSD nodes will be upgraded to have 32 cores ( 64 threads ). Disk and RAM will remain same ( 128G and 22X8T Samsung SSD ) Actually , I want to use OSD nodes to run OSD damons and not any other demons and which is why I am thinking of having 4 additional Dell servers as mentioned above. Please advise if this plan will be better. Thanks, Gagan On Wed, Apr 9, 2025 at 8:12 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
We would start deploying Ceph with 4 hosts ( HP Proliant servers ) each running RockyLinux 9.
One of the hosts called ceph-adm will be smaller one and will have following hardware :-
2x4T SSD with raid 1 to install OS on.
8 Core with 3600MHz freq.
64G RAM
We are planning to run all Ceph daemons except OSD daemon like monitor , metadata ,etc on this host.
8 core == 16 threads? Are you provisioning this node because you have it laying around idle?
Note that you will want *at least* 3 Monitor (monitors) daemons, which must be on different nodes. 5 is better, but at least 3. You’ll also have Grafana, Prometheus, MDS (if you’re going to CephFS vs using S3 object storage or RBD block)
8c is likely on the light side for all of that. You would also benefit from not having that node be a single point of failure. I would suggest if you can raising this node to the spec of the planned 3x OSD nodes so you have 4x equivalent nodes, and spread that non-OSD daemons across them.
Note also that your OSD nodes will also have node_exporter, crash, and other boilerplate daemons.
We will have 3 hosts to run OSD which will store actual data.
Each OSD host will have following hardware
2x4T SSD with raid 1 to install OS on.
22X8T SSD to store data ( OSDs ) ( without partition ). We will use entire disk without partitions
SAS, SATA, or NVMe SSDs? Which specific model? You really want to avoid client (desktop) models for Ceph, but you likely do not need to pay for higher endurance mixed-use SKUs.
Each OSD host will have 128G RAM ( No swap space )
Thank you for skipping swap. Some people are really stuck in the past in that regard.
Each OSD host will have 16 cores.
So 32 threads total? That is very light for 22 OSDs + other daemons. For HDD OSDs a common rule of thumb is at minimum 2x threads per, for SAS/SATA SSDs, 4, for NVMe SSDs 6. Plus margin for the OS and other processes.
All 4 hosts will connect to each via 10G nic.
Two ports with bonding? Redundant switches?
The 500T data
The specs you list above include 528 TB of *raw* space. Be advised that with three OSD nodes, you will necessarily be doing replication. For safety replication with size=3. Taking into consideration TB vs TiB and headroom, you’re looking at 133TiB of usable space. You could go with size=2 to get 300TB of usable space, but at increased risk of data unavailability or loss when drives/hosts fail or reboot.
With at least 4 OSD nodes - even if they aren’t fully populated with capacity drives — you could do EC for a more favorable raw:usable ratio, at the expense of slower writes and recovery. With 4 nodes you could in theory do 2,2 EC for 200 TiB of usable space, with 5 you could do 3,2 for 240 TiB usable, etc.
will be accessed by the clients. We need to have read performance as fast as possible.
Hope your SSDs are enterprise NVMe.
We can't afford data loss and downtime.
Then no size=2 for you.
So, we want to have a Ceph deployment which serves our purpose.
So, please advise me if the plan that I have designed will serve our purpose. Or is there a better way , please advise that.
Thanks, Gagan
We have a HP storage server with 12 SDD of 5T each and have set-up hardware RAID6 on these disks.
HP storage server has 64G RAM and 18 cores.
So, please advise how I should go about setting up Ceph on it to have best read performance. We need fastest read performance.
Thanks, Gagan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Apr 11, 2025, at 4:04 AM, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, Thanks for the reply!
We will be using CephFS to access Ceph Storage from clients. So, this will need MDS daemon also.
MDS is single-threaded, so unlike most Ceph daemons it benefits more from a high-frequency CPU than core count.
So, based on your advice, I am thinking of having 4 Dell PowerEdge servers . 3 of them will run 3 Monitor daemons and one of them will run MDS daemon.
These Dell Servers will have following hardware :-
1. 4 cores ( 8 threads ) ( Can go for 8 core and 16 threads )
2. 64G RAM
3. 2x4T Samsung SSD with RA!D 1 to install OS and run monitor and metadata services.
That probably suffices for a small cluster. Are those Samsungs enterprise?
OSD nodes will be upgraded to have 32 cores ( 64 threads ). Disk and RAM will remain same ( 128G and 22X8T Samsung SSD )
Which Samsung SSD? Using client SKUs for OSDs has a way of leading to heartbreak. 64 threads would be better for a 22x OSD node, though still a bit light. Are these SATA or NVMe?
Actually , I want to use OSD nodes to run OSD damons and not any other demons and which is why I am thinking of having 4 additional Dell servers as mentioned above.
Colocation of daemons is common these days, especially with smaller clusters.
Please advise if this plan will be better.
That’ll work, but unless you already have those quite-modest 4x non-OSD nodes sitting around idle you might consider just going with the OSD nodes and bumping the CPU again so you can colocate all the daemons.
Thanks, Gagan
On Wed, Apr 9, 2025 at 8:12 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
We would start deploying Ceph with 4 hosts ( HP Proliant servers ) each running RockyLinux 9.
One of the hosts called ceph-adm will be smaller one and will have following hardware :-
2x4T SSD with raid 1 to install OS on.
8 Core with 3600MHz freq.
64G RAM
We are planning to run all Ceph daemons except OSD daemon like monitor , metadata ,etc on this host.
8 core == 16 threads? Are you provisioning this node because you have it laying around idle?
Note that you will want *at least* 3 Monitor (monitors) daemons, which must be on different nodes. 5 is better, but at least 3. You’ll also have Grafana, Prometheus, MDS (if you’re going to CephFS vs using S3 object storage or RBD block)
8c is likely on the light side for all of that. You would also benefit from not having that node be a single point of failure. I would suggest if you can raising this node to the spec of the planned 3x OSD nodes so you have 4x equivalent nodes, and spread that non-OSD daemons across them.
Note also that your OSD nodes will also have node_exporter, crash, and other boilerplate daemons.
We will have 3 hosts to run OSD which will store actual data.
Each OSD host will have following hardware
2x4T SSD with raid 1 to install OS on.
22X8T SSD to store data ( OSDs ) ( without partition ). We will use entire disk without partitions
SAS, SATA, or NVMe SSDs? Which specific model? You really want to avoid client (desktop) models for Ceph, but you likely do not need to pay for higher endurance mixed-use SKUs.
Each OSD host will have 128G RAM ( No swap space )
Thank you for skipping swap. Some people are really stuck in the past in that regard.
Each OSD host will have 16 cores.
So 32 threads total? That is very light for 22 OSDs + other daemons. For HDD OSDs a common rule of thumb is at minimum 2x threads per, for SAS/SATA SSDs, 4, for NVMe SSDs 6. Plus margin for the OS and other processes.
All 4 hosts will connect to each via 10G nic.
Two ports with bonding? Redundant switches?
The 500T data
The specs you list above include 528 TB of *raw* space. Be advised that with three OSD nodes, you will necessarily be doing replication. For safety replication with size=3. Taking into consideration TB vs TiB and headroom, you’re looking at 133TiB of usable space. You could go with size=2 to get 300TB of usable space, but at increased risk of data unavailability or loss when drives/hosts fail or reboot.
With at least 4 OSD nodes - even if they aren’t fully populated with capacity drives — you could do EC for a more favorable raw:usable ratio, at the expense of slower writes and recovery. With 4 nodes you could in theory do 2,2 EC for 200 TiB of usable space, with 5 you could do 3,2 for 240 TiB usable, etc.
will be accessed by the clients. We need to have read performance as fast as possible.
Hope your SSDs are enterprise NVMe.
We can't afford data loss and downtime.
Then no size=2 for you.
So, we want to have a Ceph deployment which serves our purpose.
So, please advise me if the plan that I have designed will serve our purpose. Or is there a better way , please advise that.
Thanks, Gagan
We have a HP storage server with 12 SDD of 5T each and have set-up hardware RAID6 on these disks.
HP storage server has 64G RAM and 18 cores.
So, please advise how I should go about setting up Ceph on it to have best read performance. We need fastest read performance.
Thanks, Gagan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, Your statement about MDS is interesting... So it's possible depending on the CPU-type that read/write operations on RBD will show a better performance than similar read/write operations on a CephFS?
MDS is single-threaded, so unlike most Ceph daemons it benefits more from a high-frequency CPU than core count.
There are a lot of variables there, including whether one uses KRBD or librbd for clients. I suspect that one can’t make a blanket statement either way.
Hi Anthony,
Your statement about MDS is interesting... So it's possible depending on the CPU-type that read/write operations on RBD will show a better performance than similar read/write operations on a CephFS?
MDS is single-threaded, so unlike most Ceph daemons it benefits more from a high-frequency CPU than core count.
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers. One more thing , I want to know is that CephFS supports mounting with FsCache on clients ? 500T data stored in the cluster will be accessed by the jobs running on the clients nodes and we need super fast read performance. For that we do have additional cache disk installed on all the clients nodes. And the way NFS V4 supports mount NFS share with FsCache on clients' hosts ,CephFS also supports that. On those 4x non-OSD nodes, I will probably run ldap and HTCondor service. But mds node will not be used for anything other than mds daemon. Thanks, Gagan On Fri, Apr 11, 2025 at 8:45 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
On Apr 11, 2025, at 4:04 AM, gagan tiwari < gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, Thanks for the reply!
We will be using CephFS to access Ceph Storage from clients. So, this will need MDS daemon also.
MDS is single-threaded, so unlike most Ceph daemons it benefits more from a high-frequency CPU than core count.
So, based on your advice, I am thinking of having 4 Dell PowerEdge servers . 3 of them will run 3 Monitor daemons and one of them will run MDS daemon.
These Dell Servers will have following hardware :-
1. 4 cores ( 8 threads ) ( Can go for 8 core and 16 threads )
2. 64G RAM
3. 2x4T Samsung SSD with RA!D 1 to install OS and run monitor and metadata services.
That probably suffices for a small cluster. Are those Samsungs enterprise?
OSD nodes will be upgraded to have 32 cores ( 64 threads ). Disk and RAM will remain same ( 128G and 22X8T Samsung SSD )
Which Samsung SSD? Using client SKUs for OSDs has a way of leading to heartbreak.
64 threads would be better for a 22x OSD node, though still a bit light. Are these SATA or NVMe?
Actually , I want to use OSD nodes to run OSD damons and not any other demons and which is why I am thinking of having 4 additional Dell servers as mentioned above.
Colocation of daemons is common these days, especially with smaller clusters.
Please advise if this plan will be better.
That’ll work, but unless you already have those quite-modest 4x non-OSD nodes sitting around idle you might consider just going with the OSD nodes and bumping the CPU again so you can colocate all the daemons.
Thanks, Gagan
On Wed, Apr 9, 2025 at 8:12 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
We would start deploying Ceph with 4 hosts ( HP Proliant servers ) each running RockyLinux 9.
One of the hosts called ceph-adm will be smaller one and will have following hardware :-
2x4T SSD with raid 1 to install OS on.
8 Core with 3600MHz freq.
64G RAM
We are planning to run all Ceph daemons except OSD daemon like monitor
,
metadata ,etc on this host.
8 core == 16 threads? Are you provisioning this node because you have it laying around idle?
Note that you will want *at least* 3 Monitor (monitors) daemons, which must be on different nodes. 5 is better, but at least 3. You’ll also have Grafana, Prometheus, MDS (if you’re going to CephFS vs using S3 object storage or RBD block)
8c is likely on the light side for all of that. You would also benefit from not having that node be a single point of failure. I would suggest if you can raising this node to the spec of the planned 3x OSD nodes so you have 4x equivalent nodes, and spread that non-OSD daemons across them.
Note also that your OSD nodes will also have node_exporter, crash, and other boilerplate daemons.
We will have 3 hosts to run OSD which will store actual data.
Each OSD host will have following hardware
2x4T SSD with raid 1 to install OS on.
22X8T SSD to store data ( OSDs ) ( without partition ). We will use entire disk without partitions
SAS, SATA, or NVMe SSDs? Which specific model? You really want to avoid client (desktop) models for Ceph, but you likely do not need to pay for higher endurance mixed-use SKUs.
Each OSD host will have 128G RAM ( No swap space )
Thank you for skipping swap. Some people are really stuck in the past in that regard.
Each OSD host will have 16 cores.
So 32 threads total? That is very light for 22 OSDs + other daemons. For HDD OSDs a common rule of thumb is at minimum 2x threads per, for SAS/SATA SSDs, 4, for NVMe SSDs 6. Plus margin for the OS and other processes.
All 4 hosts will connect to each via 10G nic.
Two ports with bonding? Redundant switches?
The 500T data
The specs you list above include 528 TB of *raw* space. Be advised that with three OSD nodes, you will necessarily be doing replication. For safety replication with size=3. Taking into consideration TB vs TiB and headroom, you’re looking at 133TiB of usable space. You could go with size=2 to get 300TB of usable space, but at increased risk of data unavailability or loss when drives/hosts fail or reboot.
With at least 4 OSD nodes - even if they aren’t fully populated with capacity drives — you could do EC for a more favorable raw:usable ratio, at the expense of slower writes and recovery. With 4 nodes you could in theory do 2,2 EC for 200 TiB of usable space, with 5 you could do 3,2 for 240 TiB usable, etc.
will be accessed by the clients. We need to have read performance as fast as possible.
Hope your SSDs are enterprise NVMe.
We can't afford data loss and downtime.
Then no size=2 for you.
So, we want to have a Ceph deployment which serves our purpose.
So, please advise me if the plan that I have designed will serve our purpose. Or is there a better way , please advise that.
Thanks, Gagan
We have a HP storage server with 12 SDD of 5T each and have set-up hardware RAID6 on these disks.
HP storage server has 64G RAM and 18 cores.
So, please advise how I should go about setting up Ceph on it to have best read performance. We need fastest read performance.
Thanks, Gagan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
Your choices are yours to make, but for what it’s worth, I would not use these. * They are client-class, not designed for enterprise workloads or duty cycle * Best I can tell this lacks PLP power loss protection, which can result in corrupted or lost data * QLC can be just smurfy for object storage workloads that are read-mostly, but can be disappointing for RBD or small objects/files * 3 year warranty instead of the 5 years typical for enterprise SKUs * Slow writes after the SLC cache portion fills, this is designed for desktop intermittent workload, not sustained enterprise workload. * Rated endurance for a 4KB random write workload is ~ 0.33 DWPD over the 3 year warranty period, which if divided by the enterprise 5 year warranty workload would be .20 DWPD. If you expect a low write workload and have VERY limited performance expectations, maybe they’d work for you, but especially don’t think you can safely do replication size=2 or EC 2/3+1. A few months ago someone in the community unilaterally sent me money *begging* me to make their cluster of these faster. Nothing I could do sort of recommending that they be replaced with a more appropriate SKU.
One more thing , I want to know is that CephFS supports mounting with FsCache on clients ?
I find some references on the net to people doing this, but have zero experience with it.
500T data stored in the cluster will be accessed by the jobs running on the clients nodes and we need super fast read performance.
Client-class media are incompatible with super fast anything. I don’t recall you mentioning the network — bonded 10GE at least?
For that we do have additional cache disk installed on all the clients nodes. And the way NFS V4 supports mount NFS share with FsCache on clients' hosts ,CephFS also supports that.
You would do better to invest in enterprise cluster tech than in band-aids that may or may not work well. {Good,Fast,Cheap} Pick Any Two. Trite but so often true.
On those 4x non-OSD nodes, I will probably run ldap and HTCondor service. But mds node will not be used for anything other than mds daemon.
Thanks, Gagan
On Fri, Apr 11, 2025 at 8:45 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
On Apr 11, 2025, at 4:04 AM, gagan tiwari < gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, Thanks for the reply!
We will be using CephFS to access Ceph Storage from clients. So, this will need MDS daemon also.
MDS is single-threaded, so unlike most Ceph daemons it benefits more from a high-frequency CPU than core count.
So, based on your advice, I am thinking of having 4 Dell PowerEdge servers . 3 of them will run 3 Monitor daemons and one of them will run MDS daemon.
These Dell Servers will have following hardware :-
1. 4 cores ( 8 threads ) ( Can go for 8 core and 16 threads )
2. 64G RAM
3. 2x4T Samsung SSD with RA!D 1 to install OS and run monitor and metadata services.
That probably suffices for a small cluster. Are those Samsungs enterprise?
OSD nodes will be upgraded to have 32 cores ( 64 threads ). Disk and RAM will remain same ( 128G and 22X8T Samsung SSD )
Which Samsung SSD? Using client SKUs for OSDs has a way of leading to heartbreak.
64 threads would be better for a 22x OSD node, though still a bit light. Are these SATA or NVMe?
Actually , I want to use OSD nodes to run OSD damons and not any other demons and which is why I am thinking of having 4 additional Dell servers as mentioned above.
Colocation of daemons is common these days, especially with smaller clusters.
Please advise if this plan will be better.
That’ll work, but unless you already have those quite-modest 4x non-OSD nodes sitting around idle you might consider just going with the OSD nodes and bumping the CPU again so you can colocate all the daemons.
Thanks, Gagan
On Wed, Apr 9, 2025 at 8:12 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
We would start deploying Ceph with 4 hosts ( HP Proliant servers ) each running RockyLinux 9.
One of the hosts called ceph-adm will be smaller one and will have following hardware :-
2x4T SSD with raid 1 to install OS on.
8 Core with 3600MHz freq.
64G RAM
We are planning to run all Ceph daemons except OSD daemon like monitor
,
metadata ,etc on this host.
8 core == 16 threads? Are you provisioning this node because you have it laying around idle?
Note that you will want *at least* 3 Monitor (monitors) daemons, which must be on different nodes. 5 is better, but at least 3. You’ll also have Grafana, Prometheus, MDS (if you’re going to CephFS vs using S3 object storage or RBD block)
8c is likely on the light side for all of that. You would also benefit from not having that node be a single point of failure. I would suggest if you can raising this node to the spec of the planned 3x OSD nodes so you have 4x equivalent nodes, and spread that non-OSD daemons across them.
Note also that your OSD nodes will also have node_exporter, crash, and other boilerplate daemons.
We will have 3 hosts to run OSD which will store actual data.
Each OSD host will have following hardware
2x4T SSD with raid 1 to install OS on.
22X8T SSD to store data ( OSDs ) ( without partition ). We will use entire disk without partitions
SAS, SATA, or NVMe SSDs? Which specific model? You really want to avoid client (desktop) models for Ceph, but you likely do not need to pay for higher endurance mixed-use SKUs.
Each OSD host will have 128G RAM ( No swap space )
Thank you for skipping swap. Some people are really stuck in the past in that regard.
Each OSD host will have 16 cores.
So 32 threads total? That is very light for 22 OSDs + other daemons. For HDD OSDs a common rule of thumb is at minimum 2x threads per, for SAS/SATA SSDs, 4, for NVMe SSDs 6. Plus margin for the OS and other processes.
All 4 hosts will connect to each via 10G nic.
Two ports with bonding? Redundant switches?
The 500T data
The specs you list above include 528 TB of *raw* space. Be advised that with three OSD nodes, you will necessarily be doing replication. For safety replication with size=3. Taking into consideration TB vs TiB and headroom, you’re looking at 133TiB of usable space. You could go with size=2 to get 300TB of usable space, but at increased risk of data unavailability or loss when drives/hosts fail or reboot.
With at least 4 OSD nodes - even if they aren’t fully populated with capacity drives — you could do EC for a more favorable raw:usable ratio, at the expense of slower writes and recovery. With 4 nodes you could in theory do 2,2 EC for 200 TiB of usable space, with 5 you could do 3,2 for 240 TiB usable, etc.
will be accessed by the clients. We need to have read performance as fast as possible.
Hope your SSDs are enterprise NVMe.
We can't afford data loss and downtime.
Then no size=2 for you.
So, we want to have a Ceph deployment which serves our purpose.
So, please advise me if the plan that I have designed will serve our purpose. Or is there a better way , please advise that.
Thanks, Gagan
We have a HP storage server with 12 SDD of 5T each and have set-up hardware RAID6 on these disks.
HP storage server has 64G RAM and 18 cores.
So, please advise how I should go about setting up Ceph on it to have best read performance. We need fastest read performance.
Thanks, Gagan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Apr 11, 2025, at 10:13, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway). Each step on the journey to enterprise SSDs made things faster. The problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant. So from my experience, I would say ceph is very slow in general compared to DAS. You need all the help you can get. If you want to use the consumer stuff, I would recommend to make a slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster. Good luck! _BB
On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <brendon@netcal.com> wrote:
On Apr 11, 2025, at 10:13, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway).
Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference. Other factors include: * Enough hosts and OSDs. Three hosts with one OSD each aren’t going to deliver a great experience * At least 6GB of available physmem per NVMe OSD * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload.
Each step on the journey to enterprise SSDs made things faster. The problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant.
Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads.
So from my experience, I would say ceph is very slow in general compared to DAS. You need all the help you can get.
If you want to use the consumer stuff, I would recommend to make a slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster.
How much drives in each?
Good luck!
_BB
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
MDS (if you’re going to CephFS vs using S3 object storage or RBD block) Hi Anthony,
Can you elaborate on this remark? Should one choose between using CephFS vs S3 Storage (as it pertains to best practices)? On Proxmox, I am. using both CephFS and RBD. Regards, [image] Anthony Fecarotta Founder & President [image] anthony@linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/> On Sun Apr 13, 2025, 04:28 PM GMT, Anthony D'Atri <mailto:anthony.datri@gmail.com> wrote:
On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <brendon@netcal.com> wrote:
On Apr 11, 2025, at 10:13, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway).
Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference.
Other factors include:
* Enough hosts and OSDs. Three hosts with one OSD each aren’t going to deliver a great experience * At least 6GB of available physmem per NVMe OSD * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload.
Each step on the journey to enterprise SSDs made things faster. The problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant.
Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads.
So from my experience, I would say ceph is very slow in general compared to DAS. You need all the help you can get.
If you want to use the consumer stuff, I would recommend to make a slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster.
How much drives in each?
Good luck!
_BB
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It’s a function of your use-case.
On Apr 14, 2025, at 8:41 AM, Anthony Fecarotta <anthony@linehaul.ai> wrote:
MDS (if you’re going to CephFS vs using S3 object storage or RBD block) Hi Anthony,
Can you elaborate on this remark?
Should one choose between using CephFS vs S3 Storage (as it pertains to best practices)?
On Proxmox, I am. using both CephFS and RBD.
Regards, [image] Anthony Fecarotta Founder & President [image] anthony@linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>
On Sun Apr 13, 2025, 04:28 PM GMT, Anthony D'Atri <mailto:anthony.datri@gmail.com> wrote:
On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <brendon@netcal.com> wrote:
On Apr 11, 2025, at 10:13, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway).
Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference.
Other factors include:
* Enough hosts and OSDs. Three hosts with one OSD each aren’t going to deliver a great experience * At least 6GB of available physmem per NVMe OSD * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload.
Each step on the journey to enterprise SSDs made things faster. The problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant.
Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads.
So from my experience, I would say ceph is very slow in general compared to DAS. You need all the help you can get.
If you want to use the consumer stuff, I would recommend to make a slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster.
How much drives in each?
Good luck!
_BB
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
HI Anthony, Based on your inputs and further digging into Ceph documentation, I am now thinking to go for 6 OSD nodes to have k=4 and m=2 EC set-up. As I mentioned., we need maximum usable space and we are more concerned about data safety and best read performance from the cluster. Writes operation will be done on a separate storage solution via NFS. So, with each OSD node having 22X4T Enterprise SSD , we will have 88X6 = 528T Raw Space. With 4X2 EC , it will hopefully provide us with 390T usable space. So, that will be enough for us to start with. So, I need to know what will be data safely level with the above set-up ( i.e. 6 OSDs with 4X2 EC ). How many OSDs ( disks ) and nodes failure , above set-up can withstand. Also, if , later, we need to add more OSD modes to get more usable space, will we need to add same size disks ( 4T ) or can we add nodes with bigger size disks ( 8T or 15T ) ? Beside OSDs server , going to have three Dell servers with 8 Core and 64G RAM to run 3 monitor daemons one on each server. One 4 core and 64G RAM with high core freq ( 4800 MHz ) server to run MDS daemon. Please advise Thanks, Gagan On Tue, Apr 15, 2025 at 8:14 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
It’s a function of your use-case.
On Apr 14, 2025, at 8:41 AM, Anthony Fecarotta <anthony@linehaul.ai> wrote:
MDS (if you’re going to CephFS vs using S3 object storage or RBD block) Hi Anthony,
Can you elaborate on this remark?
Should one choose between using CephFS vs S3 Storage (as it pertains to best practices)?
On Proxmox, I am. using both CephFS and RBD.
Regards, [image] Anthony Fecarotta Founder & President [image] anthony@linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>
On Sun Apr 13, 2025, 04:28 PM GMT, Anthony D'Atri <mailto: anthony.datri@gmail.com> wrote:
On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <brendon@netcal.com>
On Apr 11, 2025, at 10:13, gagan tiwari <
gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway).
Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can
wrote: lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference.
Other factors include:
* Enough hosts and OSDs. Three hosts with one OSD each aren’t going to
* At least 6GB of available physmem per NVMe OSD * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload.
Each step on the journey to enterprise SSDs made things faster. The
deliver a great experience problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant.
Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM
per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads.
So from my experience, I would say ceph is very slow in general
compared to DAS. You need all the help you can get.
If you want to use the consumer stuff, I would recommend to make a
slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster.
How much drives in each?
Good luck!
_BB
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Apr 21, 2025, at 6:54 AM, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:
HI Anthony, Based on your inputs and further digging into Ceph documentation, I am now thinking to go for 6 OSD nodes to have k=4 and m=2 EC set-up.
Be aware that with that architecture when you lose one drive, the cluster’s capacity will decrease by that drive’s capacity until it is restored.
As I mentioned., we need maximum usable space and we are more concerned about data safety and best read performance from the cluster. Writes operation will be done on a separate storage solution via NFS.
Different data sets? Almost sounds like a task for Aerospike.
So, with each OSD node having 22X4T Enterprise SSD
No QVOs?
we will have 88X6 = 528T Raw Space. With 4X2 EC , it will hopefully provide us with 390T usable space. So, that will be enough for us to start with.
6TB sounds like mixed-use 3DWPD SSDs? If so, those are almost certainly overkill. You’ll be fine with read-intensive SSDs which would be 7.6TB. Remember the below when planning usable space: * Storage vendors use base-10 units (TB) while humans mostly use base-2 units (TiB). So 528 TB = 480 TiB * Ceph has nearfull, backfillfull, and full ratios. The default nearfull ratio is 85%, so you will get a warning state at roughly 408TiB stored, OSDs will no longer accept backfill at roughly 432TiB stored, and will no longer accept writes at 456TiB stored. * With CephFS files smaller than, say, 128KB will currently waste a noticeable fraction of raw capacity. How large are your files?
So, I need to know what will be data safely level with the above set-up ( i.e. 6 OSDs with 4X2 EC ). How many OSDs ( disks ) and nodes failure , above set-up can withstand.
With the above topology, you can sustain one OSD failure at a time without losing data availability. You can sustain two overlapping OSD failures without losing data, but it will become unavailable until replication is restored. You can sustain one node being down and data will still be available. You can sustain two nodes being down without data loss.
Also, if , later, we need to add more OSD modes to get more usable space, will we need to add same size disks ( 4T ) or can we add nodes with bigger size disks ( 8T or 15T ) ?
Above you wrote 6T but here you write 4T, which is it? Note that a read-intensive enterprise SSD will be 3.84 TB which means 3.5 TiB. You can mix OSD drive sizes, but be aware that with a 4,2 EC profile for your bulk data you will absolutely want to add them evenly across nodes. You will want every node to have the same total capacity, otherwise some capacity may not be usable, because every node will need to place one shard of that bulk EC data. ceph config set global mon_max_pg_per_osd 1000 ^ this will help avoid certain problem scenarios when mixing drive capacities.
Beside OSDs server , going to have three Dell servers with 8 Core and 64G RAM to run 3 monitor daemons one on each server.
Ok. Better yet would be also run 2 mons on the OSD servers as well.
One 4 core and 64G RAM with high core freq ( 4800 MHz ) server to run MDS daemon.
Please advise
Thanks, Gagan
On Tue, Apr 15, 2025 at 8:14 PM Anthony D'Atri <anthony.datri@gmail.com <mailto:anthony.datri@gmail.com>> wrote:
It’s a function of your use-case.
On Apr 14, 2025, at 8:41 AM, Anthony Fecarotta <anthony@linehaul.ai <mailto:anthony@linehaul.ai>> wrote:
MDS (if you’re going to CephFS vs using S3 object storage or RBD block) Hi Anthony,
Can you elaborate on this remark?
Should one choose between using CephFS vs S3 Storage (as it pertains to best practices)?
On Proxmox, I am. using both CephFS and RBD.
Regards, [image] Anthony Fecarotta Founder & President [image] anthony@linehaul.ai <mailto:anthony@linehaul.ai> <mailto:anthony@linehaul.ai <mailto:anthony@linehaul.ai>> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>
On Sun Apr 13, 2025, 04:28 PM GMT, Anthony D'Atri <mailto:anthony.datri@gmail.com <mailto:anthony.datri@gmail.com>> wrote:
On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <brendon@netcal.com <mailto:brendon@netcal.com>> wrote:
On Apr 11, 2025, at 10:13, gagan tiwari <gagan.tiwari@mathisys-india.com <mailto:gagan.tiwari@mathisys-india.com>> wrote:
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway).
Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference.
Other factors include:
* Enough hosts and OSDs. Three hosts with one OSD each aren’t going to deliver a great experience * At least 6GB of available physmem per NVMe OSD * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload.
Each step on the journey to enterprise SSDs made things faster. The problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant.
Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads.
So from my experience, I would say ceph is very slow in general compared to DAS. You need all the help you can get.
If you want to use the consumer stuff, I would recommend to make a slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster.
How much drives in each?
Good luck!
_BB
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Sorry that was typo I meant 4T SSD not 6T On Mon, 21 Apr, 2025, 5:18 pm Anthony D'Atri, <anthony.datri@gmail.com> wrote:
On Apr 21, 2025, at 6:54 AM, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:
HI Anthony, Based on your inputs and further digging into Ceph documentation, I am now thinking to go for 6 OSD nodes to have k=4 and m=2 EC set-up.
Be aware that with that architecture when you lose one drive, the cluster’s capacity will decrease by that drive’s capacity until it is restored.
As I mentioned., we need maximum usable space and we are more concerned about data safety and best read performance from the cluster. Writes operation will be done on a separate storage solution via NFS.
Different data sets? Almost sounds like a task for Aerospike.
So, with each OSD node having 22X4T Enterprise SSD
No QVOs?
we will have 88X6 = 528T Raw Space. With 4X2 EC , it will hopefully provide us with 390T usable space. So, that will be enough for us to start with.
6TB sounds like mixed-use 3DWPD SSDs? If so, those are almost certainly overkill. You’ll be fine with read-intensive SSDs which would be 7.6TB.
Remember the below when planning usable space:
* Storage vendors use base-10 units (TB) while humans mostly use base-2 units (TiB). So 528 TB = 480 TiB * Ceph has nearfull, backfillfull, and full ratios. The default nearfull ratio is 85%, so you will get a warning state at roughly 408TiB stored, OSDs will no longer accept backfill at roughly 432TiB stored, and will no longer accept writes at 456TiB stored. * With CephFS files smaller than, say, 128KB will currently waste a noticeable fraction of raw capacity. How large are your files?
So, I need to know what will be data safely level with the above set-up ( i.e. 6 OSDs with 4X2 EC ). How many OSDs ( disks ) and nodes failure , above set-up can withstand.
With the above topology, you can sustain one OSD failure at a time without losing data availability. You can sustain two overlapping OSD failures without losing data, but it will become unavailable until replication is restored.
You can sustain one node being down and data will still be available. You can sustain two nodes being down without data loss.
Also, if , later, we need to add more OSD modes to get more usable space, will we need to add same size disks ( 4T ) or can we add nodes with bigger size disks ( 8T or 15T ) ?
Above you wrote 6T but here you write 4T, which is it? Note that a read-intensive enterprise SSD will be 3.84 TB which means 3.5 TiB.
You can mix OSD drive sizes, but be aware that with a 4,2 EC profile for your bulk data you will absolutely want to add them evenly across nodes. You will want every node to have the same total capacity, otherwise some capacity may not be usable, because every node will need to place one shard of that bulk EC data.
ceph config set global mon_max_pg_per_osd 1000
^ this will help avoid certain problem scenarios when mixing drive capacities.
Beside OSDs server , going to have three Dell servers with 8 Core and 64G RAM to run 3 monitor daemons one on each server.
Ok. Better yet would be also run 2 mons on the OSD servers as well.
One 4 core and 64G RAM with high core freq ( 4800 MHz ) server to run MDS daemon.
Please advise
Thanks, Gagan
On Tue, Apr 15, 2025 at 8:14 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
It’s a function of your use-case.
On Apr 14, 2025, at 8:41 AM, Anthony Fecarotta <anthony@linehaul.ai> wrote:
MDS (if you’re going to CephFS vs using S3 object storage or RBD block) Hi Anthony,
Can you elaborate on this remark?
Should one choose between using CephFS vs S3 Storage (as it pertains to best practices)?
On Proxmox, I am. using both CephFS and RBD.
Regards, [image] Anthony Fecarotta Founder & President [image] anthony@linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 <https://www.google.com/maps/search/1+Mid+America+Plz+Flr+3+Oakbrook+Terrace,+IL+60181?entry=gmail&source=g> [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>
On Sun Apr 13, 2025, 04:28 PM GMT, Anthony D'Atri <mailto: anthony.datri@gmail.com> wrote:
On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <brendon@netcal.com>
On Apr 11, 2025, at 10:13, gagan tiwari <
gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway).
Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can
wrote: lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference.
Other factors include:
* Enough hosts and OSDs. Three hosts with one OSD each aren’t going to
* At least 6GB of available physmem per NVMe OSD * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload.
Each step on the journey to enterprise SSDs made things faster. The
deliver a great experience problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant.
Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM
per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads.
So from my experience, I would say ceph is very slow in general
compared to DAS. You need all the help you can get.
If you want to use the consumer stuff, I would recommend to make a
slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster.
How much drives in each?
Good luck!
_BB
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Files are not of smaller size. Sizes are in GB and MB. Few files will be 2.5 K. Thanks, Gagan On Mon, Apr 21, 2025 at 8:13 PM gagan tiwari < gagan.tiwari@mathisys-india.com> wrote:
Sorry that was typo I meant 4T SSD not 6T
On Mon, 21 Apr, 2025, 5:18 pm Anthony D'Atri, <anthony.datri@gmail.com> wrote:
On Apr 21, 2025, at 6:54 AM, gagan tiwari < gagan.tiwari@mathisys-india.com> wrote:
HI Anthony, Based on your inputs and further digging into Ceph documentation, I am now thinking to go for 6 OSD nodes to have k=4 and m=2 EC set-up.
Be aware that with that architecture when you lose one drive, the cluster’s capacity will decrease by that drive’s capacity until it is restored.
As I mentioned., we need maximum usable space and we are more concerned about data safety and best read performance from the cluster. Writes operation will be done on a separate storage solution via NFS.
Different data sets? Almost sounds like a task for Aerospike.
So, with each OSD node having 22X4T Enterprise SSD
No QVOs?
we will have 88X6 = 528T Raw Space. With 4X2 EC , it will hopefully provide us with 390T usable space. So, that will be enough for us to start with.
6TB sounds like mixed-use 3DWPD SSDs? If so, those are almost certainly overkill. You’ll be fine with read-intensive SSDs which would be 7.6TB.
Remember the below when planning usable space:
* Storage vendors use base-10 units (TB) while humans mostly use base-2 units (TiB). So 528 TB = 480 TiB * Ceph has nearfull, backfillfull, and full ratios. The default nearfull ratio is 85%, so you will get a warning state at roughly 408TiB stored, OSDs will no longer accept backfill at roughly 432TiB stored, and will no longer accept writes at 456TiB stored. * With CephFS files smaller than, say, 128KB will currently waste a noticeable fraction of raw capacity. How large are your files?
So, I need to know what will be data safely level with the above set-up ( i.e. 6 OSDs with 4X2 EC ). How many OSDs ( disks ) and nodes failure , above set-up can withstand.
With the above topology, you can sustain one OSD failure at a time without losing data availability. You can sustain two overlapping OSD failures without losing data, but it will become unavailable until replication is restored.
You can sustain one node being down and data will still be available. You can sustain two nodes being down without data loss.
Also, if , later, we need to add more OSD modes to get more usable space, will we need to add same size disks ( 4T ) or can we add nodes with bigger size disks ( 8T or 15T ) ?
Above you wrote 6T but here you write 4T, which is it? Note that a read-intensive enterprise SSD will be 3.84 TB which means 3.5 TiB.
You can mix OSD drive sizes, but be aware that with a 4,2 EC profile for your bulk data you will absolutely want to add them evenly across nodes. You will want every node to have the same total capacity, otherwise some capacity may not be usable, because every node will need to place one shard of that bulk EC data.
ceph config set global mon_max_pg_per_osd 1000
^ this will help avoid certain problem scenarios when mixing drive capacities.
Beside OSDs server , going to have three Dell servers with 8 Core and 64G RAM to run 3 monitor daemons one on each server.
Ok. Better yet would be also run 2 mons on the OSD servers as well.
One 4 core and 64G RAM with high core freq ( 4800 MHz ) server to run MDS daemon.
Please advise
Thanks, Gagan
On Tue, Apr 15, 2025 at 8:14 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
It’s a function of your use-case.
On Apr 14, 2025, at 8:41 AM, Anthony Fecarotta <anthony@linehaul.ai> wrote:
MDS (if you’re going to CephFS vs using S3 object storage or RBD block) Hi Anthony,
Can you elaborate on this remark?
Should one choose between using CephFS vs S3 Storage (as it pertains to best practices)?
On Proxmox, I am. using both CephFS and RBD.
Regards, [image] Anthony Fecarotta Founder & President [image] anthony@linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 <https://www.google.com/maps/search/1+Mid+America+Plz+Flr+3+Oakbrook+Terrace,+IL+60181?entry=gmail&source=g> [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>
On Sun Apr 13, 2025, 04:28 PM GMT, Anthony D'Atri <mailto: anthony.datri@gmail.com> wrote:
On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <
> On Apr 11, 2025, at 10:13, gagan tiwari <
gagan.tiwari@mathisys-india.com> wrote:
> > Hi Anthony, > We will be using Samsung SSD 870 QVO 8TB disks on > all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway).
Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can
brendon@netcal.com> wrote: lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference.
Other factors include:
* Enough hosts and OSDs. Three hosts with one OSD each aren’t going
* At least 6GB of available physmem per NVMe OSD * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload.
Each step on the journey to enterprise SSDs made things faster. The
to deliver a great experience problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant.
Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM
per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads.
So from my experience, I would say ceph is very slow in general
compared to DAS. You need all the help you can get.
If you want to use the consumer stuff, I would recommend to make a
slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster.
How much drives in each?
Good luck!
_BB
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So, I need to know what will be data safely level with the above set-up ( i.e. 6 OSDs with 4X2 EC ). How many OSDs ( disks ) and nodes failure , above set-up can withstand.
With EC N+2 you can lose one drive or host, and the cluster will go on with degraded mode until it has been able to recreate the missing data on another OSD, if you lose two drives or hosts, I believe the EC pool with go readonly, again until it has rebuilt copies elsewhere. Still, if you have EC 4+2 and only 6 OSD hosts, this means if a host dies, the cluster can not recreate data anywhere without violating "one copy per host" default placement, so the cluster will be degraded until this host comes back or another one replaces it. For a N+M EC cluster, I would suggest having N+M+1 or even +2 number of hosts, so that you can do maintenance on a host or lose a host and still be able to recover without visiting the server room. -- May the most significant bit of your life be positive.
Hi Janne, Thanks for your advice. So, you mean with with K=4 M =2 EC, we need 8 OSD nodes to have better protection Thanks, Gagan On Tue, 22 Apr, 2025, 7:22 pm Janne Johansson, <icepic.dz@gmail.com> wrote:
So, I need to know what will be data safely level with the above set-up ( i.e. 6 OSDs with 4X2 EC ). How many OSDs ( disks ) and nodes failure , above set-up can withstand.
With EC N+2 you can lose one drive or host, and the cluster will go on with degraded mode until it has been able to recreate the missing data on another OSD, if you lose two drives or hosts, I believe the EC pool with go readonly, again until it has rebuilt copies elsewhere.
Still, if you have EC 4+2 and only 6 OSD hosts, this means if a host dies, the cluster can not recreate data anywhere without violating "one copy per host" default placement, so the cluster will be degraded until this host comes back or another one replaces it. For a N+M EC cluster, I would suggest having N+M+1 or even +2 number of hosts, so that you can do maintenance on a host or lose a host and still be able to recover without visiting the server room.
-- May the most significant bit of your life be positive.
It’s the same protection, really, just a matter of flexibility. With 4+2 EC, 7+ hosts are ideal for multiple reasons. You would likely be fine with 6 hosts, so long as you have the ability to quickly repair a host if/when it fails.
On Apr 22, 2025, at 12:03 PM, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:
Hi Janne, Thanks for your advice.
So, you mean with with K=4 M =2 EC, we need 8 OSD nodes to have better protection
Thanks, Gagan
On Tue, 22 Apr, 2025, 7:22 pm Janne Johansson, <icepic.dz@gmail.com> wrote:
So, I need to know what will be data safely level with the above set-up ( i.e. 6 OSDs with 4X2 EC ). How many OSDs ( disks ) and nodes failure , above set-up can withstand.
With EC N+2 you can lose one drive or host, and the cluster will go on with degraded mode until it has been able to recreate the missing data on another OSD, if you lose two drives or hosts, I believe the EC pool with go readonly, again until it has rebuilt copies elsewhere.
Still, if you have EC 4+2 and only 6 OSD hosts, this means if a host dies, the cluster can not recreate data anywhere without violating "one copy per host" default placement, so the cluster will be degraded until this host comes back or another one replaces it. For a N+M EC cluster, I would suggest having N+M+1 or even +2 number of hosts, so that you can do maintenance on a host or lose a host and still be able to recover without visiting the server room.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We can go with 8 OSD nodes. We just can't afford cluster not available events. Also, since we will be using CephFS to mount storage data on client nodes, so having 2 dedicated servers with mds daemon running on each will provide more protection. Correct? Thanks On Tue, 22 Apr, 2025, 10:16 pm Anthony D'Atri, <anthony.datri@gmail.com> wrote:
It’s the same protection, really, just a matter of flexibility.
With 4+2 EC, 7+ hosts are ideal for multiple reasons. You would likely be fine with 6 hosts, so long as you have the ability to quickly repair a host if/when it fails.
On Apr 22, 2025, at 12:03 PM, gagan tiwari < gagan.tiwari@mathisys-india.com> wrote:
Hi Janne, Thanks for your advice.
So, you mean with with K=4 M =2 EC, we need 8 OSD nodes to have better protection
Thanks, Gagan
On Tue, 22 Apr, 2025, 7:22 pm Janne Johansson, <icepic.dz@gmail.com> wrote:
So, I need to know what will be data safely level with the above set-up ( i.e. 6 OSDs with 4X2 EC ). How many OSDs ( disks ) and nodes failure , above set-up can withstand.
With EC N+2 you can lose one drive or host, and the cluster will go on with degraded mode until it has been able to recreate the missing data on another OSD, if you lose two drives or hosts, I believe the EC pool with go readonly, again until it has rebuilt copies elsewhere.
Still, if you have EC 4+2 and only 6 OSD hosts, this means if a host dies, the cluster can not recreate data anywhere without violating "one copy per host" default placement, so the cluster will be degraded until this host comes back or another one replaces it. For a N+M EC cluster, I would suggest having N+M+1 or even +2 number of hosts, so that you can do maintenance on a host or lose a host and still be able to recover without visiting the server room.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Janne, Thanks for your advice.
So, you mean with with K=4 M =2 EC, we need 8 OSD nodes to have better protection
As always, it is a tradeoff in cost, speed, availability, storage size and so on. What you need if the data is important is for the cluster to be able to heal itself. If you run X hosts in a Replication=X setup, OR, N+M hosts in an EC N+M setup is something that works fine when everything is fine. Unfortunately, drives die, OSes crash, hosts get failed PSUs and sometimes, just a simple maintenance thing goes slightly wrong and downtime becomes far longer than expected. In any such cases where one host is out for a long time, a cluster that has exactly minimum number of hosts will not be able to repair itself when a host is missing, because it is already at the minimum and you are now one step under the minimum. This means that from being fully functional you are one crash away from being degraded and at risk of data loss or at least a cluster that goes readonly to protect the data in case of any new unexpected surprises. If you have N+M+1 hosts or more, the cluster can recover into one of the "excess" hosts drives and at some point after, become fully functional again without your intervention. Also one thing to consider is that you should never fill a cluster to 85% or more and this will need to take crashes into account aswell so if you have EC 4+2 and 7 hosts, if they are 83% full and one OSD host goes down, the cluster will still not be able to make extra copies on the remaining 6 OSD hosts so that it goes over 85% full, so not only should you have more hosts than the EC N+M says, you should also have spare drive capacity and expand early to avoid getting into a situation where you can't repair due to almost-full drives everywhere. This is easy to see if you compare the impact of losing one OSD host when you have 6, in this case you have 16.6% of the total data needing to spread out over the remaining 5 hosts, which will be a noticeable amount. If you had a 100 OSD hosts and one crashes, you have 1% of the total data needing to be spread out over the remaining 99, and even if that is the same amount of space to be rewritten/recreated, the extra data for each hosts becomes very small. You can skip going to the datacenter, let it recover by itself, and if another host dies the week after, its still ~1.1% to be spread out over 98 hosts, again something that is very much manageable without panicking. If you have EC4+2 on 6 hosts and one dies in the middle of the night, it's time to get in the car as soon as possible.
Still, if you have EC 4+2 and only 6 OSD hosts, this means if a host dies, the cluster can not recreate data anywhere without violating "one copy per host" default placement, so the cluster will be degraded until this host comes back or another one replaces it. For a N+M EC cluster, I would suggest having N+M+1 or even +2 number of hosts, so that you can do maintenance on a host or lose a host and still be able to recover without visiting the server room.
-- May the most significant bit of your life be positive.
Hi Janne / Anthony, Thanks for the great explanation. I am going ahead with 4X2 EC with 8 OSD nodes with the hope that it will provide max protection and performance. Will have 3 more servers to run Monitor / Manager and MDS daemon on each of them. As advised by Anthony will run 2 more monitor daemons on OSD nodes to have 5 monitor daemons running. Since I am newbie to Ceph, can you guys please share doc / commands on how to set up 4X2 EC. Thanks, Gagan On Wed, Apr 23, 2025 at 12:09 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Hi Janne, Thanks for your advice.
So, you mean with with K=4 M =2 EC, we need 8 OSD nodes to have better protection
As always, it is a tradeoff in cost, speed, availability, storage size and so on. What you need if the data is important is for the cluster to be able to heal itself.
If you run X hosts in a Replication=X setup, OR, N+M hosts in an EC N+M setup is something that works fine when everything is fine. Unfortunately, drives die, OSes crash, hosts get failed PSUs and sometimes, just a simple maintenance thing goes slightly wrong and downtime becomes far longer than expected. In any such cases where one host is out for a long time, a cluster that has exactly minimum number of hosts will not be able to repair itself when a host is missing, because it is already at the minimum and you are now one step under the minimum.
This means that from being fully functional you are one crash away from being degraded and at risk of data loss or at least a cluster that goes readonly to protect the data in case of any new unexpected surprises.
If you have N+M+1 hosts or more, the cluster can recover into one of the "excess" hosts drives and at some point after, become fully functional again without your intervention. Also one thing to consider is that you should never fill a cluster to 85% or more and this will need to take crashes into account aswell so if you have EC 4+2 and 7 hosts, if they are 83% full and one OSD host goes down, the cluster will still not be able to make extra copies on the remaining 6 OSD hosts so that it goes over 85% full, so not only should you have more hosts than the EC N+M says, you should also have spare drive capacity and expand early to avoid getting into a situation where you can't repair due to almost-full drives everywhere. This is easy to see if you compare the impact of losing one OSD host when you have 6, in this case you have 16.6% of the total data needing to spread out over the remaining 5 hosts, which will be a noticeable amount. If you had a 100 OSD hosts and one crashes, you have 1% of the total data needing to be spread out over the remaining 99, and even if that is the same amount of space to be rewritten/recreated, the extra data for each hosts becomes very small. You can skip going to the datacenter, let it recover by itself, and if another host dies the week after, its still ~1.1% to be spread out over 98 hosts, again something that is very much manageable without panicking. If you have EC4+2 on 6 hosts and one dies in the middle of the night, it's time to get in the car as soon as possible.
Still, if you have EC 4+2 and only 6 OSD hosts, this means if a host dies, the cluster can not recreate data anywhere without violating "one copy per host" default placement, so the cluster will be degraded until this host comes back or another one replaces it. For a N+M EC cluster, I would suggest having N+M+1 or even +2 number of hosts, so that you can do maintenance on a host or lose a host and still be able to recover without visiting the server room.
-- May the most significant bit of your life be positive.
I haven't had the need for capacity or speed that many ceph users do, but I AM insistent on reliability, and ceph has never failed me on that point even when I've made a wreck of my hardware and/or configuration. I don't think that it was explicitly stated, but I'm pretty sure that Ceph doesn't (at least any more) have a master command node and that essentially any ceph host can be used for control, although I do keep one host as my favorite place to login to. I know of no reason at all to have asymmetric hardware for such purposes, unless that's simply using what you have lying around or you're setting up some non-OSD nodes. so you don't need storage on them. To maybe clarify what Anthony wrote, vitually no Ceph monitors are expected to be single-instance. They're either co-operative or quorum- elected. Even access points such as NFS and RGW benefit from redundancies. Hope that helps, Tim On Sun, 2025-04-13 at 12:28 -0400, Anthony D'Atri wrote:
On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <brendon@netcal.com> wrote:
On Apr 11, 2025, at 10:13, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:
Hi Anthony, We will be using Samsung SSD 870 QVO 8TB disks on all OSD servers.
I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway).
Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference.
Other factors include:
* Enough hosts and OSDs. Three hosts with one OSD each aren’t going to deliver a great experience * At least 6GB of available physmem per NVMe OSD * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload.
Each step on the journey to enterprise SSDs made things faster. The problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant.
Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads.
So from my experience, I would say ceph is very slow in general compared to DAS. You need all the help you can get.
If you want to use the consumer stuff, I would recommend to make a slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster.
How much drives in each?
Good luck!
_BB
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Anthony D'Atri
-
Anthony Fecarotta
-
Brendon Baumgartner
-
Dominique Ramaekers
-
gagan tiwari
-
Janne Johansson
-
Tim Holloway