28 Apr
2023
28 Apr
'23
2:35 a.m.
Hey guys and girls,
I'm working on a project to build storage for one of our departments,
and I want to ask you guys and girls for input on the high-level
overview part. It's a long one, I hope you read along and comment.
SUMMARY
I made a plan last year to build a 'storage solution' including ceph
and some windows VM's to expose the data over SMB to clients. A year
later I finally have the hardware, built a ceph cluster, and I'm doing
tests. Ceph itself runs great, but when I wanted to start exposing the
data using iscsi to our VMware farm, I ran into some issues. I know
the iscsi gateways will introduce some new performance bottlenecks,
but I'm seeing really slow performance, still working on that.
But then I ran into the warning on the iscsi gateway page: "The iSCSI
gateway is in maintenance as of November 2022. This means that it is
no longer in active development and will not be updated to add new
features.". Wait, what? Why!? What does this mean? Does this mean that
iSCSI is now 'feature complete' and will still be supported the next 5
years, or will it be deprecated in the future? I tried searching, but
couldn't find any info on the decision and the roadmap.
My goal is to build a future-proof setup, and using deprecated
components should not be part of that of course.
If the iscsi gateway will still be supported the next few years and I
can iron out the performance issues, I can still go on with my
original plan. If not, I have to go back to the drawing board. And
maybe you guys would advise me to take another route anyway.
GOALS
My goals/considerations are:
- we want >1PB of storage capacity for cheap (on a tight budget) for
research data. Most of it is 'store once, read sometimes'. <1% of the
data is 'hot'.
- focus is on capacity, but it would be nice to have > 200MB/s of
sequential write/read performance and not 'totally suck' on random
i/o. Yes, not very well quantified, but ah. Sequential writes are most
important.
- end users all run Windows computers (mostly VDI's) and a lot of
applications require SMB shares.
- security is a big thing, we want really tight ACL's, specific
monitoring agents, etc.
- our data is incredibly important to us, we still want the 3-2-1
backup rule. Primary storage solution, a second storage solution in a
different place, and some of the data that is not reproducible is also
written to tape. We also want to be protected from ransomware or user
errors (so no direct replication to the second storage).
- I like open source, reliability, no fork-lift upgrades, no vendor
lock-in, blah, well, I'm on the ceph list here, no need to convince
you guys ;)
- We're hiring a commercial company to do ceph maintenance and support
for when I'm on leave or leaving the company, but they won't support
clients, backup software, etc, so I want something as simple as
possible. We do have multiple Windows/VMware admins, but no other real
linux guru's.
THE INITIAL PLAN
Given these considerations, I ordered two identical clusters, each
consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2
ssd's and 10 capacity disks (EC 4:2 for the data), and each node is
connected using a 2x25Gbps bond. Ceph is running like a charm. Now I
just have to think about exposing the data to end users, and I've been
testing different setups.
My original plan was to expose for example 10x100TB rbd images using
iSCSI to our VMware farm, formatting the luns with VMFS6, and run for
example 2 Windows file servers per datastore on that with a single DFS
namespace to end users. Then backup the file servers using our
existing Veeam infrastructure to RGW running on the second cluster
with an immutable bucket. This way we would have easily defined
security boundaries: the clients can only reach the file servers, the
file servers only see their local VMDK's, ESX only sees the luns on
the iSCSI target, etc. When a file server would be compromised, it
would have no access to ceph. We have easy incremental backups,
immutability for ransomware protection, etc. And the best part is that
the ceph admin can worry about ceph, the vmware admin can focus on
ESX, VMFS and all the vmware stuff, and the Windows admins can focus
on the Windows boxes, Windows-specific ACLS and tools and Veeam
backups and stuff.
CURRENT SITUATION
I'm building out this plan now, but I'm running into issues with
iSCSI. Are any of you doing something similar? What is your iscsi
performance compared to direct rbd?
In regard to performance: If I take 2 test windows VM's, I put one on
an iSCSI datastore and another with direct rbd access using the
windows rbd driver, I create a share on those boxes and push data to
it, I see different results (of course). Copying some iso images over
SMB to the 'windows vm running direct rbd' I see around 800MB/s write,
and 200MB/s read, which is pretty okay. When I send data to the
'windows vm running on top of iscsi' it starts writing at around
350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go
faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really
acceptable.
Another really viable and performant scenario would be to have the
Windows file servers connect to rbd directly with the windows rbd
driver. It seems to work well, it's fast, and you don't have the
bottleneck that the iscsi gateway creates. But I see this driver is
still in beta. Is anyone using this in production? What are your
experiences? We would miss out on the separation of layers and thus
have less security, but at the same time, it really increases
efficiency and performance.
And if I use rbd, then vmware won't see the storage, and I cannot do
an image backup using veeam. I could of course do backups of the rbd
images, using tools like restic or backy to rgw running on the second
cluster with immutable buckets. What are your experiences? Is it easy
to do differential backups of lots of 50TB rbd images? Change rate is
usually like 0.005% per day or something ;)
By the way, we also thought about CephFS, but we have some complex
stuff going on with extended ACL's that I don't think will play nice
with CephFS, and I think it's a lot more complex to backup CephFS than
block images.
If you made it here, thank you for your time! I hope you can share
thoughts on my questions!
Angelo.
28 Apr
28 Apr
3:50 a.m.
Hey Angelo,
Just to make sure I'm understanding correctly, the main idea for the use
case is to be able to present Ceph storage to windows clients as SMB?
If so, you can absolutely use CephFS to get that done. This is something we
do all the time with our cluster configurations, if we're looking to present
ceph storage to windows clients for the use case of a file server is our
standard choice, and to your point of security/ACLs we can make use of
joining the samba server that to an existing active directory, and then
assigning permissions through Windows.
I will provide a high level overview of an average setup to hopefully
explain it better, and of course if you have any questions please let me
know. I understand that this is way different of a setup of what you
currently have planned, but it's a different choice that could prove useful
in your case.
Essentially how it works is we have ceph cluster with CephFS configured, of
which we map CephFS kernel mounts onto some gateway nodes, at which point we
expose to clients via CTDB with SMB shares (CTDB for high availability).
i.e
ceph cluster > ceph fs > map cephfs kernel mount on linux client > create
smb share on top of cephfs kernel mount > connect to samba share with
windows clients.
The SMB gateway nodes hosting samba also can be joined to an Active
Directory to allow setting Windows ACL permissions to allow more in depth
control of ACLs.
Also I will say +1 for the RBD driver on Windows, something we also make use
of a lot and have a lot of success with.
Again, please let me know if you need any insight or clarification, or have
any further questions. Hope this is of assistance.
Regards,
Bailey
-----Original Message-----
From: Angelo Höngens <angelo(a)hongens.nl>
Sent: April 27, 2023 6:06 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] architecture help (iscsi, rbd, backups?)
Hey guys and girls,
I'm working on a project to build storage for one of our departments, and I
want
to ask you guys and girls for input on the high-level overview part.
It's a long one, I hope you read along and comment.
SUMMARY
I made a plan last year to build a 'storage solution' including ceph and
some windows VM's to expose the data over SMB to clients. A year later I
finally have the hardware, built a ceph cluster, and I'm doing tests. Ceph
itself runs great, but when I wanted to start exposing the data using iscsi
to our VMware farm, I ran into some issues. I know the iscsi gateways will
introduce some new performance bottlenecks, but I'm seeing really slow
performance, still working on that.
But then I ran into the warning on the iscsi gateway page: "The iSCSI
gateway
is in maintenance as of November 2022. This means that it is no
longer in active development and will not be updated to add new features.".
Wait, what? Why!? What does this mean? Does this mean that iSCSI is now
'feature complete' and will still be supported the next 5 years, or will it
be deprecated in the future? I tried searching, but couldn't find any info
on the decision and the roadmap.
My goal is to build a future-proof setup, and using deprecated components
should not
be part of that of course.
If the iscsi gateway will still be supported the next few years and I can
iron out
the performance issues, I can still go on with my original plan. If
not, I have to go back to the drawing board. And maybe you guys would advise
me to take another route anyway.
GOALS
My goals/considerations are:
- we want >1PB of storage capacity for cheap (on a tight budget) for
research
data. Most of it is 'store once, read sometimes'. <1% of the data
is 'hot'.
- focus is on capacity, but it would be nice to have
> 200MB/s of
sequential write/read performance and not 'totally suck' on
random i/o. Yes,
not very well quantified, but ah. Sequential writes are most important.
- end users all run Windows computers (mostly
VDI's) and a lot of
applications require SMB shares.
- security is a big thing, we want really tight
ACL's, specific monitoring
agents, etc.
- our data is incredibly important to us, we still want
the 3-2-1 backup
rule. Primary storage solution, a second storage solution in a
different
place, and some of the data that is not reproducible is also written to
tape. We also want to be protected from ransomware or user errors (so no
direct replication to the second storage).
- I like open source, reliability, no fork-lift
upgrades, no vendor
lock-in, blah, well, I'm on the ceph list here, no need to
convince you guys
;)
- We're hiring a commercial company to do ceph
maintenance and support for
when I'm on leave or leaving the company, but they
won't support clients,
backup software, etc, so I want something as simple as possible. We do have
multiple Windows/VMware admins, but no other real linux guru's.
THE INITIAL PLAN
Given these considerations, I ordered two identical clusters, each
consisting of 3
monitor nodes and 8 osd nodes, Each osd node has 2 ssd's and
10 capacity disks (EC 4:2 for the data), and each node is connected using a
2x25Gbps bond. Ceph is running like a charm. Now I just have to think about
exposing the data to end users, and I've been testing different setups.
My original plan was to expose for example 10x100TB rbd images using iSCSI
to our
VMware farm, formatting the luns with VMFS6, and run for example 2
Windows file servers per datastore on that with a single DFS namespace to
end users. Then backup the file servers using our existing Veeam
infrastructure to RGW running on the second cluster with an immutable
bucket. This way we would have easily defined security boundaries: the
clients can only reach the file servers, the file servers only see their
local VMDK's, ESX only sees the luns on the iSCSI target, etc. When a file
server would be compromised, it would have no access to ceph. We have easy
incremental backups, immutability for ransomware protection, etc. And the
best part is that the ceph admin can worry about ceph, the vmware admin can
focus on ESX, VMFS and all the vmware stuff, and the Windows admins can
focus on the Windows boxes, Windows-specific ACLS and tools and Veeam
backups and stuff.
CURRENT SITUATION
I'm building out this plan now, but I'm running into issues with iSCSI. Are
any of you doing something similar? What is your iscsi performance compared
to direct rbd?
In regard to performance: If I take 2 test windows VM's, I put one on an
iSCSI
datastore and another with direct rbd access using the windows rbd
driver, I create a share on those boxes and push data to it, I see different
results (of course). Copying some iso images over SMB to the 'windows vm
running direct rbd' I see around 800MB/s write, and 200MB/s read, which is
pretty okay. When I send data to the 'windows vm running on top of iscsi' it
starts writing at around 350MB/s, but after like 10-20 seconds drops to
100MB/s and won't go faster. Reads are anywhere from 40MB/s to 80MB/s, which
is not really acceptable.
Another really viable and performant scenario would be to have the Windows
file
servers connect to rbd directly with the windows rbd driver. It seems
to work well, it's fast, and you don't have the bottleneck that the iscsi
gateway creates. But I see this driver is still in beta. Is anyone using
this in production? What are your experiences? We would miss out on the
separation of layers and thus have less security, but at the same time, it
really increases efficiency and performance.
And if I use rbd, then vmware won't see the storage, and I cannot do an
image
backup using veeam. I could of course do backups of the rbd images,
using tools like restic or backy to rgw running on the second cluster with
immutable buckets. What are your experiences? Is it easy to do differential
backups of lots of 50TB rbd images? Change rate is usually like 0.005% per
day or something ;)
By the way, we also thought about CephFS, but we have some complex stuff
going on
with extended ACL's that I don't think will play nice with CephFS,
and I think it's a lot more complex to backup CephFS than block images.
If you made it here, thank you for your time! I hope you can share thoughts
on my
questions!
Angelo.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email
to
ceph-users-leave(a)ceph.io
4:16 a.m.
There is also a direct RBD client for MS Windows, though it's relatively young.
On Apr 27, 2023, at 18:20, Bailey Allison
<ballison(a)45drives.com> wrote:
Hey Angelo,
Just to make sure I'm understanding correctly, the main idea for the use
case is to be able to present Ceph storage to windows clients as SMB?
If so, you can absolutely use CephFS to get that done. This is something we
do all the time with our cluster configurations, if we're looking to present
ceph storage to windows clients for the use case of a file server is our
standard choice, and to your point of security/ACLs we can make use of
joining the samba server that to an existing active directory, and then
assigning permissions through Windows.
I will provide a high level overview of an average setup to hopefully
explain it better, and of course if you have any questions please let me
know. I understand that this is way different of a setup of what you
currently have planned, but it's a different choice that could prove useful
in your case.
Essentially how it works is we have ceph cluster with CephFS configured, of
which we map CephFS kernel mounts onto some gateway nodes, at which point we
expose to clients via CTDB with SMB shares (CTDB for high availability).
i.e
ceph cluster > ceph fs > map cephfs kernel mount on linux client > create
smb share on top of cephfs kernel mount > connect to samba share with
windows clients.
The SMB gateway nodes hosting samba also can be joined to an Active
Directory to allow setting Windows ACL permissions to allow more in depth
control of ACLs.
Also I will say +1 for the RBD driver on Windows, something we also make use
of a lot and have a lot of success with.
Again, please let me know if you need any insight or clarification, or have
any further questions. Hope this is of assistance.
Regards,
Bailey
-----Original Message-----
From: Angelo Höngens <angelo(a)hongens.nl>
Sent: April 27, 2023 6:06 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] architecture help (iscsi, rbd, backups?)
Hey guys and girls,
I'm working on a project to build storage for one of our departments, and I
want to ask you guys and girls for input on the high-level overview part.
It's a long one, I hope you read along and comment.
SUMMARY
I made a plan last year to build a 'storage solution' including ceph and
some windows VM's to expose the data over SMB to clients. A year later I
finally have the hardware, built a ceph cluster, and I'm doing tests. Ceph
itself runs great, but when I wanted to start exposing the data using iscsi
to our VMware farm, I ran into some issues. I know the iscsi gateways will
introduce some new performance bottlenecks, but I'm seeing really slow
performance, still working on that.
But then I ran into the warning on the iscsi gateway page: "The iSCSI
gateway
is in maintenance as of November 2022. This means that it is no
longer in active development and will not be updated to add new features.".
Wait, what? Why!? What does this mean? Does this mean that iSCSI is now
'feature complete' and will still be supported the next 5 years, or will it
be deprecated in the future? I tried searching, but couldn't find any info
on the decision and the roadmap.
My goal is to build a future-proof setup, and using deprecated components
should
not be part of that of course.
If the iscsi gateway will still be supported the next few years and I can
iron out
the performance issues, I can still go on with my original plan. If
not, I have to go back to the drawing board. And maybe you guys would advise
me to take another route anyway.
GOALS
My goals/considerations are:
- we want >1PB of storage capacity for cheap (on a tight budget) for
research
data. Most of it is 'store once, read sometimes'. <1% of the data
is 'hot'.
- focus is on capacity, but it would be nice to
have > 200MB/s of
sequential write/read performance and not 'totally
suck' on random i/o. Yes,
not very well quantified, but ah. Sequential writes are most important.
- end users all run Windows computers (mostly
VDI's) and a lot of
applications require SMB shares.
- security is a big thing, we want really tight
ACL's, specific monitoring
agents, etc.
- our data is incredibly important to us, we
still want the 3-2-1 backup
rule. Primary storage solution, a second storage
solution in a different
place, and some of the data that is not reproducible is also written to
tape. We also want to be protected from ransomware or user errors (so no
direct replication to the second storage).
- I like open source, reliability, no fork-lift
upgrades, no vendor
lock-in, blah, well, I'm on the ceph list here, no need to
convince you guys
;)
- We're hiring a commercial company to do
ceph maintenance and support for
when I'm on leave or leaving the company, but
they won't support clients,
backup software, etc, so I want something as simple as possible. We do have
multiple Windows/VMware admins, but no other real linux guru's.
THE INITIAL PLAN
Given these considerations, I ordered two identical clusters, each
consisting of 3
monitor nodes and 8 osd nodes, Each osd node has 2 ssd's and
10 capacity disks (EC 4:2 for the data), and each node is connected using a
2x25Gbps bond. Ceph is running like a charm. Now I just have to think about
exposing the data to end users, and I've been testing different setups.
My original plan was to expose for example 10x100TB rbd images using iSCSI
to our
VMware farm, formatting the luns with VMFS6, and run for example 2
Windows file servers per datastore on that with a single DFS namespace to
end users. Then backup the file servers using our existing Veeam
infrastructure to RGW running on the second cluster with an immutable
bucket. This way we would have easily defined security boundaries: the
clients can only reach the file servers, the file servers only see their
local VMDK's, ESX only sees the luns on the iSCSI target, etc. When a file
server would be compromised, it would have no access to ceph. We have easy
incremental backups, immutability for ransomware protection, etc. And the
best part is that the ceph admin can worry about ceph, the vmware admin can
focus on ESX, VMFS and all the vmware stuff, and the Windows admins can
focus on the Windows boxes, Windows-specific ACLS and tools and Veeam
backups and stuff.
CURRENT SITUATION
I'm building out this plan now, but I'm running into issues with iSCSI. Are
any of you doing something similar? What is your iscsi performance compared
to direct rbd?
In regard to performance: If I take 2 test windows VM's, I put one on an
iSCSI
datastore and another with direct rbd access using the windows rbd
driver, I create a share on those boxes and push data to it, I see different
results (of course). Copying some iso images over SMB to the 'windows vm
running direct rbd' I see around 800MB/s write, and 200MB/s read, which is
pretty okay. When I send data to the 'windows vm running on top of iscsi' it
starts writing at around 350MB/s, but after like 10-20 seconds drops to
100MB/s and won't go faster. Reads are anywhere from 40MB/s to 80MB/s, which
is not really acceptable.
Another really viable and performant scenario would be to have the Windows
file
servers connect to rbd directly with the windows rbd driver. It seems
to work well, it's fast, and you don't have the bottleneck that the iscsi
gateway creates. But I see this driver is still in beta. Is anyone using
this in production? What are your experiences? We would miss out on the
separation of layers and thus have less security, but at the same time, it
really increases efficiency and performance.
And if I use rbd, then vmware won't see the storage, and I cannot do an
image
backup using veeam. I could of course do backups of the rbd images,
using tools like restic or backy to rgw running on the second cluster with
immutable buckets. What are your experiences? Is it easy to do differential
backups of lots of 50TB rbd images? Change rate is usually like 0.005% per
day or something ;)
By the way, we also thought about CephFS, but we have some complex stuff
going on
with extended ACL's that I don't think will play nice with CephFS,
and I think it's a lot more complex to backup CephFS than block images.
If you made it here, thank you for your time! I hope you can share thoughts
on my
questions!
Angelo.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email
to
ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
30 Apr
30 Apr
7:51 a.m.
Bailey,
Thanks for your extensive reply, you got me down the wormhole of CephFS
and SMB (and looking at a lot of 45drives videos and knowledge base,
Houston dashboard, reading up on CTDB, etc), and this is a really
interesting option as well! Thanks for the write-up.
By the way, are you using the RBD driver in Windows in production with
your customers?
The binaries are still called beta, and last time I tried it in a proof
of concept setup (a while back), it would never connect and always crash
out on me. After reporting an issue, I did not get a response for almost
three months before a dev responded that it was an unsupported ipv6
issue. Not a problem, and all very understandable, it's open source
software written mostly by volunteers, but I got a bit cautious about
deploying this to production ;)
Angelo.
On 27/04/2023 18:20, Bailey Allison wrote:
> Hey Angelo,
>
> Just to make sure I'm understanding correctly, the main idea for the use
> case is to be able to present Ceph storage to windows clients as SMB?
>
> If so, you can absolutely use CephFS to get that done. This is something we
> do all the time with our cluster configurations, if we're looking to present
> ceph storage to windows clients for the use case of a file server is our
> standard choice, and to your point of security/ACLs we can make use of
> joining the samba server that to an existing active directory, and then
> assigning permissions through Windows.
>
> I will provide a high level overview of an average setup to hopefully
> explain it better, and of course if you have any questions please let me
> know. I understand that this is way different of a setup of what you
> currently have planned, but it's a different choice that could prove useful
> in your case.
>
> Essentially how it works is we have ceph cluster with CephFS configured, of
> which we map CephFS kernel mounts onto some gateway nodes, at which point we
> expose to clients via CTDB with SMB shares (CTDB for high availability).
>
> i.e
>
> ceph cluster > ceph fs > map cephfs kernel mount on linux client > create
> smb share on top of cephfs kernel mount > connect to samba share with
> windows clients.
>
> The SMB gateway nodes hosting samba also can be joined to an Active
> Directory to allow setting Windows ACL permissions to allow more in depth
> control of ACLs.
>
> Also I will say +1 for the RBD driver on Windows, something we also make use
> of a lot and have a lot of success with.
>
> Again, please let me know if you need any insight or clarification, or have
> any further questions. Hope this is of assistance.
>
> Regards,
>
> Bailey
>
> -----Original Message-----
>> From: Angelo Höngens <angelo(a)hongens.nl>
>> Sent: April 27, 2023 6:06 PM
>> To: ceph-users(a)ceph.io
>> Subject: [ceph-users] architecture help (iscsi, rbd, backups?)
>>
>> Hey guys and girls,
>>
>> I'm working on a project to build storage for one of our departments, and I
> want to ask you guys and girls for input on the high-level overview part.
> It's a long one, I hope you read along and comment.
>>
>> SUMMARY
>>
>> I made a plan last year to build a 'storage solution' including ceph and
> some windows VM's to expose the data over SMB to clients. A year later I
> finally have the hardware, built a ceph cluster, and I'm doing tests. Ceph
> itself runs great, but when I wanted to start exposing the data using iscsi
> to our VMware farm, I ran into some issues. I know the iscsi gateways will
> introduce some new performance bottlenecks, but I'm seeing really slow
> performance, still working on that.
>>
>> But then I ran into the warning on the iscsi gateway page: "The iSCSI
> gateway is in maintenance as of November 2022. This means that it is no
> longer in active development and will not be updated to add new features.".
> Wait, what? Why!? What does this mean? Does this mean that iSCSI is now
> 'feature complete' and will still be supported the next 5 years, or will it
> be deprecated in the future? I tried searching, but couldn't find any info
> on the decision and the roadmap.
>>
>> My goal is to build a future-proof setup, and using deprecated components
> should not be part of that of course.
>>
>> If the iscsi gateway will still be supported the next few years and I can
> iron out the performance issues, I can still go on with my original plan. If
> not, I have to go back to the drawing board. And maybe you guys would advise
> me to take another route anyway.
>>
>> GOALS
>>
>> My goals/considerations are:
>>
>> - we want >1PB of storage capacity for cheap (on a tight budget) for
> research data. Most of it is 'store once, read sometimes'. <1% of the
data
> is 'hot'.
>> - focus is on capacity, but it would be nice to have > 200MB/s of
> sequential write/read performance and not 'totally suck' on random i/o. Yes,
> not very well quantified, but ah. Sequential writes are most important.
>> - end users all run Windows computers (mostly VDI's) and a lot of
> applications require SMB shares.
>> - security is a big thing, we want really tight ACL's, specific monitoring
> agents, etc.
>> - our data is incredibly important to us, we still want the 3-2-1 backup
> rule. Primary storage solution, a second storage solution in a different
> place, and some of the data that is not reproducible is also written to
> tape. We also want to be protected from ransomware or user errors (so no
> direct replication to the second storage).
>> - I like open source, reliability, no fork-lift upgrades, no vendor
> lock-in, blah, well, I'm on the ceph list here, no need to convince you guys
> ;)
>> - We're hiring a commercial company to do ceph maintenance and support for
> when I'm on leave or leaving the company, but they won't support clients,
> backup software, etc, so I want something as simple as possible. We do have
> multiple Windows/VMware admins, but no other real linux guru's.
>>
>> THE INITIAL PLAN
>>
>> Given these considerations, I ordered two identical clusters, each
> consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2 ssd's and
> 10 capacity disks (EC 4:2 for the data), and each node is connected using a
> 2x25Gbps bond. Ceph is running like a charm. Now I just have to think about
> exposing the data to end users, and I've been testing different setups.
>>
>> My original plan was to expose for example 10x100TB rbd images using iSCSI
> to our VMware farm, formatting the luns with VMFS6, and run for example 2
> Windows file servers per datastore on that with a single DFS namespace to
> end users. Then backup the file servers using our existing Veeam
> infrastructure to RGW running on the second cluster with an immutable
> bucket. This way we would have easily defined security boundaries: the
> clients can only reach the file servers, the file servers only see their
> local VMDK's, ESX only sees the luns on the iSCSI target, etc. When a file
> server would be compromised, it would have no access to ceph. We have easy
> incremental backups, immutability for ransomware protection, etc. And the
> best part is that the ceph admin can worry about ceph, the vmware admin can
> focus on ESX, VMFS and all the vmware stuff, and the Windows admins can
> focus on the Windows boxes, Windows-specific ACLS and tools and Veeam
> backups and stuff.
>>
>> CURRENT SITUATION
>>
>> I'm building out this plan now, but I'm running into issues with iSCSI.
Are
> any of you doing something similar? What is your iscsi performance compared
> to direct rbd?
>>
>> In regard to performance: If I take 2 test windows VM's, I put one on an
> iSCSI datastore and another with direct rbd access using the windows rbd
> driver, I create a share on those boxes and push data to it, I see different
> results (of course). Copying some iso images over SMB to the 'windows vm
> running direct rbd' I see around 800MB/s write, and 200MB/s read, which is
> pretty okay. When I send data to the 'windows vm running on top of iscsi' it
> starts writing at around 350MB/s, but after like 10-20 seconds drops to
> 100MB/s and won't go faster. Reads are anywhere from 40MB/s to 80MB/s, which
> is not really acceptable.
>>
>> Another really viable and performant scenario would be to have the Windows
> file servers connect to rbd directly with the windows rbd driver. It seems
> to work well, it's fast, and you don't have the bottleneck that the iscsi
> gateway creates. But I see this driver is still in beta. Is anyone using
> this in production? What are your experiences? We would miss out on the
> separation of layers and thus have less security, but at the same time, it
> really increases efficiency and performance.
>>
>> And if I use rbd, then vmware won't see the storage, and I cannot do an
> image backup using veeam. I could of course do backups of the rbd images,
> using tools like restic or backy to rgw running on the second cluster with
> immutable buckets. What are your experiences? Is it easy to do differential
> backups of lots of 50TB rbd images? Change rate is usually like 0.005% per
> day or something ;)
>>
>> By the way, we also thought about CephFS, but we have some complex stuff
> going on with extended ACL's that I don't think will play nice with CephFS,
> and I think it's a lot more complex to backup CephFS than block images.
>>
>> If you made it here, thank you for your time! I hope you can share thoughts
> on my questions!
>>
>> Angelo.
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email
> to ceph-users-leave(a)ceph.io
>
3 May
3 May
3:08 a.m.
Hey Angelo,
Ya, we are using the RBD driver for quite a few customers in production, and it is working
quite good!
Hahahaha, I am familiar with the bug you are talking about I think, I believe that may be
resolved by now.
I believe the driver is either out of beta now/soon to be out of beta?
I recall watching a talk by the developers where they mentioned it should be out of beta
soon or already is, in addition I want to say they are also/would also eventually be
offering a paid support for it, but can't say for certain on that.
Otherwise though like I said we are using it for quite a few customers with some really
good success.
Also, if you have any further questions re: CephFS and SMB please feel free to ask! I am
sure other people might also find the same questions and answers helpful 😊
Regards,
Bailey
-----Original Message-----
From: Angelo Hongens <angelo(a)hongens.nl>
Sent: April 29, 2023 11:21 PM
To: Bailey Allison <ballison(a)45drives.com>om>; ceph-users(a)ceph.io
Subject: [ceph-users] Re: architecture help (iscsi, rbd, backups?)
Bailey,
Thanks for your extensive reply, you got me down the wormhole of CephFS and SMB (and
looking at a lot of 45drives videos and knowledge base, Houston dashboard, >reading up
on CTDB, etc), and this is a really interesting option as well! Thanks for the write-up.
By the way, are you using the RBD driver in Windows in production with your customers?
The binaries are still called beta, and last time I tried it in a proof of concept setup
(a while back), it would never connect and always crash out on me. After reporting >an
issue, I did not get a response for almost three months before a dev responded that it was
an unsupported ipv6 issue. Not a problem, and all very understandable, >it's open
source software written mostly by volunteers, but I got a bit cautious about deploying
this to production ;)
Angelo.
On 27/04/2023 18:20, Bailey Allison wrote:
Hey Angelo,
Just to make sure I'm understanding correctly, the main idea for the
use case is to be able to present Ceph storage to windows clients as SMB?
If so, you can absolutely use CephFS to get that done. This is
something we do all the time with our cluster configurations, if we're
looking to present ceph storage to windows clients for the use case of
a file server is our standard choice, and to your point of
security/ACLs we can make use of joining the samba server that to an
existing active directory, and then assigning permissions through Windows.
I will provide a high level overview of an average setup to hopefully
explain it better, and of course if you have any questions please let
me know. I understand that this is way different of a setup of what
you currently have planned, but it's a different choice that could
prove useful in your case.
Essentially how it works is we have ceph cluster with CephFS
configured, of which we map CephFS kernel mounts onto some gateway
nodes, at which point we expose to clients via CTDB with SMB shares (CTDB for high
availability).
i.e
ceph cluster > ceph fs > map cephfs kernel mount on linux client >
create smb share on top of cephfs kernel mount > connect to samba
share with windows clients.
The SMB gateway nodes hosting samba also can be joined to an Active
Directory to allow setting Windows ACL permissions to allow more in
depth control of ACLs.
Also I will say +1 for the RBD driver on Windows, something we also
make use of a lot and have a lot of success with.
Again, please let me know if you need any insight or clarification, or
have any further questions. Hope this is of assistance.
Regards,
Bailey
-----Original Message-----
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
From: Angelo Höngens <angelo(a)hongens.nl>
Sent: April 27, 2023 6:06 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] architecture help (iscsi, rbd, backups?)
Hey guys and girls,
I'm working on a project to build storage for one of our departments,
and I
want to ask you guys and girls for input on the high-level overview part.
It's a long one, I hope you read along and comment.
SUMMARY
I made a plan last year to build a 'storage solution' including ceph
and
some windows VM's to expose the data over SMB to clients. A year later
I finally have the hardware, built a ceph cluster, and I'm doing
tests. Ceph itself runs great, but when I wanted to start exposing the
data using iscsi to our VMware farm, I ran into some issues. I know
the iscsi gateways will introduce some new performance bottlenecks,
but I'm seeing really slow performance, still working on that.
But then I ran into the warning on the iscsi gateway page: "The iSCSI
gateway
is in maintenance as of November 2022. This means that it is
no longer in active development and will not be updated to add new features.".
Wait, what? Why!? What does this mean? Does this mean that iSCSI is
now 'feature complete' and will still be supported the next 5 years,
or will it be deprecated in the future? I tried searching, but
couldn't find any info on the decision and the roadmap.
My goal is to build a future-proof setup, and using deprecated
components
should not be part of that of course.
If the iscsi gateway will still be supported the next few years and I
can
iron out the performance issues, I can still go on with my original
plan. If not, I have to go back to the drawing board. And maybe you
guys would advise me to take another route anyway.
GOALS
My goals/considerations are:
- we want >1PB of storage capacity for cheap (on a tight budget) for
research
data. Most of it is 'store once, read sometimes'. <1% of the
data is 'hot'.
- focus is on capacity, but it would be nice to
have > 200MB/s of
sequential write/read performance and not 'totally
suck' on random
i/o. Yes, not very well quantified, but ah. Sequential writes are most important.
- end users all run Windows computers (mostly
VDI's) and a lot of
applications require SMB shares.
- security is a big thing, we want really tight
ACL's, specific
monitoring
agents, etc.
- our data is incredibly important to us, we
still want the 3-2-1
backup
rule. Primary storage solution, a second storage solution in a
different place, and some of the data that is not reproducible is also
written to tape. We also want to be protected from ransomware or user
errors (so no direct replication to the second storage).
- I like open source, reliability, no fork-lift
upgrades, no vendor
lock-in, blah, well, I'm on the ceph list here, no need to
convince
you guys
;)
- We're hiring a commercial company to do
ceph maintenance and
support for
when I'm on leave or leaving the company, but they won't
support
clients, backup software, etc, so I want something as simple as
possible. We do have multiple Windows/VMware admins, but no other real linux guru's.
THE INITIAL PLAN
Given these considerations, I ordered two identical clusters, each
consisting of 3
monitor nodes and 8 osd nodes, Each osd node has 2
ssd's and
10 capacity disks (EC 4:2 for the data), and each node is connected
using a 2x25Gbps bond. Ceph is running like a charm. Now I just have
to think about exposing the data to end users, and I've been testing different
setups.
My original plan was to expose for example 10x100TB rbd images using
iSCSI
to our VMware farm, formatting the luns with VMFS6, and run for
example 2 Windows file servers per datastore on that with a single DFS
namespace to end users. Then backup the file servers using our
existing Veeam infrastructure to RGW running on the second cluster
with an immutable bucket. This way we would have easily defined
security boundaries: the clients can only reach the file servers, the
file servers only see their local VMDK's, ESX only sees the luns on
the iSCSI target, etc. When a file server would be compromised, it
would have no access to ceph. We have easy incremental backups,
immutability for ransomware protection, etc. And the best part is that
the ceph admin can worry about ceph, the vmware admin can focus on
ESX, VMFS and all the vmware stuff, and the Windows admins can focus
on the Windows boxes, Windows-specific ACLS and tools and Veeam backups and stuff.
CURRENT SITUATION
I'm building out this plan now, but I'm running into issues with
iSCSI. Are
any of you doing something similar? What is your iscsi performance
compared to direct rbd?
In regard to performance: If I take 2 test windows VM's, I put one on
an
iSCSI datastore and another with direct rbd access using the windows
rbd driver, I create a share on those boxes and push data to it, I see
different results (of course). Copying some iso images over SMB to the
'windows vm running direct rbd' I see around 800MB/s write, and
200MB/s read, which is pretty okay. When I send data to the 'windows
vm running on top of iscsi' it starts writing at around 350MB/s, but
after like 10-20 seconds drops to 100MB/s and won't go faster. Reads
are anywhere from 40MB/s to 80MB/s, which is not really acceptable.
Another really viable and performant scenario would be to have the
Windows
file servers connect to rbd directly with the windows rbd driver. It
seems to work well, it's fast, and you don't have the bottleneck that
the iscsi gateway creates. But I see this driver is still in beta. Is
anyone using this in production? What are your experiences? We would
miss out on the separation of layers and thus have less security, but
at the same time, it really increases efficiency and performance.
And if I use rbd, then vmware won't see the storage, and I cannot do
an
image backup using veeam. I could of course do backups of the rbd
images, using tools like restic or backy to rgw running on the second
cluster with immutable buckets. What are your experiences? Is it easy
to do differential backups of lots of 50TB rbd images? Change rate is
usually like 0.005% per day or something ;)
By the way, we also thought about CephFS, but we have some complex
stuff
going on with extended ACL's that I don't think will play nice with
CephFS, and I think it's a lot more complex to backup CephFS than block images.
If you made it here, thank you for your time! I hope you can share
thoughts
on my questions!
Angelo.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email
to ceph-users-leave(a)ceph.io
28 Apr
28 Apr
5:35 a.m.
Hi Angelo,
Just some thoughts to consider from our experience with similar setups:
1. Use Proxmox instead of VMWare, or anything KVM based. These VMs can
consume Ceph directly, and provide the same level of service (some may say
better) for live ,migration, hyperconvergence etc. Then you run Windows
VMs in KVM, bring RBD storage to them as virtual disks and share out as
needed.
2. Use NFS - all modern Windows OSs support it. You can use any NFS
gateway you like, or set up your own machine or cluster (which is what we
did with Storcium) and export your storage as needed.
3. If you must use VMWare, you can present datastores via NFS as well, this
has a lot of indirection but is easier to manage.
--
Alex Gorbachev
ISS Storcium
https://www.iss-integration.com
On Thu, Apr 27, 2023 at 5:06 PM Angelo Höngens <angelo(a)hongens.nl> wrote:
Hey guys and girls,
I'm working on a project to build storage for one of our departments,
and I want to ask you guys and girls for input on the high-level
overview part. It's a long one, I hope you read along and comment.
SUMMARY
I made a plan last year to build a 'storage solution' including ceph
and some windows VM's to expose the data over SMB to clients. A year
later I finally have the hardware, built a ceph cluster, and I'm doing
tests. Ceph itself runs great, but when I wanted to start exposing the
data using iscsi to our VMware farm, I ran into some issues. I know
the iscsi gateways will introduce some new performance bottlenecks,
but I'm seeing really slow performance, still working on that.
But then I ran into the warning on the iscsi gateway page: "The iSCSI
gateway is in maintenance as of November 2022. This means that it is
no longer in active development and will not be updated to add new
features.". Wait, what? Why!? What does this mean? Does this mean that
iSCSI is now 'feature complete' and will still be supported the next 5
years, or will it be deprecated in the future? I tried searching, but
couldn't find any info on the decision and the roadmap.
My goal is to build a future-proof setup, and using deprecated
components should not be part of that of course.
If the iscsi gateway will still be supported the next few years and I
can iron out the performance issues, I can still go on with my
original plan. If not, I have to go back to the drawing board. And
maybe you guys would advise me to take another route anyway.
GOALS
My goals/considerations are:
- we want >1PB of storage capacity for cheap (on a tight budget) for
research data. Most of it is 'store once, read sometimes'. <1% of the
data is 'hot'.
- focus is on capacity, but it would be nice to have > 200MB/s of
sequential write/read performance and not 'totally suck' on random
i/o. Yes, not very well quantified, but ah. Sequential writes are most
important.
- end users all run Windows computers (mostly VDI's) and a lot of
applications require SMB shares.
- security is a big thing, we want really tight ACL's, specific
monitoring agents, etc.
- our data is incredibly important to us, we still want the 3-2-1
backup rule. Primary storage solution, a second storage solution in a
different place, and some of the data that is not reproducible is also
written to tape. We also want to be protected from ransomware or user
errors (so no direct replication to the second storage).
- I like open source, reliability, no fork-lift upgrades, no vendor
lock-in, blah, well, I'm on the ceph list here, no need to convince
you guys ;)
- We're hiring a commercial company to do ceph maintenance and support
for when I'm on leave or leaving the company, but they won't support
clients, backup software, etc, so I want something as simple as
possible. We do have multiple Windows/VMware admins, but no other real
linux guru's.
THE INITIAL PLAN
Given these considerations, I ordered two identical clusters, each
consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2
ssd's and 10 capacity disks (EC 4:2 for the data), and each node is
connected using a 2x25Gbps bond. Ceph is running like a charm. Now I
just have to think about exposing the data to end users, and I've been
testing different setups.
My original plan was to expose for example 10x100TB rbd images using
iSCSI to our VMware farm, formatting the luns with VMFS6, and run for
example 2 Windows file servers per datastore on that with a single DFS
namespace to end users. Then backup the file servers using our
existing Veeam infrastructure to RGW running on the second cluster
with an immutable bucket. This way we would have easily defined
security boundaries: the clients can only reach the file servers, the
file servers only see their local VMDK's, ESX only sees the luns on
the iSCSI target, etc. When a file server would be compromised, it
would have no access to ceph. We have easy incremental backups,
immutability for ransomware protection, etc. And the best part is that
the ceph admin can worry about ceph, the vmware admin can focus on
ESX, VMFS and all the vmware stuff, and the Windows admins can focus
on the Windows boxes, Windows-specific ACLS and tools and Veeam
backups and stuff.
CURRENT SITUATION
I'm building out this plan now, but I'm running into issues with
iSCSI. Are any of you doing something similar? What is your iscsi
performance compared to direct rbd?
In regard to performance: If I take 2 test windows VM's, I put one on
an iSCSI datastore and another with direct rbd access using the
windows rbd driver, I create a share on those boxes and push data to
it, I see different results (of course). Copying some iso images over
SMB to the 'windows vm running direct rbd' I see around 800MB/s write,
and 200MB/s read, which is pretty okay. When I send data to the
'windows vm running on top of iscsi' it starts writing at around
350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go
faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really
acceptable.
Another really viable and performant scenario would be to have the
Windows file servers connect to rbd directly with the windows rbd
driver. It seems to work well, it's fast, and you don't have the
bottleneck that the iscsi gateway creates. But I see this driver is
still in beta. Is anyone using this in production? What are your
experiences? We would miss out on the separation of layers and thus
have less security, but at the same time, it really increases
efficiency and performance.
And if I use rbd, then vmware won't see the storage, and I cannot do
an image backup using veeam. I could of course do backups of the rbd
images, using tools like restic or backy to rgw running on the second
cluster with immutable buckets. What are your experiences? Is it easy
to do differential backups of lots of 50TB rbd images? Change rate is
usually like 0.005% per day or something ;)
By the way, we also thought about CephFS, but we have some complex
stuff going on with extended ACL's that I don't think will play nice
with CephFS, and I think it's a lot more complex to backup CephFS than
block images.
If you made it here, thank you for your time! I hope you can share
thoughts on my questions!
Angelo.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
30 Apr
30 Apr
8:28 a.m.
Thanks Alex, interesting perspectives.
I already thought about proxmox as well, and that would also work quite
nicely. I think that would be the most performant option to put VM's on
RBD.
But my entire goal was to run SMB servers on top of that hypervisor
layer, to serve SMB shares to Windows.
So I think Bailey's suggestion makes more sense, to use CephFS with
Linux SMB gateways, which cuts out a layer in between, greatly improving
performance.
That also has the benefit of being able to use a single 1PB CephFS
filesystem served by multiple SMB gateways instead of my initial plan of
having like 10x100TB windows SMB file servers (I would not dare have a
single 1PB windows VM with an NTFS disk)
Angelo.
On 27/04/2023 20:05, Alex Gorbachev wrote:
> Hi Angelo,
>
> Just some thoughts to consider from our experience with similar setups:
>
> 1. Use Proxmox instead of VMWare, or anything KVM based. These VMs can
> consume Ceph directly, and provide the same level of service (some may
> say better) for live ,migration, hyperconvergence etc. Then you run
> Windows VMs in KVM, bring RBD storage to them as virtual disks and share
> out as needed.
>
> 2. Use NFS - all modern Windows OSs support it. You can use any NFS
> gateway you like, or set up your own machine or cluster (which is what
> we did with Storcium) and export your storage as needed.
>
> 3. If you must use VMWare, you can present datastores via NFS as well,
> this has a lot of indirection but is easier to manage.
>
> --
> Alex Gorbachev
> ISS Storcium
> https://www.iss-integration.com <https://www.iss-integration.com>
>
>
>
> On Thu, Apr 27, 2023 at 5:06 PM Angelo Höngens <angelo(a)hongens.nl
> <mailto:angelo@hongens.nl>> wrote:
>
> Hey guys and girls,
>
> I'm working on a project to build storage for one of our departments,
> and I want to ask you guys and girls for input on the high-level
> overview part. It's a long one, I hope you read along and comment.
>
> SUMMARY
>
> I made a plan last year to build a 'storage solution' including ceph
> and some windows VM's to expose the data over SMB to clients. A year
> later I finally have the hardware, built a ceph cluster, and I'm doing
> tests. Ceph itself runs great, but when I wanted to start exposing the
> data using iscsi to our VMware farm, I ran into some issues. I know
> the iscsi gateways will introduce some new performance bottlenecks,
> but I'm seeing really slow performance, still working on that.
>
> But then I ran into the warning on the iscsi gateway page: "The iSCSI
> gateway is in maintenance as of November 2022. This means that it is
> no longer in active development and will not be updated to add new
> features.". Wait, what? Why!? What does this mean? Does this mean that
> iSCSI is now 'feature complete' and will still be supported the next 5
> years, or will it be deprecated in the future? I tried searching, but
> couldn't find any info on the decision and the roadmap.
>
> My goal is to build a future-proof setup, and using deprecated
> components should not be part of that of course.
>
> If the iscsi gateway will still be supported the next few years and I
> can iron out the performance issues, I can still go on with my
> original plan. If not, I have to go back to the drawing board. And
> maybe you guys would advise me to take another route anyway.
>
> GOALS
>
> My goals/considerations are:
>
> - we want >1PB of storage capacity for cheap (on a tight budget) for
> research data. Most of it is 'store once, read sometimes'. <1% of the
> data is 'hot'.
> - focus is on capacity, but it would be nice to have > 200MB/s of
> sequential write/read performance and not 'totally suck' on random
> i/o. Yes, not very well quantified, but ah. Sequential writes are most
> important.
> - end users all run Windows computers (mostly VDI's) and a lot of
> applications require SMB shares.
> - security is a big thing, we want really tight ACL's, specific
> monitoring agents, etc.
> - our data is incredibly important to us, we still want the 3-2-1
> backup rule. Primary storage solution, a second storage solution in a
> different place, and some of the data that is not reproducible is also
> written to tape. We also want to be protected from ransomware or user
> errors (so no direct replication to the second storage).
> - I like open source, reliability, no fork-lift upgrades, no vendor
> lock-in, blah, well, I'm on the ceph list here, no need to convince
> you guys ;)
> - We're hiring a commercial company to do ceph maintenance and support
> for when I'm on leave or leaving the company, but they won't support
> clients, backup software, etc, so I want something as simple as
> possible. We do have multiple Windows/VMware admins, but no other real
> linux guru's.
>
> THE INITIAL PLAN
>
> Given these considerations, I ordered two identical clusters, each
> consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2
> ssd's and 10 capacity disks (EC 4:2 for the data), and each node is
> connected using a 2x25Gbps bond. Ceph is running like a charm. Now I
> just have to think about exposing the data to end users, and I've been
> testing different setups.
>
> My original plan was to expose for example 10x100TB rbd images using
> iSCSI to our VMware farm, formatting the luns with VMFS6, and run for
> example 2 Windows file servers per datastore on that with a single DFS
> namespace to end users. Then backup the file servers using our
> existing Veeam infrastructure to RGW running on the second cluster
> with an immutable bucket. This way we would have easily defined
> security boundaries: the clients can only reach the file servers, the
> file servers only see their local VMDK's, ESX only sees the luns on
> the iSCSI target, etc. When a file server would be compromised, it
> would have no access to ceph. We have easy incremental backups,
> immutability for ransomware protection, etc. And the best part is that
> the ceph admin can worry about ceph, the vmware admin can focus on
> ESX, VMFS and all the vmware stuff, and the Windows admins can focus
> on the Windows boxes, Windows-specific ACLS and tools and Veeam
> backups and stuff.
>
> CURRENT SITUATION
>
> I'm building out this plan now, but I'm running into issues with
> iSCSI. Are any of you doing something similar? What is your iscsi
> performance compared to direct rbd?
>
> In regard to performance: If I take 2 test windows VM's, I put one on
> an iSCSI datastore and another with direct rbd access using the
> windows rbd driver, I create a share on those boxes and push data to
> it, I see different results (of course). Copying some iso images over
> SMB to the 'windows vm running direct rbd' I see around 800MB/s write,
> and 200MB/s read, which is pretty okay. When I send data to the
> 'windows vm running on top of iscsi' it starts writing at around
> 350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go
> faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really
> acceptable.
>
> Another really viable and performant scenario would be to have the
> Windows file servers connect to rbd directly with the windows rbd
> driver. It seems to work well, it's fast, and you don't have the
> bottleneck that the iscsi gateway creates. But I see this driver is
> still in beta. Is anyone using this in production? What are your
> experiences? We would miss out on the separation of layers and thus
> have less security, but at the same time, it really increases
> efficiency and performance.
>
> And if I use rbd, then vmware won't see the storage, and I cannot do
> an image backup using veeam. I could of course do backups of the rbd
> images, using tools like restic or backy to rgw running on the second
> cluster with immutable buckets. What are your experiences? Is it easy
> to do differential backups of lots of 50TB rbd images? Change rate is
> usually like 0.005% per day or something ;)
>
> By the way, we also thought about CephFS, but we have some complex
> stuff going on with extended ACL's that I don't think will play nice
> with CephFS, and I think it's a lot more complex to backup CephFS than
> block images.
>
> If you made it here, thank you for your time! I hope you can share
> thoughts on my questions!
>
> Angelo.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> <mailto:ceph-users@ceph.io>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> <mailto:ceph-users-leave@ceph.io>
>
8:34 a.m.
Hi Angelo,
You can always use Samba to serve shares, it works well with AD, if that is
needed. You may want to benchmark your prototypes in an as close to
production setting as possible.
--
Alex Gorbachev
ISS Storcium
iss-integration.com
On Sat, Apr 29, 2023 at 10:58 PM Angelo Hongens <angelo(a)hongens.nl> wrote:
Thanks Alex, interesting perspectives.
I already thought about proxmox as well, and that would also work quite
nicely. I think that would be the most performant option to put VM's on
RBD.
But my entire goal was to run SMB servers on top of that hypervisor
layer, to serve SMB shares to Windows.
So I think Bailey's suggestion makes more sense, to use CephFS with
Linux SMB gateways, which cuts out a layer in between, greatly improving
performance.
That also has the benefit of being able to use a single 1PB CephFS
filesystem served by multiple SMB gateways instead of my initial plan of
having like 10x100TB windows SMB file servers (I would not dare have a
single 1PB windows VM with an NTFS disk)
Angelo.
On 27/04/2023 20:05, Alex Gorbachev wrote:
Hi Angelo,
Just some thoughts to consider from our experience with similar setups:
1. Use Proxmox instead of VMWare, or anything KVM based. These VMs can
consume Ceph directly, and provide the same level of service (some may
say better) for live ,migration, hyperconvergence etc. Then you run
Windows VMs in KVM, bring RBD storage to them as virtual disks and share
out as needed.
2. Use NFS - all modern Windows OSs support it. You can use any NFS
gateway you like, or set up your own machine or cluster (which is what
we did with Storcium) and export your storage as needed.
3. If you must use VMWare, you can present datastores via NFS as well,
this has a lot of indirection but is easier to manage.
--
Alex Gorbachev
ISS Storcium
https://www.iss-integration.com <https://www.iss-integration.com>
On Thu, Apr 27, 2023 at 5:06 PM Angelo Höngens <angelo(a)hongens.nl
<mailto:angelo@hongens.nl>> wrote:
Hey guys and girls,
I'm working on a project to build storage for one of our departments,
and I want to ask you guys and girls for input on the high-level
overview part. It's a long one, I hope you read along and comment.
SUMMARY
I made a plan last year to build a 'storage solution' including ceph
and some windows VM's to expose the data over SMB to clients. A year
later I finally have the hardware, built a ceph cluster, and I'm
doing
tests. Ceph itself runs great, but when I
wanted to start exposing
the
data using iscsi to our VMware farm, I ran
into some issues. I know
the iscsi gateways will introduce some new performance bottlenecks,
but I'm seeing really slow performance, still working on that.
But then I ran into the warning on the iscsi gateway page: "The iSCSI
gateway is in maintenance as of November 2022. This means that it is
no longer in active development and will not be updated to add new
features.". Wait, what? Why!? What does this mean? Does this mean
that
iSCSI is now 'feature complete' and
will still be supported the next
5
years, or will it be deprecated in the
future? I tried searching, but
couldn't find any info on the decision and the roadmap.
My goal is to build a future-proof setup, and using deprecated
components should not be part of that of course.
If the iscsi gateway will still be supported the next few years and I
can iron out the performance issues, I can still go on with my
original plan. If not, I have to go back to the drawing board. And
maybe you guys would advise me to take another route anyway.
GOALS
My goals/considerations are:
- we want >1PB of storage capacity for cheap (on a tight budget) for
research data. Most of it is 'store once, read sometimes'. <1% of the
data is 'hot'.
- focus is on capacity, but it would be nice to have > 200MB/s of
sequential write/read performance and not 'totally suck' on random
i/o. Yes, not very well quantified, but ah. Sequential writes are
most
important.
- end users all run Windows computers (mostly VDI's) and a lot of
applications require SMB shares.
- security is a big thing, we want really tight ACL's, specific
monitoring agents, etc.
- our data is incredibly important to us, we still want the 3-2-1
backup rule. Primary storage solution, a second storage solution in a
different place, and some of the data that is not reproducible is
also
written to tape. We also want to be protected
from ransomware or user
errors (so no direct replication to the second storage).
- I like open source, reliability, no fork-lift upgrades, no vendor
lock-in, blah, well, I'm on the ceph list here, no need to convince
you guys ;)
- We're hiring a commercial company to do ceph maintenance and
support
for when I'm on leave or leaving the
company, but they won't support
clients, backup software, etc, so I want something as simple as
possible. We do have multiple Windows/VMware admins, but no other
real
linux guru's.
THE INITIAL PLAN
Given these considerations, I ordered two identical clusters, each
consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2
ssd's and 10 capacity disks (EC 4:2 for the data), and each node is
connected using a 2x25Gbps bond. Ceph is running like a charm. Now I
just have to think about exposing the data to end users, and I've
been
testing different setups.
My original plan was to expose for example 10x100TB rbd images using
iSCSI to our VMware farm, formatting the luns with VMFS6, and run for
example 2 Windows file servers per datastore on that with a single
DFS
namespace to end users. Then backup the file
servers using our
existing Veeam infrastructure to RGW running on the second cluster
with an immutable bucket. This way we would have easily defined
security boundaries: the clients can only reach the file servers, the
file servers only see their local VMDK's, ESX only sees the luns on
the iSCSI target, etc. When a file server would be compromised, it
would have no access to ceph. We have easy incremental backups,
immutability for ransomware protection, etc. And the best part is
that
the ceph admin can worry about ceph, the
vmware admin can focus on
ESX, VMFS and all the vmware stuff, and the Windows admins can focus
on the Windows boxes, Windows-specific ACLS and tools and Veeam
backups and stuff.
CURRENT SITUATION
I'm building out this plan now, but I'm running into issues with
iSCSI. Are any of you doing something similar? What is your iscsi
performance compared to direct rbd?
In regard to performance: If I take 2 test windows VM's, I put one on
an iSCSI datastore and another with direct rbd access using the
windows rbd driver, I create a share on those boxes and push data to
it, I see different results (of course). Copying some iso images over
SMB to the 'windows vm running direct rbd' I see around 800MB/s
write,
and 200MB/s read, which is pretty okay. When
I send data to the
'windows vm running on top of iscsi' it starts writing at around
350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go
faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really
acceptable.
Another really viable and performant scenario would be to have the
Windows file servers connect to rbd directly with the windows rbd
driver. It seems to work well, it's fast, and you don't have the
bottleneck that the iscsi gateway creates. But I see this driver is
still in beta. Is anyone using this in production? What are your
experiences? We would miss out on the separation of layers and thus
have less security, but at the same time, it really increases
efficiency and performance.
And if I use rbd, then vmware won't see the storage, and I cannot do
an image backup using veeam. I could of course do backups of the rbd
images, using tools like restic or backy to rgw running on the second
cluster with immutable buckets. What are your experiences? Is it easy
to do differential backups of lots of 50TB rbd images? Change rate is
usually like 0.005% per day or something ;)
By the way, we also thought about CephFS, but we have some complex
stuff going on with extended ACL's that I don't think will play nice
with CephFS, and I think it's a lot more complex to backup CephFS
than
block images.
If you made it here, thank you for your time! I hope you can share
thoughts on my questions!
Angelo.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
28 Apr
28 Apr
9:42 p.m.
Hello Angelo
You can try PetaSAN
www.petasan.org
We support scale out iscsi with Ceph and is actively developed.
/Maged
On 27/04/2023 23:05, Angelo Höngens wrote:
Hey guys and girls,
I'm working on a project to build storage for one of our departments,
and I want to ask you guys and girls for input on the high-level
overview part. It's a long one, I hope you read along and comment.
SUMMARY
I made a plan last year to build a 'storage solution' including ceph
and some windows VM's to expose the data over SMB to clients. A year
later I finally have the hardware, built a ceph cluster, and I'm doing
tests. Ceph itself runs great, but when I wanted to start exposing the
data using iscsi to our VMware farm, I ran into some issues. I know
the iscsi gateways will introduce some new performance bottlenecks,
but I'm seeing really slow performance, still working on that.
But then I ran into the warning on the iscsi gateway page: "The iSCSI
gateway is in maintenance as of November 2022. This means that it is
no longer in active development and will not be updated to add new
features.". Wait, what? Why!? What does this mean? Does this mean that
iSCSI is now 'feature complete' and will still be supported the next 5
years, or will it be deprecated in the future? I tried searching, but
couldn't find any info on the decision and the roadmap.
My goal is to build a future-proof setup, and using deprecated
components should not be part of that of course.
If the iscsi gateway will still be supported the next few years and I
can iron out the performance issues, I can still go on with my
original plan. If not, I have to go back to the drawing board. And
maybe you guys would advise me to take another route anyway.
GOALS
My goals/considerations are:
- we want >1PB of storage capacity for cheap (on a tight budget) for
research data. Most of it is 'store once, read sometimes'. <1% of the
data is 'hot'.
- focus is on capacity, but it would be nice to have > 200MB/s of
sequential write/read performance and not 'totally suck' on random
i/o. Yes, not very well quantified, but ah. Sequential writes are most
important.
- end users all run Windows computers (mostly VDI's) and a lot of
applications require SMB shares.
- security is a big thing, we want really tight ACL's, specific
monitoring agents, etc.
- our data is incredibly important to us, we still want the 3-2-1
backup rule. Primary storage solution, a second storage solution in a
different place, and some of the data that is not reproducible is also
written to tape. We also want to be protected from ransomware or user
errors (so no direct replication to the second storage).
- I like open source, reliability, no fork-lift upgrades, no vendor
lock-in, blah, well, I'm on the ceph list here, no need to convince
you guys ;)
- We're hiring a commercial company to do ceph maintenance and support
for when I'm on leave or leaving the company, but they won't support
clients, backup software, etc, so I want something as simple as
possible. We do have multiple Windows/VMware admins, but no other real
linux guru's.
THE INITIAL PLAN
Given these considerations, I ordered two identical clusters, each
consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2
ssd's and 10 capacity disks (EC 4:2 for the data), and each node is
connected using a 2x25Gbps bond. Ceph is running like a charm. Now I
just have to think about exposing the data to end users, and I've been
testing different setups.
My original plan was to expose for example 10x100TB rbd images using
iSCSI to our VMware farm, formatting the luns with VMFS6, and run for
example 2 Windows file servers per datastore on that with a single DFS
namespace to end users. Then backup the file servers using our
existing Veeam infrastructure to RGW running on the second cluster
with an immutable bucket. This way we would have easily defined
security boundaries: the clients can only reach the file servers, the
file servers only see their local VMDK's, ESX only sees the luns on
the iSCSI target, etc. When a file server would be compromised, it
would have no access to ceph. We have easy incremental backups,
immutability for ransomware protection, etc. And the best part is that
the ceph admin can worry about ceph, the vmware admin can focus on
ESX, VMFS and all the vmware stuff, and the Windows admins can focus
on the Windows boxes, Windows-specific ACLS and tools and Veeam
backups and stuff.
CURRENT SITUATION
I'm building out this plan now, but I'm running into issues with
iSCSI. Are any of you doing something similar? What is your iscsi
performance compared to direct rbd?
In regard to performance: If I take 2 test windows VM's, I put one on
an iSCSI datastore and another with direct rbd access using the
windows rbd driver, I create a share on those boxes and push data to
it, I see different results (of course). Copying some iso images over
SMB to the 'windows vm running direct rbd' I see around 800MB/s write,
and 200MB/s read, which is pretty okay. When I send data to the
'windows vm running on top of iscsi' it starts writing at around
350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go
faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really
acceptable.
Another really viable and performant scenario would be to have the
Windows file servers connect to rbd directly with the windows rbd
driver. It seems to work well, it's fast, and you don't have the
bottleneck that the iscsi gateway creates. But I see this driver is
still in beta. Is anyone using this in production? What are your
experiences? We would miss out on the separation of layers and thus
have less security, but at the same time, it really increases
efficiency and performance.
And if I use rbd, then vmware won't see the storage, and I cannot do
an image backup using veeam. I could of course do backups of the rbd
images, using tools like restic or backy to rgw running on the second
cluster with immutable buckets. What are your experiences? Is it easy
to do differential backups of lots of 50TB rbd images? Change rate is
usually like 0.005% per day or something ;)
By the way, we also thought about CephFS, but we have some complex
stuff going on with extended ACL's that I don't think will play nice
with CephFS, and I think it's a lot more complex to backup CephFS than
block images.
If you made it here, thank you for your time! I hope you can share
thoughts on my questions!
Angelo.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
393
days inactive
398
days old
8 comments
6 participants
participants (6)
-
Alex Gorbachev -
Angelo Hongens -
Angelo Höngens
-
Anthony D'Atri -
Bailey Allison -
Maged Mokhtar