11 Mar
2020
11 Mar
'20
7:46 p.m.
I have a hybrid environment and need to share with both Linux and
Windows clients. For my previous iterations of file storage, I exported
nfs and samba shares directly from my monolithic file server. All Linux
clients used nfs and all Windows clients used samba. Now that I've
switched to ceph, things are a bit more complicated. I built a gateway
to export nfs and samba as needed, and connect that as a client to my
ceph cluster.
After having file locking problems with kernel nfs, I made the switch to
nfs-ganesha, which has helped immensely. For Linux clients that have
high I/O needs, like desktops and some web servers, I connect to ceph
directly for those shares. For all other Linux needs, I use nfs from the
gateway. For all Windows clients (desktops and a small number of
servers), I use samba exported from the gateway.
Since my ceph cluster went live in August, I have had some kind of
strange (to me) error at least once a week, almost always related to the
gateway client. Last night, it was MDS_CLIENT_OLDEST_TID. Since we're on
Spring Break at my university and not very busy, I decided to
unmount/remount the ceph share, requiring stopping nfs and samba
services. Stopping nfs-ganesha took a while, but it finally completed
with no complaints from the ceph cluster. Stopping samba took longer and
gave me MDS_SLOW_REQUEST and MDS_CLIENT_LATE_RELEASE on the mds. It
finally finished, and I was able to unmount/remount the ceph share and
that finally cleared all the errors.
This is leading me to believe that samba on the gateway and all the
clients attaching to that is putting a strain on the connection back to
ceph. Which finally brings me to my question: is there a better way to
export samba to my clients using the ceph back end? Or is this as good
as it gets and I just have to put up with the seemingly frequent errors?
I can live with the errors and have been able to handle them so far, but
I know people who have much bigger clusters and many more clients than
me (by an order of magnitude) and don't see nearly as many errors as I
do. Which is why I'm trying to figure out what is special about my setup.
All my ceph nodes are running latest nautilus on Centos 7 (I just
updated last week to 14.2.8), as is the gateway host. I'm mounting ceph
directly on the gateway (by way of the kernel using cephfs, not
rados/rbd) to a single mount point and exporting from there.
My searches so far have not turned up anything extraordinarily useful,
so I'm asking for some guidance here. Any advice is welcome.
Thanks.
Seth
--
Seth Galitzer
Systems Coordinator
Computer Science Department
Kansas State University
http://www.cs.ksu.edu/~sgsax
sgsax(a)ksu.edu
785-532-7790
13 Mar
13 Mar
9:25 a.m.
On 3/11/20 11:16 PM, Seth Galitzer wrote:
I have a hybrid environment and need to share with
both Linux and
Windows clients. For my previous iterations of file storage, I
exported nfs and samba shares directly from my monolithic file server.
All Linux clients used nfs and all Windows clients used samba. Now
that I've switched to ceph, things are a bit more complicated. I built
a gateway to export nfs and samba as needed, and connect that as a
client to my ceph cluster.
After having file locking problems with kernel nfs, I made the switch
to nfs-ganesha, which has helped immensely. For Linux clients that
have high I/O needs, like desktops and some web servers, I connect to
ceph directly for those shares. For all other Linux needs, I use nfs
from the gateway. For all Windows clients (desktops and a small number
of servers), I use samba exported from the gateway.
Since my ceph cluster went live in August, I have had some kind of
strange (to me) error at least once a week, almost always related to
the gateway client. Last night, it was MDS_CLIENT_OLDEST_TID. Since
we're on Spring Break at my university and not very busy, I decided to
unmount/remount the ceph share, requiring stopping nfs and samba
services. Stopping nfs-ganesha took a while, but it finally completed
with no complaints from the ceph cluster. Stopping samba took longer
and gave me MDS_SLOW_REQUEST and MDS_CLIENT_LATE_RELEASE on the mds.
It finally finished, and I was able to unmount/remount the ceph share
and that finally cleared all the errors.
This is leading me to believe that samba on the gateway and all the
clients attaching to that is putting a strain on the connection back
to ceph. Which finally brings me to my question: is there a better way
to export samba to my clients using the ceph back end? Or is this as
good as it gets and I just have to put up with the seemingly frequent
errors? I can live with the errors and have been able to handle them
so far, but I know people who have much bigger clusters and many more
clients than me (by an order of magnitude) and don't see nearly as
many errors as I do. Which is why I'm trying to figure out what is
special about my setup.
All my ceph nodes are running latest nautilus on Centos 7 (I just
updated last week to 14.2.8), as is the gateway host. I'm mounting
ceph directly on the gateway (by way of the kernel using cephfs, not
rados/rbd) to a single mount point and exporting from there.
My searches so far have not turned up anything extraordinarily useful,
so I'm asking for some guidance here. Any advice is welcome.
You can connect to your cluster directly from userland, without kernel.
Use Samba vfs_ceph for this.
k
9:56 a.m.
Hi Seth,
I don't know if this helps you, but I'll share what we do. We present a
large amount of CephFS using NFS and SMB and a handful of cephfs direct
clients, and rarely encounter issues with either frontends or CephFS.
However, the 'gateway' is multiple servers - we use 2x ganesha servers with
ceph FSAL (2 of them) to export NFS. At this stage nothing fancy (no HA),
but will certainly look to do that sometime this year. I actually expect
you get better NFS perf exporting using VFS FSAL with kernel mounted cephfs.
SMB is same as yours, kernel cephfs but mounted on several machines each
running samba, coordinated using a CTDB cluster, and rr DNS for client
connections.
We don't mix smb and nfs dirs (no one has asked yet), so i'm not sure how
nice this would play. But it is likely to work/not work as well as any
other FS.
There are some tweaks particularly in ganesha if you are using the ceph
FSAL, like disabling caching (see
https://github.com/nfs-ganesha/nfs-ganesha/blob/master/src/config_samples/c…
)
It's hard to comment without knowing more details about your setup, but it
sounds like you are doing NFS and SMB from the same machine, I'd consider
splitting up those functions, and sizing the servers appropriately. And
most importantly, make sure your MDS has lots of RAM and the metadata pool
is on flash only.
Raf
On Thu, 12 Mar 2020 at 03:17, Seth Galitzer <sgsax(a)ksu.edu> wrote:
I have a hybrid environment and need to share with
both Linux and
Windows clients. For my previous iterations of file storage, I exported
nfs and samba shares directly from my monolithic file server. All Linux
clients used nfs and all Windows clients used samba. Now that I've
switched to ceph, things are a bit more complicated. I built a gateway
to export nfs and samba as needed, and connect that as a client to my
ceph cluster.
After having file locking problems with kernel nfs, I made the switch to
nfs-ganesha, which has helped immensely. For Linux clients that have
high I/O needs, like desktops and some web servers, I connect to ceph
directly for those shares. For all other Linux needs, I use nfs from the
gateway. For all Windows clients (desktops and a small number of
servers), I use samba exported from the gateway.
Since my ceph cluster went live in August, I have had some kind of
strange (to me) error at least once a week, almost always related to the
gateway client. Last night, it was MDS_CLIENT_OLDEST_TID. Since we're on
Spring Break at my university and not very busy, I decided to
unmount/remount the ceph share, requiring stopping nfs and samba
services. Stopping nfs-ganesha took a while, but it finally completed
with no complaints from the ceph cluster. Stopping samba took longer and
gave me MDS_SLOW_REQUEST and MDS_CLIENT_LATE_RELEASE on the mds. It
finally finished, and I was able to unmount/remount the ceph share and
that finally cleared all the errors.
This is leading me to believe that samba on the gateway and all the
clients attaching to that is putting a strain on the connection back to
ceph. Which finally brings me to my question: is there a better way to
export samba to my clients using the ceph back end? Or is this as good
as it gets and I just have to put up with the seemingly frequent errors?
I can live with the errors and have been able to handle them so far, but
I know people who have much bigger clusters and many more clients than
me (by an order of magnitude) and don't see nearly as many errors as I
do. Which is why I'm trying to figure out what is special about my setup.
All my ceph nodes are running latest nautilus on Centos 7 (I just
updated last week to 14.2.8), as is the gateway host. I'm mounting ceph
directly on the gateway (by way of the kernel using cephfs, not
rados/rbd) to a single mount point and exporting from there.
My searches so far have not turned up anything extraordinarily useful,
so I'm asking for some guidance here. Any advice is welcome.
Thanks.
Seth
--
Seth Galitzer
Systems Coordinator
Computer Science Department
Kansas State University
http://www.cs.ksu.edu/~sgsax
sgsax(a)ksu.edu
785-532-7790
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
*Rafael Lopez*
Devops Systems Engineer
Monash University eResearch Centre
T: +61 3 9905 9118
M: +61 (0)427682670 <%2B61%204%2027682%20670>
E: rafael.lopez(a)monash.edu
3:35 p.m.
Note that we have had issues with deadlocks when re-exporting CephFS
via Samba. It appears to only occur with Mac clients, though. In some
cases it has hung on a request for a high-level directory and hung
that branch for all clients.
On Fri, Mar 13, 2020 at 1:56 AM Konstantin Shalygin <k0ste(a)k0ste.ru> wrote:
>
>
> On 3/11/20 11:16 PM, Seth Galitzer wrote:
> > I have a hybrid environment and need to share with both Linux and
> > Windows clients. For my previous iterations of file storage, I
> > exported nfs and samba shares directly from my monolithic file server.
> > All Linux clients used nfs and all Windows clients used samba. Now
> > that I've switched to ceph, things are a bit more complicated. I built
> > a gateway to export nfs and samba as needed, and connect that as a
> > client to my ceph cluster.
> >
> > After having file locking problems with kernel nfs, I made the switch
> > to nfs-ganesha, which has helped immensely. For Linux clients that
> > have high I/O needs, like desktops and some web servers, I connect to
> > ceph directly for those shares. For all other Linux needs, I use nfs
> > from the gateway. For all Windows clients (desktops and a small number
> > of servers), I use samba exported from the gateway.
> >
> > Since my ceph cluster went live in August, I have had some kind of
> > strange (to me) error at least once a week, almost always related to
> > the gateway client. Last night, it was MDS_CLIENT_OLDEST_TID. Since
> > we're on Spring Break at my university and not very busy, I decided to
> > unmount/remount the ceph share, requiring stopping nfs and samba
> > services. Stopping nfs-ganesha took a while, but it finally completed
> > with no complaints from the ceph cluster. Stopping samba took longer
> > and gave me MDS_SLOW_REQUEST and MDS_CLIENT_LATE_RELEASE on the mds.
> > It finally finished, and I was able to unmount/remount the ceph share
> > and that finally cleared all the errors.
> >
> > This is leading me to believe that samba on the gateway and all the
> > clients attaching to that is putting a strain on the connection back
> > to ceph. Which finally brings me to my question: is there a better way
> > to export samba to my clients using the ceph back end? Or is this as
> > good as it gets and I just have to put up with the seemingly frequent
> > errors? I can live with the errors and have been able to handle them
> > so far, but I know people who have much bigger clusters and many more
> > clients than me (by an order of magnitude) and don't see nearly as
> > many errors as I do. Which is why I'm trying to figure out what is
> > special about my setup.
> >
> > All my ceph nodes are running latest nautilus on Centos 7 (I just
> > updated last week to 14.2.8), as is the gateway host. I'm mounting
> > ceph directly on the gateway (by way of the kernel using cephfs, not
> > rados/rbd) to a single mount point and exporting from there.
> >
> > My searches so far have not turned up anything extraordinarily useful,
> > so I'm asking for some guidance here. Any advice is welcome.
>
> You can connect to your cluster directly from userland, without kernel.
> Use Samba vfs_ceph for this.
>
>
>
> k
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:16 p.m.
Hello,
we have a CTDB based HA Samba in our Ceph Management Solution.
It works like a charm and we connect it to existing active directories as
well.
It's based on vfs_ceph and you can read more about how to configure it
yourself on
https://www.samba.org/samba/docs/current/man-html/vfs_ceph.8.html.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Fr., 13. März 2020 um 13:06 Uhr schrieb Nathan Fish <lordcirth(a)gmail.com
:
Note that we have had issues with deadlocks when
re-exporting CephFS
via Samba. It appears to only occur with Mac clients, though. In some
cases it has hung on a request for a high-level directory and hung
that branch for all clients.
On Fri, Mar 13, 2020 at 1:56 AM Konstantin Shalygin <k0ste(a)k0ste.ru>
wrote:
On 3/11/20 11:16 PM, Seth Galitzer wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I have a hybrid environment and need to share
with both Linux and
Windows clients. For my previous iterations of file storage, I
exported nfs and samba shares directly from my monolithic file server.
All Linux clients used nfs and all Windows clients used samba. Now
that I've switched to ceph, things are a bit more complicated. I built
a gateway to export nfs and samba as needed, and connect that as a
client to my ceph cluster.
After having file locking problems with kernel nfs, I made the switch
to nfs-ganesha, which has helped immensely. For Linux clients that
have high I/O needs, like desktops and some web servers, I connect to
ceph directly for those shares. For all other Linux needs, I use nfs
from the gateway. For all Windows clients (desktops and a small number
of servers), I use samba exported from the gateway.
Since my ceph cluster went live in August, I have had some kind of
strange (to me) error at least once a week, almost always related to
the gateway client. Last night, it was MDS_CLIENT_OLDEST_TID. Since
we're on Spring Break at my university and not very busy, I decided to
unmount/remount the ceph share, requiring stopping nfs and samba
services. Stopping nfs-ganesha took a while, but it finally completed
with no complaints from the ceph cluster. Stopping samba took longer
and gave me MDS_SLOW_REQUEST and MDS_CLIENT_LATE_RELEASE on the mds.
It finally finished, and I was able to unmount/remount the ceph share
and that finally cleared all the errors.
This is leading me to believe that samba on the gateway and all the
clients attaching to that is putting a strain on the connection back
to ceph. Which finally brings me to my question: is there a better way
to export samba to my clients using the ceph back end? Or is this as
good as it gets and I just have to put up with the seemingly frequent
errors? I can live with the errors and have been able to handle them
so far, but I know people who have much bigger clusters and many more
clients than me (by an order of magnitude) and don't see nearly as
many errors as I do. Which is why I'm trying to figure out what is
special about my setup.
All my ceph nodes are running latest nautilus on Centos 7 (I just
updated last week to 14.2.8), as is the gateway host. I'm mounting
ceph directly on the gateway (by way of the kernel using cephfs, not
rados/rbd) to a single mount point and exporting from there.
My searches so far have not turned up anything extraordinarily useful,
so I'm asking for some guidance here. Any advice is welcome.
You can connect to your cluster directly from userland, without kernel.
Use Samba vfs_ceph for this.
k
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
5:19 p.m.
Can you also create snapshots via the vfs_ceph solution?
-----Original Message-----
Sent: 13 March 2020 14:46
Subject: [ceph-users] Re: Is there a better way to make a samba/nfs
gateway?
Hello,
we have a CTDB based HA Samba in our Ceph Management Solution.
It works like a charm and we connect it to existing active directories
as well.
It's based on vfs_ceph and you can read more about how to configure it
yourself on
https://www.samba.org/samba/docs/current/man-html/vfs_ceph.8.html.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht
Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
11:38 p.m.
Thanks to all who have offered advise on this. I have been looking at
using vfs_ceph in samba, but I'm unsure how to get it on Centos7. As I
understand it, it's optional at compile time. When searching for a
package for it, I see one glusterfs (samba-vfs-glusterfs), but nothing
for ceph. Is it just enabled in the Centos samba package, or will I have
to compile my own samba binaries?
Thanks.
Seth
On 3/13/20 8:46 AM, Martin Verges wrote:
Hello,
we have a CTDB based HA Samba in our Ceph Management Solution.
It works like a charm and we connect it to existing active
directories as well.
It's based on vfs_ceph and you can read more about how to configure it
yourself on
https://www.samba.org/samba/docs/current/man-html/vfs_ceph.8.html.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io>
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Fr., 13. März 2020 um 13:06 Uhr schrieb Nathan Fish
<lordcirth(a)gmail.com <mailto:lordcirth@gmail.com>>:
Note that we have had issues with deadlocks when re-exporting CephFS
via Samba. It appears to only occur with Mac clients, though. In some
cases it has hung on a request for a high-level directory and hung
that branch for all clients.
On Fri, Mar 13, 2020 at 1:56 AM Konstantin Shalygin <k0ste(a)k0ste.ru
<mailto:k0ste@k0ste.ru>> wrote:
kernel.
--
Seth Galitzer
Systems Coordinator
Computing and Information Sciences
Kansas State University
sgsax(a)ksu.edu
785-532-7790
On 3/11/20 11:16 PM, Seth Galitzer wrote:
> I have a hybrid environment and need to share with both Linux and
> Windows clients. For my previous iterations of file storage, I
> exported nfs and samba shares directly from my monolithic file
server.
> All Linux clients used nfs and all Windows
clients used samba. Now
> that I've switched to ceph, things are a bit more complicated.
I
built
> a gateway to export nfs and samba as needed,
and connect that as a
> client to my ceph cluster.
>
> After having file locking problems with kernel nfs, I made the
switch
> to nfs-ganesha, which has helped immensely.
For Linux clients that
> have high I/O needs, like desktops and some web servers, I
connect to
> ceph directly for those shares. For all
other Linux needs, I
use nfs
> from the gateway. For all Windows clients
(desktops and a small
number
> of servers), I use samba exported from the
gateway.
>
> Since my ceph cluster went live in August, I have had some kind of
> strange (to me) error at least once a week, almost always
related to
> the gateway client. Last night, it was
MDS_CLIENT_OLDEST_TID. Since
> we're on Spring Break at my university and not very busy, I
decided
to
> unmount/remount the ceph share, requiring
stopping nfs and samba
> services. Stopping nfs-ganesha took a while, but it finally
completed
> with no complaints from the ceph cluster.
Stopping samba took
longer
> and gave me MDS_SLOW_REQUEST and
MDS_CLIENT_LATE_RELEASE on the
mds.
> It finally finished, and I was able to
unmount/remount the ceph
share
> and that finally cleared all the errors.
>
> This is leading me to believe that samba on the gateway and all the
> clients attaching to that is putting a strain on the connection
back
> to ceph. Which finally brings me to my
question: is there a
better way
> to export samba to my clients using the ceph
back end? Or is
this as
> good as it gets and I just have to put up
with the seemingly
frequent
> errors? I can live with the errors and have
been able to handle
them
> so far, but I know people who have much
bigger clusters and
many more
> clients than me (by an order of magnitude)
and don't see nearly as
> many errors as I do. Which is why I'm trying to figure out what is
> special about my setup.
>
> All my ceph nodes are running latest nautilus on Centos 7 (I just
> updated last week to 14.2.8), as is the gateway host. I'm mounting
> ceph directly on the gateway (by way of the kernel using
cephfs, not
> rados/rbd) to a single mount point and
exporting from there.
>
> My searches so far have not turned up anything extraordinarily
useful,
so
I'm asking for some guidance here. Any advice is welcome.
You can connect to your cluster directly from userland, without Use Samba vfs_ceph for this.
k
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
7:57 a.m.
On 3/14/20 3:08 AM, Seth Galitzer wrote:
Thanks to all who have offered advise on this. I have
been looking at
using vfs_ceph in samba, but I'm unsure how to get it on Centos7. As I
understand it, it's optional at compile time. When searching for a
package for it, I see one glusterfs (samba-vfs-glusterfs), but nothing
for ceph. Is it just enabled in the Centos samba package, or will I
have to compile my own samba binaries?
This work [1] was made by vincent <vincent(a)cojot.name> you should build
packages by yourself with cephfs and ctdb flags.
[1] https://nova.polymtl.ca/~coyote/dist/samba/
k
12:40 p.m.
On 3/16/20 5:21 AM, Konstantin Shalygin wrote:
On 3/13/20 8:49 PM, Marc Roos wrote:
Just out of curiosity: We are currently running a samba server with RBD
disks as a VM on our proxmox/ceph cluster.
I see the advantage of having vfs_ceph_snapshots of the samba user-data.
But then again: re-sharing data using samba vfs_ceph adds a layer of
complexity to the setup.
Anyone here running samba with vfs_ceph? Experiences?
MJ
Can you also create snapshots via the vfs_ceph
solution?
Yes! Since Samba 4.11 this supported via vfs_ceph_snapshots module.
1:38 p.m.
On 3/16/20 4:10 PM, mj wrote:
Just out of curiosity: We are currently running a samba server with
RBD disks as a VM on our proxmox/ceph cluster.
I see the advantage of having vfs_ceph_snapshots of the samba
user-data. But then again: re-sharing data using samba vfs_ceph adds a
layer of complexity to the setup.
vfs_ceph is not a layer of complexity, IMHO, moreover in vfs case is
need to be less prepared, e.g. your env don't care about kernel version,
mount options, etc.
Anyone here running samba with vfs_ceph? Experiences?
Actually many of production clusters use this setup.
k
5 Apr
5 Apr
4:17 a.m.
I think I may have cheated...
I setup the ceph iscsi gateway in HA mode, then a freenas server. Connected
the freenas server to the iscsi targets and poof, I have a universal NFS
share(s). I stood up a few freenas servers to share various loads. We also
use the iscsi gateways for direct esxi host connections ( but I will warn
you the esxi iscsi connections seem a little wonkie ). I havent upgraded to
the latest ceph yet, so perhaps more improvements to iscsi have been made
since this was done back in October. FreeNAS talks to both windows and
linux clients, no issues and securely.
Regards,
-Brent
Existing Clusters:
Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )
US Production(HDD): Nautilus 14.2.2 with 11 osd servers, 3 mons, 4 gateways,
2 iscsi gateways
UK Production(HDD): Nautilus 14.2.2 with 12 osd servers, 3 mons, 4 gateways
US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons, 3 gateways,
2 iscsi gateways
-----Original Message-----
From: Seth Galitzer <sgsax(a)ksu.edu>
Sent: Wednesday, March 11, 2020 12:16 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Is there a better way to make a samba/nfs gateway?
I have a hybrid environment and need to share with both Linux and Windows
clients. For my previous iterations of file storage, I exported nfs and
samba shares directly from my monolithic file server. All Linux clients used
nfs and all Windows clients used samba. Now that I've switched to ceph,
things are a bit more complicated. I built a gateway to export nfs and samba
as needed, and connect that as a client to my ceph cluster.
After having file locking problems with kernel nfs, I made the switch to
nfs-ganesha, which has helped immensely. For Linux clients that have high
I/O needs, like desktops and some web servers, I connect to ceph directly
for those shares. For all other Linux needs, I use nfs from the gateway. For
all Windows clients (desktops and a small number of servers), I use samba
exported from the gateway.
Since my ceph cluster went live in August, I have had some kind of strange
(to me) error at least once a week, almost always related to the gateway
client. Last night, it was MDS_CLIENT_OLDEST_TID. Since we're on Spring
Break at my university and not very busy, I decided to unmount/remount the
ceph share, requiring stopping nfs and samba services. Stopping nfs-ganesha
took a while, but it finally completed with no complaints from the ceph
cluster. Stopping samba took longer and gave me MDS_SLOW_REQUEST and
MDS_CLIENT_LATE_RELEASE on the mds. It finally finished, and I was able to
unmount/remount the ceph share and that finally cleared all the errors.
This is leading me to believe that samba on the gateway and all the clients
attaching to that is putting a strain on the connection back to ceph. Which
finally brings me to my question: is there a better way to export samba to
my clients using the ceph back end? Or is this as good as it gets and I just
have to put up with the seemingly frequent errors?
I can live with the errors and have been able to handle them so far, but I
know people who have much bigger clusters and many more clients than me (by
an order of magnitude) and don't see nearly as many errors as I do. Which is
why I'm trying to figure out what is special about my setup.
All my ceph nodes are running latest nautilus on Centos 7 (I just updated
last week to 14.2.8), as is the gateway host. I'm mounting ceph directly on
the gateway (by way of the kernel using cephfs, not
rados/rbd) to a single mount point and exporting from there.
My searches so far have not turned up anything extraordinarily useful, so
I'm asking for some guidance here. Any advice is welcome.
Thanks.
Seth
--
Seth Galitzer
Systems Coordinator
Computer Science Department
Kansas State University
http://www.cs.ksu.edu/~sgsax
sgsax(a)ksu.edu
785-532-7790
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email
to ceph-users-leave(a)ceph.io
1504
days inactive
1528
days old
11 comments
8 participants
participants (8)
-
Brent Kennedy -
Konstantin Shalygin -
Marc Roos -
Martin Verges -
mj -
Nathan Fish -
Rafael Lopez -
Seth Galitzer