17 Jun
2020
17 Jun
'20
7:39 p.m.
(I sent this to a smaller list of folks yesterday, but I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part
of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or
something).
It doesn't take very long to build it (typically only a minute or two on
my box), and we could disable the parts that ceph doesn't care about
(other FSALs primarily). We could also have vstart just error out when
you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build-
time dependencies, and would require some extra cmake or scripting
wizardry. Nothing insurmountable, but it might represent a maintenance
burden going forward, particularly for something that's basically only
going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do
optional submodules, so we might have to look at other methods of
pulling in the ganesha tree, or just live with it as a submodule that
only gets used when ganesha is enabled.
Thoughts?
--
Jeff Layton <jlayton(a)redhat.com>
17 Jun
17 Jun
7:46 p.m.
On Wed, Jun 17, 2020 at 10:09 AM Jeff Layton <jlayton(a)redhat.com> wrote:
(I sent this to a smaller list of folks yesterday, but I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
Is it not using stable APIs from those two shared libraries? i.e. QEMU
is compiled and linked against librbd/librados, but I can set
"LD_LIBRARY_PATH" to pull in development versions if needed.
I think we ought to consider making a nfs-ganesha
build an optional part
of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or
something).
It doesn't take very long to build it (typically only a minute or two on
my box), and we could disable the parts that ceph doesn't care about
(other FSALs primarily). We could also have vstart just error out when
you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build-
time dependencies, and would require some extra cmake or scripting
wizardry. Nothing insurmountable, but it might represent a maintenance
burden going forward, particularly for something that's basically only
going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do
optional submodules, so we might have to look at other methods of
pulling in the ganesha tree, or just live with it as a submodule that
only gets used when ganesha is enabled.
Thoughts?
--
Jeff Layton <jlayton(a)redhat.com>
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
--
Jason
8:12 p.m.
On Wed, 2020-06-17 at 10:16 -0400, Jason Dillaman wrote:
On Wed, Jun 17, 2020 at 10:09 AM Jeff Layton
<jlayton(a)redhat.com> wrote:
It is using stable APIs, and yeah that would be a simpler solution.
We do sometimes change how ganesha uses those APIs, or add new
interfaces that ganesha might call (and only conditionally build in),
but that's more rare and can be handled in other ways. Let me look into
just setting LD_LIBRARY_PATH.
Thanks,
--
Jeff Layton <jlayton(a)redhat.com>
(I sent this to a smaller list of folks
yesterday, but I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
Is it not using stable APIs from those two shared libraries? i.e. QEMU
is compiled and linked against librbd/librados, but I can set
"LD_LIBRARY_PATH" to pull in development versions if needed.
8:17 p.m.
On Wed, Jun 17, 2020 at 10:09 AM Jeff Layton <jlayton(a)redhat.com> wrote:
(I sent this to a smaller list of folks yesterday, but
I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part
of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or
something).
It doesn't take very long to build it (typically only a minute or two on
my box), and we could disable the parts that ceph doesn't care about
(other FSALs primarily). We could also have vstart just error out when
you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build-
time dependencies, and would require some extra cmake or scripting
wizardry. Nothing insurmountable, but it might represent a maintenance
burden going forward, particularly for something that's basically only
going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do
optional submodules, so we might have to look at other methods of
pulling in the ganesha tree, or just live with it as a submodule that
only gets used when ganesha is enabled.
libcephfs and librgw have semi-stable APIs by now. (And IIRC libcephfs has
versioned symbols.) So there's some risk that if a versioned symbol
changes that an old build of ganesha would continue to use an old API. But
one of the whole points of using versioned symbols — IMO — is to make
people think really hard about changing the API.
How often does it happen in reality that someone runs a ganesha built
against mimic/13 or nautilus/14 with octopus/15? I can't imagine it really
happens that often. And if it does, I think there's a different problem;
one that should be fixed.
I'm not really a git submodule and cmake expert, so I don't really know how
much work it would be to add ganesha and the ntirpc submodule into the ceph
build. And build them with all the right options for whatever the
particular environment is. I'm sure it's doable though.
It doesn't really affect ganesha. Ganesha packages will still get built
independent of whatever ceph is doing. I'd kinda hate to see there be
YAPTDGF (yet another place to download ganesha from.)
Personally I'd rather try to figure out proper solutions to the concern you
raise
--
Kaleb
18 Jun
18 Jun
2:29 p.m.
On 17-6-2020 16:09, Jeff Layton wrote:
(I sent this to a smaller list of folks yesterday, but
I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part
of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or
something).
It doesn't take very long to build it (typically only a minute or two on
my box), and we could disable the parts that ceph doesn't care about
(other FSALs primarily). We could also have vstart just error out when
you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build-
time dependencies, and would require some extra cmake or scripting
wizardry. Nothing insurmountable, but it might represent a maintenance
burden going forward, particularly for something that's basically only
going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do
optional submodules, so we might have to look at other methods of
pulling in the ganesha tree, or just live with it as a submodule that
only gets used when ganesha is enabled.
To my (big) surprise is there a FreeBSD package for nfs-ganesha so it
would be perhaps
a nice addon there as well.
Have never looked at it, so I don't know if it plays nice with the
native NFS implementation
that FreeBSD brings with its kernel. And it surely will require some
specific tinkering but
that is a nice challenge.
CMake wizardry is just a one-time obstacle, usually a not too big hurdle.
I would be certainly helped by a CMake switch that turns this off,
including all tests
that are added to test the nfs-ganesha implementation.
But like you say: Things are getting bigger and bigger.
--WjW
To my
11:27 p.m.
On 6/18/20 1:59 AM, Willem Jan Withagen wrote:
On 17-6-2020 16:09, Jeff Layton wrote:
I'm not sure what the current state of nfs-ganesha on FreeBSD is, but it
has had attention from various folks in the past. It would be good to
see more active maintenance of nfs-ganesha on FreeBSD.
Frank
(I sent this to a smaller list of folks
yesterday, but I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part
of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or
something).
It doesn't take very long to build it (typically only a minute or two on
my box), and we could disable the parts that ceph doesn't care about
(other FSALs primarily). We could also have vstart just error out when
you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build-
time dependencies, and would require some extra cmake or scripting
wizardry. Nothing insurmountable, but it might represent a maintenance
burden going forward, particularly for something that's basically only
going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do
optional submodules, so we might have to look at other methods of
pulling in the ganesha tree, or just live with it as a submodule that
only gets used when ganesha is enabled.
To my (big)Â surprise is there a FreeBSD package for nfs-ganesha so it
would be perhaps
a nice addon there as well.
Have never looked at it, so I don't know if it plays nice with the
native NFS implementation
that FreeBSD brings with its kernel. And it surely will require some
specific tinkering but
that is a nice challenge.
CMake wizardry is just a one-time obstacle, usually a not too big hurdle.
I would be certainly helped by a CMake switch that turns this off,
including all tests
that are added to test the nfs-ganesha implementation.
19 Jun
19 Jun
3:17 p.m.
On 18-6-2020 19:57, Frank Filz wrote:
On 6/18/20 1:59 AM, Willem Jan Withagen wrote:
As far as I can tell is the last work in nfs-ganesha done by the Gandi
people.
And the latest release they have in Ports is 2.7, with an matching
kernel module Port for not supported syscalls.
Problem there is that the port is based on Python 2.7, which is
deprecated on all ends.
That 2.7 ganesha version is already EOL according the data on GitHub.
Not sure if ceph-dashboard expects a more recent version?
--WjW
On 17-6-2020 16:09, Jeff Layton wrote:
I'm not sure what the current state of nfs-ganesha on FreeBSD is, but
it has had attention from various folks in the past. It would be good
to see more active maintenance of nfs-ganesha on FreeBSD. (I sent this to a smaller list of folks
yesterday, but I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right
off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional
part
of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or
something).
It doesn't take very long to build it (typically only a minute or
two on
my box), and we could disable the parts that ceph doesn't care about
(other FSALs primarily). We could also have vstart just error out when
you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build-
time dependencies, and would require some extra cmake or scripting
wizardry. Nothing insurmountable, but it might represent a maintenance
burden going forward, particularly for something that's basically only
going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you
can do
optional submodules, so we might have to look at other methods of
pulling in the ganesha tree, or just live with it as a submodule that
only gets used when ganesha is enabled.
To my (big)Â surprise is there a FreeBSD package for nfs-ganesha so
it would be perhaps
a nice addon there as well.
Have never looked at it, so I don't know if it plays nice with the
native NFS implementation
that FreeBSD brings with its kernel. And it surely will require some
specific tinkering but
that is a nice challenge.
CMake wizardry is just a one-time obstacle, usually a not too big
hurdle.
I would be certainly helped by a CMake switch that turns this off,
including all tests
that are added to test the nfs-ganesha implementation. 
12:45 p.m.
On Wed, Jun 17, 2020 at 10:09 PM Jeff Layton <jlayton(a)redhat.com> wrote:
(I sent this to a smaller list of folks yesterday, but I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part
of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or
something).
we tried to shorten the time running the "make check". that's why we
have qa/suites/rados/standalone tests. personally, i don't want to add
tests which cannot be categorized into unit test to "make check" even
if they only take less than 2 minutes to build.
probably we could add a task either performed by jenkins or by
teuthology for building and testing nfs-ganesha?
It doesn't take very long to build it (typically only a minute or two on
my box), and we could disable the parts that ceph doesn't care about
(other FSALs primarily). We could also have vstart just error out when
you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build-
time dependencies, and would require some extra cmake or scripting
wizardry. Nothing insurmountable, but it might represent a maintenance
burden going forward, particularly for something that's basically only
going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do
optional submodules, so we might have to look at other methods of
pulling in the ganesha tree, or just live with it as a submodule that
only gets used when ganesha is enabled.
adding ganesha as a subtree or a submodule does not make sense to me.
i see ganesha as a consumer of libcephfs and librados. in the long
run, if we go this way, the ceph repo will be bloated like a balloon.
not to mention, it's already very big now..
Thoughts?
--
Jeff Layton <jlayton(a)redhat.com>
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
--
Regards
Kefu Chai
3:06 p.m.
On 19-6-2020 09:15, kefu chai wrote:
On Wed, Jun 17, 2020 at 10:09 PM Jeff Layton
<jlayton(a)redhat.com> wrote:
Yup, that would be one of the worrying consequences.
I previously complained about the large volume of stuff we incorporate and
compile for the project.
What sort of tipped the balance in favor, was the "challenge" to keep
the libs
in sync with the ganesha part. And as I read it, forcing the user to
recompile
ganesha to get a match with the used versions.
Perhaps that {is, would be} fixed by versioning the libraries?
--WjW
(I sent this to a smaller list of folks
yesterday, but I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part
of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or
something).
we tried to shorten the time running the "make check".
that's why we
have qa/suites/rados/standalone tests. personally, i don't want to add
tests which cannot be categorized into unit test to "make check" even
if they only take less than 2 minutes to build.
probably we could add a task either performed by jenkins or by
teuthology for building and testing nfs-ganesha?
It doesn't take very long to build it
(typically only a minute or two on
my box), and we could disable the parts that ceph doesn't care about
(other FSALs primarily). We could also have vstart just error out when
you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build-
time dependencies, and would require some extra cmake or scripting
wizardry. Nothing insurmountable, but it might represent a maintenance
burden going forward, particularly for something that's basically only
going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do
optional submodules, so we might have to look at other methods of
pulling in the ganesha tree, or just live with it as a submodule that
only gets used when ganesha is enabled.
adding ganesha as a subtree or a submodule
does not make sense to me.
i see ganesha as a consumer of libcephfs and librados. in the long
run, if we go this way, the ceph repo will be bloated like a balloon.
not to mention, it's already very big now..
6:02 p.m.
New subject: [NFS-Ganesha-Devel] Re: should we make nfs-ganesha an optional part of a ceph build?
On Fri, 2020-06-19 at 15:15 +0800, kefu chai wrote:
Agreed. I decided to drop this idea for now. The better solution is to
just allow vstart to run ganesha from an alternate location (as Jason
sort of suggested). It turns out that we already do set LD_LIBRARY_PATH
in vstart, so we do use the libraries from the ceph build.
--
Jeff Layton <jlayton(a)redhat.com>
On Wed, Jun 17, 2020 at 10:09 PM Jeff Layton
<jlayton(a)redhat.com> wrote:
The main reason I wanted it was for local testing (a'la vstart). For
teuthology, we usually have rather up-to-date ganesha packages. That's
not always the case with distro packaging (consider Centos/RHEL).
(I sent this to a smaller list of folks
yesterday, but I think it
probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up
nfs-ganesha via vstart.sh. The current implementation requires that
ganesha already be installed on the box (usually via distro packaging),
but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely
different version of libcephfs and librados. Even if you build right off
of ceph master branch, you won't get the benefit of any recent client
bugfixes when you want to test ganesha. You'd have to build new ganesha
packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part
of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or
something).
we tried to shorten the time running the "make check". that's why we
have qa/suites/rados/standalone tests. personally, i don't want to add
tests which cannot be categorized into unit test to "make check" even
if they only take less than 2 minutes to build.
probably we could add a task either performed by jenkins or by
teuthology for building and testing nfs-ganesha?
It doesn't
take very long to build it (typically only a minute or two on
my box), and we could disable the parts that ceph doesn't care about
(other FSALs primarily). We could also have vstart just error out when
you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build-
time dependencies, and would require some extra cmake or scripting
wizardry. Nothing insurmountable, but it might represent a maintenance
burden going forward, particularly for something that's basically only
going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do
optional submodules, so we might have to look at other methods of
pulling in the ganesha tree, or just live with it as a submodule that
only gets used when ganesha is enabled.
adding ganesha as a subtree or a submodule does not make sense to me.
i see ganesha as a consumer of libcephfs and librados. in the long
run, if we go this way, the ceph repo will be bloated like a balloon.
not to mention, it's already very big now..
7:06 p.m.
New subject: [NFS-Ganesha-Devel] Re: should we make nfs-ganesha an optional part of a ceph build?
On Fri, Jun 19, 2020 at 8:33 AM Jeff Layton <jlayton(a)redhat.com> wrote:
The main reason I wanted it was for local testing (a'la vstart). For
teuthology, we usually have rather up-to-date ganesha packages. That's
not always the case with distro packaging (consider Centos/RHEL).
huh? The CentOS Storage SIG has the latest releases of ganesha. I make a
point of making sure that new releases are updated promptly there.
--
Kaleb
1407
days inactive
1409
days old
10 comments
6 participants
participants (6)
-
Frank Filz -
Jason Dillaman -
Jeff Layton -
Kaleb Keithley -
kefu chai -
Willem Jan Withagen