Hi All, I recently experienced an issue while trying to use FIO on Ceph(master branch). When Ceph is started, the mon host is something like [v2:128.XXX.XXX.XXX:12345, v1:128.XXX.XXX.XXX:12346] if I try to use the FIO now, the “addr cannot be parsed” error will occur. Then I change the mon host to 128.XXX.XXX.XXX:12345, ceph will also show a connection error too. I checked the IP and port, which are used correctly. RBD bench works fine, only FIO has this issue. Please advice. Thanks, Yiming
Which version of fio are you using? -Sam On Mon, Feb 10, 2020 at 5:24 AM Yiming Zhang <yzhan298@ucsc.edu> wrote:
Hi All,
I recently experienced an issue while trying to use FIO on Ceph(master branch). When Ceph is started, the mon host is something like [v2:128.XXX.XXX.XXX:12345, v1:128.XXX.XXX.XXX:12346] if I try to use the FIO now, the “addr cannot be parsed” error will occur.
Then I change the mon host to 128.XXX.XXX.XXX:12345, ceph will also show a connection error too. I checked the IP and port, which are used correctly.
RBD bench works fine, only FIO has this issue.
Please advice.
Thanks, Yiming
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Wed, Feb 12, 2020 at 3:33 AM Yiming Zhang <yzhan298@ucsc.edu> wrote:
The fio-3.17-116-gf4cd, compiled from source.
which librados was fio engine compiled with? the v2 msgr was introduced by nautilus (v14). also, would be better if you could avoid top-posting.
Available IO engines: splice rados rbd http
Thanks, Yiming
On Feb 11, 2020, at 11:03 AM, Sam Just <sjust@redhat.com> wrote:
version of fio are you us
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Regards Kefu Chai
On Feb 12, 2020, at 1:24 AM, kefu chai <tchaikov@gmail.com> wrote:
On Wed, Feb 12, 2020 at 3:33 AM Yiming Zhang <yzhan298@ucsc.edu> wrote:
The fio-3.17-116-gf4cd, compiled from source.
which librados was fio engine compiled with? the v2 msgr was introduced by nautilus (v14).
Compiled with both usr/bin/librados-config 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e) and v14.0.0-16058-geb4513a (eb4513a62aec3d80f739b9c1ff1374a559bf6b0f) octopus (dev) The problem persists. Maybe the we I compile fio is wrong? How should I specify the librados version in fio compilation? Thanks, Yiming
also, would be better if you could avoid top-posting.
Available IO engines: splice rados rbd http
Thanks, Yiming
On Feb 11, 2020, at 11:03 AM, Sam Just <sjust@redhat.com> wrote:
version of fio are you us
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Regards Kefu Chai
On Wed, Feb 12, 2020 at 2:06 PM Yiming Zhang <yzhan298@ucsc.edu> wrote:
On Feb 12, 2020, at 1:24 AM, kefu chai <tchaikov@gmail.com> wrote:
On Wed, Feb 12, 2020 at 3:33 AM Yiming Zhang <yzhan298@ucsc.edu> wrote:
The fio-3.17-116-gf4cd, compiled from source.
which librados was fio engine compiled with? the v2 msgr was introduced by nautilus (v14).
Compiled with both usr/bin/librados-config 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e) and v14.0.0-16058-geb4513a (eb4513a62aec3d80f739b9c1ff1374a559bf6b0f) octopus (dev) The problem persists.
If you run "ldd" against your fio binary, which librados2.so is it picking up? If you have a 10.2.11 librados2.so on your system higher in your library search path, it will be used.
Maybe the we I compile fio is wrong? How should I specify the librados version in fio compilation?
Thanks, Yiming
also, would be better if you could avoid top-posting.
Available IO engines: splice rados rbd http
Thanks, Yiming
On Feb 11, 2020, at 11:03 AM, Sam Just <sjust@redhat.com> wrote:
version of fio are you us
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Regards Kefu Chai
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason
On Feb 12, 2020, at 11:08 AM, Jason Dillaman <jdillama@redhat.com> wrote:
On Wed, Feb 12, 2020 at 2:06 PM Yiming Zhang <yzhan298@ucsc.edu> wrote:
On Feb 12, 2020, at 1:24 AM, kefu chai <tchaikov@gmail.com> wrote:
On Wed, Feb 12, 2020 at 3:33 AM Yiming Zhang <yzhan298@ucsc.edu> wrote:
The fio-3.17-116-gf4cd, compiled from source.
which librados was fio engine compiled with? the v2 msgr was introduced by nautilus (v14).
Compiled with both usr/bin/librados-config 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e) and v14.0.0-16058-geb4513a (eb4513a62aec3d80f739b9c1ff1374a559bf6b0f) octopus (dev) The problem persists.
If you run "ldd" against your fio binary, which librados2.so is it picking up? If you have a 10.2.11 librados2.so on your system higher in your library search path, it will be used.
Yes, I can see the shared libs: librbd.so.1 => /usr/lib/x86_64-linux-gnu/librbd.so.1 (0x00007fa3a4e4a000) librados.so.2 => /usr/lib/x86_64-linux-gnu/librados.so.2 (0x00007fa3a4af9000) I recompiled ceph and make install all libs, and got memory issue with fio: fio-3.17-116-gf4cd Starting 1 process Segmentation fault (core dumped) -ym
Maybe the we I compile fio is wrong? How should I specify the librados version in fio compilation?
Thanks, Yiming
also, would be better if you could avoid top-posting.
Available IO engines: splice rados rbd http
Thanks, Yiming
On Feb 11, 2020, at 11:03 AM, Sam Just <sjust@redhat.com> wrote:
version of fio are you us
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Regards Kefu Chai
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason
On Wed, Feb 12, 2020 at 12:32:20PM -0800, Yiming Zhang wrote: ...
librbd.so.1 => /usr/lib/x86_64-linux-gnu/librbd.so.1 (0x00007fa3a4e4a000) librados.so.2 => /usr/lib/x86_64-linux-gnu/librados.so.2 (0x00007fa3a4af9000)
I recompiled ceph and make install all libs, and got memory issue with fio: fio-3.17-116-gf4cd Starting 1 process Segmentation fault (core dumped)
-ym ...
You probably used the wrong ceph headers. The default ceph install is going to go into /usr/local. System packages, which on many modern distributions will include ceph dependencies, will go into /usr. If you've got both running around, you need to make sure that the headers you use at compile time match the libraries you use at runtime. A memory fault is a typical symptom when you botched this. You might be able to just de-install the system provided packages; but check first to see it doesn't remove something you want to keep. At build time, you might look for lines like these -I $H/include -L $H/lib -Wl,-rpath,$H/lib on the compile line(s) to point at headers and runtime libraries. Note that many modern build systems hide the actual compile commands by default. Sometimes can also use "LD_LIBRARY_PATH" at runtime to point at the libraries you want; but first use "objdump -p " to verify there's no "RPATH". If there is an rpath, you may be able to use "chrpath" to change it. -Marcus Watts
On Feb 12, 2020, at 1:27 PM, Marcus Watts <mwatts@redhat.com> wrote:
On Wed, Feb 12, 2020 at 12:32:20PM -0800, Yiming Zhang wrote: ...
librbd.so.1 => /usr/lib/x86_64-linux-gnu/librbd.so.1 (0x00007fa3a4e4a000) librados.so.2 => /usr/lib/x86_64-linux-gnu/librados.so.2 (0x00007fa3a4af9000)
I recompiled ceph and make install all libs, and got memory issue with fio: fio-3.17-116-gf4cd Starting 1 process Segmentation fault (core dumped)
-ym ...
You probably used the wrong ceph headers.
The default ceph install is going to go into /usr/local. System packages, which on many modern distributions will include ceph dependencies, will go into /usr. If you've got both running around, you need to make sure that the headers you use at compile time match the libraries you use at runtime. A memory fault is a typical symptom when you botched this.
That’s right. After separating the dependencies, the core dumped error goes away.
You might be able to just de-install the system provided packages; but check first to see it doesn't remove something you want to keep. At build time, you might look for lines like these -I $H/include -L $H/lib -Wl,-rpath,$H/lib on the compile line(s) to point at headers and runtime libraries. Note that many modern build systems hide the actual compile commands by default.
Sometimes can also use "LD_LIBRARY_PATH" at runtime to point at the libraries you want; but first use "objdump -p " to verify there's no "RPATH". If there is an rpath, you may be able to use "chrpath" to change it.
-Marcus Watts
On 12:32 Wed 12 Feb, Yiming Zhang wrote: > > On Wed, Feb 12, 2020 at 2:06 PM Yiming Zhang <yzhan298@ucsc.edu> wrote: > >>>> The fio-3.17-116-gf4cd, compiled from source. > >>> > >>> which librados was fio engine compiled with? the v2 msgr was > >>> introduced by nautilus (v14). > >> You can use "strace" tool to show the library loaded at run time. For example, below command will show all the loaded library at run time. $ strace fio --section=with_rwlcache changcheng.fio 2>&1 | tee strace.log $ grep '^open' strace.log | grep lib "make install" command may install the library under /usr/local directory, sometimes this will cause some error. > Yes, I can see the shared libs: > librbd.so.1 => /usr/lib/x86_64-linux-gnu/librbd.so.1 (0x00007fa3a4e4a000) > librados.so.2 => /usr/lib/x86_64-linux-gnu/librados.so.2 (0x00007fa3a4af9000) > > I recompiled ceph and make install all libs, and got memory issue with fio: > fio-3.17-116-gf4cd > Starting 1 process > Segmentation fault (core dumped) 1. Enable fio run time debug option 2. Enable system core dump to get coredump file and check it. > > -ym > _______________________________________________ > Dev mailing list -- dev@ceph.io > To unsubscribe send an email to dev-leave@ceph.io
> On Feb 12, 2020, at 5:19 PM, Liu, Changcheng <changcheng.liu@intel.com> wrote: > > On 12:32 Wed 12 Feb, Yiming Zhang wrote: >>> On Wed, Feb 12, 2020 at 2:06 PM Yiming Zhang <yzhan298@ucsc.edu> wrote: >>>>>> The fio-3.17-116-gf4cd, compiled from source. >>>>> >>>>> which librados was fio engine compiled with? the v2 msgr was >>>>> introduced by nautilus (v14). >>>> > You can use "strace" tool to show the library loaded at run time. > For example, below command will show all the loaded library at run time. > $ strace fio --section=with_rwlcache changcheng.fio 2>&1 | tee strace.log > $ grep '^open' strace.log | grep lib > "make install" command may install the library under /usr/local > directory, sometimes this will cause some error. > I tried to use strace, and the problem is in the rados_connection. If I switch to the ceph v12, the same fio works just fine. For v14, I noticed that `mon addr` is replaced with `mon host`. What’s the proper way to deal with it? Thanks, -ym >> Yes, I can see the shared libs: >> librbd.so.1 => /usr/lib/x86_64-linux-gnu/librbd.so.1 (0x00007fa3a4e4a000) >> librados.so.2 => /usr/lib/x86_64-linux-gnu/librados.so.2 (0x00007fa3a4af9000) >> >> I recompiled ceph and make install all libs, and got memory issue with fio: >> fio-3.17-116-gf4cd >> Starting 1 process >> Segmentation fault (core dumped) > 1. Enable fio run time debug option > 2. Enable system core dump to get coredump file and check it. >> >> -ym >> _______________________________________________ >> Dev mailing list -- dev@ceph.io >> To unsubscribe send an email to dev-leave@ceph.io
On 15:20 Thu 13 Feb, Yiming Zhang wrote:
>
>
> > On Feb 12, 2020, at 5:19 PM, Liu, Changcheng <changcheng.liu@intel.com> wrote:
> >
> > On 12:32 Wed 12 Feb, Yiming Zhang wrote:
> >>> On Wed, Feb 12, 2020 at 2:06 PM Yiming Zhang <yzhan298@ucsc.edu> wrote:
> >>>>>> The fio-3.17-116-gf4cd, compiled from source.
> >>>>>
> >>>>> which librados was fio engine compiled with? the v2 msgr was
> >>>>> introduced by nautilus (v14).
> >>>>
> > You can use "strace" tool to show the library loaded at run time.
> > For example, below command will show all the loaded library at run time.
> > $ strace fio --section=with_rwlcache changcheng.fio 2>&1 | tee strace.log
> > $ grep '^open' strace.log | grep lib
> > "make install" command may install the library under /usr/local
> > directory, sometimes this will cause some error.
> >
> I tried to use strace, and the problem is in the rados_connection.
> If I switch to the ceph v12, the same fio works just fine.
> For v14, I noticed that `mon addr` is replaced with `mon host`. What’s the
> proper way to deal with it?
You can refer to below document:
https://github.com/ceph/ceph/blob/v14.2.7/src/sample.ceph.conf
BTW, it seems that you've sovled the problem in the email sent to
mwatts@redhat.com
Note:
For Ceph
1. If there's no need to do development on Ceph, you can the Ceph
released by Linux distribution vendor.
2. If you want to development on Ceph, prefer to:
1) Use vstart.sh environment.
Or
2) Build Ceph source code into standard package to avoid
build/link/load strange problem.
Ceph has supplied the rule to build the source code into debian or
rpm package format.
For fio
Prefer to build the source code into standard package. For example, I
create below repository to build fio into debian/ubuntu package:
https://github.com/changchengx/fio/tree/debian_tag_3.17
For rpm package, you need sovle it by your self.
Cross validation is important to make clear whether the problem
happen in fio or Ceph before digging further.
B.R.
Changcheng
>
> Thanks,
> -ym
> >> Yes, I can see the shared libs:
> >> librbd.so.1 => /usr/lib/x86_64-linux-gnu/librbd.so.1 (0x00007fa3a4e4a000)
> >> librados.so.2 => /usr/lib/x86_64-linux-gnu/librados.so.2 (0x00007fa3a4af9000)
> >>
> >> I recompiled ceph and make install all libs, and got memory issue with fio:
> >> fio-3.17-116-gf4cd
> >> Starting 1 process
> >> Segmentation fault (core dumped)
> > 1. Enable fio run time debug option
> > 2. Enable system core dump to get coredump file and check it.
> >>
> >> -ym
> >> _______________________________________________
> >> Dev mailing list -- dev@ceph.io
> >> To unsubscribe send an email to dev-leave@ceph.io
>
participants (6)
-
Jason Dillaman
-
kefu chai
-
Liu, Changcheng
-
Marcus Watts
-
Sam Just
-
Yiming Zhang