On Wed, Feb 12, 2020 at 3:33 AM Yiming Zhang <yzhan298(a)ucsc.edu> wrote: which librados was fio engine compiled with? the v2 msgr was introduced by nautilus (v14). also, would be better if you could avoid top-posting. -- Regards Kefu Chai

On Wed, Feb 12, 2020 at 2:06 PM Yiming Zhang <yzhan298(a)ucsc.edu> wrote: If you run "ldd" against your fio binary, which librados2.so is it picking up? If you have a 10.2.11 librados2.so on your system higher in your library search path, it will be used. -- Jason

On Wed, Feb 12, 2020 at 12:32:20PM -0800, Yiming Zhang wrote: ... ... You probably used the wrong ceph headers. The default ceph install is going to go into /usr/local. System packages, which on many modern distributions will include ceph dependencies, will go into /usr. If you've got both running around, you need to make sure that the headers you use at compile time match the libraries you use at runtime. A memory fault is a typical symptom when you botched this. You might be able to just de-install the system provided packages; but check first to see it doesn't remove something you want to keep. At build time, you might look for lines like these -I $H/include -L $H/lib -Wl,-rpath,$H/lib on the compile line(s) to point at headers and runtime libraries. Note that many modern build systems hide the actual compile commands by default. Sometimes can also use "LD_LIBRARY_PATH" at runtime to point at the libraries you want; but first use "objdump -p " to verify there's no "RPATH". If there is an rpath, you may be able to use "chrpath" to change it. -Marcus Watts

On 12:32 Wed 12 Feb, Yiming Zhang wrote: You can use "strace" tool to show the library loaded at run time. For example, below command will show all the loaded library at run time. $ strace fio --section=with_rwlcache changcheng.fio 2>&1 | tee strace.log $ grep '^open' strace.log | grep lib "make install" command may install the library under /usr/local directory, sometimes this will cause some error. 1. Enable fio run time debug option 2. Enable system core dump to get coredump file and check it. > > -ym > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

On 15:20 Thu 13 Feb, Yiming Zhang wrote: You can refer to below document: https://github.com/ceph/ceph/blob/v14.2.7/src/sample.ceph.conf BTW, it seems that you've sovled the problem in the email sent to mwatts(a)redhat.com Note: For Ceph 1. If there's no need to do development on Ceph, you can the Ceph released by Linux distribution vendor. 2. If you want to development on Ceph, prefer to: 1) Use vstart.sh environment. Or 2) Build Ceph source code into standard package to avoid build/link/load strange problem. Ceph has supplied the rule to build the source code into debian or rpm package format. For fio Prefer to build the source code into standard package. For example, I create below repository to build fio into debian/ubuntu package: https://github.com/changchengx/fio/tree/debian_tag_3.17 For rpm package, you need sovle it by your self. Cross validation is important to make clear whether the problem happen in fio or Ceph before digging further. B.R. Changcheng > > Thanks, > -ym > >> Yes, I can see the shared libs: > >> librbd.so.1 => /usr/lib/x86_64-linux-gnu/librbd.so.1 (0x00007fa3a4e4a000) > >> librados.so.2 => /usr/lib/x86_64-linux-gnu/librados.so.2 (0x00007fa3a4af9000) > >> > >> I recompiled ceph and make install all libs, and got memory issue with fio: > >> fio-3.17-116-gf4cd > >> Starting 1 process > >> Segmentation fault (core dumped) > > 1. Enable fio run time debug option > > 2. Enable system core dump to get coredump file and check it. > >> > >> -ym > >> _______________________________________________ > >> Dev mailing list -- dev(a)ceph.io > >> To unsubscribe send an email to dev-leave(a)ceph.io >