Make rgw use tcmalloc/jemalloc
Hi folks, How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc? Thanks Yixin
On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks,
How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc?
how can you tell that it's still using libc? $ ldd bin/radosgw | grep tcmalloc libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000)
Thanks Yixin _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
the cmake variable ALLOCATOR can be used to request a specific library: https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420 rgw and other ceph binaries refer to ALLOC_LIBS to add that link dependency On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks,
How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc?
how can you tell that it's still using libc?
$ ldd bin/radosgw | grep tcmalloc libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000)
Thanks Yixin _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Because when I ran ldd, neither tcmalloc nor jemalloc showed up. And I used gdb to set a breakpoint at one of malloc in rgw. It showed it is calling __GI_libc_malloc().I basically followed the instruction in README.md to build ceph for vstart purpose. I wonder what I missed.
On Oct 20, 2023, at 2:29 PM, Casey Bodley <cbodley@redhat.com> wrote:
the cmake variable ALLOCATOR can be used to request a specific library: https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420
rgw and other ceph binaries refer to ALLOC_LIBS to add that link dependency
On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks,
How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc?
how can you tell that it's still using libc?
$ ldd bin/radosgw | grep tcmalloc libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000)
Thanks Yixin _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Yes, we tried ALLOCATOR as well. The result is the same. In fact, without specifying ALLOCATOR, cmake is able to find tcmalloc that is installed and use it. However, although libtcmalloc.so is listed in the “LINK_LIBRARIES” part of radosgw, it is somehow ignored until I moved boost/lib/libboost_thread.a to be in front of it. This change can be either manually done in build.ninja or by adding “Boost::thread” in front of $(ALLOC_LIBS) in src/rgw/CMakeLists.txt for radosgwd. Why the position of boost thread library makes a difference is beyond me. By the way, we are using boost 1.75 and gperftools 2.7. Thanks, Yixin
On Oct 20, 2023, at 2:29 PM, Casey Bodley <cbodley@redhat.com> wrote:
the cmake variable ALLOCATOR can be used to request a specific library: https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420
rgw and other ceph binaries refer to ALLOC_LIBS to add that link dependency
On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks,
How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc?
how can you tell that it's still using libc?
$ ldd bin/radosgw | grep tcmalloc libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000)
Thanks Yixin _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Casey, For standard ceph packages, are they built with gold or lld linker, instead of the GNU linker? I found that if gold is used, it doesn't suffer this problem of ignoring tcmalloc. Thanks,Yixin On Friday, October 20, 2023 at 02:30:15 p.m. EDT, Casey Bodley <cbodley@redhat.com> wrote: the cmake variable ALLOCATOR can be used to request a specific library: https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420 rgw and other ceph binaries refer to ALLOC_LIBS to add that link dependency On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks,
How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc?
how can you tell that it's still using libc?
$ ldd bin/radosgw | grep tcmalloc libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000)
Thanks Yixin _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
pretty sure we use the default gcc linker it's been a while since we've done performance comparisons between libc- and tcmalloc. it would be nice to revisit those and see whether tcmalloc is still necessary. if it is, we need some kind of regression test coverage to make sure our builds are using it On Mon, Oct 23, 2023 at 1:46 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi Casey,
For standard ceph packages, are they built with gold or lld linker, instead of the GNU linker? I found that if gold is used, it doesn't suffer this problem of ignoring tcmalloc.
Thanks, Yixin
On Friday, October 20, 2023 at 02:30:15 p.m. EDT, Casey Bodley <cbodley@redhat.com> wrote:
the cmake variable ALLOCATOR can be used to request a specific library: https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420
rgw and other ceph binaries refer to ALLOC_LIBS to add that link dependency
On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks,
How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc?
how can you tell that it's still using libc?
$ ldd bin/radosgw | grep tcmalloc libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000)
Thanks Yixin _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Mark Kogan has confirmed benefit from tcmalloc in at least some workloads quite recently, iirc. Matt On Mon, Oct 23, 2023 at 2:59 PM Casey Bodley <cbodley@redhat.com> wrote:
pretty sure we use the default gcc linker
it's been a while since we've done performance comparisons between libc- and tcmalloc. it would be nice to revisit those and see whether tcmalloc is still necessary. if it is, we need some kind of regression test coverage to make sure our builds are using it
On Mon, Oct 23, 2023 at 1:46 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi Casey,
For standard ceph packages, are they built with gold or lld linker,
instead of the GNU linker? I found that if gold is used, it doesn't suffer this problem of ignoring tcmalloc.
Thanks, Yixin
On Friday, October 20, 2023 at 02:30:15 p.m. EDT, Casey Bodley <
cbodley@redhat.com> wrote:
the cmake variable ALLOCATOR can be used to request a specific library:
https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420
rgw and other ceph binaries refer to ALLOC_LIBS to add that link
dependency
On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks,
How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc
installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc?
how can you tell that it's still using libc?
$ ldd bin/radosgw | grep tcmalloc libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000)
Thanks Yixin _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Maybe our build somehow confused gnu linker and made it ignore tcmalloc and used libc. Anyway,the libc (libc.so.6) seems to be problematic. We can easily cause rgw to OOM by blastinga flood of uploads, which doesn't happen when tcmalloc is linked. And this is with v17.2.5.So, performance aside, tcmalloc seems to be on the safer side. On Monday, October 23, 2023 at 02:58:06 p.m. EDT, Casey Bodley <cbodley@redhat.com> wrote: pretty sure we use the default gcc linker it's been a while since we've done performance comparisons between libc- and tcmalloc. it would be nice to revisit those and see whether tcmalloc is still necessary. if it is, we need some kind of regression test coverage to make sure our builds are using it On Mon, Oct 23, 2023 at 1:46 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi Casey,
For standard ceph packages, are they built with gold or lld linker, instead of the GNU linker? I found that if gold is used, it doesn't suffer this problem of ignoring tcmalloc.
Thanks, Yixin
On Friday, October 20, 2023 at 02:30:15 p.m. EDT, Casey Bodley <cbodley@redhat.com> wrote:
the cmake variable ALLOCATOR can be used to request a specific library: https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420
rgw and other ceph binaries refer to ALLOC_LIBS to add that link dependency
On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks,
How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc?
how can you tell that it's still using libc?
$ ldd bin/radosgw | grep tcmalloc libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000)
Thanks Yixin _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
I can't speak for RGW specifically, but we've done periodic tests over the years in other situations (primarily the OSD code) and we see pretty significant performance and memory usage improvements with tcmalloc. I think the most recent example was a test from last fall of client side performance in qemu+librbd: https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/ Simply LD_PRELOADing tcmalloc increased client side 4k randread performance from ~53.5K IOPS to ~80K IOPS. Unless you've taken specific steps to improve memory allocation behavior yourselves, I suspect RGW will continue to benefit from tcmalloc as well. Mark On 10/23/23 13:57, Casey Bodley wrote:
pretty sure we use the default gcc linker
it's been a while since we've done performance comparisons between libc- and tcmalloc. it would be nice to revisit those and see whether tcmalloc is still necessary. if it is, we need some kind of regression test coverage to make sure our builds are using it
On Mon, Oct 23, 2023 at 1:46 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi Casey,
For standard ceph packages, are they built with gold or lld linker, instead of the GNU linker? I found that if gold is used, it doesn't suffer this problem of ignoring tcmalloc.
Thanks, Yixin
On Friday, October 20, 2023 at 02:30:15 p.m. EDT, Casey Bodley <cbodley@redhat.com> wrote:
the cmake variable ALLOCATOR can be used to request a specific library: https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420
rgw and other ceph binaries refer to ALLOC_LIBS to add that link dependency
On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77@yahoo.ca> wrote:
Hi folks,
How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc?
how can you tell that it's still using libc?
$ ldd bin/radosgw | grep tcmalloc libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000)
Thanks Yixin _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (4)
-
Casey Bodley
-
Mark Nelson
-
Matt Benjamin
-
Yixin Jin