Because when I ran ldd, neither tcmalloc nor jemalloc showed up. And I used gdb to set a breakpoint at one of malloc in rgw. It showed it is calling __GI_libc_malloc().I basically followed the instruction in README.md to build ceph for vstart purpose. I wonder what I missed.

Hi Casey, For standard ceph packages, are they built with gold or lld linker, instead of the GNU linker? I found that if gold is used, it doesn't suffer this problem of ignoring tcmalloc. Thanks,Yixin On Friday, October 20, 2023 at 02:30:15 p.m. EDT, Casey Bodley <cbodley(a)redhat.com> wrote: the cmake variable ALLOCATOR can be used to request a specific library: https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420 rgw and other ceph binaries refer to ALLOC_LIBS to add that link dependency On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley(a)redhat.com> wrote: _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

Mark Kogan has confirmed benefit from tcmalloc in at least some workloads quite recently, iirc. Matt On Mon, Oct 23, 2023 at 2:59 PM Casey Bodley <cbodley(a)redhat.com> wrote: -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

I can't speak for RGW specifically, but we've done periodic tests over the years in other situations (primarily the OSD code) and we see pretty significant performance and memory usage improvements with tcmalloc. I think the most recent example was a test from last fall of client side performance in qemu+librbd: https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/ Simply LD_PRELOADing tcmalloc increased client side 4k randread performance from ~53.5K IOPS to ~80K IOPS. Unless you've taken specific steps to improve memory allocation behavior yourselves, I suspect RGW will continue to benefit from tcmalloc as well. Mark On 10/23/23 13:57, Casey Bodley wrote: > pretty sure we use the default gcc linker > > it's been a while since we've done performance comparisons between > libc- and tcmalloc. it would be nice to revisit those and see whether > tcmalloc is still necessary. if it is, we need some kind of regression > test coverage to make sure our builds are using it > > On Mon, Oct 23, 2023 at 1:46 PM Yixin Jin <yjin77(a)yahoo.ca> wrote: >> >> Hi Casey, >> >> For standard ceph packages, are they built with gold or lld linker, instead of the GNU linker? I found that if gold is used, it doesn't suffer this problem of ignoring tcmalloc. >> >> Thanks, >> Yixin >> >> On Friday, October 20, 2023 at 02:30:15 p.m. EDT, Casey Bodley <cbodley(a)redhat.com> wrote: >> >> >> the cmake variable ALLOCATOR can be used to request a specific >> library: https://github.com/ceph/ceph/blob/f080406/CMakeLists.txt#L376-L420 >> >> rgw and other ceph binaries refer to ALLOC_LIBS to add that link dependency >> >> On Fri, Oct 20, 2023 at 2:20 PM Casey Bodley <cbodley(a)redhat.com> wrote: >>> >>> On Fri, Oct 20, 2023 at 1:54 PM Yixin Jin <yjin77(a)yahoo.ca> wrote: >>>> >>>> Hi folks, >>>> >>>> How could I make rgw built with tcmalloc/jemalloc? I had tcmalloc installed. Although build.ninja has libtcmalloc.so listed for bin/radosgw, it is still using libc’s malloc. Then I had jemalloc installed and the result is the same. What is the proper step to make rgw built with tcmalloc/jemalloc? >>> >>> how can you tell that it's still using libc? >>> >>> $ ldd bin/radosgw | grep tcmalloc >>> libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f0bd3000000) >>> >>>> >>>> Thanks >>>> Yixin >>>> _______________________________________________ >>>> Dev mailing list -- dev(a)ceph.io >>>> To unsubscribe send an email to dev-leave(a)ceph.io >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io