More details (and full backtrace) in https://tracker.ceph.com/issues/44570 We are able to reproduce this 100% of the time when trying to upload a large file, radowgw just segfaults about 50MB in. Here is the backtrace from the main thread, all threads are in the backtrace on the trac bug. There are a few different backtraces that we have seen...

Thread 1 (Thread 0x64a6e4bba700 (LWP 14952)): #0 raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x000013f502628651 in reraise_fatal (signum=11) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/global/signal_handler.cc:326 #2 handle_fatal_signal (signum=11) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/global/signal_handler.cc:326 #3 <signal handler called> #4 0x000064a6ef114963 in ceph::buffer::v14_2_0::list::buffers_t::clear_and_dispose (this=0x13f5082457b8) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/include/buffer.h:648 #5 ceph::buffer::v14_2_0::list::buffers_t::clone_from (other=..., this=0x13f5082457b8) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/include/buffer.h:639 #6 ceph::buffer::v14_2_0::list::operator= (other=..., this=0x13f5082457b8) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/include/buffer.h:1003 #7 ceph::buffer::v14_2_0::list::operator= (other=..., this=0x13f5082457b8) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/include/buffer.h:1000 #8 Objecter::handle_osd_op_reply (this=0x13f507c75700, m=0x13f50820eb00) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/osdc/Objecter.cc:3502 #9 0x000064a6ef115cab in Objecter::ms_dispatch (this=0x13f507c75700, m=0x13f50820eb00) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/osdc/Objecter.cc:966 #10 0x000064a6ef1347d7 in non-virtual thunk to Objecter::ms_fast_dispatch(Message*) () at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/osdc/Objecter.h:2110 #11 0x000064a6ef0e5440 in Dispatcher::ms_fast_dispatch2 (this=0x13f507c75708, m=...) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/common/RefCountedObj.h:41 #12 0x000064a6e67d530f in Messenger::ms_fast_dispatch (m=..., this=<optimized out>) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8_build/boost/include/boost/container/new_allocator.hpp:165 #13 DispatchQueue::fast_dispatch (this=0x13f506fcf558, m=...) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/DispatchQueue.cc:72 #14 0x000064a6e68fa75e in DispatchQueue::fast_dispatch (m=0x13f50820eb00, this=<optimized out>) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8_build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:67 #15 ProtocolV2::handle_message (this=<optimized out>) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/ProtocolV2.cc:1501 #16 0x000064a6e690cec8 in ProtocolV2::handle_read_frame_dispatch (this=0x13f5080fb700) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/ProtocolV2.cc:1128 #17 0x000064a6e690d245 in ProtocolV2::handle_read_frame_epilogue_main (this=0x13f5080fb700, buffer=..., r=<optimized out>) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/ProtocolV2.cc:1352 #18 0x000064a6e68f203f in ProtocolV2::run_continuation (this=0x13f5080fb700, continuation=...) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/ProtocolV2.cc:45 #19 0x000064a6e68b898e in std::function<void (char*, long)>::operator()(char*, long) const (__args#1=<optimized out>, __args#0=<optimized out>, this=0x13f508165610) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.2.0/include/g++-v9/bits/std_function.h:685 #20 AsyncConnection::process (this=0x13f508165200) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/AsyncConnection.cc:450 #21 0x000064a6e69181b8 in EventCenter::process_events (this=this@entry=0x13f506f53700, timeout_microseconds=<optimized out>, timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x64a6e4bb7d28) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.2.0/include/g++-v9/bits/basic_ios.h:282 #22 0x000064a6e691e25c in NetworkStack::<lambda()>::operator() (__closure=0x13f506f93808, __closure=0x13f506f93808) at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/Stack.cc:53 #23 std::_Function_handler<void(), NetworkStack::add_thread(unsigned int)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.2.0/include/g++-v9/bits/std_function.h:300 #24 0x000064a6e5ca2130 in std::execute_native_thread_routine (__p=0x13f506f93800) at /var/tmp/portage/sys-devel/gcc-9.2.0-r2/work/gcc-9.2.0/libstdc++-v3/src/c++11/thread.cc:80 #25 0x000064a6e5e2c482 in start_thread (arg=<optimized out>) at pthread_create.c:486 #26 0x000064a6e5abc5cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95