12 Mar
2020
12 Mar
'20
3:16 a.m.
More details (and full backtrace) in https://tracker.ceph.com/issues/44570
We are able to reproduce this 100% of the time when trying to upload a large file, radowgw
just segfaults about 50MB in. Here is the backtrace from the main thread, all threads are
in the backtrace on the trac bug.
There are a few different backtraces that we have seen...
> Thread 1 (Thread 0x64a6e4bba700 (LWP 14952)):
> #0 raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:50
> #1 0x000013f502628651 in reraise_fatal (signum=11) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/global/signal_handler.cc:326
> #2 handle_fatal_signal (signum=11) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/global/signal_handler.cc:326
> #3 <signal handler called>
> #4 0x000064a6ef114963 in ceph::buffer::v14_2_0::list::buffers_t::clear_and_dispose
(this=0x13f5082457b8) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/include/buffer.h:648
> #5 ceph::buffer::v14_2_0::list::buffers_t::clone_from (other=...,
this=0x13f5082457b8) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/include/buffer.h:639
> #6 ceph::buffer::v14_2_0::list::operator= (other=..., this=0x13f5082457b8) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/include/buffer.h:1003
> #7 ceph::buffer::v14_2_0::list::operator= (other=..., this=0x13f5082457b8) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/include/buffer.h:1000
> #8 Objecter::handle_osd_op_reply (this=0x13f507c75700, m=0x13f50820eb00) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/osdc/Objecter.cc:3502
> #9 0x000064a6ef115cab in Objecter::ms_dispatch (this=0x13f507c75700,
m=0x13f50820eb00) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/osdc/Objecter.cc:966
> #10 0x000064a6ef1347d7 in non-virtual thunk to Objecter::ms_fast_dispatch(Message*)
() at /var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/osdc/Objecter.h:2110
> #11 0x000064a6ef0e5440 in Dispatcher::ms_fast_dispatch2 (this=0x13f507c75708, m=...)
at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/common/RefCountedObj.h:41
> #12 0x000064a6e67d530f in Messenger::ms_fast_dispatch (m=..., this=<optimized
out>) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8_build/boost/include/boost/container/new_allocator.hpp:165
> #13 DispatchQueue::fast_dispatch (this=0x13f506fcf558, m=...) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/DispatchQueue.cc:72
> #14 0x000064a6e68fa75e in DispatchQueue::fast_dispatch (m=0x13f50820eb00,
this=<optimized out>) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8_build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:67
> #15 ProtocolV2::handle_message (this=<optimized out>) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/ProtocolV2.cc:1501
> #16 0x000064a6e690cec8 in ProtocolV2::handle_read_frame_dispatch
(this=0x13f5080fb700) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/ProtocolV2.cc:1128
> #17 0x000064a6e690d245 in ProtocolV2::handle_read_frame_epilogue_main
(this=0x13f5080fb700, buffer=..., r=<optimized out>) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/ProtocolV2.cc:1352
> #18 0x000064a6e68f203f in ProtocolV2::run_continuation (this=0x13f5080fb700,
continuation=...) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/ProtocolV2.cc:45
> #19 0x000064a6e68b898e in std::function<void (char*, long)>::operator()(char*,
long) const (__args#1=<optimized out>, __args#0=<optimized out>,
this=0x13f508165610) at
/usr/lib/gcc/x86_64-pc-linux-gnu/9.2.0/include/g++-v9/bits/std_function.h:685
> #20 AsyncConnection::process (this=0x13f508165200) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/AsyncConnection.cc:450
> #21 0x000064a6e69181b8 in EventCenter::process_events
(this=this@entry=0x13f506f53700, timeout_microseconds=<optimized out>,
timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x64a6e4bb7d28) at
/usr/lib/gcc/x86_64-pc-linux-gnu/9.2.0/include/g++-v9/bits/basic_ios.h:282
> #22 0x000064a6e691e25c in NetworkStack::<lambda()>::operator()
(__closure=0x13f506f93808, __closure=0x13f506f93808) at
/var/tmp/portage/sys-cluster/ceph-14.2.8/work/ceph-14.2.8/src/msg/async/Stack.cc:53
> #23 std::_Function_handler<void(), NetworkStack::add_thread(unsigned
int)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at
/usr/lib/gcc/x86_64-pc-linux-gnu/9.2.0/include/g++-v9/bits/std_function.h:300
> #24 0x000064a6e5ca2130 in std::execute_native_thread_routine (__p=0x13f506f93800) at
/var/tmp/portage/sys-devel/gcc-9.2.0-r2/work/gcc-9.2.0/libstdc++-v3/src/c++11/thread.cc:80
> #25 0x000064a6e5e2c482 in start_thread (arg=<optimized out>) at
pthread_create.c:486
> #26 0x000064a6e5abc5cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95