16 Aug
2023
16 Aug
'23
5:29 a.m.
2023-08-15T18:15:55.356+0000 7f7916ef3700 -1 *** Caught signal (Aborted) **
in thread 7f7916ef3700 thread_name:radosgw
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
1: /lib64/libpthread.so.0(+0x12ce0) [0x7f79da065ce0]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f79d905809b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7f79d905e53c]
6: /lib64/libstdc++.so.6(+0x96597) [0x7f79d905e597]
7: /lib64/libstdc++.so.6(+0x9652e) [0x7f79d905e52e]
8: (spawn::detail::continuation_context::resume()+0x87) [0x7f79e4b70a17]
9:
(boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void
(*)(), boost::asio::strand<boost::asio::io_context::executor_type> >,
std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type>
> >, std::tuple<boost::system::error_code,
std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type>
> > > >, std::allocator<void>,
boost::asio::detail::scheduler_operation>::do_complete(void*,
boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned
long)+0x25a) [0x7f79e4b77b6a]
10:
(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type
const>::operator()()+0x8d) [0x7f79e4b7f93d]
11:
(boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type
const>, boost::asio::detail::recycling_allocator<void,
boost::asio::detail::thread_info_base::default_tag>,
boost::asio::detail::scheduler_operation>::do_complete(void*,
boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned
long)+0x96) [0x7f79e4b7fca6]
12: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f2)
[0x7f79e4b73ad2]
13: /lib64/libradosgw.so.2(+0x430376) [0x7f79e4b56376]
14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f79d908aba3]
15: /lib64/libpthread.so.0(+0x81ca) [0x7f79da05b1ca]
16: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- begin dump of recent events ---
-9999> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595
0.001000006s s3:get_obj init permissions
-9998> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595
0.001000006s s3:get_obj recalculating target
-9997> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595
0.001000006s s3:get_obj reading permissions
-9996> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj init op
-9995> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj verifying op mask
-9994> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj verifying op permissions
-9993> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier ->
rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15,
is_admin=0) mask=49
-9992> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Searching permissions for uid=suliang
-9991> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Found permission: 15
-9990> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Searching permissions for group=1 mask=49
-9989> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Permissions for group not found
-9988> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Searching permissions for group=2 mask=49
-9987> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Permissions for group not found
-9986> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier
-> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=,
perm_mask=15, is_admin=0), owner=suliang, perm=1
-9985> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj verifying op params
-9984> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj pre-executing
-9983> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj executing
16 Aug
16 Aug
4:29 p.m.
thanks Louis,
that looks like the same backtrace as
https://tracker.ceph.com/issues/61763. that issue has been on 'Need
More Info' because all of the rgw logging was disabled there. are you
able to share some more log output to help us figure this out?
under "--- begin dump of recent events ---", it dumps the last 10000
lines of log output. i would love to see the final ~100 lines of that
output leading up to the crash (with any potentially-sensitive
information hidden)
On Tue, Aug 15, 2023 at 10:30 PM Louis Koo <zhucan.k8s(a)gmail.com> wrote:
2023-08-15T18:15:55.356+0000 7f7916ef3700 -1 *** Caught signal (Aborted) **
in thread 7f7916ef3700 thread_name:radosgw
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
1: /lib64/libpthread.so.0(+0x12ce0) [0x7f79da065ce0]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f79d905809b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7f79d905e53c]
6: /lib64/libstdc++.so.6(+0x96597) [0x7f79d905e597]
7: /lib64/libstdc++.so.6(+0x9652e) [0x7f79d905e52e]
8: (spawn::detail::continuation_context::resume()+0x87) [0x7f79e4b70a17]
9:
(boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void
(*)(), boost::asio::strand<boost::asio::io_context::executor_type> >,
std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type>
> >, std::tuple<boost::system::error_code,
std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type>
> > > >, std::allocator<void>,
boost::asio::detail::scheduler_operation>::do_complete(void*,
boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned
long)+0x25a) [0x7f79e4b77b6a]
10:
(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type
const>::operator()()+0x8d) [0x7f79e4b7f93d]
11:
(boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type
const>, boost::asio::detail::recycling_allocator<void,
boost::asio::detail::thread_info_base::default_tag>,
boost::asio::detail::scheduler_operation>::do_complete(void*,
boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned
long)+0x96) [0x7f79e4b7fca6]
12: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f2)
[0x7f79e4b73ad2]
13: /lib64/libradosgw.so.2(+0x430376) [0x7f79e4b56376]
14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f79d908aba3]
15: /lib64/libpthread.so.0(+0x81ca) [0x7f79da05b1ca]
16: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- begin dump of recent events ---
-9999> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595
0.001000006s s3:get_obj init permissions
-9998> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595
0.001000006s s3:get_obj recalculating target
-9997> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595
0.001000006s s3:get_obj reading permissions
-9996> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj init op
-9995> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj verifying op mask
-9994> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj verifying op permissions
-9993> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier ->
rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15,
is_admin=0) mask=49
-9992> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Searching permissions for uid=suliang
-9991> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Found permission: 15
-9990> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Searching permissions for group=1 mask=49
-9989> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Permissions for group not found
-9988> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Searching permissions for group=2 mask=49
-9987> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj Permissions for group not found
-9986> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595
0.002000013s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier
-> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=,
perm_mask=15, is_admin=0), owner=suliang, perm=1
-9985> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj verifying op params
-9984> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj pre-executing
-9983> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595
0.002000013s s3:get_obj executing
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12 Sep
12 Sep
6:29 a.m.
radosgw crash again with:
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
1: /lib64/libpthread.so.0(+0x12ce0) [0x7fdffc5d4ce0]
2: (rgw::ARN::ARN(rgw_bucket const&)+0x42) [0x7fe00727dbc2]
3: (verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket
const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*,
boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy,
std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy,
std::allocator<rgw::IAM::Policy> > const&, unsigned long)+0xa2)
[0x7fe0072ce412]
4: (verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)+0x83)
[0x7fe0072cf243]
5: (RGWListBucket::verify_permission(optional_yield)+0x12e) [0x7fe0074a2d3e]
6: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*,
optional_yield, bool)+0x81b) [0x7fe00714effb]
7: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, rgw::auth::StrategyRegistry const&,
RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >*, std::chrono::duration<unsigned long,
std::ratio<1l, 1000000000l> >*, int*)+0x2891) [0x7fe0071531b1]
8: /lib64/libradosgw.so.2(+0x440703) [0x7fe0070d5703]
9: make_fcontext()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
4:44 p.m.
Hello,
That was solved in 16.2.11 in tracker [1] with fix [2].
Best regards
Tobias
[1] https://tracker.ceph.com/issues/55765
[2] https://github.com/ceph/ceph/pull/47194/commits
On 12 Sep 2023, at 05:29, Louis Koo
<zhucan.k8s(a)gmail.com> wrote:
radosgw crash again with:
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
1: /lib64/libpthread.so.0(+0x12ce0) [0x7fdffc5d4ce0]
2: (rgw::ARN::ARN(rgw_bucket const&)+0x42) [0x7fe00727dbc2]
3: (verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket
const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*,
boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy,
std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy,
std::allocator<rgw::IAM::Policy> > const&, unsigned long)+0xa2)
[0x7fe0072ce412]
4: (verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)+0x83)
[0x7fe0072cf243]
5: (RGWListBucket::verify_permission(optional_yield)+0x12e) [0x7fe0074a2d3e]
6: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*,
optional_yield, bool)+0x81b) [0x7fe00714effb]
7: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, rgw::auth::StrategyRegistry const&,
RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >*, std::chrono::duration<unsigned long,
std::ratio<1l, 1000000000l> >*, int*)+0x2891) [0x7fe0071531b1]
8: /lib64/libradosgw.so.2(+0x440703) [0x7fe0070d5703]
9: make_fcontext()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
250
days inactive
277
days old
3 comments
3 participants
participants (3)
-
Casey Bodley -
Louis Koo -
Tobias Urdin