[ceph v16.2.10] radosgw crash
2023-08-15T18:15:55.356+0000 7f7916ef3700 -1 *** Caught signal (Aborted) ** in thread 7f7916ef3700 thread_name:radosgw ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f79da065ce0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f79d905809b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f79d905e53c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f79d905e597] 7: /lib64/libstdc++.so.6(+0x9652e) [0x7f79d905e52e] 8: (spawn::detail::continuation_context::resume()+0x87) [0x7f79e4b70a17] 9: (boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x25a) [0x7f79e4b77b6a] 10: (boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator()()+0x8d) [0x7f79e4b7f93d] 11: (boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::recycling_allocator<void, boost::asio::detail::thread_info_base::default_tag>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x96) [0x7f79e4b7fca6] 12: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f2) [0x7f79e4b73ad2] 13: /lib64/libradosgw.so.2(+0x430376) [0x7f79e4b56376] 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f79d908aba3] 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79da05b1ca] 16: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -9999> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj init permissions -9998> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj recalculating target -9997> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj reading permissions -9996> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj init op -9995> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op mask -9994> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op permissions -9993> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15, is_admin=0) mask=49 -9992> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for uid=suliang -9991> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Found permission: 15 -9990> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for group=1 mask=49 -9989> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Permissions for group not found -9988> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for group=2 mask=49 -9987> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Permissions for group not found -9986> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15, is_admin=0), owner=suliang, perm=1 -9985> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op params -9984> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj pre-executing -9983> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj executing
thanks Louis, that looks like the same backtrace as https://tracker.ceph.com/issues/61763. that issue has been on 'Need More Info' because all of the rgw logging was disabled there. are you able to share some more log output to help us figure this out? under "--- begin dump of recent events ---", it dumps the last 10000 lines of log output. i would love to see the final ~100 lines of that output leading up to the crash (with any potentially-sensitive information hidden) On Tue, Aug 15, 2023 at 10:30 PM Louis Koo <zhucan.k8s@gmail.com> wrote:
2023-08-15T18:15:55.356+0000 7f7916ef3700 -1 *** Caught signal (Aborted) ** in thread 7f7916ef3700 thread_name:radosgw
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f79da065ce0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f79d905809b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f79d905e53c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f79d905e597] 7: /lib64/libstdc++.so.6(+0x9652e) [0x7f79d905e52e] 8: (spawn::detail::continuation_context::resume()+0x87) [0x7f79e4b70a17] 9: (boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x25a) [0x7f79e4b77b6a] 10: (boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator()()+0x8d) [0x7f79e4b7f93d] 11: (boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::recycling_allocator<void, boost::asio::detail::thread_info_base::default_tag>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x96) [0x7f79e4b7fca6] 12: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f2) [0x7f79e4b73ad2] 13: /lib64/libradosgw.so.2(+0x430376) [0x7f79e4b56376] 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f79d908aba3] 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79da05b1ca] 16: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events --- -9999> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj init permissions -9998> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj recalculating target -9997> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj reading permissions -9996> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj init op -9995> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op mask -9994> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op permissions -9993> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15, is_admin=0) mask=49 -9992> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for uid=suliang -9991> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Found permission: 15 -9990> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for group=1 mask=49 -9989> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Permissions for group not found -9988> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for group=2 mask=49 -9987> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Permissions for group not found -9986> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15, is_admin=0), owner=suliang, perm=1 -9985> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op params -9984> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj pre-executing -9983> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj executing _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
radosgw crash again with: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fdffc5d4ce0] 2: (rgw::ARN::ARN(rgw_bucket const&)+0x42) [0x7fe00727dbc2] 3: (verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*, boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, unsigned long)+0xa2) [0x7fe0072ce412] 4: (verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)+0x83) [0x7fe0072cf243] 5: (RGWListBucket::verify_permission(optional_yield)+0x12e) [0x7fe0074a2d3e] 6: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x81b) [0x7fe00714effb] 7: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x2891) [0x7fe0071531b1] 8: /lib64/libradosgw.so.2(+0x440703) [0x7fe0070d5703] 9: make_fcontext() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Hello, That was solved in 16.2.11 in tracker [1] with fix [2]. Best regards Tobias [1] https://tracker.ceph.com/issues/55765 [2] https://github.com/ceph/ceph/pull/47194/commits
On 12 Sep 2023, at 05:29, Louis Koo <zhucan.k8s@gmail.com> wrote:
radosgw crash again with: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fdffc5d4ce0] 2: (rgw::ARN::ARN(rgw_bucket const&)+0x42) [0x7fe00727dbc2] 3: (verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*, boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, unsigned long)+0xa2) [0x7fe0072ce412] 4: (verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)+0x83) [0x7fe0072cf243] 5: (RGWListBucket::verify_permission(optional_yield)+0x12e) [0x7fe0074a2d3e] 6: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x81b) [0x7fe00714effb] 7: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x2891) [0x7fe0071531b1] 8: /lib64/libradosgw.so.2(+0x440703) [0x7fe0070d5703] 9: make_fcontext() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Casey Bodley
-
Louis Koo
-
Tobias Urdin