Folks, we are seeing very slow MDSes on a CephFS that has nearly nothing to do. Slow as in „200 metadata requests per second“ slow when putting a bit of load onto it. We managed to get a wallclock of the MDS: [root@node01 gdbpmp]# ./gdbpmp.py -i gdbpmp.data Thread: 7 (mds-rank-fin) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% Finisher::finisher_thread_entry() + 94.12% MDSIOContextBase::complete |+ 88.24% MDSRank::respawn ||+ 88.24% Context::complete || + 88.24% MDSDaemon::respawn || + 88.24% ceph::logging::Log::dump_recent() || + 52.94% ceph::logging::Log::_flush(std::vector<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >&, bool) || |+ 23.53% __snprintf_chk || ||+ 23.53% __vsnprintf_internal || || + 17.65% __vfprintf_internal || || + 5.88% __strchrnul_evex || || + 5.88% _IO_default_xsputn || |+ 17.65% ceph::logging::Log::_flush_logbuf() || ||+ 17.65% ceph::logging::Log::_log_safe_write(std::basic_string_view<char, std::char_traits<char> >) || || + 17.65% safe_write || || + 17.65% write || |+ 5.88% __tz_convert || | + 5.88% __tzfile_compute || | + 5.88% __strcmp_evex || + 5.88% __memmove_avx512_unaligned_erms |+ 5.88% std::lock_guard<ceph::fair_mutex>::lock_guard | + 5.88% ceph::fair_mutex::lock | + 5.88% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 5.88% pthread_cond_wait@@GLIBC_2.3.2 | + 5.88% __futex_abstimed_wait_common + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&) + 5.88% pthread_cond_wait@@GLIBC_2.3.2 + 5.88% __futex_abstimed_wait_common Thread: 10 (mds-rank-progr) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% MDSRank::ProgressThread::entry + 100.00% std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex>, MDSRank::ProgressThread::entry()::<lambda()> > + 100.00% std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex> > + 88.24% std::condition_variable::wait(std::unique_lock<std::mutex>&) |+ 88.24% pthread_cond_wait@@GLIBC_2.3.2 | + 88.24% __futex_abstimed_wait_common + 11.76% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock + 11.76% std::unique_lock<ceph::fair_mutex>::lock + 11.76% ceph::fair_mutex::lock + 11.76% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&) + 11.76% pthread_cond_wait@@GLIBC_2.3.2 + 11.76% __futex_abstimed_wait_common Thread: 11 (mds-log-trim) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% MDLog::log_trim_upkeep + 100.00% std::_V2::condition_variable_any::wait_for<std::unique_lock<ceph::fair_mutex>, long, std::ratio<1l, 1000l> > + 100.00% std::_V2::condition_variable_any::wait_until<std::unique_lock<ceph::fair_mutex>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > + 82.35% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock |+ 82.35% std::unique_lock<ceph::fair_mutex>::lock | + 82.35% ceph::fair_mutex::lock | + 82.35% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 82.35% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 82.35% pthread_cond_wait@@GLIBC_2.3.2 | + 82.35% __futex_abstimed_wait_common + 17.65% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > + 17.65% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > + 17.65% std::__condvar::wait_until + 17.65% pthread_cond_clockwait@GLIBC_2.30 + 17.65% __futex_abstimed_wait_common Thread: 12 (mds-cache-trim) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% MDCache::upkeep_main + 70.59% std::scoped_lock<ceph::fair_mutex>::scoped_lock |+ 70.59% ceph::fair_mutex::lock | + 70.59% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 70.59% pthread_cond_wait@@GLIBC_2.3.2 | + 70.59% __futex_abstimed_wait_common + 23.53% std::condition_variable::wait_for<long, std::ratio<1l, 1000000000l> > |+ 23.53% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > | + 23.53% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > | + 23.53% std::__condvar::wait_until | + 23.53% pthread_cond_clockwait@GLIBC_2.30 | + 23.53% __futex_abstimed_wait_common + 5.88% MDCache::check_memory_usage + 5.88% MemoryModel::sample + 5.88% MemoryModel::_sample + 5.88% std::getline<char, std::char_traits<char>, std::allocator<char> > + 5.88% std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char , std::char_traits<char>, std::allocator<char> >&, char) + 5.88% std::basic_filebuf<char, std::char_traits<char> >::underflow() + 5.88% std::__basic_file<char>::xsgetn(char*, long) + 5.88% read Thread: 15 (safe_timer) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% CommonSafeTimerThread<std::mutex>::entry() + 100.00% CommonSafeTimer<std::mutex>::timer_thread() + 88.24% pthread_cond_clockwait@GLIBC_2.30 |+ 88.24% __futex_abstimed_wait_common + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&) + 11.76% pthread_cond_wait@@GLIBC_2.3.2 + 11.76% __futex_abstimed_wait_common Thread: 17 (safe_timer) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% CommonSafeTimerThread<ceph::fair_mutex>::entry() + 100.00% CommonSafeTimer<ceph::fair_mutex>::timer_thread() + 70.59% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock() |+ 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 70.59% pthread_cond_wait@@GLIBC_2.3.2 | + 70.59% __futex_abstimed_wait_common + 29.41% pthread_cond_clockwait@GLIBC_2.30 + 29.41% __futex_abstimed_wait_common Thread: 20 (ms_dispatch) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% DispatchQueue::DispatchThread::entry() + 100.00% DispatchQueue::entry() + 100.00% MDSDaemon::ms_dispatch2 + 94.12% std::lock_guard<ceph::fair_mutex>::lock_guard |+ 94.12% ceph::fair_mutex::lock | + 94.12% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 94.12% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 94.12% pthread_cond_wait@@GLIBC_2.3.2 | + 94.12% __futex_abstimed_wait_common + 5.88% MDSRankDispatcher::ms_dispatch + 5.88% MDSRank::_dispatch + 5.88% MDSRank::_advance_queues + 5.88% MDSContext::complete + 5.88% Context::complete + 5.88% Server::handle_client_readdir + 5.88% CInode::encode_inodestat + 5.88% encode<optmetadata_singleton<optmetadata_client_t<std::allocator>, std::allocator>, std::allocator> + 5.88% optmetadata_multiton<optmetadata_singleton<optmetadata_client_t<std::allocator>, std::allocator>, std::allocator>::encode + 5.88% ceph::buffer::v15_2_0::list::append_hole(unsigned int) + 5.88% ceph::buffer::v15_2_0::list::refill_append_space(unsigned int) + 5.88% mempool::pool_t::adjust_count(long, long) Thread: 21 (io_context_pool) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::i o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(sh ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(s hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) + 100.00% std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1 }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&, ceph::async::io_context_pool::start(sho rt)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short): :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{la mbda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta rt(short)::{lambda()#1}&&) + 100.00% make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd a(auto:1&&, (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) const + 100.00% std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const + 100.00% boost::asio::io_context::run + 100.00% boost::asio::detail::scheduler::run(boost::system::error_code&) [clone .constprop.0] [clone .isra.0] + 100.00% boost::asio::detail::scheduler::do_run_one + 94.12% boost::asio::detail::conditionally_enabled_event::wait |+ 94.12% boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> | + 94.12% pthread_cond_wait@@GLIBC_2.3.2 | + 94.12% __futex_abstimed_wait_common + 5.88% boost::asio::detail::scheduler_operation::complete + 5.88% boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, std::allocator<voi d>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) + 5.88% boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>::operator()() + 5.88% boost::asio::detail::executor_op<boost::asio::detail::binder0<MonClient::handle_config(MConfig*)::{lambda()#2}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete (void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) [clone .lto_priv.0] + 5.88% ceph::common::ConfigProxy::set_mon_vals(ceph::common::CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<cha r, std::char_traits<char>, std::allocator<char> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__c xx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > co nst&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) [clone .isra.0] + 5.88% md_config_t::set_mon_vals(ceph::common::CephContext*, ConfigValues&, ConfigTracker const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_ _cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) + 5.88% md_config_t::update_legacy_vals(ConfigValues&) + 5.88% md_config_t::update_legacy_val(ConfigValues&, Option const&, std::variant<long ConfigValues::*, unsigned long ConfigValues::*, std::__cxx11::basic_string<char, std::char_traits<char>, s td::allocator<char> > ConfigValues::*, double ConfigValues::*, bool ConfigValues::*, entity_addr_t ConfigValues::*, entity_addrvec_t ConfigValues::*, uuid_d ConfigValues::*>) + 5.88% md_config_t::_get_val(ConfigValues const&, Option const&, boost::container::small_vector<std::pair<Option const*, std::variant<std::monostate, std::__cxx11::basic_string<char, std::cha r_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long , std::ratio<1l, 1000l> >, Option::size_t, uuid_d> const*>, 4ul, void, void>*, std::basic_ostream<char, std::char_traits<char> >*) const + 5.88% md_config_t::_get_val_nometa[abi:cxx11](ConfigValues const&, Option const&) const Thread: 22 (io_context_pool) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::i o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(sh ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(s hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) + 100.00% std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1 }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&, ceph::async::io_context_pool::start(sho rt)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short): :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{la mbda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta rt(short)::{lambda()#1}&&) + 100.00% make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd a(auto:1&&, (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) const + 100.00% std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const + 100.00% boost::asio::io_context::run + 100.00% boost::asio::detail::scheduler::run(boost::system::error_code&) [clone .constprop.0] [clone .isra.0] + 100.00% boost::asio::detail::scheduler::do_run_one + 100.00% boost::asio::detail::conditionally_enabled_event::wait + 100.00% boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> + 100.00% pthread_cond_wait@@GLIBC_2.3.2 + 100.00% __futex_abstimed_wait_common Thread: 26 (msgr-worker-2) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% std::_Function_handler<void (), NetworkStack::add_thread(Worker*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) + 94.12% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) |+ 94.12% epoll_wait + 5.88% AsyncConnection::process() + 5.88% ProtocolV2::run_continuation(Ct<ProtocolV2>&) + 5.88% ProtocolV2::handle_message() + 5.88% decode_message(ceph::common::CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&, boost::intrusive_ptr<Conne ction>) + 5.88% MMgrMap::decode_payload() + 5.88% MgrMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) + 5.88% MgrMap::ModuleInfo::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) + 5.88% std::enable_if<denc_traits<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocat or<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, void>::supported&&(!denc_traits<std::set<std::__cxx11::basic_string<char, std::cha r_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_trait s<char>, std::allocator<char> > > >, void>::need_contiguous), void>::type ceph::decode<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cx x11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, denc_traits<std::set<s td::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__c xx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, void> >(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx1 1::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, ceph::buffer::v15_2_0:: list::iterator_impl<true>&) + 5.88% ceph::buffer::v15_2_0::ptr::release() Sidenote: while recording this, the MDS became so slow that the MDS cluster declared it dead … As can be seen, the MDS is waiting in futex to no end, but how can we find out what it is waiting for? FS has many small files but nearly no activity …. Thanks in advance Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.