- ceph-users - lists.ceph.io

clarification on Fast-EC BUG in tentacle
by Sascha Lucas 15 Feb '26

15 Feb '26

Dear ceph users, I like to get some clarification regarding the Fast-EC BUG in tentacle[1]. The clyso article mentions "on CephFS", which gave the impression that just CephFS as a application is affected, but others like RBD are not. I assume and may found out, that RDB is also affected. In a cephadm test environment I started a qemu-VM on an EC-data-pool with ec-overwrite and ec-optimisations set and issued fstrim (qemu discard=unmap via librbd) which immediately brings down some OSDs (see log below). Also the clyso article points to issue 71642 "crash if allow_ec_optimizations set but with allow_ec_overwrites not set"[2], which from its title seems misleading because others and I have allow_ec_overwrites set. Thanks to clyso's known-bugs collection! Otherwise it's hard to get an overview what problem one might hit currently ;-). And also thanks for providing fixed images on harbor.clyso.com/custom-ceph/ceph/ceph:v20.2.0-fast-ec-path-hf.2. It seems the fix[3] is from Jul 11, 2025, but still awaiting backport to, in the mean time released, stable version tentacle? Sounds odd. Thanks, Sascha. [1] https://docs.clyso.com/docs/kb/known-bugs/tentacle/#osd-crash-when-enabling… [2] https://tracker.ceph.com/issues/71642 [3] https://github.com/ceph/ceph/commit/51679b948bd9bb45cce9d778b0555f45ff808cd8 ODS crash log ======8<-------------------------- osd-1[197069]: terminate called after throwing an instance of 'std::out_of_range' osd-1[197069]: what(): Key not found osd-1[197069]: *** Caught signal (Aborted) ** osd-1[197069]: in thread 7f2dd5c65640 thread_name:tp_osd_tp osd-1[197069]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) osd-1[197069]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2df4da8c30] osd-1[197069]: 2: /lib64/libc.so.6(+0x8d03c) [0x7f2df4df603c] osd-1[197069]: 3: raise() osd-1[197069]: 4: abort() osd-1[197069]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2df5b45b21] osd-1[197069]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2df5b5153c] osd-1[197069]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2df5b515a7] osd-1[197069]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2df5b51809] osd-1[197069]: 9: /usr/bin/ceph-osd(+0x41b012) [0x55ef3f2b1012] osd-1[197069]: 10: (ECTransaction::WritePlanObj::WritePlanObj(hobject_t const&, PGTransaction::ObjectOperation const&, ECU> osd-1[197069]: 11: /usr/bin/ceph-osd(+0x99cdaa) [0x55ef3f832daa] osd-1[197069]: 12: (ECCommon::get_write_plan(ECUtil::stripe_info_t const&, PGTransaction&, ECCommon::ReadPipeline&, ECComm> osd-1[197069]: 13: (ECBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::uniq> osd-1[197069]: 14: /usr/bin/ceph-osd(+0x7d980f) [0x55ef3f66f80f] osd-1[197069]: 15: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x3ae) [0x55ef3f5f002e] osd-1[197069]: 16: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xf6a) [0x55ef3f5cd63a] osd-1[197069]: 17: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2d5f) [0x55ef3f5bdf5f] osd-1[197069]: 18: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x19> osd-1[197069]: 19: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)> osd-1[197069]: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8bc) [0x55ef3f51611c] osd-1[197069]: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x23a) [0x55ef3fa98e1a] osd-1[197069]: 22: /usr/bin/ceph-osd(+0xc033d4) [0x55ef3fa993d4] osd-1[197069]: 23: /lib64/libc.so.6(+0x8b2fa) [0x7f2df4df42fa] osd-1[197069]: 24: /lib64/libc.so.6(+0x110400) [0x7f2df4e79400] osd-1[197069]: 2026-02-15T00:10:48.350+0000 7f2dd5c65640 -1 *** Caught signal (Aborted) ** osd-1[197069]: in thread 7f2dd5c65640 thread_name:tp_osd_tp --------------------------------------

1 0

tracker.ceph.com blocks edge browsers
by Sake Ceph 13 Feb '26

13 Feb '26

Hi! Is there a reason for tracker.ceph.com to block edge browsers? I use it on my Android phone and when I remove in the user agent the edge part, everything works fine. Or use a chrome user agent, everything works. Any help would be appreciated :) Kind regards Sake

1 0

Three Days Left For Ceph Days Raleigh CFP Submissions!
by Anthony Middleton 13 Feb '26

13 Feb '26

ACT NOW! Ceph Days Raleigh CFP Closes Feb. 16th! This Ceph Days offers a full day dedicated to sharing Ceph's transformative power and fostering the vibrant Ceph community in Raleigh, NC! Submit your talk here: https://ceph.io/en/community/events/2026/ceph-days-raleigh/

1 0

Ceph MDS super slow
by Martin Gerhard Loschwitz 13 Feb '26

13 Feb '26

Folks, we are seeing very slow MDSes on a CephFS that has nearly nothing to do. Slow as in „200 metadata requests per second“ slow when putting a bit of load onto it. We managed to get a wallclock of the MDS: [root@node01 gdbpmp]# ./gdbpmp.py -i gdbpmp.data Thread: 7 (mds-rank-fin) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% Finisher::finisher_thread_entry() + 94.12% MDSIOContextBase::complete |+ 88.24% MDSRank::respawn ||+ 88.24% Context::complete || + 88.24% MDSDaemon::respawn || + 88.24% ceph::logging::Log::dump_recent() || + 52.94% ceph::logging::Log::_flush(std::vector<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >&, bool) || |+ 23.53% __snprintf_chk || ||+ 23.53% __vsnprintf_internal || || + 17.65% __vfprintf_internal || || + 5.88% __strchrnul_evex || || + 5.88% _IO_default_xsputn || |+ 17.65% ceph::logging::Log::_flush_logbuf() || ||+ 17.65% ceph::logging::Log::_log_safe_write(std::basic_string_view<char, std::char_traits<char> >) || || + 17.65% safe_write || || + 17.65% write || |+ 5.88% __tz_convert || | + 5.88% __tzfile_compute || | + 5.88% __strcmp_evex || + 5.88% __memmove_avx512_unaligned_erms |+ 5.88% std::lock_guard<ceph::fair_mutex>::lock_guard | + 5.88% ceph::fair_mutex::lock | + 5.88% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 5.88% pthread_cond_wait@@GLIBC_2.3.2 | + 5.88% __futex_abstimed_wait_common + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&) + 5.88% pthread_cond_wait@@GLIBC_2.3.2 + 5.88% __futex_abstimed_wait_common Thread: 10 (mds-rank-progr) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% MDSRank::ProgressThread::entry + 100.00% std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex>, MDSRank::ProgressThread::entry()::<lambda()> > + 100.00% std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex> > + 88.24% std::condition_variable::wait(std::unique_lock<std::mutex>&) |+ 88.24% pthread_cond_wait@@GLIBC_2.3.2 | + 88.24% __futex_abstimed_wait_common + 11.76% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock + 11.76% std::unique_lock<ceph::fair_mutex>::lock + 11.76% ceph::fair_mutex::lock + 11.76% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&) + 11.76% pthread_cond_wait@@GLIBC_2.3.2 + 11.76% __futex_abstimed_wait_common Thread: 11 (mds-log-trim) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% MDLog::log_trim_upkeep + 100.00% std::_V2::condition_variable_any::wait_for<std::unique_lock<ceph::fair_mutex>, long, std::ratio<1l, 1000l> > + 100.00% std::_V2::condition_variable_any::wait_until<std::unique_lock<ceph::fair_mutex>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > + 82.35% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock |+ 82.35% std::unique_lock<ceph::fair_mutex>::lock | + 82.35% ceph::fair_mutex::lock | + 82.35% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 82.35% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 82.35% pthread_cond_wait@@GLIBC_2.3.2 | + 82.35% __futex_abstimed_wait_common + 17.65% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > + 17.65% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > + 17.65% std::__condvar::wait_until + 17.65% pthread_cond_clockwait(a)GLIBC_2.30 + 17.65% __futex_abstimed_wait_common Thread: 12 (mds-cache-trim) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% MDCache::upkeep_main + 70.59% std::scoped_lock<ceph::fair_mutex>::scoped_lock |+ 70.59% ceph::fair_mutex::lock | + 70.59% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 70.59% pthread_cond_wait@@GLIBC_2.3.2 | + 70.59% __futex_abstimed_wait_common + 23.53% std::condition_variable::wait_for<long, std::ratio<1l, 1000000000l> > |+ 23.53% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > | + 23.53% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > | + 23.53% std::__condvar::wait_until | + 23.53% pthread_cond_clockwait(a)GLIBC_2.30 | + 23.53% __futex_abstimed_wait_common + 5.88% MDCache::check_memory_usage + 5.88% MemoryModel::sample + 5.88% MemoryModel::_sample + 5.88% std::getline<char, std::char_traits<char>, std::allocator<char> > + 5.88% std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char , std::char_traits<char>, std::allocator<char> >&, char) + 5.88% std::basic_filebuf<char, std::char_traits<char> >::underflow() + 5.88% std::__basic_file<char>::xsgetn(char*, long) + 5.88% read Thread: 15 (safe_timer) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% CommonSafeTimerThread<std::mutex>::entry() + 100.00% CommonSafeTimer<std::mutex>::timer_thread() + 88.24% pthread_cond_clockwait(a)GLIBC_2.30 |+ 88.24% __futex_abstimed_wait_common + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&) + 11.76% pthread_cond_wait@@GLIBC_2.3.2 + 11.76% __futex_abstimed_wait_common Thread: 17 (safe_timer) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% CommonSafeTimerThread<ceph::fair_mutex>::entry() + 100.00% CommonSafeTimer<ceph::fair_mutex>::timer_thread() + 70.59% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock() |+ 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 70.59% pthread_cond_wait@@GLIBC_2.3.2 | + 70.59% __futex_abstimed_wait_common + 29.41% pthread_cond_clockwait(a)GLIBC_2.30 + 29.41% __futex_abstimed_wait_common Thread: 20 (ms_dispatch) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% DispatchQueue::DispatchThread::entry() + 100.00% DispatchQueue::entry() + 100.00% MDSDaemon::ms_dispatch2 + 94.12% std::lock_guard<ceph::fair_mutex>::lock_guard |+ 94.12% ceph::fair_mutex::lock | + 94.12% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 94.12% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 94.12% pthread_cond_wait@@GLIBC_2.3.2 | + 94.12% __futex_abstimed_wait_common + 5.88% MDSRankDispatcher::ms_dispatch + 5.88% MDSRank::_dispatch + 5.88% MDSRank::_advance_queues + 5.88% MDSContext::complete + 5.88% Context::complete + 5.88% Server::handle_client_readdir + 5.88% CInode::encode_inodestat + 5.88% encode<optmetadata_singleton<optmetadata_client_t<std::allocator>, std::allocator>, std::allocator> + 5.88% optmetadata_multiton<optmetadata_singleton<optmetadata_client_t<std::allocator>, std::allocator>, std::allocator>::encode + 5.88% ceph::buffer::v15_2_0::list::append_hole(unsigned int) + 5.88% ceph::buffer::v15_2_0::list::refill_append_space(unsigned int) + 5.88% mempool::pool_t::adjust_count(long, long) Thread: 21 (io_context_pool) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::i o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(sh ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(s hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) + 100.00% std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1 }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&, ceph::async::io_context_pool::start(sho rt)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short): :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{la mbda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta rt(short)::{lambda()#1}&&) + 100.00% make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd a(auto:1&&, (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) const + 100.00% std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const + 100.00% boost::asio::io_context::run + 100.00% boost::asio::detail::scheduler::run(boost::system::error_code&) [clone .constprop.0] [clone .isra.0] + 100.00% boost::asio::detail::scheduler::do_run_one + 94.12% boost::asio::detail::conditionally_enabled_event::wait |+ 94.12% boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> | + 94.12% pthread_cond_wait@@GLIBC_2.3.2 | + 94.12% __futex_abstimed_wait_common + 5.88% boost::asio::detail::scheduler_operation::complete + 5.88% boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, std::allocator<voi d>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) + 5.88% boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>::operator()() + 5.88% boost::asio::detail::executor_op<boost::asio::detail::binder0<MonClient::handle_config(MConfig*)::{lambda()#2}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete (void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) [clone .lto_priv.0] + 5.88% ceph::common::ConfigProxy::set_mon_vals(ceph::common::CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<cha r, std::char_traits<char>, std::allocator<char> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__c xx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > co nst&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) [clone .isra.0] + 5.88% md_config_t::set_mon_vals(ceph::common::CephContext*, ConfigValues&, ConfigTracker const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_ _cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) + 5.88% md_config_t::update_legacy_vals(ConfigValues&) + 5.88% md_config_t::update_legacy_val(ConfigValues&, Option const&, std::variant<long ConfigValues::*, unsigned long ConfigValues::*, std::__cxx11::basic_string<char, std::char_traits<char>, s td::allocator<char> > ConfigValues::*, double ConfigValues::*, bool ConfigValues::*, entity_addr_t ConfigValues::*, entity_addrvec_t ConfigValues::*, uuid_d ConfigValues::*>) + 5.88% md_config_t::_get_val(ConfigValues const&, Option const&, boost::container::small_vector<std::pair<Option const*, std::variant<std::monostate, std::__cxx11::basic_string<char, std::cha r_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long , std::ratio<1l, 1000l> >, Option::size_t, uuid_d> const*>, 4ul, void, void>*, std::basic_ostream<char, std::char_traits<char> >*) const + 5.88% md_config_t::_get_val_nometa[abi:cxx11](ConfigValues const&, Option const&) const Thread: 22 (io_context_pool) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::i o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(sh ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(s hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) + 100.00% std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1 }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&, ceph::async::io_context_pool::start(sho rt)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short): :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{la mbda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta rt(short)::{lambda()#1}&&) + 100.00% make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd a(auto:1&&, (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) const + 100.00% std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const + 100.00% boost::asio::io_context::run + 100.00% boost::asio::detail::scheduler::run(boost::system::error_code&) [clone .constprop.0] [clone .isra.0] + 100.00% boost::asio::detail::scheduler::do_run_one + 100.00% boost::asio::detail::conditionally_enabled_event::wait + 100.00% boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> + 100.00% pthread_cond_wait@@GLIBC_2.3.2 + 100.00% __futex_abstimed_wait_common Thread: 26 (msgr-worker-2) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% std::_Function_handler<void (), NetworkStack::add_thread(Worker*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) + 94.12% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) |+ 94.12% epoll_wait + 5.88% AsyncConnection::process() + 5.88% ProtocolV2::run_continuation(Ct<ProtocolV2>&) + 5.88% ProtocolV2::handle_message() + 5.88% decode_message(ceph::common::CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&, boost::intrusive_ptr<Conne ction>) + 5.88% MMgrMap::decode_payload() + 5.88% MgrMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) + 5.88% MgrMap::ModuleInfo::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) + 5.88% std::enable_if<denc_traits<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocat or<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, void>::supported&&(!denc_traits<std::set<std::__cxx11::basic_string<char, std::cha r_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_trait s<char>, std::allocator<char> > > >, void>::need_contiguous), void>::type ceph::decode<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cx x11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, denc_traits<std::set<s td::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__c xx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, void> >(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx1 1::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, ceph::buffer::v15_2_0:: list::iterator_impl<true>&) + 5.88% ceph::buffer::v15_2_0::ptr::release() Sidenote: while recording this, the MDS became so slow that the MDS cluster declared it dead … As can be seen, the MDS is waiting in futex to no end, but how can we find out what it is waiting for? FS has many small files but nearly no activity …. Thanks in advance Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

reef 18.2.8 QE validation status
by Yuri Weinstein 13 Feb '26

13 Feb '26

Note: this is an attempt to test and publish post EOL reef point release Details of this release are summarized here: https://tracker.ceph.com/issues/73906#note-1 Release Notes - TBD LRC upgrade - N/A distro => jammy focal centos9 windows bookworm Seeking approvals/reviews for: smoke - Laura approved? rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? (rados is also running on build 2 == build 1 + https://github.com/ceph/ceph/pull/66056 per Igor's request) rgw - Adam E approved? fs - Venky approved? orch - Adam King approved? rbd - Ilya approved? krbd - Ilya approved? upgrade/pacific-x (reef) - can this be deprecated? Josh? Neha? (same as in 18.2.5) upgrade/quincy-x (reef) - Laura, Prashant please take a look. (same as in 18.2.5) powercycle - Brad approved? ceph-volume - see orch, Guillaume approved? Notably upgrades suites show same results as for 18.2.5 (tested in March 2025) To compare, see 18.2.5 tests https://tracker.ceph.com/issues/70563#note-1 TIA

4 5

Re: mgmt-gateway and oauth2-proxy for radosgw?
by Casey Bodley 13 Feb '26

13 Feb '26

hi Florian, On Thu, Feb 12, 2026 at 11:05 AM Florian Haas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi everyone, > > a question about the mgmt-gateway and oauth2-proxy services, because we > want to incorporate one or both of them into our Ceph training course. > > I understand that as of Tentacle, mgmt-gateway exists as an > administrative simplification for access to the Dashboard and the > orchestrated monitoring stack,[0] and that it has OpenID Connect (OIDC) > support via oauth2-proxy. The docs[1] assert that oauth2-proxy serves as > "an advanced method for managing authentication and access control for > Ceph applications". > > We've also had OIDC support in radosgw since Octopus (I believe), > however it is tied to an implementation of a subset of STS[2], > configuration is rather non-trivial[3], and as far as I understand it is > presently not integrated with oauth2-proxy. > > Now my question is: will oauth2-proxy eventually be integrated with > radosgw to replace and remove the STS dependency, or are the two OIDC > integrations expected to coexist in parallel? i don't know that radosgw has any plans for oidc integration outside of sts and AssumeRoleWithWebIdentity. the benefit of sts for this integration is that it works with unmodified aws clients the user account feature in squid added fine-grained api-based control over oidc providers, roles, and their associated iam policy: https://docs.ceph.com/en/squid/radosgw/account/ if configuration was your main obstacle, i would hope that the aws-compatible tooling like https://docs.aws.amazon.com/cli/latest/reference/iam/create-open-id-connect… would make that easier > > Thanks! > > Cheers, > Florian > > [0] https://docs.ceph.com/en/latest/cephadm/services/mgmt-gateway/ > [1] https://docs.ceph.com/en/latest/cephadm/services/oauth2-proxy/ > [2] https://docs.ceph.com/en/latest/radosgw/oidc/ > [3] > https://community.ibm.com/community/user/blogs/deepak-thorat/2024/03/20/cep… > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Data is still actively corrupted!
by Eugen Block 13 Feb '26

13 Feb '26

Hi, you seem to hit this issue [0] with the read_from_replica option, see the announcement in [1]. But I haven't looked in detail, not sure if there's a way to fix it and if read_from_replica=balance has the same effect. Regards, Eugen [0] https://tracker.ceph.com/issues/73997 [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/JI2ZRF7A3PW… Am Do., 12. Feb. 2026 um 10:46 Uhr schrieb dominik.baack via ceph-users < ceph-users(a)ceph.io>: > Hi, > > thanks for your reply. > > Mounting was done without the 'root_squash' option, here is the > corresponding fstab entry: > > > 192.168.251.2,192.168.251.3,192.168.251.4,192.168.251.5,192.168.251.6,192.168.251.7:/ > > /cephfs ceph > name=gpu01,secretfile=/etc/ceph/gpu01.key,noatime,_netdev,recover_session=clean,read_from_replica=balance > > 0 0 > > Dominik > > > Am 2026-02-12 09:51, schrieb goetze: > > Hi ! > > > > Have you mounted cephfs with the 'root_squash' option set? If so, > > remove that option. I may be wrong here, but as far as I know, this is > > still considered unsafe and can lead to data corruption since the > > necessary code changes have not yet made it into the mainstream linux > > kernel. > > > > Carsten > > ------------------------------------------------------------------ > > Carsten Goetze > > Computer Graphics tel: +49 531 391-2109 > > TU Braunschweig fax: +49 531 391-2103 > > Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de > > D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze > > > >> Am 12.02.2026 um 07:15 schrieb Dominik Baack via ceph-users > >> <ceph-users(a)ceph.io>: > >> > >> Hi, > >> > >> I noticed now that files are still actively corrupted / replaced by > >> empty files when open saved. > >> > >> Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. > >> Ceph servers are still running 20.2 Tentacle (deployed via cephadm) > >> with an Ubuntu 24.04 host. > >> > >> Options currently set are to noout, norebalance, noscrub and > >> nodeep-scrub > >> > >> I am currently setting up a read only mount to copy all existing > >> data for a backup. > >> > >> I have currently no clue whats going on, I currently was able to > >> observe this behavior only on the nodes. As those were upgraded as > >> well (MLNX driver, nvidia-fs,...) can it be network? > >> > >> Any Idea how to recover from this? > >> > >> Cheers > >> Dominik > >> > >> Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users: > >> > >>> Hi, > >>> > >>> after an controlled shutdown of the whole cluster do to external > >>> circumstances we decided to update from 19.2 to 20.2 after the > >>> restart. The system was health before and after the update. > >>> The nodes mounting the filesystem were not equally lucky and were > >>> partially shutdown hard. Storage was kept running additional > >>> ~30min after node shutdown, all inflight operations should have > >>> finished. > >>> > >>> Now we discover the some of the user files seem to be replaced > >>> with zeros. For example: > >>> > >>> stat .gitignore > >>> File: .gitignore > >>> Size: 4429 Blocks: 9 IO Block: 4194304 > >>> regular file > >>> Device: 0,48 Inode: 1100241384598 Links: 1 > >>> > >>> hexdump -C .gitignore > >>> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > >>> |................| > >>> * > >>> 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| > >>> 0000114d > >>> > >>> Scanning for files containing only zeros show several issues of > >>> files that were likely accessed before or during the shutdown of > >>> the nodes. > >>> > >>> How should I progress from here? > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Why did this situation occur?
by Jayanth Reddy 12 Feb '26

12 Feb '26

Hi, Just restart your manager daemons using `ceph orch restart mgr` and see the updated status. AFAIK Quincy and earlier versions had problems with mgrs where stale data was shown and a simple daemon restart would fix it. Additionally, it is extremely recommended to use a monitoring system for the Operating System as well as Ceph. Ceph dashboard also should shows you problems with Ceph and your monitoring system for the OS should catch any abnormal disk usage. Thanks ________________________________ From: 苏察哈尔灿 via ceph-users <ceph-users(a)ceph.io> Sent: Friday, February 13, 2026 7:55:51 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Why did this situation occur? Please help me, I have a CEPH cluster with six nodes. Yesterday, one of the nodes reported insufficient disk space (image01.png), but using the command "ceph -s", it kept showing normal (image02.png). Before this, there was also a situation where the disk space of a certain node was insufficient. At that time, using the "ceph -s" command could normally display the alarm and the involved nodes. Why is this situation occurring now? Filesystem                    Type  Size  Used Avail Use% Mounted on tmpfs                         tmpfs   3.2G  3.6M  3.1G   1% /run /dev/mapper/ubuntu--vg-ubuntu--lv ext4    98G   93G   863M  100% / tmpfs                         tmpfs   16G     0     16G    0% /dev/shm tmpfs                         tmpfs   5.0M    0     5.0M   0% /run/lock /dev/sdj2                     ext4    2.0G  253M  1.6G   14% /boot /dev/sdj1                     vfat    1.1G  6.1M  1.1G   1% /boot/efi tmpfs                         tmpfs   3.2G  4.0K  3.2G   1% /run/user/0 overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/b5a55615f548ae182d48dee0eced64eb2ed43c37d696d951618e66b01b700/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/3c6ecfc4d8a3561835504c3aaee7f69ea7a117c26d3ec59ea1ceabe5994fd57/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/a504698e7cc386ebefd506402638c2f8af83ad9a35a8d0c49a7c287d8d26c24/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/380d370bc8313724d95b8fc7c70d7283c69439b014ce82936b245ee6b810c/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/71ba8a75e186df7542a2fa9667580e4b2b0db2fb78eacfea3711abe6c9d22ad/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8daea1eaddbc8c3b0f7afa35b58524ec3adcfcf48a304f988e5adbec2c4ffa/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/04f8e8635d8bc88c953b00815bc3234f26499f6eb066ae998121bf0dd4a03b4/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/bafbe7cbad3f4ab94a92eb5f098591ac316de90def61c2ca1b69559d96b62/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/4a0aa4bad8692e397ca98ed91a042e22caeb7509d3825f597bb37e9fb98a1cca/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/9c2e8075696b5edb2a38e4f8d7e866ada67b5d238ff73f8ea7b0f0669cf06660a/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8251b5ef9df5394947e5d8018a163822aac69ab1fe378cfa7359cb42d1788db/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/0762e8c175c6b568042a8e8a12b90bb46dc76b9bc41ccc972ee56c6670e2f5/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/c7f232a9a0abe297fbcfe58f5242754078a669b826726a4ed7146d1e5619cefc/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8186f868a3e87299a360a60f780cef452556fe00ace340433f11e7f922c12124/merged cluster:   id: f68c4e2e-e880-11ef-b13e-512ad216ec40   health: HEALTH_OK services:   mon: 5 daemons, quorum ceph-node001,ceph-node006,ceph-node002,ceph-node003,ceph-node005 (age 44h)   mgr: ceph-node001.zcyayo(active, since 12H), standbys: ceph-node002.wvpazs   mds: 1/1 daemons up, 1 standby   osd: 60 osds: 60 up (since 5w), 60 in (since 7M) data:   volumes: 1/1 healthy   pools: 6 pools, 2625 pg   objects: 42.59M objects, 160 TiB   usage: 477 TiB used, 396 TiB / 873 TiB avail   pg:     2591 active+clean     28 active+clean+scrubbing+deep     6 active+clean+scrubbing ceph version 17.2.7 (b12291d11b049b2f35a32e0de30d70e9a4c060d2), quincy (stable) CLASS    SIZE      AVAIL     USED      RAW USED  %RAW USED hdd      873 TiB   396 TiB   477 TiB   477 TiB   54.68 TOTAL    873 TiB   396 TiB   477 TiB   477 TiB   54.68 POOL               ID    PGS     STORAGED   OBJECTS   USED       %USED    MAX AVAIL .mgr               1     1       978 MiB    161       2.9 GiB    0        108 TiB vmware-server      2     1024    155 TiB    41.5M     464 TiB    58.79    108 TiB cephfs-metadata    4     32      195 MiB    69        584 MiB    0        108 TiB cephfs-data        5     512     1.9 GiB    490       5.7 GiB    0        108 TiB .nfs               6     32      0 B        0         0 B        0        108 TiB backup-pool        8     1024    4.1 TiB    1.0M      12 TiB     3.63     108 TiB NAME                     PORTS            RUNNING  REFRESHED  AGE   PLACEMENT alertmanager             ?:9093,9094      1/1      12s ago    12M   count:1 crash                    -                6/6      6m ago     12M   * graphana                 ?:3000           1/1      12s ago    12M   count:1 mds.cephfs-vmware-server -                2/2      4m ago     12M   count:2 mgr                      -                2/2      4m ago     12M   count:2 mon                      -                5/5      6m ago     12M   count:5 node-exporter            ?:9100           6/6      6m ago     12M   * osd                      -                60       -          -     <unmanaged> prometheus               ?:9095           1/1      12s ago    12M   count:1 NAME                              HOST             PORTS                  STATUS      REFRESHED  AGE   MEM USE  MEM LIM  VERSTION  IMAGE ID              CONTAINER ID alertmanager.ceph-node001         ceph-node001     *:9093,9094            running (12M)  10m ago  12M   32.2K    -        0.25.0                c8568f9114cd2         0bda2adb384 crash.ceph-node001                ceph-node001     -                      running (12M)  10m ago  12M   3479k    -        17.2.8                259b35566514          6bdcadd825e5 crash.ceph-node002                ceph-node002     -                      running (5M)   3m ago   12M   5227k    -        17.2.8                259b35566514          4f315378f8fb crash.ceph-node003                ceph-node003     -                      running (5M)   3m ago   12M   4328k    -        17.2.8                259b35566514          c89f20d5244 crash.ceph-node004                ceph-node004     -                      running (7M)   5m ago   7M    10.9M    -        17.2.8                259b35566514          e3f0a0a49d6a crash.ceph-node005                ceph-node005     -                      running (5M)   3m ago   12M   4551k    -        17.2.8                259b35566514          a3028d149a9c crash.ceph-node006                ceph-node006     -                      running (12M)  5m ago   12M   4219k    -        17.2.8                259b35566514          5458ecc46771 grafana.ceph-node001              ceph-node001     *:3000                 running (12M)  10m ago  12M   133M     -        9.4.7                 954c88fa6188          5536bc4858e4 mds.cephfs-vmware-server.ceph-node002.gryudx ceph-node002     -                      running (5M)   3m ago   12M   15.3M    -        17.2.8                259b35566514          b7bd424d9996 mds.cephfs-vmware-server.ceph-node005.qyxswr ceph-node005     -                      running (5M)   3m ago   12M   25.8M    -        17.2.8                259b35566514          3be6f2e7c7ce mgr.ceph-node001.zcyayo           ceph-node001     *:9283                 running (12M)  10m ago  12M   891M     -        17.2.8                259b35566514          daa1ea0d1168 mgr.ceph-node002.uvpazs           ceph-node002     *:8443,9283            running (5M)   3m ago   12M   57.3M    -        17.2.8                259b35566514          81a8cd1fe2c3 mon.ceph-node001                  ceph-node001     -                      running (7M)   10m ago  12M   477M     2048M    17.2.8                259b35566514          3e5c22bfedd4 mon.ceph-node002                  ceph-node002     -                      running (5M)   10m ago  12M   398M     2048M    17.2.8                259b35566514          c15e5b4c21c8 mon.ceph-node003                  ceph-node003     -                      running (12M)  5m ago   12M   290M     2048M    17.2.8                259b35566514          72facfdf82a8 mon.ceph-node005                  ceph-node005     -                      running (7M)   3m ago   7M    832M     2048M    17.2.8                259b35566514          3f94eb72a5e3 mon.ceph-node006                  ceph-node006     -                      running (12M)  5m ago   12M   285M     2048M    17.2.8                259b35566514          97a4a1d76116 node-exporter.ceph-node001        ceph-node001     *:9100                 running (12M)  10m ago  12M   17.3M    -        1.5.0                 0da6a335fe13          81b476bbe6fc node-exporter.ceph-node002        ceph-node002     *:9100                 running (5M)   3m ago   12M   20.2M    -        1.5.0                 0da6a335fe13          ba20a215dd70 node-exporter.ceph-node003        ceph-node003     *:9100                 running (5M)   3m ago   12M   17.4M    -        1.5.0                 0da6a335fe13          ef75144ce2b3 node-exporter.ceph-node004        ceph-node004     *:9100                 running (7M)   5m ago   7M    20.7M    -        1.5.0                 0da6a335fe13          a04852da78e8 node-exporter.ceph-node005        ceph-node005     *:9100                 running (12M)  3m ago   12M   17.1M    -        1.5.0                 0da6a335fe13          d07533213e5c node-exporter.ceph-node006        ceph-node006     *:9100                 running (12M)  5m ago   12M   16.8M    -        1.5.0                 0da6a335fe13          23f1ceec45a osd.0                             ceph-node001     -                      running (12M)  10m ago  12M   1885M    1287M    17.2.8                259b35566514          9b33a19ff54c osd.1                             ceph-node001     -                      running (12M)  10m ago  12M   1711M    1287M    17.2.8                259b35566514          49eea8cdb06d osd.2                             ceph-node001     -                      running (12M)  10m ago  12M   2118M    1287M    17.2.8                259b35566514          25cdd9efdd95 osd.3                             ceph-node001     -                      running (12M)  10m ago  12M   1639M    1287M    17.2.8                259b35566514          ed50fbe841f osd.4                             ceph-node001     -                      running (12M)  10m ago  12M   1828M    1287M    17.2.8                259b35566514          56109a380a52 osd.5                             ceph-node001     -                      running (8w)   10m ago  12M   2335M    1287M    17.2.8                259b35566514          b1f65b1df237 osd.6                             ceph-node001     -                      running (12M)  10m ago  12M   2088M    1287M    17.2.8                259b35566514          ca7d1a2835ab osd.7                             ceph-node001     -                      running (12M)  10m ago  12M   2055M    1287M    17.2.8                259b35566514          fafd75e72d5f osd.8                             ceph-node001     -                      running (12M)  10m ago  12M   2033M    1287M    17.2.8                259b35566514          1cdc6b774429 osd.9                             ceph-node002     -                      running (5M)   3m ago   12M   2307M    1185M    17.2.8                259b35566514          117c76b6d10 osd.10                            ceph-node002     -                      running (5M)   3m ago   12M   1683M    1185M    17.2.8                259b35566514          d1ce2269d143 osd.11                            ceph-node002     -                      running (5M)   3m ago   12M   2008M    1185M    17.2.8                259b35566514          b69fbf088ee osd.12                            ceph-node002     -                      running (5M)   3m ago   12M   2442M    1185M    17.2.8                259b35566514          df71fa4dd602 osd.13                            ceph-node002     -                      running (5M)   3m ago   12M   1803M    1185M    17.2.8                259b35566514          86d54722e9ab osd.14                            ceph-node002     -                      running (5M)   3m ago   12M   1999M    1185M    17.2.8                259b35566514          ab1f5734cf6f My cluster is version 17.2.7 (quincy). The basic information of the Ceph cluster is provided in the other attached pictures. Thank you! _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Why did this situation occur?
by 苏察哈尔灿 12 Feb '26

12 Feb '26

Please help me, I have a CEPH cluster with six nodes. Yesterday, one of the nodes reported insufficient disk space (image01.png), but using the command "ceph -s", it kept showing normal (image02.png). Before this, there was also a situation where the disk space of a certain node was insufficient. At that time, using the "ceph -s" command could normally display the alarm and the involved nodes. Why is this situation occurring now? Filesystem                    Type  Size  Used Avail Use% Mounted on tmpfs                         tmpfs   3.2G  3.6M  3.1G   1% /run /dev/mapper/ubuntu--vg-ubuntu--lv ext4    98G   93G   863M  100% / tmpfs                         tmpfs   16G     0     16G    0% /dev/shm tmpfs                         tmpfs   5.0M    0     5.0M   0% /run/lock /dev/sdj2                     ext4    2.0G  253M  1.6G   14% /boot /dev/sdj1                     vfat    1.1G  6.1M  1.1G   1% /boot/efi tmpfs                         tmpfs   3.2G  4.0K  3.2G   1% /run/user/0 overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/b5a55615f548ae182d48dee0eced64eb2ed43c37d696d951618e66b01b700/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/3c6ecfc4d8a3561835504c3aaee7f69ea7a117c26d3ec59ea1ceabe5994fd57/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/a504698e7cc386ebefd506402638c2f8af83ad9a35a8d0c49a7c287d8d26c24/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/380d370bc8313724d95b8fc7c70d7283c69439b014ce82936b245ee6b810c/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/71ba8a75e186df7542a2fa9667580e4b2b0db2fb78eacfea3711abe6c9d22ad/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8daea1eaddbc8c3b0f7afa35b58524ec3adcfcf48a304f988e5adbec2c4ffa/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/04f8e8635d8bc88c953b00815bc3234f26499f6eb066ae998121bf0dd4a03b4/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/bafbe7cbad3f4ab94a92eb5f098591ac316de90def61c2ca1b69559d96b62/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/4a0aa4bad8692e397ca98ed91a042e22caeb7509d3825f597bb37e9fb98a1cca/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/9c2e8075696b5edb2a38e4f8d7e866ada67b5d238ff73f8ea7b0f0669cf06660a/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8251b5ef9df5394947e5d8018a163822aac69ab1fe378cfa7359cb42d1788db/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/0762e8c175c6b568042a8e8a12b90bb46dc76b9bc41ccc972ee56c6670e2f5/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/c7f232a9a0abe297fbcfe58f5242754078a669b826726a4ed7146d1e5619cefc/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8186f868a3e87299a360a60f780cef452556fe00ace340433f11e7f922c12124/merged cluster:   id: f68c4e2e-e880-11ef-b13e-512ad216ec40   health: HEALTH_OK services:   mon: 5 daemons, quorum ceph-node001,ceph-node006,ceph-node002,ceph-node003,ceph-node005 (age 44h)   mgr: ceph-node001.zcyayo(active, since 12H), standbys: ceph-node002.wvpazs   mds: 1/1 daemons up, 1 standby   osd: 60 osds: 60 up (since 5w), 60 in (since 7M) data:   volumes: 1/1 healthy   pools: 6 pools, 2625 pg   objects: 42.59M objects, 160 TiB   usage: 477 TiB used, 396 TiB / 873 TiB avail   pg:     2591 active+clean     28 active+clean+scrubbing+deep     6 active+clean+scrubbing ceph version 17.2.7 (b12291d11b049b2f35a32e0de30d70e9a4c060d2), quincy (stable) CLASS    SIZE      AVAIL     USED      RAW USED  %RAW USED hdd      873 TiB   396 TiB   477 TiB   477 TiB   54.68 TOTAL    873 TiB   396 TiB   477 TiB   477 TiB   54.68 POOL               ID    PGS     STORAGED   OBJECTS   USED       %USED    MAX AVAIL .mgr               1     1       978 MiB    161       2.9 GiB    0        108 TiB vmware-server      2     1024    155 TiB    41.5M     464 TiB    58.79    108 TiB cephfs-metadata    4     32      195 MiB    69        584 MiB    0        108 TiB cephfs-data        5     512     1.9 GiB    490       5.7 GiB    0        108 TiB .nfs               6     32      0 B        0         0 B        0        108 TiB backup-pool        8     1024    4.1 TiB    1.0M      12 TiB     3.63     108 TiB NAME                     PORTS            RUNNING  REFRESHED  AGE   PLACEMENT alertmanager             ?:9093,9094      1/1      12s ago    12M   count:1 crash                    -                6/6      6m ago     12M   * graphana                 ?:3000           1/1      12s ago    12M   count:1 mds.cephfs-vmware-server -                2/2      4m ago     12M   count:2 mgr                      -                2/2      4m ago     12M   count:2 mon                      -                5/5      6m ago     12M   count:5 node-exporter            ?:9100           6/6      6m ago     12M   * osd                      -                60       -          -     <unmanaged> prometheus               ?:9095           1/1      12s ago    12M   count:1 NAME                              HOST             PORTS                  STATUS      REFRESHED  AGE   MEM USE  MEM LIM  VERSTION  IMAGE ID              CONTAINER ID alertmanager.ceph-node001         ceph-node001     *:9093,9094            running (12M)  10m ago  12M   32.2K    -        0.25.0                c8568f9114cd2         0bda2adb384 crash.ceph-node001                ceph-node001     -                      running (12M)  10m ago  12M   3479k    -        17.2.8                259b35566514          6bdcadd825e5 crash.ceph-node002                ceph-node002     -                      running (5M)   3m ago   12M   5227k    -        17.2.8                259b35566514          4f315378f8fb crash.ceph-node003                ceph-node003     -                      running (5M)   3m ago   12M   4328k    -        17.2.8                259b35566514          c89f20d5244 crash.ceph-node004                ceph-node004     -                      running (7M)   5m ago   7M    10.9M    -        17.2.8                259b35566514          e3f0a0a49d6a crash.ceph-node005                ceph-node005     -                      running (5M)   3m ago   12M   4551k    -        17.2.8                259b35566514          a3028d149a9c crash.ceph-node006                ceph-node006     -                      running (12M)  5m ago   12M   4219k    -        17.2.8                259b35566514          5458ecc46771 grafana.ceph-node001              ceph-node001     *:3000                 running (12M)  10m ago  12M   133M     -        9.4.7                 954c88fa6188          5536bc4858e4 mds.cephfs-vmware-server.ceph-node002.gryudx ceph-node002     -                      running (5M)   3m ago   12M   15.3M    -        17.2.8                259b35566514          b7bd424d9996 mds.cephfs-vmware-server.ceph-node005.qyxswr ceph-node005     -                      running (5M)   3m ago   12M   25.8M    -        17.2.8                259b35566514          3be6f2e7c7ce mgr.ceph-node001.zcyayo           ceph-node001     *:9283                 running (12M)  10m ago  12M   891M     -        17.2.8                259b35566514          daa1ea0d1168 mgr.ceph-node002.uvpazs           ceph-node002     *:8443,9283            running (5M)   3m ago   12M   57.3M    -        17.2.8                259b35566514          81a8cd1fe2c3 mon.ceph-node001                  ceph-node001     -                      running (7M)   10m ago  12M   477M     2048M    17.2.8                259b35566514          3e5c22bfedd4 mon.ceph-node002                  ceph-node002     -                      running (5M)   10m ago  12M   398M     2048M    17.2.8                259b35566514          c15e5b4c21c8 mon.ceph-node003                  ceph-node003     -                      running (12M)  5m ago   12M   290M     2048M    17.2.8                259b35566514          72facfdf82a8 mon.ceph-node005                  ceph-node005     -                      running (7M)   3m ago   7M    832M     2048M    17.2.8                259b35566514          3f94eb72a5e3 mon.ceph-node006                  ceph-node006     -                      running (12M)  5m ago   12M   285M     2048M    17.2.8                259b35566514          97a4a1d76116 node-exporter.ceph-node001        ceph-node001     *:9100                 running (12M)  10m ago  12M   17.3M    -        1.5.0                 0da6a335fe13          81b476bbe6fc node-exporter.ceph-node002        ceph-node002     *:9100                 running (5M)   3m ago   12M   20.2M    -        1.5.0                 0da6a335fe13          ba20a215dd70 node-exporter.ceph-node003        ceph-node003     *:9100                 running (5M)   3m ago   12M   17.4M    -        1.5.0                 0da6a335fe13          ef75144ce2b3 node-exporter.ceph-node004        ceph-node004     *:9100                 running (7M)   5m ago   7M    20.7M    -        1.5.0                 0da6a335fe13          a04852da78e8 node-exporter.ceph-node005        ceph-node005     *:9100                 running (12M)  3m ago   12M   17.1M    -        1.5.0                 0da6a335fe13          d07533213e5c node-exporter.ceph-node006        ceph-node006     *:9100                 running (12M)  5m ago   12M   16.8M    -        1.5.0                 0da6a335fe13          23f1ceec45a osd.0                             ceph-node001     -                      running (12M)  10m ago  12M   1885M    1287M    17.2.8                259b35566514          9b33a19ff54c osd.1                             ceph-node001     -                      running (12M)  10m ago  12M   1711M    1287M    17.2.8                259b35566514          49eea8cdb06d osd.2                             ceph-node001     -                      running (12M)  10m ago  12M   2118M    1287M    17.2.8                259b35566514          25cdd9efdd95 osd.3                             ceph-node001     -                      running (12M)  10m ago  12M   1639M    1287M    17.2.8                259b35566514          ed50fbe841f osd.4                             ceph-node001     -                      running (12M)  10m ago  12M   1828M    1287M    17.2.8                259b35566514          56109a380a52 osd.5                             ceph-node001     -                      running (8w)   10m ago  12M   2335M    1287M    17.2.8                259b35566514          b1f65b1df237 osd.6                             ceph-node001     -                      running (12M)  10m ago  12M   2088M    1287M    17.2.8                259b35566514          ca7d1a2835ab osd.7                             ceph-node001     -                      running (12M)  10m ago  12M   2055M    1287M    17.2.8                259b35566514          fafd75e72d5f osd.8                             ceph-node001     -                      running (12M)  10m ago  12M   2033M    1287M    17.2.8                259b35566514          1cdc6b774429 osd.9                             ceph-node002     -                      running (5M)   3m ago   12M   2307M    1185M    17.2.8                259b35566514          117c76b6d10 osd.10                            ceph-node002     -                      running (5M)   3m ago   12M   1683M    1185M    17.2.8                259b35566514          d1ce2269d143 osd.11                            ceph-node002     -                      running (5M)   3m ago   12M   2008M    1185M    17.2.8                259b35566514          b69fbf088ee osd.12                            ceph-node002     -                      running (5M)   3m ago   12M   2442M    1185M    17.2.8                259b35566514          df71fa4dd602 osd.13                            ceph-node002     -                      running (5M)   3m ago   12M   1803M    1185M    17.2.8                259b35566514          86d54722e9ab osd.14                            ceph-node002     -                      running (5M)   3m ago   12M   1999M    1185M    17.2.8                259b35566514          ab1f5734cf6f My cluster is version 17.2.7 (quincy). The basic information of the Ceph cluster is provided in the other attached pictures. Thank you!

1 0

ceph
by 苏察哈尔灿 12 Feb '26

12 Feb '26

Please help me, I have a CEPH cluster with six nodes. Yesterday, one of the nodes reported insufficient disk space (image01.png), but using the command "ceph -s", it kept showing normal (image02.png). Before this, there was also a situation where the disk space of a certain node was insufficient. At that time, using the "ceph -s" command could normally display the alarm and the involved nodes. Why is this situation occurring now?  My cluster is version 17.2.7 (quincy). The basic information of the Ceph cluster is provided in the other attached pictures. Thank you!

1 0