Something else to try might be to delete the .mgr pool and let it be recreated.
On Jun 28, 2026, at 9:31 AM, Eugen Block via ceph-users <ceph-users@ceph.io> wrote:
Hi,
I don’t have a response to all your questions, but will try to answer a few.
Zitat von Chris Palmer via ceph-users <ceph-users@ceph.io>:
Due to a number of power and heat events, I had to power down a small ceph cluster last night. There was only time to do a operating system shutdown of all nodes.
When the systems rebooted a few hours later, all was ok except that all 3 MGRs would abort and restart until they reached the restart limit. This left the system with no MGRs and no orchestration. All OSD, MDS, RGW & monitoring subsystems were working as normally as could be expected without an MGR.
Analysis of the log files showed each MGR aborting with the assertion below. Every one was in the telemetry thread. I disabled telemetry and the MGRs could be restarted normally. Questions:
* What caused the assertion error? Was it related to the total shutdown? * Aborting the entire MGR seems overkill. Disabling the telemetry module and raising a warning seems more proportionate. * When running with no active MGR the client IO stats in ceph -s were frozen (at plausible non-zero values). Is that expected with no active MGR?
Yes, OSDs report their stats to the MGR which is responsible for displaying those in tje status.
* With no active MGR a large number of deep-scrubs started - roughly 3 times the maximum number we ever see were actively deep-scrubbing, with none in the scrub wait state, and response times were horrible. Is this expected with no MGR to control it properly? I had to temporarily disable scrubs and deep-scrubs as a workaround.
I see the same after restarting an entire cluster (mostly in small lab environments), I don’t think that has to do with no active mgr. Although mgr and osd have each a deep-scrub interval, the OSDs are responsible for scheduling. So this is most likely just how OSDs work after a cluster reboot.
I haven't yet tried 20.2.2, but I couldn't see anything obvious in the release notes. (Although there were about 3 PRs that I thought might have had the potential to upset exception handling...).
Thanks, Chris
{ "crash_id": "2026-06-26T01:46:00.265207Z_b0579062-d65c-4a51-8dbc-d9837209aa80", "timestamp": "2026-06-26T01:46:00.265207Z", "process_name": "ceph-mgr", "entity_name": "mgr.ceph2", "ceph_version": "20.2.1", "utsname_hostname": "ceph2", "utsname_sysname": "Linux", "utsname_release": "7.0.12-201.fc44.x86_64", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Jun 11 01:30:16 UTC 2026", "utsname_machine": "x86_64", "os_name": "CentOS Stream", "os_id": "centos", "os_version_id": "9", "os_version": "9", "assert_condition": "cursor != root", "assert_func": "virtual void PyFormatter::close_section()", "assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h", "assert_line": 84, "assert_thread_name": "telemetry", "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: In function 'virtual void PyFormatter::close_section()' thread 7f752186e640 time 2026-06-26T01:46:00.263380+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: 84: FAILED ceph_assert(cursor != root)\n", "backtrace": [ "/lib64/libc.so.6(+0x3fc30) [0x7f75a9eb8c30]", "/lib64/libc.so.6(+0x8d05c) [0x7f75a9f0605c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f75aa9ed95b]", "/usr/bin/ceph-mgr(+0x6f7a5) [0x55ac462367a5]", "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x15a3) [0x55ac462d3143]", "/usr/bin/ceph-mgr(+0x12ba12) [0x55ac462f2a12]", "/lib64/libpython3.9.so.1.0(+0x15b4b7) [0x7f75ab5e34b7]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", "/lib64/libpython3.9.so.1.0(+0x14fa85) [0x7f75ab5d7a85]", "/lib64/libpython3.9.so.1.0(+0x138725) [0x7f75ab5c0725]", "_PyObject_CallMethod_SizeT()", "(PyModuleRunner::serve()+0x66) [0x55ac463759e6]", "(PyModuleRunner::PyModuleRunnerThread::entry()+0x139) [0x55ac463761e9]", "/lib64/libc.so.6(+0x8b31a) [0x7f75a9f0431a]" ] } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io