Due to a number of power and heat events, I had to power down a small ceph cluster last night. There was only time to do a operating system shutdown of all nodes. When the systems rebooted a few hours later, all was ok except that all 3 MGRs would abort and restart until they reached the restart limit. This left the system with no MGRs and no orchestration. All OSD, MDS, RGW & monitoring subsystems were working as normally as could be expected without an MGR. Analysis of the log files showed each MGR aborting with the assertion below. Every one was in the telemetry thread. I disabled telemetry and the MGRs could be restarted normally. Questions: * What caused the assertion error? Was it related to the total shutdown? * Aborting the entire MGR seems overkill. Disabling the telemetry module and raising a warning seems more proportionate. * When running with no active MGR the client IO stats in ceph -s were frozen (at plausible non-zero values). Is that expected with no active MGR? * With no active MGR a large number of deep-scrubs started - roughly 3 times the maximum number we ever see were actively deep-scrubbing, with none in the scrub wait state, and response times were horrible. Is this expected with no MGR to control it properly? I had to temporarily disable scrubs and deep-scrubs as a workaround. I haven't yet tried 20.2.2, but I couldn't see anything obvious in the release notes. (Although there were about 3 PRs that I thought might have had the potential to upset exception handling...). Thanks, Chris { "crash_id": "2026-06-26T01:46:00.265207Z_b0579062-d65c-4a51-8dbc-d9837209aa80", "timestamp": "2026-06-26T01:46:00.265207Z", "process_name": "ceph-mgr", "entity_name": "mgr.ceph2", "ceph_version": "20.2.1", "utsname_hostname": "ceph2", "utsname_sysname": "Linux", "utsname_release": "7.0.12-201.fc44.x86_64", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Jun 11 01:30:16 UTC 2026", "utsname_machine": "x86_64", "os_name": "CentOS Stream", "os_id": "centos", "os_version_id": "9", "os_version": "9", "assert_condition": "cursor != root", "assert_func": "virtual void PyFormatter::close_section()", "assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h", "assert_line": 84, "assert_thread_name": "telemetry", "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: In function 'virtual void PyFormatter::close_section()' thread 7f752186e640 time 2026-06-26T01:46:00.263380+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: 84: FAILED ceph_assert(cursor != root)\n", "backtrace": [ "/lib64/libc.so.6(+0x3fc30) [0x7f75a9eb8c30]", "/lib64/libc.so.6(+0x8d05c) [0x7f75a9f0605c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f75aa9ed95b]", "/usr/bin/ceph-mgr(+0x6f7a5) [0x55ac462367a5]", "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x15a3) [0x55ac462d3143]", "/usr/bin/ceph-mgr(+0x12ba12) [0x55ac462f2a12]", "/lib64/libpython3.9.so.1.0(+0x15b4b7) [0x7f75ab5e34b7]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", "/lib64/libpython3.9.so.1.0(+0x14fa85) [0x7f75ab5d7a85]", "/lib64/libpython3.9.so.1.0(+0x138725) [0x7f75ab5c0725]", "_PyObject_CallMethod_SizeT()", "(PyModuleRunner::serve()+0x66) [0x55ac463759e6]", "(PyModuleRunner::PyModuleRunnerThread::entry()+0x139) [0x55ac463761e9]", "/lib64/libc.so.6(+0x8b31a) [0x7f75a9f0431a]" ] }