I've just applied routine Centos 9 updates to one node of a Reef 18.2.4 system (package install). They include some python3 updates that break the MGR in at least two ways. When the MGR starts in standby, it immediately logs the following two crashes. Only the dashboard and restful services are enabled (prometheus is not enabled). We have mgr/dashboard/FEATURE_TOGGLE_DASHBOARD = false. { "archived": "2024-11-19 10:30:46.031489", "backtrace": [ " File \"/usr/share/ceph/mgr/dashboard/__init__.py\", line 60, in <module>\n from .module import Module, StandbyModule # noqa: F401", " File \"/usr/share/ceph/mgr/dashboard/module.py\", line 51, in <module>\n patch_cherrypy(cherrypy.__version__)", " File \"/usr/share/ceph/mgr/dashboard/cherrypy_backports.py\", line 197, in patch_cherrypy\n accept_exceptions_from_builtin_ssl(ver)", " File \"/usr/share/ceph/mgr/dashboard/cherrypy_backports.py\", line 113, in accept_exceptions_from_builtin_ssl\n patch_builtin_ssl_wrap(v, accept_ssl_errors)", " File \"/usr/share/ceph/mgr/dashboard/cherrypy_backports.py\", line 75, in patch_builtin_ssl_wrap\n from cherrypy.wsgiserver.ssl_builtin import BuiltinSSLAdapter as builtin_ssl", "ModuleNotFoundError: No module named 'cherrypy.wsgiserver'" ], "ceph_version": "18.2.4", "crash_id": "2024-11-19T09:19:40.427015Z_f96f4d70-2112-47b5-96f7-5e9bd463e8eb", "entity_name": "mgr.ceph1", "mgr_module": "dashboard", "mgr_module_caller": "PyModule::load_subclass_of", "mgr_python_exception": "ModuleNotFoundError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mgr", "stack_sig": "9c98ebf1b6831bfca2823f54c9e6be01306090c2a7749def2ed8a15167fc527a", "timestamp": "2024-11-19T09:19:40.427015Z", "utsname_hostname": "ceph1.xxxxxx", "utsname_machine": "x86_64", "utsname_release": "6.1.112-1.el9.elrepo.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Mon Sep 30 13:59:36 EDT 2024" } { "archived": "2024-11-19 10:30:46.056007", "backtrace": [ " File \"/usr/share/ceph/mgr/prometheus/__init__.py\", line 2, in <module>\n from .module import Module, StandbyModule", " File \"/usr/share/ceph/mgr/prometheus/module.py\", line 38, in <module>\n v = Version(cherrypy.__version__)", " File \"/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py\", line 277, in __init__\n raise InvalidVersion(\"Invalid version: '{0}'\".format(version))", "pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: 'unknown'" ], "ceph_version": "18.2.4", "crash_id": "2024-11-19T09:19:42.188291Z_c112de10-cdd5-4ed3-86b5-04dffd660cb8", "entity_name": "mgr.ceph1", "mgr_module": "prometheus", "mgr_module_caller": "PyModule::load_subclass_of", "mgr_python_exception": "InvalidVersion", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mgr", "stack_sig": "7fb0c6c17573887e8772c311aac1f3547c8366e50d6f2f8d8bd38deb6e0e9405", "timestamp": "2024-11-19T09:19:42.188291Z", "utsname_hostname": "ceph1.xxxxxx", "utsname_machine": "x86_64", "utsname_release": "6.1.112-1.el9.elrepo.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Mon Sep 30 13:59:36 EDT 2024" } The MGR can be made active, but attempts to use the dashboard generate more of these. The culprit turns out to be python3-jaraco-text from the epel repository, which upgraded from python3-jaraco-text-3.2.0-6.el9.noarch to python3-jaraco-text-4.0.0-2.el9.noarch. (It also installed python3-jaraco-context-6.0.1-3.el9.noarch as a new dependency). Reverting to python3-jaraco-text-3.2.0 has avoided the problem for now, but that's not a sustainable long-term fix.