Quite a few regressions turned up in CI this week, most of them fallout from the CephX CVE fixes. Here's a summary and status. A few unrelated breakages landed in the same window and are marked [not CephX] so nobody chases the wrong cause. Everything below is fixed on main. Where a release branch is listed as "n/a", that branch never had the breakage. == Make Check Failures (ceph-pull-requests / ceph-pull-requests-arm64) == = mon/MonClient: delete move assignment = Symptoms: * Build dies 4-16 minutes in, before any test runs; "ninja: build stopped: subcommand failed." in red in the Console Log * Header error pointing at src/mon/MonClient.h in the Jenkins Web UI (line number varies by branch: main 70, tentacle 67, squid 62) * Also fails "ceph API tests" the same way ``` FAILED: src/CMakeFiles/common-objs.dir/mon/MonClient.cc.o /ceph/src/mon/MonClient.h:70:18: error: explicitly defaulted move assignment operator is implicitly deleted [-Werror,-Wdefaulted-function-deleted] 70 | MonConnection& operator=(MonConnection&&) = default; | ^ /ceph/src/mon/MonClient.h:151:16: note: move assignment operator of 'MonConnection' is implicitly deleted because field 'monc_lock' is of reference type 'ceph::mutex &' ninja: build stopped: subcommand failed. ``` Cause: the CephX work added a "ceph::mutex& monc_lock" reference member to MonConnection, which makes the explicitly defaulted move assignment operator implicitly deleted. clang rejects that under -Werror. Nothing move-assigns a MonConnection, so it is now = delete. Fixes: * main: https://github.com/ceph/ceph/pull/71165 (merged) * umbrella: n/a - CephX changes are not on umbrella yet; they arrive with https://github.com/ceph/ceph/pull/71172 and this fix will need to go with them * tentacle: https://github.com/ceph/ceph/pull/71191 (merged) * squid: https://github.com/ceph/ceph/pull/71190 (merged) = mgr/cephadm: IndexError during upgrade key rotation = Symptoms: * "4 - run-tox-mgr (Failed)" in the CTest summary ``` FAILED cephadm/tests/test_upgrade.py::test_upgrade_daemons_offline_hosts - IndexError: list index out of range FAILED cephadm/tests/test_upgrade.py::test_upgrade_run[False] - IndexError: list index out of range FAILED cephadm/tests/test_upgrade.py::test_upgrade_run[True] - orchestrator._interface.OrchestratorError: Invalid container image 'to_image@repo_digest' FAILED cephadm/tests/test_upgrade.py::test_do_upgrade_limit_exhausted_marks_complete_without_scope_check E IndexError: list index out of range cephadm/upgrade.py:1795: IndexError ``` Cause: _upgrade_daemons() now calls _rotate_mgr_mon_auth_keys() on every pass. With no mon daemons in the cache, _detect_need_upgrade() reports "no mons need upgrade", which is taken as "all mons upgraded", so it sets auth_allowed_ciphers, rotates mgr keys, and then indexes mon_daemons[0] on an empty list. Now it skips the rotation for that pass and retries later. Fixes: * main: https://github.com/ceph/ceph/pull/71165 (merged) * umbrella: n/a - arrives with #71172 * tentacle: https://github.com/ceph/ceph/pull/71191 (merged) * squid: https://github.com/ceph/ceph/pull/71190 (merged) = common/entity_name: raw bytes instead of type names = Symptoms: * "35 - run-cli-tests (Failed)"; src/test/cli/ceph-conf/invalid-args.t fails * Any user-facing message listing valid entity types is garbage, e.g. from `ceph-conf --name total.garbage` ``` /ceph/src/test/cli/ceph-conf/invalid-args.t: failed --- /ceph/src/test/cli/ceph-conf/invalid-args.t +++ /ceph/src/test/cli/ceph-conf/invalid-args.t.err - error parsing 'total.garbage': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client + error parsing 'total.garbage': expected string of the form TYPE.ID, valid types are: , \x01, \x04, \x02, \x10, \x08 (esc) ``` Cause: a real bug, not just a test expectation. STR_TO_ENTITY_TYPE became a vector of (entity_type_t, std::string) pairs but get_valid_types_as_str() kept streaming .first, i.e. the uint8_t type value, instead of .second. Fixes: * main: https://github.com/ceph/ceph/pull/71165 (merged) * umbrella: n/a - arrives with #71172 * tentacle: https://github.com/ceph/ceph/pull/71191 (merged) * squid: https://github.com/ceph/ceph/pull/71190 (merged) = test/cli: stale cram expectations for the new auth options = Symptoms: * "35 - run-cli-tests (Failed)", 10 .t files reported as failed with a diff of expected vs actual tool output ``` /ceph/src/test/cli/ceph-authtool/help.t: failed + -t, --key-type KEY_TYPE set the key-type # Ran 15 tests, 0 skipped, 1 failed. /ceph/src/test/cli/monmaptool/help.t: failed + [--auth-service-cipher <cipher>] + [--auth-allowed-ciphers <cipher1,cipher2,...>] + [--auth-preferred-cipher <cipher>] /ceph/src/test/cli/monmaptool/add-exists.t: failed + auth_epoch 0 + auth_service_cipher aes256k + auth_allowed_ciphers aes256k + auth_preferred_cipher aes256k /ceph/src/test/cli/monmaptool/feature-set-unset-list.t: failed - supported: [...,umbrella(4096),nvmeof_beacon_diff(4294967296)] + supported: [...,umbrella(4096),cephx_auth_aes256k(2147483648),nvmeof_beacon_diff(4294967296)] ``` Cause: ceph-authtool gained -t/--key-type, monmaptool gained the --auth-* options and now prints auth_epoch / auth_service_cipher / auth_allowed_ciphers / auth_preferred_cipher, and the cephx_auth_aes256k mon feature bit shows up in feature lists - none of which were reflected in the cram expectations. Fixes: * main: https://github.com/ceph/ceph/pull/71165 (merged) * umbrella: n/a - arrives with #71172 * tentacle: https://github.com/ceph/ceph/pull/71191 (merged) * squid: https://github.com/ceph/ceph/pull/71190 (merged) Note for backporters: the feature lists are release-specific, so monmaptool/feature-set-unset-list.t cannot be cherry-picked verbatim - rebuild the expected output from that branch's src/mon/mon_types.h. = qa/tasks: flake8 findings = Symptoms: * "323 - run-tox-qa (Failed)" ``` flake8: commands[0] /ceph/qa> flake8 --select=F,E9 --exclude=venv,.tox ./tasks/ceph.py:1733:5: F841 local variable 'testdir' is assigned to but never used ./tasks/ceph.py:1797:5: F841 local variable 'clusters' is assigned to but never used ./tasks/ceph.py:1839:5: F841 local variable 'firstmon' is assigned to but never used ./tasks/ceph.py:2280:13: F841 local variable 'firstmon' is assigned to but never used ./tasks/radosbench.py:113:13: F541 f-string is missing placeholders ./tasks/radosbench.py:166:9: F841 local variable 'e' is assigned to but never used flake8: FAIL code 1 ``` Fixes: * main: https://github.com/ceph/ceph/pull/71165 (merged) * umbrella: n/a - arrives with #71172 * tentacle: https://github.com/ceph/ceph/pull/71191 (merged) * squid: https://github.com/ceph/ceph/pull/71190 (merged) = cephadm: unformatted OSD key rotation code (check-black) = Symptoms: * "22 - run-tox-cephadm (Failed)" with flake8/mypy/py3 all OK and only check-black failing ``` would reformat /ceph/src/cephadm/cephadmlib/daemons/ceph.py Oh no! 💥 💔 💥 1 file would be reformatted, 50 files would be left unchanged. check-black: FAIL code 1 ``` Cause: the OSD keyring rotation changes were not run through `tox -e format-black` (black >= 23, < 24, -l78). Fixes: * main: https://github.com/ceph/ceph/pull/71165 (merged) * umbrella: n/a - arrives with #71172 * tentacle: https://github.com/ceph/ceph/pull/71191 (merged) * squid: https://github.com/ceph/ceph/pull/71190 (merged) = test/osd: missing stringify.h include (tentacle and squid only) = Symptoms: * Compile failure that only appears once the MonClient.h break above is fixed ``` FAILED: src/test/osd/CMakeFiles/unittest_osdmap.dir/TestOSDMap.cc.o /ceph/src/test/osd/TestOSDMap.cc:2829:39: error: use of undeclared identifier 'stringify' ninja: build stopped: subcommand failed. ``` Cause: TestOSDMap.cc uses stringify() but never included include/stringify.h; it compiled via a transitive include that the CephX merge removed. main already had the include. Fixes: * main: n/a - already has the include (073584877f5) * umbrella: n/a * tentacle: https://github.com/ceph/ceph/pull/71191 (merged) * squid: https://github.com/ceph/ceph/pull/71190 (merged) = mgr/cephadm: mypy attr-defined errors (tentacle only) = Symptoms: * "4 - run-tox-mgr (Failed)" with py3 passing and only mypy failing ``` cephadm/upgrade.py: note: In member "_mark_upgrade_complete" of class "CephadmUpgrade": cephadm/upgrade.py:1418: error: "CephadmUpgrade" has no attribute "_unset_noautoscale" [attr-defined] cephadm/upgrade.py:1419: error: "UpgradeState" has no attribute "noautoscale_set" [attr-defined] Found 2 errors in 1 file (checked 33 source files) mypy: FAIL code 1 ``` Cause: the tentacle CephX series carried over a hunk of main's _mark_upgrade_complete() that belongs to the PG autoscaler feature (main's da4b361e63c), which was never backported. Nothing on tentacle sets noautoscale_set, so the block is unreachable dead code; it is being dropped rather than backporting a feature to a stable branch. Fixes: * main: n/a - has the full feature * umbrella: n/a * tentacle: https://github.com/ceph/ceph/pull/71206 (open) * squid: n/a = cephadm/tests: intermittent pyfakefs failure [not CephX] = Symptoms: * "22 - run-tox-cephadm (Failed)" in roughly 1 run in 4 on arm64 and 1 in 11 on x86, passing on a retest * One root error followed by 80-90 identical cascading errors ``` E RuntimeError: dictionary changed size during iteration .tox/py3/lib/python3.10/site-packages/pyfakefs/fake_filesystem_unittest.py:831: RuntimeError E AttributeError: 'NoneType' object has no attribute 'add_real_directory' tests/fixtures.py:89: AttributeError ============ 9 failed, 379 passed, 17 warnings, 87 errors in 35.40s ============ ``` Cause: not arch-specific and not CephX. TestBootstrap leaves NamedTemporaryFiles on the pyfakefs fake filesystem; their finalizers raise at GC time, pytest's unraisable-exception hook then lazily imports tracemalloc from inside that GC pass, and if it lands while pyfakefs is iterating sys.modules the fs fixture setup dies. Because pyfakefs' Patcher is a ref-counted singleton, every later fs test then fails with the NoneType error. Fixed by pre-importing tracemalloc in src/cephadm/tests/conftest.py. Fixes: * main: https://github.com/ceph/ceph/pull/71163 (merged) * umbrella: needed, not yet posted * tentacle: https://github.com/ceph/ceph/pull/71189 (open) * squid: https://github.com/ceph/ceph/pull/71188 (open) == GitHub Actions Failures == = Check for missing .qa links = Symptoms: * The "Check for missing .qa links" check fails on every PR against the branch and prints the patch it wants applied ``` Please apply the following patch: diff --git a/qa/suites/rados/upgrade/cephx/X/ms_bind/.qa b/qa/suites/rados/upgrade/cephx/X/ms_bind/.qa new file mode 120000 +../.qa/ ... ##[error]Process completed with exit code 1. ``` Cause: the new qa/suites/rados/upgrade/cephx suites were added without the .qa symlinks in X/ms_bind, X/tasks/07-client-auth-keys, X/tasks/08-allowed-ciphers and the release-X counterparts. Fixes: * main: https://github.com/ceph/ceph/pull/71165 (merged) * umbrella: n/a - arrives with #71172 * tentacle: https://github.com/ceph/ceph/pull/71191 (merged) * squid: https://github.com/ceph/ceph/pull/71190 (merged) == Other Jenkins Job Failures == = ceph windows tests: OpenSSL 3 headers (tentacle and squid only) = Symptoms: * "ceph windows tests" fails on every PR against the branch ``` FAILED: src/auth/CMakeFiles/common-auth-objs.dir/Crypto.cc.obj /home/ubuntu/ceph/src/auth/Crypto.cc:39:10: fatal error: 'openssl/core_names.h' file not found ninja: build stopped: subcommand failed. ``` Cause: the CephX code includes <openssl/core_names.h>, an OpenSSL 3.x header, but win32_deps_build.sh on the stable branches still pinned sslTag="OpenSSL_1_1_1c". main has built the Windows deps with openssl-3.0.16 since May 2025. Fixes: * main: n/a - already on openssl-3.0.16 * umbrella: n/a * tentacle: https://github.com/ceph/ceph/pull/71204 (merged) * squid: https://github.com/ceph/ceph/pull/71203 (merged) = ceph dashboard cephadm e2e: chrome apt repo [not CephX] = Symptoms: * "ceph dashboard cephadm e2e tests" fails after 1-2 minutes, before any test runs ``` + sudo rm /etc/apt/sources.list.d/google-chrome.list rm: cannot remove '/etc/apt/sources.list.d/google-chrome.list': No such file or directory Build step 'Execute shell' marked build as failure ``` Cause: install-e2e-test-deps.sh hand-wrote the Google repo into google-chrome.list, the same file the google-chrome-stable package manages. As of 151.0.7922.169-1 its postinst migrates that legacy .list to a deb822 google-chrome.sources and deletes the .list, so our cleanup `rm` aborted the script under `set -e`. The repo is now added and removed with add-apt-repository. Fix: * ceph-build: https://github.com/ceph/ceph-build/pull/2685 (merged) Note: this job can still fail for unrelated reasons (Cypress specs, kcli cluster bring-up); those failures also occur on main PRs. == Teuthology Run Failures == = Rocky 10.2 release [not CephX] = Symptoms: * Every job on a rocky_10 fragment fails in the install task ``` teuthology.exceptions.CommandFailedError: Command failed on smithi174 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ...' ``` * The useful part is further up in the log ``` Error: Problem 1: package ceph-osd-2:21.3.0-2013.g4a522c91.el10.x86_64 from Ceph requires ceph-osd-crimson = 2:21.3.0-2013.g4a522c91.el10, but none of the providers can be installed - nothing provides c-ares(x86-64) >= 1.28.0 needed by ceph-osd-crimson-2:21.3.0-2013.g4a522c91.el10.x86_64 from Ceph Problem 2: package ceph ... requires ceph-osd ... Problem 3: package ceph-osd-classic ... requires ceph-osd ... Problem 4: package ceph-volume ... requires ceph-osd ... ``` * Confirm which minor the node actually booted, near the top of teuthology.log ``` INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="10.1" INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="Rocky Linux 10.1 (Red Quartz)" ``` Cause: our packages are built inside a container image they are targeted for. E.g., Rocky 10 packages are built in an UNTAGGED (so latest) Rocky 10 container image. The Ceph container image itself is also based on the latest Rocky container image. We only had a Rocky 10.1 FOG image. When 10.2 came out, our CI correctly started building Ceph packages and containers based on it, which made package dependencies change. But FOG lagged behind. I've since captured a smithi and trial_rocky_10 FOG image that is based on 10.2. I've also made a modification to the distro-mirror OpenShift service that will e-mail my team when the rocky/10 symlink moves to a new minor version so we can capture a new FOG image. Fixes: * teuthology: https://github.com/ceph/teuthology/pull/2246 (merged) * main: https://github.com/ceph/ceph/pull/71169 (merged) * umbrella: https://github.com/ceph/ceph/pull/71195 (open) * tentacle: https://github.com/ceph/ceph/pull/71194 (open) * squid: n/a Note: jobs that land on a node already running an older Rocky 10 minor are not re-imaged, so they will keep hitting this until those nodes are re-imaged onto the 10.2 image. = fs:upgrade:mds_upgrade_sequence: roleless cluster with mon roles = Symptoms: * Every job in upgrade:cephfs (fs:upgrade:mds_upgrade_sequence) fails during cephadm bootstrap; 32/55 failed and 0 passed in the run that surfaced it ``` INFO:tasks.cephadm:Waiting for 5 mons in monmap... File ".../qa/tasks/cephadm.py", line 1028, in ceph_mons while proceed(): teuthology.exceptions.MaxWhileTries: reached maximum tries (180) after waiting for 180 seconds ``` * In the mon journal on the affected host ``` Processor -- bind unable to bind to v2:10.20.193.155:3300/0: (98) Address already in use Processor -- bind was unable to bind. Trying again in 5 seconds ``` * A giveaway in the same log is host-named mons alongside the role-named ones, e.g. "mon.trial155 calling monitor election" next to mon.a / mon.b / mon.c Cause: "qa: list mons/mgrs for cephadm install" replaced the suite's host.a/host.b roles with explicit mon/mgr roles but left 'roleless: true' in the 0-from yamls. In roleless mode the cephadm task fabricates host-named mons on top of mon.a/b/c (5 mons over 2 hosts) and bootstraps without --orphan-initial-daemons, so cephadm's default mon spec races with the explicit placement and deploys a host-named mon on the ports assigned to mon.b. mon.b then crash-loops and the monmap never reaches 5 mons. The suite is now fully role-based, and the cephadm task fails fast with a clear message if a roleless cluster defines mon roles instead of timing out after 180s. Fixes: * main: https://github.com/ceph/ceph/pull/71197 (merged) * umbrella: n/a - umbrella's copy of the suite is still host-based and self-consistent; it will only need this if the roles change is backported there * tentacle: backport prepared, PR to follow * squid: n/a - suite is already fully role-based there -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway@ibm.com IBM