A couple updates: If you are seeing "build-with-container.py: error: unrecognized arguments: --image-variant=packages” in your build logs, you need to pull https://github.com/ceph/ceph/commit/25efee4ba3de897bd91019d18ad9de72da00b4cd into your branch. "error: Failed build dependencies:” - I am investigating this right now. Definitely a result of messing with crimson/debug flags and vars. Only an issue in main. From: David Galloway <David.Galloway@ibm.com> Date: Saturday, December 20, 2025 at 6:40 PM To: sepia@ceph.io <sepia@ceph.io>, dev@ceph.io <dev@ceph.io> Subject: Re: Update on Known Issues 1. I finally got the Windows CI image rebuilt. The “ceph windows tests” GitHub Check is fixed<https://jenkins.ceph.com/job/ceph-windows-pull-requests/70892/>! 1. Even better news, while it’s probably ugly and still needs testing on non-UEFI systems, teuthology jobs on Ubuntu 22.04 and CentOS 9 are running<https://pulpito.ceph.com/dgalloway-2025-12-20_21:11:18-smoke-tentacle-distro-default-trial/> on the new testnodes! You still need -m trial -t wip-dg-maas when scheduling. e.g., teuthology-suite -s smoke -m trial --suite-repo https://git.ceph.com/ceph.git --ceph-repo https://git.ceph.com/ceph.git -c tentacle --subset 1/100 --limit 10 --force-priority --newest 10 -t wip-dg-maas -v 1. CentOS 9 builds via ceph-dev-pipeline are fixed<https://jenkins.ceph.com/job/ceph-dev-pipeline/2304/pipeline-overview/> after merging https://github.com/ceph/ceph-build/pull/2497 I think this mostly gets development unblocked. Please let me know if anything needs attention. From: David Galloway <David.Galloway@ibm.com> Date: Thursday, December 18, 2025 at 11:07 PM To: sepia@ceph.io <sepia@ceph.io>, dev@ceph.io <dev@ceph.io> Subject: Update on Known Issues All, Here are a list of known issues we are still working through. Windows jobs - There was a Windows qcow2 image built in a Jenkins job in 2023 that was served from the LRC. I didn’t think anything was still being used from that service so did not bother to mirror it. I am working on rebuilding that Windows image to serve again so Windows PR jobs can run. Ceph API - ModuleNotFoundError: No module named ‘distutils’ - Fix merged https://github.com/ceph/teuthology/pull/2116. Rebase. CentOS 9 Builds - error: Failed build dependencies: - We think maybe something in this PR? https://github.com/ceph/ceph/pull/66229. Maybe builds will pass once https://github.com/ceph/ceph-build/pull/2497 is merged. Reef builds - Reef was not prepared for all Noble builders. There are a number of fixes that need to be back ported to support reef make check on the new builders. Working on it but lower priority. "Status code: 503 for http://apt-mirror.front.sepia.ceph.com/lab-extras/10/repodata/repomd.xml” - Pull in https://github.com/ceph/ceph/pull/66656 or one of its backports. Teuthology CentOS 9 jobs - We continue to struggle with the right way to handle booting into a newly installed kernel using BLS. I have a method working that requires two reboots so I’m trying to work around that. UEFI work is being done here https://github.com/ceph/teuthology/pull/2117 We still have a few fixes needed to work reliably with MaaS - https://github.com/ceph/teuthology/pull/2105 In more positive news: * make check on the new Jenkins builders is under 40 minutes * arm64 make check on the new builders is under an hour * Package and container builds are under an hour We are furiously working our way through these issues. -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway@ibm.com IBM