1. I finally got the Windows CI image rebuilt.  The “ceph windows tests” GitHub Check is fixed!

  1. Even better news, while it’s probably ugly and still needs testing on non-UEFI systems, teuthology jobs on Ubuntu 22.04 and CentOS 9 are running on the new testnodes!  You still need -m trial -t wip-dg-maas​ when scheduling.  e.g.,

      teuthology-suite -s smoke -m trial --suite-repo https://git.ceph.com/ceph.git --ceph-repo https://git.ceph.com/ceph.git -c tentacle --subset 1/100 --limit 10 --force-priority --newest 10 -t wip-dg-maas -v

  1. CentOS 9 builds via ceph-dev-pipeline are fixed after merging https://github.com/ceph/ceph-build/pull/2497

I think this mostly gets development unblocked.  Please let me know if anything needs attention.


From: David Galloway <David.Galloway@ibm.com>
Date: Thursday, December 18, 2025 at 11:07 PM
To: sepia@ceph.io <sepia@ceph.io>, dev@ceph.io <dev@ceph.io>
Subject: Update on Known Issues

All,

Here are a list of known issues we are still working through.

Windows jobs - There was a Windows qcow2 image built in a Jenkins job in 2023 that was served from the LRC.  I didn’t think anything was still being used from that service so did not bother to mirror it.  I am working on rebuilding that Windows image to serve again so Windows PR jobs can run.
Ceph API - ModuleNotFoundError: No module named ‘distutils’ - Fix merged https://github.com/ceph/teuthology/pull/2116.  Rebase.
CentOS 9 Builds - error: Failed build dependencies: - We think maybe something in this PR? https://github.com/ceph/ceph/pull/66229.  Maybe builds will pass once https://github.com/ceph/ceph-build/pull/2497 is merged.
Reef builds - Reef was not prepared for all Noble builders.  There are a number of fixes that need to be back ported to support reef make check​ on the new builders.  Working on it but lower priority.
"Status code: 503 for http://apt-mirror.front.sepia.ceph.com/lab-extras/10/repodata/repomd.xml” - Pull in https://github.com/ceph/ceph/pull/66656 or one of its backports.

Teuthology CentOS 9 jobs - We continue to struggle with the right way to handle booting into a newly installed kernel using BLS.  I have a method working that requires two reboots so I’m trying to work around that.  UEFI work is being done here https://github.com/ceph/teuthology/pull/2117

We still have a few fixes needed to work reliably with MaaS - https://github.com/ceph/teuthology/pull/2105

In more positive news:

We are furiously working our way through these issues.

-- 

David Galloway

Ceph Engineering Labs – Infrastructure Architect

david.galloway@ibm.com

IBM