Teuthology Update teuthology runs are able to be scheduled again. To do so, either do not specify a machine type or add -m trial as that is the new testnode machine type. No -t (teuthology branch) parameter is required. The new testnodes have two 1.92TB NVMe devices intended to be OSDs. By default, each will get two 700GB logical volumes and one will get a third 100GB logical volume intended to be the /var/lib partition. You can override this by providing an override in your teuthology yaml. See "osd_selector_lvm:”in the testnodes ansible README: https://github.com/ceph/ceph-cm-ansible/tree/main/roles/testnode. Add’l hardware information: https://wiki.sepia.ceph.com/doku.php?id=hardware:trial. We have FOG images for Ubuntu 20 (Focal), 22 (Jammy), and 24 (Noble) as well as CentOS 9. Still working on Rocky 10. FOG is not a permanent solution. We still intend on migrating to MaaS but I didn’t want to hold up development any longer so we will continue figuring out the database lock contention in the background. If you hit Dead or Failed jobs that appear to be infrastructure-related, please file a tracker ticket: https://tracker.ceph.com/projects/lab/issues/new Red Hat Migrated Gear I filed tickets with IBM Labs last week to get the next batch of servers re-racked. This includes smithi, braggi, adami, and the ARM64 machines. I will prioritize bringing the Jenkins builders online first. We still need to meet with IBM Networking and possibly get some optics purchased in order to tie the migrated switches into the lab fabric. That network architecture call is scheduled for today. So at this time, I do not have an ETA for making any of the gear that migrated from Red Hat available. OVERALL As of now, there are no blockers I am aware of that should prevent ongoing Ceph development. Folks who have asked me are sharing the new “sockeni” development machines. PR checks are running, and teuthology jobs can finally be ran. If you are blocked by anything, please let me know. From: David Galloway <David.Galloway@ibm.com> Date: Thursday, January 15, 2026 at 10:53 AM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Re: Sepia Lab Update Teuthology Update I’ve made progress with FOG. I am just working on modifying the MaaS-installed OS image so that I can capture it and subsequently deploy it to other hosts with FOG. We used to deploy a machine with cobbler, then prep the image, put a bunch of sketchy scripts in /etc/rc.local, then capture that image. MaaS does a bunch of cloud-init stuff we need to undo and change. Progress! From: David Galloway <David.Galloway@ibm.com> Date: Wednesday, January 14, 2026 at 8:36 PM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Re: Sepia Lab Update Teuthology Update The MaaS provisioner PR and UEFI / BLS PR have been merged. The MaaS database and/or one of its services continues to get overwhelmed and wreak all sorts of havoc when a suite is scheduled. Dan is going to reach out to MaaS devs for assistance. In the meantime, I continue to struggle with PXE booting the new trial machines with FOG. From: David Galloway <David.Galloway@ibm.com> Date: Tuesday, January 13, 2026 at 11:08 AM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Re: Sepia Lab Update Teuthology Update The MaaS integration PR continues to be reviewed. There are about 3 separate fixes that are also required to get all of this working correctly so testing all of those at the same time has been troublesome. I am continuing to work on deploying FOG as a backup but have hit an issue with PXE booting that I’m working with FOG Developers on. https://forums.fogproject.org/topic/18087/the-ddp-package-file-was-not-found... From: David Galloway <David.Galloway@ibm.com> Date: Thursday, January 8, 2026 at 3:42 PM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Re: Sepia Lab Update It has come to my attention there are outstanding questions about what should be working, what’s not, timelines, etc. I’ll do my best to address the questions and send continued and more frequent updates. To Recap The “Sepia” lab was shut down on December 15 and most of the hardware was relocated to an IBM datacenter in Poughkeepsie, NY. It arrived on December 19 and is in the process of being re-racked. Power is in place under the floors already. Rack elevations are figured out. The lab admins are just in the process of re-racking and re-cabling which is going to take some time. The folio are re-racked and cabled. robsoni, ivan, braggi, and adami are in progress. smithi and vossi are next. Based on my experience with IBM getting the Octo lab’s Red Hat switches online in Tucson, I estimate another month. What moved / didn’t move? Moved: adami, braggi, confusa, folio, gibba, ivan, mako, officinalis, reesi, robsoni, smithi, sulcata, vossi, ssdstore, gitbuilder, hv, cnv NOT MOVED: incerta, irvingi, mira, omani, refarch, senta Is Wireguard the new permanent VPN solution? We aren’t sure yet. We have observed if a VPN client rsyncs data over the tunnel, all VPN clients (including the rsync..er) start to experience >=50% packet loss. We need debugging time to narrow down the bottleneck and engage IBM Networking and/or the ISP and/or IBM Security and/or Palo Alto Networks. In the meantime, Adam on my team is setting up a proof of concept for https://goteleport.com/. What I can say is it’s very unlikely we will reuse the exact same OpenVPN configuration we used for the RDU lab. My PR Checks / Jenkins builds are taking forever Yes, we went from ~30 bare metal x86 builders to 8. Initially, in December when the RDU lab first shut down, it appeared those 8 were enough to keep up with the workload but January has been much busier than anticipated. The Ceph Windows PR check failures added to the load because they kept failing and created a long backlog. They also take the most amount of time to run by far. Roughly 2 hours per PR, whereas actual package and container builds are under an hour. “make check” is under an hour. Adam K is also spinning up some additional baremetal builders today to help with the load. Can I push to ceph-ci? Yes, and you’ll even get a build. Eventually. Great, can I test it using teuthology? Not yet. As I hope you’ve already heard, we decided to switch to using Canonical’s Metal-As-A-Service (MaaS) for deploying OSes. Unfortunately, we were not given enough time to deploy it, along with all the requisite services (i.e., pulpito, paddles, teuthology, LRC), and get it working with the new testnodes before the Sepia lab shut down. So we’re figuring out the quirks now. The main issue I have been fighting for weeks is MaaS can not handle the burstiness of teuthology job scheduling. Essentially, one server deployment holds a lock on a postgres database, and enough database transactions pile up waiting for that lock to the point that subsequent API calls can eventually time out. So, if we schedule 30 deployments at the same time, and another machine is already PXE booting and cloud-init is asking MaaS for its kickstart/preseed, it occasionally times out, the deployment fails, and the teuthology job dies. I am deciding we have spent enough time spinning wheels and am in the process of deploying FOG as an interim measure to get us unblocked. Either additional teuthology development will be required to throttle deployments (which I don’t think it a reasonable ask), or we will need assistance from Canonical with no guaranteed timeline for resolution. Are teuthology job logs available to be analyzed? Logs for jobs ran prior to December 15 are still on the old LRC which needs to be re-racked, re-cabled, and switches reconfigured still. Logs for new teuthology runs will be on the new LRC. If you want the new LRC mounted to your dev box, let me know. It requires installing ceph-common and I don’t want to interfere with existing packaging/repos/etc. without your prior approval. I need a Dev Box “sockeni” are the new dev boxes. Look at https://wiki.sepia.ceph.com/doku.php?id=hardware:sockeni#users. Based on which team members you most closely align with, let me know, and I’ll add your user and SSH key. Sharing is caring for now. We’ll look at sharing, spreading things out, etc. once the Red Hat gear is brought back online. Additional Perspective There are still over 200 servers in Tucson that are unusable due to a combination of PTO, switch port misconfigs, last minute subnet/architecture changes, MaaS growing pains, repeated power outages, etc. so my team is still spread really thin. Let me know if you have any questions. From: David Galloway <David.Galloway@ibm.com> Date: Wednesday, January 7, 2026 at 8:52 AM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Re: Sepia Lab Update Quick update: Ceph Windows checks on GitHub PRs This was broken, then fixed yesterday, then broke again overnight. Some background - I forgot the Windows jobs relied on a qcow2 image that we build and had hosted via filedump.ceph.com from the LRC. I built this image, deployed a new “apt-mirror” service in the Openshift cluster and restored the lab-extras repo and this Windows qcow2 image to it. The service originally used an nginx container image from Docker Hub. Our Openshift cluster enforces image signature/trust policy for external registries, and the pull was rejected, so the nginx pod failed to start and the service went down. That caused lab-extras to be unavailable resulting in "Failed to download metadata for repo ''lab-extras'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried” errors. The Windows PR checks also failed with "ERROR: The WINDOWS_VM_IP env variable is not set” This has been resolved by importing and mirroring the nginx image into the internal Openshift registry and pointing the apt-mirror service to use it. I also modified the Deployment to 3 replicas and added a “Pod Disruption Budget” to prevent all replicas from going down at the same time. Teuthology Dead jobs After getting conserver running, I was able to see cloud-init during MaaS provisioning is occasionally timing out trying to pull the testnode’s preseed (kickstart). This happens under load. Our current theory is the database is getting hammered and MaaS can’t handle all the concurrent requests so we will be tuning that today. Hopefully the last few lingering job Dead issues will be cleared up then. Thanks for your patience. These last few hurdles are difficult to find until things are under load but we’re nearly there. From: David Galloway <David.Galloway@ibm.com> Date: Monday, January 5, 2026 at 4:27 PM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Re: Sepia Lab Update I stand correct on teuthology being ready… There are still some bugs that need to be worked out in the maas provisioner. Hold off for now. From: David Galloway <David.Galloway@ibm.com> Date: Monday, January 5, 2026 at 2:33 PM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Sepia Lab Update Just wanted to provide a quick update on the lab status. * Incoming connections to the lab got resolved as I was typing this e-mail. This means pulpito is back, qa-proxy is back, new VPN connections can be made. Apparently traffic got rerouted to our backup/redundant link and the firewall was not active for that route. BGP was moved back to the primary link. * There are some development machines available for use. They are all installed and assigned at this point so additional users will have to share. I’m trying to keep teammates on the same boxes since you/they are more likely to keep in touch and not step on each other’s toes. If you need a resource, see https://wiki.sepia.ceph.com/doku.php?id=hardware:sockeni#users and let me know which machine you need. * I got conserver running last week. All of the new testnode (“trial”) machines needed their BMC firmware updated to get serial-over-LAN working. Conserver running means we should have worker_logs being populated during teuthology jobs. However, the dispatcher doesn’t seem to have picked up that change. We’re looking into it. * Speaking of teuthology jobs, there are still a few of the new testnodes giving us trouble. Getting console logs working was a precursor to working out the remaining kinks with deploying the actual hosts. * The ‘wip-dg-maas’ branch still has some changes that need to make it back into the official MaaS PR<https://github.com/ceph/teuthology/pull/2105> * The Red Hat migrated gear arrived in Poughkeepsie and was unloaded on 12/19. The folio, robsoni, LRC, and Jenkins builders are highest priority to re-rack and bring online at the moment. smithis will be next. Overall, you should be able to schedule teuthology runs using the wip-dg-maas teuthology branch from soko04.front.sepia.ceph.com. You may just need to tolerate some Dead jobs for now as we work through the remaining infra issues. Apologies for missing the CSC call this morning. Was not feeling well enough to stare at a screen. Let me know if there are any questions. -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway@ibm.com IBM