The plumbing in Poughskeepsie was fixed and the Sepia lab has been brought back up. Jenkins builders will churn through the queue. Teuthology jobs are running. A known issue from an unrelated maintenance, if you see "502 Server Error: Bad Gateway for url: https://1.chacra.ceph.com<https://1.chacra.ceph.com/repos/ceph/reef/e6ab650721089ed287ed684011c35b550bf20fe7/centos/9/flavors/default/repo>”, it’s fixed. Same issue in the container stage of ceph-dev-pipeline: + curl -fs -L https://shaman.ceph.com/api/repos/ceph-iscsi/main/latest/centos/9/repo -o /etc/yum.repos.d/ceph-iscsi.repo Error: building at STEP "RUN set -ex && curl -s -L https://shaman.ceph.com/api/repos/tcmu-runner/main/latest/centos/9/repo?arch...) -o /etc/yum.repos.d/tcmu-runner.repo && case "${CEPH_REF}" in quincy|reef) curl -fs -L https://download.ceph.com/ceph-iscsi/3/rpm/el9/ceph-iscsi.repo -o /etc/yum.repos.d/ceph-iscsi.repo ; ;; main|*) curl -fs -L https://shaman.ceph.com/api/repos/ceph-iscsi/main/latest/centos/9/repo -o /etc/yum.repos.d/ceph-iscsi.repo ; ;; esac": while running runtime: exit status 22 In a teuthology job, same issue: 2026-02-16T23:36:46.794 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on trial013 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' trial029 unfortunately resulted in a few Dead jobs. Let me know if you see any other issues but please be patient. From: David Galloway <David.Galloway@ibm.com> Date: Friday, February 13, 2026 at 7:15 PM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Re: Emergency Lab Maintenance 2/12 - DELAYED 24h The teuthology dispatcher service has been instructed to stop. Running jobs will finish but no queued jobs will start again until late Monday or possibly Tuesday. From: David Galloway <David.Galloway@ibm.com> Date: Friday, February 13, 2026 at 4:11 PM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Re: Emergency Lab Maintenance 2/12 - DELAYED 24h I’m pushing this back a couple hours. Teuthology jobs will stop at 7PM Eastern. Jenkins will stop processing new jobs at 8PM. Hopefully this gives the queue time to clean out a bit as things calm down for the weekend. From: David Galloway <David.Galloway@ibm.com> Date: Thursday, February 12, 2026 at 11:17 AM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Re: Emergency Lab Maintenance 2/12 - DELAYED 24h Was just made aware the earliest the vendor can perform the repair is Monday so the lab shutdown is postponed to Friday afternoon. From: David Galloway <David.Galloway@ibm.com> Date: Wednesday, February 11, 2026 at 6:04 PM To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io> Subject: Emergency Lab Maintenance 2/12 All, This afternoon, we were informed of a water leak in one of the cooling pipes serving the Poughkeepsie lab. The vendor will need to drain and repair (or potentially replace) the affected pipe. During this work, cooling in the lab will be unavailable, which requires us to power down all systems in the environment. The earliest available maintenance window from the vendor is Friday. As a result, we will begin shutting down Sepia on Thursday afternoon. We will start this process tomorrow, 2/12, at 4:00 PM Eastern. At that time, I will stop the teuthology dispatcher so that currently running jobs can complete, but no new jobs will be picked up. Jenkins will be placed in offline mode so active builds can finish while new jobs remain queued. A few hours later, we will proceed with shutting down all services and servers, including the sockeni dev boxes. The vendor has estimated approximately six hours to complete the maintenance, and IBM anticipates an additional hour to confirm that datacenter cooling systems are fully restored and stable before we begin powering systems back on. Please feel free to reach out with any questions or concerns. -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway@ibm.com IBM