This afternoon, we were informed of a water leak in one of the cooling pipes serving the Poughkeepsie lab. The vendor will need to drain and repair (or potentially replace) the affected pipe. During this work, cooling in the lab will be unavailable, which requires us to power down all systems in the environment.

The earliest available maintenance window from the vendor is Friday. As a result, we will begin shutting down Sepia on Thursday afternoon.

We will start this process tomorrow, 2/12, at 4:00 PM Eastern. At that time, I will stop the teuthology dispatcher so that currently running jobs can complete, but no new jobs will be picked up. Jenkins will be placed in offline mode so active builds can finish while new jobs remain queued. A few hours later, we will proceed with shutting down all services and servers, including the sockeni dev boxes.

The vendor has estimated approximately six hours to complete the maintenance, and IBM anticipates an additional hour to confirm that datacenter cooling systems are fully restored and stable before we begin powering systems back on.

Please feel free to reach out with any questions or concerns.