Earlier today, Yuri made me aware of a failed ceph-dev-pipeline Jenkins job.  Upon investigating, I discovered all three chacra nodes were unhealthy so the jobs had nowhere to push packages and failed.

The root cause was the chacra instances were recently redeployed.  chacra has a purge_repos config option that defines how long to keep repos for.  chacra periodically cleans up old repos of dev builds after 14 days.  The instances were inadvertently redeployed without this config option so the data drives began filling up.  Once each node reached 85%, they told shaman they were unhealthy.

I redeployed the instances with purge_repos=true, cleaned up the data drives, and retriggered any failed builds.

Total outage time was between 2026-08-12 04:11:19 and 2026-08-13 21:14:12 UTC.

To prevent this in the future, I’ve added alerts to alertmanager to alert us of both a chacra node reporting unhealthy as well as a 75% disk utilization alert for the chacra nodes.