Around 15:24 UTC today, I noticed calls to shaman.ceph.com were getting refused. Upon investigation, I observed the nginx service on shaman.ceph.com (the load balancer in front of 1.shaman and 2.shaman) was in failed state. Oct 01 06:37:37 shaman systemd[1]: Stopping nginx.service - A high performance web server and a reverse proxy server... Oct 01 06:37:37 shaman nginx[2274090]: 2025/10/01 06:37:37 [emerg] 2274090#2274090: host not found in upstream "1.shaman.ceph.com" in /etc/nginx/sites-enabled/01-shaman.conf:35 Oct 01 06:37:37 shaman nginx[2274090]: nginx: configuration file /etc/nginx/nginx.conf test failed I manually restarted the service at 15:16:48 UTC. Some immediate actions we’re taking to improve this story: * Improved alerting. We used to have an nginx service that would have caught this and sent alerts but gluster ate the VM’s disk. We have a Grafana instance but its alerts are too noisy. I’ve asked Adam to create a way to prioritize alerts for outages like this. * Put entries in /etc/hosts for the downstream shaman hosts Next year, I aim to launch another status portal like we had at status.sepia.ceph.com. Any branches pushed between ~06:00 and 15:16 UTC should be force pushed to retrigger a build. -- David Galloway Ceph Engineering Labs – Infrastructure Architect +1 989 295 0091 - Mobile david.galloway@ibm.com IBM