I’ve deployed Anubis on the reverse proxy host that serves pulpito/pulpit-ng to block the botnet attack.
RCA:
A scraper botnet (~95K unique IPs, spoofed browser user agents) ramped pulpito-ng.ceph.com from ~1K to 40K requests/hour starting around 6am Eastern, including ~99K hits on job history pages. Every one of those pages makes
pulpito-ng call paddles' /jobs/?description= endpoint,
whose LIKE '%...%' query scans the entire
8.4M-row jobs table at 3-5s each. That kept all 16 paddles gunicorn workers permanently busy, so nothing got answered, including the liveness probe, and
Openshift kill-looped the pods every ~7 minutes.
From: David Galloway <David.Galloway@ibm.com>
Date: Wednesday, August 5, 2026 at 9:42 AM
To: sepia@ceph.io <sepia@ceph.io>
Subject: Paddles/Pulpito Outage
Paddles availability has been flapping over the past few hours due to a botnet attack. I’m working on setting up some form of mitigation.
--
David Galloway
Ceph Engineering Labs – Infrastructure Architect
david.galloway@ibm.com
IBM