Paddles availability has been flapping over the past few hours due to a botnet attack. I’m working on setting up some form of mitigation. -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway@ibm.com IBM
I’ve deployed Anubis on the reverse proxy host that serves pulpito/pulpit-ng to block the botnet attack. RCA: A scraper botnet (~95K unique IPs, spoofed browser user agents) ramped pulpito-ng.ceph.com from ~1K to 40K requests/hour starting around 6am Eastern, including ~99K hits on job history pages. Every one of those pages makes pulpito-ng call paddles' /jobs/?description= endpoint, whose LIKE '%...%' query scans the entire 8.4M-row jobs table at 3-5s each. That kept all 16 paddles gunicorn workers permanently busy, so nothing got answered, including the liveness probe, and Openshift kill-looped the pods every ~7 minutes. From: David Galloway <David.Galloway@ibm.com> Date: Wednesday, August 5, 2026 at 9:42 AM To: sepia@ceph.io <sepia@ceph.io> Subject: Paddles/Pulpito Outage Paddles availability has been flapping over the past few hours due to a botnet attack. I’m working on setting up some form of mitigation. -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway@ibm.com IBM
I've removed the "job history" links for the time being. I'm a little suspicious that this could be someone pointing AI agents at the tool to do some sort of analysis. But Anubis seems like a good move regardless.
participants (2)
-
David Galloway
-
Zack Cerza