AI Bots Are DDoSing Bug Trackers: How to Protect Your Public Dev Infrastructure

The Gentoo Bugzilla Incident: What Actually Happened

On January 5, 2025, Gentoo's Bugzilla instance went dark. Not due to a zero-day exploit or a malicious DDoS attack, but because an AI scraper bot hammered the server with requests until it fell over. Michał Górny, a Gentoo developer, documented the incident with the kind of exhausted precision that only a systems engineer who's been paged on a Sunday can muster.

The culprit wasn't a sophisticated adversary. It was an AI training data scraper—likely someone building a dataset of software bugs for model fine-tuning—configured with zero rate limiting, zero respect for robots.txt, and zero understanding that Bugzilla is not a static website. Every request hit the full application stack: database queries, template rendering, authentication checks. The bot wasn't just downloading pages; it was forcing the server to do real work for every single request.

The server didn't stand a chance. Gentoo's infra team had to take Bugzilla offline entirely to stop the bleeding. This is the new normal: public developer infrastructure is being treated as free training data by an arms race of AI companies and hobbyist scrapers who either don't know or don't care about the operational cost they impose.

Why This Matters for Forward Deployed Engineers

If you're an FDE—or any engineer shipping software that faces the public internet—this isn't a Gentoo problem. It's your problem. Forward Deployed Engineers sit at the exact intersection where this hurts most: you're deploying into customer environments, often behind firewalls, but increasingly exposing APIs, dashboards, and tools to the wider internet. Your customers' bug trackers, internal wikis, and CI/CD dashboards are all potential targets.

Three things make this especially relevant for FDEs:

1. You're the first responder. When a customer's self-hosted instance gets scraped into oblivion, you're the one who gets the 2 AM Slack message. You need to know what to reach for before it happens.

2. You deploy in constrained environments. Unlike a SaaS company with a dedicated SRE team and Cloudflare Enterprise, you're often working inside a customer's VPC with whatever tooling they've approved. Rate limiting might need to live in nginx configs, not a WAF dashboard.

3. The data is sensitive. Bug trackers contain vulnerability discussions, internal architecture notes, and sometimes credentials accidentally pasted into comments. Scraping isn't just a load problem—it's a data exfiltration vector. For more on operating inside customer security perimeters, see our deep dive on the Palantir-style FDE embed.

The Attack Surface: What Makes Dev Tools Vulnerable

Bug trackers, wikis, and CI dashboards share a dangerous trait: they're read-heavy applications with expensive page generation. Every bug report page on Bugzilla might execute 20+ database queries, render templates, check permissions, and format attachments. A single scraper hitting at 100 requests per second can easily overwhelm a modest server.

Compare this to a static blog. A scraper hitting a static HTML page is basically a file server workload—cheap. But Bugzilla, Jira, GitLab, Phabricator, and similar tools are dynamic applications. Each request is a mini-transaction. The economics are completely different.

Here's the attack surface breakdown:

Search endpoints deserve special attention. A scraper that discovers your search form can generate arbitrary query complexity. ?q=a, ?q=b, ?q=c—each one triggers a full-text search across your entire bug database. This is how a single bot can generate more load than your entire legitimate user base.

Defense Architecture: Rate Limiting and Request Filtering

The defense strategy has to work at multiple layers because scrapers target different parts of the stack. Here's what a layered defense looks like:

Layer 1: IP-based rate limiting. This is table stakes. Use fail2ban, nginx's limit_req_zone, or a reverse proxy to cap requests per IP. The key insight: scraper bots often come from a single IP or a small range. A limit of 30 requests per minute per IP will stop most naive scrapers cold while legitimate users won't notice.

Layer 2: User-Agent filtering. Most scrapers identify themselves honestly—or use a default library UA string like python-requests/2.31.0. Block known scraper UAs at the edge. Maintain a deny list. But don't rely on this alone; sophisticated scrapers spoof Chrome UAs.

Layer 3: Path-based throttling. Apply stricter limits to expensive endpoints. Your search page might get a 5 req/min/IP limit while static assets get 100 req/min. This is where understanding your application's cost-per-endpoint pays off.

Layer 4: Cache shield. If a page can be served from cache, it should be. More on this below.

Implementing a Practical Shield with Open Tools

Let's get concrete. Here's an nginx configuration that implements the first three layers without any external services. This is the kind of config you can drop into a customer's environment with zero dependencies beyond what's already running.

# Define a shared memory zone for rate limiting
limit_req_zone $binary_remote_addr zone=general:10m rate=30r/m;
limit_req_zone $binary_remote_addr zone=expensive:10m rate=5r/m;

# Map for blocking known scraper UAs
map $http_user_agent $block_ua {
    default 0;
    "~*python-requests" 1;
    "~*scrapy" 1;
    "~*curl/7" 1;
    "~*Go-http-client" 1;
    "~*node-fetch" 1;
    "~*axios" 1;
    "~*okhttp" 1;
    "~*Java/1" 1;
}

server {
    # Block known scraper UAs immediately
    if ($block_ua) {
        return 403;
    }

    # General rate limit for most endpoints
    location / {
        limit_req zone=general burst=10 nodelay;
        limit_req_status 429;
        proxy_pass http://app_server;
    }

    # Stricter limit for search endpoints
    location /search {
        limit_req zone=expensive burst=2 nodelay;
        limit_req_status 429;
        proxy_pass http://app_server;
    }

    location /bug {
        limit_req zone=expensive burst=3 nodelay;
        limit_req_status 429;
        proxy_pass http://app_server;
    }
}

This configuration uses $binary_remote_addr (binary IP) for efficient memory usage in the shared zone. The burst parameter allows short spikes above the rate—important for legitimate users who might open multiple tabs. nodelay means excess requests get a 429 immediately rather than being queued.

The UA block list targets common HTTP client libraries. These are the libraries scrapers use because they're the defaults. A serious scraper will spoof Chrome's UA, but many won't bother—and blocking the low-effort ones reduces load significantly.

Caching Strategies That Deflect Scraper Traffic

Rate limiting stops the flood, but caching makes your application resilient even under heavy legitimate load. The goal is to serve as many requests as possible without touching the application server.

Reverse proxy caching with nginx. For mostly-read applications like bug trackers, you can cache responses aggressively with short TTLs:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=bugzilla:100m 
                 max_size=10g inactive=60m use_temp_path=off;

location / {
    proxy_cache bugzilla;
    proxy_cache_key "$scheme$request_method$host$request_uri";
    proxy_cache_valid 200 302 5m;
    proxy_cache_valid 404 1m;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503;
    proxy_cache_bypass $http_cache_control;
    add_header X-Cache-Status $upstream_cache_status;
    proxy_pass http://app_server;
}

The proxy_cache_use_stale directive is the unsung hero here. If your app server goes down, nginx will serve stale cached content instead of errors. For a bug tracker, a 5-minute-old page is infinitely better than a 502. The X-Cache-Status header lets you debug cache behavior easily.

Application-level caching. If you control the application code (or can add middleware), cache expensive query results. Bug listing pages that show "all open bugs in component X" don't need to be regenerated for every request. A 60-second Redis cache on these queries can reduce database load by 90% or more.

Edge caching for static assets. This is basic but often overlooked: CSS, JS, and image files should have far-future Cache-Control headers and be served from a CDN or at minimum a separate location block with aggressive caching.

The Ethical Scraper: Robots.txt Is Not Enough

There's an uncomfortable truth here: robots.txt is a voluntary standard. It's like a "Please Don't Rob Me" sign on your front door. The well-behaved bots (Google, Bing, Internet Archive) respect it. The AI scrapers that crashed Gentoo's Bugzilla? They ignore it entirely.

This creates a collective action problem. If your service is public, you're subsidizing someone's training data pipeline. The cost is real: compute cycles, bandwidth, and the engineering time to recover from overload incidents. But the scrapers capture all the value.

What can you do beyond rate limiting?

• Require authentication for expensive endpoints. If your bug tracker's search requires a login, scrapers can't hit it anonymously. This is heavy-handed but effective.
• Implement proof-of-work challenges. Before serving a response, require the client to solve a small computational puzzle. This is what Cloudflare's "I Am Under Attack" mode does. It's hostile to legitimate users but stops bots cold.
• Serve tarpits. When you detect a scraper, instead of blocking them, serve an infinite stream of garbage data at 1 byte per second. They'll tie up their own resources and eventually give up. This is satisfying but may violate your own terms of service with your hosting provider.

For FDEs building internal tools that get exposed to customers, the right answer is usually authentication plus rate limiting. If you're building something like a RAG chatbot over internal documentation, you already have authentication in place—make sure your rate limiting is equally robust.

Balanced Take: Scraping vs. Service Integrity

Let's be fair: the people scraping bug trackers aren't necessarily malicious. They're probably engineers like us, trying to build better AI models for code generation or vulnerability detection. There's genuine value in having models trained on real bug reports. The problem is the tragedy of the commons: individual scrapers don't bear the cost of their requests, so they have no incentive to be gentle.

The solution isn't to lock everything behind authentication. Public bug trackers are public for good reasons: transparency, community contribution, and searchability. The solution is to make them resilient by default.

Here's what a resilient posture looks like:

1. Assume you will be scraped. Design your infrastructure accordingly. Cache everything you can. Rate-limit everything you can't.
2. Monitor for anomalies. A sudden spike in traffic from a single IP or to a single endpoint pattern is a scraper. Alert on it.
3. Have a kill switch. If a scraper is taking you down, you need the ability to block it in under 60 seconds. This means your rate limiting config should be hot-reloadable, not requiring a full deploy.
4. Design for degraded service. If your app server is overloaded, serve stale cached content. If that's not possible, serve a static "under heavy load" page. A partially working bug tracker is better than a completely offline one.

This pattern of building resilient, self-hosted tools applies far beyond bug trackers. When you're deploying an LLM feature behind a Fortune 500 firewall, the same principles of caching, rate limiting, and graceful degradation keep your service alive when things go wrong—whether it's a scraper, a misconfigured internal client, or just unexpected popularity.

FAQ

Q: Can't I just use Cloudflare to solve this?

Cloudflare's DDoS protection and bot management are excellent, but they're not always available. In customer environments, you may not be allowed to route traffic through a third party. And Cloudflare's free tier won't catch everything. The nginx-based approaches above work anywhere.

Q: Won't rate limiting block legitimate users behind a NAT?

Yes, shared IPs (corporate offices, universities) can hit rate limits more quickly. Mitigate this by setting generous burst values, using session cookies as a secondary rate limit key, or whitelisting known good IP ranges. For most applications, the occasional false positive 429 is preferable to the server crashing.

Q: How do I know if I'm being scraped right now?

Check your access logs for patterns: single IPs requesting every page in sequence, requests concentrated on search endpoints, UA strings that are HTTP client libraries, or traffic that ignores robots.txt. Tools like goaccess or ELK stack make this analysis straightforward.

Q: Is this really an AI problem specifically?

AI training data collection has dramatically increased the volume and sophistication of scraping. Traditional search engine crawlers are relatively polite and respect rate limits. AI scrapers are often run by individuals or small teams who either don't know how to be polite or are actively trying to collect data as fast as possible. The scale is different, and the disregard for service health is more common.

Q: What's the one thing I should implement today?

IP-based rate limiting in your reverse proxy. It's 10 lines of nginx config, it catches 80% of naive scrapers, and it protects against many other problems (brute force attacks, runaway scripts). Start there, then add caching.