Protect Django From Bots and Scrapers With a WAF

Problem statement

Bots make up a large share of traffic to most public sites: scrapers copying content and prices, credential-stuffing tools trying leaked passwords on the login page, spam bots filling signup and contact forms, and vulnerability scanners probing /wp-admin and /.env. They waste server capacity, distort analytics, and create security risk. Defences inside Django (rate limits, lockouts, CAPTCHAs) help, but stopping abuse before it reaches your servers is cheaper. A web application firewall (WAF) at the CDN or load balancer filters traffic at the edge.

1. Put the site behind a WAF: Cloudflare (WAF, Bot Fight Mode, rate limiting), AWS WAF on CloudFront or an ALB, or similar.
2. Enable managed rule sets (OWASP core rules, known bad bots, IP reputation) in log mode first, then block.
3. Add rate-based rules for sensitive paths: login, signup, password reset, search, and APIs.
4. Challenge rather than block on ambiguous traffic (managed challenge, Turnstile) for human-facing pages.
5. Block noise paths Django never serves (/wp-login.php, /.env, /phpmyadmin) at the edge.
6. Keep Django-side protection too: django-axes for logins, throttling in DRF, CSRF, and form spam protection.

Step-by-step solution

1. Start with managed rules in log mode

Enable the provider's managed rule groups: Cloudflare Managed Ruleset and OWASP Core Ruleset, or AWS Managed Rules (Core rule set, Known bad inputs, Amazon IP reputation list, Bot Control). Run them in log or count mode for a few days and review what would be blocked. Admin forms and rich-text editors sometimes trigger SQL-injection or XSS signatures; add narrowly scoped exceptions (specific path plus authenticated session) rather than disabling rules globally.

2. Rate-limit sensitive endpoints

A Cloudflare rate limiting rule:

(http.request.uri.path eq "/accounts/login/" and http.request.method eq "POST")
→ 10 requests per 1 minute per IP → managed challenge for 10 minutes

An AWS WAF rate-based rule can scope down to the same path and method with a limit per 5 minutes per IP. Apply similar rules to signup, password reset, contact forms, search, and expensive API endpoints.

3. Challenge bots on human pages

Challenges (Cloudflare Managed Challenge, Turnstile, AWS WAF CAPTCHA or challenge actions) let real browsers through after a quick check while stopping simple bots. Use them on login, signup, and checkout. Do not challenge API endpoints or webhooks: machine clients cannot solve them; authenticate and rate-limit those instead, and exclude webhook paths (Stripe, GitHub) from bot rules.

4. Block obvious noise

Scanners request paths that do not exist in Django. Blocking them at the edge saves Django from rendering thousands of 404s:

(http.request.uri.path contains "/wp-" or http.request.uri.path contains ".php" or http.request.uri.path eq "/.env") → block

5. Keep Django's own defences

The WAF is a layer, not a replacement. In Django:

• django-axes locks out repeated failed logins per user and IP.
• DRF throttling (AnonRateThrottle, UserRateThrottle) limits API use per client.
• Turnstile or a honeypot field on signup and contact forms.
• Restore real client IPs from the CDN headers so Django-side limits see the right address.

Verification check: WAF analytics show blocked and challenged requests on login and probe paths, a scripted burst of login POSTs from one IP triggers the challenge, real users can still log in, and Django logs show fewer 404s from scanners.

Explanation

A WAF inspects HTTP requests at the edge against rule sets: signatures for known attack payloads, IP and ASN reputation, bot detection based on fingerprints and behaviour, and rate counters per client. Because it runs on the CDN or load balancer, abusive traffic is dropped or challenged before consuming Django workers, database connections, or bandwidth. Django-level controls remain essential for logic the edge cannot see, such as per-account lockouts and per-user API quotas.

Edge cases / notes

• Shared IPs (corporate NAT, mobile carriers) can trip IP-based limits; prefer challenges to hard blocks for human pages.
• Search engine crawlers should be allowed; managed bot rules usually verify good bots.
• Webhooks and health checks need explicit exceptions.
• Cost. AWS WAF bills per rule and request; Bot Control adds per-request fees.

Next steps

• Add server-side limits with Rate Limit Django With Nginx.
• Lock out password guessing with Lock Out Brute-Force Logins With django-axes.

FAQ

How do I stop bots on my Django site?

Use an edge WAF with managed rules, rate limits, and challenges on sensitive pages, plus Django-side lockouts and throttles.

Will a WAF block Google?

Managed bot rules allow verified search engine crawlers. Check settings before enabling strict bot modes.

Should APIs be behind bot challenges?

No. Machine clients cannot solve challenges. Use authentication and rate limits returning 429.

Do I still need django-axes behind a WAF?

Yes. Edge limits stop volume; axes stops slow, distributed guessing against specific accounts.