All articles

BLOG / BOT TRAFFIC

Website Bot Traffic: Measure Abuse and Preserve Useful Bots

Separate useful crawlers from abusive automation, measure action outcomes beyond request counts, and roll out controls without disrupting legitimate visitors.

A document lane and repeated action-request lanes remain visibly separate.A document lane and repeated action-request lanes remain visibly separate.

Website bot traffic is traffic generated by software rather than a person directly navigating each request. It includes useful search crawlers, authorized monitoring, integrations, and abusive automation. The operational goal is to preserve useful access while reducing harmful actions, not to make an estimated “bot percentage” as small as possible.

Start with server-side evidence for the action that costs you something. A spike in page requests, a surge of failed logins, and repeated trial-credit claims are different problems. They need different measurements and may need different controls.

Separate useful automation from abuse

Traffic class Example Useful response
Useful public crawling A search engine fetches product documentation Verify the crawler where necessary and preserve intended access.
Authorized automation A customer's integration calls a documented API Use integration credentials, authorization, and appropriate quotas.
Abusive automation Repeated account creation claims a limited trial Protect the trial-granting action and review confirmed outcomes.
Uncertain activity An unfamiliar client generates unusual request volume Investigate context before assigning intent.

A user-agent string is a claim, not proof of identity. Google documents how to verify its crawlers using published IP ranges or DNS verification. Use the relevant provider's procedure rather than trusting a familiar name in a header.

Keep server-to-server integrations separate from browser-only paths. Requiring browser evidence on a documented machine API can break customers without addressing abuse on your signup form. Classify the route before choosing the verification method.

Measure requests, sessions, and business effects separately

Your analytics tool and access logs can report different totals because they observe different events. A page analytics script only counts what reaches and runs that script. Server logs may also include assets, failed requests, health checks, and retries. Neither view alone tells you how many harmful actions succeeded.

Create an action-level baseline. For signup, record attempts, completed accounts, granted trials, and accounts later confirmed abusive. For login, separate failed authentication from successful sessions and confirmed account compromise. For checkout, distinguish abandoned attempts, completed orders, and duplicate effects.

Illustrative measurement example: during one test period, an endpoint receives 1,000 signup requests and creates 200 accounts. Those numbers do not establish that 800 requests were malicious: they may include retries, validation failures, and legitimate abandonment. Attach a stable operation identifier and outcome classification before calculating an abuse rate. These figures illustrate the accounting problem; they are not MANDATE production results.

Choose denominators explicitly. “Rejected requests divided by all requests” measures policy activity. “Confirmed abusive accounts divided by reviewed completed accounts” measures a different outcome and depends on the quality and timing of review. Show both the review coverage and the time window.

Why simple signals are insufficient

Shared networks can put many legitimate people behind one address. An authenticated account can still be compromised. A real browser can still be automated. A low request rate can still consume a valuable coupon or inventory reservation. These examples make one point: a single convenient attribute rarely captures intent.

Use network, browser, account, and action evidence according to the problem, while limiting collection to what you need. Sensitive evidence belongs behind appropriate access controls and retention rules. Customer-facing investigation views should explain grouped reasons rather than expose raw browser traces.

For account-access attacks, OWASP recommends layered defenses and warns against depending on client-side checks alone. Browser verification supplements authentication safeguards; it does not replace strong authentication.

Protect the action where abuse creates value

A public page read and a credit-granting request do not need identical policies. Apply coarse traffic controls where they help protect capacity, then place business rules at the operation that grants value. An application should still decide whether a customer qualifies for a promotion or may reserve inventory.

For browser-originated actions, verification can contribute evidence before that decision. The backend should evaluate the proof in its current context instead of trusting a client-side “passed” flag. If the request is a retry, reconcile it with the existing business operation rather than accidentally creating another effect.

A rollout checklist that protects legitimate traffic

  1. Choose one route. State the harmful outcome you want to reduce and the legitimate completion you must preserve.

  2. Record a baseline. Include devices, browser groups, traffic sources, and support reports relevant to that route.

  3. Observe proposed decisions. Keep existing access controls responsible for enforcement while you investigate uncertain results.

  4. Review exceptions. Check failed script loads, timeouts, accessibility issues, legitimate integrations, and repeated submissions.

  5. Enforce narrowly. Introduce a defined policy only when its consequences are understood.

  6. Retain rollback and compare outcomes. A traffic decrease is useful only in the context of successful customer actions and confirmed abuse.

Keep useful crawlers on your public content working throughout the rollout. A control meant for signup should not accidentally challenge the pages that help people discover and evaluate your product.

Where MANDATE fits

MANDATE focuses on invisible browser verification for high-value web actions, using opaque proof and hosted decisions against live server-side state. That is a narrower problem than classifying or blocking all website bot traffic. Your CDN, authentication, integration controls, and business rules retain their jobs.

For route-specific examples, see new-account protection and account-access protection. For the rollout method, read observe before enforce. If you are evaluating tools, the next step is a bot detection software evaluation grounded in your own action outcomes.

AI-assisted editorial revision. References checked for this revision.

Investigate the costly action next: Use the credential-stuffing workflow for distributed login attempts, or the card-testing route checklist for repeated provider-bound attempts. For an evaluation of available control layers, use the web application bot-protection guide. A request-volume change alone is not proof that abuse or customer harm changed.

More from the blog

FREE TRIAL ABUSE

Free Trial Abuse Playbook for SaaS and Ticketing

Detect free trial abuse without blocking legitimate evaluators. Define eligibility, measure repeat redemptions, and roll out controls with a rollback plan.

TICKET BUYING BOTS

Ticket Buying Bots: Detection and Defense Guide

Learn how ticket buying bots operate and how security teams can detect and mitigate automated abuse with invisible browser verification strategies.

PROMO ABUSE

Stopping Promo Abuse with Browser Proof

Learn how automated promo abuse distorts marketing metrics and drains budgets. Discover how server-verified browser proof stops multi-accounting