Bot traffic: good bots, bad bots and how to tell them apart
A large share of visits to any website are not people. Which bots you want, which you do not, how to tell them apart in your data, and what to do about each.
The short answer
A large share of the requests any website receives come from software, not people. Some of that software is exactly what you want: the crawler that puts you in search results, the fetcher that builds a link preview, the AI assistant reading your page to answer a question, your own monitoring. Some of it is what you do not: scanners looking for known holes, scrapers copying content, scripts trying stolen passwords on your login, spammers filling forms, fake traffic inflating numbers.
The word bot covers both. The job is to tell them apart and treat each accordingly.
Good bots and bad bots
| Good bots | What they do for you | Bad bots | What they do to you |
|---|---|---|---|
| Search engine crawlers | Index your pages so people find you | Vulnerability scanners | Look for outdated software to exploit |
| AI answer crawlers | Read pages to cite you in answers | Credential stuffers | Try leaked passwords on your login |
| Link preview fetchers | Show your page nicely when shared | Form spammers | Fill your forms with junk and probes |
| Monitoring services | Tell you when the site is down | Scrapers | Copy content, prices and product data |
| Accessibility and SEO tools you run | Audit your own site | Fake traffic and referral spam | Pollute analytics; advertise in your reports |
How to tell them apart
- Identification. Good bots say who they are in the user agent and publish the address ranges they come from, so a claim can be verified. Bad bots pretend to be ordinary browsers.
- Behaviour. Good bots read pages at a polite rate and follow your robots rules. Bad bots hit login pages, forms and admin paths, request hundreds of pages a minute, or ignore rules entirely.
- Targets. A visitor pattern that only touches your login, your forms or paths that do not exist is not a visitor.
- Analytics footprint. Most bots do not run analytics scripts, so they are invisible there and visible in server or CDN logs. Bots that do appear in analytics show zero engagement and odd origins.
What to do about each
- Good bots: allow them, give them a clean sitemap and a machine-readable index, and make sure your pages are fast for them too.
- Impolite but legitimate bots: rate limits, so a crawler that asks too fast slows down without being banned.
- Malicious bots: stop them at the edge. A CDN with bot rules filters known bad actors and suspicious patterns before they reach your site. Forms get honeypots, rate limits and a silent check; logins get rate limits and two-factor.
- Fake traffic in analytics: enable known-bot filtering in the analytics tool and ignore referrers you do not recognise.
What this means for you
Bots are not a problem to eliminate; they are traffic to sort. Allow and help the ones that bring you visitors, slow the impolite, and stop the malicious at the network edge before they cost you anything. If your site sits behind a CDN with bot rules and your forms and logins have the basic protections, most of this happens without you thinking about it, which is the goal.
Frequently asked questions
How much of our traffic is bots?
Often a large share of raw requests, though most of it never appears in analytics because analytics scripts do not run for bots. Your server or CDN logs show the real picture. A sudden jump in traffic with no matching rise in conversions or human behaviour is usually bots.
Should we block bots to protect our content?
Blocking well-behaved crawlers costs visibility in search and in AI answers, and does not stop bad actors, who ignore robots rules anyway. Allow the crawlers you benefit from, rate-limit the impolite, and use edge bot protection for the malicious. Blanket blocking is the wrong tool.
Our analytics shows visits from strange places with zero seconds on page. What is that?
Typically bots that do run scripts, or referral spam: automated visits designed to put a website's name in your reports so you look it up. Filter known bot traffic in your analytics settings, and ignore referrers you do not recognise rather than visiting them.
Sources
- Cloudflare Learning Center: What is a bot? (accessed 2026-09-11)
- Google Search Central: Overview of Google crawlers and fetchers (accessed 2026-09-11)