Should you block AI crawlers? What it costs you in visibility
Blocking AI crawlers feels like protection and works like invisibility. What the bots do, what you give up, and the middle path we chose for our own site.
Our position, stated plainly
Two kinds of crawler
| Kind | What it does | What blocking it means |
|---|---|---|
| Training crawler | Collects pages to improve future models | Your content does not contribute to training. Your visibility today is not directly affected |
| Answer crawler | Fetches a page because a user asked a question now, so the assistant can read and cite it | You do not appear in that answer. The competitor who allowed it does |
Most blocking advice treats them as one thing. They are named separately in robots rules by every major provider, and the decision about each is different. Blocking training crawlers is a defensible choice about your content’s future use. Blocking answer crawlers is opting out of a channel.
What being cited looks like
When someone asks an assistant which platform to choose for their store, or why their form gets spam, the assistant fetches a few pages and summarises them, with links. Those pages were written to answer the question plainly, were reachable by the crawler, and were structured so the answer was easy to find. Being one of them is the new version of ranking. It is also why this knowledge base exists.
The real risks, and what actually addresses them
- Content copied wholesale. A licensing and legal matter. A robots rule does not prevent it, because the parties who would do it ignore robots rules. Publish, monitor, pursue misuse when it happens.
- Server load. A static site on an edge network does not notice crawlers. A server-rendered site uses a rate limit for impolite bots. Neither requires disappearing.
- Sensitive pages. Anything that should not be public should not be public, for any crawler or any human. Authentication, not robots rules.
- Misrepresentation. An assistant summarising you badly. The fix is clearer content and structured data, so the summary is right.
What we did for this site
- Allowed the crawlers, training and answer kinds, explicitly, in the robots file.
- Published an index at a machine-readable address that lists every article with a one-line summary, and a full-text version for assistants that want depth.
- Structured every page: article, breadcrumb and FAQ data that says what the page is, when it was updated, and what questions it answers.
- Wrote answer-first, so the first paragraph of every article answers the question the title asks.
- Kept private things private by not publishing them, not by asking crawlers to look away.
What this means for you
Decide separately about training crawlers and answer crawlers, and know that most businesses that block “AI” have blocked both. If your website exists to bring you customers, allow the answer crawlers, make your pages easy to understand, and handle copying, load and privacy with the tools built for them. Being invisible is not a security posture.
Frequently asked questions
What is the difference between a training crawler and an answer crawler?
A training crawler collects pages to improve future models. An answer crawler fetches a page because a user asked a question right now and the assistant wants to cite a source. Blocking the first is a decision about training data. Blocking the second removes you from the answers your customers are reading today. Many businesses block both without knowing there are two.
Will AI assistants steal our content if we allow them?
They quote and link, much as search engines have done with snippets for years. Wholesale copying is a licensing and legal matter, not something a robots rule prevents, since bad actors ignore robots rules anyway. Allowing well-behaved crawlers and pursuing misuse when it happens is the same posture taken with search.
Does allowing crawlers create server load?
For a static site served from an edge network, negligible. For a server-rendered site, a crawler is just another visitor, and a rate limit handles the impolite ones. Load is a reason to fix hosting, not to disappear from AI answers.
How do we make sure AI understands our pages correctly?
Clear, answer-first content with headings, structured data on each page, and a machine-readable index such as an llms.txt file that lists what you publish and why. That is what we did for this knowledge base, and it is also what makes the pages better for human readers and for search.
Sources
- Google Search Central: Overview of Google crawlers and fetchers (accessed 2026-09-11)
- OpenAI: Overview of OpenAI crawlers (accessed 2026-09-11)