Sitemaps and robots.txt: the two files every site needs
A sitemap tells crawlers what exists; robots.txt says what to leave alone. What each file does, what goes in it, and the mistakes that hide sites from search.
The short answer
Two small files at the root of your domain shape how crawlers see your site. The sitemap is a list of the pages you want found, with the date each last changed. The robots.txt file is a set of requests to crawlers about which parts of the site to leave alone. Both are simple, both should be generated automatically, and one of them, when wrong, can keep a whole site out of search for months without anyone noticing.
What goes in each
| File | Contains | Should not contain |
|---|---|---|
| sitemap.xml | Every page you want indexed, its canonical address, and when it last changed | Pages marked noindex, redirects, error pages, private pages, duplicate addresses |
| robots.txt | Rules per crawler for paths that should not be crawled, and the address of the sitemap | A blanket block on everything; paths you are trying to keep secret; anything you expect to be enforced |
The mistakes that hide sites
- The staging block that went live. A robots.txt that disallows everything is normal on a test copy and disastrous on the real site. Migrations carry it over. Search traffic falls for weeks before someone looks.
- The hand-maintained sitemap. Made once at launch, never updated. New pages are not in it, deleted pages still are, and the change dates are years old. Worse than none, because it tells crawlers the site is static.
- Sitemap full of the wrong pages. Redirected addresses, noindex pages, filtered and paginated duplicates. It wastes crawl attention and muddies what the site is about.
- Blocking the crawlers you want. Rules that block AI answer crawlers or preview fetchers, copied from a template, removing the site from places it wanted to be.
How a well-built site handles them
The sitemap is generated from the site itself at build time, from the list of real pages with their actual modification dates: a new page appears without anyone remembering, and a deleted page disappears. Robots.txt does not have to be generated in the same way. Where it holds no path rules, there is nothing in it that can go out of date, so a fixed file naming the sitemap address is enough. That is how this site does it: the sitemap generated, robots.txt fixed and pointing at it. What must never be hand-maintained is the file that tracks the pages.
What this means for you
Check both files today, then check Search Console to see whether they were read and what was found. If either is hand-maintained, missing or wrong, ask for both to be generated automatically from the site. Two small files, no ongoing effort, and one of the most common causes of “our site disappeared from Google” removed for good.
Frequently asked questions
Do we need a sitemap if the site is small?
It helps at any size and costs nothing when generated automatically. It tells search engines which pages exist and when they changed, so new and updated pages are found sooner. On a small site the main benefit is the change dates; on a large one it is discovery.
Can I use robots.txt to hide a page?
No. It asks well-behaved crawlers not to visit the page; it does not prevent anyone from opening it, and it does not remove it from search if other sites link to it. To keep a page out of search, use a noindex instruction on the page. To keep it private, put it behind a login.
How do I check ours?
Open yourdomain.com/robots.txt and yourdomain.com/sitemap.xml in a browser. The first should be short and must not contain a rule blocking everything. The second should list your real pages with recent dates. Search Console reports whether both were read and what was found.
Sources
- Google Search Central: Learn about sitemaps (accessed 2026-09-11)
- Google Search Central: Introduction to robots.txt (accessed 2026-09-11)