SEO & GEO2 min read
robots.txt and llms.txt mistakes that hide AI-built sites
A single line in robots.txt can remove a site from every answer engine. Generated sites ship with a few of these lines more often than you would think.
Key takeaways
Disallow: /underUser-agent: *blocks every crawler, including AI crawlers you may want.- Staging rules often survive into production when an agent copies config between environments.
- Each AI company uses named user agents. Decide deliberately which to allow.
- llms.txt is a short plain-text map for language models. An empty or missing file is a missed opportunity, not an error.
What robots.txt does and does not do
robots.txt tells well-behaved crawlers which paths they may fetch. It is not access control: anything it hides is still public to anyone who asks. And it is matched by user agent, so a rule for * applies to every crawler that has no more specific group. Google's introduction is the clearest reference.
The mistakes we see most
Blocking everything. A staging config that should never have shipped:
User-agent: * Disallow: /
Blocking AI crawlers by accident. A template that blocks a list of bots, copied without reading it:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
- No sitemap line, so crawlers have to discover pages by following links.
- A sitemap that lists localhost or preview URLs instead of production ones.
- Disallowing
/_next/or/assets/, which can stop renderers from loading the CSS and scripts they need. - Using robots.txt to hide private pages. It advertises them instead.
Know which AI agents you are allowing
AI companies publish separate user agents for different purposes. OpenAI, for example, documents GPTBot for training, OAI-SearchBot for search, and ChatGPT-User for live browsing on a user's behalf. Blocking one does not block the others. Google-Extended is a control token: it governs whether Google may use your content for its AI models, without affecting Google Search.
There is no single right policy. Decide which uses you want, write explicit groups for those agents, and keep a sitemap line at the bottom:
User-agent: * Allow: / Disallow: /api/ Disallow: /dashboard/ Sitemap: https://your-site.example/sitemap.xml
llms.txt: a short map for language models
The llms.txt proposal is a Markdown file at /llms.txt that summarises what your site is and links to its most useful pages. It is a proposal, not a standard, and no major AI company has committed to reading it. It is still cheap to add, and it gives any tool that does read it a clean summary instead of a scraped page.
# Your Product > One sentence on what it does and who it is for. ## Docs - [Getting started](https://your-site.example/docs/start): Set up in five minutes - [Pricing](https://your-site.example/pricing): Plans and limits
One public audit, two kinds of gap
The same habits that leave a site hard to find, such as copied config, leftover staging rules, and nobody reading the raw responses, also leave it exposed. A quick look at what your production host actually serves to an anonymous client often turns up both: a robots.txt that blocks everyone, and a /.env that answers 200.
Routelint's SEO and GEO check reads your robots.txt and reports access for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot, and checks for a sitemap and llms.txt. The free security scan covers the other side.
Questions
- Should I block GPTBot in robots.txt?
- It depends on whether you want your content used for model training. Blocking GPTBot does not block OAI-SearchBot or ChatGPT-User, which OpenAI documents separately for search and live browsing.
- Do AI companies read llms.txt?
- llms.txt is a community proposal. Support varies and no major provider has committed to it, but it is cheap to publish and useful to tools that do read it.
The free SEO and GEO check fetches raw HTML the way a non-rendering crawler does, then reads robots.txt, the sitemap, and llms.txt.
Sources
- Introduction to robots.txt — Google Search Central
- Overview of OpenAI crawlers — OpenAI
- The /llms.txt file — llmstxt.org
Published November 1, 2026 by the Routelint team.