Home·Guides·What robots.txt and llms.txt should say
What robots.txt and llms.txt should say
Routelint·
A shipped app is readable when robots.txt allows the public crawlers and llms.txt lists the pages a person can open.
robots.txt belongs at the site root. Allow GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot on the public pages. Disallow the signed-in app and the API. Name the sitemap on its own line. A file that disallows every path hides the site. A missing file leaves the rules unreadable.
llms.txt belongs at /llms.txt. It is a short plain-text map: what the product is, the public pages, and a contact URL. The HTML of those pages still needs real sentences in the first response, because some crawlers never run JavaScript.
Cursor, Lovable, and v0 ship the interface. They do not write these two files. Paste the public URL into the check. The scores and the row names are immediate. The evidence behind each crawler opens after you sign in.