Is your app invisible to AI crawlers?

In The rise of the AI crawler, Vercel analysed crawler traffic across its network. In one month GPTBot made about 569 million requests and Anthropic's crawler about 370 million. Neither, nor any other major AI crawler they tested, rendered JavaScript. That includes OpenAI's crawlers, ClaudeBot, Meta's crawler, Bytespider, and PerplexityBot.

ChatGPT's and Claude's crawlers did download JavaScript files. They did not run them. The exceptions were Google's Gemini, which uses Googlebot's rendering infrastructure, and AppleBot.

Many prompt-to-app tools generate a client-rendered single-page app. The server sends a near-empty HTML shell and a script tag, and the browser builds the page:

<!doctype html>
<html>
  <head><title>My App</title></head>
  <body>
    <div id="root"></div>
    <script src="/assets/index-4f2a.js"></script>
  </body>
</html>

A person sees a full landing page. A crawler that does not run JavaScript sees a title and an empty div. Your product description, pricing, and docs never reach the model that might have cited them.

This is easy to miss because Google renders JavaScript. A client-rendered site can be indexed and rank in Google Search, and its owner reasonably assumes it is visible everywhere. For answer engines that rely on their own non-rendering crawlers, it is not.

• Server-render or statically generate public pages: home, pricing, features, docs, and blog. Keep client rendering for the signed-in app.
• If you must stay client-side, pre-render public routes to static HTML at build time.
• Put the main content, headings, and links in the initial HTML. Content behind tabs or read-more buttons should be in the markup and hidden with CSS, not fetched on click.
• Add structured data and an llms.txt file that summarises your key pages in plain text.

The quickest test is to look at what the server sends before any script runs:

curl -s https://your-site.example | sed -e 's/<[^>]*>//g' | wc -w

If that number is close to zero on a page that looks full in the browser, AI crawlers see nothing. Routelint's SEO and GEO check does this automatically. It fetches raw HTML without running scripts, reports the word count for each crawled page, and reads your robots.txt rules for the major AI crawlers.