Your site is probably invisible to AI crawlers

We build a research tool for people who read a lot. So it was uncomfortable to discover that our own marketing pages had nothing to read.

Here is the check. Fetch a page the way something without a JavaScript engine would, and strip the tags:

curl -sSL https://example.com/about \
  | sed -n '/<body/,/<\/body>/p' \
  | sed 's/<[^>]*>//g'

If that prints your copy, you are fine. If it prints nothing, every word on that page arrives only after a browser downloads and executes your bundle.

Ours printed nothing.

Why this is worse than it used to be

The standard answer is that Google renders JavaScript, so a single-page app is fine. That was mostly true when Google was the only reader that mattered.

It is not the only reader any more. When somebody asks an assistant which tool to use, the crawlers behind that answer — GPTBot, ClaudeBot, PerplexityBot — fetch the HTML and read it. They do not run your bundle. To them an empty <div id="root"> is an empty page, and an empty page has nothing to recommend.

Even for Google the situation is softer than people assume: rendering happens on a second pass, at Google's convenience, and it is a queue rather than a promise.

Two things that were quietly wrong

Nothing was prerendered. The fix is unglamorous: after the production build, boot the bundle in headless Chromium, visit each public route, and write the rendered DOM back out as real HTML. Roughly a hundred lines.

One detail is easy to miss. Landing pages usually animate sections in as they scroll into view, which means those sections sit at opacity: 0 until they have actually entered the viewport. Capture without scrolling and you faithfully bake your entire page in as invisible. The prerenderer has to scroll the whole page first.

The static server was undoing it. Our container served the build with serve -s. That -s rewrites every request to /index.html before it looks at the filesystem — so the prerendered about/index.html sat on disk, correct and complete, and was never served once. We only caught it because we tested the serving layer separately from the build.

The replacement is a short config that lists the routes which genuinely need the client-side shell, and lets everything else be served as a file. A side effect worth having: unknown URLs now return a real 404 instead of 200 with an empty page, which is what stops search engines treating dead links as live ones.

What it cost

The page grew from a 10 KB shell to 31 KB gzipped — with the entire page inside it, rather than a promise to fetch one. That is a good trade in both directions: readers see content sooner, and crawlers see content at all.

If you run a single-page app with a marketing site attached, run the curl above before you spend anything on content or ads. There is no point buying traffic to a page that cannot be read.