Headless sites and AI crawlers: is your site an empty page to ChatGPT?
Most AI crawlers don't run JavaScript. If your headless or app-style site builds content in the browser, they may see nothing at all. Here's how to check in five minutes and what to fix.
Headless sites can be the best-performing sites in AI search, or completely invisible to it. The difference comes down to one question: is your content in the HTML your server sends, or does JavaScript build it in the browser?
Why this matters now
When Googlebot visits a page, it can run the JavaScript and see the finished result. That’s why many client-rendered sites rank fine in Google and nobody noticed a problem.
Most AI crawlers don’t do that. GPTBot, ClaudeBot and PerplexityBot fetch the HTML and read what’s there. If the page is an empty <div id="root"></div> plus a bundle of scripts, that’s all they get (TripleDart’s research, drawing on Vercel and Cloudflare data).
So a site can rank on Google and still be invisible to ChatGPT, Claude and Perplexity.
Check your site in five minutes
You need a terminal. Pick three important pages: your homepage, a key service or product page, and an article.
1. Fetch the raw HTML the way a crawler does.
curl -s -A "GPTBot" https://yoursite.com/your-page | grep -c "a sentence from the middle of that page"If it prints 0, the text isn’t in the raw HTML. AI crawlers can’t see it.
2. Check your metadata and structured data.
curl -s https://yoursite.com/your-page | grep -E -o '<title>[^<]*</title>|application/ld\+json'You want to see a real title and at least one application/ld+json block. JSON-LD injected by Google Tag Manager or a client-side script won’t show up here, and AI crawlers won’t see it either.
3. Check you’re not blocking the bots.
curl -s https://yoursite.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yoursite.com/A 403 or a challenge page means your CDN or firewall is blocking the bot. Some CDNs and security tools block AI crawlers by default, so check the bot settings even if you never changed them.
No terminal? Open the page in Chrome, turn off JavaScript in DevTools (Settings, then Debugger, then Disable JavaScript), and reload. What you see is roughly what AI crawlers see.
How to fix it
The fix is to send finished HTML from the server. Every major headless framework can do it. You usually don’t need a rewrite, just the right rendering mode on the pages that matter.
| Framework | What to use | Watch out for |
|---|---|---|
| Next.js (App Router) | Server Components with static generation or ISR | Content fetched in useEffect or marked "use client" at the top of a page |
| Next.js (Pages Router) | getStaticProps or getServerSideProps |
Pages that fetch data only on the client |
| Astro | Static output (the default) | Content loaded inside client-only islands |
| Nuxt | ssr: true (the default) or prerendering |
<ClientOnly> wrapping real content |
| SvelteKit | Prerender or SSR (the default) | ssr = false on content routes |
| Gatsby | Static build (the default) | Content fetched at runtime in the browser |
| React SPA (Vite, Create React App) | Move content pages to a framework above, or add prerendering | Everything, by default |
Then make sure three things are in the server HTML on every content page:
- The main content, including headings and body text.
- The
<title>, meta description and canonical tag. - Your JSON-LD, rendered on the server, not injected by a tag manager.
Interactive pieces like filters, calculators and carts can stay client-side. They don’t need to be read; the content around them does.
Our verdict: Do it now
If you run a headless site, this is the single highest-value check you can make for AI search, and it takes five minutes. If the test fails, the fix is usually a rendering setting, not a rebuild.
We built SEOWhat.now as fully static HTML with Astro for exactly this reason. Want us to run the check on your site? Ask us, or see how we help.