Next.js and AI Crawlers: What GPTBot Actually Gets

Next.js keeps a list of bots that get their metadata in the <head>. Bingbot is on it. Twitterbot and Slackbot are on it. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot aren't.

We wanted to know what that costs on a real build, and whether the advice everyone repeats about Next.js and AI crawlers ("use SSR, avoid client components") holds up. Most of what's published on this topic cites one Vercel study from December 2024 and stops there. So we scaffolded a fresh app, wrote 13 rendering patterns, requested each one with eight user agents and wrote down where every sentence landed. Some of the results surprised us. 'use client' turned out fine. The trouble was somewhere else.

Do AI crawlers run JavaScript?

AI crawlers such as GPTBot, ClaudeBot and PerplexityBot have not been shown to run JavaScript: they read the HTML your server sends. Vercel and MERJ's network study found no major AI crawler executing scripts, and one unconfirmed 2026 report suggests OpenAI's bots may be starting. Google is the exception, because it renders pages.

In that study, ChatGPT's crawler downloaded script files in 11.50% of requests and Claude's in 23.84%, and neither executed them. The data is almost two years old, though, and the picture has started to move.

Our read: treat the September report as a reason to keep watching your logs, and keep building for the crawler that reads bytes. Google's JavaScript SEO documentation describes a rendering step for pages it crawls, which is why a site can rank in Google and still come up blank in ChatGPT.

13 Next.js patterns, requested as GPTBot

The setup: create-next-app 16.4.3 with React 19.3.0, a production build under next start, tested on 1 November 2026. Slow patterns wait 3 seconds with a timer standing in for a database call. Each page carries a unique marker sentence, and we parsed the raw response to see whether that marker sat in normal HTML, inside a hidden block, inside a script tag, or nowhere.

Turning Cache Components off (the scaffold ships with it on) changed one row: pattern 11's body text moved into normal HTML because nothing else was holding it back. An async page with no Suspense at all, which only builds with Cache Components off, put everything in normal HTML and made every visitor wait the full 3 seconds for the first byte.

'use client' is fine. Where the text gets set is the problem

Pattern 3 is the row that contradicts the common advice. A client component is still rendered to HTML on the server, so a 'use client' pricing table with its text in the JSX reached GPTBot intact. Add a useEffect that sets the same text after mount (pattern 4) and the crawler gets your loading state in its place.

That distinction matters when you're reading code an AI tool wrote. You don't need to strip 'use client' from your app. Search for text that arrives through useEffect, useState fed by a browser fetch, or ssr: false.

Patterns 7 and 9 are the quiet ones, and we think they do the most damage on marketing sites. Pricing tabs, a monthly and annual toggle, the FAQ accordion: each looks finished on screen and each drops its closed panels from the HTML when written as conditional rendering. FAQ answers are written as direct answers, the shape an answer engine can lift. Pattern 8 gets the same interaction with a plain <details> element and keeps every answer in the response. (We cover the rest of what makes a page quotable in how to get cited by ChatGPT.)

Pattern 6 needs a caveat in the other direction. The sentence is technically in the bytes, inside the self.__next_f script. Vercel's study noted that content like this may still get picked up, since models can read non-HTML text. We wouldn't bet a product description on it.

The title that moved: streaming metadata and htmlLimitedBots

Streaming metadata is how Next.js avoids holding the page while generateMetadata waits on data: it sends the shell first and appends the tags to the <body> when they're ready. htmlLimitedBots is the setting that lists which user agents get blocking metadata in the <head> instead, and setting it replaces the default list.

Plain Googlebot gets the streamed version too, and that's fine for it, because Google renders. The AI crawlers get the same bytes with no evidence of that step. A parser that reads the title from the head finds nothing. One that closes the connection at 2 seconds never sees a title at all, and we measured exactly that: the title wasn't in the bytes at the 1-second or 2-second mark.

Whether this hurts you depends on how slow your metadata is. On a page where generateMetadata resolves fast or comes from a cache (pattern 12), the tags land in the head for everyone. The risk concentrates on routes that fetch per request: product pages, profile pages, anything that hits a database with a connection() or a cookie read before it can name itself.

Hidden blocks: in the bytes doesn't mean read

A hidden block is the <div hidden id="S:0"> React streams at the end of the response, holding Suspense content until a script swaps it into place. No JavaScript means no swap. So patterns 2 and 10 leave a crawler with a visible "Loading" and the real content sitting in a hidden element.

We don't know what OpenAI's or Anthropic's extraction does with that element. So we ran the streamed page, with distinct text in every paragraph, through four open-source tools that turn HTML into text:

Two kept it, two dropped it, and the split is telling: the plain converters keep everything, and the two built to isolate article content threw the streamed part away. Neither pair tells you what a specific AI crawler does. Both tell you that streamed text has a real chance of being lost somewhere between fetch and answer. That's enough to keep your main sentence out of a Suspense boundary.

Two fixes for a slow title, measured

Same route, same 3-second data call, requested as GPTBot:

Caching wins wherever the data can tolerate it. A product name doesn't change per request, so list your known slugs and the crawler gets a finished page. This is the code we measured, with the data call swapped in:

async function load(slug: string) {
  'use cache'
  cacheLife('hours')
  return db.product(slug)
}

export async function generateStaticParams() {
  return db.productSlugs()
}

export async function generateMetadata({ params }) {
  const { slug } = await params
  const p = await load(slug)
  return { title: p.name, description: p.summary }
}

Reach for htmlLimitedBots when the metadata truly varies per request. Copy the default regex from the Next.js source first, because your value replaces it, and append GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User. You'll be trading a late title for a late first byte, so if the slow call takes more than a second or two, fix the call.

The First-Response Test: four checks for any URL

This is the check we now run on a page we care about. It takes about two minutes and needs nothing beyond a terminal.

curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.4)" https://yoursite.com/page -o page.html
grep -c "a sentence from your page" page.html
grep -obE '<title|</head>' page.html | head -2
grep -o '<div hidden id="S:[0-9]*"' page.html | wc -l

1. Is the sentence in the bytes? A count of 0 means it's set in the browser: patterns 4, 5, 7 or 9. Pick a sentence with no apostrophes or quotes, because React escapes those in the HTML and grep will miss them.
2. Is it outside script tags? Open page.html and find the sentence. If it only appears inside self.__next_f.push, you have pattern 6.
3. Does <title come before </head>? If </head> prints first, the metadata streamed: pattern 11.
4. Are there zero hidden blocks holding your main content? A nonzero count is normal on dynamic pages. Check that the sentence you care about isn't inside one.

We ran it on five tabkeel.com pages (home, pricing, an article, a tool and the connector page): title in the head on all five, no hidden blocks, between 484 and 3,395 words of visible text. That's a statically rendered marketing site, so it's the easy case.

When a page fails, hand your coding agent the specific failure. This is the shape of prompt that works:

Fetched as a crawler with no JavaScript, this route's FAQ answers are missing from the HTML because the accordion renders them conditionally. Rewrite it with native details and summary elements, keep the styling, and make sure every answer is present in the server response.

To check a whole site instead of one URL, the AI readability test at tabkeel.com/tools/ai-readability looks at served HTML the way these crawlers do. The full exam also flags any page shipping under 220 characters of text without JavaScript, which is how patterns 4 and 5 show up at page scale. One gap you should know about: today it doesn't flag a streamed title, a hidden block or a closed tab, so the curl check above is still the way to catch those.

What about Lovable and plain Vite apps?

As a control we built a default Vite 8 and React 19 app and requested it as GPTBot. The response was the index.html title and an empty <div id="root">. Zero words of page text. The marker sentence existed only in the JavaScript bundle.

If you built on Lovable, check which stack you're on. Its documentation covers upgrading older React and Vite projects to TanStack Start, and third-party guides report server rendering as the default for new projects since May 2026. Lovable says older projects stay readable to search engines through pre-rendering. Whether an AI crawler gets the same treatment is a question for your own URL: run the curl command with its user agent and you'll know in ten seconds. We cover the fixes for a client-rendered site in how to make your website readable to AI. And since these crawlers already spend a third of their fetches on missing pages, the soft 404 behavior of the same Next.js setup is worth checking in the same sitting.

If ChatGPT describes you wrongly even though your HTML passes every check, the problem has moved past rendering, and controlling what ChatGPT says about your brand picks up from there.

Frequently asked questions