How to Make Your Website Readable to AI

Run any AI-built homepage through a curl request with no browser attached and you find out fast whether it was built for people or for machines too. Most weren't. In the exam corpus behind Tabkeel's pre-launch checklist for AI-built sites, seven in ten sites carried at least one AI-readability failure. Not because anyone did anything wrong on purpose. Because "readable to AI" isn't a setting Lovable, v0 or Cursor turns on by default, and nothing on screen tells you it's off.

This page is the deep dive on that one front: what "readable to AI" actually means, the twelve checks that decide it, which four to fix today, and a real finding from the exam corpus that never made it into the checklist post. If you're weighing whether AI can name your company at all, that's a different and later problem, covered in how to get cited by ChatGPT and AI search. This page is about the gate before that one: can a machine read the page at all.

What "readable to AI" actually means

AI readability is whether a language model, without running any JavaScript, can retrieve a page's content, determine what it is, and extract a usable answer from it. It is not the same discipline as generative engine optimization. GEO asks whether your content gets quoted in an answer. AI readability asks a smaller, earlier question: does the model even receive anything to quote.

The distinction matters because most advice skips straight to GEO, structuring answers, adding schema, chasing citations, while the reader's own site fails at step zero. A perfectly quotable paragraph behind a blocked crawler produces exactly one citation: none. So this page stays at step zero on purpose.

Two mechanisms decide whether a machine reader gets in at all: whether it's allowed to fetch the page, and whether the page hands over real content when it does.

How to make your website readable to AI, step by step

In order, worst problem first:

1. Allow the search and answer AI crawlers in robots.txt (OAI-SearchBot, PerplexityBot, Claude-User and the rest of the search class below).
2. Confirm your homepage's real content, not just the title tag, is in the HTML your server sends before any JavaScript runs.
3. Replace a generic framework title and add a real, descriptive <h1> to the served HTML.
4. Fix a broken or localhost og:image so link previews render everywhere your URL gets shared.
5. Add JSON-LD structured data describing what the product is.
6. Add an llms.txt file last. It's real, but it's the lightest-weight check on this list.

Each step is its own section below, with the exact evidence Tabkeel's exam looks for and the fix for each.

The crawler you block without meaning to

robots.txt has one line that ends AI visibility on the spot: Disallow: / under the wrong user-agent. The trouble is there isn't one kind of AI bot to allow or block. There are two, and they do opposite things to your site if you get the wrong one.

Search and answer crawlers fetch a page live, the moment someone asks a question that touches it. Block one of these and you disappear from that answer today, not eventually. Training crawlers instead collect pages to shape a future model. Block one of these and nothing changes about how you're described right now; a future version of that model just learns a little less about you. Templates that "protect content from AI" in one blanket line almost always catch both classes, which is how a site ends up invisible to the exact channel its landing page promises to win.

Here's the full list Tabkeel checks, split by what blocking each one actually costs you:

The top six are the ones to check first, because those are the only rows where blocking has a same-day consequence. Everything below the line is a legitimate, deliberate choice to opt out of training data. Just make it a decision, not an accident inherited from a template someone else wrote.

The other way in: content that exists without JavaScript

Allowing the right crawlers gets you nowhere if the page they fetch is empty. Most AI crawlers, OAI-SearchBot and PerplexityBot included, do not execute JavaScript. They read whatever HTML your server sends on the first request and move on. A page that assembles its headline, copy and pricing in the browser hands them a shell.

Google is the partial exception. It renders JavaScript on a delayed second pass, so a client-rendered page usually gets indexed eventually, just slower and with weaker signals in the meantime. Answer engines mostly skip that second pass entirely. What they get on the first fetch is what they get.

This is why "it looks fine, I checked it in Chrome" is the least useful QA step available to you here. Chrome runs the JavaScript. The crawler doesn't. The only honest test is to ask for the page the way a crawler does and read what comes back.

A real finding: the title nobody meant to publish

Not every AI-readability failure looks dramatic. One of the more common ones in the exam corpus is almost funny once you see it, and I've pulled the same one off more than one founder's homepage now: a page that renders perfectly, reads well, converts fine, and ships a <title> tag that still says Vite + React or Create Next App.

It happens because the title tag lives in a template file nobody opens again after the first commit. The founder wrote three paragraphs of homepage copy and never touched the one line that tells a machine what the page is before it reads a single word of that copy. To a person landing on the page, nothing looks wrong. To a model or a search engine filing the page by its title, the site is called "Vite + React App," and everything downstream inherits that mistake: the browser tab, the search snippet, the link preview when someone shares the URL, and the first signal an AI crawler uses to decide what it just fetched.

Tabkeel flags this as a high-severity finding, the same tier as a blocked answer-engine crawler, because the evidence is unambiguous and the damage is total: every reader of that title, human or machine, gets nothing. The fix ships as a prompt, not a diff, because the right words depend on the product:

Replace the framework-default title tag on the home page with a real one: "<Product>: <what it does in a few words>", 50 to 60 characters, written directly into the served HTML rather than set only by client-side JavaScript. Give every route its own descriptive title instead of one static value copied everywhere.

Ten minutes of work, once you know to look. The exam corpus keeps finding it because nothing in a normal QA pass, click through the site, check the layout on mobile, check the buttons, ever puts eyes back on a tag that isn't visible anywhere in the UI.

The AI Readability Checklist, ranked by severity

Twelve checks make up the front, and they don't carry equal weight. Fix top to bottom, not in the order you happen to notice them.

Notice where /llms.txt sits. It's the file every "AI SEO" checklist leads with, and it's ranked seventh here, medium severity, well below a blocked crawler or a missing headline. That ordering is deliberate, and the next section is why.

Where llms.txt actually fits

llms.txt is a plain text file at your site's root, written for language models rather than search engines, listing what your product is and which pages matter. It's a young convention, proposed in 2024, and adoption among the AI tools it targets is still uneven enough that treating it as a ranking lever oversells what it does.

The honest numbers back that up. Ahrefs analyzed 137,000 domains in June 2026 and found that 97% of published llms.txt files got zero requests the following month; of the small share that did get traffic, most of it came from SEO audit tools, not AI assistants. Rankability's June 2026 tracking puts adoption at 8.7% of the world's top 1,000 sites. Read together: it's a growing convention with real but small uptake, and close to no measured payoff yet, not a proven ranking lever.

None of that makes it worthless. It costs about ten minutes, and when a tool does read it, it's the cheapest accurate self-description you'll ever publish, no guessing required on the model's part. Write it, keep it honest about what the product does, and don't let it eat the afternoon that should go to the crawler and rendering checks above it. That's the entire case for llms.txt: cheap insurance, not a strategy.

Test it yourself in five minutes

You don't need a tool to check the two checks that matter most. Both take under a minute each.

1. Check what a crawler actually receives. Run curl -A "GPTBot" https://yoursite.com from a terminal, or use view-source in your browser (not "Inspect", which shows the DOM after JavaScript ran). If your headline, pricing and product description aren't in that raw response, no AI crawler is seeing them either.
2. Check whether the right bots are blocked. Open yoursite.com/robots.txt directly and search it for the crawler names in the table above. A blanket Disallow: / under *, or under any of the six search and answer crawlers, is the finding to fix first.
3. Check your title tag against your address bar. Look at the browser tab, not the page. If it still says the name of your framework or starter template, that's the generic-title finding from above, live on your own site.
4. Check for a JSON-LD block. View-source your homepage and search for application/ld+json. If it isn't there, run the page through Schema.org's validator to confirm, then add SoftwareApplication or Organization schema with your name, description and URL.

Four checks, five minutes, and you'll know more about how AI reads your site than most launch checklists ever ask you to find out. For the full twelve-check pass with clickable evidence for each finding, the free AI readability tool runs the same checks Tabkeel's exam does, without an account.

Mistakes that undo the fix

Fixing the low-severity rows first because they're easier. Adding a meta description while a search crawler is still blocked is real work spent on a page nothing is reading yet. Severity order exists so you stop guessing.

Treating llms.txt as the whole job. It's the file every generic AI-SEO guide leads with because it's new and easy to explain, not because it outranks a blocked crawler or an empty served page. Write it last, not first.

Checking readability once, at launch, and never again. A redesign, a new page builder, or a migration to a different framework can silently reintroduce a client-only render or overwrite a robots.txt rule someone set deliberately months earlier. Readability isn't a box you tick once.

Confusing readability with citability. Passing every check on this page makes your site legible. It doesn't make it quotable. Getting an answer engine to actually cite a legible page is the next, separate job, and it's what the citation playbook covers in full.

Where this fits in a launch

AI readability is one front. A site can pass all twelve of these checks and still fail on indexability, billing integrity, or a 404 page that quietly returns HTTP 200. The full seven-front pass, including this one, is what the pre-launch checklist for AI-built sites walks end to end, with the corpus data behind all seven. Run the free AI readability tool against a URL for just this front, or point the full Tabkeel exam at the site for all seven, evidence and paste-ready fixes included, no signup required.

Frequently asked questions