Technical SEO for AI

Technical SEO for AI comes down to making sure AI crawlers can reach and read your site, that your pages load fast enough and that changes arrive as quickly as possible at the search engines feeding AI answers.

That sounds like an extension of ordinary technical SEO and for a large part it is. The difference sits in the crawlers themselves: alongside Googlebot there are now GPTBot, ClaudeBot and PerplexityBot doing the rounds, each with its own rules, its own speed and its own limits. A site that works fine for Google can still be invisible to those crawlers, simply because nobody ever checked whether they get in. That makes this chapter less an addition to your checklist and more a check you would otherwise never carry out.

Make sure AI crawlers have access

• Check your robots.txt. GPTBot, ClaudeBot, PerplexityBot and Google-Extended each have their own user agent and blocking one of them shuts out that entire platform without anything showing on the outside.
• Check that your server lets them through. Some firewalls or CDN settings turn away unknown bots automatically, even when robots.txt allows them. That often happens silently, without an error message you can act on.
• Look at your server logs. There you see in black and white which AI crawlers already come by and which never make it past the front door. A crawler that never visits can never cite your site, however good the content is.
• Repeat this check periodically. New crawlers appear regularly and a setting that is correct today can shut out a new platform six months from now.

Render your content server-side

• JavaScript is a risk. A crawler that runs no JavaScript sees an empty page when your content is only built up in the browser. For Googlebot that is less often a problem these days, though far from every AI crawler renders as thoroughly.
• Server-side rendering or prerendering solves this. The HTML the crawler receives then already contains the full text, regardless of whether it runs scripts.
• Test it simply. Look at the page source without JavaScript, or use a tool that shows what a crawler sees, to check that your text is really in there. When the core text is missing from that view, it is missing for part of the AI crawlers too.

Work on speed, and not only for visitors

• Crawlers have a time limit as well. A slow server means a crawler takes fewer pages per visit, which slows indexing down on large sites in particular.
• Speed remains a direct ranking factor. Core Web Vitals such as loading time and interaction speed count towards how Google and other search engines judge your page, separately from how well your text is written.
• A fast site is a citable site. AI models often work with a time limit per source page, so a page that loads slowly risks being skipped in favour of a competitor that finishes on time.

Report changes with IndexNow

• What IndexNow does. It is a protocol that lets you pass a change straight to a search engine, instead of waiting until a crawler comes by on its own.
• Who uses it. Bing, Yandex, Seznam and Naver support IndexNow, while Google does not do so to this day, despite having tested it since 2021.
• Why that still matters for AI. Bing feeds the index behind ChatGPT Search, Copilot and parts of Perplexity, so a fast notification there can get a new or changed page into AI answers sooner than waiting for a crawler to notice it by chance.
• The effort is limited. Setting up IndexNow usually costs a single technical change, while the return lands on precisely the platforms that do not run through Google.

What gives no leverage: schema and llms.txt

• Schema markup is hygiene. Ahrefs tracked 1,885 pages that added schema between August 2025 and March 2026 and, compared with 4,000 control pages, found virtually no effect on AI citations, with a slight drop for Google AI Overviews and a negligible rise for AI Mode and ChatGPT.
• llms.txt is rarely fetched. The same Ahrefs looked at 137,000 sites in 2026 and found that 97 per cent of llms.txt files were never fetched by an AI crawler.
• What this means for your priorities. Use schema for rich results and for machine understanding and put llms.txt in place as cheap hygiene while you are at it, though expect neither to raise your visibility in AI answers in any demonstrable way.
• Why this still gets recommended so often. Both are easy to sell as a quick win, while the four points above demand more technique and more time. That makes them popular in advice, even without data behind them.

What does count is measurable: crawlers that have access, content they can read, a site that loads fast enough and a quick notification when something changes. That is less appealing than adding a new file, and it is what the data supports so far. Anyone who does the easy work first and leaves those four proven points alone chooses the illusion of progress over the work that actually makes a difference. That fits with what we see in hoe AI bepaalt wie het noemt: access and readability are the condition, and authority then decides who gets picked.

Want to know exactly whether AI crawlers can reach and read your site? Request the paid AI visibility audit.