Search the phrase "AI-agent-ready website" and almost everything you find is selling something: a platform, an audit, a monitoring subscription. That is not a reason to dismiss the topic, but it does mean the available advice is shaped by what is sellable rather than by what matters.

We recently did this work on our own site — a Laravel application with roughly 3,500 public pages — and measured it before and after. This is what actually mattered, in order, along with the one mistake we made that would have quietly undone a large part of the rest.

What "readable by an AI agent" actually means

Three different kinds of software are being talked about here, and they have different needs. Conflating them is why so much advice is vague.

  • Crawlers that collect pages for training or for an index. They want stable URLs and permission.
  • Answer engines that retrieve a handful of pages to compose an answer for someone right now. They want extractable claims and something to attribute.
  • Agents acting on a person's behalf — finding a supplier, comparing programmes, checking a requirement. They want to navigate and query, not read prose.

The good news is that the first two are served almost entirely by doing ordinary web fundamentals properly. Only the third asks for anything genuinely new.

Start with the fundamentals, because they are most of the score

Server-render the content

An agent that does not run JavaScript sees your HTML and nothing else. If your page ships an empty container that gets filled after hydration, that page is blank to a meaningful share of the software reading it.

This is easy to check and easy to get falsely reassured about. Fetch your own page with a plain HTTP request — no browser — and look at what comes back inside <main>. If your headline and body text are not there as text, nothing else in this guide will help.

One H1, and headings in order

Heading structure is how any reader that is not looking at your CSS understands the shape of a document. One <h1> stating what the page is; <h2>s for the real sections; no skipping from h2 to h4 because the smaller size looked better.

Give your <h2>s stable id attributes too. It costs nothing and it lets anything — an answer engine, a person, another page — link to a specific section rather than to the top of a long document.

Say what the page is in the title and description

This sounds too obvious to mention. It is also the single largest defect we found on our own site: 33 pages — every page for eleven certification programmes — rendered the bare site name as their title and shared one identical site-wide meta description. The per-page copy already existed in the database. It was simply never rendered.

Worth checking directly rather than assuming, because this class of bug is invisible in a browser: the page looks fine, and only the tab and the source reveal it.

Structured data: telling a machine what you are

Semantic HTML tells a machine how your page is organised. Schema.org JSON-LD tells it what the page is about, and — more importantly — what your organisation is.

Two things matter more than which types you pick:

Emit an Organization node on every page, with one stable @id. Our organisation identity originally lived inside a fallback block that only rendered on pages which had not set their own SEO tags. The effect was precisely backwards: the identity disappeared from exactly the pages that mattered most — the articles, the guides, the programme pages — and survived only on pages that had nothing specific to say. Give the organisation an absolute @id like https://example.com/#organization, emit it site-wide, and have every other node reference it rather than restating a slightly different copy of it. That is what lets an engine resolve your name to one entity instead of several.

Only mark up what is actually on the page. A Course node with no course behind it, a rating with no ratings, an FAQ that appears nowhere in the visible content — these are worse than no markup. They are the category of thing that gets structured-data privileges withdrawn.

Beyond that: BreadcrumbList on anything nested, Article with datePublished and dateModified on written content, and the specific type that fits your content where one exists.

robots.txt: the mistake worth learning from ours

We rewrote our robots.txt to state explicitly that AI crawlers were welcome, listing each one:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /paid-course-content/

This is wrong, and dangerously so.

Under the Robots Exclusion Protocol (RFC 9309), a crawler obeys exactly one group: the most specific User-agent line that matches it. It does not merge that group with the wildcard group. So the file above tells GPTBot and ClaudeBot that their rules are "Allow: /" and nothing else — exempting those two crawlers, and only those two, from every exclusion below, including the one protecting paid content.

We had written a file that did the opposite of what it appeared to say, and it read as more careful than the version that was actually correct.

The fix is to have one group that applies to everyone, and to express the AI-crawler decision as a documented comment rather than as a structural exemption:

# Decision: AI and answer-engine crawlers are allowed. Our free research is
# published in order to be found and cited. To opt out of a specific crawler,
# add a group for it containing EVERY Disallow line below plus Disallow: /.
# Never a partial group - see RFC 9309 section 2.2.1.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /paid-course-content/

If you do want to restrict one crawler specifically, its group must repeat the full rule set plus the extra restriction. The opt-out tokens worth knowing are Google-Extended, Applebot-Extended, GPTBot, ClaudeBot, CCBot, Bytespider and meta-externalagent.

Two more robots.txt rules that are easy to get wrong: never block CSS or JavaScript, because a crawler that cannot load them cannot render your page as a person sees it; and declare your sitemap with a Sitemap: line.

llms.txt: useful, unofficial, and easy to get wrong

llms.txt is a proposed convention: a markdown file at your root that indexes your most important public resources for language models, so they can orient without crawling everything.

It is a proposal, not a standard, and no major engine has committed to honouring it. That is an argument for keeping it cheap, not for skipping it.

The failure mode is worse than absence. Our previous llms.txt was hand-written and had drifted: it described five research guides and zero of our eleven certification programmes, because it had been written before those existed and never revisited. A file whose entire purpose is telling a model what exists is actively harmful when it is wrong, precisely because it looks authoritative and gets trusted without checking.

So: generate it from live data, or do not publish it. Ours is now produced by a command that reads the database, so its counts and listings cannot disagree with the site. We go into the format, the decision rule and llms-full.txt in llms.txt: what it is and whether you need one.

Two things worth putting in yours that most examples omit:

  • How to cite you. Attribution guidance is the thing an engine most needs and can least infer — which page URL to link, what to call you, what your content is not.
  • What you are not. Ours states that nothing on the site is legal, tax, accounting or compliance advice. That is a boundary worth stating in the file a model reads first.

Provenance: give an engine something it can attribute

An answer engine composing a response has to decide whose claim to repeat. Pages that make attribution easy have an obvious advantage over pages that do not.

Four things, on the page and in the markup:

  • A named author. An organisation name counts; anonymity does not.
  • A published date.
  • A last-verified date, which is not the same thing. For anything touching law, policy or fast-moving technology, "when did someone last check this was still true" is the more useful date, and almost nobody publishes it.
  • Sources, listed and linked. If your claim rests on a statute or a standard, cite the statute or the standard — not an article about it.

State a summary explicitly, too. An engine lifts a stated conclusion far more reliably than it reconstructs one from six paragraphs of narrative. A short "the answer is" block and a list of key takeaways above the body do more for extractability than any amount of keyword placement.

APIs: turning scraping into reading

This is the highest-effort item and the last to reach for. It is only worth doing if you have structured data an agent would genuinely want — a directory, a catalogue, a dataset.

If you do, a small read-only JSON API is better for everyone than being scraped: the agent gets clean data, and you get rate limiting, a stable contract, and the ability to state licensing terms.

What we found mattered in practice:

  • A service index at a predictable path that lists every endpoint with the question each one answers. An agent choosing between eight endpoints needs more than eight names.
  • An OpenAPI description, with a real prose description on every path. Generated schemas with empty descriptions are decorative.
  • Discovery via RFC 9727 — a /.well-known/api-catalog document, so an agent holding only your domain finds the API in one hop instead of guessing at /openapi.json, /swagger.json and /api/docs.
  • Identical visibility rules to your web pages. This is the one that bites. It is very easy for an API to expose a draft, an unpublished record or a paid resource that the corresponding page correctly hides. Mirror the page's rule in the query, and write the test that proves it.
  • Rate limiting from day one. An unthrottled public endpoint is a free denial of service.

What we would skip

Honesty about what did not earn its place:

  • Custom ai: meta tags. No engine documents support for them. Harmless, but not work.
  • Marking up everything you can. Schema types you cannot support with real page content are a liability, not a signal.
  • Writing for machines instead of people. Every technique above also makes the page better for a human reader — clearer structure, stated conclusions, visible sources. Anything that improves the machine-readability while making the page worse to read is a bad trade, and usually a sign of gaming rather than clarity.

A checklist you can actually run

In rough order of value per hour spent:

  1. Fetch a page without a browser. Confirm the body text is in the HTML.
  2. Check every page template has its own <title> and meta description. Do not assume — look at the source.
  3. One H1 per page; headings in order; stable ids on sections.
  4. Emit Organization site-wide with a stable absolute @id; reference it from every other node.
  5. Add BreadcrumbList to nested pages and Article with both dates to written content.
  6. Validate every JSON-LD block parses and carries @context.
  7. Read your robots.txt as a crawler would: one group per crawler, no merging.
  8. Confirm CSS and JS are not blocked, and that a Sitemap: line is present.
  9. Check your sitemaps carry your real public hostname. Ours advertised a local .test domain for months, because generation read the app's configured URL.
  10. Put author, published date, last-verified date and sources on every substantive page.
  11. Add a stated summary above the body on long pages.
  12. Generate llms.txt from live data, or leave it out.
  13. Only then consider an API.

The bottom line

There is no separate discipline called AI-agent optimisation. There is web craft done properly — server-rendered, semantically structured, honestly marked up, with stable URLs and stated provenance — plus a small number of genuinely new affordances, of which a generated llms.txt and a discoverable read-only API are the two worth the effort.

The techniques that help a machine understand your site are, almost without exception, the same ones that help a person. When a technique only helps the machine, that is usually a signal you have drifted from making yourself understood into trying to game something. The first is durable. The second has never lasted.