Loading patterns…
Agent-Readable Web (llms.txt / NLWeb)
Supply-side contracts that expose a site's content to agents directly instead of forcing them to scrape rendered HTML. llms.txt (Jeremy Howard, 2024) is a curated Markdown index at /llms.txt plus .md shadow pages that give an LLM high-density, navigation-free content at inference time. Microsoft NLWeb (2025, led by Schema.org co-creator R.V. Guha) goes further with an /ask natural-language endpoint that returns structured JSON grounded in the site's own Schema.org data, with every NLWeb instance doubling as an MCP server. Together they define the agent-readable site, the read-path counterpart to robots.txt for the agentic web.
In 30 seconds
- What
- Site publishes curated Markdown at /llms.txt and .md shadow pages, plus optional /ask endpoint returning structured JSON, so agents retrieve content without scraping HTML.
- When to use
- You operate a service or content site and want agents to access your data reliably, accurately, and at scale without parsing rendered pages.
- Watch out
- Requires you to maintain two content surfaces in sync; stale or incomplete /llms.txt defeats the purpose and agents fall back to scraping anyway.
Ask the AI expert about this pattern
Opens the assistant with your question prefilled. You review it before sending.
Agent-Readable Web (llms.txt / NLWeb): Overview
Supply-side contracts that expose a site's content to agents directly instead of forcing them to scrape rendered HTML. llms.txt (Jeremy Howard, 2024) is a curated Markdown index at /llms.txt plus .md shadow pages that give an LLM high-density, navigation-free content at inference time. Microsoft NLWeb (2025, led by Schema.org co-creator R.V. Guha) goes further with an /ask natural-language endpoint that returns structured JSON grounded in the site's own Schema.org data, with every NLWeb instance doubling as an MCP server. Together they define the agent-readable site, the read-path counterpart to robots.txt for the agentic web.
- Curated /llms.txt Markdown index of a site
- .md shadow pages for clean, high-density content
- NLWeb /ask natural-language query endpoint
- Structured JSON grounded in Schema.org data
- Every NLWeb instance also acts as an MCP server
- Retrieval without brittle HTML scraping
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.
References
The papers, specifications, and repositories this pattern is based on.
From the engineer behind this catalog
Get your agent architecture reviewed
This page documents one pattern. Your system runs dozens, and most failures live in how they fit together. Have the whole design reviewed against the 288 patterns in this catalog: architecture, reliability, evaluation and cost, every finding mapped to the pattern that fixes it.
€750 instead of €1,500, one week, written report and walkthrough call, until 30 September