Loading...
Agent-Readable Web (llms.txt / NLWeb)
Supply-side contracts that expose a site's content to agents directly instead of forcing them to scrape rendered HTML. llms.txt (Jeremy Howard, 2024) is a curated Markdown index at /llms.txt plus .md shadow pages that give an LLM high-density, navigation-free content at inference time. Microsoft NLWeb (2025, led by Schema.org co-creator R.V. Guha) goes further with an /ask natural-language endpoint that returns structured JSON grounded in the site's own Schema.org data, with every NLWeb instance doubling as an MCP server. Together they define the agent-readable site, the read-path counterpart to robots.txt for the agentic web.
Loading technique guide…
Agent-Readable Web (llms.txt / NLWeb)
Supply-side contracts that expose a site's content to agents directly instead of forcing them to scrape rendered HTML. llms.txt (Jeremy Howard, 2024) is a curated Markdown index at /llms.txt plus .md shadow pages that give an LLM high-density, navigation-free content at inference time. Microsoft NLWeb (2025, led by Schema.org co-creator R.V. Guha) goes further with an /ask natural-language endpoint that returns structured JSON grounded in the site's own Schema.org data, with every NLWeb instance doubling as an MCP server. Together they define the agent-readable site, the read-path counterpart to robots.txt for the agentic web.
Loading technique guide…