# Lyrenth

> Lyrenth is the AI-readable web index. Any public URL becomes one clean AIDocument: markdown, structure, and token economics in one JSON contract. Built for agents.

Lyrenth is the AI-readable web index: a search index of the
public web built for AI. One standing index of 2B+ pages, each page
served as a stable AIDocument: clean markdown, source trace, cache
truth, page structure, and token economics in one JSON contract. The
same envelope for every page, for every agent. A query endpoint (send
a question, get the best-matching AIDocuments) is coming. Built for
agents. Origin-friendly by default.

This file follows the llms.txt convention (https://llmstxt.org/). If
you're an AI agent reading this, you can pivot from here to the
canonical pages below; or, if you want every URL on the web in this
same shape, read it through the index: POST https://api.lyrenth.com/v1/aidocument with
{"url":"..."}.

## Documentation

- [API quickstart](https://www.lyrenth.com/docs/quickstart): call POST /v1/aidocument from curl, Python, TypeScript, Go, Java, C#, PHP, Ruby, or Rust.
- [API reference](https://www.lyrenth.com/docs/api): every endpoint a key can call, with request fields, response shapes, status codes, and error codes.
- [AIDocument shape](https://www.lyrenth.com/docs/aidocument): the canonical v2 JSON envelope /v1/aidocument returns.
- [MCP server](https://www.lyrenth.com/docs/mcp): the hosted endpoint at https://api.lyrenth.com/mcp, or npx -y lyrenth-mcp locally.
- [Framework integrations](https://www.lyrenth.com/docs/frameworks): LangChain and LlamaIndex packages, and tools for the Vercel AI SDK, eve, and the OpenAI SDK.
- [Documentation home](https://www.lyrenth.com/docs): the index of everything above.
- [Pricing](https://www.lyrenth.com/pricing): Free, Starter Beta, and Pro Beta limits for AIDocuments, API rate, and verified domains.

## Blog

- [Blog index](https://www.lyrenth.com/blog)
- [An independent measurement of AI search crawlers, and where Lyrenth sits](https://www.lyrenth.com/blog/independent-measurement-ai-search-crawlers): Known Agents measures which crawlers do the most AI search indexing across thousands of sites. The list is Huawei, Amazon, Apple, Anthropic, Meta, OpenAI, and, a few rows down, us.
- [The token economics of RAG over live web data](https://www.lyrenth.com/blog/rag-token-economics-live-web): Most RAG cost is not the model, it is what you feed it. A measured look at raw HTML versus a readable document, and where the money actually goes.
- [MCP or a REST API: which one should your agent use to read the web?](https://www.lyrenth.com/blog/mcp-vs-api-for-web-reading): MCP and a plain HTTP API do the same job in different places. When to expose a capability as a tool, when to call the endpoint, and why most teams run both.
- [Agentic browsers vs crawlers: which one does your agent actually need?](https://www.lyrenth.com/blog/agentic-browsers-vs-crawlers): Crawlers read the web so a question can be answered later. Agentic browsers use the web now. Picking the wrong one costs latency, tokens, and access.
- [How fresh is an indexed page? Staleness, recrawls, and forcing a refresh](https://www.lyrenth.com/blog/freshness-ai-web-index): Cached reads are fast and cheap, but pages change. How freshness works in an AI web index: documented cache ages, change-driven recrawls, and per-request refresh.
- [More than markdown: getting structure from web pages your agent can use](https://www.lyrenth.com/blog/structured-web-data-for-llms): Clean text is half the job. How AIDocument's structure, headings, links, and media, lets agents navigate, cite, and act on pages instead of just reading them.
- [The September 15 readiness checklist for agent builders](https://www.lyrenth.com/blog/september-15-agent-builder-checklist): Five checks to run on your agent's web access before Cloudflare's new crawler defaults land: inventory, identity, failure handling, freshness, and fallback.
- [Reading the web without getting blocked: why politeness beats tricks](https://www.lyrenth.com/blog/polite-web-access-for-agents): Agents get blocked for behaving like strangers. What polite machine access looks like, why evasion is a treadmill, and how identified reading stays welcome.
- [Caching web content for AI agents: what to cache, for how long, and when to force a refresh](https://www.lyrenth.com/blog/caching-web-content-for-ai-agents): A practical guide to caching the web for agents: why every agent team ends up building a cache, what staleness actually costs, and what a shared index changes.
- [Two billion documents, and why the number matters more after September 15](https://www.lyrenth.com/blog/two-billion-documents): Lyrenth's index crossed two billion AI-ready documents this week, on a public live counter. Here is what an index that size changes for agents as the web's defaults turn restrictive.
- [September 15 is coming: what actually changes for AI crawlers](https://www.lyrenth.com/blog/cloudflare-september-15-what-changes): Cloudflare's new defaults land on September 15. Here is what gets blocked, who is affected, what stays allowed, and what agent builders and site owners should do before the date.
- [One crawl serves everyone: the economics of a shared web index](https://www.lyrenth.com/blog/one-crawl-serves-everyone-economics): Why one crawl serving every reader changes the cost of machine-read web data: real token numbers, cache math, and what shared infrastructure means for origins.
- [llms.txt, explained for developers](https://www.lyrenth.com/blog/llms-txt-explained-for-developers): What llms.txt is, the exact file format, how it differs from robots.txt and sitemaps, who adopted it, and how to write one that AI assistants actually use.
- [How robots.txt applies to AI agents in 2026](https://www.lyrenth.com/blog/robots-txt-ai-agents-2026): What robots.txt actually governs in 2026, how AI crawler tokens work, recipes to allow search and block training, and why verification matters as much as rules.
- [The web is getting metered for machines: what agent builders should know before September 15](https://www.lyrenth.com/blog/metered-web-what-agent-builders-should-know): Cloudflare's new defaults block agent fetchers on ad pages and per-fetch access is getting priced. How agent builders keep reading the web through an index.
- [Cloudflare will block AI crawlers by default on September 15: what site owners should actually do](https://www.lyrenth.com/blog/cloudflare-blocks-ai-crawlers-what-site-owners-should-do): Cloudflare blocks Training and Agent crawlers by default on ad pages from September 15, 2026. What changes, who it affects, and a site-owner checklist.
- [See exactly which AI bots read your site (and which just say they do)](https://www.lyrenth.com/blog/which-ai-bots-read-your-site): User-agent strings are easy to fake. Here is how to verify which AI bots really read your site, and how AI bot traffic analytics show it without grep.
- [Crawl-on-miss: how an index reads pages it has never seen](https://www.lyrenth.com/blog/crawl-on-miss-explained): Crawl on miss, explained: how Lyrenth serves every page from its index, whether cached or freshly indexed, and when a re-crawl happens before serving.
- [LangChain document loaders for live web pages: a clean pattern](https://www.lyrenth.com/blog/langchain-live-web-pages-loader): A custom LangChain web loader that reads pages as clean AIDocuments instead of raw HTML: about 30 lines of Python, with real token numbers.
- [Lyrenth vs building your own scraper stack: the real 12-month bill](https://www.lyrenth.com/blog/diy-scraper-stack-real-cost): What web scraping infrastructure cost actually looks like over 12 months: fetch workers, browser pools, engineer time, and the shared-index alternative.
- [Is your site readable by AI? The readiness score, explained](https://www.lyrenth.com/blog/ai-readiness-score-explained): Run an AI readability website check with Lyrenth's readiness score: 0 to 10, the mean of per-page audits. What it measures and how to raise it.
- [Give your agent a web-reading tool in 30 lines](https://www.lyrenth.com/blog/web-reading-tool-in-30-lines): LLM tool use for web reading, end to end: one tool schema for OpenAI and Anthropic plus a 30-line Python handler that returns clean AIDocument Markdown.
- [We measured the token cost of famous pages](https://www.lyrenth.com/blog/token-cost-of-famous-pages): What is the token count of a web page for an LLM? We measured five famous pages: raw HTML vs AIDocument tokens, with real per-read costs.
- [What is an AI-readable web index?](https://www.lyrenth.com/blog/what-is-an-ai-readable-web-index): What an AI-readable web index is, how it differs from scraping and from a classic search index, and why one crawl should serve every machine reader.
- [Add web reading to Claude Desktop, Claude Code, or Cursor with one MCP block](https://www.lyrenth.com/blog/add-web-reading-to-claude-cursor-mcp): A five-minute quickstart: drop one MCP block into your assistant and it reads any URL as a clean AIDocument. Covers read_url, read_urls, and check_usage.
- [Agents don't browse, they read: what an index for machine readers looks like](https://www.lyrenth.com/blog/agents-dont-browse-they-read): Humans browse the web through layout and pixels; AI agents read it as text, structure, and provenance. That difference inverts how an index should be built.
- [How to feed web pages to an LLM without blowing the context window](https://www.lyrenth.com/blog/feed-web-pages-to-llm-without-blowing-context): Raw HTML burns your context window on nav, scripts, and boilerplate. Here is how to feed web pages to an LLM as clean AIDocuments, with real token numbers.
- [The publisher's guide to AI bots, robots.txt, and getting represented correctly](https://www.lyrenth.com/blog/publishers-guide-ai-bots-and-verification): Who reads your site now, what a well-behaved AI-index bot looks like, how robots.txt and llms.txt apply, and how to verify your domain with Lyrenth for free.
- [What is an AIDocument? One clean shape for agents that read the web](https://www.lyrenth.com/blog/what-is-an-aidocument): An AIDocument is one stable JSON shape for any web page: Markdown body, title, description, structure, and measured economics. Here is how it works.
- [Why your scraper gets empty HTML from JavaScript sites (and what rendering actually takes)](https://www.lyrenth.com/blog/why-scrapers-get-empty-html-from-javascript-sites): Fetch a React or SPA site and you get a hollow div, not content. Here is why JavaScript sites return empty HTML, and what real rendering takes at index scale.

## API endpoints

- [GET /v1/stats](https://api.lyrenth.com/v1/stats): live count of indexed documents (public).
- [POST /v1/aidocument](https://www.lyrenth.com/docs/quickstart): read any URL through the index as an AIDocument (Bearer auth); served from the standing index, crawled only on a miss.
- [POST /v1/aidocument/batch](https://www.lyrenth.com/docs/api): the same read for up to 20 URLs in one call, with per-URL error isolation (Bearer auth).
- [GET /v1/read?url=...](https://www.lyrenth.com/docs/api): the same read as /v1/aidocument, but a GET that returns clean Markdown. Paste-and-go for agents and shell (Bearer auth).
- [POST /v1/submit](https://www.lyrenth.com/docs/api): queue a URL for background indexing.
- [GET /v1/quota](https://www.lyrenth.com/docs/api): your plan and remaining credits (Bearer auth).
- [GET /aidocument.schema.json](https://api.lyrenth.com/aidocument.schema.json): JSON Schema (draft-07) for the v2 envelope.

## Index data

- [Web index stats](https://www.lyrenth.com/web-index/stats): aggregate signal coverage across our corpus.

## For agents

- [Integrations](https://www.lyrenth.com/docs/integrations): read the web through Lyrenth from an MCP client, a GET /v1/read, or the Python (pip install lyrenth) and TypeScript (npm install lyrenth) SDKs, with LangChain, LlamaIndex, Vercel AI SDK and eve adapters.
- [Agents doing-file](https://www.lyrenth.com/agents.md): what this service is, how an agent gets access, and exactly which calls a key can make.
- [Agent manifest (JSON)](https://www.lyrenth.com/api/agent-manifest): brand, capabilities, endpoints, document shape.
- [AIDocument JSON Schema (draft-07)](https://api.lyrenth.com/aidocument.schema.json): canonical contract for the v2 response shape served by POST /v1/aidocument.
- [Bot identification](https://www.lyrenth.com/bot): we crawl with User-Agent "AIWebIndex/2.0". RFC 9309 robots.txt is enforced (if we cannot read your robots.txt, we do not crawl you); add a Disallow rule to opt out, or allowlist to be indexed. Machine-readable TDM / AI-training reservations (TDM Reservation Protocol, robots noai/notrain) exclude a domain from corpus licensing, honored worldwide.
- [Open Graph image](https://www.lyrenth.com/opengraph-image): 1200x630 PNG, brand identity.

## For site owners

- [Verify your domain](https://www.lyrenth.com/sites/add): AI agents already read your pages. Verify and you author the canonical AIDocument every agent gets when it reads you through us, see exactly what AI extracts from each page today, and push a change to make it live in seconds. Additive, not exclusive: the labs can still read you directly, and we never ask you to make Lyrenth your only path to AI. Free, no revenue share. See the [crawler policy](https://www.lyrenth.com/crawler-policy).

## Optional

- [Long-form (every page concatenated as markdown)](https://www.lyrenth.com/llms-full.txt)
- [Sitemap (XML)](https://www.lyrenth.com/sitemap.xml)
- [robots.txt](https://www.lyrenth.com/robots.txt)