The crawler behind the Lyrenth index.
AIWebIndex/2.0 is the crawler that builds Lyrenth, the AI-readable web index: a search index of the public web that serves every page as one clean, attributed AIDocument. It is an indexer, not a model trainer: we do not train foundation models on the content we crawl. It identifies itself honestly and is verifiable three ways on this page, honors RFC 9309 robots.txt including Crawl-delay, keeps per-domain rate limits, and fails closed when robots.txt cannot be read. Agents read the resulting AIDocuments through POST /v1/aidocument.
AIDocument is
an open format.
The index is a universal adapter for the existing web, no website changes required. Our crawler returns the AIDocument shape, which we publish openly so anyone can implement, fork, or extend it.
- ›User-Agent identifier (
AIWebIndex/2.0) - ›AIDocument JSON format for extracted content
- ›Verification (
.well-known/aiwebindex-verify.txtand DNS TXT)
Lyrenth, operated by Aleksma Ai, Inc., is the reference commercial implementation. The format itself is open: Aleksma Ai, Inc. holds no patents on the core AIDocument shape or the verification mechanics, and pledges not to seek such patents. Anyone is free to implement, fork, or extend AIDocument.
User-Agent strings
User-initiated API fetches (a customer submitted that exact URL) identify separately so your logs can tell the two apart:
Verification probes use a slightly different UA so you can distinguish crawl fetches from ownership checks:
UA strings can be spoofed. Confirm a request is genuinely ours three ways:
- 1Source IP is on our published list at /bot/ip-ranges.json.
- 2Requests carry a Web Bot Auth signature (RFC 9421); our public key directory is at api.lyrenth.com/.well-known/http-message-signatures-directory.
- 3Every IP has forward-confirmed reverse DNS under
lyrenth.com.
robots.txt, backoff, rate limits
Our crawler implements RFC 9309 and honors Disallow rules, Crawl-delay, HTTP 429/503 backoff, Sitemap directives, and per-domain rate caps with a 2-second cooldown floor. The simplest opt-out is your robots.txt:
Prefer the opposite? Verify your domain and you author the canonical AIDocument agents receive. Verify your domain →
Two methods.
Either works.
Pick whichever propagates first. Our verifier queries 1.1.1.1, 8.8.8.8, and 9.9.9.9 in parallel so misconfigured local resolvers do not block your verification.
DNS TXT record
Recommended. Usually propagates in under 60 seconds.
.well-known file
HTTPS only. Plain HTTP fetches are rejected.
Reading this with
an AI agent?
We publish machine-readable manifests so you do not have to scrape this page. Each one pivots to the same canonical AIDocument shape.
AIDocument
The full JSON envelope, every field and error code, documented.
Read the contract →