You probably landed here from a server log.
Lyrenth is a structured index of the public web. Our background crawler discovers and fetches public pages at polite, rate-limited speeds to build the index, and AI agents also resolve specific URLs on demand through POST /v1/aidocument. Either way we fetch and clean each page once, then serve the canonical AIDocument from cache. Every request honors robots.txt and per-domain rate limits (details below).
AIDocument is an open format.
Lyrenth is a universal adapter for the existing web. No website changes required. Our crawler returns the AIDocument shape, which we publish openly so anyone can implement, fork, or extend it. The open spec covers:
- User-Agent identifier (
AIWebIndex/2.0) - AIDocument JSON format for extracted content
- Verification mechanism (
.well-known/aiwebindex-verify.txtand DNS TXT records)
Lyrenth, operated by Aleksma AI Inc., is the reference commercial implementation. The format itself is open: Aleksma AI Inc. holds no patents on the core AIDocument shape or the verification mechanics, and pledges not to seek such patents. Anyone is free to implement, fork, or extend AIDocument.
Site operators allowlisting AIWebIndexvia robots.txt or firewall rules allow any compliant crawler — not just Lyrenth. This means an open ecosystem of compatible implementations can grow over time.
The full AIDocument spec will be formalized at aiwebindex.org. Today, that domain currently redirects to www.lyrenth.com while the spec site is being built.
User-Agent strings
User-Agent: AIWebIndex/2.0 (+https://www.lyrenth.com/bot; AI-readable web index)
Verification probes use a slightly different UA so you can distinguish crawl fetches from ownership checks:
User-Agent: AIWebIndex/2.0 verification (+https://www.lyrenth.com/bot)
Robots.txt, backoff, and rate limits
Our crawler implements RFC 9309 and honors Disallow rules, Crawl-delay directives, HTTP 429 / 503 backoff, Sitemap directives, and per-domain rate caps with a 2 second cooldown floor. The simplest opt-out is your robots.txt:
User-agent: AIWebIndex Disallow: /
Or use granular Disallow: /path/ rules to block specific sections. We re-fetch your robots.txt at most once per 24 hours; email us if you need an immediate cache invalidation.
Two methods. Either works.
Pick whichever propagates first. Our verifier queries 1.1.1.1, 8.8.8.8, and 9.9.9.9 in parallel so misconfigured local resolvers don't block your verification.
DNS TXT record
_aiwebindex-verify.<your-domain> TXT "aiwi-verify=<token>"
Recommended. Usually propagates in under 60 seconds.
.well-known file
https://<your-domain>/.well-known/aiwebindex-verify.txt contents: aiwi-verify=<token>
HTTPS only. Plain HTTP fetches are rejected.
Reading this with an AI agent?
We publish machine-readable manifests so you don't have to scrape this page. All three pivot to the same canonical AIDocument shape.