Crawler policy.

Who we are

Lyrenth is operated by Aleksma Ai, Inc., a Delaware corporation.

Contact: hello@lyrenth.com

Two fetch types, one rule

Lyrenth is an index, and every fetch we make is Lyrenth's own act of building it. We fetch URLs in two ways, and both follow the same rule: every fetch honors your robots.txt. We disclose both here so your server logs always make sense.

1. Scheduled crawling. Our background indexer chooses what to fetch and when, from sitemaps, link graphs, and prior fetches. It fully implements RFC 9309: a Disallow for our user-agent means we do not crawl those paths. Everything in “What we respect” below applies.

2. On-demand index fetches. When a customer asks our API for a URL the index does not yet hold, or asks for a fresh copy, Lyrenth fetches the page itself to index it, then serves it from the index. The customer asks the index; only Lyrenth fetches, and the same rules apply, robots.txt included. A URL your robots.txt disallows for our user-agent is not fetched, not indexed, and not served through the API, on any path; the request is refused with an explanation instead. These fetches use a distinct User-Agent (see “User-Agents” below) only so your logs can tell scheduled crawling from on-demand index fills.

If a Disallow rule is in place for our user-agent, you will not see fetches for those paths from either fetch type.

We additionally throttle or refuse on-demand fetches against any origin when aggregate traffic risks disrupting it, and we maintain a domain refusal list, honored across all customers, for origins that request it.

What we do

Lyrenth builds a structured index of publicly accessible web pages and serves it to AI systems and developers. We:

• parse structured data,
• resolve pages into the canonical AIDocument format,
• and cache AIDocuments to minimize unnecessary repeated requests.

If 1,000 AI agents request the same URL, our goal is to reduce that to a minimal number of origin fetches whenever technically possible. Our cache is designed to reduce total load on your origin, not add to it.

Verify your domain

AI agents already crawl your pages and extract them however they like. Every agent that reads you through Lyrenth gets an AIDocument built from your own page. Verifying your domain shows you how AI reads each page: your dashboard gives a readability score and the signals behind it, and, with our edge worker, which AI agents visit. We read your sitemap and crawl its pages ahead of pages found through links. When you change a page, the change webhook or the Recheck button (2 credits a page) puts it at the front of our crawl queue, so the copy agents read is fetched again from your page.

Verification is also a licensing grant; we say this plainly. Verifying a domain places it in our Verified Index and grants Lyrenth the right to license its canonical content to AI labs and model companies, including for payment. Lyrenth retains the proceeds; there is no revenue share. The free owner toolkit (verification, dashboard, AI Readiness, change webhook) is the consideration for that grant. The full mechanics, including how to leave and what leaving does and does not undo, are in Terms §8.2.

This is additive, not exclusive. The labs can keep reading you directly. Agents that read through Lyrenth are served from our cache, so that demand reaches your origin as a single fetch rather than a fresh crawl per agent.

What we respect today

These commitments apply to every fetch we make, scheduled and on-demand alike, with one exception to the rate caps, noted in the list.

Our crawler implements RFC 9309 (the formal robots.txt specification) and honors:

• Disallow rules inside robots.txt (section 6.1, MUST).
• Crawl-delay directives (section 6.2, MAY).
• Sitemap: directives inside robots.txt.
• HTTP 429 backoff and HTTP 503 temporary unavailability responses.
• Per-domain rate caps with a 2-second cooldown floor, raised by Crawl-delay where declared. The one exception is pages a verified site owner asks us to fetch again (with Recheck or the change webhook): those skip the wait and the Crawl-delay, and robots.txt still applies.
• Machine-readable reservations of TDM / AI-training rights: TDM Reservation Protocol meta tags (tdm-reservation, tdmrep) and robots-meta noai / noimageai / notrain / noml directives. These exclude a domain from corpus licensing entirely, worldwide.

If we cannot read your robots.txt at all and have no cached copy, we treat the site as disallowed, stricter than RFC 9309 requires. We keep a robots.txt we could read for 24 hours, so we fetch it about once a day. One we could not read is asked again after a minute, and one your server refused us (for example with a 403) after six hours.

We do not attempt to bypass paywalls, login requirements, authentication systems, or technical access controls, on either fetch type.

Traps and hidden links

Some sites plant links that no person can see or click, so that anything following them identifies itself as automated. A crawler that follows one gets classified as hostile for doing exactly what the page invited it to do. We treat avoiding that as our responsibility, not the site owner's.

• Links come only from a real HTML parse. We read anchors from a spec-compliant parse of the document. A URL sitting inside a script body is not a link to us, and text that merely looks like a URL is not followed.
• Links hidden from people are dropped before they are queued. If an anchor, or an ancestor a few levels above it, carries an inline display:none, visibility:hidden, or the hidden attribute, the link is removed from discovery rather than crawled.
• Known honeypot namespaces are refused outright. Paths under a reserved trap namespace, such as /.within.website/, cannot enter our crawl queue, are refused on every read path including the unauthenticated demo endpoint, and cannot be reached through a redirect. Nothing we run will fetch one.

Two limits, stated plainly because a promise is only useful if its edges are known. We examine inline styles and attributes only, so a link hidden by a class in an external stylesheet is not detected. And aria-hidden is deliberately not treated as invisibility, because component frameworks apply it to links that are perfectly visible. Both choices err toward missing a trap rather than dropping a real page.

None of this makes us perfect. A trap we have not seen before, a way of hiding a link that we do not detect, or an ordinary bug on our side can still send one of our fetches somewhere it should not have gone. We would rather say that plainly than promise something we cannot guarantee.

If it happens, tell us at hello@lyrenth.com. We treat it as our mistake to correct: we stop the fetching, remove what we already stored, and fix the cause so the same thing cannot happen again. We start on it as soon as we read the message rather than scheduling it for later. The same address is the right one if you operate a trap namespace we do not yet know about, and we will add it.

User-Agents

Our autonomous crawler identifies itself as:

On-demand index fetches identify themselves as:

A Disallow rule for AIWebIndex governs both: each User-Agent above matches the AIWebIndex token, and both honor your robots.txt on every fetch. The two strings exist only so your logs can tell scheduled crawling from on-demand index fills.

Verifying it's really us

Anyone can put our name in a User-Agent string. To confirm a request genuinely came from Lyrenth, in order of strength:

• Signature. Our requests carry a Web Bot Auth signature (RFC 9421 HTTP Message Signatures, Ed25519). Our public key directory is at https://api.lyrenth.com/.well-known/http-message-signatures-directory; the signature covers the request authority and is verifiable against that key. This is the same scheme Cloudflare's verified-bots program uses. Cloudflare has verified both of our user agents as Search bots.
• IP + reverse DNS. We publish our crawler and fetcher IP ranges in machine-readable form at https://lyrenth.com/bot/ip-ranges.json, updated whenever our infrastructure changes, and every IP has forward-confirmed reverse DNS under lyrenth.com.
• Requests from outside those ranges, or without a valid signature, that claim our User-Agent are impostors; we'd appreciate a report at hello@lyrenth.com.

This also makes firewall decisions precise: allowlist or blocklist our published ranges rather than guessing.

Lyrenth is a search-style indexer, not a model trainer. We fetch public pages to build a fresh, canonical index that we serve to agents with attribution and a link back to the source; we do not train foundation models on the content we crawl, and we honor machine-readable AI-training reservations (see “What we respect” above).

Data we extract

We may extract: public HTML, visible page content, OpenGraph metadata, JSON-LD structured data, schema.org metadata, and publicly available semantic markup.

We do not intentionally extract: passwords, authenticated content, private user data, or content behind login walls.

Opt-out options

Different mechanisms do different things. Here is exactly what each one achieves:

robots.txt

Effect: stops all future fetching of the disallowed paths, scheduled and on-demand alike: a disallowed URL is not fetched, not indexed, and not served on request. Prospective only: content already in the index remains until you also request removal (below).

AI/TDM rights reservation

A TDM Reservation Protocol meta tag (tdm-reservation / tdmrep) or a robots-meta noai / notrain directive on your pages.

Effect: excludes your domain from corpus licensing entirely, worldwide, in addition to whatever crawling restrictions you declare. This is the strongest machine-readable signal we honor.

Email request

Domain owners may request a domain-wide opt-out at hello@lyrenth.com.

Effect: the strongest option overall. We stop future crawling, suppress your existing content from the API, any future search surface, and future corpus deliveries, and add your domain to the refusal list honored across every fetch type. Removal is prospective: it cannot recall datasets already delivered to a licensee or affect models already trained. See our Privacy policy for the full removal mechanics.

Firewall blocking

Block our published IP ranges (/bot/ip-ranges.json).

Effect: prevents fetches at the network level, both fetch types. Blunt but absolute.

Verified owners

Verified site owners exclude paths the same way as every other site: with a Disallow rule for AIWebIndex in robots.txt, which every fetch honors.

Effect: the same as robots.txt above, for the paths you disallow.

Abuse reporting

If you believe our crawler is ignoring robots.txt, causing excessive traffic, hammering your infrastructure, or behaving unexpectedly, contact hello@lyrenth.com. Include log excerpts with timestamps and source IPs if you can; that lets us distinguish our traffic from impostors immediately. We actively monitor abuse reports and aim to respond within 24 hours.

Operational reference

robots.txt: allow our crawler explicitly

(Your rules for other crawlers are unaffected; only add a User-agent: * block if you intend to set policy for all bots.)

Verification methods

Domain owners confirm ownership in one of two ways, and either one is enough: a DNS TXT record at _aiwebindex-verify.<your-domain> with the value aiwi-verify=<token>, or a file at https://<your-domain>/.well-known/aiwebindex-verify.txt that contains the same value. You get the token when you add the domain in the dashboard.