Crawler policy.

Who we are

Lyrenth is operated by Aleksma Ai, Inc., a Delaware corporation.

Crawler and index infrastructure are primarily hosted on Hetzner infrastructure in Frankfurt and Falkenstein, Germany.

Contact: hello@lyrenth.com

Two fetch types, one rule

Lyrenth is an index, and every fetch we make is Lyrenth's own act of building it. We fetch URLs in two ways, and both follow the same rule: every fetch honors your robots.txt. We disclose both here so your server logs always make sense.

1. Scheduled crawling. Our background indexer chooses what to fetch and when, from sitemaps, link graphs, and prior fetches. It fully implements RFC 9309: a Disallow for our user-agent means we do not crawl those paths. Everything in “What we respect” below applies.

2. On-demand index fetches. When a customer asks our API for a URL the index does not yet hold, or asks for a fresh copy, Lyrenth fetches the page itself to index it, then serves it from the index. The customer asks the index; only Lyrenth fetches, and the same rules apply, robots.txt included. A URL your robots.txt disallows for our user-agent is not fetched, not indexed, and not served through the API, on any path; the request is refused with an explanation instead. These fetches use a distinct User-Agent (see “User-Agents” below) only so your logs can tell scheduled crawling from on-demand index fills.

If a Disallow rule is in place for our user-agent, you will not see fetches for those paths from either fetch type.

We additionally throttle or refuse on-demand fetches against any origin when aggregate traffic risks disrupting it, and we maintain a domain refusal list, honored across all customers, for origins that request it.

What we do

Lyrenth builds a structured index of publicly accessible web pages and serves it to AI systems and developers. We:

• parse structured data,
• resolve pages into the canonical AIDocument format,
• and cache AIDocuments to minimize unnecessary repeated requests.

If 1,000 AI agents request the same URL, our goal is to reduce that to a minimal number of origin fetches whenever technically possible. Our cache is designed to reduce total load on your origin, not add to it.

Verify to author your canonical version

AI agents already crawl your pages and extract them however they like. Verifying your domain lets you take the wheel: you author the canonical AIDocument every agent gets when it reads you through Lyrenth, your dashboard shows exactly what AI currently makes of each page, and you can push a change so the version agents read is corrected in seconds.

Verification is also a licensing grant; we say this plainly. Verifying a domain places it in our Verified Index and grants Lyrenth the right to license its canonical content to AI labs and model companies, including for payment. Lyrenth retains the proceeds; there is no revenue share. The free owner toolkit (verification, dashboard, AI Readiness, corrections, change webhook) is the consideration for that grant. The full mechanics, including how to leave and what leaving does and does not undo, are in Terms §8.2.

This is additive, not exclusive. The labs can keep reading you directly. Agents that read through Lyrenth are served from our cache, so that demand reaches your origin as a single fetch rather than a fresh crawl per agent.

What we respect today

These commitments apply to every fetch we make, scheduled and on-demand alike.

Our crawler implements RFC 9309 (the formal robots.txt specification) and honors:

• Disallow rules inside robots.txt (section 6.1, MUST).
• Crawl-delay directives (section 6.2, MAY).
• Sitemap: directives inside robots.txt.
• HTTP 429 backoff and HTTP 503 temporary unavailability responses.
• Per-domain rate caps with a 2-second cooldown floor, raised by Crawl-delay where declared.
• Machine-readable reservations of TDM / AI-training rights: TDM Reservation Protocol meta tags (tdm-reservation, tdmrep) and robots-meta noai / noimageai / notrain / noml directives. These exclude a domain from corpus licensing entirely, worldwide.

If we cannot read your robots.txt at all and have no cached copy, we treat the site as disallowed, stricter than RFC 9309 requires. We fetch a site's robots.txt at most once per 24 hours.

We do not attempt to bypass paywalls, login requirements, authentication systems, or technical access controls, on either fetch type.

User-Agents

Our autonomous crawler identifies itself as:

On-demand index fetches identify themselves as:

A Disallow rule for AIWebIndex governs both: each User-Agent above matches the AIWebIndex token, and both honor your robots.txt on every fetch. The two strings exist only so your logs can tell scheduled crawling from on-demand index fills.

Verifying it's really us

Anyone can put our name in a User-Agent string. To confirm a request genuinely came from Lyrenth, in order of strength:

• Signature. Our requests carry a Web Bot Auth signature (RFC 9421 HTTP Message Signatures, Ed25519). Our public key directory is at https://api.lyrenth.com/.well-known/http-message-signatures-directory; the signature covers the request authority and is verifiable against that key. This is the same scheme Cloudflare's verified-bots program uses.
• IP + reverse DNS. We publish our crawler and fetcher IP ranges in machine-readable form at https://lyrenth.com/bot/ip-ranges.json, updated whenever our infrastructure changes, and every IP has forward-confirmed reverse DNS under lyrenth.com.
• Requests from outside those ranges, or without a valid signature, that claim our User-Agent are impostors; we'd appreciate a report at hello@lyrenth.com.

This also makes firewall decisions precise: allowlist or blocklist our published ranges rather than guessing at Hetzner address space.

Lyrenth is a search-style indexer, not a model trainer. We fetch public pages to build a fresh, canonical index that we serve to agents with attribution and a link back to the source; we do not train foundation models on the content we crawl, and we honor machine-readable AI-training reservations (see “What we respect” above).

Data we extract

We may extract: public HTML, visible page content, OpenGraph metadata, JSON-LD structured data, schema.org metadata, and publicly available semantic markup.

We do not intentionally extract: passwords, authenticated content, private user data, or content behind login walls.

Opt-out options

Different mechanisms do different things. Here is exactly what each one achieves:

robots.txt

Effect: stops all future fetching of the disallowed paths, scheduled and on-demand alike: a disallowed URL is not fetched, not indexed, and not served on request. Prospective only: content already in the index remains until you also request removal (below).

AI/TDM rights reservation

A TDM Reservation Protocol meta tag (tdm-reservation / tdmrep) or a robots-meta noai / notrain directive on your pages.

Effect: excludes your domain from corpus licensing entirely, worldwide, in addition to whatever crawling restrictions you declare. This is the strongest machine-readable signal we honor.

Email request

Domain owners may request a domain-wide opt-out at hello@lyrenth.com.

Effect: the strongest option overall. We stop future crawling, suppress your existing content from the API, any future search surface, and future corpus deliveries, and add your domain to the refusal list honored across every fetch type. Removal is prospective: it cannot recall datasets already delivered to a licensee or affect models already trained. See our Privacy policy for the full removal mechanics.

Firewall blocking

Block our published IP ranges (/bot/ip-ranges.json).

Effect: prevents fetches at the network level, both fetch types. Blunt but absolute.

Verified owner controls

Verified site owners may manage per-path exclusions through dashboard controls where available.

Effect: path-level control over what the index serves and what enters the Verified Index corpus, without removing the whole domain.

Abuse reporting

If you believe our crawler is ignoring robots.txt, causing excessive traffic, hammering your infrastructure, or behaving unexpectedly, contact hello@lyrenth.com. Include log excerpts with timestamps and source IPs if you can; that lets us distinguish our traffic from impostors immediately. We actively monitor abuse reports and aim to respond within 24 hours.

Operational reference

robots.txt: allow our crawler explicitly

(Your rules for other crawlers are unaffected; only add a User-agent: * block if you intend to set policy for all bots.)

Verification methods

Verified domain owners may confirm ownership through DNS TXT verification, meta tag verification, or dashboard verification workflows.