llms.txt: A Practical Guide to the Emerging Convention

llms.txt is an emerging proposal for placing a concise, Markdown-formatted guide at /llms.txt. Its goal is to help language models and agents find the most useful information on a website at inference time.

That description is intentionally narrower than many articles about the convention. llms.txt is not a replacement for robots.txt, not an access-control system, and not a guaranteed way to influence training, indexing, attribution, or citations.

What the proposal actually defines

The primary llms.txt proposal describes a Markdown document with a simple order:

1. An H1 containing the project or site name. This is the only required element.
2. An optional blockquote with a short summary.
3. Optional explanatory paragraphs or lists.
4. Optional H2 sections containing curated links, each with a useful description.
5. An optional section literally named Optional for secondary resources that can be skipped when shorter context is needed.

A compact example looks like this:

# Example Outdoor

> Product guides, fit information, and care instructions for trail equipment.

Use product pages for current specifications. Use policy pages for current shipping and return information.

## Product guides

- [Trail running shoes](https://example.com/guides/trail-shoes): Choosing tread, fit, and cushioning
- [Pack fitting](https://example.com/guides/pack-fit): How to size and adjust a hiking pack

## Policies

- [Shipping](https://example.com/shipping): Current destinations and delivery guidance

## Optional

- [Company history](https://example.com/about): Background on the organization

The important pattern is curation: a short explanation plus links that help a machine reader reach authoritative source material.

What llms.txt does not do

The distinction matters because incorrect expectations lead to incorrect product and policy decisions.

It does not replace robots.txt

robots.txt communicates crawl permissions to user agents that choose to honor it. llms.txt provides context and navigation. If you need to allow or disallow crawling, keep that policy in robots.txt and in the controls offered by the relevant crawler operators. The standardized behavior of robots.txt is defined by RFC 9309.

It does not enforce training policy

Adding prose about training preferences to a file does not, by itself, create a technical enforcement mechanism. Legal rights, crawler controls, contracts, and platform policy remain separate questions.

It does not require attribution

The proposal does not define a field that forces an AI product to use a particular citation format. A consuming system may ignore the file entirely, use only some links, or present information without a source link.

It does not prove adoption

Publishing the file proves that your site offers it. It does not prove that a specific model, agent, or crawler has fetched or used it. Server and edge logs are still the appropriate place to observe requests.

Where llms.txt can be useful

The convention is most useful when a site has many possible entry points and a small number of resources are especially authoritative.

Good candidates include:

• documentation sites with a clear getting-started path;
• e-commerce sites with product guides and frequently changing policy pages;
• publishers with topic hubs and canonical explainers;
• organizations with public reports, reference material, or support documentation.

The file should be short enough to scan, specific enough to route a reader, and maintained alongside the content it describes.

A practical implementation checklist

1. Start with the questions your site should answer

List the topics where your organization has authoritative, current information. Avoid turning the file into a full sitemap.

2. Link to canonical source pages

Choose stable URLs. If a policy changes frequently, link to the live policy rather than copying details that will drift.

3. Describe each link

The description should explain what a reader will find and when to use it. Labels such as Documentation or Products are less useful than a sentence about scope.

4. Keep permissions elsewhere

Review robots.txt, authentication, and crawler-specific controls separately. Do not assume that text in llms.txt blocks access.

5. Verify and maintain the route

Confirm that /llms.txt returns Markdown or plain text successfully, does not redirect unexpectedly, and contains no private URLs. Revisit it when site structure changes.

6. Measure requests honestly

If your edge or server logs show a request to /llms.txt, you know the file was fetched. You still do not know how the requester used it unless that system provides additional evidence.

How SigilEdge handles llms.txt

SigilEdge can serve tenant-configured content at the domain’s /llms.txt path. That keeps the guide close to the same onboarding and configuration workflow used for crawler-facing enrichment.

We describe it as an emerging convention because that is what it is. The product does not present the file as a citation switch, a training opt-out, or a substitute for established crawl controls.

Sources and review method

Black Magic Consulting, LLC reviewed this guide against the primary proposal and the standardized robots exclusion protocol. Claims about adoption are deliberately excluded because publishing or observing a request does not prove downstream use.

• The primary llms.txt proposal
• RFC 9309: Robots Exclusion Protocol
• Introduction to robots.txt — Google Search Central
• AI features and your website — Google Search Central

The bottom line

Use llms.txt as a concise, machine-readable guide to the most useful parts of your website. Keep it grounded, curated, and current. Use other mechanisms for access policy, and measure only what your infrastructure can actually observe.

---

Want to manage llms.txt alongside structured, grounded crawler-facing content? Explore the SigilEdge developer surface or apply for early access.