llms.txt and Markdown Versions: The Spec, Who Uses It, and Implementing It in ASP.NET Core

Version note: This post is based on the v2 state of the llms.txt proposal at llmstxt.org (August 2026); the code examples were built and run with .NET 10, EF (Entity Framework) Core 10.0.14, ReverseMarkdown 6.3.0 and Lighthouse 13.6.0.

Guess First

You've decided to publish a /llms.txt for the Bookstore blog. You have three questions. First, does this file help you get into Google's AI (Artificial Intelligence) results, that is, AI Overviews? Second, should the links in llms.txt point to HTML (HyperText Markup Language) pages or to something else? Third, when you convert a post to Markdown, what happens to a <pre><code> block in it and to a relative link like /books/async-in-practice?

What llms.txt Is and Isn't

llms.txt is a format Jeremy Howard proposed on September 3, 2024: a Markdown file placed at the site root, written so LLMs (Large Language Model) and the tools that use them can understand the site quickly. The proposal's page was last updated on August 10, 2026 and describes itself as v2. It's a community proposal, not a spec approved by a standards body.

It shouldn't be confused with robots.txt. robots.txt tells bots what they may crawl; llms.txt carries no permission or prohibition. It's a content map and a summary. The place for a "don't use me in AI training" decision is the robots.txt tokens we saw in the third post of the series, not llms.txt.

Spec v2: Structure and Discovery

The file's structure is ordered and short. An optional BOM (Byte Order Mark), then an H1 (a first-level heading, # in Markdown) with the project or site name; this is the only required element in the file. Then a blockquote with a short summary, zero or more description sections without headings, and zero or more H2 (second-level heading, ##) sections containing file lists. Each list item carries a required [name](url) link and optionally notes after a :. The H2 named "Optional" has a special meaning: secondary links that can be skipped when a shorter context is needed.

The proposal also recommends that every page have a Markdown version at the same URL (Uniform Resource Locator): page.html.md or page.md; for URLs without a file name, index.html.md. So the answer to the second opening question: links in llms.txt pointing to the Markdown versions fits the spec's intent better.

v2 defines two relations for discovery: rel="alternate" with type="text/markdown" points to the page's Markdown version, and rel="describedby" points to the llms.txt file that covers the page. Both can be given as an HTML <link> element or an HTTP (Hypertext Transfer Protocol) Link header; the header method also works for non-HTML files and can be set at the server or CDN (Content Delivery Network) level. An llms.txt covers all pages under the path where it lives.

The Honest Status: Who Uses It?

On the Google side the answer is clear. Google's AI optimization guide says you don't need new machine-readable files, AI text files or Markdown to appear in Google Search; the note added on June 15, 2026 says creating llms.txt is "completely fine", but because Google Search ignores these files it neither helps nor harms visibility. That's the answer to the first opening question: it doesn't help you get into AI Overviews.

For the other major AI search engines, as of November 2026 I found no official statement in OpenAI's, Anthropic's or Perplexity's crawler documentation that llms.txt is used. A bot requesting /llms.txt in your server log doesn't prove the file is used in an answer either; it only shows it was downloaded.

The format's real use is elsewhere: developer documentation, coding assistants and agents. Giving a coding assistant a library's llms.txt offers it a clean content map without wrestling with HTML. And there's Google's own tool: the Agentic Browsing category in Lighthouse 13.6.0 includes an llms-txt audit. According to the audit's source code, if the file returns 4xx the audit counts as "not applicable", and if it returns 5xx or 429 it fails; if the file exists it looks for at least one H1, at least one Markdown link and at least 50 characters. We'll open the whole category in the sixth post of the series.

An llms.txt Endpoint in ASP.NET Core

The Bookstore blog's posts are in a database via EF Core. Generating llms.txt from the same data instead of writing it by hand means the maintenance takes care of itself:

app.MapGet("/llms.txt", async (BlogDbContext db, CancellationToken ct) =>
{
    List<BlogPost> posts = await db.Posts.AsNoTracking()
        .OrderBy(p => p.Series).ThenByDescending(p => p.PublishedAt)
        .ToListAsync(ct);

    StringBuilder sb = new();
    sb.AppendLine("# Bookstore Blog");
    sb.AppendLine();
    sb.AppendLine("> Hands-on posts about the books we sell: tested code, measured results, dated version notes.");
    foreach (IGrouping<string, BlogPost> series in posts.GroupBy(p => p.Series))
    {
        sb.AppendLine();
        sb.AppendLine($"## {series.Key}");
        sb.AppendLine();
        foreach (BlogPost post in series)
        {
            sb.AppendLine($"- [{post.Title}]({SiteUrl}/blog/{post.Slug}.md): {post.Summary}");
        }
    }
    sb.AppendLine();
    sb.AppendLine("## Optional");
    sb.AppendLine();
    sb.AppendLine($"- [Book catalog]({SiteUrl}/): every book we sell, as HTML");
    return Results.Text(sb.ToString(), "text/plain; charset=utf-8");
}).CacheOutput(policy => policy.Tag("blog"));

Real output:

# Bookstore Blog

> Hands-on posts about the books we sell: tested code, measured results, dated version notes.

## Book Reviews

- [Review: Testing Without Fear](https://bookstore.example/blog/testing-without-fear-review.md): What the book gets right about contract tests, and the two chapters we skipped.
- [Review: Modeling Bounded Contexts](https://bookstore.example/blog/modeling-bounded-contexts-review.md): A practical book on splitting a domain, read by a team that actually split one.

## Field Notes

- [Backpressure in Our Order Service](https://bookstore.example/blog/backpressure-in-our-order-service.md): We applied the bounded channel example from Async in Practice to our order service.

## Optional

- [Book catalog](https://bookstore.example/): every book we sell, as HTML

Posts are grouped by series under H2s, links go to the Markdown versions, secondary content sits under "Optional". I chose text/plain as the content type; it stays readable in the browser, and the file is valid Markdown as plain text anyway.

A Markdown Endpoint per Post

The Markdown version needs a library to convert the HTML body; I used ReverseMarkdown 6.3.0. ASP.NET Core routing supports a complex segment like {slug}.md, so the HTML and Markdown versions of the same slug sit side by side:

private static readonly Converter HtmlToMarkdown = new(new Config
{
    GithubFlavored = true,
    Tags = { Unknown = Config.UnknownTagsOption.Bypass },
});

app.MapGet("/blog/{slug}.md", async (string slug, BlogDbContext db, CancellationToken ct) =>
{
    BlogPost? post = await db.Posts.AsNoTracking().FirstOrDefaultAsync(p => p.Slug == slug, ct);
    if (post is null)
    {
        return Results.NotFound();
    }

    DateTime lastChange = post.UpdatedAt ?? post.PublishedAt;
    // relative links mean nothing outside the site, so make them absolute first
    string html = post.BodyHtml.Replace("href=\"/", $"href=\"{SiteUrl}/");
    string markdown = $"""
        # {post.Title}

        > {post.Summary}

        Published: {post.PublishedAt:yyyy-MM-dd} · Updated: {lastChange:yyyy-MM-dd} · Canonical: {SiteUrl}/blog/{post.Slug}

        {HtmlToMarkdown.Convert(html)}
        """;
    return Results.Text(markdown, "text/markdown; charset=utf-8");
}).CacheOutput(policy => policy.Tag("blog"));

The post's HTML body has a <pre><code> block with class="language-csharp", encoded characters such as <Order> inside it, and a relative link. The real Markdown output:

# Backpressure in Our Order Service

> We applied the bounded channel example from Async in Practice to our order service.

Published: 2026-10-02 · Updated: 2026-10-20 · Canonical: https://bookstore.example/blog/backpressure-in-our-order-service

## The bounded channel

We capped the queue at `100` items so producers wait instead of piling up orders:

```csharp
Channel<Order> orders = Channel.CreateBounded<Order>(
    new BoundedChannelOptions(100) { FullMode = BoundedChannelFullMode.Wait });
```

Read the [book page](https://bookstore.example/books/async-in-practice) for the full chapter list.

The answer to the third opening question: with the GithubFlavored setting, the code block becomes a fenced code block with its language, the encoded < and > characters are decoded, and indentation is preserved. The relative link, however, doesn't become absolute on its own; in my first attempt it stayed as (/books/async-in-practice). Since a relative link is useless in a file read outside the site, I made it absolute before the conversion. The date and canonical line is deliberate too: the Markdown version is the only place where a reader or a tool can see how current the content is and its real address.

Discovery: the Link Header and Output Cache

The HTML page announces the Markdown version and llms.txt with the v2 relations:

response.Headers.Link =
    $"</blog/{slug}.md>; rel=\"alternate\"; type=\"text/markdown\", </llms.txt>; rel=\"describedby\"";

$ curl -si http://localhost:5481/blog/backpressure-in-our-order-service | grep -i "^link"
Link: </blog/backpressure-in-our-order-service.md>; rel="alternate"; type="text/markdown", </llms.txt>; rel="describedby"

Both endpoints are in Output Cache with the "blog" tag. We covered Output Cache's mechanics in the "What's New in ASP.NET Core with .NET 7" post of the ASP.NET Core Web API (Application Programming Interface) series; what matters here is invalidation. When the CMS (Content Management System) publishes or edits a post, it calls an endpoint that clears the tag:

// called by the CMS after a post is published or edited
app.MapPost("/internal/blog-cache/evict", async (IOutputCacheStore cache, CancellationToken ct) =>
{
    await cache.EvictByTagAsync("blog", ct);
    return Results.NoContent();
});

$ curl -s -D - -o /dev/null http://localhost:5481/llms.txt
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
$ curl -s -D - -o /dev/null http://localhost:5481/llms.txt
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Age: 2
$ curl -s -o /dev/null -w "evict: %{http_code}\n" -X POST http://localhost:5481/internal/blog-cache/evict
evict: 204
$ curl -s -D - -o /dev/null http://localhost:5481/llms.txt
HTTP/1.1 200 OK

On the second request the Age: 2 header shows the response came from the cache; the request after the evict is generated again. In a real app this internal endpoint must be behind authentication; I left it open here for brevity.

As a final check I ran Lighthouse 13.6.0 against the local page with only this audit: npx lighthouse http://localhost:5481/blog/backpressure-in-our-order-service --only-audits=llms-txt. Result: llms-txt, "llms.txt follows recommendations", score 1.

mertozen.com's llms.txt: What We Did, What We Can't Measure

This site already publishes /llms.txt and /llms-full.txt; both are generated from CMS data in SeoController and cached for an hour. According to the source code, llms.txt has an H1, a blockquote summary, the 10 newest posts, the 5 newest videos, the About and Contact pages and the list of developer tools. llms-full.txt lists all posts in a category and series hierarchy, in reading order. The structure follows the spec and meets the three conditions Lighthouse looks for.

There's also room for improvement against spec v2. The links go to HTML pages; posts have no Markdown version. Only TR addresses are listed; there's no separate section or separate /en/llms.txt for the English posts under /en/. Page heads use <link rel="llms-txt">; that value isn't defined in the spec, and v2's equivalent is rel="describedby". In the same place, rel="alternate" type="text/markdown" points to llms-full.txt; in the spec that relation is defined for the page's own Markdown version, not for a site-wide file.

And the honest part: we can't measure what these files contribute to the site appearing in any AI answer. Google Search says it doesn't use them, the other major engines don't say they do, and a download in the log doesn't prove use. Today the file's value is on the side of coding assistants and agents, and that use leaves no measurable traffic either.

Production Notes

Automate the maintenance or don't start. An llms.txt updated by hand goes stale within a few months; it should be generated from the same data as the content, in the same deploy.

Think about Markdown versions against duplicate content risk. Google's note says Google can crawl many file types but doesn't treat them specially. Putting canonical information in the Markdown endpoint and, if needed, adding X-Robots-Tag: noindex keeps these versions from competing with the HTML page in search results; details in the "noindex and X-Robots-Tag" post of the Technical SEO (Search Engine Optimization) series.

Keep the file short and ordered by importance. That's exactly what the spec's "Optional" section is for: the links to drop first when a tool has to shrink its context should live there. Put the most important series and pages first instead of dumping the whole archive into one file.

When not to bother: if you run a content site and can't measure the cost, llms.txt ranks behind indexing problems in Search Console, slow pages and weak content quality.

Common Pitfalls

Treating llms.txt as an alternative to robots.txt. It carries no permission or prohibition.

Thinking llms.txt is a Google ranking factor. Google states explicitly that it ignores it.

Not testing code blocks and links in the Markdown conversion. Encoded characters, language info and relative links are where it breaks most often.

Dumping the whole site into one giant file. The spec suggests a short map and a structure that can be shortened with "Optional"; per-page Markdown versions exist for the full text.

A Small Experiment

★☆☆ If your site has an llms.txt, check whether its H1, blockquote and links follow the spec; if not, write a three-line draft.

★★☆ Add a post containing an HTML table to the Markdown endpoint and see how ReverseMarkdown converts it. What happens to a table without a header row?

★★★ Add language support to Bookstore: /en/llms.txt lists English posts, /llms.txt lists Turkish posts; every HTML page's Link header points to the llms.txt in its own language as describedby. Verify both languages with curl.

In the next post we look at the bigger picture llms.txt is part of: what a browser agent reading your page sees, why the accessibility tree matters, and what Lighthouse's Agentic Browsing category checks.