XML Sitemaps and IndexNow: Generating Dynamic Sitemaps in ASP.NET Core and Announcing Changes

Version note: The code in this post was built and run with .NET 10 (ASP.NET Core 10), EF (Entity Framework) Core 10.0.14 (Microsoft.EntityFrameworkCore.Sqlite) and Angular 22.2; protocol details are taken from sitemaps.org, Google Search Central and indexnow.org as of November 2026.

Guess First

Bookstore's catalog has 120,000 books; 108,000 of them are live, the rest are drafts. An intern wrote the first sitemap: every URL (Uniform Resource Locator) has <changefreq>daily</changefreq>, <priority>1.0</priority> and, as <lastmod>, the moment the sitemap was generated. After every publish it also sends a request to Google's sitemap ping address.

Three questions. Will Google crawl the book pages every day because of this sitemap? Do 108,000 URLs fit in a single file? And what does the ping request achieve today? The answers are in the protocol itself and in Google's sitemap documentation.

In the previous two posts of the series we looked at tools that stop the crawler and remove pages from the index. A sitemap works in the opposite direction: it says "these URLs exist and changed on this date". In this post we'll generate an XML (Extensible Markup Language) sitemap in ASP.NET Core by streaming from EF Core, measure it, and set up IndexNow to announce changes instantly.

The Protocol: Limits and the Fields Google Reads

The sitemap protocol is defined at sitemaps.org. The basic rules:

• A sitemap file can contain at most 50,000 URLs and be at most 50 MB uncompressed. Larger sites split their files and list them in a sitemap index file.
• The file is encoded as UTF-8 (Unicode Transformation Format, 8-bit); URLs are written as absolute URLs including protocol and host. Google crawls URLs exactly as written.
• Unless submitted through Search Console, a sitemap's scope is the folder it lives in and below. That's why Google recommends keeping the file at the root.
• The file can also be gzip-compressed and served as .xml.gz; the 50 MB limit applies to the uncompressed size.

The really interesting part is what each field means to Google:

The answer to the first opening question is right here: changefreq and priority mean nothing to Google; writing them is harmless but wasted bytes. Google also states plainly that submitting a sitemap is only a hint, with no guarantee that the file will be downloaded or every URL crawled. What Bing and other engines do with these fields is a matter for their own documentation; since one sitemap serves all of them, writing fields that are valid in the protocol isn't forbidden, just don't expect an effect from Google.

Using lastmod Honestly

Google's lastmod rule: the value is used if it's "consistently and verifiably" accurate; verification means comparing it with the page's actual last modification. A significant change to the main content, structured data or links counts; updating the copyright year in the footer doesn't. The intern's sitemap writes the generation time on every URL. When Google sees over a few crawls that these dates don't match real changes on the pages, it stops trusting the site's lastmod values; the real loss is that the signal from pages that genuinely changed gets lost in the noise.

This has a counterpart in the data model. Most tables have an UpdatedAt column that is also updated when the stock count, a view counter or an internal flag changes. None of those change the page's content. At Bookstore we keep a separate field for the sitemap:

public sealed class Book
{
    public int Id { get; set; }
    public required string Slug { get; set; }
    public required string Title { get; set; }
    public bool IsPublished { get; set; }
    public DateTime PublishedAt { get; set; }
    public DateTime? ContentUpdatedAt { get; set; } // only real content changes
}

lastmod = ContentUpdatedAt ?? PublishedAt. When the book's description, title or author information changes, ContentUpdatedAt is updated; when the stock changes, it isn't. The date format is W3C (World Wide Web Consortium) Datetime; day precision (2026-11-08) is enough.

A Streaming Sitemap in ASP.NET Core

Building a sitemap for 108,000 URLs as an XDocument in memory and turning it into a string works, but every request means tens of MB of temporary memory and pressure on the large object heap. An XmlWriter that writes straight to the response stream and an IAsyncEnumerable that reads from EF Core row by row remove that cost:

public static class SitemapWriter
{
    public const int UrlsPerFile = 50_000;
    private const string Ns = "http://www.sitemaps.org/schemas/sitemap/0.9";
    private const string Xhtml = "http://www.w3.org/1999/xhtml";
    private const string Site = "https://bookstore.example";

    // Async for non-blocking writes, no BOM at the start of the file
    private static readonly XmlWriterSettings Settings = new() { Async = true, Encoding = new UTF8Encoding(false) };

    public static async Task WriteBooksAsync(Stream output, BookstoreDb db, int part, CancellationToken ct)
    {
        IAsyncEnumerable<(string Slug, DateTime LastMod)> rows = db.Books
            .AsNoTracking()
            .Where(b => b.IsPublished)
            .OrderBy(b => b.Id)
            .Skip((part - 1) * UrlsPerFile)
            .Take(UrlsPerFile)
            .Select(b => ValueTuple.Create(b.Slug, b.ContentUpdatedAt ?? b.PublishedAt))
            .AsAsyncEnumerable();

        await using XmlWriter xml = XmlWriter.Create(output, Settings);
        await xml.WriteStartDocumentAsync();
        await xml.WriteStartElementAsync(null, "urlset", Ns);
        await xml.WriteAttributeStringAsync("xmlns", "xhtml", null, Xhtml);

        int written = 0;
        await foreach ((string slug, DateTime lastMod) in rows.WithCancellation(ct))
        {
            string tr = $"{Site}/books/{slug}";
            string en = $"{Site}/en/books/{slug}";
            await xml.WriteStartElementAsync(null, "url", Ns);
            await xml.WriteElementStringAsync(null, "loc", Ns, tr);
            await xml.WriteElementStringAsync(null, "lastmod", Ns, lastMod.ToString("yyyy-MM-dd"));
            await WriteAlternateAsync(xml, "tr", tr);
            await WriteAlternateAsync(xml, "en", en);
            await WriteAlternateAsync(xml, "x-default", tr);
            await xml.WriteEndElementAsync();

            if (++written % 1000 == 0)
                await xml.FlushAsync(); // send to the client in chunks, do not buffer 10 MB
        }
        await xml.WriteEndElementAsync();
    }
}

Three details. First, ASP.NET Core forbids synchronous writes to the response body by default (AllowSynchronousIO is off), so every write call on the XmlWriter is async. Second, if Encoding isn't set, XmlWriter puts a BOM (Byte Order Mark) at the start of the file; on my first try the index file started with an invisible U+FEFF character. Most parsers tolerate it, but there's no need for it. Third, the xhtml:link lines in every URL carry hreflang information in the sitemap, because Bookstore has Turkish and English versions. The rules of hreflang are in the tenth post of the series; here we'll only look at its effect on file size.

The index file computes the number of parts from the count of live books, and the endpoints are bound to an output cache policy:

builder.Services.AddOutputCache(o =>
    o.AddPolicy("sitemap", p => p.Expire(TimeSpan.FromHours(1)).Tag("sitemap")));

app.MapGet("/sitemap.xml", async (HttpContext ctx, BookstoreDb db) =>
{
    ctx.Response.ContentType = "application/xml; charset=utf-8";
    await SitemapWriter.WriteIndexAsync(ctx.Response.Body, db, ctx.RequestAborted);
}).CacheOutput("sitemap");

app.MapGet("/sitemaps/books-{part:int:min(1)}.xml", async (int part, HttpContext ctx, BookstoreDb db) =>
{
    ctx.Response.ContentType = "application/xml; charset=utf-8";
    await SitemapWriter.WriteBooksAsync(ctx.Response.Body, db, part, ctx.RequestAborted);
}).CacheOutput("sitemap");

I seeded a SQLite database with 120,000 books (ten percent drafts) and sent the requests:

$ curl -s http://localhost:20105/sitemap.xml
<?xml version="1.0" encoding="utf-8"?><sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>https://bookstore.example/sitemaps/books-1.xml</loc></sitemap><sitemap><loc>https://bookstore.example/sitemaps/books-2.xml</loc></sitemap><sitemap><loc>https://bookstore.example/sitemaps/books-3.xml</loc></sitemap></sitemapindex>

books-1: 200 19210154 bytes 0.369588s
books-2: 200 19290150 bytes 0.284150s
books-3: 200 3112150 bytes 0.047874s
books-1 again: 19210154 bytes 0.011258s

books-1 urls:    50000
books-2 urls:    50000
books-3 urls:     8000

$ gzip -c books-1.xml | wc -c
  764545

The answer to the second opening question: 108,000 URLs don't fit in one file; they're split into three. The size side is instructive too: 50,000 URLs with hreflang take 19.2 MB. That's about 384 bytes per URL, most of it the three xhtml:link lines. Far from the 50 MB limit, but with long URLs and more languages you can hit the size limit before 50,000 URLs; in that case lower the number of URLs per file. gzip shrinks the same file to 765 KB; for large sitemaps, enabling response compression or serving .xml.gz cuts bandwidth significantly. The timings are single measurements on this machine (a Mac with Apple's M5 (Apple Silicon) chip, local SQLite file): 0.37 seconds for the first request, 0.011 seconds for the second one served from the output cache.

Output Cache and Invalidation on Publish

There's no need to generate the sitemap from the database on every request; but a one-hour cache means a newly published book enters the sitemap an hour late. With a tagged output cache we get both: the publish endpoint evicts the sitemap tag.

app.MapPost("/admin/books/{id:int}/publish", async (int id, BookstoreDb db, IOutputCacheStore cache,
    IndexNowClient client, CancellationToken ct) =>
{
    Book? book = await db.Books.FindAsync([id], ct);
    if (book is null) return Results.NotFound();

    book.IsPublished = true;
    book.PublishedAt = DateTime.UtcNow;
    await db.SaveChangesAsync(ct);

    await cache.EvictByTagAsync("sitemap", ct); // next sitemap request is fresh
    await client.SubmitAsync([$"https://bookstore.example/books/{book.Slug}"], ct);
    return Results.NoContent();
});

$ curl -s -X POST -o /dev/null -w "publish: %{http_code}\n" http://localhost:20105/admin/books/10/publish
publish: 204
books-1 after publish: 19210142 bytes 0.348453s
<url><loc>https://bookstore.example/books/book-10</loc><lastmod>2026-11-08</lastmod>

The request after publishing bypassed the cache, was regenerated in 0.35 seconds, and book-10 is in the file with today's date. The shrinking file size should catch your eye: a new URL was added, yet the file got 12 bytes smaller. That's because the parts are split with Skip/Take; every book that gets inserted pushes a book at the boundary into the next file. So every publish changes the contents of all three files. A more stable alternative is to split the parts by Id range (Ids 1 to 50,000 in the first file, and so on): files may contain fewer than 50,000 URLs, but a new book only changes the last file.

You also need to know two default limits of the output cache. The default OutputCacheOptions.MaximumBodySize is 64 MB; larger responses are never cached. The default SizeLimit is 100 MB; the three sitemap parts alone take 41.6 MB, and together with other endpoints sharing the same cache it's easy to approach that limit. The mechanics of output caching are in the ASP.NET Core Web API (Application Programming Interface) series post "What's New in ASP.NET Core with .NET 7: MapGroup, TypedResults, Rate Limiting and Output Caching".

IndexNow: Announcing Changes Instantly

The answer to the third opening question: Google's sitemap ping endpoint was deprecated in 2023; sending requests to that address means nothing today. Google recommends submitting the sitemap through Search Console, the Sitemap line in robots.txt and the Search Console API; after a change, Google decides for itself when to read the sitemap again.

IndexNow takes a different approach: the site POSTs changed URLs directly to one of the participating search engines. The protocol's site lists Microsoft Bing, Naver, Seznam.cz, Yandex and Yep as participants; URLs sent to one are shared with the other participants. Google isn't on that list. So IndexNow is useful for Bing and Yandex traffic, and doesn't replace the sitemap for Google.

The protocol has two parts. A key for ownership verification: a value of 8 to 128 characters containing letters, digits and hyphens, published at the site root in a file named {key}.txt that contains only the key. The notification is a POST with a JSON (JavaScript Object Notation) body; at most 10,000 URLs per request:

public sealed record IndexNowOptions(string Endpoint, string Host, string Key);

public sealed class IndexNowClient(HttpClient http, IndexNowOptions options, ILogger<IndexNowClient> logger)
{
    public async Task SubmitAsync(IReadOnlyList<string> urls, CancellationToken ct)
    {
        // protocol limit: at most 10,000 URLs per POST
        foreach (string[] batch in urls.Chunk(10_000))
        {
            var payload = new
            {
                host = options.Host,
                key = options.Key,
                keyLocation = $"https://{options.Host}/{options.Key}.txt",
                urlList = batch,
            };
            using HttpResponseMessage response = await http.PostAsJsonAsync(options.Endpoint, payload, ct);
            logger.LogInformation("IndexNow {Count} URLs -> {Status}", batch.Length, (int)response.StatusCode);
        }
    }
}

// IndexNow key file: https://bookstore.example/{key}.txt
app.MapGet($"/{indexNow.Key}.txt", () => Results.Text(indexNow.Key, "text/plain; charset=utf-8"));

The example in this post doesn't send a real request to a search engine; bookstore.example isn't our domain, and a real engine would reject the request because it couldn't find the key file at that address. I pointed the endpoint at a stand-in in the same application that logs the incoming body. The real payload sent by the publish request above:

mock IndexNow received: {"host":"bookstore.example","key":"5f3c9a1e7b2d4c6f8a0e1b3d5c7f9a2e","keyLocation":"https://bookstore.example/5f3c9a1e7b2d4c6f8a0e1b3d5c7f9a2e.txt","urlList":["https://bookstore.example/books/book-10"]}
info: Bookstore.Sitemap.IndexNowClient[0]
      IndexNow 1 URLs -> 202

The codes a real engine may return are defined in the protocol: 200 received, 202 received but key validation pending, 400 invalid format, 403 invalid key (file missing or content doesn't match), 422 URLs don't belong to the host, 429 too many requests. 200 only says the request was received; not that the URL will be crawled or indexed. In production, instead of making this call synchronously inside the publish request, hand it off to a background job or an outbox; an engine's slow response shouldn't keep the editor's publish button waiting.

A Build-Time Sitemap in an Angular SSR Project

If the content of an Angular application using SSR (Server-Side Rendering) is known at build time (prerendered routes), the sitemap can be a build step too. The Angular CLI (Command Line Interface) writes the routes it prerenders to dist/bookstore-ng/prerendered-routes.json; a small Node.js script turns that into a sitemap:

// after `ng build`: turn the prerendered routes into dist/.../browser/sitemap.xml
import { readFileSync, statSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';

const SITE = 'https://bookstore.example';
const dist = 'dist/bookstore-ng';
const { routes } = JSON.parse(readFileSync(join(dist, 'prerendered-routes.json'), 'utf8'));

const urls = Object.keys(routes).map(route => {
  // lastmod: the build time of the generated HTML is not a content change, so we skip it
  return `  <url><loc>${SITE}${route === '/' ? '/' : route}</loc></url>`;
});

const xml = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${urls.join('\n')}
</urlset>
`;
writeFileSync(join(dist, 'browser', 'sitemap.xml'), xml);
console.log(`sitemap.xml: ${urls.length} URLs, ${statSync(join(dist, 'browser', 'sitemap.xml')).size} bytes`);

$ npm run build
> ng build && node scripts/sitemap.mjs
Prerendered 4 static routes.
sitemap.xml: 4 URLs, 357 bytes

$ curl -sI http://localhost:20110/sitemap.xml | grep -iE "^HTTP|content-type|cache-control"
HTTP/1.1 200 OK
Cache-Control: public, max-age=3600
Content-Type: application/xml

The script doesn't write lastmod, because the only date it has is the build time; having every URL say "changed" on every deploy is exactly the untrustworthy lastmod I described above. If you can get real dates from the data source (CMS or API), add them. The cache header comes from the setHeaders rule we added to server.ts in the previous post; without that rule, the sitemap would also be served with a one-year cache. For content rendered on the server or changing often, a build-time sitemap isn't enough; generate it with a route on the Express side or from the ASP.NET Core API behind it.

mertozen.com's sitemap.xml

This site's sitemap is also produced by a controller action, built with XDocument and kept in IMemoryCache for an hour. According to the source code: two <url> entries per page, Turkish and English, each with tr, en and x-default alternates; for posts, lastmod = the later of the publish date and the update date in the CMS (Content Management System). That's the right approach as described above. Three observations:

First, every entry has changefreq and priority (home page daily/0.8, posts monthly/0.7). Google ignores them; no harm, no benefit.

Second, the accuracy of lastmod depends on the accuracy of the update date in the CMS. If the CMS moves every post's update date to the same day during a bulk import or schema change, the sitemap reports the same day for every post too. In a sample taken on 7 November while the series was being planned, the lastmod values were noted to be the same day on all pages; a situation like that can trigger the loss of trust described above. If a post's content hasn't changed, its update date shouldn't change either.

Third, the sitemap action fetches posts from the CMS with a cap of 500 records (GetPostsAsync("tr", 1, 500, ...)). The day the number of published posts passes 500, the oldest posts silently drop out of the sitemap; no error, no warning. The split sitemap structure in this post is exactly the fix for that limit. The decision to change it is the site owner's.

Production Notes

Put only canonical URLs that return 200 and that you want indexed in the sitemap. Listing a URL that redirects, carries noindex or has a canonical pointing elsewhere sends Google conflicting signals; Search Console's sitemap report flags those URLs separately.

Announce the sitemap both with the Sitemap line in robots.txt and in Search Console. Search Console's sitemap report shows when Google last read the file, how many URLs it found and any errors; with a split sitemap you can track each part separately.

Monitor the performance of the sitemap endpoint. Googlebot and other crawlers request the sitemap regularly; an uncached query that takes 0.35 seconds becomes a noticeable database load with a few parts and a few crawlers. Output cache with tagged invalidation solves this.

Filter out drafts and content with a future publish date. If Where(b => b.IsPublished) isn't enough (if there are scheduled publishes), add a PublishedAt <= now condition as well; a URL in the sitemap that isn't live means a crawl request that returns 404.

Common Pitfalls

Writing the generation time into lastmod. A sitemap that says every URL changes every day makes the change signal completely worthless.

Trying to steer crawling with changefreq and priority. Google ignores both.

Writing relative URLs or putting the sitemap in a subfolder. <loc>/books/dune</loc> is invalid; a file under /sitemaps/ only covers that folder unless submitted through Search Console.

Counting the 50,000 URL limit and forgetting the 50 MB, or the other way around. Both apply at once; in sitemaps with hreflang, size grows faster than you'd expect.

Setting up IndexNow for Google. Google isn't a participant; IndexNow is for Bing, Yandex and the other participants.

A Small Experiment

★☆☆ Download your own site's sitemap and count three things: how many URLs there are, how many distinct lastmod values there are, and how many URLs share the most frequent lastmod value. If one date repeats across most URLs, find out why.

★★☆ Change WriteBooksAsync to split by Id range instead of Skip/Take, then publish a book and compare the sizes of the three parts before and after. How many files changed?

★★★ Take the IndexNow call out of the publish endpoint: publishing writes the changed URL into an outbox table; a BackgroundService sends the URLs in the outbox every minute in batches of 10,000, backs off when it gets 429, and marks successful submissions. Show with a test why the outbox is needed (the scenario where the publish was saved but the notification couldn't be sent).

The sitemap tells Google which URL you want; but if the same content lives at several URLs, which one will Google pick? In the next post we'll look at the sources of duplicate content, rel=canonical and the other canonical signals, with a canonical tag helper in ASP.NET Core and the canonical link during SSR in Angular, in code.