HTTP Status Codes and Soft 404s: How Google Interprets 404, 410, 5xx and 429

Version note: The code in this post was built and run with .NET 10 (ASP.NET Core 10) and Angular 22.2; Google's interpretation of status codes is taken from Google's crawling documentation as of November 2026.

Guess First

When Bookstore's book detail page can't find a book, it shows a nice "Sorry, this book could not be found" page and returns 200. Books permanently withdrawn from the catalog get the same page. One night the database doesn't respond for ten minutes; meanwhile, because the code "can't find" the books, it returns the same page with 200 again. On top of that, so that crawlers don't wear out the server, the team has put a per-minute, per-IP (Internet Protocol) limit on book pages and returns 429 to anyone exceeding it.

Four questions. What does Google see those "not found" pages as? Should withdrawn books return 404 or 410, and does it matter? What should be returned instead of 200 during the database outage? And what does the 429 limit do to Googlebot?

In earlier posts we told Google what we want with URLs (Uniform Resource Locators), robots.txt, noindex, sitemaps and canonicals. The status code is the most basic piece of information the server gives in every response: does this URL exist, has it moved, is it temporarily broken? Most of Google's crawling and indexing decisions look at this three-digit number first.

How Google Interprets Each Class

A summary of Google's HTTP (HyperText Transfer Protocol) status code documentation:

Three lessons come out of this table. First, 4xx means "this content doesn't exist" and has lasting consequences: the URL leaves the index. Second, 5xx and 429 mean "not right now, come back later"; Google keeps the URL in the index for a while and lowers the crawl rate. For 500, 502 and 503, the slowdown is proportional to the number of URLs returning errors. Third, although 429 is a 4xx, Google puts it in the server error class.

The answer to the 404 vs 410 question is here too: according to Google's documentation, every 4xx code except 429 is handled the same way, so 404 and 410 are the same to Google. Semantically, 410 means "permanently removed" and is more informative for other clients (caches, link checkers); don't expect different behavior from Google.

There's another note, added to the JavaScript SEO (Search Engine Optimization) documentation in December 2025: pages returning a status code other than 200 might not be sent to rendering. So content that a 404 page produces with JavaScript might never be processed; that's correct behavior, but it means trying to "rescue" with JavaScript a page that should be 200 but mistakenly returns 404 won't work.

Soft 404: The "Not Found" Page That Says 200

The answer to the first opening question: Google flags these pages as soft 404s. A soft 404 is a response that returns 2xx but whose content is an error message, an empty page or "not found"; it shows up as a separate row in Search Console's "Pages" report. The problem cuts both ways: Google evaluates each one as a separate page and crawls it again and again, wasting crawl budget; and if the classification is wrong, "not found" pages may stay in the index. Bookstore's first version:

// the soft 404: a "not found" page that says 200
app.MapGet("/soft/books/{slug}", (string slug) =>
    books.TryGetValue(slug, out string? title)
        ? Results.Content($"<h1>{title}</h1>", "text/html")
        : Results.Content("<h1>Sorry, this book could not be found</h1>", "text/html"));

$ curl -s -i http://localhost:20107/soft/books/neuromancer
HTTP/1.1 200 OK
Content-Type: text/html
<h1>Sorry, this book could not be found</h1>

The right way is to show the user the same nice page while returning the correct status code. In ASP.NET Core it's enough for the endpoint to return only the status code and let UseStatusCodePagesWithReExecute produce the page:

// empty-bodied 4xx/5xx -> re-execute /not-found, but keep the original status code
app.UseStatusCodePagesWithReExecute("/not-found");

Dictionary<string, string> books = new() { ["dune"] = "Dune", ["foundation"] = "Foundation" };
HashSet<string> removed = ["hyperion"]; // permanently withdrawn from the catalog

app.MapMethods("/books/{slug}", [HttpMethods.Get, HttpMethods.Head], (string slug, HttpContext ctx) =>
{
    if (maintenance)
    {
        // temporary: "come back later", do not drop the page from the index
        ctx.Response.Headers.RetryAfter = "120";
        return Results.Text("Back soon", statusCode: StatusCodes.Status503ServiceUnavailable);
    }
    if (books.TryGetValue(slug, out string? title))
        return Results.Content($"<h1>{title}</h1>", "text/html");
    return removed.Contains(slug)
        ? Results.StatusCode(StatusCodes.Status410Gone)
        : Results.NotFound();
});

app.Map("/not-found", (HttpContext ctx) =>
    Results.Content($"<h1>Page not found</h1><p>Status: {ctx.Response.StatusCode}</p>", "text/html"));

$ curl -s -i http://localhost:20107/books/dune
HTTP/1.1 200 OK
Content-Type: text/html
<h1>Dune</h1>

$ curl -s -i http://localhost:20107/books/hyperion
HTTP/1.1 410 Gone
Content-Type: text/html
<h1>Page not found</h1><p>Status: 410</p>

$ curl -s -i http://localhost:20107/books/neuromancer
HTTP/1.1 404 Not Found
Content-Type: text/html
<h1>Page not found</h1><p>Status: 404</p>

The user sees the same friendly page, the crawler gets the right code. UseStatusCodePagesWithReExecute only kicks in for responses with an empty body; if the endpoint writes a body itself (like the soft 404 above), it leaves it alone. The re-executed /not-found handler can read the original status code from ctx.Response.StatusCode; if you want different messages for 404 and 410, branch there. Results.NotFound() and the other IResult types are covered in the ASP.NET Core Web API (Application Programming Interface) series post "Status Codes and Responses in ASP.NET Core: Ok, CreatedAtAction and ProblemDetails".

5xx, 503 and Retry-After: Saying Temporary Problems Are Temporary

The third opening question is the most expensive one. Saying "there's no book" while the database isn't responding tells Google those pages don't exist. If the outage lasts long, or crawling is heavy at that moment, existing pages may be processed as if they had become 404s and drop out of the index. The right language for a temporary problem is 503 and, if available, a Retry-After header:

$ MAINTENANCE=1 dotnet run --no-launch-profile --urls http://localhost:20107
$ curl -s -i http://localhost:20107/books/dune
HTTP/1.1 503 Service Unavailable
Content-Length: 9
Content-Type: text/plain; charset=utf-8
Retry-After: 120
Back soon

According to Google's documentation, crawling slows down on 5xx and URLs stay in the index for a while; if the error continues for a long time, they're dropped. That's why 503 is the right tool for short outages, not for maintenance lasting days. The point to watch here is distinguishing the type of failure: "the data source didn't respond" and "the data source responded and there's no record" are two different situations and must be two different paths in the code. A repository or HTTP client layer returning null for both is the classic design mistake that erases this distinction.

429 and Rate Limiting: The Cost of Slowing Googlebot Down

Bookstore's rate limiter is set up with ASP.NET Core's built-in middleware:

builder.Services.AddRateLimiter(o =>
{
    o.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
    o.OnRejected = (ctx, _) =>
    {
        ctx.HttpContext.Response.Headers.RetryAfter = "10";
        return ValueTask.CompletedTask;
    };
    o.AddFixedWindowLimiter("books", w => { w.PermitLimit = 3; w.Window = TimeSpan.FromSeconds(10); });
});

$ for i in 1 2 3 4; do curl -s -o /dev/null -w "request $i: %{http_code} retry-after=%header{retry-after}\n" http://localhost:20107/limited/books/dune; done
request 1: 200 retry-after=
request 2: 200 retry-after=
request 3: 200 retry-after=
request 4: 429 retry-after=10

The default RejectionStatusCode is 503; here 429 was chosen explicitly. The answer to the fourth opening question comes from the table: Google treats 429 as a signal that the server is overloaded and slows crawling down; URLs that keep getting 429 may eventually drop out of the index. Google's guide to reducing the crawl rate recommends returning 500, 503 or 429 for a few hours in an urgent overload; but a permanent rate limit catching Googlebot means the site throttles its own crawling. If you apply rate limits to bot traffic, it's better to verify the real Googlebot (in the eleventh post of the series) and give it a wider limit, while being strict with fake user agents. Also, crawling can't be slowed with 401 or 403; Google handles those as 4xx and removes the URLs from the index.

HEAD Requests and 405

In Minimal APIs, MapGet matches only GET. Link checkers, some crawlers and uptime monitors, however, send HEAD to check whether a page exists:

$ curl -sI http://localhost:20107/get-only/books/dune
HTTP/1.1 405 Method Not Allowed
Allow: GET

$ curl -sI http://localhost:20107/books/dune
HTTP/1.1 200 OK
Content-Type: text/html

The second endpoint accepts GET and HEAD together with MapMethods; Kestrel sends no body in a HEAD response, and the status and headers stay the same as GET. A 405 doesn't affect the GET requests Google uses for Search, but tools that check with HEAD report the page as broken. In MVC (Model-View-Controller), the [HttpGet] attribute also matches only GET; the fix is adding HEAD to every action or doing it in bulk with a convention.

Angular SSR: The "200 for Everything" Problem

The classic SPA (Single-Page Application) problem: the server returns the same index.html with 200 for every path, and the "not found" decision is made in the browser with JavaScript. For a crawler, this means every possible URL on the site exists. In Bookstore's Angular application, the route that serves the same component with RenderMode.Client shows this exactly:

$ curl -s -o /dev/null -w "%{http_code}\n" http://localhost:20110/client/books/hyperion
200
$ curl -s -o /dev/null -w "%{http_code}\n" http://localhost:20110/books/hyperion
404

The second route runs with SSR (Server-Side Rendering), and when the component can't find the book it sets the status code on the server. Angular's RESPONSE_INIT token gives access to the response's status and headers during rendering; the token is only populated on the server during a request, and is null during the build, prerendering and in the browser:

export class BookDetail {
  readonly slug = input.required<string>();
  protected readonly book = computed(() => BOOKS.find(b => b.slug === this.slug()));
  private readonly seo = inject(SeoService);
  private readonly responseInit = inject(RESPONSE_INIT, { optional: true });

  constructor() {
    effect(() => {
      const b = this.book();
      if (b) {
        this.seo.update({ title: b.title, description: b.description, path: `/books/${b.slug}` });
      } else {
        // unknown slug: real 404 on the server, noindex for the client
        if (this.responseInit) {
          this.responseInit.status = 404;
        }
        this.seo.update({ title: 'Not found', description: '', path: `/books/${this.slug()}`, noindex: true });
      }
    });
  }
}

For paths that match no route at all, setting the status statically on the server route is enough:

{ path: '**', renderMode: RenderMode.Server, status: 404 },

$ curl -s -D - -o /dev/null http://localhost:20110/nope | head -1
HTTP/1.1 404 Not Found

The book route uses RenderMode.Prerender to produce HTML (HyperText Markup Language) at build time for the three known books; when a slug not on the list arrives, Angular renders that request on the server by default, and the 404 decision is made there. For an application that has to run entirely client-side, Google recommends two options: for unknown content, redirect with JavaScript to a URL where the server returns 404, or add noindex with JavaScript. Both depend on rendering and aren't as reliable as a real 404 under SSR.

Status Codes on mertozen.com

The last two releases of this site's source code fix exactly the topic of this post. In the observation on 7 November 2026, HEAD requests sent to the site returned 405; in v1.12.1 an IActionModelConvention (HeadMatchesGetConvention) was added that also attaches HEAD to every MVC action accepting GET. The reason in the commit message: link checkers and crawlers were getting 405.

The fix in v1.12.2 is more important. The site reads its content from a CMS (Content Management System) API; previously, when the CMS temporarily returned an error, the blog detail action returned 404 because it "couldn't find" the post. So a few minutes of hiccup in the CMS, if it coincided with a crawl, could report existing posts to Google as "not there". In the new release, a CMS failure (429, 5xx or timeout) is carried up as a separate exception, and a middleware answers it with 503, Retry-After: 30, Cache-Control: no-store and a noindex in the page; the "record genuinely doesn't exist" case stays a 404. That's the real-life counterpart, and the fix, of the database outage scenario from the opening.

On the 404 side, the site uses UseStatusCodePagesWithReExecute("/404"); every empty-bodied 4xx/5xx response comes back with a friendly page while keeping the original status code, and that page also carries noindex, follow. In other words, the same structure we built above.

Production Notes

Watch the "Soft 404" row in Search Console's "Pages" report. If you see pages that really exist in that row, your page content looks like an error page to Google (very little content, a "not found"-like text); empty category pages are a common example.

In the Crawl Stats report (under Search Console settings), look at the breakdown by response code and the server response time. An increase in the 5xx rate followed by a drop in crawl requests is a sign that Google slowed down to protect your server.

Test the failure path. Write separate integration tests for the "no record" and "data source didn't respond" cases; assert that the second returns 503 and the first returns 404. In code without tests, these two cases getting mixed up is only noticed in production after traffic drops.

Don't return 503 for days during long maintenance. Google drops URLs as the error persists; for long maintenance, serving a readable maintenance page with 200 or keeping a read-only copy of the site up is less risky.

Common Pitfalls

Returning the "not found" page with 200. It becomes a soft 404; it wastes Google's time and its classification isn't up to you.

Turning a temporary data source error into a 404. Existing pages look nonexistent to Google.

Redirecting every deleted page to the home page with 301. Google's site move guide says to avoid redirecting many pages to an irrelevant destination (such as the home page); such redirects may be reported as soft 404s. If there's no relevant new page, return 404 or 410.

Trying to throttle crawling with 401, 403 or a permanent 429. 401/403 remove URLs from the index; a continuous 429 eventually leads to the same result.

Returning 200 for unknown routes in an SPA. If you use SSR or prerendering, give the ** route status: 404, and set the status with RESPONSE_INIT for dynamic content.

A Small Experiment

★☆☆ Request three nonexistent URLs on your own site (curl -s -o /dev/null -w "%{http_code}"): a random path, a nonexistent slug, a nonexistent file extension. Are they all 404?

★★☆ Bind Bookstore's book endpoint to an IBookCatalog service; the service returns null for "not found" and a dedicated exception for "the source didn't respond". Write a middleware that turns the exception into 503 with Retry-After, and test both cases with WebApplicationFactory.

★★★ Partition the rate limiter by user agent and verified bot status: unverified "Googlebot" user agents fall into a strict limit, the real Googlebot into a wider limit, other clients into the default limit. Put the verification function behind an interface for now, to be replaced with the real one after the eleventh post of the series.

We deliberately left the 3xx in the middle of the status code family for last. In the next post we'll set up the difference between 301, 302, 307 and 308, the effect of redirects on canonicals, chains, and old slug → new slug redirects fed from the database with ASP.NET Core's URL Rewriting Middleware.