Google Search Essentials: Technical Requirements, Spam Policies and Key Best Practices

Version note: This post is based on the Google Search Essentials and spam policy pages as of November 2026; the code examples were built and run with .NET 10.

Guess First

Bookstore has shipped two features: readers can write reviews of books, and they can create a "reading list" and share its link with friends. Three months later, strange queries show up in Search Console: drug and gambling searches that have nothing to do with books, all leading to pages under /lists/. Who do you think Google blames here: the people who wrote the spam, or the site? Which one-line header would have been enough to prevent it? And one more: do the links in those reviews count as your site's "vote"?

The Three Parts of Search Essentials

Google Search Essentials is today's version of the document that used to be called the "Webmaster Guidelines". It has three parts:

• Technical requirements: the minimum Google Search needs from a page to be able to show it.
• Spam policies: behaviors that can cause a page or site to rank lower or be removed from results entirely.
• Key best practices: the things that most affect how your site looks and ranks.

The order matters: the first two are "must have", the third is "good to have". The same warning repeats throughout the document: meeting these doesn't mean Google will crawl, index or serve your content. Search Essentials is not a contract, it is an eligibility list.

Technical Requirements: Truly the Minimum

The list has three items, and most sites meet them without even noticing:

Googlebot isn't blocked. The page must be publicly available and Googlebot must be able to reach it; pages that require a login aren't crawled. Google adds an important note here: a URL (Uniform Resource Locator) blocked by robots.txt can still appear in results. If you want to keep a page out of the index, use noindex and allow crawling.

The page works. Google only indexes pages served with an HTTP (Hypertext Transfer Protocol) 200 (success) status code; client and server error pages aren't indexed. The place to check is the URL Inspection tool in Search Console.

The content is indexable. The text must be in a file type Google supports, and the content must comply with the spam policies.

Three short items, but for a developer each hides a class of bugs: a staging Disallow: / that accidentally ships to production, an SPA (Single-Page Application) that returns 200 for everything, or a page whose content lives only inside an image. Diagnosing them is the topic of the sixth post in this series; the code-side fixes are in the Technical SEO (Search Engine Optimization) series.

A Map of the Spam Policies

Google collects the spam policies for web search on a single page. You don't need to memorize them all; it's enough for a developer to know where they touch which policy:

Besides these, the list also has keyword stuffing, link spam, scraping, thin affiliation, misleading functionality, and machine-generated traffic, which covers sending automated queries to Google without permission. Note that last one: it also covers tools that scrape Google results pages to track rankings.

What Changed Between 2024 and 2026?

In March 2024 three new policies were added: scaled content abuse, site reputation abuse and expired domain abuse. The wording of scaled content abuse matters in particular: the rule applies "no matter how it's created". So the problem isn't producing with AI, it's mass-producing without adding value; using generative AI tools to generate many pages without adding value is listed explicitly among the examples.

On April 13, 2026 back button hijacking was added to the malicious practices policy: a site manipulating browser history or other browser functions so the user can't immediately go back to the page they came from. Code in an SPA that pushes fake history entries with history.pushState "so the user won't leave" is now a direct policy violation.

Since April 14, 2026 Google states that it may use spam reports submitted by users to take manual action; on April 23 it clarified when and why that can happen.

On May 15, 2026 Google stated explicitly that the spam policies also apply to generative AI responses in Google Search: attempts to manipulate AI Overviews and AI Mode responses are spam too. We'll open this topic separately in the GEO (Generative Engine Optimization) series.

In August 2026 the enforcement of site reputation abuse inside the EEA (European Economic Area) changed: within the EEA such pages may be treated as a section separate from the main domain instead of receiving a manual action, and rank on their own merits. Outside the EEA the manual action path remains.

On October 26, 2026 the user-generated spam guide was updated with blocking indexing of link-shared content and monitoring abuse in Search Console. That is exactly Bookstore's reading list story.

Manual Actions and Algorithmic Detection

Google detects violations "through automated systems and, as needed, human review". On the automated side, spam detection systems such as SpamBrain run; they don't send you a notification, the ranking just drops or the page disappears. Human review, on the other hand, results in a manual action, and Google reports it in the Manual actions report and the message center in Search Console.

This distinction is critical for diagnosis. If there's a manual action, you know what to do: fix the problem and send a reconsideration request from Search Console. If there's no manual action and traffic dropped, the cause could be a spam system, a core update or a purely technical bug. We'll talk about core updates and when you should wait in the third post of the series.

The Developer's Angle: User Content and Link-Shared Pages

Back to the opening story. Two of the recommendations in Google's user-generated spam guide are pure code: keep pages meant for sharing or utility out of the index by default, and add ugc or nofollow to links in untrusted content. Reading lists exist for the person who has the link, not for Search. In ASP.NET Core, expressing that with endpoint metadata is a clean approach:

namespace Bookstore.Web.Seo;

public sealed record NoIndexMetadata;

public static class NoIndexExtensions
{
    public static TBuilder NoIndex<TBuilder>(this TBuilder builder) where TBuilder : IEndpointConventionBuilder =>
        builder.WithMetadata(new NoIndexMetadata());

    public static IApplicationBuilder UseNoIndexHeader(this IApplicationBuilder app) =>
        app.Use(async (context, next) =>
        {
            context.Response.OnStarting(() =>
            {
                if (context.GetEndpoint()?.Metadata.GetMetadata<NoIndexMetadata>() is not null)
                {
                    context.Response.Headers["X-Robots-Tag"] = "noindex";
                }
                return Task.CompletedTask;
            });
            await next();
        });
}

app.UseNoIndexHeader();

// reading lists are shared by link, they are not meant for Search
app.MapGet("/lists/{token}", (string token) =>
        Results.Content($"<h1>Reading list {token}</h1>", "text/html"))
    .NoIndex();

The header is added right before the response starts, based on the endpoint's metadata. The real request:

$ curl -si http://localhost:5481/lists/q7Hk2
HTTP/1.1 200 OK
Content-Length: 27
Content-Type: text/html
Date: Sun, 08 Nov 2026 09:06:25 GMT
Server: Kestrel
X-Robots-Tag: noindex

<h1>Reading list q7Hk2</h1>

Note: we don't block this page with robots.txt. If we did, Googlebot couldn't fetch the page and couldn't see the header either; a URL that gets external links could appear in results without its content being read. noindex only works while crawling is allowed. The details of X-Robots-Tag and applying it to files that aren't HTML (HyperText Markup Language), such as PDFs are in the "noindex and X-Robots-Tag" post of the Technical SEO series.

The second step is for links inside reviews: encode the content, generate the links yourself, and set the rel value based on the author's trust level.

using System.Net;
using System.Text.RegularExpressions;

namespace Bookstore.Web.Seo;

public static partial class ReviewLinkRenderer
{
    [GeneratedRegex(@"https?://[^\s<>""]+")]
    private static partial Regex UrlPattern();

    public static string Render(string reviewText, bool authorIsTrusted)
    {
        string rel = authorIsTrusted ? "ugc" : "ugc nofollow";
        string encoded = WebUtility.HtmlEncode(reviewText);
        return UrlPattern().Replace(encoded, m => $"<a href=\"{m.Value}\" rel=\"{rel}\">{m.Value}</a>");
    }
}

Great book &lt;3 see <a href="https://cheap-pills.example/buy" rel="ugc nofollow">https://cheap-pills.example/buy</a>
Errata: <a href="https://bookstore.example/errata/42" rel="ugc">https://bookstore.example/errata/42</a>

The link in the new account's review gets ugc nofollow, the trusted reader's link gets only ugc. Google's guide recommends the same logic at page level too: start content from new users without reputation with noindex and allow indexing as they build reputation. The WebUtility.HtmlEncode that runs first is also an XSS (Cross-Site Scripting) defense; that topic is covered in the Web Security (OWASP Top 10) series.

Key Best Practices

The third part of Search Essentials is a short list: create helpful, reliable, people-first content; use the words people would search for in prominent places such as the title, main heading, alt text and link text; make your links crawlable; promote your site in relevant communities; follow the specific guidance for images, video, structured data and JavaScript; enable the search features that fit your site; and use the right method for content that shouldn't appear. The first item on that list is the entry point to the ranking systems and E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness), which we'll talk about in the third post of the series.

Production Notes

Verify Search Console and turn on its notifications. Manual action and security issue notifications go only to verified property owners; on an unverified site you'll learn you've been hacked from a customer, not from Google.

For every new feature that produces user content, decide "should this appear in Search?" and make noindex the default in code. Google's guide also recommends monitoring: a sudden jump in indexed pages, impressions or queries unrelated to your platform in the Page indexing and Performance reports is the earliest sign of abuse.

Watch traffic to share URLs in your server log. If reading lists suddenly get requests from thousands of external referrers overnight, someone is using those pages for spam.

Common Pitfalls

Thinking in "white hat / black hat" folklore. That binary is industry slang; it has no counterpart in Google's documents. What is binding are the spam policies themselves. When you discuss a tactic sold as "gray", the question shouldn't be "which hat?" but "which policy does it fall under?".

Treating the policies as only the SEO team's job. Back button hijacking, cloaking, sneaky redirects and hacked content are born directly in frontend and backend code. Someone who knows these policies should be in the code review.

Seeing AI-generated content as a problem by itself, or the opposite, saying "Google doesn't object to AI" and pushing out hundreds of template pages. The rule looks at scale and the value added for users, not at how the content was produced. This myth gets its own treatment in the seventh post of the series.

Deleting the spam and walking away. If you clean up hacked or spam-filled pages without closing the hole, the same content comes back; if there's a manual action, fix the root cause before sending a reconsideration request.

A Small Experiment

★☆☆ List every page type in your own project that is produced by users or shared by link. For each, answer "should it be indexed?" in a single word.

★★☆ Make NoIndexMetadata work for Razor Pages too: add the metadata to a page with a convention and verify the header with curl -si.

★★★ Extend ReviewLinkRenderer: if the author has fewer than 5 approved reviews, add <meta name="robots" content="noindex"> to the review page; at 5 or more, allow indexing. Show both cases with a real request.

We've seen the rules; in the next post we move to their other side, how Google ranks: ranking systems, core updates and the much misunderstood E-E-A-T.