Technical SEO Audits: Screaming Frog, Lighthouse SEO Audits, Bing Site Scan, the URL Inspection API and Your Own Crawler in CI

Version note: The code in this post was built and tested with .NET 10 (ASP.NET Core 10.0.15) and AngleSharp 1.10.3; Lighthouse 13.6.0 was run with Playwright's Chromium. The Screaming Frog, Bing Site Scan and URL Inspection API sections rely on these tools' official documentation as read in November 2026.

Guess First

The catalog page of Bookstore, the online bookstore used as the shared example across this site's series, scores 100 in Lighthouse's SEO category in the draft code of the previous post. In the same draft the page has four filter links, and all four go to an address that returns 404.

Why didn't Lighthouse see that? And who checks a site's technical SEO health as a whole rather than page by page, and how?

This is the last post of the Technical SEO series. The way to see that the rules we built over the previous fifteen posts (status codes, redirects, canonicals, hreflang, sitemaps, robots.txt, images, video, filter URLs) don't break in the next deploy is an audit. We'll look at what each tool sees and write our own crawler that runs in CI.

The Audit Checklist and What Each Tool Can See

The questions a technical SEO audit asks line up with the posts of the series. The tools look at them from three different places: the rendered state of a single page (Lighthouse), the site as a whole as reached through its links (Screaming Frog, Bing Site Scan, your own crawler) and Google's own record in the index (URL Inspection).

The table in short: no single tool is enough. Lighthouse shows the page itself, a crawler shows the structure of the site, and URL Inspection shows Google's result.

Lighthouse's SEO Category: One Page, After Rendering

Lighthouse 13.6.0's SEO category has ten automated audits: is-crawlable, document-title, meta-description, http-status-code, link-text, crawlable-anchors, robots-txt, image-alt, hreflang and canonical; plus a manual structured-data audit that doesn't count towards the score. In the package's own configuration the weight of is-crawlable is set so that if this audit fails on its own, the whole category fails too. The category's description states its limits: these are basic search engine optimization recommendations, and many other factors that affect ranking aren't scored here. What each SEO audit does is in Lighthouse's documentation on developer.chrome.com; meta-description, for example, only checks that the element exists and isn't empty, and doesn't judge the quality of the description.

I audited three of Bookstore's pages and the catalog page from the previous post's draft (with Playwright's Chromium):

$ npx lighthouse "http://127.0.0.1:52213/books/genre/science-fiction?page=2" --only-categories=seo \
    --output=json --output-path=page2-draft.json --chrome-path="$CHROMIUM" --chrome-flags="--headless=new" --quiet
$ jq -r '"\(input_filename): seo=\(.categories.seo.score*100), lighthouse \(.lighthouseVersion)"' dune.json video.json page2.json page2-draft.json
dune.json: seo=100, lighthouse 13.6.0
video.json: seo=100, lighthouse 13.6.0
page2.json: seo=100, lighthouse 13.6.0
page2-draft.json: seo=100, lighthouse 13.6.0

The filter links on the draft page looked like this:

$ curl -s "http://127.0.0.1:52213/books/genre/science-fiction?page=2" | grep -o '<a href="CatalogQuery[^>]*>' | head -2
<a href="CatalogQuery { Genre = science-fiction, Language = , Format = Hardcover, Sort = title, Page = 1, IsIndexable = False }.ToUrl(p)">
<a href="CatalogQuery { Genre = science-fiction, Language = , Format = Paperback, Sort = title, Page = 1, IsIndexable = False }.ToUrl(p)">

In href="@(Model.Query with { ... }).ToUrl(p)", Razor evaluated @( ) as an expression, printed the record's ToString() and treated .ToUrl(p) as plain text; the correct form is @((Model.Query with { ... }).ToUrl(p)). The answer to the opening question: crawlable-anchors checks whether a link is in a crawlable form (is there an href, is it javascript:), not whether the target returns 404. Lighthouse audits a single page; it doesn't follow the links leaving it. That's why "Lighthouse SEO 100" means "the page follows the basic rules", not "the site is healthy" or "rankings will be good". The difference between the DOM after JavaScript and the initial HTML was covered in the eighth post of the series.

Your Own Crawler: Sitemap and Link Graph

To see what Lighthouse can't, you have to walk the site like a crawler. Bookstore's audit tool, tools/Bookstore.SeoAudit, is a console application fed from two sources: discovery through links, that is a BFS starting from the home pages, and the sitemap. The ResolveAsync that follows redirect chains in the link graph we wrote in the "Internal Linking" post of the Content SEO series is reused here as is.

The first design decision is about addresses. The application under test runs on http://127.0.0.1:52213, but the canonical, hreflang and sitemap URLs in the HTML are https://bookstore.example, as in production. Changing the app's base URL for the test environment would mean testing different HTML from production's. Instead, the crawler maps the two addresses:

// Maps the application under test (e.g. http://127.0.0.1:52213) to the production address (https://bookstore.example):
// canonical, hreflang and sitemap URLs in the HTML stay as in production, requests go to the local app.
public sealed class Fetcher(Uri target, Uri site)
{
    private readonly HttpClient _http = new(new HttpClientHandler { AllowAutoRedirect = false }) { Timeout = TimeSpan.FromSeconds(10) };

    // is it the site's own URL? If so path + query, otherwise null (external link)
    public string? ToPath(string url, string basePath = "/")
    {
        Uri resolved = new(new Uri(site, basePath), url);
        bool own = resolved.Authority == site.Authority || resolved.Authority == target.Authority;
        return own && resolved.Scheme.StartsWith("http") ? resolved.PathAndQuery : null;
    }

    // follows the redirect chain: [first path, ..., last path] and the final status
    public async Task<(List<string> Hops, int Status)> ResolveAsync(string path)
    {
        List<string> hops = [path];
        for (int i = 0; i < 5; i++)
        {
            Page page = await GetAsync(hops[^1]);
            if (page.Status is < 300 or > 399 || page.Location is null)
                return (hops, page.Status);
            hops.Add(ToPath(page.Location, hops[^1])!);
        }
        return (hops, 0); // a chain longer than 5 hops or a loop
    }
}

The crawler follows the Disallow rules of robots.txt, so Bookstore's filter URLs are never requested. On every page it checks the canonical, the meta description and the src and alt of each <img>, and follows internal links together with their redirect chains. Then it asks four questions for every URL in the sitemap:

// 2) sitemap: every URL must be 200, indexable, its own canonical and reachable through links
foreach (string path in sitemap)
{
    Page page = await fetcher.GetAsync(path);
    if (page.Status != 200)
    {
        Add(Severity.Error, "sitemap-status", path, $"listed in sitemap but returns {page.Status}");
        continue;
    }
    if (IsNoindex(page))
        Add(Severity.Error, "sitemap-noindex", path, "listed in sitemap but has noindex");
    string? canonical = Canonical(page);
    if (canonical != fetcher.ToSiteUrl(path))
        Add(Severity.Error, "sitemap-canonical", path, $"canonical is {canonical ?? "missing"}");
    if (!_reachable.Contains(path))
        Add(Severity.Warning, "orphan", path, "in sitemap but no internal link reaches it");
}

The hreflang check is the reciprocity rule from the tenth post of the series: every alternate must return 200 and link back to this page.

// hreflang: every alternate must return 200 and link back to this page
foreach (IElement link in page.Html!.QuerySelectorAll("link[rel=alternate][hreflang]"))
{
    string href = link.GetAttribute("href")!;
    if (href == self)
        continue;
    Page alternate = await fetcher.GetAsync(fetcher.ToPath(href)!);
    if (alternate.Status != 200)
        Add(Severity.Error, "hreflang-target", page.Path, $"{link.GetAttribute("hreflang")} -> {href} returns {alternate.Status}");
    else if (alternate.Html?.QuerySelectorAll("link[rel=alternate][hreflang]").All(l => l.GetAttribute("href") != self) != false)
        Add(Severity.Error, "hreflang-return", page.Path, $"{href} has no hreflang link back");
}

The First Run: What Did It Find?

I ran the crawler for the first time against the application carrying the previous post's draft code:

$ dotnet run --project tools/Bookstore.SeoAudit -- http://127.0.0.1:52213/ https://bookstore.example/ > audit.txt
$ head -4 audit.txt | cut -c1-150
SEO audit http://127.0.0.1:52213/ (as https://bookstore.example/)
sitemap URLs: 96, pages reached by links: 104, requests: 136

ERROR broken-link         /books/genre/classics             /books/genre/CatalogQuery%20%7B%20Genre%20=%20classics,%20Language%20=%20,%20Format%20=%20
$ grep -c broken-link audit.txt; grep -c duplicate-title audit.txt; grep -c orphan audit.txt
40
41
8
$ grep -E "duplicate-title +/books/dune |orphan +/books/genre/science-fiction/tr" audit.txt
WARN  duplicate-title     /books/dune                       "Dune | Frank Herbert | Bookstore" also on /en/books/dune
WARN  orphan              /books/genre/science-fiction/tr   in sitemap but no internal link reaches it
$ tail -1 audit.txt
40 errors, 49 warnings

Three different findings, three different lessons.

The 40 broken links were the Razor bug that Lighthouse scored 100: four filter links on each of ten catalog pages. The bug was fixed before the previous post was published, and the code there is the fixed version; still, the first thing to catch it was the crawler, not a test, because the tests only looked at links we had thought of in advance. A test that checks the filter links came with the fix.

The 8 orphans were language pages that are in the sitemap but linked from nowhere (such as /books/genre/science-fiction/tr). In the previous post we put these pages in the sitemap as filters to be indexed and forgot to link to them from the genre page. The genre page now has "Turkish" and "English" links, following the same rule as the sitemap: only for languages that narrow the list.

The 41 duplicate titles were the tool's own mistake: the Turkish and English pages of a book carry the same title (the book title and author are the same in both languages), and they're language versions linked with hreflang, not competing pages. The rule was changed to group by language. Every audit tool's rules need tuning to the structure of the site; the same goes for the warnings of Screaming Frog or Site Scan.

After the fixes:

$ dotnet run --project tools/Bookstore.SeoAudit -- http://127.0.0.1:52213/ https://bookstore.example/ seo-report.md
SEO audit http://127.0.0.1:52213/ (as https://bookstore.example/)
sitemap URLs: 96, pages reached by links: 114, requests: 114


0 errors, 0 warnings

When there's nothing to find, how do you know the crawler can find anything? The tool's own test sets up a small site with known problems: a URL in the sitemap that returns 301, a page with noindex, a page whose canonical points elsewhere, an hreflang target that doesn't link back, one link to a 404 and one through a redirect, an image without alt and a page linked from nowhere. The test expects each to be found exactly once:

$ dotnet test tests/Bookstore.SeoAudit.Tests --logger "console;verbosity=normal" | grep Passed
  Passed Bookstore.SeoAudit.Tests.AuditorTests.Finds_every_seeded_problem [95 ms]
     Passed: 1

CI: A Gate Before Deploy and a PR Comment

The crawler exits with a non-zero code and can write a Markdown report; those two features turn it into a CI step. Bookstore's GitHub Actions workflow runs the tests on every pull request, starts the app on the runner on 127.0.0.1, runs the crawler and writes the report both to the job summary and as a comment on the PR:

      - name: Start Bookstore.Web on 127.0.0.1
        run: |
          dotnet run --project src/Bookstore.Web --no-launch-profile --urls http://127.0.0.1:52213 &
          for i in $(seq 1 60); do curl -sf -o /dev/null http://127.0.0.1:52213/ && exit 0; sleep 1; done
          exit 1
        env:
          ASPNETCORE_ENVIRONMENT: Development

      # HTML as in production (canonical https://bookstore.example), requests go to the local app
      - name: SEO crawl
        id: crawl
        continue-on-error: true
        run: dotnet run --project tools/Bookstore.SeoAudit -- http://127.0.0.1:52213/ https://bookstore.example/ seo-report.md

      - name: Job summary
        if: always()
        run: cat seo-report.md >> "$GITHUB_STEP_SUMMARY"

      # PRs from forks don't get write permission on the token; comment only on branches of the same repo
      - name: PR comment
        if: always() && github.event.pull_request.head.repo.full_name == github.repository
        run: gh pr comment "$PR" --body-file seo-report.md --edit-last --create-if-none
        env:
          GH_TOKEN: ${{ github.token }}
          PR: ${{ github.event.pull_request.number }}

      - name: Fail on errors
        if: steps.crawl.outcome == 'failure'
        run: exit 1

I checked the workflow with actionlint 1.7.12 and it reported no errors. A few details: the crawl step runs with continue-on-error, so even when there are errors the report gets written and lands on the PR, and the job breaks in the last step. The --edit-last --create-if-none options of gh pr comment update the bot's last comment instead of opening a new one on every push. The job gets pull-requests: write permission to write comments; for PRs from forks the GITHUB_TOKEN doesn't get that permission, so the comment step only runs on PRs opened from the same repo, while the job summary is written in every case.

This section is based on the official documentation.

Screaming Frog SEO Spider

Screaming Frog is the most widely used desktop crawler for technical SEO audits. The free version is limited to 500 URLs per crawl, crawls can't be saved, and configuration and advanced features require a licence; according to the product's FAQ and user guide, JavaScript rendering is among the licensed features as well. A licence removes the URL limit.

In JavaScript rendering mode the tool renders pages in a headless Chrome and can store both the raw and the rendered HTML. Its JavaScript tab flags, page by page, fields such as the title, meta description, H1, canonical or robots directives that only appear after rendering or are changed by JavaScript; it also shows the word count difference between the raw and the rendered HTML. It's a tool that asks the "what's in the initial HTML?" question from the eighth post of the series for the whole site.

Its relationship with your own crawler is a division of labour, not competition. Screaming Frog is strong for exploration: seeing all the problems of an unfamiliar site for the first time, browsing through charts, starting with hundreds of ready-made checks. Your own crawler is for regression: checking a few rules specific to your site on every PR, free and repeatable.

This section is based on the official documentation.

Bing Webmaster Tools Site Scan

Site Scan is an on-demand site audit tool inside Bing Webmaster Tools; for a verified site, Bing's crawler walks the site and reports common technical SEO issues. According to the help page, a scan starts with one of three scopes: walking the site through its links like Bingbot (all subdomains can be included), scanning the URLs in a single sitemap, or a list of URLs. The maximum number of pages to scan, a link depth limit, requests per second and URL parameters to ignore can be configured; ignoring robots.txt is an option too. Bing calculates the monthly scan quota for a site from its own signals and doesn't accept requests to increase it.

The report sorts issues into three severity levels: errors that can affect indexability, warnings that can affect SEO health, and the lowest-priority notices. The pages affected by each issue are sorted by link depth and can be downloaded as CSV; the results of up to five scans are kept. Scans use the Bingbot user agent. What it means for Bookstore: a second opinion that needs no setup, showing not what Google can't see but how Bing's own crawler sees the site.

This section is based on the official documentation.

The Search Console URL Inspection API

Your own crawler and Screaming Frog show what the site publishes; only Google can say what it does with it. The API of Search Console's URL Inspection tool is a POST to https://searchconsole.googleapis.com/v1/urlInspection/index:inspect: the body carries the URL to inspect (inspectionUrl), the Search Console property (siteUrl; with a trailing / for URL-prefix properties, in the form sc-domain:bookstore.example for domain properties) and an optional message language (languageCode). Authorization is OAuth with the webmasters or webmasters.readonly scope.

The indexStatusResult in the response is the most valuable part for an audit: verdict (PASS, FAIL, NEUTRAL), coverageState, robotsTxtState, indexingState, pageFetchState, lastCrawlTime, googleCanonical and userCanonical. Comparing the last two answers, in bulk, the question from the fifth post of the series: "is the canonical Google chose the same as the one you declared?" The limits are clear too: according to the method documentation, only the status of the version in Google's index is returned, and a live URL test isn't possible; the quota is 2,000 requests per day and 600 per minute per property. On a site like Bookstore with thousands of books, that means checking a sample rather than every URL: newly published books, pages the crawler warned about, and a few examples from each template.

This section is based on the official documentation.

In Production

Run the crawler against production too, but slowly and identifying itself. The CI scan runs against the local app; reverse proxies, CDNs and server rules in production (like the migration verification in the twelfth post of the series) can give different results. In a production scan, limit the request rate and put the tool's name in the user agent.

Tune the rules to the structure of the site, but write down why. In Bookstore paginated lists are exempt from duplicate titles because Google's pagination documentation allows it, and language versions are exempt because they're linked with hreflang. An exemption without a written reason will eventually cover up real problems too.

Separate warnings from errors. Broken links and 404s in the sitemap should stop a deploy; an orphan page or a missing description should show up on the PR as a warning but not block the merge.

Common Pitfalls

Making the Lighthouse SEO score the target. 100 shows that the page follows the basic rules; it says nothing about working links, the structure of the site or rankings.

Crawling staging and generalizing the result to production. Staging has a different robots.txt, base URL and data.

Trying to fix every warning of an audit tool without questioning it. First check whether the rule makes sense on your site.

Running the tool only once, before a migration or a big change. An audit pays off when it runs on every deploy.

Trying to scan the whole site with the URL Inspection API. The daily quota isn't enough for that; the API is for the one thing a crawler can't find, Google's decision.

A Small Experiment

★☆☆ Run Lighthouse's SEO category on three different templates of your own site, then request the links leaving those pages one by one with curl -s -o /dev/null -w "%{http_code}". Is there a broken link on a page Lighthouse scored 100?

★★☆ Add a redirect chain rule to Auditor: an internal link that goes through two or more hops is an error, a single hop a warning. Add the chain /old2 -> /old -> /a to the fake site in AuditorTests and update the test.

★★★ Add a "Google's eye" mode to the crawler: at the end of the crawl, query up to 50 of the pages that produced warnings through the URL Inspection API and report the ones where googleCanonical differs from userCanonical. Use a rate limiter that keeps requests below 600 a minute so you stay within the quota, and provide the API credentials only as a CI secret.

The Technical SEO series ends here. We started with URL design; we managed crawling and indexing with robots.txt, noindex, sitemaps and canonicals, built the server's response correctly with status codes, redirects and JavaScript rendering, and grew to scale with Angular SSR, hreflang, crawl budget and site migrations; in the last block we added images, video, catalog filters and an audit that checks all of it on every deploy. At every step we relied on Google's own documentation and Bookstore's real responses. Natural next stops from here: the Structured Data and Schema.org series for the appearance on the results page, the Web Performance series for page speed, the Content SEO series for the content itself, the GEO series for how AI answer engines read the site, and the Analytics and Measurement series for measuring the impact of all of this.