Measuring the Impact of an SEO Change: Before/After Analysis, Seasonality, Annotations and Honest Reporting

Version note: The code in this post was built and run with .NET 10; all data in the examples is synthetic. This post is based on the official documentation: Google's traffic-drop guide, the website testing guide, and the Search Console annotation and data anomalies pages, as of 15 November 2026.

Guess First

Two changes were made on the blog of Bookstore, the online bookstore used as the shared example across this site's series. On 2 September, Article structured data with dateModified was added to two posts; four weeks later those two posts had 43 percent more clicks. On 14 October, three posts got new titles; four weeks later those three posts had 12 percent more clicks. The team wants to report both as wins. Which one really worked? Both, or neither? Which other number would you look at to decide?

Hypothesis First, Then the Change

Measuring an effect starts before the change. Before making it, write down four things: which pages change, which metric you expect to move, in which direction and roughly by how much, and how long you will wait. For the title rewrite the hypothesis could be: "On these three posts CTR will rise while impressions stay the same; after four weeks we will compare against pages that were not changed." A hypothesis written afterwards makes it far too easy to build a story that explains whatever the data shows.

Then mark the change. Since 17 November 2025 you can add annotations to the Search Console Performance chart: up to 120 characters, 200 notes per property, notes older than 500 days are deleted; owners and full users add and delete them, restricted users only see them. The note is shared with every user, so write no personal information in it. Do not leave the annotation as the only record: keep a change file in the repository and match it with your deploy log. In Bookstore this file is a simple CSV:

date,description,pages
2026-09-02,Article structured data with dateModified on two posts,/blog/cs8618-nullable-uyarisi|/blog/linq-groupby
2026-10-14,title rewrite on three posts,/blog/ef-core-migration-rehberi|/blog/dependency-injection-nedir|/blog/async-await-nasil-calisir

Finally, wait. Google's traffic-drop guide says some changes show their effect within days and others within months, recommends waiting a few weeks before reassessing, and adds that even a good change does not guarantee an improvement. For Core Web Vitals changes, CrUX's 28-day window already means a waiting period.

Seasonality and Outside Factors

What breaks a before/after comparison most is that things outside your control change in the same period as your change. Typical ones for a technical blog:

• Day of the week: traffic drops on weekends. Choose whole-week windows and, as the help page advises, do not compare a Monday with a Saturday.
• Holidays and periods: public holidays, religious holidays, summer vacation and schools reopening. A rise in September often comes from the calendar, not from something you did.
• Core updates and other ranking changes: check Google's ranking update history; covered in the SEO Fundamentals post "How Google Ranks".
• Data corrections: Search Console's Data anomalies page. For example, the impression logging error between 13 May 2025 and 27 April 2026 affected impressions, CTR and position.
• Measurement changes: when the consent banner, the tag or the log format changes, the chart breaks without any change in traffic; we saw this in the fifth post.
• Interest itself: Google's guide suggests checking Google Trends to see whether a drop is specific to you or part of a broader trend.

Before/After with a Control Group

There is a simple way to remove most of these factors: track similar pages that were not changed, as a control group, over the same period. If the season, a holiday or an update affects both groups together, the ratio of the test group to the control group does not change. The analysis below does exactly that: it leaves out the change day, takes four whole weeks before and after, and divides the change of the test group by the change of the control group:

    // The change day itself is left out. Windows are whole weeks, so each weekday counts equally.
    public static ImpactResult Analyze(
        IReadOnlyList<DailyPage> days, IReadOnlySet<string> testPages, DateOnly changeDay,
        int windowWeeks = 4, int baselineWeeks = 12)
    {
        DateOnly afterFrom = changeDay.AddDays(1);
        DateOnly afterTo = afterFrom.AddDays(windowWeeks * 7 - 1);
        DateOnly beforeTo = changeDay.AddDays(-1);
        DateOnly beforeFrom = beforeTo.AddDays(-(windowWeeks * 7) + 1);

        GroupWindow Sum(bool test, DateOnly from, DateOnly to)
        {
            List<DailyPage> rows = days
                .Where(d => d.Date >= from && d.Date <= to && testPages.Contains(d.Page) == test)
                .ToList();
            return new GroupWindow(rows.Sum(d => d.Clicks), rows.Sum(d => d.Impressions));
        }

        double WeekRatio(DateOnly weekStart)
        {
            GroupWindow test = Sum(true, weekStart, weekStart.AddDays(6));
            GroupWindow control = Sum(false, weekStart, weekStart.AddDays(6));
            return (double)test.Clicks / control.Clicks;
        }

        GroupWindow testBefore = Sum(true, beforeFrom, beforeTo);
        GroupWindow testAfter = Sum(true, afterFrom, afterTo);
        GroupWindow controlBefore = Sum(false, beforeFrom, beforeTo);
        GroupWindow controlAfter = Sum(false, afterFrom, afterTo);

        // Test clicks relative to control clicks: shared effects (season, holiday, update) cancel out.
        double effect = (double)testAfter.Clicks / testBefore.Clicks
                        / ((double)controlAfter.Clicks / controlBefore.Clicks) - 1;

        List<double> baseline = Enumerable.Range(1, baselineWeeks)
            .Select(w => WeekRatio(changeDay.AddDays(-7 * w)))
            .ToList();
        List<double> after = Enumerable.Range(0, windowWeeks)
            .Select(w => WeekRatio(afterFrom.AddDays(7 * w)))
            .ToList();

        return new ImpactResult(testBefore, testAfter, controlBefore, controlAfter, effect, baseline, after);
    }
}

A single ratio is not enough, because on a small site the ratio already moves from week to week. So the code computes the test/control ratio for each of the 12 weeks before the change and compares the ratios of the four weeks after it against that range. If all of them are outside the range, it says the effect stands out from normal variation; if not, it does not.

The input is data/daily-pages.csv: daily clicks and impressions for ten pages from June to November 2026. The dataset was produced with tools/synthetic/daily_pages.py and is not data from a real site; but we built a known truth into it: on the three retitled posts, CTR rises by 15 percent within a week after 14 October. Everything else is season, day of the week, the 29 October holiday and random variation. First the structured data change:

$ dotnet run --project src/Bookstore.Analytics.Cli -- impact data/daily-pages.csv data/changes.csv 2026-09-02
change 2026-09-02: Article structured data with dateModified on two posts (2 pages)
windows: 28 days before vs 28 days after, change day excluded

           clicks before   after   change  ctr before   after
test                 501     716   +42.9%       7.09%   7.15%
control             1130    1618   +43.2%       4.30%   4.42%

test vs control, clicks: -0.2%   ctr: -2.0%
weekly test/control ratio, 12 weeks before: 0.412 .. 0.518
weekly test/control ratio, 4 weeks after:  0.438  0.450  0.450  0.431
the weeks after the change stay within normal variation: no evidence of an effect

The test group rose 43 percent, but the control group also rose 43 percent: the seasonal effect we put into the dataset (low in summer, high in September) lifted both groups together; on a real site this corresponds to the end of the summer vacation. The test/control ratio did not change, and the four weeks after stay within the range of the 12 weeks before. We did not put an effect here, and the analysis does not find one. Reporting this change as a win would be reporting the calendar as a win. What structured data does and does not affect is covered in detail in the Structured Data and Schema.org series.

Then the title change:

$ dotnet run --project src/Bookstore.Analytics.Cli -- impact data/daily-pages.csv data/changes.csv 2026-10-14
change 2026-10-14: title rewrite on three posts (3 pages)
windows: 28 days before vs 28 days after, change day excluded

           clicks before   after   change  ctr before   after
test                 994    1110   +11.7%       4.25%   4.77%
control             1372    1415    +3.1%       5.68%   5.78%

test vs control, clicks: +8.3%   ctr: +10.3%
weekly test/control ratio, 12 weeks before: 0.665 .. 0.781
weekly test/control ratio, 4 weeks after:  0.744  0.788  0.726  0.883
the weeks after the change stay within normal variation: no evidence of an effect

Here the result is subtler. Against the control group, the test group is 8.3 percent ahead in clicks and 10.3 percent in CTR; one of the four weeks is outside the earlier range, three are inside. The analysis concludes "not enough evidence". Yet the data contains a real 15 percent CTR increase. That is not a flaw of the method but the reality of a small site: a few hundred clicks a week on three posts are not enough to separate a 10-15 percent effect from random variation in four weeks. The honest options are to extend the window, apply the change to more pages, or report "there is a positive sign, but it is not confirmed".

The tests pin down the code's two key behaviors: a 50 percent seasonal lift that hits both groups does not show as an effect, while a real doubling stands out from normal variation:

$ dotnet test --filter "FullyQualifiedName~ChangeImpactAnalyzerTests"
Passed!  - Failed:     0, Passed:     3, Skipped:     0, Total:     3, Duration: 8 ms - Bookstore.Analytics.Tests.dll (net10.0)

SEO Split Tests and the Limits of a Small Site

The more structured version of the control group idea is an SEO split test: randomly split similar pages into two groups and change only one of them. Here Googlebot and users see the same page, so cloaking is not an issue. For user-based A/B tests, Google's website testing guide applies: do not show Googlebot different content, give variant URLs a rel="canonical" to the original, use 302 instead of 301 if you redirect to a variant, and do not keep the test running longer than needed; since Googlebot generally does not accept cookies, in cookie-based tests it sees the variant a cookieless browser gets.

Split tests work well on large sites, because there are hundreds of similar pages and thousands of clicks a day. On a site of a few dozen posts such as the Bookstore blog, the groups stay small and the problem we just saw gets bigger. On a small site the realistic goal is to catch large effects and to accept saying "we don't know" for small ones.

Building the Weekly Report from the Archive

The analysis input comes in the same shape out of the SQL Server archive we filled from the Search Console API in the third post of the series:

SELECT DataDate AS [date], Page AS page, SUM(Clicks) AS clicks, SUM(Impressions) AS impressions
FROM dbo.SearchAnalyticsDaily
WHERE SearchType = 'web'
GROUP BY DataDate, Page
ORDER BY DataDate, Page;

The weekly job first writes the new days to the archive, then saves this query's output as CSV and runs the impact command for every change in changes.csv that is at least 28 days old. Because the query comes from the detail table, it does not include anonymized queries; since the test and control groups are affected the same way, the ratio comparison is largely unaffected, but note it when you report absolute numbers.

Honest Reporting

Pick the metric according to who reads the report. For management, the non-branded click trend from Search Console and a few sentences of conclusion; for the team, detail at the page and query level. What every impact report needs:

• The hypothesis and the change date, with its record in the change file.
• The windows: how many days, which days, whether the change day is excluded.
• The test and control groups, the raw numbers of both and the ratio.
• The range of normal variation and where the result sits against it.
• Outside factors in the same period: updates, holidays, data anomalies, measurement changes.
• The wording of the result: "it worked", "there is a sign but it is not confirmed" or "no effect seen".

The level of evidence matters especially for claims. Take the sentence "our CTR fell because of AI Overviews": Search Console shows AI Overviews and AI Mode data inside the "Web" search type and there is no way to separate those clicks; the generative AI report gives impressions only. On top of that, the 2025-2026 impression correction changed CTR separately. In this situation what you have is at most a coincidence in time, not causation. The honest report says: "CTR fell in the period when AI Overviews became widespread; Search Console data does not let us separate the two." The AI side of measurement is in the GEO post "Measuring GEO".

Production Notes

Make changes one after another. Leave at least one measurement window between two changes on the same pages; the change file makes that order visible.

Choose the control group in advance. Saying "these pages will be the control" after seeing the results means choosing the result you want.

Keep the raw data. Attach the CSV the report relies on and the command you used; months later, when someone asks, you should be able to produce the same number again.

Update the result. Look again in week eight at a change that came out "not enough evidence" in week four; if no new change came in between, uncertainty shrinks as the window grows.

Common Pitfalls

Counting a one-day spike as a win. A single day's clicks can jump because of news, a social media share or a bot; work with whole weeks.

Making five changes at once. If titles, structured data, internal links and images change in the same week, no analysis can tell which one worked.

A before/after comparison without a control group. As the September example showed, the calendar is the strongest "change".

Confusing "no effect" with "no evidence". An effect you cannot measure on a small site is not an effect that does not exist; write the two separately in the report.

A Small Experiment

★☆☆ Raise the effect in tools/synthetic/daily_pages.py from 15 percent to 40 percent, regenerate the CSV and run the impact command. Does the result change?

★★☆ Make the window length a parameter of ChangeImpactAnalyzer and compare the title change with 2, 4 and 6-week windows. In which window does it first stand out from normal variation?

★★★ Add a permutation test instead of the weekly ratio: shift the change date randomly into earlier weeks and count how often an effect of the same size appears. Test it with synthetic data and report the result as "in how many of how many tries".

The Analytics and Measurement series ends here. We have seen which question each data source answers, how to read and automate Search Console's numbers, how to set up GA4 correctly in an SPA, how consent affects the numbers, the field data behind Core Web Vitals and how to measure the effect of a change honestly. The common conclusion is simple: write the question first, know where every number comes from, and where you are not sure, say so in the report.