Deciding with Experiments: A/B Tests, Feature Flags and Statistical Pitfalls
An A/B test is trustworthy when its sample is computed before the experiment and you wait for it. We test Bookstore's shipping cost hypothesis with a Microsoft.FeatureManagement variant flag and see in C# output that it needs 3,954 carts per arm, how the SRM check works, the confidence interval of a 2.24-point lift, and how peeking daily raises false positives in no-effect A/A tests from 5% to 32%.
Version note: The code in this post was built and run with .NET 10 and Microsoft.FeatureManagement 4.8.0.
Guess First
The Bookstore team started testing the hypothesis from post 02: does showing the shipping cost on the product page raise the share of people who complete an order after adding to the cart? Half of the visitors see the old page, half the page with the shipping cost. At the end of week one the dashboard shows: in the control group 33.0% of those who added to the cart placed an order, on the new page 36.1%; p-value 0.03.
The product owner is excited: "It's statistically significant, let's stop the experiment and roll it out to everyone." Is that the right decision? And consider this: the team has looked at this dashboard every morning since the first day of the experiment. Does that change what the p-value means?
From Hypothesis to Experiment
An A/B test, a controlled experiment, is the strongest way to test a hypothesis: you split users randomly into two groups, show the change to one and not to the other; thanks to random assignment, the cause of the difference between them is the change itself. But whether the experiment works depends on a few decisions made before writing the code.
The first decision is the primary metric: a single metric determines the experiment's success, and it isn't picked afterward. For Bookstore it's the conversion from the cart_item_added step to the order_placed step in the funnel from post 07; today's value is 33.3%. The second decision is the guardrails: average basket value and the product page's 95th percentile response time must not get worse. The third is the randomization unit: the user, not the session, because one person seeing the shipping cost on one visit and not the next day breaks both the experience and the measurement. The fourth is duration and sample: how many users, for how long? That answer is computed before the experiment.
Kohavi, Tang and Xu's book "Trustworthy Online Controlled Experiments" (2020) discusses each of these decisions in depth; this post is an introduction, and that book is the reference for the topic.
Sample Size and Minimum Detectable Effect
How many users an experiment needs depends on three things: the baseline rate, the smallest difference you want to detect, the MDE (Minimum Detectable Effect), and the error rates you accept. The common choice is alpha = 0.05 (the probability of saying "there is a difference" when there isn't) and power = 0.80 (the probability of catching a difference as large as the MDE when it exists). Since our hypothesis says "at least three points," the MDE is three points.
The formula is the standard approach for a difference of two proportions: n per arm = (z(1 − α/2) + z(1 − β))² × [p1(1 − p1) + p2(1 − p2)] / (p2 − p1)². In the code below, SampleSizePerArm computes this: for a 33.3% baseline and a three-point MDE, 3,954 carts per arm, 7,908 in total. According to the funnel in post 07, 5,192 people add to the cart at Bookstore in four weeks, about 1,300 a week. The experiment has to run for at least six weeks.
This calculation also says why the team shouldn't decide in the first week: one week has only 1,300 carts, a sixth of the required sample. Had you wanted to halve the MDE to 1.5 points, the sample would roughly quadruple; detecting small effects gets expensive with the square of the traffic.
Code: Assignment with a Variant Feature Flag
To assign users to variants we use Microsoft.FeatureManagement's variant feature flags. The flag mechanics, the details of IVariantFeatureManager and the ASP.NET Core integration were covered in the Deployment Strategies series' feature flag post; here we use only what experiment design and analysis need. Configuration:
{
"feature_management": {
"feature_flags": [
{
"id": "ShippingCostOnProductPage",
"enabled": true,
"variants": [
{ "name": "Control" },
{ "name": "ShowShipping" }
],
"allocation": {
"percentile": [
{ "variant": "Control", "from": 0, "to": 50 },
{ "variant": "ShowShipping", "from": 50, "to": 100 }
],
"seed": "shipping-cost-2026-11",
"default_when_enabled": "Control"
}
}
]
}
}allocation.percentile spreads users over a 0–100 range and gives 0–50 to Control and 50–100 to ShowShipping. The spread is based on a deterministic hash computed from the user id and the seed value; the same user gets the same variant on every call, and changing the seed reshuffles users for the next experiment. The statistics helpers:
namespace Bookstore.Experiments;
public sealed record ProportionTest(double Diff, double Low, double High, double P);
public static class Stats
{
// CDF of the standard normal distribution (Abramowitz-Stegun 7.1.26 erf approximation, error < 1.5e-7)
public static double NormalCdf(double z)
{
double x = Math.Abs(z) / Math.Sqrt(2);
double t = 1 / (1 + 0.3275911 * x);
double erf = 1 - ((((1.061405429 * t - 1.453152027) * t + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.Exp(-x * x);
return z >= 0 ? 0.5 * (1 + erf) : 0.5 * (1 - erf);
}
// z test for the difference of two proportions; two-sided p-value
public static ProportionTest TwoProportions(int c1, int n1, int c2, int n2)
{
double p1 = (double)c1 / n1, p2 = (double)c2 / n2;
double pooled = (double)(c1 + c2) / (n1 + n2);
double z = (p2 - p1) / Math.Sqrt(pooled * (1 - pooled) * (1.0 / n1 + 1.0 / n2));
double se = Math.Sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2);
return new ProportionTest(p2 - p1, p2 - p1 - 1.96 * se, p2 - p1 + 1.96 * se, 2 * (1 - NormalCdf(Math.Abs(z))));
}
// Sample needed per arm: alpha 0.05 (two-sided), power 0.80
public static int SampleSizePerArm(double baseline, double mde) =>
(int)Math.Ceiling(Math.Pow(1.959964 + 0.841621, 2)
* (baseline * (1 - baseline) + (baseline + mde) * (1 - baseline - mde)) / (mde * mde));
// Sample ratio mismatch: chi-square against the expected 50/50 split (1 degree of freedom)
public static double SrmPValue(int a, int b)
{
double expected = (a + b) / 2.0;
double chi2 = Math.Pow(a - expected, 2) / expected + Math.Pow(b - expected, 2) / expected;
return 2 * (1 - NormalCdf(Math.Sqrt(chi2)));
}
}The program does four steps in order: it computes the required sample, assigns that many simulated users to variants through the flag (we simulate the true effect as +3 points), analyzes the result, and finally measures the effect of peeking with 2,000 A/A experiments. The project's .csproj has to copy appsettings.json to the output folder (<None Update="appsettings.json" CopyToOutputDirectory="PreserveNewest" />); we forgot that on the first run and got a FileNotFoundException.
using System.Globalization;
using Bookstore.Experiments;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.FeatureManagement;
using Microsoft.FeatureManagement.FeatureFilters;
CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;
IConfiguration config = new ConfigurationBuilder().AddJsonFile("appsettings.json").Build();
ServiceCollection services = new();
services.AddSingleton(config);
services.AddFeatureManagement();
await using ServiceProvider provider = services.BuildServiceProvider();
IVariantFeatureManager features = provider.GetRequiredService<IVariantFeatureManager>();
// 1) Before the experiment: how many users do we need?
double baseline = 0.333; // 07'deki funnel: cart_item_added -> order_placed
double mde = 0.03; // smallest difference we want to detect: 3 points
int perArm = Stats.SampleSizePerArm(baseline, mde);
Console.WriteLine($"Needed per arm: {perArm} carts (baseline {baseline * 100:F1}%, MDE {mde * 100:F0} points)");
// 2) Assignment: the same user always gets the same variant
Dictionary<string, int> assigned = new() { ["Control"] = 0, ["ShowShipping"] = 0 };
Dictionary<string, int> converted = new() { ["Control"] = 0, ["ShowShipping"] = 0 };
Random outcome = new(11);
for (int i = 1; i <= 2 * perArm; i++)
{
TargetingContext user = new() { UserId = $"customer-{i:D6}" };
Variant variant = await features.GetVariantAsync("ShippingCostOnProductPage", user, CancellationToken.None);
assigned[variant.Name]++;
// Simulation: let the true effect be +3 points
double rate = variant.Name == "ShowShipping" ? baseline + 0.03 : baseline;
if (outcome.NextDouble() < rate)
converted[variant.Name]++;
}
Variant again = await features.GetVariantAsync("ShippingCostOnProductPage", new TargetingContext { UserId = "customer-000042" }, CancellationToken.None);
Console.WriteLine($"customer-000042 -> {again.Name} (stable on every call)");
Console.WriteLine($"Assigned: Control {assigned["Control"]}, ShowShipping {assigned["ShowShipping"]}, " +
$"SRM p = {Stats.SrmPValue(assigned["Control"], assigned["ShowShipping"]):F3}");
// 3) Analysis: difference, 95% confidence interval, p-value
ProportionTest r = Stats.TwoProportions(converted["Control"], assigned["Control"], converted["ShowShipping"], assigned["ShowShipping"]);
Console.WriteLine($"Control {100.0 * converted["Control"] / assigned["Control"]:F1}%, ShowShipping {100.0 * converted["ShowShipping"] / assigned["ShowShipping"]:F1}%");
Console.WriteLine($"Difference {r.Diff * 100:F2} points, 95% CI [{r.Low * 100:F2}, {r.High * 100:F2}], p = {r.P:F4}");
// 4) Peeking: 2,000 A/A experiments with NO effect; what if we look daily and stop at p < 0.05?
Random aa = new(7);
int days = 42, perDay = (int)Math.Ceiling(2.0 * perArm / days);
int stoppedEarly = 0, significantAtEnd = 0;
for (int run = 0; run < 2000; run++)
{
int cA = 0, nA = 0, cB = 0, nB = 0;
bool stopped = false;
for (int day = 1; day <= days; day++)
{
for (int k = 0; k < perDay; k++)
{
if (aa.Next(2) == 0) { nA++; if (aa.NextDouble() < baseline) cA++; }
else { nB++; if (aa.NextDouble() < baseline) cB++; }
}
if (!stopped && Stats.TwoProportions(cA, nA, cB, nB).P < 0.05)
stopped = true;
}
if (stopped) stoppedEarly++;
if (Stats.TwoProportions(cA, nA, cB, nB).P < 0.05) significantAtEnd++;
}
Console.WriteLine($"A/A tests with no real effect: false positives looking once at the end = {significantAtEnd / 20.0:F1}%, " +
$"peeking daily = {stoppedEarly / 20.0:F1}%");The real output:
Needed per arm: 3954 carts (baseline 33.3%, MDE 3 points)
customer-000042 -> Control (stable on every call)
Assigned: Control 4008, ShowShipping 3900, SRM p = 0.225
Control 33.5%, ShowShipping 35.7%
Difference 2.24 points, 95% CI [0.14, 4.33], p = 0.0367
A/A tests with no real effect: false positives looking once at the end = 5.2%, peeking daily = 31.9%Sample Ratio Mismatch
Before the analysis, the first check is whether assignment happened as expected. In a 50/50 split, 4,008 of 7,908 users landed in Control and 3,900 in ShowShipping. Is that difference chance? The chi-square test says p = 0.225: a difference of this size is common with random assignment, no problem.
SRM (Sample Ratio Mismatch) raises the alarm when this test's p-value is very small, say below 0.001. If there is an SRM, the experiment's result can't be trusted, because the groups are no longer random. The causes are usually technical: the new variant's page loads more slowly so some users' events never arrive, a bot filter affects one variant more than the other, a CDN (Content Delivery Network) cache shows one variant's page to everyone, or the consent banner behaves differently in one variant. The paper by Fabijan and colleagues from Microsoft's experimentation platform team, published at KDD (Knowledge Discovery and Data Mining) 2019, shows how common SRM is and gives a taxonomy of its causes. The rule is simple: if there's an SRM, don't interpret the result; find the cause first.
Analysis: Difference, Confidence Interval, p-value
Conversion is 33.5% in Control and 35.7% in ShowShipping: a difference of 2.24 points. The 95% confidence interval is [0.14, 4.33], p = 0.0367. We read it like this: the data supports that showing the shipping cost raises conversion; the size of the true effect could be as small as 0.1 points or as large as 4.3.
This result has two subtle points. First, in the simulation we set the true effect to +3 points, yet the measured difference is 2.24. That is exactly sampling error, and it shows a single experiment's point estimate can be far from the true value. Second, the lower bound of the confidence interval is very close to zero. The team hasn't proven the "raises it by three points" hypothesis; it has reached "there is an increase, its size is uncertain."
The decision isn't left to the p-value alone. The guardrails are checked: did average basket value drop, did the product page slow down? We didn't model guardrails in this simulation; in a real experiment they're reported in the same table, and if one has clearly gotten worse, the gain in the primary metric isn't enough on its own. Cost counts too: calculating the shipping cost on the product page has a maintenance cost. With this result the Bookstore team rolled the change out to everyone and decided to watch the funnel for the next four weeks.
The Peeking Problem
Back to the opening story: the team looked at the dashboard every morning from day one and wanted to stop the first time it saw p < 0.05. The last line of the program shows why that's dangerous. We simulated 2,000 A/A experiments with no real difference at all: both groups convert at the same 33.3%. Looking once at the end of the experiment, 5.2% of the experiments came out "significant" by mistake; that's exactly what alpha = 0.05 promises. Looking every day and stopping on the first day p < 0.05 appeared, 31.9% of the experiments were wrongly declared "winners."
Here's why: a p-value is computed as if you'll look once at a sample size fixed in advance. Every look is a new chance, and random fluctuation tends to dip below the threshold at some point during the experiment. The probability of dipping at least once in forty-two looks is far higher than five percent.
The fix is either to fix the sample in advance and not decide until you reach it, or to use sequential testing methods designed for continuous monitoring. Looking at the dashboard isn't forbidden; you should look to see whether a guardrail is in disaster. What's forbidden is looking at the primary metric and declaring early victory.
When Not to Run an A/B Test
A/B tests are powerful, but they aren't the right tool for every question. With low traffic an experiment takes months: at Bookstore an experiment targeting only guest users needs about 1.4 times as long, eight or nine weeks, for the same MDE, because only two thirds of those adding to the cart are guests and the guests' baseline rate (27.9%) is different. If you want to detect a smaller effect, the duration stretches into years. In that case the qualitative methods from post 03, interviews and prototype tests, give more information, faster, than an experiment.
Irreversible changes aren't suited to experiments. A pricing policy, a brand change or a promise made to customers can't be split into two groups and then rolled back; when the experiment ends you can't tell the control group "actually this wasn't for you."
There are ethical and regulatory limits. Showing different prices for the same product to different users is risky for customer trust and consumer law. For experiments involving personal data, the KVKK (Turkey's Personal Data Protection Law) and GDPR (General Data Protection Regulation) checklist from post 07 applies as is. When in doubt, ask the legal team.
In Production
Write the experiment plan before the experiment: hypothesis, primary metric, guardrails, MDE, sample, duration and decision rule. This single page prevents the argument "actually we should have looked at that other metric" when the result arrives. The success metric section of the one-pager in post 09 carries exactly this.
Write the assignment into events. If the variant a user landed in isn't stored as a property in the event table from post 07 (e.g. experiment and variant), you can't split the funnel by variant during analysis. Adding these properties to the taxonomy is a prerequisite of the experiment.
Remove the flag when the experiment ends. Once the winning variant is rolled out to everyone, the flag and the control arm's code should enter the backlog as a cleanup item; experiment flags left in code for years are the most common source of the flag debt described in the Deployment Strategies series.
Common Pitfalls
The novelty effect: users react more to something new in the first days simply because it's new. A change's effect can look big in the first week and then fade. The opposite happens too: users who don't want to give up their habits resist a new design in the first days. Running the experiment for at least one or two full weeks and plotting the effect over time lets you see both effects.
Fishing in segments afterward: when the result isn't significant, looking segment by segment, "what about only mobile users? only Istanbul? only Tuesdays?", will always find a "significant" result somewhere if you look at enough segments. If segments weren't written into the plan before the experiment, what you find is a hypothesis for a new experiment, not evidence.
Looking at many metrics and picking one is another form of the same mistake. If you look at twenty metrics, one is expected to cross the 5% threshold even when there's no effect at all. Choose the primary metric in advance.
Finally, reading a non-significant result as "no effect." An experiment with 80% power misses a real effect one time in five. In our first attempt, testing one-step checkout instead of the shipping cost, a real +3 point effect ended at p = 0.055, just above the threshold. "We found no evidence" and "there is no effect" are not the same sentence.
A Small Experiment
★☆☆ Lower the MDE in the program from 0.03 to 0.015 and find the required sample. With Bookstore's 1,300 carts a week, how many weeks would this experiment take?
★★☆ Deliberately set the allocation in appsettings.json to 0–55 and 55–100, but run the analysis expecting 50/50. Does the SRM test catch it? From how many users does it start catching it?
★★★ Add a simple correction to the peeking simulation: on daily looks use a much stricter threshold than p < 0.05 (e.g. 0.005), and 0.05 again on the last day. What happens to the false positive rate? This is a rough version of the idea behind sequential testing methods.
We've seen deciding with experiments. In the last post of the series we'll discuss how a decision coming out of discovery turns into backlog items: the one-pager and PRD (Product Requirements Document) skeleton, the success metric and non-goal sections, and the questions a developer should ask in a PRD review, through Bookstore's cart abandonment work.