Story Points and Relative Estimation: Planning Poker, T-Shirt Sizing and the Contested Legacy of Story Points

Version note: This post is based on the Scrum Guide 2020 and Ron Jeffries' Story Points Revisited (2019); the calculation example was built and run with .NET 10.

Make a Guess First

Thursday, 19 November, refinement meeting. Selin, the Product Owner, reads the Institutional Orders stories of Bookstore, the online bookstore used as the shared example across this site's series, one by one. Next up: "As a purchasing officer I want to paste my book list and see the validation result for every row." Emre and Ayşe turn over their cards at the same time. Emre's says 3, Ayşe's says 8.

A moment of silence. Selin asks: "The middle is 5, shall we write 5?" Do you think taking the average is right? What could explain the gap between the two numbers: did one of them misunderstand the work, or are they thinking about different work? And one more: how much calendar time does this 8 correspond to? Four days, two weeks, or is the question itself wrong?

Why Relative Estimation?

People are bad at estimating absolute sizes and fairly good at comparing. Saying how tall a building is in meters is hard; seeing that the building next to it is twice as tall is easy. In software, "how many hours does this story take?" asks you to squeeze interruptions, review, who will do it and that week's meetings into one number. "Is this story bigger or smaller than the coupon story we did last month?" looks only at the work itself.

That is why relative estimation separates the size of the work from its duration. The team orders stories relative to each other and derives duration later from historical data: how many units of work does the team finish in an average sprint? Part of the optimism we saw in the first post drops out this way, because nobody says "I can do this in three days"; past speed already includes interruptions and forgotten work.

Where Did Story Points Come From?

The best account of the story point's history comes from Ron Jeffries, one of its first users, in his 2019 post Story Points Revisited. As Jeffries tells it, Extreme Programming teams first estimated stories in time and then in "ideal days". An ideal day was how long a pair would take if nobody bothered them. A load factor converted this to real calendar time, and it was usually around 3: one ideal day of work took three real days.

The problem was that people outside the work did not understand the factor. "Why does a day of work take three days?" kept coming back. So the team started calling ideal days "points"; once the unit was gone, so was the question. Jeffries writes that the main purpose of those points was deciding how much work to take into an iteration. The most quoted sentence of the post is his too: "I like to say that I may have invented story points, and if I did, I'm sorry now." (Ron Jeffries, Story Points Revisited, 2019). Today Jeffries suggests dropping story points, and even estimating stories at all, where possible, and instead finding the most valuable work and slicing stories into small pieces.

The exact "invention" of points is uncertain; Jeffries leaves it open himself with "may have". What is certain is this: points were not born to escape duration, but to escape people outside the work misreading duration estimates. That also explains why nearly all of today's distortions come from trying to turn points back into time.

Planning Poker

James Grenning first described planning poker in a short paper in 2002. The problem he wanted to solve is familiar: in a release planning meeting two people discuss a story for twenty minutes, the rest of the room drifts off, and in the end the estimate does not change. In Grenning's version the customer reads the story, there is a short clarification if needed, every programmer writes an estimate on a card without showing anyone, and the cards are turned over together. If everyone wrote the same number there is nothing to discuss and the team moves on. If the numbers differ, the people with the extreme estimates open the discussion and the team tries to reach consensus. If there is no consensus, Grenning's advice is relaxed: defer the story, split it or take the low estimate; it is one story out of many.

Grenning's deck had cards for 1, 2, 3, 5, 7 and 10 ideal days plus an infinity card. For a story bigger than two weeks you played infinity and asked the customer to split it. The technique spread widely with Mike Cohn's 2005 book Agile Estimating and Planning; common decks today hold values such as 0, 1, 2, 3, 5, 8, 13, 20, 40 and 100, plus a question mark card for "I don't understand this".

The gaps in the scale have a simple reason: the bigger the work, the less precise the estimate. You can feel the difference between 3 and 4; you cannot feel the difference between 13 and 14. Grenning says the same about the gaps in his own deck. The scale saves the team from arguing about precision that isn't there.

But the real value of planning poker is not the number; it is the gap when the cards are turned over. Back to Bookstore's 3 and 8. Ayşe explains: school lists mix ISBN-10 and ISBN-13, the same book shows up on two rows, and some lists have no ISBN at all, just the title. Emre thought of a clean list containing only valid ISBN-13s. Both are right; they are estimating different work. Averaging to 5 would have thrown that information away. The team splits the story in three: pasting and validating an ISBN-13 list is 3, converting ISBN-10 and merging duplicate rows is 2, matching rows without an ISBN by title is 8. The last one is still big and uncertain; we will deal with it in the third post.

T-Shirt Sizing and Affinity Estimation

Not every situation needs numbers. With t-shirt sizing, stories get sizes such as XS, S, M, L and XL. It suits early stages, for roughly comparing the big pieces on a story map; nobody argues about exactly how many points an L is. If the sizes need to become numbers later, the team should do that as a deliberate decision; otherwise sizes quietly turn into points.

Affinity estimation is used to size many stories quickly. Story cards are laid out on a table or a board, and the team silently orders them from small to large; anyone who wants to move a card moves it, and a card that keeps moving is set aside for discussion. The ordered row is then cut into groups and each group gets a size. Discussing the 30 Institutional Orders stories one by one with planning poker takes an afternoon; affinity gives a rough sizing in half an hour, leaving poker for the contested stories.

Both techniques need a reference. The Bookstore team keeps a reference set of finished stories: for 1 point "add the publisher field to the book detail page", for 3 points "coupon code entry in the cart", for 5 points "the return request form", for 8 points "fetching the carrier's tracking number". Every new story is compared with these four. Without a reference set the meaning of a point drifts from sprint to sprint; without noticing, the team gives the same work bigger and bigger numbers.

The Scrum Guide and How Story Points Get Distorted

The Scrum Guide 2020 never mentions story points, velocity or even the word "estimation". When it describes Product Backlog refinement, it says details such as description, order and size are added to items, and that the Developers who will do the work are responsible for the sizing. The Product Owner can influence them by helping them understand trade-offs, but does not give the number. For Sprint Planning it says the better the Developers know their past performance, their upcoming capacity and their Definition of Done, the more confident they will be in their forecasts. So Scrum asks for sizing but leaves the method to the team. What velocity does and does not show inside Scrum is covered in the Scrum series post "Scrum Metrics: What Velocity, Burndown and Sprint Goal Success Do and Don't Tell You".

The contested legacy of story points comes from the distortions that settle into that gap. Three are very common:

Converting points to hours. The moment a "1 point = 4 hours" rule appears, all the benefit of points is gone: the team is estimating time again, just under a different name. And the optimism from the first post returns. If points need to be converted to time, that happens at team level through historical velocity, not at story level.

Comparing teams. Points mean something only relative to each team's own reference set. One team's 40 points is not more work than another team's 20. Management that compares teams by velocity soon gets inflated points.

Making points a performance measure. Once higher velocity becomes the goal, the team makes points bigger; the number goes up, delivered value does not. This is a well-known case of Goodhart's law, the observation that a measure stops being a good measure once it becomes a target, and the Scrum series metrics post covers it in detail.

#NoEstimates: Is Counting Enough?

Woody Zuill started the #NoEstimates hashtag in 2012; Vasco Duarte developed the topic further in a book of the same name. The name is misleading: its advocates do not ban estimation altogether. What they propose is this: slice stories until they are roughly the same size, then count stories instead of summing points. If the number of stories finished per sprint, the throughput, says the same as the point total anyway, the time spent in estimation sessions is saved.

That is a claim, and we can check it against the Bookstore team's data. The points of every story that reached Done in the last 10 sprints are on record. First we put this data and the 30 stories of Institutional Orders Release 1, as sized in planning poker, into a class:

namespace Bookstore.Planning;

public sealed record Sprint(int Number, int[] DonePoints)
{
    public int Velocity => DonePoints.Sum();
    public int Throughput => DonePoints.Length;
}

public static class SprintHistory
{
    // Bookstore team, last 10 sprints (2 weeks): points of every story that reached Done
    public static readonly Sprint[] Last10 =
    [
        new(14, [5, 3, 8, 2, 3]),
        new(15, [3, 5, 2, 5, 2]),
        new(16, [8, 3, 5, 3, 2, 3]),
        new(17, [5, 5, 3, 3, 2, 1]),
        new(18, [8, 3, 3]),
        new(19, [5, 3, 3, 5, 2, 3, 1]),
        new(20, [3, 5, 8, 2, 2]),
        new(21, [5, 3, 5, 2, 3, 5]),
        new(22, [8, 5, 3, 2]),
        new(23, [3, 5, 3, 2, 5, 3, 1]),
    ];

    // Institutional Orders Release 1: 30 stories after planning poker
    public static readonly int[] Release1 =
    [
        1, 2, 2, 2, 2, 2, 3, 3, 3, 3,
        3, 3, 3, 3, 5, 5, 5, 5, 5, 5,
        5, 5, 5, 5, 5, 8, 8, 8, 8, 8,
    ];
}

Then we answer the same question two ways: how many sprints does Release 1 take by point total, and how many by story count?

namespace Bookstore.Planning;

public static class PointsVsCount
{
    public static void Run()
    {
        Sprint[] sprints = SprintHistory.Last10;
        double[] velocity = [.. sprints.Select(s => (double)s.Velocity)];
        double[] throughput = [.. sprints.Select(s => (double)s.Throughput)];

        Console.WriteLine("Sprint  points  stories");
        foreach (Sprint s in sprints)
            Console.WriteLine($"  {s.Number,4}  {s.Velocity,6}  {s.Throughput,7}");
        Console.WriteLine($"  mean  {velocity.Average(),6:F1}  {throughput.Average(),7:F1}");
        Console.WriteLine($"Pearson r (points vs stories) = {Correlation(velocity, throughput):F2}");

        // The same question two ways: how many sprints does Release 1 take?
        int[] backlog = SprintHistory.Release1;
        Console.WriteLine();
        Console.WriteLine($"Release 1: {backlog.Length} stories, {backlog.Sum()} points");
        Console.WriteLine($"  by points : {backlog.Sum() / velocity.Average():F1} sprints");
        Console.WriteLine($"  by count  : {backlog.Length / throughput.Average():F1} sprints");

        // Counting works when the story size mix looks like the past
        int[] done = [.. sprints.SelectMany(s => s.DonePoints)];
        Console.WriteLine();
        Console.WriteLine($"Average story size: history {done.Average():F2}, Release 1 {backlog.Average():F2}");
        foreach (int size in new[] { 1, 2, 3, 5, 8 })
            Console.WriteLine($"  {size} pt: history {Share(done, size),3:F0}%   Release 1 {Share(backlog, size),3:F0}%");
    }

    static double Share(int[] stories, int size) => stories.Count(p => p == size) * 100.0 / stories.Length;

    static double Correlation(double[] x, double[] y)
    {
        double mx = x.Average(), my = y.Average();
        double cov = x.Zip(y, (a, b) => (a - mx) * (b - my)).Sum();
        double sx = Math.Sqrt(x.Sum(a => (a - mx) * (a - mx)));
        double sy = Math.Sqrt(y.Sum(b => (b - my) * (b - my)));
        return cov / (sx * sy);
    }
}

We add case "count": PointsVsCount.Run(); break; to the switch in Program.cs and run it:

dotnet run --project src/Bookstore.Planning -- count

Sprint  points  stories
    14      21        5
    15      17        5
    16      24        6
    17      19        6
    18      14        3
    19      22        7
    20      20        5
    21      23        6
    22      18        4
    23      22        7
  mean    20.0      5.4
Pearson r (points vs stories) = 0.81

Release 1: 30 stories, 130 points
  by points : 6.5 sprints
  by count  : 5.6 sprints

Average story size: history 3.70, Release 1 4.33
  1 pt: history   6%   Release 1   3%
  2 pt: history  20%   Release 1  17%
  3 pt: history  37%   Release 1  27%
  5 pt: history  28%   Release 1  37%
  8 pt: history   9%   Release 1  17%

The Pearson correlation between points and story count per sprint is 0.81. Sprints with more points usually have more stories too; the clearest case is Sprint 18: three stories, 14 points. Ten sprints is a small sample, so the number itself is not precise, but the direction is clear: for this team, counting stories says most of what summing points says.

In the Release 1 forecast, though, the two methods diverge: 6.5 sprints by points, 5.6 by count. The reason is in the last lines. In the past the average story was 3.70 points, while Release 1 averages 4.33; the share of 8-point stories went from 9% to 17%. Institutional Orders stories are bigger than what the team is used to. Counting assumes the historical story size mix continues; when that assumption breaks it gives an optimistic date. That is exactly why #NoEstimates insists on slicing first. If the team splits the five 8-point stories in two, the count and point methods move closer again.

When does it make sense? For a team that regularly slices stories small, has historical throughput data and can answer the date question with Monte Carlo, estimation sessions are largely unnecessary. The forecast in the Kanban series post "Service Level Expectation and Monte Carlo" works with exactly this data. For a new team, an epic of unusual size or a technical piece that is hard to split, the size conversation still produces valuable information.

In Production

Set a rule for bugs and technical work up front and stick to it. Some teams don't point bugs and measure velocity only with work that creates new value; bugs then reduce capacity and velocity shows it. Others point everything; velocity becomes steadier but overstates the pace of new features. Both are defensible; mixing them is not. If you point bugs in one sprint and not in the next, the velocity chart tells you nothing.

Keep the reference set alive. Every few sprints, pick new examples from finished stories and drop the stale ones. When someone joins the team, show the reference set explicitly in the first planning poker rounds; otherwise the newcomer brings the scale of their old team.

Timebox planning poker. If there is no consensus on a story after two rounds, that signals the story is not understood well enough. Instead of stretching the discussion, split the story or get the questions answered before the next refinement.

Common Pitfalls

• Taking the average: the middle of 3 and 8 is 5, but you lose the fact that two people are thinking of different work; talk about the reason for the gap first.
• Anchoring on the most senior person: if cards are not turned over together, or the most senior person speaks first, everyone else drifts towards their number.
• Converting points to hours: every conversion at story level wipes out the benefit of relative estimation.
• Inconsistency with bugs and technical work: pointing them in one sprint and not the next makes velocity meaningless.
• Comparing velocity across teams or making it a target: points inflate, delivered work stays the same.
• Counting without slicing: if story sizes differ from the past, a throughput forecast stays optimistic.

A Small Experiment

★☆☆ Pick four of your team's last ten finished stories as a reference set, one each for 1, 3, 5 and 8 points. In the next refinement, size new stories only by comparing them with these four.

★★☆ Split the five 8-point stories in the SprintHistory.Release1 array into a 5 and a 3 and run the program again. How much closer did the count-based and point-based forecasts get?

★★★ Add a loop to PointsVsCount that computes the correlation over a sliding window, for example the last 6 sprints, instead of the last 10. Does the correlation change over time? If it does, what might have changed in the team's slicing habits?

Points and counts work well for many stories that look alike. But how do you estimate work like matching rows without an ISBN by title, something the team has never done and that carries big uncertainty on its own? In the next post we look at methods beyond points: work breakdown, three-point and PERT, Wideband Delphi and reference class forecasting.