Outcomes over Output: Problem Statements, Hypotheses and Jobs to Be Done

Version note: This post is conceptual; the definition of outcome follows Joshua Seiden's "Outcomes Over Output" (2019), and Jobs to Be Done follows Clayton Christensen's work.

Guess First

There is a number on Bookstore's analytics screen: in the last four weeks, 66.7% of visitors who added a book to the cart left without completing the order. In the meeting, the general manager names the fix right away: "Let's do one-click checkout. Amazon has it, why don't we?" The team writes an epic called "One-click checkout" into next quarter's plan.

Three questions: Will one-click checkout lower cart abandonment? What information do we have right now to know that? And consider this: how many of the visitors who add a book to the cart and leave are already registered customers with a saved card? If most of them are first-time guests, what does "one click" mean to them?

Output, Outcome, Impact

The distinction at the center of this post is three words, each the result of the previous one:

Output is what the team produces: a feature, a screen, an endpoint. "Guest checkout in production" is an output, and it is under the team's control; you could ship it tomorrow if you wanted. An outcome is a change in people's behavior as a result of that output. In "Outcomes Over Output," Joshua Seiden defines an outcome as a change in human behavior that drives business results. "Cart abandonment dropped from 66.7% to 60%" is an outcome; the team can influence it but can't control it directly, because in the end the customer decides. Impact is the outcome's effect on the business: revenue, profit, market share. A rise in monthly online revenue is an impact, and it is affected by many factors outside the team, from campaigns to exchange rates.

The level you pick when giving a team a goal shapes how the team works. An output goal ("ship one-click checkout in the first quarter") focuses the team on delivering, and after delivery nobody looks at the result; that is the description of the feature factory from the previous post. An impact goal ("grow online revenue by 10%") is too far away to connect with the team's daily decisions. The outcome level sits between the two, where the team can both influence and measure.

Why Is Targeting Output So Tempting?

Because output is easy to see, easy to measure and easy to celebrate. "We shipped one-click checkout" can be shown at a sprint review; "cart abandonment dropped three points" takes weeks of data and may never happen. Managers are also often used to naming solutions: requesting the solution in their head instead of describing a problem feels like a clearer instruction.

The price is building the wrong solution the right way. One-click checkout is a solution designed for customers with a saved card. If most of the people abandoning carts are first-time guests and their real issue is the shipping fee they see at the last step, the team builds a nice feature for a quarter and cart abandonment doesn't budge. Worse, nobody notices; because the goal was the output, and the output was delivered.

A Good Problem Statement

The first step toward focusing on outcomes is separating the problem from the solution. A good problem statement says four things and contains no solution: who (which user segment), in what situation, what happens (the observed behavior and its evidence) and what it costs.

Bad example: "Our customers can't pay with one click." This isn't really a problem but the absence of a solution; the solution is hidden inside it. Good example: "In the last four weeks, 66.7% of visitors who added a book to the cart left without completing the order. Abandonment is 72.1% for guest users and 56.3% for registered customers; about 60% of abandonment happens in the cart, before the payment step. With current traffic, every three-point improvement means about 155 extra orders in four weeks."

The second statement names no solution, but it narrows the solution space: the problem is bigger for guests, and most people give up before they even see the payment step. The last sentence says with a number why the problem matters; we'll see how to use numbers like that in the prioritization post. The numbers in these sentences (the last four weeks' cart and order counts) are the output of the funnel we will compute with SQL (Structured Query Language) from event data in post 07; until then, treat them as read off the team's analytics screen.

The Hypothesis Format

Once the problem is defined, each solution idea is written as a hypothesis. A common format is: "We believe that [this change] will move [this metric] [in this direction, by this much] for [these users]; we will confirm it with [this measurement] within [this time]."

An example for Bookstore: "We believe that showing the shipping cost on the product page and in the cart, before the payment step, will lower cart abandonment by at least three points for guest users; we will confirm it with the add-to-cart-to-order funnel within four weeks, checking that average basket value doesn't drop."

Every part of this sentence has a job. The word "believe" admits that this is a guess, not knowledge. The user segment says where to look for the result. The metric and the size make the hypothesis falsifiable: if it doesn't drop by three points, the hypothesis is refuted. The time limit prevents waiting forever. The last part is a guardrail: while lowering cart abandonment, we don't want to lower average basket value. We'll open up guardrail metrics in post 06.

Writing more than one hypothesis for the same problem is a good habit: "if we remove the mandatory sign-up step," "if we save card details for registered customers." The general manager's one-click checkout idea also sits in this list as a hypothesis; the only difference is that it will now be judged by the same rules as the others.

Jobs to Be Done

One way to define the problem correctly is to ask not what the customer buys but what they are trying to accomplish. Clayton Christensen's Jobs to Be Done (JTBD) idea says exactly this: people "hire" products to make a certain progress in a certain situation. In Christensen's best-known example, people buying a milkshake on their morning commute weren't really buying a drink but a filling, one-handed way to get through a long, boring drive; the milkshake's competitor wasn't another milkshake but bananas and bagels.

Someone buying a book at Bookstore isn't always buying "a book." Some are looking for a gift for a friend whose birthday is tomorrow, some are trying to get the books on their university reading list before term starts, some want the next novel on their reading list. The same payment step means very different things for these three jobs. For the gift buyer the real question is the delivery date; for the textbook buyer it's the total cost and shipping.

Job Stories and User Stories

One way to bring the JTBD idea into the backlog is the job story format: "When [situation], I want to [motivation], so I can [expected outcome]." For example: "When I remember at the last minute that my friend's birthday is tomorrow, I want to order a book quickly while seeing the delivery date, so I can be sure my gift arrives in time."

A classic user story usually takes the form "As a [role], I want [something], so that [benefit]": "As a customer, I want to pay quickly, so that I save time." The difference between the two formats is the situation. A user story starts from a role, and the role is often as broad as "customer"; a job story starts from a situation, and the situation is the information that shapes the solution the most. A team reading the job story above arrives at "show the delivery date before the payment step" on its own; a team reading the user story arrives at one-click checkout.

This isn't a format war. Many teams use both: job stories in discovery to understand the problem, user stories in delivery to write backlog items. The details of user stories, use cases and acceptance criteria are the subject of the upcoming Requirements Analysis series; this series doesn't go there.

Framing the Cart Abandonment Problem

Now let's go back to the opening meeting and frame the problem without jumping to a solution. Here is what the team knows: abandonment is 66.7%; 72.1% for guests, 56.3% for registered customers; about 60% of abandonment happens before the payment step. What it doesn't know is why. So the first job is not to pick a solution but to look for the answer to "why?"

The team decides to look at three sources. First, data: splitting the funnel by user segment (guest, registered) and by the step where abandonment happens. Second, support tickets: the last three months of tickets mentioning "can't place an order" and "shipping fee." Third, customers: short interviews over the next two weeks with six customers who abandoned their cart in the last month. Every "why" from these three sources will become a branch of the Opportunity Solution Tree in the next post.

The one-click checkout idea didn't go in the bin; it stays on the hypothesis list. But now the team will judge it on evidence, not because "the general manager asked." And at first glance, since most cart abandonment comes from guests, a solution aimed at registered customers with saved cards won't be at the top of the list.

In Production

If you can't measure an outcome, you can't set it as a goal. For the goal "lower cart abandonment" to be meaningful, today's value first has to be computed reliably: which events, which time window, how guests and registered users are separated. Setting the goal without writing this definition down invites the later argument "by our calculation, it dropped."

Outcomes move slowly. The outcome doesn't change on the day you ship the output; it takes weeks of data. So write the time limit into the hypothesis up front and don't declare a result before it's up. We'll discuss deciding with experiments, along with their statistical pitfalls, in post 08.

An outcome goal left on its own produces side effects. The easiest way to lower cart abandonment is to make adding to the cart harder: fewer people add, those who do were going to buy anyway, and the rate improves. That's why every outcome goal needs at least one guardrail metric next to it.

Common Pitfalls

The first pitfall is writing a solution as if it were a problem. "Users can't filter search results" looks like a problem but hides the filter solution inside it. The real problem may be "users can't find the book they're looking for on the first page," and the fix may be better ranking, not filters.

The second is counting activity instead of outcomes. "The banner showing the shipping cost was viewed 40,000 times" is not an outcome; it doesn't say anyone's behavior changed. Views, clicks and opens are intermediate signals on the way to an outcome.

The third is writing an unfalsifiable hypothesis. "Showing the shipping cost will improve the user experience" can never turn out wrong, because it doesn't say what will change, or by how much. A hypothesis should contain a number that refutes it if it doesn't happen.

The fourth is confusing JTBD with personas. "Ayşe, 35, lives in Istanbul, loves crime fiction" is a persona; JTBD cares about what Ayşe is trying to accomplish this evening. The same person buys a gift one day and a textbook another; the job changes, the person doesn't.

A Small Experiment

★☆☆ Read the first five items in your team's backlog and mark whether each describes an output or an outcome. For one of the outputs, write the outcome behind it in one sentence.

★★☆ For a problem in your product, write the four-part problem statement from this post: who, in what situation, what happens (with a number), what it costs. Make sure no solution is left in it; have a colleague read it and ask which solution it implies.

★★★ Write three different hypotheses for the same problem, each in this post's format: change, segment, metric and size, measurement and time, guardrail. Discuss which one is the cheapest to test.

We separated the problem from the solution and learned to write hypotheses. In the next post we go looking for the answer to "why?": we'll see continuous discovery, how a good customer interview is done, and how to map the cart abandonment problem with an Opportunity Solution Tree.