Prioritizing Technical Debt and Making the Case for Refactoring: Hotspot Analysis, Measurement and Talking to the Product Owner

Version note: The script in this post was built and run as a file-based app with .NET 10 (SDK 10.0.402) and Microsoft.CodeAnalysis.CSharp 5.12.0; the git output was taken with Git 2.54.

Guess First: You Have Two Weeks, Where Do You Spend Them?

The Product Owner finally said, "this sprint and the next, we can set aside part of the team's capacity for technical debt." This series' legacy parts are on the table, and everyone has a favorite. In InvoiceRenderer, a single Render method produces every line of the invoice; everyone who opens it winces. LegacyShippingClient dates from 2014, compiles with a SYSLIB0014 warning and uses an obsolete API. PricingEngine.Calculate is full of nested conditionals. OrderImportJob depends on static helpers and DateTime.Now. The catalog API returns entities directly as JSON.

There are two voices on the team. One says "let's fix the worst code: InvoiceRenderer", the other says "let's get rid of the obsolete API: LegacyShippingClient". Which one would you choose? More importantly, how would you justify your choice to the Product Owner?

The Hotspot Idea: Frequently Changed and Complex

The core idea of Adam Tornhill's Your Code as a Crime Scene (second edition, 2024) is this: the quality of code only costs something when you touch that code. A complex file nobody opens doesn't slow anyone down today, however ugly it is. A complex file that changes every week charges a tax on every change: it takes long to understand, the risk of bugs is high, review is hard. A hotspot is where these two dimensions meet: code that changes often and is complex.

We have the data for both dimensions. Change frequency is in the Git history: how many commits touched a file in the last year? Complexity is in the current version of the code: Tornhill uses simple measures such as line count and indentation depth; in this post we'll use Cognitive Complexity. Defined in a white paper G. Ann Campbell wrote for SonarSource, this measure counts what makes code harder to read: +1 for every if, loop, catch and switch that breaks the flow, extra points for each level of nesting, and +1 for sequences of logical operators. Unlike cyclomatic complexity, it counts a switch once however many cases it has, and makes nested structure more expensive.

Measuring the Bookstore Repo: git log and a Small C# Script

For this post I built a Bookstore repo from the legacy files we used throughout the series and gave it a year of change history: campaigns, partner files, catalog fields. The numbers below are the script's real output on that repo; when you run the same script on your repo, you see your own history. The script uses the file-based app feature that came with .NET 10: a single .cs file, no project file, the package comes through the #:package directive, and it runs with dotnet run hotspots.cs -- ../bookstore. The first part is change frequency:

#:package Microsoft.CodeAnalysis.CSharp@5.12.0
#:property PublishAot=false

// 1. Change frequency: how many commits touched each C# file in the period.
List<string[]> commits = ParseCommits(Git(repo, "log", $"--since={since}", "--name-only", "--format=>%h", "--", "*.cs"));
Dictionary<string, int> revisions = commits.SelectMany(files => files)
    .GroupBy(file => file)
    .ToDictionary(g => g.Key, g => g.Count());

// Each commit starts with a ">hash" line, followed by the files it touched.
static List<string[]> ParseCommits(string log) =>
    log.Split('>', StringSplitOptions.RemoveEmptyEntries)
       .Select(c => c.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries).Skip(1).ToArray())
       .Where(files => files.Length > 0)
       .ToList();

The second part is complexity. With Roslyn we build each file's syntax tree and run every method through a CSharpSyntaxWalker. It is a plain version that applies most of the rules, not all of them; recursion and some language-specific rules are left out:

// Simplified Cognitive Complexity (G. Ann Campbell, SonarSource): +1 for each break in linear flow,
// +nesting level for structures nested inside others; else/else if add +1 without the nesting part.
sealed class CognitiveComplexity : CSharpSyntaxWalker
{
    private int _nesting;
    private int _score;

    private void Structure(SyntaxNode? condition, SyntaxNode body)
    {
        _score += 1 + _nesting;
        Visit(condition);
        Nested(body);
    }

    public override void VisitForEachStatement(ForEachStatementSyntax node) => Structure(node.Expression, node.Statement);
    public override void VisitCatchClause(CatchClauseSyntax node) => Structure(node.Filter, node.Block);

    // Each new sequence of && or || costs +1: a && b && c is 1, a && b || c is 2.
    public override void VisitBinaryExpression(BinaryExpressionSyntax node)
    {
        if (node.Kind() is SyntaxKind.LogicalAndExpression or SyntaxKind.LogicalOrExpression
            && !(node.Parent is BinaryExpressionSyntax parent && parent.Kind() == node.Kind()))
            _score += 1;
        base.VisitBinaryExpression(node);
    }
    // ... if/else, switch, ?:, goto, lambdas and local functions
}

Before trusting a metric you wrote yourself, test it against known results. I translated the white paper's two examples (a method that sums primes with nested loops and a switch with four cases) to C# and gave them to the script; the values in the paper are 7 and 1:

$ dotnet run hotspots.cs -- samples.cs
SumOfPrimes: 7
GetWords: 1

The Answer: Neither InvoiceRenderer nor LegacyShippingClient

I ran the script on the repo with the last twelve months of history:

$ dotnet run hotspots.cs -- ../bookstore
88 commits since 12.months

File                                         Revs CogC Score  Most complex member
Pricing/PricingEngine.cs                       41   28  1148  Calculate (28)
Invoicing/InvoiceRenderer.cs                   23   41   943  Render (41)
Catalog/Program.cs                             15    6    90  <top-level> (6)
Import/OrderImportJob.cs                       13    6    78  ExecuteAsync (3)
Catalog/CsvExport.cs                            3    1     3  Write (1)

The most complex file really is InvoiceRenderer: its single method scores 41. But it changed 23 times in a year; PricingEngine changed 41 times. Multiplying change frequency by complexity puts PricingEngine at the top of the list. LegacyShippingClient isn't even on it:

$ dotnet run hotspots.cs -- ../bookstore/src/Bookstore.Shipping/Legacy/LegacyShippingClient.cs
Endpoint.get: 0
Endpoint.set: 0
GetRate: 0
CreateShipment: 0
Post: 0
$ git log --since=12.months --oneline -- src/Bookstore.Shipping/Legacy/LegacyShippingClient.cs
ee192e0 fix: carrier timeout isbn13
9890041 fix: carrier timeout partner-a

It uses an obsolete API and nobody likes it, but it isn't complex and it changed twice in a year. Rewriting it today would be spending that makes no change cheaper for the team. The migration in the sixth post only made sense when the carrier shut down the old service, that is, when the code really had to change.

The score is not an exact ranking but an attention list. The difference between 1148 and 943 doesn't mean "one is 20 percent more important than the other"; it means these two files stand an order of magnitude apart from the rest of the list. The data in the next section settles the decision.

Change Coupling: Files That Change Together

The third part of the script counts pairs of files that change in the same commit. The degree is the number of shared commits divided by the average number of commits of the two files:

Coupled files                                                  Shared Degree
Shipping/Checkout/CheckoutService.cs <-> Web/Program.cs             5   100%
Catalog/Book.cs <-> Catalog/Program.cs                              9    67%
Invoicing/InvoiceRenderer.cs <-> Pricing/PricingEngine.cs          19    59%
Pricing/CartLine.cs <-> Pricing/PricingEngine.cs                    6    26%

Not every coupling is bad. CheckoutService changing together with the Program.cs that wires it to an endpoint is expected; they are two ends of the same feature. Book.cs and the catalog Program.cs change together because the endpoints return the entity directly; in the seventh post we saw why that is risky. The real finding is the third line: 19 of InvoiceRenderer's 23 changes are in the same commit as PricingEngine. Two classes in different folders, owned by different teams, change together without importing each other. The commit messages tell why:

$ git log --since=12.months --oneline --grep="invoice total differs from cart"
05e48c9 fix: invoice total differs from cart for audiobooks
f7476c6 fix: invoice total differs from cart for black-friday
a20fc1f fix: invoice total differs from cart for summer
66f803e fix: invoice total differs from cart for back-to-school
6ddc00f fix: invoice total differs from cart for ebooks
d3eadb7 fix: invoice total differs from cart for audiobooks
6a8794e fix: invoice total differs from cart for kids
5c84a44 fix: invoice total differs from cart for partner-a

Eight times in a year, the invoice total didn't match the cart total. InvoiceRenderer recalculates the coupon discount and the tax on its own; every new campaign is added first to PricingEngine and then (often incompletely) to InvoiceRenderer. This is the shotgun surgery from the Clean Code series post "Code Smell Catalog", made visible by data. The target of the investment is now clear: first make the rules in PricingEngine readable as in the fourth post, then have InvoiceRenderer print PricingEngine's result instead of calculating the price itself. The rest of Render, that is the header, the line format and the gift wrap logic, can keep waiting.

When to Refactor

The hotspot list says what to invest in; the question of when has three answers, and all three work together. The first is continuous small clean-up (the boy scout rule): when you touch a file for a task, leave it a little better than you found it. Since hotspots are the files touched most often anyway, this habit helps them most. The second is preparatory refactoring, which we covered in the first post: if the new campaign rule is hard to fit into PricingEngine, change the structure first and then add the rule. The third is allocated time: explicitly set-aside capacity, like at the start of this post, for work that takes more than a few days and doesn't fit inside a feature.

For allocated time to work, technical debt has to be visible. The debt metaphor Ward Cunningham introduced in 1992 helps here: debt has a principal (the cost of fixing it) and interest (the extra cost paid on every change). Hotspot analysis shows the debt with the highest interest. How to keep technical debt visible in the Product Backlog was covered in the Scrum series post "Engineering Practices in Scrum: Branching, CI, Feature Flags and Technical Debt"; we won't repeat it here.

Explaining It to the Product Owner: Cost of Change, Lead Time, Failure Rate

The sentence "the code is really bad, we need to refactor" tells the Product Owner nothing; they can't judge whether the code is bad and they don't have to. In their language there are three questions: how long does a new feature take, how predictable is it, and how often does it come back with a bug? A refactoring proposal should answer one of these three questions.

For Bookstore the conversation goes something like this. In the last year 41 changes went into the pricing side, and eight times a customer received an invoice that differed from the price in the cart; each one means a support ticket, a fix and a deploy. Every campaign is written twice, in two separate places. Each of the five campaigns planned for the next quarter will go the same way. The proposal: for two sprints, use part of the capacity to make the invoice take its price from the pricing engine. The measure of success: the time campaign work takes from start to production, and the number of "invoice doesn't match" bugs.

These measures correspond to two of the DORA metrics: lead time for changes and change failure rate. For anyone who wants a general basis, there is the 2022 "Code Red" study by Adam Tornhill and Markus Borg. Looking at 39 proprietary codebases, it reports about 15 times more defects in low-quality code, and tasks taking longer and less predictably there. It should also be said that the authors are from CodeScene, the company that produces the quality metric used. Your own data is always more convincing.

Measuring Before and After

The strongest way to make the case for refactoring is to come to the next conversation with data. That means taking the measurement before the work starts. For Bookstore the starting point is in this post's output: PricingEngine.Calculate at 28 points, change coupling of 59 percent with InvoiceRenderer, eight "invoice doesn't match" fixes in a year. The average and longest duration of pricing work also has to come from the issue tracker; that data isn't in the repo.

Three months later we run the same script on the same repo with --since moved to after the refactoring and look at these questions: does InvoiceRenderer still change together with PricingEngine? Did Calculate's score drop? Did campaign work get shorter, did the "invoice doesn't match" tickets stop? If the answer is yes, the next technical debt conversation will be much shorter. If not, we learned early that we invested in the wrong place; that is valuable information too.

Common Pitfalls

Investing in the worst code. The main lesson of this post: the investment goes not to the ugliest code but to the complex code that is touched most. Rewriting LegacyShippingClient today wouldn't have gained anything for the team's daily work.

Saving the work up for a "refactoring sprint". Collecting technical debt into one sprint means the interest keeps being paid until that sprint comes; and most of the time that sprint never comes. Continuous clean-up and preparatory refactoring do most of the work; allocated time is only for large, planned work.

Turning the metric into a target. If you set a target like "Cognitive Complexity will go below 15", the team will lower the number by splitting methods into meaningless pieces. Hotspot analysis is a decision tool, not a performance indicator.

Measuring without cleaning up the history. A bulk format commit that touches every file, or moving files around, distorts change counts. This script uses git log with file names; a renamed file's old history isn't added to its new name. If you analyze after a big move, set the period to start after the move or filter out bulk commits.

Presenting a number without context. "PricingEngine's score is 1148" means nothing to the Product Owner; "eight times in the last year a customer got a wrong invoice, and every new campaign is written in two places" is a business problem.

A Small Experiment

★☆☆ Find the ten files that changed most in your repo over the last year. Is the file the team calls "the worst code" on this list? If not, what would rewriting it today make cheaper?

★★☆ Run this post's hotspots.cs on your own repo. For one of the top five hotspots, look at the change coupling list: which file does it change with most often, and is that an expected relationship?

★★★ Pick a hotspot and write a one-page proposal for the Product Owner: what is the problem, which data shows it, what is the proposal, how long will it take, how will success be measured? Try writing it without technical terms.

The series ends here. Over nine posts we treated refactoring not as a list of techniques but as a journey: first a safety net that measures behavior, then small, reversible steps, breaking dependencies in legacy code, carrying out large changes on trunk with Mikado and branch by abstraction, moving contracts with expand-contract, modernizing Angular code with schematics and by hand, and finally choosing with data where to start all of this. The common thread was always the same: at every step the system works, and a test or a measurement tells you what changed.