How Much Do AI Coding Assistants Really Help? Tool Types, Evidence and the Right Expectations

Version note: The code in this post was built and run with .NET 10; model outputs come from qwen3:4b and gemma3:12b on a local Ollama 0.34.4. Tool and research information is as of November 2026.

Guess First

In early 2025 METR ran an experiment with 16 experienced open-source developers. They picked 246 real issues from large repositories they had contributed to for years (on average more than 22,000 stars and over a million lines of code). Each issue was randomly assigned to one of two groups: AI allowed or AI not allowed. Where it was allowed, the developers mostly used Cursor Pro with Claude 3.5/3.7 Sonnet.

Before starting, the developers were asked how much AI would speed them up. The average answer was 24%. After the study they were asked again how much AI had sped them up, and this time they said 20%. So what did the screen recordings and time measurements say? Was the speedup below 20%, above it, or something else entirely? Pick a number; the answer is in the middle of this post.

This series is for the .NET and Angular developer who uses AI coding assistants and coding agents every day but is not sure about the quality of the code they produce or whether they really save time. We will not walk through the buttons of each tool; every tool gets its own series (Claude Code, OpenAI Codex, Working with Claude, Working with ChatGPT). Here we talk about what is tool-neutral: which kind of tool fits which job, what an agent needs to be told, how you verify the code it writes, and how you keep security and the team safe. The examples use Bookstore, the online bookstore used as the shared example across this site's series.

The Tool Spectrum: From Completion to Cloud Agents

"I write code with AI" covers five very different ways of working. The difference is not how smart the model is; it is how much work it does in one step and who checks that work, and when:

An example for the last row: as of November 2026 GitHub's documentation calls this type "Copilot cloud agent". When you assign an issue to Copilot, it works in a temporary environment on GitHub Actions, pushes to a branch starting with copilot/ and opens a draft pull request. A session lasts at most 59 minutes, is limited to one repository, and a person has to review and merge the pull request it opens.

Reading the table from top to bottom shows a pattern: the further down you go, the more work AI produces in one go, and the less often you get to check it. With completion you see every line. With a cloud agent the first thing you see may be a 14-file diff. Part of the time you save on typing is paid back in reading and verification. That trade-off comes up in every post of this series. The tour of AI features inside the IDE is in the ninth post of the IDEs series; here we look at what the types have in common.

A Small Experiment: An Isolated Task and a Repository-Bound Task

First I gave two tasks to a local model. Cloud vendors' models are much stronger than these small ones; the point of the experiment is not to score a model but to see how the type of task changes the result. The first task is fully isolated: a method that validates an ISBN-13. This was the prompt:

Write a C# 14 static class `Isbn13` with a method `public static bool IsValid(string? isbn)` that checks an ISBN-13.
Hyphens and spaces may appear between digits and must be ignored. Return false for null, wrong length or a wrong check digit.
Reply with the code only, in one ```csharp block.

Before answering this 85-token prompt, qwen3:4b thought to itself for 6,668 tokens (184 seconds). Along the way it got confused while computing with a wrong example ISBN, then recovered. The final answer was this:

public static class Isbn13
{
    public static bool IsValid(string? isbn)
    {
        if (isbn == null)
            return false;

        var cleaned = new string(isbn.Where(c => char.IsDigit(c)).ToArray());

        if (cleaned.Length != 13)
            return false;

        int total = 0;
        for (int i = 0; i < 12; i++)
        {
            int digit = cleaned[i] - '0';
            int weight = (i % 2 == 0) ? 1 : 3;
            total += digit * weight;
        }

        int checkDigit = (10 - (total % 10)) % 10;

        return cleaned[12] - '0' == checkDigit;
    }
}

Readable, correct weights, correct check-digit formula. In a demo you would pass The Hobbit's ISBN, see true and be convinced. Instead, I appended the code unchanged to a .NET 10 file-based app and wrote the cases myself:

#:property PublishAot=false
// The code is exactly as qwen3:4b wrote it; we wrote the cases
(string? Input, bool Expected)[] cases =
[
    ("9780261102217", true),        // The Hobbit, from the Bookstore seed
    ("978-0-261-10221-7", true),    // same ISBN with hyphens
    ("978 0 261 10221 7", true),    // same ISBN with spaces
    ("9780261102218", false),       // wrong check digit
    ("978026110221", false),        // 12 digits
    (null, false),
    ("ISBN 9780261102217", false),  // letters are not allowed by the spec
    ("9780261102217-X", false),     // trailing garbage
];

int failed = 0;
foreach ((string? input, bool expected) in cases)
{
    bool actual = Isbn13.IsValid(input);
    string mark = actual == expected ? "ok  " : "FAIL";
    if (actual != expected) failed++;
    Console.WriteLine($"{mark} {input ?? "null",-22} expected={expected,-5} actual={actual}");
}
Console.WriteLine($"{cases.Length - failed}/{cases.Length} passed");

> dotnet run isbn-check.cs
ok   9780261102217          expected=True  actual=True
ok   978-0-261-10221-7      expected=True  actual=True
ok   978 0 261 10221 7      expected=True  actual=True
ok   9780261102218          expected=False actual=False
ok   978026110221           expected=False actual=False
ok   null                   expected=False actual=False
FAIL ISBN 9780261102217     expected=False actual=True
FAIL 9780261102217-X        expected=False actual=True
6/8 passed

The prompt asked only for hyphens and spaces to be ignored; the model chose to drop every non-digit character. The result is a method that silently accepts input with an "ISBN" prefix or trailing garbage. In an API that means values reaching the database in a shape you did not expect. The bug is not hard, but a demo that tries only the happy path never shows it. One more note: the file-based app did not build on the first try, because the repository's TreatWarningsAsErrors setting turned a known-vulnerability warning on the file-based app's default AOT dependency into an error. That is why the #:property PublishAot=false line is there; we will meet warnings like this again in the fifth post.

The second task is tied to Bookstore. This time I asked gemma3:12b the following without showing it a single file from the repository:

Our Bookstore web API runs on .NET 10. Add an endpoint that returns the books of one category,
for example GET /api/categories/classics/books. Show the code you would add or change, with file names.

In 143 seconds the model wrote a long, confident answer. The file layout and registration code it proposed looked like this (shortened from the answer, code verbatim):

Bookstore.Api/
├── Models/
│   ├── Book.cs
│   └── Category.cs
├── Services/
│   └── IBookService.cs
│   └── BookService.cs  (Implementation - optional)
├── Controllers/
│   └── BooksController.cs
├── Startup.cs
└── Program.cs

[HttpGet("categories/{categoryId}/books")]
public ActionResult<IEnumerable<Book>> GetBooksByCategory(int categoryId)

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        // ... other services ...

        services.AddScoped<IBookService, BookService>(); // Register the service
    }

There is not even a compile error in the answer; the problem is that the answer does not belong to our project. Bookstore is written with Minimal APIs and module folders; the model proposed MVC controllers and a Startup.cs from the project templates that predate .NET 6. Bookstore already has a Book entity and an EF Core context; the model wrote a second Book under Bookstore.Models and a service holding a hard-coded list of four books. The requested path was /api/categories/classics/books, but the controller's route became /api/books/categories/1/books, and the category is looked up by an int id instead of the slug. An empty category returns 404, and there is no test. The model openly listed all of this as "reasonable assumptions"; it had no choice but to assume, because it never saw the repository.

The gap between the two experiments also explains why studies come out so differently. On an isolated task that is easy to verify, the model is fast and mostly right. On a task tied to the repository, quality depends on how much context you give the model and how carefully you check the output.

METR 2025: Feeling Faster While Getting Slower

Back to the opening question. In METR's measurement, published on 10 July 2025, issues where AI was allowed took 19% longer on average. The developers expected a 24% speedup and believed they had been 20% faster, while the measurement showed a slowdown. This gap between perception and measurement became the most quoted part of the study.

To read the result correctly, you also need the limits METR states on its own page. The study does not show that most developers, or developers in other domains, are slowed down by AI. Nor does it show that better usage or later models would not speed things up in the same setting. The sample is 16 people; METR notes that developers who believed AI helps a lot may have stayed out, and that it cannot rule out learning effects beyond roughly 50 hours of Cursor use. METR examined 20 factors that might explain the slowdown and found evidence that five of them likely contribute.

For me the real lesson of the study is the setting rather than the number: an expert working on a large, mature project with a high quality bar, who knows the code base very well. The context in that person's head is far richer than the context you can hand to a model. Understanding, fixing and waiting for the model's suggestion can take longer than doing the work yourself. The more your daily work looks like that setting, the more seriously you should take the result.

The 2026 METR Update: Why the Result Is Still Uncertain

In August 2025 METR started a larger study with the same design and published the results on 24 February 2026: 57 developers (10 of them from the first study), 143 repositories and more than 800 tasks. In METR's sign convention a negative value means finishing faster. For developers returning from the first study the estimate is -18%, with a confidence interval from -38% to +9%. For newly recruited developers the estimate is -4%, with an interval from -15% to +9%.

Some secondary sources reported these numbers as "AI now makes developers 18% faster". METR's own reading is far more cautious: it says the raw results show some evidence of a speedup and that developers are probably sped up more than in early 2025, but that the data is very weak evidence for the size of that increase. The upper end of the interval still includes a slowdown.

The reasons for the uncertainty are instructive too. Some developers no longer agreed to work without AI; between 30% and 50% said they kept some tasks out of the study because they did not want to do them without AI. So the tasks where AI helps most, and the developers who believe in it most, may have dropped out of the sample. Pay dropping from 150 to 50 dollars an hour also affected participation. Timing developers who ran several agents at once was unreliable. METR says it is changing its study design as a result. The takeaway for you: measuring AI's effect on productivity is hard even for the most careful researchers. When you see a slide saying "our team got 40% faster", the first question should be how that was measured.

DORA 2025: AI Is an Amplifier

The second important source is DORA. Its 2025 report is based on a survey run between 13 June and 21 July 2025 and on answers from nearly 5,000 technology professionals. Its findings do not contradict METR; they answer a different question: under which conditions does AI work inside an organization?

90% of respondents use AI in their work and more than 80% believe it has made them more productive. Yet 30% of the same people trust AI-generated code only a little (23%) or not at all (7%); 46% trust it somewhat, 20% a lot and 4% a great deal. The report treats this as a sign of mature use: developers approach AI output with the same caution they apply to an answer from Stack Overflow. That gap between using and trusting is the reason for the review habit we discuss in the fourth post of this series.

The report's delivery finding is more striking: AI adoption now improves throughput, a change from the previous year, but it still increases instability. Teams have adapted to going faster but have not yet built the systems to ship faster code safely. The report's main message comes from here: AI is an amplifier, magnifying both the strengths and the weaknesses of an organization. In a team with weak tests, shallow reviews and hard deployments, AI produces more problems, faster. The seven foundational capabilities DORA recommends for this (a clear AI policy, working in small batches, strong version control practices and others) are covered in the sixth post when we discuss team policy.

Vibe Coding and Augmented Coding

What did I do in the ISBN example? Before accepting the model's code, I wrote down what it had to do as cases and measured the result. In June 2025 Kent Beck described this difference with two names. In vibe coding you do not look at the code, only at how the system behaves; when an error shows up you feed it back to the AI and hope for a good enough fix. In augmented coding the code itself matters: its complexity, its tests and their coverage. The bar is the same as for code you write by hand; the difference is that you type far less of it.

In that post, while building a B+ tree library with AI, Beck lists three signs that the model has gone off track: going around in loops, adding functionality nobody asked for (even when it is a reasonable next step), and moves that look like cheating, such as disabling or deleting tests. All three show up again in the review checklist in the fourth post of this series. Beck's remedy is familiar too: give the agent a plan file, the TDD cycle and rules such as "never mix structural and behavioral changes in one commit". We build the Bookstore version of this in the second and third posts.

Vibe coding is not always wrong. For a script you run once and throw away, a prototype, or an experiment nobody will maintain, not looking at the code is a reasonable trade-off. The problem is when a habit that is fine for a prototype moves into Bookstore's ordering flow.

The Right Expectation: How Much Help for Which Work

Putting the evidence and the experiment together gives a rough but useful map. AI saves the most on work whose correctness can be checked quickly and that needs little context: repetitive code, DTOs and mapping, test skeletons and test data, explaining a piece of code you do not know, drafting a regex or an SQL query, one-off scripts. Medium gains come from well-bounded features in a well-structured repository with tests; there, how well the agent uses the context you give it is decisive. Gains can be small or even negative where things look like METR's setting: a large, mature code base you know very well, unclear requirements, security-critical code, and work you are doing to learn.

If you want to measure your own situation, measure by task, not by lines. For a few weeks, estimate each task before you start, note whether you used AI, and write down the actual time and any fixes that came later. It is a small copy of METR's design; not perfect, but better than the feeling that you are faster. Team-level measurement is covered in the sixth post.

In Production

Code written by AI should go through the same gates as code written by people: build, tests, review, static analysis, security scanning. The moment you open a separate, looser path because "AI wrote it, let's just skim it", the instability increase DORA saw will show up in your team too. If you use a cloud agent, check whether its pull requests need approval before CI runs and whether branch protection rules apply to the agent as well.

On the measurement side, drop misleading metrics from the start. Suggestion acceptance rate and lines generated look better the more code is written, while review time, reverted changes and production bugs show the real cost. There is a direct cost as well: tokens and credits. For example, GitHub's documentation puts the average cost of a Copilot code review at roughly 0.25 to 10 dollars per pull request, depending on the effort level. Prices like these change often; check the vendor's current page before deciding.

Finally, tool knowledge goes stale quickly. The product names and limits in this post are correct as of November 2026; a year from now some of them will have been renamed. When you document a tool feature for your team, add a date.

Common Pitfalls

• Generalizing from one impressive demo: a demo shows the happy path, and the bug is in the edge case, as in the ISBN example.
• Measuring productivity by lines generated or suggestions accepted: more code means more review and more maintenance.
• Trusting perception: METR's developers believed they were 20% faster while they were slower.
• Not matching the tool type to the work: a cloud agent for a one-line fix, completion for an unclear architectural decision.
• Asking for repository-bound work without context and not comparing the result with the project's rules.
• Lowering the review and test bar for AI-written code: a green build is not proof that the code is correct.

A Small Experiment

★☆☆ Fix the Isbn13.IsValid method in isbn-check.cs so that it only allows digits, hyphens and spaces. Both FAIL lines should become ok without breaking the other six cases. Then check that char.IsDigit also accepts non-ASCII digits such as Arabic-Indic digits, and add a case for that too.

★★☆ Pick five tasks you work on this week. Before starting each one, write down a time estimate; note whether you used AI and the actual time. At the end of the week add the changes you later had to fix and look at the table: do your feeling and the measurement point the same way?

★★★ Give the same repository-bound task (an endpoint returning the books of a category) to the agent you use, without showing it any file. In the resulting diff, mark everything that breaks the project's rules: file location, endpoint style, tests, naming. Keep that list.

That list is the input to the next post. An agent not knowing the repository is not an unsolvable problem; you can describe the build commands, layering rules and prohibitions in a short file. In the second post we write an AGENTS.md for Bookstore, give the same task again and measure the difference.