Prompt Engineering in Practice: Treating Prompts as Specifications, Code and Tested Components

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 11.0.0, OllamaSharp 5.5.2 and xUnit v3 4.0.3. Model outputs come from gemma3:4b, a small open model (4.3 billion parameters, Q4_K_M quantization) running locally on Ollama 0.34.4; nothing was run against a cloud provider such as OpenAI or Anthropic.

Guess First

Gül from customer service has a request: when an incoming message is a return request and contains an order number, both should be filled in automatically when the ticket is opened. Emre writes a prompt and tries the first message: "Merhaba, 10428 numaralı siparişimdeki Dune hasarlı geldi, iade etmek istiyorum." In English: the copy of Dune in order 10428 arrived damaged and the customer wants to return it. The model returns {"isReturnRequest": true, "orderNumber": "10428"}. Emre opens a pull request and writes "return detection with an LLM, works" in the description.

If you tried the same prompt on twelve different messages, how many would come out right? Among them is someone asking about the return policy, someone waiting for the refund of a book they returned last month, someone who wrote an invoice amount instead of an order number, and someone who only wrote "Merhaba". Pick a number, then read on.

This post sets up the way of working we will use for the rest of the series: treating a prompt as a specification, as a piece of code and as a tested component. Every task goes through the same loop: task definition, success criteria, a small test set, prompt, measurement and fix.

Why "Looks Like It Works" Is Not Enough

The single message Emre tried is the easiest case for the prompt: the word for return is right there, and the order number has five digits and appears once. An LLM classifying that message correctly does not show that the prompt captures the task; it only shows that it works on the most obvious example. The equivalent in ordinary code would be calling a method with one input and saying "tested". Nobody accepts that, yet it happens with prompts all the time, because the model always produces a plausible-looking answer and errors are hard to spot by reading.

What decides a prompt's behaviour is not only the text you wrote: the model, the model version, temperature, the output format and the input distribution decide it together. When one of them changes, the behaviour can change too. So the only honest way to say a prompt "works" is to measure it on a test set that resembles real input and to write the result down as a number.

A Prompt Is a Specification, So Write the Criteria First

In the LLM Fundamentals series, "Writing Prompts for LLMs (1): Clear Instructions, Delimiters and Few-shot" covered the parts of a prompt: role, clear instructions, delimiters that separate input from instructions, examples and output format. "Writing Prompts for LLMs (2)" covered step-by-step thinking and reasoning models. This series does not repeat those principles; the focus is on how we build a prompt for a task and how we measure it.

In practice, two things should be written down before you start on the prompt. The first is the task definition: what counts as a "return request", does an exchange count, is asking about the status of an existing return a return request? Those answers are a decision for Gül's team, not for the model. The second is the success criterion: with which measurement and at what threshold do we call it "good enough"? Anthropic's prompt engineering guide also treats a clear success criterion, a way to test against it and a first draft prompt as prerequisites before improving a prompt, and recommends criteria that are measurable and specific to the task.

Our criterion for this task: on the twelve-message test set both fields must be correct; when the message has no order number the field must stay null, and the model must never invent a number. The second rule matters more than the first: a wrong order number is more expensive than an empty field, because Gül might act on the wrong order.

The Test Set: Twelve Messages, Each One a Question

The test set is a JSONL file; each line has an input, the expected output and a short note on why the case is there:

{"id": "rc-01", "input": "Merhaba, 10428 numaralı siparişimdeki Dune hasarlı geldi, iade etmek istiyorum.", "expected": {"isReturnRequest": true, "orderNumber": "10428"}, "note": "the first message we tried"}
{"id": "rc-04", "input": "İade etmeyeceğim, sadece faturayı tekrar gönderebilir misiniz? Sipariş 10602.", "expected": {"isReturnRequest": false, "orderNumber": "10602"}, "note": "negation"}
{"id": "rc-06", "input": "Geçen ay iade ettiğim kitabın parası hesabıma yatmadı. Sipariş 10345.", "expected": {"isReturnRequest": false, "orderNumber": "10345"}, "note": "status of an earlier refund"}
{"id": "rc-09", "input": "1.240,50 TL ödediğim kitaplar ıslak geldi, hepsini geri göndereceğim.", "expected": {"isReturnRequest": true, "orderNumber": null}, "note": "price is not an order number"}
{"id": "rc-11", "input": "İade politikanız kaç gün?", "expected": {"isReturnRequest": false, "orderNumber": null}, "note": "policy question"}

The rest of the set has a shipping delay, a return without an order number, a request written in English that never uses the word "return", an exchange request, a new order that contains an ISBN, a notice of the legal right of withdrawal that never mentions returns, and a greeting with no content. Twelve cases is few, but each one answers a different question. The criterion for picking them is situations we see in real tickets and that the prompt could misread. Growing the test set and building a golden set are covered in detail in the last post of the series.

Code That Measures: PromptVersion and EvalRunner

Measuring by hand produces mistakes even on the first try; nobody carefully reads and compares twelve JSON outputs. So the prompt and the code that measures it live in the same solution. A prompt is a record, together with its version and the output format it expects:

using Microsoft.Extensions.AI;

namespace Bookstore.Prompts.Evaluation;

// A prompt is versioned together with the output format it expects
public sealed record PromptVersion(
    string Task,
    string Version,
    string System,
    Func<string, string> BuildUser,
    ChatResponseFormat? Format = null)
{
    public List<ChatMessage> Render(string input) =>
        [new(ChatRole.System, System), new(ChatRole.User, BuildUser(input))];
}

The runner is provider-agnostic: it takes an IChatClient, sends every case to the model and hands the output to a task-specific grade function. Fixed settings (temperature 0, a fixed seed) come from outside and do not change during a measurement:

public sealed class EvalRunner(IChatClient client, ChatOptions baseOptions)
{
    public async Task<EvalReport> RunAsync<T>(
        PromptVersion prompt,
        IReadOnlyList<TestCase<T>> cases,
        Func<TestCase<T>, string, Verdict> grade,
        CancellationToken cancellationToken = default)
    {
        ChatOptions options = baseOptions.Clone();
        options.ResponseFormat = prompt.Format;

        List<CaseResult> results = [];
        foreach (TestCase<T> testCase in cases)
        {
            long start = Stopwatch.GetTimestamp();
            ChatResponse response = await client.GetResponseAsync(
                prompt.Render(testCase.Input), options, cancellationToken);
            Verdict verdict = grade(testCase, response.Text);
            results.Add(new CaseResult(testCase.Id, verdict, response.Text, Stopwatch.GetElapsedTime(start)));
        }
        return new EvalReport(prompt.Task, prompt.Version, results);
    }
}

Because we use IChatClient, the same runner works tomorrow with a cloud provider's client; only the line that creates the client in the Lab project changes. In this series the model runs on a local Ollama; its address and the model name are read from the BOOKSTORE_LLM_ENDPOINT and BOOKSTORE_LLM_MODEL environment variables, with http://localhost:11434 and gemma3:4b as defaults. On Windows, installing Ollama and running ollama pull gemma3:4b is all you need. Testing IChatClient code with a fake client is covered in "Testing Code That Uses IChatClient" in the AI with .NET series.

The grade function is deterministic: it parses the output as JSON, compares both fields with the expected values and writes what went wrong on one line. Output that does not parse is also a failure, and we count it.

Running the Loop: From v1 to v3

Emre's prompt is v1. It has one system sentence and a "reply with JSON" instruction; JSON mode is on, but nothing says what the fields mean:

// v1: the prompt that "worked" on the first message we tried
public static readonly PromptVersion V1 = new(
    Task: "return-check",
    Version: "v1",
    System: "You help the Bookstore customer service team.",
    BuildUser: message =>
        $"Does the customer want to return a book? Reply with JSON containing isReturnRequest and orderNumber.\n\n{message}",
    Format: ChatResponseFormat.Json);

First, let's do what Emre did, with only the first message:

dotnet run --project src/Bookstore.Prompts.Lab -- return-check v1 --first
return-check v1 | gemma3:4b | 1 cases
PASS rc-01
1/1 passed (100%)

One hundred percent. Now the whole set:

dotnet run --project src/Bookstore.Prompts.Lab -- return-check v1
return-check v1 | gemma3:4b | 12 cases
PASS rc-01
PASS rc-02
PASS rc-03
PASS rc-04
PASS rc-05
FAIL rc-06  isReturnRequest: expected false, got true
PASS rc-07
PASS rc-08
FAIL rc-09  orderNumber: expected null, got 1.240,50
PASS rc-10
FAIL rc-11  isReturnRequest: expected false, got true
FAIL rc-12  isReturnRequest: expected false, got true
8/12 passed (67%)

Four failures, all four because the task definition is not in the prompt. The model treats every message that mentions returns as a return request: the return policy question (rc-11) and the message asking about money for an old return (rc-06) come out true. Even "Merhaba", which carries no information, is true (rc-12). Worst of all is rc-09: the model put the invoice amount into the order number field. The JSON is valid, the field names are right, the value is wrong.

v2 adds the task definition to the prompt. Here the application's own parts join the ones we know from LLM Fundamentals: the meaning and the boundaries of each field, an explicit null rule for missing information, and an output contract. We give the output contract not only as text but also as a JSON schema; ForJsonSchema<T>() generates the schema from the record and Ollama constrains the output to match it (the mechanism is in the structured output post of the LLM Fundamentals series):

// v2: definitions, a null rule and an output contract
public static readonly PromptVersion V2 = new(
    Task: "return-check",
    Version: "v2",
    System: """
        You classify one customer message for the Bookstore customer service team.

        ## Definitions
        - isReturnRequest is true when the customer wants to start a new return or an exchange
          for a book they received: damaged, wrong edition, missing pages, changed their mind,
          or they use their legal right of withdrawal.
        - isReturnRequest is false for questions about the return policy, the status of an
          earlier return or refund, shipping delays, invoices and new orders.
        - orderNumber is the 5-digit Bookstore order number in the message, without "#".
          ISBNs (13 digits), prices and phone numbers are not order numbers.
        - If the message contains no order number, orderNumber is null. Never guess one.

        ## Output
        Return only a JSON object that matches the schema. No explanation.
        """,
    BuildUser: message => $"<message>\n{message}\n</message>",
    Format: ChatResponseFormat.ForJsonSchema<ReturnCheckResult>());

Every definition was written together with Gül; "an exchange counts as a return" is the team's decision, not the model's. The prompt is in English and the messages are in Turkish; with small models, English instructions usually give more consistent results, but that too needs to be confirmed by measuring. The result (I removed the PASS lines):

dotnet run --project src/Bookstore.Prompts.Lab -- return-check v2
FAIL rc-06  isReturnRequest: expected false, got true
FAIL rc-09  orderNumber: expected null, got 12405
10/12 passed (83%)

Two cases are fixed, but rc-09 has become more instructive. The prompt says "prices are not order numbers, never guess one"; the model still turned the digits of 1.240,50 into the five-digit 12405. The schema guarantees only the shape: the field is a string with five digits. It does not guarantee that the value came from the message. Writing such a rule into the prompt is not enough; it has to be verified in code:

// The model may only copy an order number, never compose one
public static partial class ReturnCheckGuard
{
    public static ReturnCheckResult Apply(ReturnCheckResult result, string message)
    {
        if (result.OrderNumber is null)
        {
            return result;
        }
        string digits = result.OrderNumber.Trim().TrimStart('#');
        bool copied = OrderNumberPattern().Matches(message).Any(m => m.Value == digits);
        return copied ? result with { OrderNumber = digits } : result with { OrderNumber = null };
    }

    [GeneratedRegex(@"(?<!\d|\d[.,])\d{5}(?!\d|[.,]\d)")]
    private static partial Regex OrderNumberPattern();
}

The guard finds five-digit numbers that stand on their own in the message; the digits inside "1.240,50" and a 13-digit ISBN do not match that pattern. If the number the model returned is not one of them, the guard clears the field. Measuring v2 with the guard (return-check v2 --guard) gives 11/12; rc-06 is left.

rc-06 is a definition boundary: "the money for the book I returned has not arrived" mentions returns but is not a new return request. The definition covers this case explicitly, and the model still says true. In v3 we add two examples that show this boundary. The examples were not taken from the test set; putting a test-set message into the prompt as an example would be like writing the answers between the exam questions.

// v3: v2 plus two boundary examples (not taken from the test set)
public static readonly PromptVersion V3 = V2 with
{
    Version = "v3",
    System = V2.System + """

        ## Examples
        <message>The refund for the book I sent back two weeks ago is still missing. Order 20117.</message>
        {"isReturnRequest": false, "orderNumber": "20117"}

        <message>I paid 389,90 TL and the cover is torn. I want to send it back.</message>
        {"isReturnRequest": true, "orderNumber": null}
        """,
};

dotnet run --project src/Bookstore.Prompts.Lab -- return-check v3 --guard
return-check v3 | gemma3:4b | 12 cases
PASS rc-01
PASS rc-02
PASS rc-03
PASS rc-04
PASS rc-05
PASS rc-06
PASS rc-07
PASS rc-08
PASS rc-09
PASS rc-10
PASS rc-11
PASS rc-12
12/12 passed (100%)

As a table, the loop looks like this:

12/12 should not be read as "done". We wrote v3's examples after seeing v2's failures; the prompt was shaped by these twelve messages. The real distribution will produce other failures. The job is to add new failures from production to the test set and to rerun the whole set on every prompt change. These results come from a single small model, with temperature 0 and a fixed seed; a large cloud model might do better with v1, but that too should be measured on the same set instead of guessed.

Bookstore's Prompt Backlog

For Bookstore, the online bookstore used as the shared example across this site's series, the tasks we will cover in the rest of the series were collected in a backlog from the team's requests. Each one goes through the same loop:

• Extracting books, quantities and a delivery date from institutional order e-mails (Nevin Hanım's e-mails; extraction).
• Routing the tickets in Gül's inbox to the returns, shipping, payments and institutional teams (classification and routing).
• Translating book descriptions from Turkish to English with a glossary and without breaking placeholders.
• Shortening a long campaign text into a push notification, and summarizing an incident report separately for management and for customers.
• Extracting the return terms from a forty-page supplier contract, with source quotes.
• Answering questions about the return policy based on the contract and the policy text.
• Generating read-only SQL from natural language for the reporting screen.
• The system prompt of a customer assistant that can change orders and leaves return approval to a person.

The last two posts go deeper into managing and measuring prompts in code: keeping prompts in files as templates, versioning them together with the model snapshot, LLM-as-judge and a regression gate in CI.

Production Notes

Pin the measurement settings. Temperature, seed, model name and model version are part of the report; if one of them changes, the old number and the new one cannot be compared. OpenAI's prompt engineering guide also recommends pinning production applications to dated model snapshots and rerunning the eval set when the model version changes.

Keep the raw output. The Lab writes the raw model output of every run as JSONL into the BOOKSTORE_RUN_LOG folder. You only understand a failure like the 12405 in rc-09 by looking at the raw output; one line in the report is not enough.

Give rules the prompt cannot own to code. "The order number must come from the message" is a business rule; the prompt can tell the model, but it cannot guarantee it. Every rule that can be verified (format, the value appearing in the source, ranges) is checked in code; the prompt only makes it easier for the model to get it right.

Think about cost. This set runs in a few seconds on a local model; with a cloud provider every run means tokens. Running small deterministic checks on every change and large sets when needed is the topic of the last post of the series.

Common Pitfalls

Calling a prompt done after one example. Emre's 1/1 and the set's 8/12 came from the same prompt. The first message anyone tries is almost always the easiest case.

Mistaking valid JSON for a correct answer. JSON mode and JSON schema guarantee the shape of the output, not its content. In rc-09 the schema was satisfied and the value was invented.

Leaving the definition to the model. Where a concept like "return request" ends is a business decision. If the prompt does not state it, the model uses its own definition, and that definition may not be the same for every message.

Using test-set cases as examples in the prompt. The score goes up, but the measurement loses its meaning. Write examples separately, and keep the test set separate even while developing the prompt.

Looking only at the total score. The difference between 10/12 and 11/12 means nothing until you see which case changed. A fix can solve one case and break another; a per-case report shows that.

A Small Experiment

★☆☆ Add two messages to the test set: a return request that mentions two order numbers, and a customer who writes the order number out in words, "on bin dört yüz yirmi sekiz". Write the expected output first, then run v3. Look at what the guard does with these cases.

★★☆ Pick another small model with BOOKSTORE_LLM_MODEL and run v1, v2 and v3 on the same set. Which version's gain shrinks when the model changes? Show in a table whether the prompt or the model made the real difference.

★★★ Set temperature to 0.7, remove the seed and run v3 five times. Write a small report that counts in how many runs each case passed. Which cases are unstable? That number tells you which part of the test set really pushes the prompt.

In the next post we move to the first task in the backlog: extracting a structured order from Nevin Hanım's free-text e-mail. Missing fields, Turkish date expressions such as "gelecek salı", next Tuesday, and letting the model say it does not know will be the main topics there.