Improving Prompts by Measuring: Test Sets, Rule-Based Checks, LLM-as-Judge, Microsoft.Extensions.AI.Evaluation and Regression in CI with promptfoo

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 11.0.0, Microsoft.Extensions.AI.Evaluation and Microsoft.Extensions.AI.Evaluation.Quality 11.0.0, OllamaSharp 5.5.2, xUnit v3 4.0.3 and promptfoo 0.125.1 (Node.js 24). Model outputs come from two small open models running locally on Ollama 0.34.4: gemma3:4b (4.3 billion parameters) and qwen3:4b (4 billion parameters), both Q4_K_M.

Guess First

One task was left in the backlog from the first post of this series: answering customer questions about the return policy of Bookstore, the online bookstore used as the shared example across this site's series, based on the policy text. The policy is a short text of eight clauses: a 14-day right of withdrawal, 30 days for damaged books, 49.90 TL return shipping, refunds within 10 business days at the latest, a 10% limit on institutional orders and a few exceptions. Emre wrote two prompts. v1 is one sentence plus the policy. v2 has rules: at most three sentences, start with "Evet" or "Hayır", the Turkish yes or no, when possible, do the necessary calculation, do not invent anything that is not in the policy.

We will measure them with a golden test set of ten questions. Three guesses: does v2 give more correct answers than v1? How well do simple rules that check whether the required information appears agree with the answers a person marks as correct? And if we ask another small model "is this correct?", on how many answers does it agree with the person?

The Golden Set: Reference Answer, Required Facts and a Human Label

A golden set, test cases whose correct answer was written and approved by the team, goes one step beyond the sets we built in this series. Each question has three things: a reference answer written by the returns team, facts that must appear in the answer, and a "correct/incorrect" label given by a person after the answer was generated. The questions resemble real customer messages and push the boundaries: a return after 10 days (within the period), a return after 20 days (outside it), 40 books from a 200-book order (needs a calculation), a return from abroad that the policy never mentions (the correct answer is "I don't know").

{"id": "pq-04", "input": "Okulumuz 200 kitaplık kurumsal sipariş verdi. 40 kitabı iade edebilir miyiz?", "expected": {"reference": "Hayır, 40 kitap iade edilemez. Kurumsal siparişlerde iade, sipariş adedinin en fazla yüzde 10'u, yani 200 kitapta en fazla 20 kitapla sınırlıdır ve kurumsal satış ekibinin onayı gerekir.", "facts": ["20", "kurumsal satış"], "label": ""}}
{"id": "pq-09", "input": "Yurt dışındaki adresime gönderilen kitabı nasıl iade ederim?", "expected": {"reference": "Politikada yurt dışı gönderimlerin iadesiyle ilgili bir kural yok; bu soruyu müşteri hizmetlerine iletmek gerekir.", "facts": ["müşteri hizmetleri|yer almıyor|bilgi bulunmuyor|belirtilmemiş|kapsamıyor"], "label": ""}}

The answers are generated once and stored; every measurement below reads the same saved answers. So you can compare different graders on the same answers without calling the model again. I read each of the thirty answers (v1 and v2 with gemma3:4b, v2 also with qwen3:4b) as someone who knows the policy and labeled it. The result: v1 gemma3:4b 8/10, v2 gemma3:4b 5/10, v2 qwen3:4b 10/10.

Here is the answer to the first guess. v2's "start with Evet or Hayır" rule backfired with the small model. To the customer asking about a book received 10 days ago (pq-01) it said "since your return period is 10 days you cannot use this right"; to the customer asking after 20 days (pq-08) it started with "Evet" and said "you have no right of return" in the second sentence. v1's two errors are of another kind: for the expired return it implied "you can return it if you pay the shipping fee", and for the question about abroad it invented a procedure that is not in the policy.

Rule-Based Checks: Cheap, Fast and with Blind Spots

The first measurement layer is deterministic: do the required facts appear, does the answer exceed 400 characters, does the answer contain a number that is not in the policy, the question or the reference?

public static List<string> Check(string question, string answer, PolicyExpectation expected, string policy)
{
    List<string> issues = [];
    foreach (string fact in expected.Facts)
    {
        // "a|b" means either form counts
        if (!fact.Split('|').Any(f => answer.Contains(f, StringComparison.OrdinalIgnoreCase)))
        {
            issues.Add($"missing fact: {fact}");
        }
    }
    if (answer.Length > MaxLength)
    {
        issues.Add($"too long: {answer.Length} chars");
    }
    HashSet<string> known = [.. Numbers(policy), .. Numbers(question), .. Numbers(expected.Reference)];
    List<string> invented = [.. Numbers(answer).Where(n => !known.Contains(n)).Distinct()];
    if (invented.Count > 0)
    {
        issues.Add($"numbers not in the policy: {string.Join(", ", invented)}");
    }
    return issues;
}

dotnet run --project src/Bookstore.Prompts.Lab -- policy-qa check v2
FAIL pq-01 missing fact: 49,90|müşteriye ait|size ait
FAIL pq-04 missing fact: kurumsal satış
FAIL pq-09 missing fact: müşteri hizmetleri|yer almıyor|bilgi bulunmuyor|belirtilmemiş|kapsamıyor
FAIL pq-10 missing fact: Bookstore öde|hasarlı
policy-qa v2 | gemma3:4b | checks 6/10

The rules are wrong in both directions. v2's answer to pq-08 passed because it contains "14 gün", but it is a contradictory answer that starts with "Evet". On the other hand many correct answers failed because they do not contain a detail from the reference (the institutional sales approval, who pays shipping for a damaged book). The rules are still useful: they can catch v1's 812-character answer, an invented number or a completely missing fact without calling the model, in milliseconds. This is the layer that runs on every pull request.

LLM-as-Judge: The Rubric Is a Prompt Too

To check meaning, we ask a model to compare the answer with the reference. Anthropic's guide on developing tests recommends a detailed rubric, a short and parseable verdict, and having the grader reason before the verdict when grading with an LLM; and, where possible, a different model from the one that produced the answer. So the judge prompt is a versioned file like every other prompt in the catalog:

---
task: policy-judge
version: v1
model: qwen3:4b
model-digest: 359d7dd4bcda
temperature: 0
seed: 7
---
# system
You grade one answer of a customer assistant against a reference answer written by the
Bookstore returns team.

## Rubric
- correct: the answer reaches the same conclusion as the reference and states every number the
  question needs (days, fees, quantities) with the same value. Extra wording is fine.
- incorrect: the conclusion differs, a needed number is missing or different, or the answer adds a
  claim that contradicts the reference or a promise that the reference does not make.

Write your reasoning first, then the verdict.

A judge's value is measured by how well it agrees with human labels. We graded the same thirty answers with two judges:

dotnet run --project src/Bookstore.Prompts.Lab -- policy-qa judge gemma3:4b
judge gemma3:4b: agrees with the person on 21/30; passes the person failed 2, fails the person passed 7, no verdict 0
dotnet run --project src/Bookstore.Prompts.Lab -- policy-qa judge qwen3:4b
DISAGREE pq-08 v1 gemma3:4b: judge no verdict, person incorrect
DISAGREE pq-03 v2 gemma3:4b: judge incorrect, person correct
DISAGREE pq-05 v2 gemma3:4b: judge correct, person incorrect
DISAGREE pq-06 v2 gemma3:4b: judge correct, person incorrect
DISAGREE pq-08 v2 gemma3:4b: judge no verdict, person incorrect
DISAGREE pq-10 v2 gemma3:4b: judge incorrect, person correct
DISAGREE pq-01 v2 qwen3:4b: judge incorrect, person correct
judge qwen3:4b: agrees with the person on 23/30; passes the person failed 2, fails the person passed 3, no verdict 2

qwen3:4b made the same decision as the person on 23 of thirty answers, gemma3:4b on 21. The details matter more than the totals. The two answers both judges passed by mistake are the same: the v2 answers to pq-05 and pq-06. The first says "although the return period is 30 days", the second invents a process, "you will be asked to pay based on the price of the remaining books". Neither contradicts the reference, but both contradict the policy; the judge only sees the reference, so it cannot know. Four of gemma3:4b's seven false alarms are on qwen3:4b's short answers: it counted a short, correct answer that does not repeat a detail from the reference as incomplete. In a set of 30 answers we cannot read this as a definite bias, but it is a signal to watch.

The "no verdict" lines are a lesson of their own. In qwen3:4b's first run a single judge call took 8 minutes 30 seconds: the model kept writing into the reasoning field of the JSON. We put an output limit on the judge call and counted an answer that hit the limit and could not be parsed as "no verdict". A 1,024-token limit also cut valid answers, so we raised it to 4,096; two calls still hit the limit. Both are the question about a return after 20 days, the case that challenges the judge most. Never count an undecided judge as a pass; such answers go to a person.

The Judge's Biases: Order and Its Own Answers

A judge has two well-known biases: looking at the order when comparing two answers, and favoring answers that resemble its own. Order is easy to measure: you ask about the same pair in both orders; a consistent judge picks the same answer both times. We gave the gemma3:4b answers of v1 and v2 to qwen3:4b in pairs, with "length and style do not matter" in the prompt:

dotnet run --project src/Bookstore.Prompts.Lab -- policy-qa pairwise qwen3:4b
pq-01: v1 first -> v1, v2 first -> v1
pq-02: v1 first -> v1, v2 first -> v1
pq-03: v1 first -> v1, v2 first -> v1
pq-04: v1 first -> none, v2 first -> none
pq-05: v1 first -> v1, v2 first -> none  (order decided)
pq-06: v1 first -> none, v2 first -> none
pq-07: v1 first -> v2, v2 first -> v2
pq-08: v1 first -> v2, v2 first -> v1  (order decided)
pq-09: v1 first -> v2, v2 first -> v1  (order decided)
pq-10: v1 first -> none, v2 first -> v1  (order decided)
consistent 4/10; answer in slot A won 5 of 20 comparisons

Only four of the ten pairs gave the same result in both orders. In pq-08 and pq-09 the winner changed purely with the order: both times the answer in the second position won. In 9 of the 14 comparisons that reached a decision the second answer was chosen; in six comparisons the judge hit the output limit and gave no decision at all. If the pairwise comparison had been done in one order only, nearly half of the results in this set would have come from the order of the answers rather than the answers themselves. The fix is simple but twice as expensive: ask about every pair in both orders and count only consistent decisions.

There is no clear evidence in this set for a judge favoring its own answers. The gemma3:4b judge agreed with the person on 15 of 20 of its own answers and on 6 of 10 of qwen3:4b's answers; but the difference may also come from its tendency to count short answers as incomplete. The practical rule stays the same: where possible, choose a judge different from the model that produced the answers, and regularly compare the judge's decisions with human labels.

The Same Measurement with Microsoft.Extensions.AI.Evaluation

We wrote our own graders; .NET also has a standard framework for this. Microsoft.Extensions.AI.Evaluation brings evaluators together behind the IEvaluator interface; the ready-made LLM-based evaluators (RelevanceEvaluator, CompletenessEvaluator, EquivalenceEvaluator, GroundednessEvaluator and the agent-focused ones) are in the Quality package, and storing results and generating reports is in the Reporting package. The API in detail is in the AI with .NET post "Testing Code That Uses IChatClient"; here we only look at how our own checks move into this framework. The rule-based checks return a BooleanMetric as an IEvaluator:

public ValueTask<EvaluationResult> EvaluateAsync(
    IEnumerable<ChatMessage> messages,
    ChatResponse modelResponse,
    ChatConfiguration? chatConfiguration = null,
    IEnumerable<EvaluationContext>? additionalContext = null,
    CancellationToken cancellationToken = default)
{
    KeyFactsContext context = additionalContext?.OfType<KeyFactsContext>().Single()
        ?? throw new ArgumentException("KeyFactsContext is required", nameof(additionalContext));
    string question = messages.Last(m => m.Role == ChatRole.User).Text;
    List<string> issues = PolicyChecks.Check(question, modelResponse.Text, context.Expected, context.Policy);

    BooleanMetric metric = new(MetricName, issues.Count == 0, issues.Count == 0 ? "all checks passed" : string.Join("; ", issues));
    metric.Interpretation = new EvaluationMetricInterpretation(
        issues.Count == 0 ? EvaluationRating.Good : EvaluationRating.Unacceptable, failed: issues.Count > 0);
    return ValueTask.FromResult(new EvaluationResult(metric));
}

It runs side by side with the built-in EquivalenceEvaluator on the same saved answers; the reference answer is passed with EquivalenceEvaluatorContext:

IEvaluator evaluator = new CompositeEvaluator(new KeyFactsEvaluator(), new EquivalenceEvaluator());
ChatConfiguration judge = new(judgeClient);
...
EvaluationResult result = await evaluator.EvaluateAsync(messages, response, judge,
    [new KeyFactsContext(testCase.Expected, Policy), new EquivalenceEvaluatorContext(testCase.Expected.Reference)]);

We first used qwen3:4b as the judge. For all of the first three answers the evaluator could not read a score and marked the metric Inconclusive; instead of a score in the format the evaluator expects, the model returned an empty answer. We stopped the run and made gemma3:4b the judge. Here the judge is the same model that produced the answers it grades; that is a trade-off to keep in mind when reading the results:

dotnet run --project src/Bookstore.Prompts.Lab -- policy-qa meai v2 gemma3:4b
pq-01 key facts False, equivalence 2 (Poor)
pq-02 key facts True, equivalence 3 (Average)
pq-03 key facts True, equivalence 5 (Exceptional)
pq-04 key facts False, equivalence 5 (Exceptional)
pq-05 key facts True, equivalence 2 (Poor)
pq-06 key facts True, equivalence 5 (Exceptional)
pq-07 key facts True, equivalence 3 (Average)
pq-08 key facts True, equivalence 2 (Poor)
pq-09 key facts False, equivalence 2 (Poor)
pq-10 key facts False, equivalence 3 (Average)

Equivalence is a score from 1 to 5. Compared with the human labels, it shows how much the threshold matters: with "4 and above passes", three of the five correct answers fail; with "3 and above", the decision matches the person on nine of ten answers, and the only difference is pq-06, the invented process both judges missed. That is different information from the "correct/incorrect" verdict of our own evaluator: turning a score into a decision with a threshold is also a step that needs calibration.

A Prompt x Model Matrix and CI with promptfoo

The second tool for the same measurement is promptfoo: an open source eval tool that runs on Node.js, compares prompts and models in a matrix and can run in CI. Instead of copying the prompts and the test set by hand, we export them from the catalog, so there is still a single source:

dotnet run --project src/Bookstore.Prompts.Lab -- policy-qa export-promptfoo eval/promptfoo
exported 2 prompts and 10 tests to eval/promptfoo

# Prompt x model matrix for policy-qa; prompts and tests are exported from the catalog
description: Bookstore policy-qa regression
prompts:
  - id: file://prompts/policy-qa-v1.json
    label: policy-qa v1
  - id: file://prompts/policy-qa-v2.json
    label: policy-qa v2
providers:
  - id: ollama:chat:gemma3:4b
    label: gemma3:4b
    config:
      temperature: 0
      seed: 7
  - id: ollama:chat:qwen3:4b
    label: qwen3:4b
    config:
      temperature: 0
      seed: 7
      think: false
defaultTest:
  options:
    # qwen3:4b writes its reasoning before </think> even with think off
    transform: "output.includes('</think>') ? output.split('</think>').pop().trim() : output.trim()"
  assert:
    - type: javascript
      value: "output.length <= 400"
      metric: length
tests: file://tests.yaml

Each test carries the required facts as javascript assertions. We ran promptfoo with the local models, with sharing and telemetry off (--no-share, PROMPTFOO_DISABLE_TELEMETRY=1) and the Ollama address given with OLLAMA_BASE_URL:

npx promptfoo@0.125.1 eval -c promptfooconfig.yaml --no-share --no-cache -j 1 -o results.json
Results:
  ✓ 20 passed (50.00%)
  ✗ 20 failed (50.00%)
  0 errors (0%)
Duration: 15m 28s (concurrency: 1)

Counting per cell in the results file: v1 gemma3:4b 5/10, v2 gemma3:4b 6/10, v1 qwen3:4b 2/10, v2 qwen3:4b 7/10. The gemma3:4b and v2 qwen3:4b columns gave exactly the same scores as the rule checks on the .NET side; same prompt, same seed, same rules. What the matrix adds is v1 with qwen3:4b: with the prompt that has no short-answer rule, most of qwen3:4b's answers exceeded 400 characters. A small example of why a prompt and a model have to be evaluated together: v1 is good with one model and bad with the other.

The promptfoo command exited with code 100 because at least one test failed; in CI that is a regression gate as it is. The threshold can be set with PROMPTFOO_PASS_RATE_THRESHOLD. The workflow has two jobs: tests that call no model on every pull request, and the model eval only for pull requests with a prompt change label, on a runner where the models are installed:

  model-eval:
    needs: unit-tests
    runs-on: [self-hosted, ollama]
    if: contains(github.event.pull_request.labels.*.name, 'prompt-change')
    env:
      OLLAMA_BASE_URL: http://127.0.0.1:11434
      PROMPTFOO_DISABLE_TELEMETRY: '1'
      PROMPTFOO_DISABLE_UPDATE: '1'
      PROMPTFOO_PASS_RATE_THRESHOLD: '80'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '10.0.x'
      - uses: actions/setup-node@v4
        with:
          node-version: '24'
      - run: dotnet run --project src/Bookstore.Prompts.Lab -- policy-qa export-promptfoo eval/promptfoo
      - run: npx promptfoo@0.125.1 eval -c eval/promptfoo/promptfooconfig.yaml --no-share -j 1 -o eval/promptfoo/results.json

This workflow is a starting point. promptfoo's official GitHub Action (promptfoo/promptfoo-action@v1) can compare the prompt before and after a change and post the result as a comment on the pull request; according to its documentation it needs Node.js 22.22 or later. This section is based on the official documentation.

Upgrading the Model: Deciding with the Same Set

The measurements in this post turn into a decision table. Say v1 with gemma3:4b runs in production. Two candidates are on the table: a new prompt (v2) and a new model (qwen3:4b). Everything was measured with the same ten questions, the same saved answers and the same human labels:

Rolling out v2 with gemma3:4b means trading correctness for short answers; rejected. v2 with qwen3:4b gives the most correct result, but on this machine the median answer time is seven times longer and one answer took 54 seconds. The decision cannot be made on the score alone: are return policy questions asked in a live chat, or answered by e-mail? In a live chat this latency is not acceptable; for an e-mail draft it can be. A middle path is using both models together as in the third post: the fast model answers, and if the rules or the judge see a problem, the answer goes to the slow model. Either way, the decision is recorded in the pull request with a dated test set result, and the new model's digest is written into the prompt file. OpenAI's guide also recommends running the same eval set again when the model version changes; with a cloud provider the table gets a cost column too.

Production Notes

Separate the measurement layers by cost. Rule-based checks and unit tests that call no model run on every pull request; they take seconds and cost nothing. Judge-based eval is expensive and slow because it calls a model: grading the thirty answers in this post with qwen3:4b took minutes locally, and with a cloud provider every run costs tokens. Run it only when a prompt file, the model or the test set changes, in a nightly run, or in a job triggered by a label.

Measure the judge too. The judge's agreement rate with people should be stored as part of the test set. When you change the judge prompt or the judge model, the first thing to do is recompute this rate; otherwise the measurement itself changes silently.

Do not let the prompt memorize the test set. In this post we wrote v2's rules without looking at the answers. If you fix a prompt by looking at the errors on a set, that set's score is now an optimistic estimate; make the decision on a separate holdout set, cases that are not used while developing the prompt and are kept for the final measurement only. The moment you look at the holdout and change the prompt, that set becomes a development set too.

Learn from production. Customer complaints, answers a person corrected and answers the judge scored low are candidate cases for the golden set. When growing the set, the team should still write the reference answer and the label; turning the model's answer into the reference copies the model's mistake into the test.

Common Pitfalls

Looking at a single score. The rules gave v2 gemma3:4b 6/10 and the person 5/10, but not for the same answers. Without seeing per case which layer caught what, a score is misleading.

Setting a threshold without calibrating the judge. The answers where the judge disagrees with people are not random; they cluster in certain types. Without knowing that, a rule like "merge if the judge says 80% correct" gives false confidence.

Forgetting that the judge knows no more than the reference. The answer saying "although the return period is 30 days" passed the judge because 30 days does not appear in the reference. You need to give the judge the context it needs (the policy text) or leave that kind of error to the rules.

Running expensive evals on every commit. Cost and waiting drive the team away from evals; separate the layers.

Judging a model upgrade by its score alone. Latency and cost are part of the decision too; measure them with the same set, in the same table.

A Small Experiment

★☆☆ Write v2 of the judge prompt: also give it the policy text and compare the agreement rate of v1 and v2 with the person on the same thirty answers. Does v2 catch the "30 days" answer in pq-05?

★★☆ Set aside three questions of the set as a holdout. Write policy-qa v3 by looking at the remaining seven questions, and measure v1, v2 and v3 first on the seven questions, then on the holdout. The difference between the two scores shows how much you fitted the prompt to the set.

★★★ Add an llm-rubric assertion to the promptfoo config and make the local qwen3:4b the grading provider. Compare its results case by case with the .NET judge in this post; do two judges with the same rubric give the same verdict on the same answers?

With this post the Prompt Engineering in Practice series ends. Across nine posts we ran the same loop for every task: task definition, success criterion, a small test set, prompt, measurement and correction. From extraction to agents, we tied the model's output to a check in code for every task; in the last two posts we set up versioning, reviewing and measuring a prompt like a piece of code. The closest posts to continue with are the RAG series post "Measuring RAG", the AI Agents and MCP post "Evaluating Agents" and the LLM Application Security post on red-team regression in CI, "Guardrail Architecture and Security Testing".