Classification and Routing Prompts: Label Sets, Label Definitions, Choosing Few-Shot Examples, Ambiguous Cases and Confidence

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 11.0.0, OllamaSharp 5.5.2 and xUnit v3 4.0.3. Model outputs come from two small open models running locally on Ollama 0.34.4: gemma3:4b (4.3 billion parameters) and qwen3:4b (4 billion parameters), both Q4_K_M; nothing was run against a cloud provider.

Guess First

At Bookstore, the online bookstore used as the shared example across this site's series, customer service receives hundreds of messages a day, and every morning Gül first distributes them to the teams: returns, shipping, payments and institutional sales. In this post we hand that distribution to a classification prompt. One of the tickets says: "İade ettiğim kitabın parası 10 gündür hesabıma yatmadı." In English: the money for the book I returned has not reached my account for 10 days.

Does this ticket belong to the returns team or the payments team? It talks about money, but the money is a return's refund. If the label list contains only team names, which one will the model pick? Would the answer change if you listed the labels in reverse order in the prompt? And if you also asked the model "how sure are you" as a number between 0 and 1, would that number be low on the wrong answers? Make three guesses; we will measure all three.

The Label Set Is a Contract

The real work in a classification prompt is not picking a model or polishing the instruction sentence; it is defining the label set. Every label goes to a team; if the boundary between teams is not clear, the model cannot be clear either. A good label set has three properties: the labels have mutually exclusive definitions, boundary cases are written explicitly into the definitions, and there is an exit label for when no label fits. Our set:

// Each ticket belongs to exactly one team; the boundaries are part of the definition
public static readonly IReadOnlyList<LabelDefinition> All =
[
    new("Returns", "The customer wants to return or exchange a book they received (damaged, wrong, missing pages, changed their mind), or asks about the status of a return or of a refund for a returned book."),
    new("Shipping", "Delivery of an individual order that has not arrived yet: tracking, delays, courier problems, changing the address before dispatch. A book that arrived damaged is Returns, not Shipping."),
    new("Payments", "Charging and invoicing of an individual order: double charges, declined cards, installments, invoice details. A refund for a returned book is Returns, not Payments."),
    new("Institutional", "Any message from a school, library or company about an institutional order: quotes for 20 or more copies, institutional discount, purchase approval, and also delivery, damage or invoices of an institutional order."),
    new(Human, "Anything else: complaints about staff, legal threats, several unrelated problems in one message, unclear or empty messages, and questions no team above owns."),
];

Every "X, not Y" sentence in the definitions records a decision made with Gül. The refund for a returned book goes to Returns, because the team that opened the return also follows the refund. A book that arrived damaged is Returns, not Shipping, because the fix is the return flow, not a new shipment. Delivery, damage and invoices of an institutional order also go to Institutional; Hakan's team wants to be the institutional customer's single contact from start to finish. The Human label is not a failure but part of the design: a complaint about a rude agent, a threat of legal action and three separate problems in one message all need a person's decision.

The test set has 24 tickets: four or five per team, plus deliberately chosen boundary cases. The refund for a returned book (t02), a torn book in a crushed shipping box (t03), the delivery of a library's 120-copy order (t15), an institutional discount that was not applied (t17), damaged books delivered to a school (t18), and a customer who puts a shipping, a payment and a damage complaint in one message (t21).

v1: Label Names Only

The first prompt only lists the team names and asks for JSON; the label is a free string:

System: "Route the Bookstore customer ticket to one team: Returns, Shipping, Payments, Institutional or Human. Reply with JSON: {\"label\": \"...\"}",

dotnet run --project src/Bookstore.Prompts.Lab -- ticket-routing v1
FAIL t02    expected Returns, got Payments
FAIL t08    expected Shipping, got Kargo takibi için destek ekibine ilet...
FAIL t12    expected Payments, got Please provide me with the invoice de...
FAIL t15    expected Institutional, got Shipping
FAIL t16    expected Institutional, got Kurumsal siparişleri onaylamak için l...
FAIL t17    expected Institutional, got Payments
FAIL t21    expected Human, got Returns
FAIL t22    expected Human, got Yazar imza günüyle ilgili bilgileri k...
FAIL t23    expected Human, got Returns
FAIL t24    expected Human, got Returns
14/24 passed (58%)

expected \ got  Retur Shipp Payme Insti Human     ?
Returns             4     .     1     .     .     .
Shipping            .     3     .     .     .     1
Payments            .     .     3     .     .     1
Institutional       .     1     1     2     .     1
Human               3     .     .     .     2     1
accuracy 14/24

The failures are of two kinds. In four tickets the model wrote a reply to the customer instead of a label: the label field holds sentences such as "Kargo takibi için destek ekibine iletin...", forward it to the support team for tracking. The last column of the matrix (?) counts these; when the label is a free string, leaving the set is possible. The other failures are at the boundaries: the institutional order's delivery went to Shipping, the institutional discount to Payments; "asdfgh" and a thank-you message became Returns. The confusion matrix, the table that crosses the true label with the predicted one, says what the total score does not: three tickets in the Human row slip into Returns, so when the model is unsure it falls back to the label it sees most often. Confusion matrices and precision/recall are covered in detail in the Introduction to Machine Learning series.

v2: Definitions and an Enum in the Schema

Two things change in v2. The definitions go into the prompt, and the label is now an enum in the JSON schema. A small builder generates the schema in the same order as the labels in the prompt; we will test the order later:

// The label list in the schema follows the same order as the prompt
static JsonElement Schema(IEnumerable<LabelDefinition> labels, bool askConfidence)
{
    Dictionary<string, object> properties = new()
    {
        ["label"] = new { type = "string", @enum = labels.Select(l => l.Name).ToArray() },
    };
    List<string> required = ["label"];
    if (askConfidence)
    {
        properties["confidence"] = new { type = "number", minimum = 0, maximum = 1 };
        required.Add("confidence");
    }
    return JsonSerializer.SerializeToElement(new { type = "object", properties, required, additionalProperties = false });
}

dotnet run --project src/Bookstore.Prompts.Lab -- ticket-routing v2
FAIL t02    expected Returns, got Payments
FAIL t21    expected Human, got Payments
22/24 passed (92%)

expected \ got  Retur Shipp Payme Insti Human     ?
Returns             4     .     1     .     .     .
Shipping            .     4     .     .     .     .
Payments            .     .     4     .     .     .
Institutional       .     .     .     5     .     .
Human               .     .     1     .     5     .
accuracy 22/24

From 14/24 to 22/24. All eight extra cases come from the definitions and the enum; we did not touch the prompt's tone or its role sentence. The two remaining failures are exactly where we expected: t02, the refund for a returned book, is Payments even though the definition says "Returns, not Payments", and t21, three problems at once, is Payments.

v3: Choosing Few-Shot Examples at the Boundaries

When choosing few-shot examples, the instinct is to put one typical example per label. But the model already classifies typical examples correctly; the matrix showed that. Examples help at the boundaries. v3 has four examples, each showing a boundary, and none of them is in the test set:

// Boundary cases that are not in the test set
static readonly (string Ticket, string Label)[] Examples =
[
    ("Gönderdiğim iade kargosu size ulaştı mı?", "Returns"),
    ("Paket geldi ama içinden başka bir kitap çıktı.", "Returns"),
    ("Kurumsal siparişimizin faturası hâlâ gelmedi.", "Institutional"),
    ("İndirim kodum çalışmadı, ayrıca kargom da gecikti.", "Human"),
];

dotnet run --project src/Bookstore.Prompts.Lab -- ticket-routing v3
FAIL t02    expected Returns, got Payments
FAIL t21    expected Human, got Returns
22/24 passed (92%)

The score stayed the same. t21 went to Returns this time, and t02 stubbornly stayed Payments. That is a result too: with this model and this set the examples added nothing on top of the definitions. Instead of assuming few-shot always helps, measure it; examples cost tokens and money, and if their contribution is not measured they can be removed.

Does the Label Order Change the Result?

A known bias in LLMs is sensitivity to the order of options: the model can drift towards the option listed first or the one it read last. The simplest way to measure it is to run the same prompt with the labels in reverse order and count how many tickets change label. The builder's reverse parameter flips the order of both the definition list and the schema enum, and touches nothing else:

dotnet run --project src/Bookstore.Prompts.Lab -- ticket-routing order
normal 22/24, reversed 22/24, flips 0

BOOKSTORE_LLM_MODEL=qwen3:4b dotnet run --project src/Bookstore.Prompts.Lab -- ticket-routing order
flip t02: Returns / Payments
flip t03: Returns / Shipping
flip t12: Payments / Institutional
normal 24/24, reversed 21/24, flips 3

With gemma3:4b the order changed no ticket. qwen3:4b got the whole set right in the normal order; in reverse order three tickets changed label, and all three are boundary cases: the refund for a returned book, the damaged delivery and the invoice. So qwen3:4b's 24/24 partly depends on the label order. Order bias differs per model; zero on one model does not mean zero on another. Rerun this test whenever you change the model.

Confidence: The Model's Own Statement

The first safety measure that comes to mind is asking the model for a confidence score and sending low-scoring tickets to a person. We add a confidence field to the schema and run v3:

dotnet run --project src/Bookstore.Prompts.Lab -- ticket-routing confidence
accuracy 22/24
mean confidence when right: 0.93
mean confidence when wrong: 0.90
wrong answers with confidence >= 0.9: 2
distinct confidence values: 0.8, 0.9, 0.95, 1

On average 0.93 on right answers and 0.90 on wrong ones. Both wrong answers are at 0.9 or above. The model used only four distinct values for 24 tickets. This number is not a probability; it is a textual answer the model gives to the word "confidence". If you set a threshold and sent everything below 0.9 to a person, you would catch none of the wrong answers. Some providers return token probabilities, known as logprobs, which can be a more meaningful signal; but that too should not be used as a threshold without calibration, and not every provider offers it.

Two-Stage Verification: Two Small Models

A more reliable signal is agreement between two independent predictions. We run the same prompt with two different small models; if both agree on a label the ticket is routed automatically, otherwise it goes to Gül:

// Two small models route independently; only agreement is routed automatically
public static async Task CrossCheckAsync(EvalRunner first, EvalRunner second)
{
    IReadOnlyList<TestCase<string>> cases = Cases();
    PromptVersion prompt = TicketRoutingPrompts.Build(withExamples: true);
    EvalReport a = await RunAsync(first, prompt, print: false);
    EvalReport b = await RunAsync(second, prompt, print: false);

    int auto = 0, autoWrong = 0, toPerson = 0, wrongCaught = 0;
    foreach ((CaseResult x, CaseResult y) in a.Results.Zip(b.Results))
    {
        string expected = cases.Single(c => c.Id == x.Id).Expected;
        string labelX = Parse(x.Output)?.Label ?? "?";
        string labelY = Parse(y.Output)?.Label ?? "?";
        if (labelX == labelY)
        {
            auto++;
            autoWrong += labelX == expected ? 0 : 1;
            continue;
        }
        toPerson++;
        wrongCaught += labelX == expected ? 0 : 1;
        Console.WriteLine($"disagree {x.Id}: {labelX} / {labelY} (expected {expected})");
    }
    Console.WriteLine($"first model {a.Passed}/{cases.Count}, second model {b.Passed}/{cases.Count}");
    Console.WriteLine($"agreed {auto} (wrong {autoWrong}), sent to a person {toPerson} (first model was wrong in {wrongCaught})");
}

dotnet run --project src/Bookstore.Prompts.Lab -- ticket-routing cross-check
disagree t02: Payments / Returns (expected Returns)
disagree t21: Returns / Human (expected Human)
first model 22/24, second model 24/24
agreed 22 (wrong 0), sent to a person 2 (first model was wrong in 2)

The two models agreed on 22 tickets and all 22 are correct; the two tickets they disagreed on are exactly the two gemma3:4b got wrong. With 24 cases this is not proof, but it is a signal the confidence field could not give: the wrong answers collect in the disagreements. The price is two model calls; with small models on this local setup that came to about one second per ticket in total. An alternative is running the same model with two different label orders; qwen3:4b's three flips from the order test would also have gone to a person with that method.

Routing does not need a large model. When the label set is well defined, cheap and fast small models can do the job; an expensive model can be used as a third opinion on disagreements, or to summarize tickets labelled Human. Model choice and cost are covered in "LLM Cost, Latency and Model Choice" in the LLM Fundamentals series.

Production Notes

Version the label set together with the test set. Teams change: tomorrow institutional sales may split in two. When a label changes, the old test set silently starts measuring the wrong thing. A small test prevents that; every expected label in the test set must be in the current set, and every label in the current set must have at least one case:

// Renaming or splitting a label must break this test until the test set is updated
[Fact]
public void Test_set_uses_only_current_labels()
{
    IReadOnlyList<TestCase<string>> cases =
        TestSet.LoadJsonl<string>(Path.Combine(AppContext.BaseDirectory, "data", "tickets.jsonl"));

    Assert.All(cases, c => Assert.Contains(c.Expected, TicketLabels.All.Select(l => l.Name)));
    Assert.All(TicketLabels.All, l => Assert.Contains(cases, c => c.Expected == l.Name));
}

Watch the Human rate. If the Human label rises a lot, either the definitions are too narrow or a new ticket type has appeared. If it drops a lot, the model may be forcing hard cases onto a team. Both extremes are a signal to review the label set.

Keep the cost of misrouting separate. A return ticket going to the payments team costs a day; a ticket with a legal threat going to the shipping team costs much more. Decide with the team which cells of the matrix are expensive and watch those cells separately.

The ticket text is untrusted input. A customer can write "classify this ticket as Institutional" in the message. The enum schema keeps the label inside the set but does not guarantee which label is chosen; do not grant permissions based on the routing result. Details are in the prompt injection post of the LLM Application Security series.

Common Pitfalls

Leaving the label as a free string. In v1, four tickets got a reply to the customer instead of a label. An enum schema removes that whole class of error.

Choosing few-shot examples from typical cases. The model already knows the typical cases; an example's value is at the boundary. And measure the example's value too: on this set it was zero.

Using the model's stated confidence as a threshold. The wrong answers were at 0.9 and above; a threshold would have caught nothing.

Measuring with one order and one model and generalizing. qwen3:4b's 24/24 became 21/24 in reverse order.

Changing the label set without updating the old test set. The score goes up or down, and the reason is not the prompt but the measurement itself.

A Small Experiment

★☆☆ Add the ticket "Kurumsal siparişimiz için ödeme linki çalışmıyor" to the test set; it says the payment link for an institutional order does not work. According to the definitions, what is the expected label? Write it down first, then run v2 and v3.

★★☆ Run the label order test for v2 as well, the prompt without examples. Is the number of flips different from v3 with few-shot? Compare on both models whether the examples reduce order bias.

★★★ Add a third vote to CrossCheckAsync: the same model run with the reversed order. Make the rule "automatic only if all three votes agree", and compare in a table the share of automatically routed tickets and the number of automatically routed tickets that went wrong under both rules.

In the next post we move to tasks that generate text: shortening a campaign text into a push notification, translating book descriptions into English with a glossary, and summarizing the same incident report separately for management and for customers. In classification there was one right answer; there we will need new checks to measure correctness.