Extraction Prompts: Structured Data from Free Text, Missing Fields, Ambiguity and "I Don't Know"

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 11.0.0, OllamaSharp 5.5.2 and xUnit v3 4.0.3. Model outputs come from small open models running locally on Ollama 0.34.4: gemma3:4b (4.3 billion parameters) and, for comparison, qwen3:4b (4 billion parameters), both Q4_K_M; nothing was run against a cloud provider.

Guess First

Nevin Hanım is the purchasing officer of a school and an institutional customer of Bookstore, the online bookstore used as the shared example across this site's series. She does not use the order form; she orders by e-mail. Hakan's institutional sales team reads every e-mail and types it into the Institutional Order screen by hand. What they want sounds simple: pull the books, quantities and delivery date out of the e-mail, fill in the form, and let Hakan only approve it.

Sent: 2026-11-12 (Thursday)

Merhaba,
9. sınıflar için 40 adet Dune (ISBN 9780441013593) ve 35 adet Emma istiyoruz. Kitapların gelecek salıya kadar okulda olması gerekiyor.

Nevin Hanım
Lale Koleji Satın Alma

In English: 40 copies of Dune with ISBN 9780441013593 and 35 copies of Emma for the 9th grade, at the school by next Tuesday.

This e-mail has no ISBN for Emma. If isbn is a required string in the JSON schema, what will the model write? An empty string, Dune's number, or a number that does not exist? Which day is "gelecek salı", next Tuesday? Can the model work that out, or should the calculation be left to code? Write down your guess; below we will see the real answers on a set of eight e-mails.

v1: The Schema Is Right, the Values Are Wrong

The first attempt is the most common setup for extraction: a record, a JSON schema generated from it and a one-sentence instruction. Every field is required and none can be null:

// v1 schema: every field is required and non-nullable
public sealed record OrderRequestV1(List<RequestedItemV1> Items, string DeliveryDate);

public sealed record RequestedItemV1(string Title, string Isbn, int Quantity);

public static readonly PromptVersion V1 = new(
    Task: "order-extraction",
    Version: "v1",
    System: "Extract the book order from the email as JSON.",
    BuildUser: email => email,
    Format: ChatResponseFormat.ForJsonSchema<OrderRequestV1>());

The model has no way of knowing today's date, so the day the e-mail was sent always travels at the top of the input, in the Sent: line. The test set has eight e-mails from Nevin Hanım, each with its own difficulty: a book whose quantity is "a few", a request to "repeat last year's order exactly", an "ISBNs are attached" note, a quantity updated from 40 to 45, "twenty-five copies" written in words and an ISBN written with hyphens. The scorer counts each field separately; a single PASS/FAIL per e-mail does not show where the prompt is weak.

dotnet run --project src/Bookstore.Prompts.Lab -- order-extraction v1
order-extraction v1 | gemma3:4b | 8 cases
FAIL ox-01  Emma: isbn null -> 9780441013593; date (model) 2026-11-17 -> null
FAIL ox-02  Middlemarch: quantity null -> 3; date (model) null -> 2026-11-22
FAIL ox-03  Frankenstein: quantity null -> 1
PASS ox-04
FAIL ox-05  date (model) 2026-11-25 -> null
FAIL ox-06  Emma: isbn null -> 25; date (model) 2026-12-15 -> null
FAIL ox-07  The Hobbit: quantity null -> 0; date (model) null -> 2026-11-30
FAIL ox-08  date (model) 2026-12-02 -> 2026-11-30
1/8 passed (13%)

items found       12/12 (extra: 0)
quantity correct  9/12 (invented: 3)
isbn correct      10/12 (invented: 2)
date by model     2/8

Every book was found; the problem is in the values. Three of the raw outputs tell the story well:

{"items": [{"title": "Jane Eyre", "isbn": "Ekte", "quantity": 30}, {"title": "Pride and Prejudice", "isbn": "Ekte", "quantity": 30}], "deliveryDate": "Acelemiz yok"}
{"items": [{"title": "Emma", "isbn": "25 adet", "quantity": 25}], "deliveryDate": "15 Aralık'tan önce teslim edilmesi gerekiyor"}
{"items": [{"title": "Middlemarch", "isbn": "978-0-14-143954-9", "quantity": 20}, {"title": "The Hobbit", "isbn": "N/A", "quantity": 0}], "deliveryDate": "2026-11-30"}

Each one matches the schema. isbn had to be a string, and the model wrote strings: "Ekte", which means attached, "25 adet", which means 25 copies, and "N/A". quantity had to be an integer: 0 for The Hobbit, whose quantity is not known, 3 for "a few" in ox-02, 1 for "same as last year". In ox-01 Dune's ISBN was copied onto Emma. A field that cannot be null forces the model to invent something when the information is missing; since the schema offers no "leave it empty" option, the model writes the most plausible-looking value. The date field is in the same position: sometimes the model wrote an ISO date, sometimes a sentence like "Acelemiz yok", we are in no hurry. Only two of the eight dates are correct.

v2: Nullable Fields, Field Descriptions and a Source Quote

In v2 we change the schema. There are three rules: a field is null when the information is missing; the meaning of each field is written in the schema; and every item carries the sentence it came from, word for word. The Description attributes end up as description in the JSON schema that Microsoft.Extensions.AI generates, so a field description becomes part of the prompt for the model:

// v2 schema: nullable fields, descriptions and a source quote per item
public sealed record OrderRequest(
    [property: Description("One entry per book the sender wants to order now.")]
    List<RequestedItem> Items,
    [property: Description("The delivery date phrase copied exactly from the email, e.g. \"by next Tuesday\". null if no date is mentioned.")]
    string? DeliveryDateText,
    [property: Description("Your reading of DeliveryDateText as yyyy-MM-dd, using the Sent date. null if no date is mentioned.")]
    string? DeliveryDate);

public sealed record RequestedItem(
    [property: Description("Book title as written in the email.")]
    string Title,
    [property: Description("ISBN-13 digits only, only if written in the email body. Otherwise null.")]
    string? Isbn,
    [property: Description("Number of copies requested now. null if the email gives no exact number.")]
    int? Quantity,
    [property: Description("The sentence from the email this item comes from, copied word for word.")]
    string SourceQuote);

There are two fields for the date. DeliveryDateText is the exact phrase from the e-mail ("gelecek salıya kadar"); in production, code will calculate the date from that phrase. DeliveryDate is the model's own calculation; we ask for it only for comparison, so that one run can answer "model or code". The system prompt also gets rules specific to this kind of e-mail:

## Rules
- Extract only what the email states. Do not use outside knowledge to fill fields.
- A field the email does not state is null. Words like "a few", "same as last year"
  or "we will confirm later" are not quantities: use null.
- Write numbers given in words as digits ("twenty-five" is 25).
- If the sender changes an earlier quantity, use the new one.
- ISBN: digits only, without hyphens, and only if it is written in the email body.
  "ISBNs are attached" means null.
- Copy SourceQuote and DeliveryDateText exactly as they appear in the email.

items found       8/12 (extra: 0)
quantity correct  8/12 (invented: 0)
isbn correct      8/12 (invented: 0)
date by model     2/8
date by code      6/8
quotes in email   6/8

Invented values dropped to zero, but four books disappeared. In three e-mails the model returned an empty items list: ox-02 with two books of unknown quantity, ox-06 with "twenty-five copies" written in words, and the short ox-08. In ox-04 it also wrote the string "null" instead of JSON null in sourceQuote. This is a common side effect with small models: when the "write null if you don't know" rule is strong, the model skips an item it is unsure about altogether. In extraction two error types work against each other: hallucination and omission. A prompt change can reduce one while increasing the other, which is why both have to be counted separately.

v3: "List Every Book" and a Single Example

v3 adds two lines and one example to v2's rules. The example is not one of the test-set e-mails; it shows a book whose quantity is written in words and a book whose quantity is unknown, side by side:

- Every book the sender wants to order goes into Items, even when its quantity is null.
- Use JSON null, never the string "null".

## Example
<email>
Sent: 2026-10-05 (Monday)

Hello, we need twelve copies of Persuasion and some copies of Walden, I will confirm
the number. ISBN for Walden: 978-0-00-000000-2. Delivery by 20 October please.
</email>
{"items": [
  {"title": "Persuasion", "isbn": null, "quantity": 12,
   "sourceQuote": "we need twelve copies of Persuasion"},
  {"title": "Walden", "isbn": "9780000000002", "quantity": null,
   "sourceQuote": "some copies of Walden, I will confirm the number"}],
 "deliveryDateText": "by 20 October", "deliveryDate": "2026-10-20"}

dotnet run --project src/Bookstore.Prompts.Lab -- order-extraction v3
order-extraction v3 | gemma3:4b | 8 cases
PASS ox-01
PASS ox-02
FAIL ox-03  Frankenstein: quote not in email
PASS ox-04
PASS ox-05
PASS ox-06
PASS ox-07
PASS ox-08
7/8 passed (88%)

items found       12/12 (extra: 0)
quantity correct  12/12 (invented: 0)
isbn correct      12/12 (invented: 0)
date by model     4/8
date by code      8/8
quotes in email   11/12

Items, quantities and ISBNs are complete. The failure in ox-03 is interesting: the model wrote the quote as "Geçen yıl aldığımız Frankenstein siparişını", while the e-mail says "siparişini". One letter changed, and the quote no longer appears in the e-mail. That is exactly the benefit of asking for a word-for-word quote: code can check whether the model really read the text by checking that the quote appears in the source. If the quote does not match, the item is not processed automatically; it goes to Hakan.

Dates: Should the Model Calculate, or the Code?

In the same run the model got four of the eight dates right; the code, working from the phrase the model copied, got all eight. The difference is not a coincidence. Turning a phrase like "next Tuesday" into a date takes two steps: finding the phrase and doing calendar arithmetic. The first is language work, and LLMs are good at it. The second is an exact calculation, and small models are inconsistent at it: for "önümüzdeki çarşamba", next Wednesday, sent on Wednesday 25 November, the model said 30 November; the right answer is 2 December. In code the calculation is a few lines:

foreach ((string stem, DayOfWeek day) in Days)
{
    if (!text.Contains(stem))
    {
        continue;
    }
    int ahead = ((int)day - (int)sent.DayOfWeek + 7) % 7;
    DateOnly next = sent.AddDays(ahead == 0 ? 7 : ahead);
    // "Gelecek cuma" said on a Monday: this Friday or the one after? Do not guess.
    bool sameWeek = ISOWeek.GetWeekOfYear(next.ToDateTime(TimeOnly.MinValue))
        == ISOWeek.GetWeekOfYear(sent.ToDateTime(TimeOnly.MinValue));
    bool saysNext = text.Contains("gelecek") || text.Contains("önümüzdeki") || text.Contains("haftaya");
    return saysNext && sameWeek ? new(null, true) : new(next, false);
}

Code has another advantage: it can recognise ambiguity. ox-02 was sent on a Monday and asks for delivery "gelecek cuma", next Friday. In Turkish that can mean the Friday of the same week or the following one. In that case the resolver does not invent a date; it returns Ambiguous, and the order screen shows a "confirm the delivery date" warning. In v1 the model had written 22 November for ox-02, which is a Sunday. In v3 it left the date empty, but for ox-04, which says "we are in no hurry", it wrote the day the e-mail was sent, 20 November. A phrase the resolver does not recognise, such as "towards the end of the month", also means null and a person's approval. This resolver covers only the phrase types in the set; in production, log every phrase it does not recognise and extend its coverage from those logs.

The general rule: give the language part of normalization to the model and the calculation part to code. Unit conversion, money amounts, stripping hyphens from an ISBN and date arithmetic are deterministic code work. Let the model copy the raw phrase; that phrase also stays in the log, and when debugging you can see what was turned into what.

The Validator Says What the Schema Cannot

I also ran v3 with another small model, qwen3:4b. The total is 6/8; one of the failures is in ox-01 and is the same as v1's: Dune's ISBN was written onto Emma. The prompt says "ISBN only if it is written in the e-mail"; the ISBN is written in the e-mail, just next to another book. Neither the schema nor the prompt can catch that. The validator checks whether the title closest to the ISBN in the e-mail is the item's own title:

else if (emailDigits.IndexOf(isbn, StringComparison.Ordinal) is int at and >= 0)
{
    // The ISBN must sit next to this item's title, not next to another book
    string? nearest = NearestTitle(extraction, email.Body, digitPositions[at]);
    if (nearest is not null && !string.Equals(nearest, item.Title, StringComparison.OrdinalIgnoreCase))
    {
        issues.Add($"{item.Title}: ISBN {isbn} is written next to {nearest}");
    }
}

The validator also checks the ISBN-13 check digit, that the quantity is positive, that the quote appears in the e-mail and that at least one item was found. If it finds a problem, the e-mail is not processed automatically:

ox-01 review: Emma: ISBN 9780441013593 is written next to Dune
ox-02 auto
ox-03 auto
ox-04 auto
ox-05 auto
ox-06 auto
ox-07 auto
ox-08 auto

The same run also shows a limit: in ox-06 qwen3:4b wrote only "r" as the book title, and the validator let it through, because the letter "r" appears in the e-mail and the item has no ISBN. The fix for that gap is not the prompt but matching the title against the records in Catalog. An item that does not match goes to Hakan too.

Production Notes

Measure per field. The difference between 7/8 and 1/8 per e-mail does not tell you which field improved or why. Count invented and lost values separately; they cost different things. A wrong quantity means a wrong shipment, while a lost item is a gap Hakan will notice and add.

Make human approval part of the design. The goal of extraction is not to remove Hakan but to turn his job from "read and type" into "check and approve". Every validator warning appears next to its field on the screen, with the quote beside it, so Hakan can check without opening the e-mail.

Pass the input as it is, but bound it. E-mail signatures, quoted old messages and attachments bloat the context. The older message can be necessary for the model, as in ox-05's "we said 40, let's make it 45"; still, sending the last message plus the previous order in structured form is more controlled than sending the whole thread.

Think about personal data. Institutional e-mails contain names, phone numbers and addresses. Know what goes into the prompt and into the logs; masking sensitive data before it reaches the model is a topic of the LLM Application Security series.

Common Pitfalls

Trying to express missing information with a required field. int Quantity cannot carry an "unknown" value; the model writes 0, 1 or 3. Every field that might be unknown should be nullable, and its description should say what null means.

Mistaking schema validity for correctness. All eight of v1's outputs matched the schema; only one was correct.

Reducing hallucination while increasing loss. In v2 invention went to zero and four items disappeared. Someone looking at a single metric would have called v2 an improvement.

Leaving calendar arithmetic to the model. The model finds the phrase "next Tuesday" correctly but can get wrong which day it falls on. Have the model copy the phrase and calculate the date in code.

Asking for a quote and never checking it. Until code looks for the quote in the e-mail, it is just a field that looks reassuring.

A Small Experiment

★☆☆ Add an e-mail sent on 20 November that says "ayın 5'ine kadar", by the 5th of the month, to the test set. What does the resolver return, and what does the model write? Write the expected value yourself first.

★★☆ In v3, remove the example and keep only the two new rules. Does the number of lost items go back to v2's, or stay at v3's? That shows whether the improvement came from the rules or from the example.

★★★ Add Catalog matching to ExtractionValidator: write a small ICatalogLookup interface that holds a list of titles, mark items whose title does not match as "review", and show with a test that qwen3:4b's "r" output in ox-06 is now caught.

In the next post we go back to Gül's inbox: a classification prompt that routes tickets to teams. Mutually exclusive label definitions, choosing few-shot examples from boundary cases, whether the label order changes the result, and why the model's own "confidence" is of no use will all be measured there.