Working with Long Documents and Context: Placement, Asking for Quotes, Chunking and Ordering for Prompt Caching

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 11.0.0, OllamaSharp 5.5.2 and xUnit v3 4.0.3. Model outputs and timings come from gemma3:4b, a small open model (4.3 billion parameters, Q4_K_M) running locally on Ollama 0.34.4; nothing was run against a cloud provider. OpenAI's and Anthropic's prompt caching rules were read from their official documentation on 15 November 2026.

Guess First

The purchasing team of Bookstore, the online bookstore used as the shared example across this site's series, keeps asking the same kinds of questions about the contract signed with the book supplier: within how many days must we report a damaged book, how much unsold stock can we return, who pays for shipping? The contract has 31 clauses and three annexes; about 14,500 words, roughly forty printed pages. Some answers are in Clause 17, some in the amendment protocol at the end, Ek-3; Ek-3 changes two provisions of Clause 17.

The model has a 128 thousand token context window; putting the whole contract into the prompt and asking the question is the easiest route. Guess three things: how many tokens is the contract? Should the document go before or after the question and the instructions? If you ask the model to back its answer with quotes from the contract, does accuracy go up?

Context Budget: Count the Tokens First

The contract is in Turkish, and gemma3's tokenizer splits Turkish text into more pieces than English: 14,500 words come to 33,496 tokens. Find this number by measuring, not guessing, because how much of the context window you can use comes from the environment you run, not from the number the model claims. In Ollama the context length is set per request with num_ctx; the default is too small for this contract. In the Lab we pass it through ChatOptions.AdditionalProperties:

// Ollama's context window is set per request; the default is too small for this contract
public static ChatOptions Options(int contextTokens)
{
    ChatOptions options = LabClient.Options();
    options.AdditionalProperties = new() { ["num_ctx"] = contextTokens };
    return options;
}

When I asked the same questions with num_ctx 32,768 I got no error at all; answers came back and the JSON was valid. But Ollama's log had this line, and the Usage.InputTokenCount of every answer was 16,387 instead of 33,600:

level=WARN source=llama_server.go:320 msg="truncating input prompt" limit=16387 prompt=33654 keep=5 new=16387

Ollama silently cut the prompt and the model never saw half of the contract. In that run only one of the six questions passed; two answers rely on the old provisions of Clause 17 instead of Ek-3. Any code that sends long documents should compare the input token count in the response with the expected number: if they differ, the answer should not be used. Cloud providers usually reject a request that exceeds the context with an error; local servers and some gateways may truncate instead. The RAG side of the context budget, meaning how many chunks fit and "I don't know" on a low score, is in "Putting Context into the Prompt" in the RAG series.

Where Does the Document Go: First or Last?

Anthropic's prompting guide recommends putting long documents at the top of the prompt, above the question, the instructions and the examples; it says that in its tests putting the question at the end improved response quality by up to 30 percent, especially with complex multi-document inputs. It also recommends tagging the document together with its source. The prompt builder can produce both placements:

// Stable part (the document) first, the changing question last, or the other way round
BuildUser: question => placement == Placement.DocumentFirst
    ? $"{document}\n\n<question>{question}</question>"
    : $"<question>{question}</question>\n\n{document}",

The document variable wraps the contract in a <document source="supplier-contract-tr.md"> tag. The test set has six questions: two provisions changed by Ek-3, three provisions from Clause 17, and a topic the contract does not cover at all, e-book returns, whose correct answer is NOT_FOUND.

Grounding the Answer in Quotes

In v1 the model writes only the answer. In v2 it first copies the relevant clauses from the contract word for word, with their clause numbers, and then writes the answer based only on those quotes. In the schema the Quotes field comes before Answer; because the model generates the JSON from top to bottom, it writes the quotes before the answer:

// Quotes come before the answer in the schema, so the model writes them first
public sealed record GroundedAnswer(List<ContractQuote> Quotes, string Answer);

The grader checks two things separately: that the expected information appears in the answer, compared with Turkish casing rules, using the tr-TR CompareInfo so that "TEDARİKÇİ" matches "tedarikçi", and that every quote really appears in the contract. The expected information is written as a stem such as "10 iş gün"; on the first try I had written "10 iş günü", and because the model said "on (10) iş güne düşmüştür" a correct answer was counted as a failure. A keyword check on Turkish text has to cover the suffixed forms too. With the document first, the two versions follow; I shortened v2's cq-02 and cq-06 lines and the long quote lists, and v1's first question took 2.2 seconds because the same prefix was still in memory from a previous run, for a reason explained below:

dotnet run --project src/Bookstore.Prompts.Lab -- contract ask v1 doc-first
PASS cq-01   2.2s in=33628 Madde 17'ye göre hasarlı bir kitabı tedarikçiye on beş (1...
PASS cq-02   3.4s in=33621 Madde 17.2'de belirtilen on beş (15) günlük bildirim süre...
PASS cq-03   1.5s in=33616 Madde 17.1'e göre, kusurlu kitapların iadesinde kargo ücr...
PASS cq-04   3.8s in=33612 Teslimatın Madde 9'da belirtilen süreyi aşması halinde TE...
FAIL cq-05   2.0s in=33614 answer "Madde 17.2'de belirtilen on beş (15) günlük bildirim süre..." lacks 6 ay|altı (6) ay|altı ay
FAIL cq-06   1.1s in=33614 expected NOT_FOUND, got "Hayır, sözleşmede e-kitapların iadesiyle ilgili bir hüküm..."
4/6 passed

dotnet run --project src/Bookstore.Prompts.Lab -- contract ask v2 doc-first
FAIL cq-01  44.9s in=33661 no quote from Ek-3 (cited: 17.1, 3.1)
FAIL cq-03   4.4s in=33649 answer "BOOKSTORE, kusurlu kitapların iadesiyle ilgili kargo ücre..." lacks tedarikçi; no quote from 17.6 (...)
FAIL cq-04   3.4s in=33645 answer "TEDARİKÇİ, iade edilen kitapları teslim tarihinden itibar..." lacks 30 gün|otuz (30) gün|otuz gün; 1/1 quotes not in the contract; no quote from 17.7 (cited: 27.2)
FAIL cq-05   4.6s in=33647 1/1 quotes not in the contract
0/6 passed

At first sight v1 is good and v2 a disaster. But a closer look at v1's 4/6 changes the picture. cq-03's answer is right, the supplier pays for shipping, but it cites "Clause 17.1"; the provision is in 17.6. cq-02's answer starts with the notice period, which has nothing to do with the question, and the right rate of 30% only comes in the second sentence. Half of cq-04's answer talks about the late-delivery penalty in Clause 22. v1's passing answers passed because the word we check appears somewhere in the answer; nothing shows that the model actually read the contract in the right place.

v2 makes the same weakness visible. Most of the quotes are not in the contract: the model rewrote clauses in its own words, mixed up clause numbers by calling the text of 17.4 "17.1", and in cq-04 invented a "Clause 27.2" that does not exist; according to that clause the credit note is issued within 15 days, not 30. In cq-05 the answer is right but the quote is not word for word. A 4-billion-parameter model struggles to quote exactly from a 33-thousand-token text; asking for quotes did not make it more accurate, but it made its mistakes catchable in code. In production, trusting one of v1's right-looking answers and missing a return deadline is far more expensive than v2 saying "I could not show the source".

What Happens When the Question Comes First?

I ran the same v2 prompt with the question first and the document last. The result was 0/6 again, but the kind of failure changed: in cq-01 the model poured out 37 quotes, 13 of which are not in the contract; in cq-03 it kept generating for 258 seconds and in the end did not even produce valid JSON. Running v1 with the question first also gives 4/6, but the answers are messier: cq-01 starts with "Madde 1'de belirtildiği üzere", as stated in Clause 1, and cq-05 ends with a meaningless sentence about "10 months". With one model and a six-question set there is not enough data to claim an accuracy difference between placements in numbers; but one difference is certain: time. With the document first, every question after the first was answered in a few seconds; with the question first, every question was processed from scratch in 35–40 seconds. The reason is the topic of the next section.

Stable Part First: Prompt Caching

When a model processes a prompt it computes an intermediate state for every token, the KV cache. If the next request starts with the same prefix, that computation does not have to be repeated. The llama.cpp server running under Ollama keeps the previous request's prefix in memory and reuses it for a new request that starts the same way. We ask the same three questions in order with both placements and measure how long the model takes to process the prompt; the duration comes from the PromptEvalDuration field of OllamaSharp's raw response:

ChatDoneResponseStream? done = response.RawRepresentation as ChatDoneResponseStream;
double promptSeconds = (done?.PromptEvalDuration ?? 0) / 1e9;

dotnet run --project src/Bookstore.Prompts.Lab -- contract cache
DocumentFirst  cq-01 total  42.7s prompt eval  35.9s tokens 33661
DocumentFirst  cq-02 total   7.8s prompt eval   0.2s tokens 33654
DocumentFirst  cq-03 total   4.1s prompt eval   0.2s tokens 33649
QuestionFirst  cq-01 total 116.8s prompt eval  35.9s tokens 33661
QuestionFirst  cq-02 total  41.3s prompt eval  36.1s tokens 33654
QuestionFirst  cq-03 total 252.4s prompt eval  36.2s tokens 33649

With the document first, the first question processes the whole contract in 35.9 seconds; for the next two questions prompt processing drops to 0.2 seconds, because everything from the system prompt to the end of the contract is identical and only the question at the end is new. With the question first, the first few tokens differ in every request and nothing after them can be reused: every question takes 36 seconds again. The token count is the same in both cases; what differs is where the cost goes. Most of cq-03's 252 seconds in total is not prompt processing but the model producing quotes without stopping.

With cloud providers the same logic turns into a line on the bill. According to OpenAI's documentation, prompt caching is on by default for supported models; the cache requires the beginning of the prompt to match exactly, for GPT-5.6 and later models the cacheable prefix must be at least 1,024 tokens, and the number of tokens read from the cache shows up in the response's usage.input_tokens_details.cached_tokens field. The guide recommends putting stable content such as instructions, tool definitions and reference text first and changing content last, and appending new messages instead of rewriting old ones. At Anthropic the prefix order is tools, system and messages; where the cache ends is marked with cache_control, at most at four points, or automatic caching is switched on at the request level. The default lifetime is five minutes and is refreshed on every hit; writing to the cache costs 1.25 times the base input price, reading costs 0.1 times on most models. If the prefix is below a minimum between 512 and 4,096 tokens depending on the model family, it is silently not cached. Details and current prices are in the Claude API series and in the official documentation; these figures are dated 15 November 2026.

The effect on prompt design is a simple rule: order the prompt from what changes least to what changes most. System prompt and tool definitions first, the reference document after them, and everything specific to the user or the request, such as the question, the date or the user name, last. Putting a value that changes on every request, such as "today's date", into the system prompt invalidates the whole prefix behind it on every request. Stable parts such as the glossary from the previous post should also stay at the start of the prompt, in the same order on every request; that is why it matters in C# that the order is stable when you iterate a Dictionary.

Chunking: Map-Reduce

If a small model cannot quote exactly from 33 thousand tokens, giving it smaller pieces is the obvious fix. Map-reduce has two steps. In the map step the contract is split into chunks of four clauses, and return-related clauses are collected word for word from each chunk. Code looks up every collected quote in the contract and drops the ones it cannot find. In the reduce step the questions are asked against a short document made only of these verified quotes. The map prompt:

System: """
    You read one part of Bookstore's supplier contract. Copy, word for word, the clauses
    in this part that are about BOOKSTORE returning books to the supplier: notice
    deadlines, limits on unsold stock, who pays return shipping, credit notes, and any
    clause that explicitly changes one of those return clauses.
    General clauses about notices, documents, cooperation, payments or penalties do not
    count. Clause is only the number, such as "17.2" or "Ek-3.1".
    Return at most 8 clauses, and an empty list if this part has none.
    """,

This prompt is the second version. The first had no "at most" limit and the chunks had six clauses; in the third chunk the model was writing more than 80 quotes when the output was cut off, and the whole run died with a JsonException. Now the output is capped at 2,048 tokens and a chunk that returns invalid JSON is skipped:

ChatOptions mapOptions = Options(16384);
mapOptions.ResponseFormat = ContractPrompts.Map.Format;
// A small model can keep listing clauses forever; cap the output and treat a cut-off answer as a failed chunk
mapOptions.MaxOutputTokens = 2048;

dotnet run --project src/Bookstore.Prompts.Lab -- contract map
map chunk 1: 8 quotes, 8 found in the contract: 1.1. Sözl..., 1.11. Söz..., 2.1. Bu m..., 2.9. Tanı..., 2.18. Tar..., 3.14. BOO..., 3.17. Tar..., 4.1. Ürün...
map chunk 5: 6 quotes, 6 found in the contract: 17.2. Has..., 17.3. Sür..., 17.4. BOO..., 17.5. İad..., 17.6. Kus..., 17.7. TED...
map chunk 9: 2 quotes, 2 found in the contract: Ek-3.1. M..., Ek-3.2. M...
FAIL cq-01 answer "Ocak 2027'de teslim alacağımız kitaplar için hasarlı bir ..." lacks 10 iş gün|on (10) iş gün|on iş gün; no quote from Ek-3 (...)
PASS cq-02 Okul siparişleri için aldığınız kitaplarda satılamayan st...
PASS cq-03 Kusurlu kitapların iadesinde kargo bedeli TEDARİKÇİ'ye ai...
PASS cq-04 TEDARİKÇİ, iade edilen kitapları teslim aldığı tarihten i...
PASS cq-05 BOOKSTORE, satılamayan kitapları yayın tarihinden itibare...
FAIL cq-06 expected NOT_FOUND, got "BOOKSTORE, satılamayan kitapları yayın tarihinden itibare..."
4/6 passed, 65 quotes collected, 68182 input tokens in total

I kept three of the nine chunks in the output. The map step is not perfect: the model returned exactly eight quotes from almost every chunk, as if it read "at most 8" as "find 8". Most of the 65 collected quotes have nothing to do with returns. But two important things are right: the six provisions of Clause 17 and the two provisions of Ek-3 were collected completely and word for word, because copying exactly from a four-clause chunk is much easier than copying from a 33-thousand-token text.

The reduce result is 4/6, and this time every passing answer comes from the right clause, with a quote that appears word for word in the contract: in cq-02, 17.4 and Ek-3.2 are quoted together to reach 30%. In cq-01 the model saw Ek-3.1 but still used the old 15-day period from 17.2; it could not make the amendment connection. In cq-06 an unrelated clause came back again instead of NOT_FOUND. The total cost is 68,182 input tokens: nine map calls and six short reduce calls. Six questions against the full document are over 200 thousand tokens without caching; with caching, in practice 33,600 tokens are processed once and then only the questions. Map-reduce is expensive for a one-question job; if many questions will be asked against the same document, the map result can be produced once and stored. What the map step does is, in a sense, find the relevant pieces; the approach that does the same job with embeddings is the topic of the RAG series.

Production Notes

Check the input token count on every request. Ollama's silent truncation is the cleanest example of a failure that can happen in any system that sends long documents. Estimate the number of tokens you send with the tokenizer or a previous measurement, compare it with Usage.InputTokenCount in the response, and do not use the answer if they do not match.

Verify quotes in code and send unverifiable answers to a person. For the purchasing team, "the source of this answer could not be found, see Clause 17 and Ek-3" is far cheaper than a wrong return deadline. If verified quotes are shown on screen with their clause numbers, the user can check the answer without opening the contract.

Handle amendment protocols separately. An annex like Ek-3 changes earlier clauses; neither the full document nor map-reduce made that connection reliably with this small model. If document annexes are kept in structured form, recording which annex changes which clause from which date, giving that relation to code is more reliable than extracting it from the model.

Protect the prompt order like a contract. A cache hit depends on the prompt being identical from the start. A developer adding one line at the top of the system prompt can make every request fall out of the cache and multiply the cost overnight. That is why tracking the number of tokens read from the cache in the providers' responses and alerting on a drop is useful.

Choose the approach by model and document size. The results in this post come from a local 4-billion-parameter model. A large cloud model may quote exactly from 33 thousand tokens much better; but the answer to "does it?" is still given by the same test set and the same quote check.

Common Pitfalls

"The context window is big, I'll put everything in." The length the model supports is not the same as the setting of the server you run; in this setup the 128-thousand-token model saw the contract cut down to 16 thousand tokens. Even when it fits, cost and time grow with document length.

Mistaking a keyword match for a correct answer. In v1's question-first run cq-02 passed; the answer described the late-delivery penalty in Clause 22 and added "thirty percent for school orders" at the end. The word was in the answer; the reasoning was wrong.

Asking for quotes and not checking them. The model can produce rewritten sentences and invented clause numbers instead of quotes; the only way to tell them apart is to look the quote up in the source.

Putting changing information at the start of the prompt. Putting the question, the date or user details first means the long document behind it is processed again on every request.

Searching Turkish text for exact words. "10 iş günü" and "10 iş güne", "yüzde otuz" and "yüzde 30" carry the same information. Write the check with stems and alternatives, and compare case with tr-TR rules.

A Small Experiment

★☆☆ Run contract ask v2 doc-first 16384 and look at the in= value in the answers. Add a small check that catches a truncated request and throws: estimate the token count of the text you send from an earlier measurement, and reject the answer if InputTokenCount is below 90 percent of it.

★★☆ Add a simple pre-filter to the map step that uses the contract's clause headings: send only sections whose heading or text contains "iade" and sections that start with "Ek-" to the map step. How do the number of collected quotes, the total input tokens and the reduce score change?

★★★ Give the clauses changed by Ek-3 to code: add a list to ContractDocument that holds "Ek-3.1 changes 17.2 from 2027-01-01", and order the quotes sent to reduce so that the amending annex comes right after the changed clause. Do the results for cq-01 and cq-02 change?

In the next post we move to prompts that produce code: generating read-only SQL from natural language for the reporting screen, plus test and code review prompts. The same principle will apply there: instead of trusting the model's output, verify it with the compiler, the parser and the database.