Summarization, Rewriting and Translation: Length Control, Tone, Glossaries and TR/EN Localization
In text-generating tasks, length, required facts, new numbers, placeholders, HTML tags and the glossary are checked in code; the model cannot count characters, and price and date formats are CultureInfo work. On Bookstore campaigns v1 passed zero of five and reached 3/5 with explicit rules and a repair loop measured by code; in book description translation detailed rules stayed at 0/6 while a short instruction with one example gave 4/6.
Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 11.0.0, OllamaSharp 5.5.2 and xUnit v3 4.0.3. Model outputs come from gemma3:4b, a small open model (4.3 billion parameters, Q4_K_M) running locally on Ollama 0.34.4; nothing was run against a cloud provider.
Guess First
At Bookstore, the online bookstore used as the shared example across this site's series, the marketing team writes campaigns as long texts, while the mobile app shows push notifications of at most 120 characters. You wrote "shorten this to under 120 characters" in the prompt and gave it five campaign texts. How many will stay under the limit, and in how many will the discount, the campaign code and the last day all survive?
The second question is about translation: book descriptions are HTML, with placeholders such as {price} and product terms such as "Kurumsal Sipariş". With a "Translate to English" instruction, will the model keep the HTML? Will it translate the term as the glossary says, or with a word of its own?
In classification there was one right answer and checking it was easy. Tasks that generate text have no single right answer, but that does not mean correctness cannot be measured. In this post, for every task we first ask "what can code check?"
What Can You Check in a Text-Generating Task?
In summarization, shortening and translation, most of the checkable rules are deterministic: length, fixed facts from the source (code, rate, date) surviving in the output, no number appearing in the output that is not in the source, placeholders and HTML tags staying the same, and following the glossary. They are written with regexes and string comparisons and can run on every run and every commit. Tone, fluency and faithfulness of meaning cannot be fully checked with rules; they need LLM-as-judge and human review, which we cover in the last post of the series. Most of the checks in this post rest on this small helper:
// "1.240,50" and "1,240.50" are the same number: compare digits only
public static HashSet<string> Numbers(string text) =>
[.. NumberPattern().Matches(text).Select(m => new string([.. m.Value.Where(char.IsDigit)]))];
// Numbers in the output that the source never mentioned
public static List<string> NewNumbers(string source, string output) =>
[.. Numbers(output).Except(Numbers(source)).Order()];Comparing numbers by their digits alone means a format difference (1.240,50 versus 1,240.50) is not counted as new information. A number the model adds to a summary, on the other hand, shows up immediately; in summarizing and shortening, the most dangerous error is adding information to the source.
Shortening a Campaign Text
v1 is one sentence: "shorten the campaign text to under 120 characters for a push notification, keep it in Turkish". The check function looks at the length, the required facts marketing marked, new numbers, exclamation marks and emoji:
dotnet run --project src/Bookstore.Prompts.Lab -- campaign v1
campaign-push v1 | gemma3:4b | 5 cases
FAIL cp-01 attempts=1 length=141 141 characters, limit is 120; exclamation mark; emoji; missing "30 Kasım"; new numbers: 120
FAIL cp-02 attempts=1 length=978 978 characters, limit is 120; exclamation mark; new numbers: 1, 120, 2, 3
FAIL cp-03 attempts=1 length=1471 1471 characters, limit is 120; exclamation mark; emoji; missing "31 Aralık"; new numbers: 1, 120, 3, 3112
FAIL cp-04 attempts=1 length=108 exclamation mark; emoji
FAIL cp-05 attempts=1 length=138 138 characters, limit is 120; exclamation mark; emoji; missing "23 Kasım"; new numbers: 120
0/5 passedZero out of five. In cp-02 and cp-03 the model wrote "Option 1, Option 2, Option 3" alternatives, English translations and a recommendation paragraph instead of a notification; the output is 978 and 1471 characters. In cp-01 and cp-05 the answer starts with "İşte 120 karakteri aşmayan bir versiyon:", here is a version under 120 characters; the "new numbers: 120" warning comes from that sentence. In cp-03 the last day "31 Aralık" became "31.12". Shortening means dropping information; if the prompt does not say which information must not be dropped, the model drops the last day first.
v2 defines what a notification is and gives the limit as a number. Giving a target of 80–110 characters and stating the absolute limit separately leaves the model some room; the model does not count length character by character, it estimates it from tokens:
System: $"""
You write Turkish push notifications for Bookstore campaigns.
## Rules
- One sentence, 80 to 110 characters including spaces. Never more than {MaxLength}.
- Keep the discount, the campaign code and the last day exactly as written.
- No emoji, no exclamation marks, no information that is not in the campaign text.
- Output only the notification text.
""",dotnet run --project src/Bookstore.Prompts.Lab -- campaign v2
FAIL cp-01 attempts=1 length=96 exclamation mark
FAIL cp-02 attempts=1 length=118 exclamation mark
FAIL cp-03 attempts=1 length=160 160 characters, limit is 120
FAIL cp-04 attempts=1 length=94 exclamation mark
PASS cp-05 attempts=1 length=107
1/5 passedAll required facts are there, no new numbers, four notifications under the limit. But the model broke the "no exclamation marks" rule four times and left cp-03 unshortened, as the first two sentences of the campaign text (160 characters).
Code Measures Length: A Repair Loop
The model cannot count characters; code can. We turn that division of labour into a loop: the model writes, code checks, failed checks are added to the same conversation as concrete feedback, and the model writes again. At most three attempts:
// Code measures, the model rewrites: failed checks go back as feedback
public sealed class ConstrainedRewriter(IChatClient client, ChatOptions options)
{
public async Task<RewriteResult> RewriteAsync(
PromptVersion prompt,
string input,
Func<string, List<string>> check,
int maxAttempts = 3,
CancellationToken cancellationToken = default)
{
List<ChatMessage> messages = prompt.Render(input);
string text = "";
List<string> issues = [];
for (int attempt = 1; attempt <= maxAttempts; attempt++)
{
ChatResponse response = await client.GetResponseAsync(messages, options, cancellationToken);
text = response.Text.Trim();
issues = check(text);
if (issues.Count == 0)
{
return new RewriteResult(text, attempt, issues);
}
messages.Add(new ChatMessage(ChatRole.Assistant, text));
messages.Add(new ChatMessage(ChatRole.User,
$"Your text has {text.Length} characters. Fix these problems and output only the new text: {string.Join("; ", issues)}"));
}
return new RewriteResult(text, maxAttempts, issues);
}
}dotnet run --project src/Bookstore.Prompts.Lab -- campaign repair
campaign-push repair | gemma3:4b | 5 cases
FAIL cp-01 attempts=3 length=96 exclamation mark
Tüm klasik romanlarda %25 indirim! KITAP25 koduyla alışveriş yapın, 30 Kasım'a kadar kaçırmayın.
PASS cp-02 attempts=3 length=118
Öğretmenler Günü'ne özel: okul ve kütüphaneler için Kurumsal Sipariş ile %5 indirim, kod: OGRETMEN5, 24 Kasım'a kadar.
FAIL cp-03 attempts=3 length=134 134 characters, limit is 120
Hediye Paketi seçen herkese paket ücretsiz, üstelik 2 kitap alana üçüncü kitapta %50 indirim. HEDIYE50 kodu 31 Aralık'a kadar geçerli.
PASS cp-04 attempts=2 length=94
Dune serisinin yeni ciltli baskısı için DUNE15 koduyla %15 indirim kazanın, 12 Aralık'a kadar.
PASS cp-05 attempts=1 length=107
Bu hafta sonu tüm Jane Austen kitaplarında %20 indirim. AUSTEN20 kodu 23 Kasım Pazar 23.59'a kadar geçerli.
3/5 passedThe loop took 1/5 to 3/5, but it teaches two lessons. First, in cp-01 the model kept the exclamation mark in all three attempts. A format rule like "no exclamation marks" can be applied by code in one line (turning the exclamation mark into a full stop); spending three calls to ask the model for it is wasteful. Second, cp-03 went from 160 to 134 and stayed there: the model dropped the first sentence but did not shorten the rest. The loop needs a ceiling, and text that reaches the ceiling should go to a person; here someone from marketing will write cp-03 by hand. Every attempt in the loop is a model call; track the cost together with the number of attempts.
Who Is the Summary For?
"Summarize this report" is an incomplete instruction; the reader decides what the summary contains. For management, the same incident report should show impact, cause and action, with numbers. A notification to a customer should say what happened to their order and what Bookstore offers; how many orders were affected, what the coupons cost and which job failed are of no interest to the customer. So we write two separate prompts:
public static readonly PromptVersion Customer = new(
Task: "incident-summary",
Version: "customer",
System: """
Write a short Turkish notification to an affected Bookstore customer based on the
incident report. At most 2 sentences and 300 characters. Say what happened to their
order and what Bookstore offers. Do not mention internal details: order counts,
costs, system or team names, root cause. Output only the notification.
""",
BuildUser: report => $"<report>\n{report}\n</report>");The check functions also differ by reader. The management summary must be exactly three bullets and must contain no number that is not in the source. The customer notification must not exceed 300 characters and must not contain the list of internal details defined per incident (order counts, costs, module and job names). The result for two incident reports:
dotnet run --project src/Bookstore.Prompts.Lab -- summary
FAIL in-01 management expected 3 bullet lines, got 4 lines
Here’s a summary of the bookstore incident report for the management team, in Turkish, using only the provided numbers and bullet points:
- : 2 gün boyunca 318 sipariş gecikmeli ve 41 müşteri siparişini iptal etti.
- : Hata, 14 Kasım'da düzeltildi.
- : Etkilenen müşterilere toplam 15.900 TL değerinde 50 TL indirim kuponu tanımlandı.
PASS in-01 customer
Siparişiniz gecikmeli teslim edildi, üzgünüz. Size 50 TL değerinde bir indirim kuponu tanımladık, lütfen kullanmayı unutmayın!
FAIL in-02 management expected 3 bullet lines, got 4 lines
Here’s a summary of the bookstore incident report for the management team, in Turkish, using only the provided numbers and bullet points:
- : 76 ödeme iki kez çekildi (18.00-19.30 arasında) ve toplam iade tutarı 31.480 TL oldu.
- : Kesinti, 3 Aralık 2026’da ödeme sağlayıcısında meydana geldi.
- : Payments ekibi, çift çekimleri 4 Aralık’ta iade etti ve iadelerin hesaplara yansıması 3 iş günü sürebilir.
PASS in-02 customer
Siparişinizdeki sorun için üzgünüz. Ücretsiz kargo kodu ile yeni bir sipariş verebilirsiniz.In the management summaries the model started writing the "impact, cause, action" headings from the prompt and left them empty (- :), and it added an English introduction sentence at the top. The numbers are right and there is no number missing from the source; the format check caught the failure. The customer notification for in-02 is more instructive: it passed the checks because it is short, has no internal detail and no new numbers. But it does not tell the customer what really matters: that the money charged twice to their card was refunded on 4 December and may take three business days to show up. Deterministic checks answer "did it say something wrong?" well; they cannot answer "did it say what it had to?". For the second, either mark the required facts one by one, as in the campaign, or use LLM-as-judge.
Translation: Glossary and Placeholders
The book descriptions in Catalog are written in Turkish and translated for the English version of the site. The six-description set has HTML tags, placeholders such as {price}, {stock} and {customerName}, three product terms (Kurumsal Sipariş, Hediye Paketi, Okul Listesi) and names that must not be translated (Bookstore, book titles). The check function looks at all of these, at whether the output starts with HTML, and at whether any Turkish letters are left in the English text. v1 is "Translate the book description from Turkish to English":
dotnet run --project src/Bookstore.Prompts.Lab -- translate v1
FAIL tr-01 text before the HTML: "Here's the English translation of the..."; HTML tags changed; glossary: "Kurumsal Sipariş" should be "Institutional Order"
FAIL tr-02 text before the HTML: "Here’s the English translation of the..."; HTML tags changed; glossary: "Hediye Paketi" should be "Gift Wrap"
FAIL tr-03 text before the HTML: "Here's the English translation of the..."; HTML tags changed; glossary: "Okul Listesi" should be "School List"
FAIL tr-04 text before the HTML: "Here's the English translation of the..."; placeholders {stock} -> {stock} {stock}; HTML tags changed; "Bookstore" was changed
FAIL tr-05 text before the HTML: "Here's the English translation of the..."; HTML tags changed; glossary: "Kurumsal Sipariş" should be "Institutional Order"; glossary: "Hediye Paketi" should be "Gift Wrap"
FAIL tr-06 text before the HTML: "Here's the translation of the Turkish..."; HTML tags changed
0/6 passed (0%)Zero out of six. The model turned the HTML into Markdown (**Dune** instead of <b>Dune</b>), added an introduction sentence to every answer and "Would you like me to translate anything else?" to the end of some. It chose its own words for the terms: Kurumsal Sipariş became "Corporate Order", Hediye Paketi "Gift Package", and Okul Listesi "Wish List". The last one is not just a terminology error but a meaning error: a school list and a wish list are different features. "Bookstore'a özel baskı" also became "exclusive to the bookstore"; the store's name turned into an ordinary noun.
In v2 we wrote detailed rules: return only HTML, keep tags and placeholders, add no numbers, use the glossary, and a "Never translate: Bookstore, Dune, Emma..." list. The result was 0/6 again, with different failures: the model wrapped the answer in a code block (```html), added id, translate and data-glossary attributes to tags, left two descriptions untranslated in Turkish, and in tr-06 replaced the prices with placeholders it invented, {price} and {original_price}. The do-not-translate list seems to have steered the small model towards a task of marking up the HTML. More rules do not always give better results; you only see that by measuring.
v3 is shorter: one paragraph of instructions, the glossary inline and a single example that shows exactly the shape of the output we want:
// v3: fewer rules, glossary inline, one worked example of the exact output shape
public static PromptVersion V3(Glossary glossary) => new(
Task: "description-translation",
Version: "v3",
System: $$"""
Translate Bookstore catalog descriptions from Turkish to English.
Return the same HTML with only the Turkish text translated: same tags, no new
attributes, no code fences, no notes. Placeholders such as {price} stay as they are.
Book titles and the store name Bookstore stay in English as written.
Glossary: {{string.Join("; ", glossary.Terms.Select(t => $"{t.Key} = {t.Value}"))}}.
Example input:
<p><b>Walden</b>, Thoreau'nun 1854 tarihli kitabı. Fiyat: {price}</p>
Example output:
<p><b>Walden</b>, Thoreau's book from 1854. Price: {price}</p>
""",
BuildUser: description => description);dotnet run --project src/Bookstore.Prompts.Lab -- translate v3
description-translation v3 | gemma3:4b | 6 cases
PASS tr-01
PASS tr-02
PASS tr-03
PASS tr-04
FAIL tr-05 HTML tags changed
FAIL tr-06 Turkish number format: 1.240,50, 989,90
4/6 passed (67%)Four descriptions are clean: the tags are the same, the placeholders are in place, the terms follow the glossary, and "Bookstore exclusive bound edition" is right. In tr-05 the model wrote a broken <i/>Modern Prometheus/) instead of <i>Modern Prometheus</i>; the tag check caught it.
Date and Money Formats Belong to Code
The tr-06 failure is a classic localization trap: inside an English sentence, "on sale: 1.240,50 TL instead of 989,90 TL". The numbers are right, the format is Turkish; an English reader may read 1.240,50 not as one thousand two hundred forty but as one point two four zero, fifty. The check looks for decimal-comma numbers in English text. The real fix, though, is not in the prompt but in the content: instead of embedding the price and the date in the text, leave them as placeholders and format them in code for each language:
// Prices and dates are formatted by code per language and passed in as placeholders
public static class LocalizedPrice
{
static readonly CultureInfo Tr = CultureInfo.GetCultureInfo("tr-TR");
static readonly CultureInfo En = CultureInfo.GetCultureInfo("en-US");
public static string Format(decimal amount, string language) => language switch
{
"tr" => string.Create(Tr, $"{amount:N2} TL"),
_ => string.Create(En, $"TRY {amount:N2}"),
};
public static string FormatDate(DateOnly date, string language) => language switch
{
"tr" => date.ToString("d MMMM yyyy", Tr),
_ => date.ToString("MMMM d, yyyy", En),
};
}LocalizedPrice.Format(1240.50m, "en") returns "TRY 1,240.50"; the test project has tests for both languages. If the campaign description is written with {originalPrice} and {campaignPrice}, the model never touches the number; translation becomes purely language work. The same rule applies to dates: in tr-06 the model translated "15 Aralık 2026" correctly as "December 15, 2026", but making the date a placeholder too is more robust than relying on that.
A Lesson from This Site's TR/EN Flow
The post you are reading also goes through a TR to EN flow, and the rules there follow the same logic as the checks in this post. Software terms stay in English in both languages, which is a glossary of its own; the explanations given in parentheses on first mention in the Turkish post do not exist in the English one; and code blocks must be identical in both languages apart from comment lines. That last rule is checked by a script: it compares the code blocks of the two files with comments removed and stops publishing on a single character of difference. So the rule does not remain an instruction; like the placeholder check, it becomes a deterministic test that runs for every post. Whoever does the translation, a person or a model, the parts that must be preserved are verified in code.
Production Notes
Version the checks together with the prompt. The MustKeep list in the campaign is data that marketing provides with the campaign text; when the prompt changes the checks do not, when the campaign changes they do.
Do not make the model fix what code can fix. Turning exclamation marks into full stops, dropping a leading "İşte..." sentence or a code-fence marker is a deterministic post-processing step. But log the original output when you do it; how often something gets cleaned up shows where the prompt is weak.
Give the repair loop a ceiling. Text that does not get fixed in three attempts may not get fixed in a fourth either; cp-03 stayed at 134 characters. Output that hits the ceiling is not published automatically; it goes to a person.
Mark required facts explicitly for text that goes to customers. If the team that writes the incident report also marks required facts for the customer notification, such as the "refund date" and "time to show up in the account", the check can answer part of "did it say what it had to?" in code too.
Common Pitfalls
Expecting the model to count characters. The model sees tokens; v1's outputs, which were supposed to be "under 120 characters", were 138, 141 and 1471 characters long.
Missing information added in a summary. Every number and name not in the source should be a warning; that is why NewNumbers is in every check.
Assuming more rules give better results. In translation the detailed v2 produced different but equally bad failures than the short v1; v3, with short instructions and one example, was the best.
Having prices and dates translated. Format localization is not language work but CultureInfo work. Use placeholders and let code do the formatting.
Mistaking passing the checks for being correct. The customer notification for in-02 passed every check and left out the key information.
A Small Experiment
★☆☆ Write a post-processing step that runs before CampaignRewrite.Check: it should turn exclamation marks into full stops and drop a leading "İşte ... :" line. Rerun v2 with this step; how many campaigns pass without entering the loop at all?
★★☆ Add the required facts of the in-02 customer notification (4 Aralık, 3 iş günü) to incidents.jsonl and extend CheckCustomer to check that list too. Then change the prompt to ask for these facts and measure the result.
★★★ Rewrite tr-06 with {originalPrice} and {campaignPrice} placeholders, add a small render step that fills the placeholders with LocalizedPrice after translation, and write a test that shows the tr and en outputs side by side.
In the next post the texts get longer: we will extract the return terms from a forty-page supplier contract. Where to put a long document in the prompt, grounding the answer in quotes, splitting the document, and putting the stable part first to benefit from prompt caching will be measured there.