Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
Improving Prompts by Measuring: Test Sets, Rule-Based Checks, LLM-as-Judge, Microsoft.Extensions.AI.Evaluation and Regression in CI with promptfoo
Measuring a prompt needs a golden set with reference answers and human labels, rule checks that run on every pull request, and an LLM-as-judge whose agreement with people has been measured. On Bookstore's return policy questions a short-answer rule cut a small model's accuracy from 8/10 to 5/10; rules and two judges agreed with the person on at most 23 of 30 answers, and a pairwise judge depended on order or gave no decision in six of ten pairs.
Managing Prompts in Code: Templates, Prompts in Files, Versioning, Pinning Model Snapshots and Reviewing Prompt Changes
Version a prompt in a file together with its model, digest, temperature and seed, embed it in the assembly, freeze released versions with a lock test and review changes through a diff and an eval result. Moving Bookstore's prompts into files, we proved without model calls that rendering stayed identical; escaping stopped none of four injection messages, and the new v4 kept the main set at 12/12 while the injection set rose from 0/4 to 2/4.
System Prompts for Agents and Tool-Using Systems: Role, Boundaries, Tool Usage Rules, Approval and Stop Conditions
An agent system prompt states the goal, authority boundary, tool rules, confirmation, when to ask and when to stop; the authority itself lives in the code behind the tools. In Bookstore's order assistant a one-line prompt cancelled two orders without confirmation, a sectioned prompt passed all eleven scenarios, and with cancellation approval moved into code via ApprovalRequiredAIFunction no order was cancelled without approval, even with the weakest prompt.
Prompts for Code: Code Generation, Test Generation, Code Review and SQL Generation — Asking for Verifiable Output
From a prompt that produces code, ask for checkable output rather than an opinion: verify SQL with SQL Server's parser and a schema check, run test cases against real code, and match review findings to lines in the diff. On Bookstore's reporting screen a schema-less prompt got 0 of 12 queries right and wrote a DELETE; with the schema, rules and a decision-first output it reached 5/12, the validator let no write through, and generated tests, not the review, caught an off-by-one.
Working with Long Documents and Context: Placement, Asking for Quotes, Chunking and Ordering for Prompt Caching
With long documents, count the tokens first, put the document first and the question last, ground the answer in word-for-word quotes verified in code, and keep the stable part in front to benefit from prompt caching. On a 33k-token supplier contract Ollama silently cut the prompt in half, the small model could not quote correctly from the full text, map-reduce reached 4/6 with verified quotes, and with the document first the second question's prompt processing fell from 35.9 to 0.2 seconds.
Summarization, Rewriting and Translation: Length Control, Tone, Glossaries and TR/EN Localization
In text-generating tasks, length, required facts, new numbers, placeholders, HTML tags and the glossary are checked in code; the model cannot count characters, and price and date formats are CultureInfo work. On Bookstore campaigns v1 passed zero of five and reached 3/5 with explicit rules and a repair loop measured by code; in book description translation detailed rules stayed at 0/6 while a short instruction with one example gave 4/6.
Classification and Routing Prompts: Label Sets, Label Definitions, Choosing Few-Shot Examples, Ambiguous Cases and Confidence
A classification prompt's accuracy depends most on mutually exclusive label definitions, an enum in the schema and a "send to a person" label; the model's own confidence does not separate wrong answers. On Gül's 24-ticket set, definitions raised the score from 14/24 to 22/24, few-shot added nothing, qwen3:4b changed three tickets when the label order was reversed, and agreement between two small models sent both wrong answers to a person.
Extraction Prompts: Structured Data from Free Text, Missing Fields, Ambiguity and "I Don't Know"
In an extraction prompt every field that might be unknown should be nullable, each value's source should be proven with a quote, and calculations such as dates belong in code; a JSON schema only guarantees the shape. On Nevin Hanım's order e-mails, v1 with required fields got one in eight right and invented ISBNs like "Ekte"; with a nullable schema, source quotes, one example, a Turkish date resolver and a validator we reach 12/12 items and 8/8 dates.
Prompt Engineering in Practice: Treating Prompts as Specifications, Code and Tested Components
The only honest way to say a prompt works is to measure it on a test set that resembles real input. Bookstore's return-detection prompt passed 1/1 on a single message but only 8/12 on a twelve-message set; we run the loop step by step with a task definition, a null rule, a JSON schema, verification in code and boundary examples, and build the EvalRunner used throughout the series.
Bridging to Python: Solving the Same Problem with scikit-learn and Running It in .NET via ONNX
A model trained with scikit-learn in Python can be converted to ONNX together with its preprocessing and run in .NET with ONNX Runtime so that it gives the same result. We build Bookstore's cancellation model with a scikit-learn Pipeline and cross-validation, map the concepts to ML.NET, export it with skl2onnx and score it in C#, showing with real output that the difference stays around 10⁻⁷, and the float type, opset and unconvertible step pitfalls.
Taking a Model to Production: PredictionEnginePool in ASP.NET Core, Model Versioning and Drift
The way to serve an ML.NET model safely in ASP.NET Core is PredictionEnginePool, a card stored with the model and drift tracked against real outcomes. We put Bookstore's delivery-time model into a minimal API and show with real output how a shared PredictionEngine returns 12% wrong answers under load, how to update the model without a restart, latency metrics, and a model that degrades while its inputs stay the same.
Model Selection and AutoML: Hyperparameters, Cross-Validation, the AutoML API and Feature Importance
Hyperparameter search and AutoML speed up model selection when the metric is right and the data has no leakage, but they cannot strengthen a weak signal. On Bookstore's cancellation risk we compare grid and random search for FastTree with cross-validation, the ML.NET AutoML API and the mlnet CLI that searches by accuracy, and show with real output how a leaky column produces an AUC of 1.0, Permutation Feature Importance and a fairness check by age and district.