Structured Output with LLMs: JSON Mode, JSON Schema and GetResponseAsync<T>

Version note: The examples in this post were tested with Ollama 0.34 on local models (gemma3:4b), using .NET 10 and Node 22.

Guess First

You ask the model for a book record as JSON and give it a JSON schema. The answer arrives, it parses, and it matches the type. Can you write this record straight to the database? In other words: is an answer that matches the schema a correct answer?

In a moment we will see an answer that matches the schema perfectly but would add a free, out-of-stock book to the store. First, let's look at how we get there.

Parsing Free Text

Staff write short notes for new books arriving at the store, and we want to create records from these notes. The most natural first attempt is to ask the model to extract the details and read the answer with a regex. We wrote the regex by looking at the answer we got at temperature: 0; in that answer the stock line looked like **Number of Copies in Stock:** 2:

import ollama from 'ollama';

const note = `New in store: 'Project Hail Mary' by Andy Weir. Paperback, 496 pages.
Now 18.99 USD (was 22.00). Only 2 copies left on the shelf.`;
const prompt = `Extract the title, author, price and number of copies in stock from this note:\n${note}`;

// Written against the answer we got at temperature 0
function parseCopies(text: string): number | null {
  const match = text.match(/\*\*Number of Copies in Stock:\*\*\s*(\d+)/);
  return match ? Number(match[1]) : null;
}

for (let i = 1; i <= 5; i++) {
  // Model defaults (temperature 1), as in many real applications
  const res = await ollama.chat({ model: 'gemma3:4b', messages: [{ role: 'user', content: prompt }] });
  console.log(`run ${i}: copies = ${parseCopies(res.message.content)}`);
}

run 1: copies = null
run 2: copies = null
run 3: copies = null
run 4: copies = null
run 5: copies = 2

In four of the five attempts the regex found nothing. The model didn't say anything wrong; it just wrote the label sometimes as "Copies in Stock" and sometimes as "Number of Copies in Stock", and sometimes used a curly apostrophe instead of a straight one. Same information, different format. As we saw in the sampling post, every token is picked from a probability distribution, and that applies to the words of the label too. Code that parses free text relies on a format the model never promised.

temperature: 0 only hides this problem. With the same model and the same prompt the format stays fixed, but when the model is updated, when you add a sentence to the prompt or when the input changes a little, the format drifts again, and you only notice in production, from fields that silently come back empty.

JSON Mode: Valid but Unpredictable

The first improvement is asking the model for JSON. In Ollama the simplest way is format: "json", that is, JSON mode. Let's send the same prompt in both forms:

import ollama from 'ollama';

const note = `New in store: 'Project Hail Mary' by Andy Weir. Paperback, 496 pages.
Now 18.99 USD (was 22.00). Only 2 copies left on the shelf.`;

const prompt = `Extract the title, author, price and number of copies in stock from this note:\n${note}`;

for (const format of [undefined, 'json']) {
  const res = await ollama.chat({
    model: 'gemma3:4b',
    messages: [{ role: 'user', content: prompt }],
    format,
    options: { temperature: 0, seed: 42 },
  });
  console.log(`--- format: ${format}`);
  console.log(res.message.content);
}

--- format: undefined
Here's the extracted information:

*   **Title:** Project Hail Mary
*   **Author:** Andy Weir
*   **Price:** 18.99 USD
*   **Number of Copies in Stock:** 2
--- format: json
{
  "title": "Project Hail Mary",
  "author": "Andy Weir",
  "price": "18.99 USD",
  "copies_in_stock": 2
}

JSON mode guarantees valid JSON, and indeed JSON.parse reads this answer without trouble. But it makes no promise about the shape of the content. The model chose the field name: copies_in_stock; if your TypeScript type expects copiesInStock, the field comes back undefined. It also wrote the price as the string "18.99 USD" instead of a number; on the next run it might write 18.99 or "$18.99". JSON mode pushes the format problem one level down; it doesn't solve it.

Pinning the Format with a JSON Schema

The real fix is to give the model the structure you expect as a JSON schema. Since Ollama 0.5.0 (December 2024), the format field also accepts a schema object instead of "json" and constrains generation to that schema. In TypeScript, defining the schema with Zod and generating it with z.toJSONSchema, instead of writing it by hand, lets you use the same definition for both the request and the validation:

import ollama from 'ollama';
import { z } from 'zod';

const Book = z.object({
  title: z.string(),
  author: z.string(),
  format: z.enum(['paperback', 'hardcover', 'ebook']),
  price: z.number(),
  copiesInStock: z.number().int(),
});
type Book = z.infer<typeof Book>;

const note = `New in store: 'Project Hail Mary' by Andy Weir. Paperback, 496 pages.
Now 18.99 USD (was 22.00). Only 2 copies left on the shelf.`;

const schema = z.toJSONSchema(Book);
console.log(JSON.stringify(schema));

const res = await ollama.chat({
  model: 'gemma3:4b',
  messages: [{
    role: 'user',
    // The docs recommend describing the schema in the prompt as well
    content: `Extract the book from this note as JSON matching this schema: ${JSON.stringify(schema)}\n\n${note}`,
  }],
  format: schema,
  options: { temperature: 0, seed: 42 },
});

console.log(res.message.content);
const book: Book = Book.parse(JSON.parse(res.message.content));
console.log(book.price * book.copiesInStock);

{"$schema":"https://json-schema.org/draft/2020-12/schema","type":"object","properties":{"title":{"type":"string"},"author":{"type":"string"},"format":{"type":"string","enum":["paperback","hardcover","ebook"]},"price":{"type":"number"},"copiesInStock":{"type":"integer","minimum":-9007199254740991,"maximum":9007199254740991}},"required":["title","author","format","price","copiesInStock"],"additionalProperties":false}
{
  "title": "Project Hail Mary",
  "author": "Andy Weir",
  "format": "paperback",
  "price": 18.99,
  "copiesInStock": 2
}
37.98

This time the field names are exactly the ones in the schema, the price is a number, and format is one of the three values we allowed. Book.parse validates the answer once more with the same Zod definition, and the result is a typed Book in TypeScript; computing the stock value needed no string handling at all. Ollama's docs recommend two more things, and we did both: describing the schema in the prompt as well and setting temperature to 0. The schema constrains generation, but the model needs to see the fields in the text too in order to understand what they mean.

GetResponseAsync<T> in C#

Microsoft.Extensions.AI shortens this by one more step. You don't write the schema; you give it a C# type:

using System.ComponentModel;
using System.Globalization;
using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");
ChatOptions options = new() { Temperature = 0, Seed = 42 };

string note = """
    New in store: 'Project Hail Mary' by Andy Weir. Paperback, 496 pages.
    Now 18.99 USD (was 22.00). Only 2 copies left on the shelf.
    """;

ChatResponse<Book> response = await client.GetResponseAsync<Book>(
    $"Extract the book from this note.\n\n{note}", options);

Console.WriteLine(response.Text);

Book book = response.Result;
decimal stockValue = book.Price * book.CopiesInStock;
Console.WriteLine($"{book.Title} | {book.Format} | stock value: {stockValue.ToString(CultureInfo.InvariantCulture)} USD");

enum BookFormat { Paperback, Hardcover, Ebook }

record Book(
    string Title,
    string Author,
    BookFormat Format,
    [property: Description("Current price in USD")] decimal Price,
    int CopiesInStock);

{ "title": "Project Hail Mary", "author": "Andy Weir","format": "Paperback", "price": 18.99, "copiesInStock": 2}
Project Hail Mary | Paperback | stock value: 37.98 USD

GetResponseAsync<Book> generates a JSON schema from the Book record, adds it to the request, deserializes the answer and gives it to you as response.Result. When I captured the outgoing request with a DelegatingHandler, the format field was this:

{"$schema":"https://json-schema.org/draft/2020-12/schema","type":"object","properties":{"title":{"type":"string"},"author":{"type":"string"},"format":{"type":"string","enum":["Paperback","Hardcover","Ebook"]},"price":{"description":"Current price in USD","type":"number"},"copiesInStock":{"type":"integer"}},"required":["title","author","format","price","copiesInStock"]}

A few details stand out. The property names were converted to camelCase in the JSON. The BookFormat enum went as an enum list of its names, not as numbers; I didn't write an extra converter. The [Description] attribute became the field's description in the schema; that is the cleanest way to tell the model what a field means. And one thing is missing: the request's messages did not mention the schema at all, so in this call the schema was not added to the prompt; it only went in the format field. The field names are clear enough that this caused no problem here; for fields with an ambiguous meaning, add a [Description] or a short description in the prompt.

A Schema Guarantees the Format, Not the Truth

Now back to the opening question. A new note: "Circe" has arrived, the price is not yet known, and the stock arrives next week. We use the same Book type, plus a draft type whose unknown fields can be null:

using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");
ChatOptions options = new() { Temperature = 0, Seed = 42 };
string circe = "Just in: 'Circe' by Madeline Miller, hardcover. Price to be announced. Stock arrives next week.";

// Every field is required and non-nullable
ChatResponse<Book> strict = await client.GetResponseAsync<Book>($"Extract the book from this note.\n\n{circe}", options);
Console.WriteLine(strict.Text);

// Unknown values may be null, and the prompt says so
ChatResponse<BookDraft> draft = await client.GetResponseAsync<BookDraft>(
    $"Extract the book from this note. Use null for anything the note does not say.\n\n{circe}", options);
Console.WriteLine(draft.Text);

enum BookFormat { Paperback, Hardcover, Ebook }

record Book(string Title, string Author, BookFormat Format, decimal Price, int CopiesInStock);

record BookDraft(string Title, string Author, BookFormat? Format, decimal? Price, int? CopiesInStock);

{ "title": "Circe","author": "Madeline Miller","format": "Hardcover","price": 0, "copiesInStock": 0}
{
  "title": "Circe",
  "author": "Madeline Miller",
  "format": "Hardcover",
  "price": null,
  "copiesInStock": null
}

The first answer matches the schema perfectly and is completely wrong. The note says the price is not known; since the model could not leave the price field empty, it wrote 0. It wrote 0 for the stock too. If you write this record to the database, a free book appears in your store and nobody can buy it because it is out of stock. The schema told the model "there must be a number here", and the model made one up.

In the second answer the fields are nullable and the prompt says "use null for anything the note does not say"; the model returned null for both unknowns. The lesson: when you design a schema, give the model a way to say "I don't know". Every field that may be missing from the source should be nullable; a required field forces the model to invent a value.

Validation and What to Do When It Fails

In a real application, data coming from the model needs to be checked at two layers: first technically, then against the business rules. Let's put both in a single method:

using System.Globalization;
using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");

string hailMary = "New in store: 'Project Hail Mary' by Andy Weir. Paperback. Now 18.99 USD (was 22.00). Only 2 copies left.";
string circe = "Just in: 'Circe' by Madeline Miller, hardcover. Price to be announced. Stock arrives next week.";

Console.WriteLine(await ExtractAsync(client, hailMary, maxOutputTokens: 200));
Console.WriteLine(await ExtractAsync(client, circe, maxOutputTokens: 200));

// A limit that is too low cuts the JSON in half
Console.WriteLine(await ExtractAsync(client, hailMary, maxOutputTokens: 15));

static async Task<string> ExtractAsync(IChatClient client, string note, int maxOutputTokens)
{
    ChatResponse<BookDraft> response = await client.GetResponseAsync<BookDraft>(
        $"Extract the book from this note. Use null for anything the note does not say.\n\n{note}",
        new ChatOptions { Temperature = 0, Seed = 42, MaxOutputTokens = maxOutputTokens });

    // 1) Is the JSON complete and does it fit the type?
    if (response.FinishReason == ChatFinishReason.Length || !response.TryGetResult(out BookDraft? draft))
    {
        return $"invalid ({response.FinishReason}): {response.Text}";
    }

    // 2) Do the values make sense for the business?
    List<string> problems = [];
    if (draft.Price is null or <= 0) problems.Add("no price");
    if (draft.CopiesInStock is null) problems.Add("no stock count");

    return problems.Count == 0
        ? $"ok: {draft.Title}, {draft.Format}, {draft.Price!.Value.ToString(CultureInfo.InvariantCulture)} USD, {draft.CopiesInStock} in stock"
        : $"needs review: {draft.Title} ({string.Join(", ", problems)})";
}

enum BookFormat { Paperback, Hardcover, Ebook }

record BookDraft(string Title, string Author, BookFormat? Format, decimal? Price, int? CopiesInStock);

ok: Project Hail Mary, Paperback, 18.99 USD, 2 in stock
needs review: Circe (no price, no stock count)
invalid (length): {
  "title": "Project Hail Mary",
  "author

The first layer checks whether the answer is complete and fits the type. In the third call we passed MaxOutputTokens = 15; the JSON was cut at "author, FinishReason became length and TryGetResult returned false. Had we read response.Result directly, we would have gotten a JsonException; when I tried it, the message was "Expected end of string, but instead reached end of data". TryGetResult lets you decide with a boolean instead of an exception.

The second layer checks whether the values fit your business. The "Circe" record is technically valid but has no price and no stock; we flagged it as "needs review" rather than as an error. The model cannot know this layer: your business rules are what say a $0 book is unacceptable.

What you do on failure depends on the kind of failure. For technical failures, one more try makes sense: with a higher limit for a truncated JSON, with the error added to the prompt for a parse error. But don't build an endless retry loop; after one or two attempts, record the request as failed. For business rule failures, retrying usually doesn't help, because the model draws the same conclusion from the same note; route those to a human. Whichever it is, the one thing you should not do is silently fill the missing fields with default values and move on.

Common Pitfalls

The first pitfall is treating the schema as text that explains what the fields mean. A schema constrains the structure; it doesn't explain the meaning. Is price the discounted price or the old one? The note had two prices and the model picked the right one, but don't leave that to chance: say it with a [Description] or a sentence in the prompt.

The second is keeping the output limit small and leaving no room for the end of the JSON. With schemas that have long description fields, MaxOutputTokens fills up easily and what remains is half an object that does not parse. Set a limit, but always check the finish reason.

The third is making every field required. As we saw above, a required field pushes the model to invent a value for information that is not in the source. Values like 0, an empty string or "Unknown" don't look like errors and leak into the database.

The fourth is mixing up the enum format between two sides that share the same data. On the C# side the enum values went as Paperback, on the Zod side as paperback. If two services read the same JSON, define the enum values explicitly in one place.

The fifth is assuming every provider offers the same support. According to Ollama's docs, Ollama Cloud does not support structured outputs, and cloud providers don't support every feature of JSON Schema either. Even if your code looks provider-independent through IChatClient, check in your provider's docs which schema features it accepts.

A Small Experiment

★☆☆ Add the field genre: z.enum(['fantasy', 'science_fiction', 'mystery', 'other']) to the Zod definition in schema.ts. Which value comes back for Project Hail Mary? Does it change when you remove the schema description from the prompt?

★★☆ Add string? Isbn and int? PublicationYear to BookDraft in s10_required.cs. For these two pieces of information that are not in the note, does the model return null or invent a value? Try the same fields without making them nullable.

★★★ Add a retry to ExtractAsync: if the finish reason is length, send the same request once more with MaxOutputTokens doubled, and return an error if it still fails. How does the output of the third call change?

We can now get typed answers from the model that a program can read. The next step is for the model to produce this structure not only as an answer but as a request for an action: tool calling. In the next post we will see, in raw form with qwen3:4b and the tools field of /api/chat, that the model does not call the function itself but only produces a call request.