Streaming LLM Responses: NDJSON, IAsyncEnumerable and Time to First Token

Version note: The examples in this post were tested with Ollama 0.34 on local models (gemma3:4b), using .NET 10 and Node 22.

Guess First

You ask the same question in two ways: once waiting for the whole answer, once receiving it piece by piece with streaming. Does the streaming request finish earlier? Does the model produce the answer faster because you are waiting for it?

Most of us have the intuition "yes, streaming is faster". When we measure, we will see that the total time is almost the same. What changes is when the user sees the first word. That difference is exactly what this post is about.

Why Streaming?

The model produces the answer token by token; remember the loop from the first post of the series: pick a token, append it, repeat. A request that waits for the whole answer has nothing in hand until that loop ends. A paragraph of a hundred tokens takes a few seconds on a local 4B model; with a long answer or a slow model that goes up to tens of seconds. Meanwhile the user stares at an empty screen.

With streaming, the server sends each token as soon as it is produced. The total time stays the same, but the perceived latency drops dramatically: the user sees the first word within a few hundred milliseconds and starts reading. A restaurant analogy: the difference between a table where all the dishes arrive at once when everything is ready and a table where the soup is served as soon as it is done. The kitchen works at the same speed, but the guest at the second table feels the wait less.

So there are two separate numbers to measure: the time to first token (TTFT) and the total time. In chat interfaces, the first one usually decides the user experience.

Ollama's Stream: NDJSON

Let's look inside the wire first. On Ollama's /api/chat endpoint, streaming is on by default; if you don't send the stream field at all, the answer arrives as a stream:

curl -s http://localhost:11434/api/chat -d '{
  "model": "gemma3:4b",
  "messages": [{ "role": "user", "content": "Name the author of The Hobbit. Reply in one short sentence." }],
  "options": { "temperature": 0, "seed": 42 }
}'

{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.536477Z","message":{"role":"assistant","content":"J"},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.630588Z","message":{"role":"assistant","content":"."},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.668004Z","message":{"role":"assistant","content":"R"},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.706532Z","message":{"role":"assistant","content":"."},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.745246Z","message":{"role":"assistant","content":"R"},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.789672Z","message":{"role":"assistant","content":"."},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.837245Z","message":{"role":"assistant","content":" Tolkien"},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.883892Z","message":{"role":"assistant","content":" wrote"},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.930959Z","message":{"role":"assistant","content":" The"},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:28.976957Z","message":{"role":"assistant","content":" Hobbit"},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:29.023111Z","message":{"role":"assistant","content":"."},"done":false}
{"model":"gemma3:4b","created_at":"2026-11-06T20:47:29.070258Z","message":{"role":"assistant","content":""},"done":true,"done_reason":"stop","total_duration":953237833,"load_duration":747458,"prompt_eval_count":23,"prompt_eval_cached_count":0,"prompt_eval_duration":417173000,"eval_count":12,"eval_duration":533648000}

This format is called NDJSON (newline-delimited JSON): every line is a valid JSON object on its own. Each line carries the new piece in message.content, and in this run each piece corresponds to exactly one token: "J", ".", "R", ".", " Tolkien". The last line comes with done: true, and the statistics are only there: prompt_eval_count 23 tokens, eval_count 12 tokens, durations in nanoseconds. eval_count / eval_duration gives about 22.5 tokens per second here.

Ollama's OpenAI-compatible /v1 endpoint sends the same stream in a different format:

curl -s http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "gemma3:4b",
  "messages": [{ "role": "user", "content": "Name the author of The Hobbit. Reply in one short sentence." }],
  "temperature": 0, "seed": 42, "stream": true
}'

data: {"id":"chatcmpl-955","object":"chat.completion.chunk","created":1793998049,"model":"gemma3:4b","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":"J"},"finish_reason":null}]}

data: {"id":"chatcmpl-955","object":"chat.completion.chunk","created":1793998049,"model":"gemma3:4b","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"content":"."},"finish_reason":null}]}

...

data: {"id":"chatcmpl-955","object":"chat.completion.chunk","created":1793998049,"model":"gemma3:4b","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

This is Server-Sent Events (SSE). I shortened the middle of the output: each event starts with the data: prefix and is separated by an empty line, the pieces are in choices[0].delta.content, and a data: [DONE] line announces the end of the stream. In its default form there are no token statistics either. You won't need to parse either format by hand; the SDKs do it for you. But when you see these lines in a proxy, in a log or in the browser's network tab, knowing what you are looking at makes your job easier.

Reading the Stream in C#: IAsyncEnumerable

Microsoft.Extensions.AI has a separate method for streaming: GetStreamingResponseAsync. Its return type is IAsyncEnumerable<ChatResponseUpdate>, an asynchronous sequence you walk with await foreach. The code below sends the same prompt first without streaming, then with streaming, and measures both times:

using System.Diagnostics;
using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");
string prompt = "Recommend 'The Hobbit' to a new reader in one short paragraph.";
ChatOptions options = new() { Temperature = 0, Seed = 42 };

// 1) Without streaming: nothing to show until the whole answer is ready
Stopwatch sw = Stopwatch.StartNew();
ChatResponse whole = await client.GetResponseAsync(prompt, options);
Console.WriteLine($"[non-streaming] first text after {sw.ElapsedMilliseconds} ms");

// 2) With streaming: print each piece as it arrives
List<ChatResponseUpdate> updates = [];
sw.Restart();
long firstTokenMs = -1;
await foreach (ChatResponseUpdate update in client.GetStreamingResponseAsync(prompt, options))
{
    if (firstTokenMs < 0 && update.Text.Length > 0)
    {
        firstTokenMs = sw.ElapsedMilliseconds;
    }
    Console.Write(update.Text);
    updates.Add(update);
}
Console.WriteLine();
Console.WriteLine($"[streaming] first text after {firstTokenMs} ms, finished after {sw.ElapsedMilliseconds} ms");

ChatResponse merged = updates.ToChatResponse();
Console.WriteLine($"updates: {updates.Count}, output tokens: {merged.Usage?.OutputTokenCount}, finish: {merged.FinishReason}");
Console.WriteLine($"same text as non-streaming: {merged.Text == whole.Text}");

[non-streaming] first text after 2819 ms
If you’re looking for a fantastic adventure, I wholeheartedly recommend “The Hobbit” by J.R.R. Tolkien! It’s a charming and exciting story about a comfortable hobbit named Bilbo Baggins who gets swept up in a thrilling quest to reclaim a treasure from a fearsome dragon. Filled with memorable characters, fantastical creatures, and a touch of magic, it’s a perfect introduction to the world of Middle-earth and a wonderfully engaging read for anyone who loves a good tale of bravery, friendship, and a little bit of mischief.
[streaming] first text after 56 ms, finished after 2804 ms
updates: 114, output tokens: 114, finish: stop
same text as non-streaming: True

The numbers answer the opening question. Without streaming we saw the first text after 2819 ms, because the first text is also the last text. With streaming the first piece arrived at 56 ms and the stream ended at 2804 ms. The total time is the same; the difference is whether the user stares at an empty screen for 2.7 seconds. One reason 56 ms is so small is that the prompt is short and the same prompt had just been processed by the previous request; with a long prompt the first token comes later.

Each ChatResponseUpdate carries one piece: update.Text is the text that just arrived. To keep the pieces after the stream ends, we collected them in a list and merged them into a single ChatResponse with the ToChatResponse() extension method. That also gives us information that only arrives toward the end, such as Usage and FinishReason. The merged text is identical to the non-streaming answer; one more place where temperature: 0 and a fixed seed pay off. If you keep a chat history, you can add the same list to it with history.AddMessages(updates).

An Async Iterator in TypeScript

In the ollama package, passing stream: true makes chat return an async iterator that you walk with for await. This time let's also read the statistics in the last chunk:

import ollama from 'ollama';

const started = performance.now();
let firstTokenMs = -1;

const stream = await ollama.chat({
  model: 'gemma3:4b',
  messages: [{ role: 'user', content: "Recommend 'The Hobbit' to a new reader in one short paragraph." }],
  options: { temperature: 0, seed: 42 },
  stream: true,
});

for await (const chunk of stream) {
  if (firstTokenMs < 0 && chunk.message.content.length > 0) {
    firstTokenMs = performance.now() - started;
  }
  process.stdout.write(chunk.message.content);

  if (chunk.done) {
    // Only the last chunk carries the statistics (durations are in nanoseconds)
    const tokensPerSecond = chunk.eval_count / (chunk.eval_duration / 1e9);
    console.log();
    console.log(`first text: ${firstTokenMs.toFixed(0)} ms, total: ${(performance.now() - started).toFixed(0)} ms`);
    console.log(`load: ${(chunk.load_duration / 1e6).toFixed(0)} ms, prompt: ${chunk.prompt_eval_count} tokens in ${(chunk.prompt_eval_duration / 1e6).toFixed(0)} ms`);
    console.log(`output: ${chunk.eval_count} tokens, ${tokensPerSecond.toFixed(1)} tokens/s, done_reason: ${chunk.done_reason}`);
  }
}

If you’re looking for a fantastic adventure, I wholeheartedly recommend “The Hobbit” by J.R.R. Tolkien! It’s a charming and exciting story about a comfortable hobbit named Bilbo Baggins who gets swept up in a thrilling quest to reclaim a treasure from a fearsome dragon. Filled with memorable characters, fantastical creatures, and a touch of magic, it’s a perfect introduction to the world of Middle-earth and a wonderfully engaging read for anyone who loves a good tale of bravery, friendship, and a little bit of mischief.
first text: 26981 ms, total: 31858 ms
load: 20658 ms, prompt: 24 tokens in 155 ms
output: 114 tokens, 23.4 tokens/s, done_reason: stop

The first token arrived after 27 seconds. The model did not change, the prompt did not change; the only difference was that the model was not in memory at that moment. load_duration says 20658 ms: the model was loaded from disk again. I ran the same code twice more; the last run was this:

first text: 1592 ms, total: 6019 ms
load: 1 ms, prompt: 24 tokens in 151 ms
output: 114 tokens, 25.7 tokens/s, done_reason: stop

Loading dropped to 1 ms and the first token to 1.6 seconds. The tokens per second stayed in the 23–26 range in both runs. So generation speed is a property of the model and the hardware; the time to first token is a property of the conditions at that moment.

Reading the Time to First Token Correctly

The time to first token is not a measure of a single thing but the sum of several waits: the request queuing on the server, the model loading into memory (load_duration), the prompt being processed (prompt_eval_duration) and the first token being generated. In the warm run, loading was 1 ms and the prompt 151 ms; yet the first token arrived at 1592 ms. The gap has no counterpart in the numbers Ollama reports. Other jobs were using the same Ollama on this machine at the time, and the request most likely waited for its turn. When I ran the C# example again at a busy moment, the timing lines were these:

[non-streaming] first text after 3946 ms
[streaming] first text after 11016 ms, finished after 15307 ms

In the streaming request, the first token arrived after 11 seconds; the code is exactly the same program that measured 56 ms. The lesson: don't judge TTFT by a single measurement. Measure it in your application, look at the distribution (percentiles rather than the average) and compare load_duration and prompt_eval_duration from the last chunk with the time you measured yourself. Those three numbers let you tell whether the latency comes from the model, from the length of the prompt or from the queue. We will expand these measurements in the post on cost and latency.

Stopping the Stream Halfway

Another benefit of streaming is that you can stop the answer before it ends: when the user presses "Stop", leaves the page or sees the answer going in the wrong direction. In C#, the tool for this is, as always, a CancellationToken:

using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");
ChatOptions options = new() { Temperature = 0, Seed = 42 };

using CancellationTokenSource cts = new();
int received = 0;
try
{
    await foreach (ChatResponseUpdate update in client.GetStreamingResponseAsync(
        "Summarize the plot of 'The Lord of the Rings' in detail.", options, cts.Token))
    {
        Console.Write(update.Text);
        received++;
        if (received == 20)
        {
            // The user pressed "Stop", for example
            cts.Cancel();
        }
    }
}
catch (OperationCanceledException)
{
    // Some clients throw here; OllamaSharp did not in this run
    Console.WriteLine("\nOperationCanceledException");
}

Console.WriteLine();
Console.WriteLine(cts.IsCancellationRequested
    ? $"cancelled after {received} updates"
    : $"completed with {received} updates");

Okay, here's a detailed summary of *The Lord of the Rings*, broken down into its
cancelled after 20 updates

Something happened here that I did not expect. When the token was cancelled, the await foreach loop simply ended; no OperationCanceledException was thrown. In this run OllamaSharp 5.5.2 treated cancellation as the end of the stream rather than as an error. The general behavior of cancellation in .NET is to throw, and another IChatClient implementation may do exactly that. That's why the code handles both: the catch block catches the exception, and the IsCancellationRequested check after the loop tells the silent ending apart. In ASP.NET Core, instead of creating this token yourself, you pass HttpContext.RequestAborted; when the user closes the connection, the model request is cancelled too.

On the TypeScript side, the iterator returned by the ollama package has its own abort() method:

import ollama from 'ollama';

const stream = await ollama.chat({
  model: 'gemma3:4b',
  messages: [{ role: 'user', content: "Summarize the plot of 'The Lord of the Rings' in detail." }],
  options: { temperature: 0, seed: 42 },
  stream: true,
});

let received = 0;
try {
  for await (const chunk of stream) {
    process.stdout.write(chunk.message.content);
    received++;
    if (received === 20) {
      // Stops only this stream; ollama.abort() would stop every stream of the client
      stream.abort();
    }
  }
} catch (error) {
  console.log();
  console.log(`${(error as Error).name} after ${received} chunks`);
}

Okay, here's a detailed summary of *The Lord of the Rings*, broken down into its
AbortError after 20 chunks

Here the behavior is different: after abort() was called, the loop threw an AbortError. The package uses an AbortController internally, and cancellation arrives as the familiar fetch error. The same job, two different signals in two SDKs. When you write cancellation code, don't assume what your SDK does before trying it once.

Common Pitfalls

The first pitfall is assuming that every network read brings a complete line when you read the stream by hand. If you parse NDJSON yourself, buffer the incoming bytes until a line break; a JSON line can be split across two reads. The same rule applies to the event separator in SSE. If you can, leave this job to the SDK.

The second is assuming that every piece is one token. In these runs 114 updates matched 114 tokens, but that is not a contract. Take the token count from eval_count in the last chunk or from the Usage of the merged response, not by counting pieces.

The third is forgetting the cost of a cancelled stream. The statistics come in the last chunk; when you cut the stream halfway, that chunk never arrives and you cannot know how many tokens were spent. With cloud providers, the tokens generated up to the moment of cancellation are usually still billed; check your provider's docs. If you track usage, count cancelled requests separately.

The fourth is not carrying cancellation all the way down the chain. Even if the user closes the tab, the request keeps running in the background if you did not pass the token to the model call. Locally that means wasted GPU time; in the cloud, wasted money.

The fifth is forgetting the layers in between. If a reverse proxy or a compression middleware buffers the response, an answer generated token by token on the server reaches the browser in one piece. Test streaming end to end, where the user actually sees it.

The sixth is treating partial text as finished text. During the stream, the text in your hands can be half a sentence, an unclosed Markdown marker or half a JSON at any moment. That is fine for printing to the screen; for parsing, writing to the database or passing it to another step, wait for the stream to end.

A Small Experiment

★☆☆ Run the curl command again with "stream": false added. How many lines came back, and where are the statistics? Then run the same command asking for a longer answer and watch the lines flow in the terminal.

★★☆ Turn stream.ts into a loop that runs the same prompt ten times. In each run, print the time to first token, load_duration and the tokens per second; compare the minimum, maximum and median values.

★★★ Tie the C# cancellation example to time instead of 20 updates with cts.CancelAfter(TimeSpan.FromSeconds(1)). Then try merging the same stream with ToChatResponse(): what happens to Usage and FinishReason in a cancelled stream?

We have seen how to read, measure and stop the stream; now we control how fast the answer arrives. Next comes the answer itself. In the next post we move on to writing prompts: clear instructions, delimiters, prompts with examples (few-shot) and describing the output format. We will put bad and good prompts side by side with real output from the same model.