Sampling in LLMs: Temperature, Top-p, Top-k and Seed

Version note: The examples in this post were tested with Ollama 0.34 on local models (gemma3:4b), using .NET 10 and Node 22.

Guess First

You ask the model the same question, in the same words, five times. Do you get the same answer five times? And if you set temperature to 0 and add a seed on top, can you promise the answer will be the same every time?

Most developers answer the first question with "it's a computer, of course it's the same". Plenty answer the second with "absolutely". We will check both with real runs in a moment; the answer to the second one is more nuanced than you might expect.

Same Question, Different Answers

We are looking for a name for a new branch of our Bookstore. The TypeScript code below asks the same question five times with the model's default settings, then five times with temperature: 0:

import ollama from 'ollama';

const prompt = 'Suggest a name for a new bookstore. Reply with the name only.';

async function ask(options: { temperature?: number; seed?: number } = {}): Promise<string> {
  const res = await ollama.chat({
    model: 'gemma3:4b',
    messages: [{ role: 'user', content: prompt }],
    options,
  });
  return res.message.content.trim();
}

console.log('--- model defaults');
for (let i = 0; i < 5; i++) console.log(await ask());

console.log('--- temperature 0');
for (let i = 0; i < 5; i++) console.log(await ask({ temperature: 0 }));

--- model defaults
The Written Page
The Paper Lantern
The Storyteller's Nook
The Written Page
The Written Page
--- temperature 0
The Book Nook
The Book Nook
The Book Nook
The Book Nook
The Book Nook

With the defaults, five attempts gave three different names. With temperature: 0, all five are the same. Here is the interesting part: the zero-temperature answer, "The Book Nook", did not show up in any of the first five attempts. When the model behaves randomly, it does not give the "best" answer each time; it picks one of the plausible answers.

As you may remember from the first post of the series, at each step the model computes a probability distribution for the next token. Picking a single token from that distribution is called sampling. Every setting in this post controls how that pick is made.

Behind the Curtain: Candidate Tokens

The Ollama version we use can return the most likely candidates at each step along with the answer: with logprobs: true and top_logprobs. A logprob is the natural logarithm of the probability; we turn it back into a percentage with Math.exp.

import ollama from 'ollama';

const res = await ollama.chat({
  model: 'gemma3:4b',
  messages: [{ role: 'user', content: 'Suggest a name for a new bookstore. Reply with the name only.' }],
  options: { temperature: 0 },
  logprobs: true,
  top_logprobs: 5,
});

// For each generated token, show the five most likely candidates
for (const step of res.logprobs!.slice(0, 3)) {
  const candidates = step.top_logprobs!
    .map((c) => `${JSON.stringify(c.token)} ${(Math.exp(c.logprob) * 100).toFixed(1)}%`)
    .join(' | ');
  console.log(`${JSON.stringify(step.token).padEnd(8)} <- ${candidates}`);
}

"The"    <- "The" 99.3% | "Ink" 0.6% | "Page" 0.0% | "Pages" 0.0% | "Chapter" 0.0%
" Book"  <- " Book" 37.6% | " Written" 25.8% | " Story" 23.1% | " Ink" 7.4% | " Paper" 2.4%
" N"     <- " N" 100.0% | " Burrow" 0.0% | "binder" 0.0% | "worm" 0.0% | " Nest" 0.0%

The output explains everything. At the first step the model is almost certain: "The" at 99.3%. At the third step " N" (the start of Nook) is close to 100% as well. The real decision happens at the second step: " Book" 37.6%, " Written" 25.8%, " Story" 23.1%. "The Written Page", "The Storyteller's Nook" and "The Paper Lantern" from the default run were born exactly from these candidates. The model finds three or four names roughly equally reasonable; which one it says depends on how the dice land at that step.

One more observation: when I ran the same request with different temperature values, these percentages did not change. So logprobs shows the model's raw distribution; temperature and the other settings are applied on top of it, at the moment of picking.

Temperature, Top-k and Top-p

Now let's calculate by hand what these settings do to the same candidate list. The code below takes the five candidates of the second step, first recomputes their probabilities with different temperature values, then checks which candidates top-p keeps:

import ollama from 'ollama';

const res = await ollama.chat({
  model: 'gemma3:4b',
  messages: [{ role: 'user', content: 'Suggest a name for a new bookstore. Reply with the name only.' }],
  options: { temperature: 0 },
  logprobs: true,
  top_logprobs: 5,
});

// Second token: the candidates that come after "The"
const candidates = res.logprobs![1].top_logprobs!;

function withTemperature(t: number): string {
  const weights = candidates.map((c) => Math.exp(c.logprob / t));
  const sum = weights.reduce((a, b) => a + b, 0);
  return candidates
    .map((c, i) => `${c.token.trim()} ${((weights[i] / sum) * 100).toFixed(0)}%`)
    .join(', ');
}

for (const t of [0.2, 0.7, 1.0, 1.5]) {
  console.log(`T=${t}: ${withTemperature(t)}`);
}

// Top-p: keep the smallest set whose cumulative probability reaches p
function nucleus(p: number): string[] {
  const kept: string[] = [];
  let total = 0;
  for (const c of candidates) {
    kept.push(c.token.trim());
    total += Math.exp(c.logprob);
    if (total >= p) break;
  }
  return kept;
}

console.log('top_p=0.5:', nucleus(0.5));
console.log('top_p=0.9:', nucleus(0.9));

T=0.2: Book 81%, Written 12%, Story 7%, Ink 0%, Paper 0%
T=0.7: Book 45%, Written 27%, Story 23%, Ink 4%, Paper 1%
T=1: Book 39%, Written 27%, Story 24%, Ink 8%, Paper 2%
T=1.5: Book 33%, Written 26%, Story 24%, Ink 11%, Paper 5%
top_p=0.5: [ 'Book', 'Written' ]
top_p=0.9: [ 'Book', 'Written', 'Story', 'Ink' ]

Temperature means dividing the model's raw scores (logits) by a number before the softmax; dividing the logprob by T and normalizing again gives the same result. As T gets smaller, the distribution gets sharper: at 0.2, " Book" climbs to 81% and the rest approach zero. As T gets larger, the distribution flattens: at 1.5, weak candidates ("Paper" at 5%) get a real chance. As T approaches zero, the most likely token is picked every time; this is called greedy decoding. A caveat: we did the math over the top five candidates only, so the T=1 line is a few points off from the raw values. Real sampling works over the whole vocabulary.

Top-k is a blunter pair of scissors: it keeps only the k most likely candidates and drops the rest. Top-p (nucleus sampling, from the 2019 paper by Holtzman et al.) looks at the total probability instead of a count: it adds candidates from the largest down and stops as soon as the total reaches p. In the output, top_p=0.5 kept only two candidates and 0.9 kept four. The advantage: when the model is sure (like the 99.3% at the first step) the pool shrinks to a single candidate, and when it is undecided the pool widens. Top-k keeps the same number of candidates every time.

The model also ships with default values for these. In Ollama you can look at the parameters in the model's Modelfile:

$ ollama show gemma3:4b --parameters
stop                           "<end_of_turn>"
temperature                    1
top_k                          64
top_p                          0.95
$ ollama show qwen3:4b --parameters
top_k                          20
top_p                          0.95
repeat_penalty                 1
stop                           "<|im_start|>"
stop                           "<|im_end|>"
temperature                    0.6

gemma3 comes with temperature 1, top_k 64 and top_p 0.95 by default; qwen3 with 0.6, 20 and 0.95. If your request does not set a value, these are used. So "I didn't set anything" really means "I'm using whatever the person who packaged the model chose", and that changes from model to model.

Seed: Making Randomness Repeatable

Sometimes you want variety but you also need to reproduce the same variety: to recreate a bug, to pin a test or to compare two prompts fairly. A seed is the starting value of the random number generator that sampling uses; the same seed means the same sequence of random numbers. On the C# side, the ChatOptions class of Microsoft.Extensions.AI carries it directly:

using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");
string prompt = "Write a one-sentence tagline for a small bookstore. Reply with the tagline only.";

long[] seeds = [42, 42, 7, 7];
foreach (long seed in seeds)
{
    ChatOptions options = new() { Temperature = 0.8f, Seed = seed };
    ChatResponse response = await client.GetResponseAsync(prompt, options);
    Console.WriteLine($"seed {seed}: {response.Text.Trim()}");
}

seed 42: Lose yourself in the pages, one story at a time.
seed 42: Lose yourself in the pages, one story at a time.
seed 7: Discover your next story at [Bookstore Name].
seed 7: Discover your next story at [Bookstore Name].

With Temperature = 0.8f, randomness is on, yet the same seed gave the same sentence and a different seed gave a different one. OllamaSharp maps these properties to Ollama's options object: Temperature → temperature, TopP → top_p, TopK → top_k, Seed → seed, MaxOutputTokens → num_predict, StopSequences → stop. I checked this by capturing the outgoing request with a DelegatingHandler.

The promise a seed makes is limited. The Microsoft.Extensions.AI docs also describe Seed as a value the service uses to control the reproducibility of results, not as a guarantee. You can only expect the same result with the same model build, the same quantization, the same hardware and backend, the same options and a similar load. Floating-point math on the GPU and the batching of requests can still make a difference. I will show a live example of this in the pitfalls section.

Limiting the Length: num_predict and Stop Sequences

Sampling is not only about which token gets picked but also about when generation stops. Normally the model stops when it produces its own end token. You can add two more controls:

using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");
string prompt = "List five classic novels, one per line, numbered 1 to 5. No extra text.";

ChatResponse full = await client.GetResponseAsync(prompt, new ChatOptions { Temperature = 0 });
Print("no limit", full);

ChatResponse cut = await client.GetResponseAsync(prompt, new ChatOptions { Temperature = 0, MaxOutputTokens = 12 });
Print("MaxOutputTokens = 12", cut);

ChatResponse stopped = await client.GetResponseAsync(prompt, new ChatOptions { Temperature = 0, StopSequences = ["4."] });
Print("StopSequences = [\"4.\"]", stopped);

static void Print(string title, ChatResponse response)
{
    Console.WriteLine($"--- {title}");
    Console.WriteLine(response.Text);
    Console.WriteLine($"finish: {response.FinishReason}, output tokens: {response.Usage?.OutputTokenCount}");
}

--- no limit
1. Pride and Prejudice
2. To Kill a Mockingbird
3. 1984
4. Jane Eyre
5. The Great Gatsby

finish: stop, output tokens: 36
--- MaxOutputTokens = 12
1. Pride and Prejudice
2. To Kill a Mock
finish: length, output tokens: 12
--- StopSequences = ["4."]
1. Pride and Prejudice
2. To Kill a Mockingbird
3. 1984

finish: stop, output tokens: 25

MaxOutputTokens (num_predict in Ollama) puts a hard ceiling on the number of generated tokens. The model was cut off in the middle of a word at "To Kill a Mock", and the finish reason became length. If you use the answer without checking that reason, you will treat a truncated text as complete; when you generate JSON, that means half an object that does not parse. We will come back to this in the structured output post.

StopSequences (stop in Ollama) stops generation when a specific text is produced. Generation stopped when "4." appeared, and note: the stop text was not included in the answer, and the finish reason came back as stop. So you cannot tell a natural ending from hitting a stop sequence by looking at that field alone. Stop sequences are useful wherever you don't want the answer to go past a pattern: a line break for a single-line answer, the start of the next tag in a template. The model's chat template uses the same mechanism; stop "<end_of_turn>" in gemma3 is exactly that.

Settings by Task

There is no single "right" setting; you have to look at what the task needs. The split I use in practice: if the answer has one correct form (classification, extraction, JSON, code transformation), turn randomness off. If the answer has more than one good form (a book blurb, a title suggestion, a tagline), turn it up a little. Keeping this in named presets instead of scattered numbers makes the code easier to read:

using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");

ChatResponse genre = await client.GetResponseAsync(
    "Which genre is 'Dune' by Frank Herbert? Reply with one word.",
    SamplingPresets.Extraction);
Console.WriteLine($"extraction: {genre.Text.Trim()}");

ChatResponse blurb = await client.GetResponseAsync(
    "Write a two-sentence shelf note for 'Dune' by Frank Herbert.",
    SamplingPresets.Creative);
Console.WriteLine($"creative: {blurb.Text.Trim()}");

static class SamplingPresets
{
    // Classification, extraction, JSON: the same input should give the same output
    public static ChatOptions Extraction => new() { Temperature = 0f, Seed = 42, MaxOutputTokens = 50 };

    // Shelf notes, taglines, suggestions: some variety is welcome
    public static ChatOptions Creative => new() { Temperature = 0.9f, TopP = 0.95f, TopK = 64, MaxOutputTokens = 120 };
}

extraction: Science Fantasy
creative: Dive into a sprawling epic of politics, ecology, and destiny in Frank Herbert's *Dune*. Prepare to be transported to a desert planet unlike any you've ever imagined!

Two observations. First, even though we said "reply with one word", the model wrote two: "Science Fantasy". Temperature = 0 makes the answer repeatable, not the model more obedient. An answer that ignores the instruction can come at zero temperature too, and it comes the same way every time. Second, we set MaxOutputTokens even in the creative preset: asking for variety does not mean letting the length get out of hand.

And a general rule: don't change temperature and top-p aggressively at the same time. Both shape the same distribution. Changing one and measuring the result beats moving both and not knowing which one helped.

Common Pitfalls

The first pitfall is treating a seed as a guarantee. The first time I ran the seed example above, the output was this:

seed 42: Lose yourself in a good book, right here.
seed 42: Lose yourself in the pages, one story at a time.
seed 7: Discover your next story at [Bookstore Name].
seed 7: Discover your next story at [Bookstore Name].

Same seed, same temperature, same prompt, two requests back to back; yet the first two lines differ. In the next three runs all four lines were consistent, as shown earlier. The cause is not visible from the outside. Other jobs were using the same Ollama server at the time, and the server's internal state (caches, batching) can change between requests. The lesson: use a seed for debugging and comparisons, but don't build business logic on the assumption that the same input always gives the same output. In tests, check the properties of the answer (format, length, expected label) instead of comparing the text exactly.

The second is forgetting that defaults change from model to model. When you switch from gemma3 to qwen3, temperature drops from 1 to 0.6 and top_k from 64 to 20 even if you touch nothing. Send the settings you want to keep stable explicitly with every request.

The third is treating temperature: 0 as a quality setting. Zero temperature picks the most likely token; the most likely token is not always the correct answer. As in the "Science Fantasy" example, an answer that ignores the instruction also comes back again and again at zero temperature. Quality is decided by the prompt, the model choice and validation; sampling only decides how to pick from the distribution.

The fourth is setting MaxOutputTokens without checking the finish reason. The limit protects cost and latency, but an answer that ends with length is incomplete. If you are going to show it to the user or parse it, look at FinishReason first.

The fifth is treating a high temperature as a "creativity" knob. As you go above 1.5, weak candidates gain ground, and past a certain point that is not creativity but odd word choices and drifting off topic. If you want variety, a temperature between 0.7 and 1 and a sensible top-p are usually enough.

A Small Experiment

★☆☆ Change the prompt in same-question.ts to "Recommend one fantasy novel. Reply with the title only." and run it five times with the defaults. How many different answers did you get? Then look at the first tokens of the same prompt with candidates.ts and find the step where the answers split.

★★☆ Run reshape.ts on the candidates of the first step, that is res.logprobs![0]. When the model is 99.3% sure, how much can even T=1.5 change the distribution? This shows why a high temperature does not affect every token equally.

★★★ Run the seed example ten times in a loop and count how often the same seed gives the same sentence. Then repeat it while sending other requests from a second terminal at the same time, and compare the two results.

In every example in this post we waited for the answer to complete and only then printed anything. In the next post we move on to streaming: how Ollama sends the answer piece by piece, how to read it with IAsyncEnumerable in C# and an async iterator in TypeScript, how to cancel the stream halfway and how to measure when the first token arrives.