LLM Cost, Latency and Model Choice: From Tokens to the Bill

Version note: The examples in this post were tested with Ollama 0.34 on local models (gemma3:4b, gemma3:12b, qwen3:4b, nomic-embed-text), using .NET 10 and Node 22.

Guess First

The Bookstore assistant gets 50,000 requests a month. In one scenario every request is short: you pass a book title and ask for a two-sentence blurb. In the other you use RAG to attach three thousand tokens of review text to every request and ask for a 150-token summary. If an output token costs five times as much as an input token, how many times more expensive is the second scenario? And one more: on the same machine, if you swap the 4-billion-parameter model for the 12-billion one, does the answer get three times slower, or worse?

We will work out both answers with real numbers in this post. So far we have focused on getting a correct answer out of an LLM; this time we look at the bill and the clock for that answer.

Cost Is Measured in Tokens

Cloud providers charge per token, not per character or per request. Prices are usually quoted per million tokens (MTok), and input and output are priced separately. Output tokens are almost always more expensive, because the model generates every output token one at a time, while it processes the input in a single pass. The cost of a request is therefore the sum of two products: input tokens times the input price, plus output tokens times the output price.

You don't need to guess the token counts; Ollama already sends prompt_eval_count and eval_count with every response. For the book blurb I used the real numbers from the measurement we will run in a moment (30 input, 62 output). The prices are examples: I picked them to be in the same range as mid-tier cloud models at the time of writing, and they are not a real quote. When you do your own math, use the current figure from the provider's pricing page.

// Use a dot as the decimal separator on every machine
CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;

PrintCost("Book description", inputTokens: 30, outputTokens: 62, requestsPerMonth: 50_000);
PrintCost("Review summary (RAG)", inputTokens: 3_000, outputTokens: 150, requestsPerMonth: 50_000);

static void PrintCost(string name, int inputTokens, int outputTokens, int requestsPerMonth)
{
    // Example prices, dollars per million tokens
    const decimal InputPricePerMTok = 2.00m;
    const decimal OutputPricePerMTok = 10.00m;

    decimal perRequest = inputTokens * InputPricePerMTok / 1_000_000m
                       + outputTokens * OutputPricePerMTok / 1_000_000m;
    decimal perMonth = perRequest * requestsPerMonth;

    Console.WriteLine($"{name,-22} {perRequest:0.000000} $/request  {perMonth,8:0.00} $/month");
}

Book description       0.000680 $/request     34.00 $/month
Review summary (RAG)   0.007500 $/request    375.00 $/month

The answer to the opening question: more than eleven times. The detail is more interesting. In the book blurb, 620 of the 680 micro-dollars come from output. In the RAG scenario the picture flips: 6,000 of the 7,500 micro-dollars are input. Even with output tokens costing five times more, once the context you put into the prompt is long enough, it decides the bill. The system prompt, the chat history, few-shot examples and the chunks that RAG brings in are sent again with every request and paid for every time. This is the "the API is stateless" fact from the context window post, measured in money.

Don't skip the CultureInfo.InvariantCulture line at the top of the code. On a machine set to Turkish, the decimal separator comes out as a comma without it, and in cost figures written to a log or a CSV that difference can break whatever system parses them next.

Latency: Three Separate Durations

When we say "the model is slow", we actually mean one of three different things. The duration fields in Ollama's response report them separately, all in nanoseconds. load_duration is the time spent loading the model from disk into memory; if the model is already in memory it is close to zero. prompt_eval_duration is the time spent processing the input. eval_duration is the time spent generating the output token by token.

The split matters because each one has a different fix. Load time lands on the first request; Ollama keeps a model in memory for five minutes by default, so the requests that follow don't pay it. Processing the input is usually fast because all tokens are processed in parallel. Generating output is sequential: each token depends on the previous one. That is why the output length usually decides the total time, roughly eval_count / tok/s. The time to first token we measured in the streaming post is close to load plus input processing.

4b or 12b: A Real Measurement

Let's ask both models the same question with the same settings, twice each, and read the durations from the response itself:

import ollama from 'ollama';

const prompt =
  'Write a two-sentence description of the book "The Pragmatic Programmer" for an online bookstore.';

// Convert nanoseconds to milliseconds
const ms = (ns: number) => (ns / 1e6).toFixed(0);

async function run(model: string) {
  const res = await ollama.chat({
    model,
    messages: [{ role: 'user', content: prompt }],
    options: { temperature: 0, seed: 42, num_predict: 120 },
  });
  const promptTps = res.prompt_eval_count / (res.prompt_eval_duration / 1e9);
  const evalTps = res.eval_count / (res.eval_duration / 1e9);
  console.log(
    `${model.padEnd(11)} load=${ms(res.load_duration)}ms ` +
      `prompt=${res.prompt_eval_count} tok (${promptTps.toFixed(0)} tok/s) ` +
      `output=${res.eval_count} tok (${evalTps.toFixed(1)} tok/s) ` +
      `total=${ms(res.total_duration)}ms`,
  );
  return res.message.content;
}

for (const model of ['gemma3:4b', 'gemma3:12b']) {
  for (let i = 0; i < 2; i++) {
    const text = await run(model);
    if (i === 1) console.log(`  -> ${text.trim()}\n`);
  }
}

gemma3:4b   load=1ms prompt=30 tok (293 tok/s) output=62 tok (43.5 tok/s) total=2632ms
gemma3:4b   load=1ms prompt=30 tok (589 tok/s) output=62 tok (43.4 tok/s) total=2046ms
  -> "The Pragmatic Programmer" is a timeless classic that teaches you how to build and maintain software effectively, focusing on practical skills and real-world problem-solving. Packed with insightful advice and actionable techniques, this book will transform the way you approach programming and help you become a more confident and productive developer.

gemma3:12b  load=3369ms prompt=30 tok (92 tok/s) output=65 tok (9.2 tok/s) total=10727ms
gemma3:12b  load=6ms prompt=30 tok (123 tok/s) output=65 tok (8.9 tok/s) total=7554ms
  -> "The Pragmatic Programmer" is a timeless guide offering practical advice and techniques for software development, focusing on real-world problem-solving rather than theoretical concepts. It's a must-read for developers of all levels seeking to improve their skills, write better code, and embrace a more pragmatic approach to their craft.

On this machine (Apple M5, 24 GB), 4b generated about 43 tokens per second and 12b about 9. The parameter count went up three times, and generation speed dropped not three but almost five times. That is the answer to the second opening question: "worse". On 12b's first call, loading the model into memory took 3.4 seconds; on the second call that cost was gone. 12b's answer is a little tidier, but is that worth five times the wait for a two-sentence blurb? Usually not.

A warning: total_duration comes out a bit larger than the sum of the three parts. The response doesn't say in detail where that difference went, so if you want to measure the time the user actually feels, also measure with your own clock on the client side. I also repeated this measurement three times, and because other requests were running on the machine at the same time, the numbers moved: between 22.7 and 43.5 tokens/s for 4b, and between 8.8 and 15.3 for 12b. In one round 12b's load time went up to 28 seconds; between two back-to-back requests the model had been evicted from memory and loaded again. Don't trust a single measurement; run it several times and look at the median.

Quantization: What Does Q4_K_M Mean?

If we stored every weight of a 4.3-billion-parameter model in 16 bits, the weights alone would take 8.6 GB. gemma3:4b is 3.3 GB on disk. Quantization explains the gap: weights are stored with fewer bits, at lower precision. Let's see how many bits per parameter the installed models use:

import ollama from 'ollama';

const { models } = await ollama.list();

for (const m of models) {
  // The value looks like "4.3B" or "137M"
  const params = parseFloat(m.details.parameter_size) * (m.details.parameter_size.endsWith('B') ? 1e9 : 1e6);
  const bitsPerParam = (m.size * 8) / params;
  console.log(
    `${m.name.padEnd(24)} ${m.details.parameter_size.padStart(6)} ` +
      `${m.details.quantization_level.padEnd(7)} ${(m.size / 1e9).toFixed(2)} GB ` +
      `${bitsPerParam.toFixed(1)} bits/param`,
  );
}

qwen3:4b                   4.0B Q4_K_M  2.50 GB 5.0 bits/param
nomic-embed-text:latest    137M F16     0.27 GB 16.0 bits/param
gemma3:4b                  4.3B Q4_K_M  3.34 GB 6.2 bits/param
gemma3:12b                12.2B Q4_K_M  8.15 GB 5.3 bits/param

nomic-embed-text is not quantized; it is F16 and keeps the full 16 bits. The others are Q4_K_M. The name comes from llama.cpp's "k-quants" family. Q4_K is about 4.5 bits per weight; the trailing M (medium) describes a mix: some sensitive tensors are stored in the higher-bit Q6_K, the rest in Q4_K. So why is gemma3:4b 6.2 bits and not 4.5? If you pass verbose: true to /api/show, you can see the type of every tensor:

type Tensor = { name: string; type: string; shape: number[] };

const res = await fetch('http://localhost:11434/api/show', {
  method: 'POST',
  body: JSON.stringify({ model: 'gemma3:4b', verbose: true }),
});
const { tensors } = (await res.json()) as { tensors: Tensor[] };

const totals = new Map<string, number>();
for (const t of tensors) {
  // Tensors starting with "v." belong to the vision encoder
  const part = t.name.startsWith('v.') ? 'vision' : 'text';
  const key = `${t.type} ${part}`;
  const count = t.shape.reduce((a, b) => a * b, 1);
  totals.set(key, (totals.get(key) ?? 0) + count);
}

for (const [key, count] of [...totals].sort((a, b) => b[1] - a[1])) {
  console.log(`${key.padEnd(12)} ${(count / 1e6).toFixed(0).padStart(5)}M params`);
}

Q4_K text     2721M params
Q6_K text     1159M params
F16 vision     416M params
F16 text         3M params
F32 vision       0M params
F32 text         0M params

The table explains it. The 416-million-parameter vision encoder (gemma3:4b can read images too) is not quantized at all; it stays in F16. Most of the Q6_K side is the 671-million-parameter token embedding table. The actual language layers are Q4_K. So the "Q4_K_M" label doesn't say the whole file is 4-bit; it names a weighted mix.

What you need to know in practice: quantization shrinks memory and disk use, fits a bigger model on the same machine, and usually speeds up generation, because less data is read from memory for every token. The price is a small loss in accuracy, and the loss grows as the bit count drops. Most default tags in the Ollama library ship as Q4_K_M; Gemma 3's -it-qat tags are versions trained with quantization in mind (quantization-aware training).

Don't Pay Twice for the Same Question

The cheapest token is the one you never send. In the Bookstore, the question "What is Clean Code about?" can arrive in exactly the same form from hundreds of users. Microsoft.Extensions.AI has a ready-made middleware for this: UseDistributedCache. It answers a request with the same messages and the same options from the cache, without sending it to the model.

using System.Diagnostics;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Options;
using OllamaSharp;

// In-memory cache for the demo; use a shared cache such as Redis in a real app
IDistributedCache cache = new MemoryDistributedCache(Options.Create(new MemoryDistributedCacheOptions()));

IChatClient ollama = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");

IChatClient client = ollama
    .AsBuilder()
    .UseDistributedCache(cache)
    .Build();

ChatOptions options = new() { Temperature = 0, Seed = 42 };

string[] questions =
[
    "In one sentence, what is the book \"Clean Code\" about?",
    "In one sentence, what is the book \"Clean Code\" about?",
    "In one sentence, what is the book \"Refactoring\" about?",
];

foreach (string question in questions)
{
    Stopwatch sw = Stopwatch.StartNew();
    ChatResponse response = await client.GetResponseAsync(question, options);
    sw.Stop();
    Console.WriteLine($"{sw.ElapsedMilliseconds,6} ms | {response.Text.Trim()}");
}

  1731 ms | "Clean Code" by Robert C. Martin is a practical guide to writing code that is easy to understand, maintain, and evolve, emphasizing readability, simplicity, and best practices.
     1 ms | "Clean Code" by Robert C. Martin is a practical guide to writing code that is easy to understand, maintain, and evolve, emphasizing readability, simplicity, and best practices.
  6990 ms | "Refactoring" is a guide to improving the internal structure of existing code without changing its external behavior, ultimately leading to more maintainable and flexible software.

The second question came back in 1 millisecond because it never reached the model. The third question was different, so it went to the model again; it took 7 seconds because of the other requests running on the machine at the same time (when I ran the program again, the same question returned in 1.4 seconds). This cache needs an exact match: "What is Clean Code about" and "What does Clean Code cover" are two separate keys. A "semantic cache" that works on similarity of meaning needs embeddings and belongs to the RAG series.

A small detail in the code comes from a real compile error. Chaining new OllamaApiClient(...).AsBuilder() made the compiler report CS0121: since OllamaApiClient implements both IChatClient and IEmbeddingGenerator, it couldn't pick between the two AsBuilder extension methods. Assigning it to an IChatClient variable first solves the problem.

The cache comes with two warnings. First, it only makes sense for deterministic requests; if you want creative text, you may not want to give everyone the same answer. Second, answers that contain user-specific data (such as order status) must not go into a cache shared between users. Cloud providers also have a "prompt caching" feature: when you send a long, fixed prompt prefix (a system prompt, a document) again, they bill that part at a discount. It caches the processing of the input, not the answer, and the details differ from provider to provider; read the docs before you rely on it.

Local or Cloud?

No single criterion makes this decision; you look at the answers to a few questions. Can the data leave the machine? If there is a constraint such as customer data, contracts or health information, a local model or a model running on your own infrastructure moves ahead. How hard is the task? For narrow jobs such as a short blurb, classification or extracting structured data, a 4b model is often enough; for long reasoning, broad knowledge and long context, large cloud models are clearly better. What are the volume and latency? Locally there is no per-token fee, but there is hardware, electricity and maintenance, and a single machine serves a limited number of requests at a time. In the cloud, scale is not your problem, but the bill grows with tokens.

A practical path: develop and test against a local model, and choose for production based on the need. To do that, your code must not be glued to a specific provider; that is exactly the topic of the next post.

Common Pitfalls

First, calculating cost by looking only at output. The system prompt, chat history and RAG context sent again with every request can make up most of the bill. Do the math from real requests with prompt_eval_count.

Second, mistaking the first request's duration for the model's speed. The first request includes load time. When you measure, send a warm-up request and keep load_duration separate.

Third, deciding on a single measurement. With other work running on the same machine, the numbers can move by a factor of two. Repeat a few times and take the median.

Fourth, assuming "a bigger model is always better". On a small, narrow task the big model may write a slightly better sentence, but it is several times slower and more expensive. Compare with your own examples, using a small evaluation set like the one in post 15.

Fifth, forgetting about memory. The size of the model file is the lower bound of the memory it will take; as the context window grows, the KV cache needs extra memory. On a 24 GB machine, an 8 GB model and a few small models may not all stay loaded at once, and Ollama evicts one of them.

A Small Experiment

★☆☆ Run the measurement script on your own machine and note the tokens per second for gemma3:4b. Then raise num_predict from 120 to 400 and ask for a longer answer. Does the total time grow in proportion to the output length?

★★☆ Do the cost math for the Bookstore's own scenario: send a request with a real system prompt and a few turns of chat history, take prompt_eval_count and eval_count, and pass them to the PrintCost function. How does the cost per request change as the chat history grows?

★★★ Ask the cached client a question, then ask the same question again with a different Temperature. Did it come from the cache or go to the model? Find out from the duration, and think about why that is the right behavior.

In this post our code was tied tightly to one place: OllamaSharp and the ollama package. In the next post we will write the same code independently of the provider: the IChatClient abstraction, Ollama's OpenAI-compatible /v1 endpoint, switching providers through configuration, storing API keys safely, and wiring it all into ASP.NET Core with DI.