LLM Hallucination and Reliability: Cite, Verify, Measure

Version note: The examples in this post were tested with Ollama 0.34 on local models (gemma3:4b, nomic-embed-text), using Node 22.

Guess First

What happens if you ask the model for the author of a book that doesn't exist? Will it say "I don't know that book", or will it make up an author and a publication year? And if it makes them up, how will you tell?

In the previous post we tied the model's answers to our own documents with RAG. In this post we step back to look at why hallucinations happen, then add source citations to the answer, check those sources in code and, most importantly, measure with numbers whether our changes actually work.

A Book That Doesn't Exist

Two questions: one about a real novel, the other about a title I made up.

import ollama from 'ollama';

// No context, no tools: only what the model learned in training
for (const question of process.argv.slice(2)) {
  const response = await ollama.chat({
    model: 'gemma3:4b',
    messages: [{ role: 'user', content: question }],
    options: { temperature: 0, seed: 42 },
  });
  console.log(`> ${question}\n${response.message.content}\n`);
}

$ npx tsx hallucination.ts "Who wrote the novel 'Dune' and when was it published?" "Who wrote the novel 'Seven Winters in Quillhaven' and when was it published?"
> Who wrote the novel 'Dune' and when was it published?
The novel "Dune" was written by **Frank Herbert**. 

It was originally published in **1965** by Chilton Books. 

Do you want to know anything more about the book or its author?

> Who wrote the novel 'Seven Winters in Quillhaven' and when was it published?
The novel "Seven Winters in Quillhaven" was written by **Melanie Bolt** and was published in **2019**. 

It's a historical romance set in 18th-century Scotland.

The first answer is correct. The second comes in the same tone, the same format and with the same confidence, but there is no novel called "Seven Winters in Quillhaven". The author's name, the year and the genre are all made up by the model. In my attempts with other invented titles the model used the names of real authors, and even produced Goodreads links for the invented books.

That is the most dangerous side of hallucination: the right and the wrong answer look the same. You can't tell which one is invented from the shape of the answer.

Why Do Hallucinations Happen?

Go back to the first post of the series: a large language model predicts the next token. Training teaches it to produce "plausible" text, not "true" text. Most of the time the two overlap, because true information is repeated often in the training data. The answer to the "Dune" question appears on countless pages, so the model completes it with confidence. For "Quillhaven" it has nothing; but the continuation of "Who wrote the novel X" is always a name and a year, so it completes the pattern.

A few factors make it worse. The model doesn't know anything after its training data ends; in the tool calling post we saw qwen3:4b take 2026 for "the distant future". It has never seen your company's private information; in the RAG post it invented a 30-day return window. Small models carry less knowledge than big ones. And a model doesn't know the limits of its own knowledge in a calibrated way; saying "I'm not sure" hasn't always been a rewarded behaviour in training.

Let's also correct a misconception: temperature: 0 doesn't prevent hallucination. Every example in this post ran with temperature 0. Temperature 0 only picks the most likely token; if the most likely continuation is wrong, you repeat the same mistake every time.

Grounding and Citing Sources

You can't remove hallucination entirely, but you can narrow its room. The first way is the RAG we saw in the previous post: the model produces the answer from the text you put in front of it, not from its own knowledge. The second is asking the model to cite its sources. For that I moved the RAG code to TypeScript and made a small module:

import ollama from 'ollama';
import { readFileSync } from 'node:fs';

export interface Chunk {
  id: number;
  text: string;
  vector: number[];
}

const dot = (a: number[], b: number[]) => a.reduce((sum, x, i) => sum + x * b[i], 0);

// Same chunking as the C# version: one chunk per paragraph, with the section title
export async function buildIndex(path: string): Promise<Chunk[]> {
  const texts: string[] = [];
  let section = '';
  for (const line of readFileSync(path, 'utf8').split('\n')) {
    if (line.startsWith('# ')) {
      section = line.slice(2);
    } else if (line.trim()) {
      texts.push(`${section}: ${line}`);
    }
  }
  const { embeddings } = await ollama.embed({
    model: 'nomic-embed-text',
    input: texts.map((t) => `search_document: ${t}`),
  });
  return texts.map((text, i) => ({ id: i + 1, text, vector: embeddings[i] }));
}

export async function retrieve(index: Chunk[], question: string, topK = 2): Promise<Chunk[]> {
  const { embeddings } = await ollama.embed({ model: 'nomic-embed-text', input: [`search_query: ${question}`] });
  return [...index].sort((a, b) => dot(embeddings[0], b.vector) - dot(embeddings[0], a.vector)).slice(0, topK);
}

Now we ask for the answer with structured output. We put a number in front of every chunk and ask the model to return, as JSON, the answer, the numbers of the sources it used and whether it found the answer at all. That way "I don't know" is no longer a free-form sentence but a readable field: found: false.

import ollama from 'ollama';
import { z } from 'zod';
import { buildIndex, retrieve, type Chunk } from './rag.js';

// The answer must say whether it was found and which sources it used
const Answer = z.object({
  found: z.boolean(),
  answer: z.string(),
  source_ids: z.array(z.number().int()),
});
type Answer = z.infer<typeof Answer>;

export async function askGrounded(index: Chunk[], question: string): Promise<{ result: Answer; chunks: Chunk[] }> {
  const chunks = await retrieve(index, question);
  const context = chunks.map((c) => `[${c.id}] ${c.text}`).join('\n');
  const response = await ollama.chat({
    model: 'gemma3:4b',
    format: z.toJSONSchema(Answer),
    options: { temperature: 0, seed: 42 },
    messages: [
      {
        role: 'system',
        content: [
          'You are the Bookstore support assistant.',
          'Answer using ONLY the numbered sources inside <context>.',
          'Put the numbers of the sources you used in source_ids.',
          'If the sources do not contain the answer, set found to false, answer "I don\'t know" and leave source_ids empty.',
          'Reply as JSON: {"found": boolean, "answer": string, "source_ids": number[]}',
        ].join('\n'),
      },
      { role: 'user', content: `<context>\n${context}\n</context>\n\nQuestion: ${question}` },
    ],
  });
  return { result: Answer.parse(JSON.parse(response.message.content)), chunks };
}

// Cheap checks in code: do the cited sources exist, and do the numbers in the answer appear in them?
export function verify(result: Answer, chunks: Chunk[]): string[] {
  const problems: string[] = [];
  const cited = chunks.filter((c) => result.source_ids.includes(c.id));
  for (const id of result.source_ids) {
    if (!chunks.some((c) => c.id === id)) {
      problems.push(`source ${id} was not in the context`);
    }
  }
  if (result.found && cited.length === 0) {
    problems.push('answer has no valid source');
  }
  const citedText = cited.map((c) => c.text).join(' ');
  // Remove citation markers like [3] first, they are not facts
  const answerText = result.answer.replace(/\[\d+\]/g, '');
  for (const num of answerText.match(/\d+(\.\d+)?/g) ?? []) {
    if (!citedText.includes(num)) {
      problems.push(`number ${num} is not in the cited sources`);
    }
  }
  return problems;
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const index = await buildIndex('policies.md');
  for (const question of process.argv.slice(2)) {
    const { result, chunks } = await askGrounded(index, question);
    console.log(`> ${question}`);
    console.log(`  context: ${chunks.map((c) => c.id).join(', ')}`);
    console.log(`  ${JSON.stringify(result)}`);
    const problems = verify(result, chunks);
    console.log(problems.length ? `  CHECK FAILED: ${problems.join('; ')}` : '  check ok');
  }
}

The verify function does two cheap checks that don't depend on the model. Are the cited source numbers really among the chunks we gave the model? Do the numbers in the answer appear in the text of the cited sources? Numbers such as return windows, prices and dates are the most expensive hallucinations for a customer, and they are easy to check.

Verifying the Answer in Code

$ npx tsx grounded.ts "How many days do I have to return a book?" "How many points do I need for a 10 USD discount?" "Can I return a book after 30 days?" "Do you ship to Germany?"
> How many days do I have to return a book?
  context: 3, 4
  {"found":true,"answer":"You can return a book within 21 days of delivery [3].","source_ids":[3]}
  check ok
> How many points do I need for a 10 USD discount?
  context: 6, 1
  {"found":true,"answer":"You need 100 points for a 5 USD discount.","source_ids":[6]}
  check ok
> Can I return a book after 30 days?
  context: 3, 4
  {"found":false,"answer":"I don't know","source_ids":[]}
  check ok
> Do you ship to Germany?
  context: 1, 2
  {"found":false,"answer":"I don't know","source_ids":[]}
  check ok

The first answer cites its source both in source_ids and as [3] in the text, and the number 21 appears in chunk 3. On the first run this line printed CHECK FAILED: number 3 is not in the cited sources: my check had taken the model's [3] marker for a number. That's why I added the replace line in the code. A small moment showing that verification code can be wrong too, and needs testing as well.

The second answer is more instructive. Every check passed: the source exists, and the numbers 100 and 5 appear in it. But the question was about a 10 USD discount; the right answer is 200 points. Instead of doing the math, the model repeated the sentence from the source and didn't answer the question. Source verification catches "does this answer contradict what the source says?", not "does this answer actually answer the question?".

The third answer is a problem too, in the opposite direction. The policy says returns are possible within 21 days; the answer to "Can I return a book after 30 days?" is clearly "no". The model said found: false. That isn't a hallucination, it's over-caution; but for the user it's a wrong answer as well. The Germany question went exactly as we wanted.

The number check has one more limit: a correct answer that repeats a number from the user's question ("No, you cannot return a book after 30 days...") would also trip it, because 30 doesn't appear in the source. Use the check as an alarm, not as a judge.

Golden Set: Small but Real Measurement

You can't decide that a prompt is "better" by looking at individual examples. A change can fix one question and break two others. The fix is a fixed set of questions whose right answers you know (a golden set), run in full after every change. Think of it as the LLM-world counterpart of unit tests: not a proof of correctness, but a net that tells you when you've broken something.

I prepared a set of ten questions: the answers to seven are in the policy (some need a small inference), three deliberately ask about things the document doesn't cover. We run the same set with three approaches: no context at all, plain RAG, and the rule-based RAG with sources from the previous section.

import ollama from 'ollama';
import { buildIndex, retrieve, type Chunk } from './rag.js';
import { askGrounded } from './grounded.js';

interface GoldenCase {
  question: string;
  expect: RegExp | 'unknown';
}

// The golden set: questions with answers we know to be right
const golden: GoldenCase[] = [
  { question: 'How many days do I have to return a book?', expect: /21/ },
  { question: 'How much does express shipping cost?', expect: /9\.90/ },
  { question: 'Is shipping free for a 50 USD order?', expect: /free|yes/i },
  { question: 'How long are gift cards valid?', expect: /3 years|three years/i },
  { question: 'Can I exchange a gift card for cash?', expect: /\b(no|cannot|can't|not)\b/i },
  { question: 'Can I return a book after 30 days?', expect: /\b(no|cannot|21)\b/i },
  { question: 'How many points do I need for a 10 USD discount?', expect: /200/ },
  { question: 'Do you ship to Germany?', expect: 'unknown' },
  { question: 'Do you sell audiobooks?', expect: 'unknown' },
  { question: 'What are the opening hours of the Ankara store?', expect: 'unknown' },
];

// Free-text refusals come in many shapes; this list grew after the first run
const saysUnknown = (text: string) =>
  /don't know|do not know|not sure|no information|do not have information|doesn['’]t (cover|include)/i.test(text);

type Variant = (index: Chunk[], question: string) => Promise<{ text: string; unknown: boolean }>;

async function chat(system: string, user: string): Promise<string> {
  const response = await ollama.chat({
    model: 'gemma3:4b',
    options: { temperature: 0, seed: 42 },
    messages: [
      { role: 'system', content: system },
      { role: 'user', content: user },
    ],
  });
  return response.message.content.trim();
}

const variants: Record<string, Variant> = {
  'no-rag': async (_index, question) => {
    const text = await chat('You are the Bookstore support assistant.', question);
    return { text, unknown: saysUnknown(text) };
  },
  'rag': async (index, question) => {
    const context = (await retrieve(index, question)).map((c) => c.text).join('\n');
    const text = await chat(
      'You are the Bookstore support assistant. Use the context to answer.',
      `<context>\n${context}\n</context>\n\nQuestion: ${question}`,
    );
    return { text, unknown: saysUnknown(text) };
  },
  'rag+rules+sources': async (index, question) => {
    const { result } = await askGrounded(index, question);
    return { text: result.answer, unknown: !result.found };
  },
};

const index = await buildIndex('policies.md');
for (const [name, run] of Object.entries(variants)) {
  let passed = 0;
  console.log(`\n== ${name}`);
  for (const c of golden) {
    const { text, unknown } = await run(index, c.question);
    const ok = c.expect === 'unknown' ? unknown : !unknown && c.expect.test(text);
    passed += ok ? 1 : 0;
    console.log(`${ok ? 'PASS' : 'FAIL'}  ${c.question}  ->  ${text.replace(/\s+/g, ' ').slice(0, 90)}`);
  }
  console.log(`score: ${passed}/${golden.length}`);
}

Scoring is deliberately simple. For questions with a known answer, the answer must match a regex and the model must not have said "I don't know". For questions whose answer isn't in the document, the model must say "I don't know". In the variant with sources we read "I don't know" from the found field, in the others from the text.

Results: Three Approaches, Ten Questions

On the first run plain RAG scored 6/10, and the three failures looked like this:

FAIL  Do you ship to Germany?  ->  I do not have information about shipping to Germany. Our standard shipping information cov
FAIL  Do you sell audiobooks?  ->  I apologize, but this information doesn’t cover our audiobook offerings.
FAIL  What are the opening hours of the Ankara store?  ->  I apologize, but the provided context doesn't include the opening hours of the Ankara stor
score: 6/10

In all three the model had behaved correctly and said it didn't know; the bug was in my scoring function. saysUnknown only looked for a few patterns like "don't know". This is exactly the fragility of scoring free text with a regex, and it's where the value of the found field from structured output shows. After widening the regex (the code above is that version) and running again:

$ npx tsx eval.ts

== no-rag
FAIL  How many days do I have to return a book?  ->  Okay, let's get that sorted out for you! You have **30 days** to return a book to us. 😊 D
FAIL  How much does express shipping cost?  ->  Okay, let's look at our express shipping costs! It depends on a few things, like the weigh
PASS  Is shipping free for a 50 USD order?  ->  Okay, let's check that for you! Generally, shipping is free on orders of $50 or more. To c
FAIL  How long are gift cards valid?  ->  Okay, let's get you the information on gift card validity! Generally, our gift cards are v
PASS  Can I exchange a gift card for cash?  ->  Okay, let's see about exchanging a gift card! Unfortunately, we generally **cannot** excha
FAIL  Can I return a book after 30 days?  ->  Okay, let’s see about that! Our standard return policy is 30 days from the date of purchas
FAIL  How many points do I need for a 10 USD discount?  ->  Okay, let's figure that out! Currently, you need 100 points for a $10 discount. Do you hav
FAIL  Do you ship to Germany?  ->  Yes, we absolutely do ship to Germany! 😊 We use a reliable shipping partner to get your o
FAIL  Do you sell audiobooks?  ->  Hello there! Thanks for your interest in our bookstore. Yes, we absolutely do sell audiobo
FAIL  What are the opening hours of the Ankara store?  ->  Okay, let’s check the opening hours for our Ankara store! Our Ankara store is currently op
score: 2/10

== rag
PASS  How many days do I have to return a book?  ->  You have 21 days to return a book if it’s unread and in its original condition.
PASS  How much does express shipping cost?  ->  Express shipping costs 9.90 USD.
PASS  Is shipping free for a 50 USD order?  ->  Yes, shipping is free for orders over 40 USD. Since your order is 50 USD, you qualify for 
PASS  How long are gift cards valid?  ->  Gift cards are valid for 3 years from the date of purchase.
PASS  Can I exchange a gift card for cash?  ->  No, gift cards cannot be exchanged for cash.
PASS  Can I return a book after 30 days?  ->  No, you cannot return a book after 30 days. Our return policy allows returns within 21 day
FAIL  How many points do I need for a 10 USD discount?  ->  You need 100 points for a 5 USD discount.
PASS  Do you ship to Germany?  ->  I do not have information about shipping to Germany. Our standard shipping information cov
PASS  Do you sell audiobooks?  ->  I apologize, but this information doesn’t cover our audiobook offerings.
PASS  What are the opening hours of the Ankara store?  ->  I apologize, but the provided context doesn't include the opening hours of the Ankara stor
score: 9/10

== rag+rules+sources
PASS  How many days do I have to return a book?  ->  You can return a book within 21 days of delivery [3].
PASS  How much does express shipping cost?  ->  Express shipping costs 9.90 USD
PASS  Is shipping free for a 50 USD order?  ->  Yes, shipping is free for orders over 40 USD.
PASS  How long are gift cards valid?  ->  Gift cards are valid for 3 years from the date of purchase.
FAIL  Can I exchange a gift card for cash?  ->  I don't know
FAIL  Can I return a book after 30 days?  ->  I don't know
FAIL  How many points do I need for a 10 USD discount?  ->  You need 100 points for a 5 USD discount.
PASS  Do you ship to Germany?  ->  I don't know
PASS  Do you sell audiobooks?  ->  I don't know
PASS  What are the opening hours of the Ankara store?  ->  I don't know
score: 7/10

The model without context scored 2/10, and both of its passes are luck: "free shipping on orders of $50 or more" isn't our rule, it's a generic guess that happened to match the regex. It happily invented shipping to Germany, an audiobook catalog and opening hours for the Ankara store. RAG changes the picture completely.

The really surprising result is in the last two sections. The variant with stricter rules and mandatory sources scored lower than plain RAG: 7 against 9. Both have zero invented answers; the difference is that the rule-based variant said "I don't know" on two questions that need one step of inference from the context. So the rules didn't reduce hallucination, they reduced helpfulness. We saw that thanks to the golden set, not by looking at individual examples. The 10 USD question fails in all three variants: questions that need arithmetic are a separate weakness for a model of this size.

Don't trust these numbers too much: ten questions is a small set, and while other jobs share the same Ollama, the text may not repeat word for word even at temperature 0. A real set would have hundreds of questions, samples of real user questions, and several runs per question. But the method is the same: a fixed set, fixed scoring, run again after every change.

Common Pitfalls

The first is taking a confident tone for correctness. The made-up author in this post and the real Frank Herbert answer came in exactly the same format. Tone is not a signal.

The second is treating temperature as a hallucination setting. Temperature 0 makes answers repeatable, not correct.

The third is taking citations for verification. The model can make up a number in source_ids too; check in code that the number really is one of the chunks you provided and that the answer is consistent with that chunk.

The fourth is measuring only hallucination. An assistant that invents nothing but says "I don't know" to everything is useless too. Put questions in the golden set whose answers are in the document but need a small inference.

The fifth is not testing the scoring code. Our first scoring unfairly cost plain RAG three questions. Check the scoring function by hand against a few sample answers too, and where you can, score structured output fields instead of free text.

The sixth is writing the golden set once and forgetting it. Every real question answered wrongly in production is a candidate test case for the set.

A Small Experiment

★☆☆ Give hallucination.ts three book titles you made up. In how many does the model say "I don't know", and in how many does it invent an author? Try the same questions with qwen3:4b and note the difference.

★★☆ Add five questions to the golden set: two that aren't in the document and three that need inference (e.g. "I ordered at 3 pm with express shipping, when will it arrive?"). How do the scores of the three variants change?

★★☆ Add the line "If the answer can be inferred from the sources with simple reasoning, answer it." to the system prompt in grounded.ts. Does the rule-based variant's score go above 7, or do invented answers start to appear?

★★★ Add a fourth variant to eval.ts: a judge (LLM-as-judge) that asks gemma3:4b in a second request "Does this answer follow from these sources? Reply yes or no.". Does the judge catch the 10 USD mistake that verify missed? How would you measure how reliable the judge itself is?

With that we have covered most of the conceptual foundation: tokens, the context window, sampling, prompts, structured output, tools, embeddings, RAG and now measuring. In the next post we move to the numbers side: we will turn tokens into cost and latency, measure tokens/s on this machine, compare the 4b and 12b models, and talk about how to decide between local and cloud.