Reranking and Query Transformation: Cross-Encoders, Query Rewriting and HyDE

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0, OllamaSharp 5.5.2, Entity Framework Core 10 (Npgsql 10.0.3) and PostgreSQL 17 + pgvector 0.8.7; the models are gemma3:4b, qwen3:4b and nomic-embed-text on Ollama.

Guess First

The hybrid search we built in the previous post put the right chunk first for 15 of the 30 questions; for 28 questions the right chunk was among the first 10 candidates. So most of the time the right answer is in our hands, just in the wrong position. If we had a small local model, gemma3:4b, read those 10 candidates one by one, asked "how well does this passage answer this question, score 0–10", and re-sorted by score, how far would the number of right chunks in first place rise from 15? And what would it cost per question in seconds?

What I measured: from 15 to 23. The cost is about 3.2 seconds per question. That trade-off is the subject of this post: retrieving wide and returning narrow (reranking), and transforming the query before searching (query rewriting, multi-query, HyDE and self-query). We'll see when each helps and when each hurts, on the same question set.

Bi-encoders versus Cross-encoders

The embedding model we've used so far is a bi-encoder: it turns the question and the chunk into vectors separately, then compares the two vectors. Since chunk vectors can be precomputed and indexed, search over millions of records takes milliseconds. The price: the model never sees the question and the chunk together. The link between "Kargom evde kimse yokken gelirse" (if my parcel arrives when nobody is home) and "adreste kimse bulunamazsa" (if nobody is found at the address) is left to two separate compressions happening to land close.

A cross-encoder reads the question and the chunk as a single input and produces a relevance score directly. It is far more accurate, but nothing can be precomputed: the model runs again for every (question, chunk) pair. That is why we chain them: fetch 20–150 candidates with a cheap bi-encoder and keyword search, sort them with an expensive cross-encoder, and put the best 3–10 into the prompt.

An example of a multilingual cross-encoder is BAAI's (Beijing Academy of Artificial Intelligence) bge-reranker-v2-m3: built on bge-m3, about 0.6 billion parameters, and according to its model card it has strong multilingual capabilities; its raw score can be mapped to 0–1 with a sigmoid. Because Ollama has no reranker endpoint and we don't download new models in this series, I didn't run a cross-encoder in this post. Instead we'll have a chat model do the same job: reranking with an LLM (Large Language Model), i.e. LLM-as-reranker.

One Reranker Interface, Two Implementations

The pipeline doesn't need to know which kind of reranker it uses:

public sealed record Candidate(string Id, string Section, string Content);
public sealed record Scored(Candidate Candidate, double Score);

public interface IReranker
{
    Task<List<Scored>> RerankAsync(string query, IReadOnlyList<Candidate> candidates, CancellationToken ct = default);
}

// LLM-as-reranker: a 0-10 score for every (question, chunk) pair (pointwise)
public sealed class LlmReranker(IChatClient chat) : IReranker
{
    const string Instructions = """
        You are a relevance judge for a Turkish bookstore help center.
        Given a question and a passage, reply with a single integer from 0 to 10:
        10 = the passage directly answers the question, 0 = the passage is unrelated.
        Reply with the number only.
        """;

    public async Task<List<Scored>> RerankAsync(string query, IReadOnlyList<Candidate> candidates, CancellationToken ct = default)
    {
        List<Scored> result = [];
        foreach (Candidate c in candidates)
        {
            ChatResponse response = await chat.GetResponseAsync(
            [
                new ChatMessage(ChatRole.System, Instructions),
                new ChatMessage(ChatRole.User, $"Question: {query}\n\nPassage:\n{c.Content}\n\nScore:")
            ], new ChatOptions { Temperature = 0, Seed = 42, MaxOutputTokens = 4 }, ct);

            double score = double.TryParse(response.Text.Trim(), NumberStyles.Float, CultureInfo.InvariantCulture, out double s) ? s : 0;
            result.Add(new Scored(c, score));
        }
        // On ties, keep the retrieval order: OrderBy is stable
        return result.OrderByDescending(s => s.Score).ToList();
    }
}

MaxOutputTokens = 4 matters: if the model is allowed to write an explanation, each pair takes seconds. A stable sort matters too; small models hand out 10 very often, and the hybrid order should survive ties.

For the cross-encoder side, here is an implementation that calls the /rerank endpoint of Hugging Face's Text Embeddings Inference (TEI) server. This code compiles but was not run for this post:

// Cross-encoder: the Hugging Face Text Embeddings Inference (TEI) /rerank endpoint
public sealed class TeiReranker(HttpClient http) : IReranker
{
    sealed record RerankRequest(string Query, IReadOnlyList<string> Texts, bool RawScores);
    sealed record RerankItem(int Index, double Score);

    // TEI expects snake_case: raw_scores
    static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web) { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower };

    public async Task<List<Scored>> RerankAsync(string query, IReadOnlyList<Candidate> candidates, CancellationToken ct = default)
    {
        HttpResponseMessage response = await http.PostAsJsonAsync("/rerank",
            new RerankRequest(query, candidates.Select(c => c.Content).ToList(), RawScores: false), Json, ct);
        response.EnsureSuccessStatusCode();
        List<RerankItem> items = (await response.Content.ReadFromJsonAsync<List<RerankItem>>(Json, ct))!;
        return items.Select(i => new Scored(candidates[i.Index], i.Score))
            .OrderByDescending(s => s.Score)
            .ToList();
    }
}

The difference shows here: TEI scores all candidates as a batch in one HTTP (Hypertext Transfer Protocol) request, while our LLM reranker makes one model call per candidate.

Measurement: 10 Candidates, 300 Calls

For each question I took 10 candidates from the previous post's hybrid search, scored them with gemma3:4b and printed the right chunk's position before and after:

$ dotnet run -- rerank-eval 10 gemma3:4b
q04 rank 4 -> 1  top: iade-politikasi#2=8 sadakat-programi#0=8 iade-politikasi#0=8
q05 rank 8 -> 2  top: iade-politikasi#0=10 iade-politikasi#3=9 kurumsal-satis-sozlesmesi#3=7
q08 rank 6 -> 1  top: kargo-ve-teslimat#0=10 sadakat-programi#3=8 iade-politikasi#0=6
q12 rank 2 -> 2  top: kargo-ve-teslimat#2=0 kargo-ve-teslimat#5=0 kargo-ve-teslimat#3=0
q14 rank 2 -> 5  top: iade-politikasi#1=8 iade-politikasi#2=6 odeme-secenekleri#1=3
q16 rank 1 -> 4  top: sadakat-programi#2=3 sadakat-programi#0=3 sadakat-programi#3=2
q26 rank 2 -> 1  top: urun-katalogu#7=10 urun-katalogu#11=10 urun-katalogu#0=10
q30 rank 4 -> 1  top: urun-katalogu#9=10 sss#2=0 sss#3=0
...
in pool@10: 28/30 | recall@1 15 -> 23 | recall@3 24 -> 25 | rerank time 96,7s (3222 ms/question)

The gain is real: recall@1 rose from 15 to 23. The children's book in q30, which hybrid pushed to fourth in the previous post, and BKS-20114 in q26, which it pushed to second, are now first. In q05 ("the book arrived torn") the right chunk rose from eighth to second. These are exactly the cases a bi-encoder can't see, the ones that require reading the question and the text together.

The losses are real too. In q14 ("can I turn my gift card into cash?") the right chunk dropped from 2 to 5; the model gave a chunk from the return policy an 8. In q16 the right chunk, first before, fell to fourth. In q12 the model gave 0 to all ten candidates; when I repeated the raw call on its own the answer was again '0', so it isn't a parse error: the model simply couldn't connect "what happens if my parcel arrives when nobody is home?" to the "undeliverable parcels" section. A 4-billion-parameter general-purpose model is not a cross-encoder trained for this job; the scores are coarse (three candidates scored 10 in q26) and sometimes plain wrong.

Also note that recall@3 only went from 24 to 25. If you put three chunks into the prompt, what the reranker really buys you isn't "finding the right chunk" but moving it to the top. That matters for the context budget we discuss in the next post: you can get the same accuracy with fewer chunks.

Latency and Cost

3.2 seconds per question, about 320 ms times 10 pairs. This was measured on a developer machine where other jobs share the same Ollama; on a GPU (Graphics Processing Unit) server in production it would be much faster, but the ratio doesn't change: double the candidates and the reranker time doubles. With cloud APIs (Application Programming Interface) the same calculation is done in tokens: 10 candidates times about 150 tokens per chunk means 1,500 extra input tokens per question. You reduce it by parallelising the calls (without exceeding the provider's rate limit) or by switching to a real cross-encoder.

Transforming the Query (1): Rewriting Follow-ups in a Conversation

In a chat, users don't write full sentences. After "Annem için hediye kartı almak istiyorum" (I want to buy a gift card for my mother) they ask "Süresi ne kadar?" (how long is it valid?). Embed that follow-up as is and the search returns everything closest to the word "duration". The fix is to turn the question into a standalone query using the conversation history:

// Turn a follow-up in a conversation into a standalone search query
public async Task<string> RewriteAsync(IEnumerable<ChatMessage> history, string followUp)
{
    string transcript = string.Join("\n", history.Select(m => $"{m.Role}: {m.Text}"));
    ChatResponse response = await chat.GetResponseAsync(
    [
        new ChatMessage(ChatRole.System, """
            Rewrite the user's last message as a standalone search query in Turkish.
            Use the conversation only to resolve references. Do not answer. Return only the query.
            """),
        new ChatMessage(ChatRole.User, $"Conversation:\n{transcript}\n\nLast message: {followUp}")
    ], Deterministic);
    return response.Text.Trim();
}

$ dotnet run -- rewrite
follow-up: Süresi ne kadar?  => rewritten (553 ms): hediye kartlarının süresi ne kadar?
  raw      : hediye-karti#1, iade-politikasi#0, sadakat-programi#2
  rewritten: hediye-karti#1, hediye-karti#0, hediye-karti#2
follow-up: Nakde çevirebilir mi?  => rewritten (839 ms): hediye kartını nakde çevirebilirim mi?
  raw      : hediye-karti#2, sadakat-programi#1, sadakat-programi#3
  rewritten: hediye-karti#2, hediye-karti#0, hediye-karti#1
follow-up: Puan kazandırır mı?  => rewritten (575 ms): hediye kartları puan kazandırır mı
  raw      : sadakat-programi#0, sadakat-programi#2, sadakat-programi#1
  rewritten: hediye-karti#0, sadakat-programi#0, hediye-karti#2

The result is more nuanced than I expected. In the first two follow-ups the raw query also puts the right chunk first, but unrelated chunks such as the return window and points expiry also make the top three; if those reach the prompt, the model may answer the "how long" question from the wrong section. The rewritten query fills the top three with gift-card chunks. The third follow-up went the other way: the answer is the sentence "gift card purchases don't earn points" in the "earning points" section. The raw query got it first; the rewrite put "gift card" up front and pushed it to second. Rewriting ties the query to the right topic, but makes questions whose answer lives in another topic's document a bit harder. Also note the grammar error, "çevirebilirim mi". Fine for search, but don't show that text to the user.

Transforming the Query (2): Multi-query and HyDE

Multi-query has the model rewrite the question three different ways, searches with each, and merges the results with RRF (Reciprocal Rank Fusion). HyDE (Hypothetical Document Embeddings) is the idea from Gao et al.'s 2022 paper: don't embed the question, embed a hypothetical answer passage the model writes for it. The passage's content may be wrong; what matters is that it resembles the documents.

// HyDE: write a hypothetical passage that answers the question, then embed it
public async Task<string> HypotheticalDocumentAsync(string question)
{
    ChatResponse response = await chat.GetResponseAsync(
    [
        new ChatMessage(ChatRole.System, "Write a short passage (2-3 sentences, in Turkish) from an online bookstore's help page that answers the question. Do not add a title."),
        new ChatMessage(ChatRole.User, question)
    ], new ChatOptions { Temperature = 0, Seed = 42, MaxOutputTokens = 160 });
    return response.Text.Trim();
}

// Search side: the hypothetical passage is embedded as a document, not as a query
string passage = await qt.HypotheticalDocumentAsync(q.Question);
transformed = await retriever.VectorAsync(await embedder.GenerateVectorAsync($"search_document: {passage}"), 5);

$ dotnet run -- hyde-eval
q01 + -> +  Kitapları satın aldıktan sonra 30 gün içinde iade edebilirsiniz. ...
q04 - -> +  Ödeme işleminiz tamamlandıktan sonra, ödeme yöntemine bağlı olarak genellikle 1-3 iş günü içinde hesabınıza geri yatar. ...
q08 - -> +  Siparişlerinizde 500 TL ve üzeri alışverişlerde kargo ücreti ödemenize gerek kalmıyor. ...
q11 - -> +  Evet, Almanya'ya kitap göndermekteyiz. Ancak, kargo ücretleri ve gümrük vergileri gibi faktörler nedeniyle ...
q26 - -> -  BKS-20114 koduyla ilgili kitap, güncel stok durumuna göre değişiklik gösterebilir. ...
hyde-eval recall@5: 25/30 -> 28/30, LLM time 94,0s (3133 ms/question)

$ dotnet run -- multi-eval
q05 - -> -  Yırtık kitap nasıl onarılır? | Kitap sayfaları yırtıldı çözüm nedir? | Yırtılmış kitap tamir yöntemleri
q08 - -> +  Hangi alışveriş tutarı için kargo ücretsiz? | Kargo ücretsiz olan minimum alışveriş tutarı ne kadar? | ...
multi-eval recall@5: 26/30 -> 27/30, LLM time 79,0s (2633 ms/question)

HyDE's result is the most instructive output in this post. With vector search alone, recall@5 rose from 25 to 28: text that looks like a document rather than a question really does land closer to document embeddings. But look inside the passages: a 30-day return window (it's 21), free shipping from 500 TL (it's 400), "yes, we ship books to Germany" (we don't). The model made up what it didn't know, and the fabrication improved retrieval. That is why a HyDE passage is used only for search; it never goes into the prompt, the logs or to the user. On the SKU (Stock Keeping Unit) question q26 HyDE is helpless too: the model knows nothing about the code, so it wrote a generic "visit the product page" passage.

Multi-query gained only one question over hybrid (26 to 27) and produced a nice example of query drift in q05: "the book arrived torn, what should I do?" became "how do I repair a torn book?". The user wants a return; the model is searching for repair tips. One question in exchange for three extra searches and an LLM call is not a good trade on this set.

Self-query: Extracting a Metadata Filter from the Question

The last technique extracts a metadata filter from the question and limits the search to that category. We ask the model for a search plan with structured output:

public enum Category { Iade, Kargo, Odeme, HediyeKarti, Sadakat, Siparis, Urun, Kurumsal }

public sealed record SearchPlan(
    [property: Description("The search query, without filter words")] string Query,
    [property: Description("Topic filter, or null if unclear")] Category? Category);

// Self-query: extract a metadata filter from the question
public async Task<SearchPlan> SelfQueryAsync(string question)
{
    ChatResponse<SearchPlan> response = await chat.GetResponseAsync<SearchPlan>(
    [
        new ChatMessage(ChatRole.System, """
            Turn the customer's question into a search plan for a bookstore help center.
            Categories: Iade (returns), Kargo (shipping), Odeme (payment, installments), HediyeKarti (gift cards),
            Sadakat (loyalty points), Siparis (orders, store pickup, customer service), Urun (product catalog), Kurumsal (corporate sales).
            """),
        new ChatMessage(ChatRole.User, question)
    ], new ChatOptions { Temperature = 0, Seed = 42, AdditionalProperties = new() { ["think"] = false } });
    return response.Result;
}

$ dotnet run -- selfquery ...
Kitap yırtık geldi, ne yapmalıyım?
  plan (11595 ms): query="Kitap yırtık geldi" category=Iade
  top3: iade-politikasi#3, iade-politikasi#4, iade-politikasi#0
Hediye kartımı nakde çevirebilir miyim?
  plan (10234 ms): query="Hediye kartımı nakde çevirebilir miyim?" category=HediyeKarti
  top3: hediye-karti#0, hediye-karti#2, hediye-karti#1
Kargom evde kimse yokken gelirse ne olur?
  plan (11260 ms): query="Kargom evde kimse yokken gelirse ne olur?" category=Kargo
  top3: kargo-ve-teslimat#3, kargo-ve-teslimat#1, kargo-ve-teslimat#5
Okulumuz için 120 adet kitap alırsak ne kadar indirim alırız?
  plan (13310 ms): query="Okulumuz için 120 adet kitap alırsak ne kadar indirim alırız?" category=Sadakat
  top3: sadakat-programi#1, sadakat-programi#3, sadakat-programi#0
Hediye kartıyla aldığım kitabı iade edersem param nereye yatar?
  plan (15863 ms): query="Hediye kartıyla aldığım kitabı iade edersem param nereye yatar?" category=Iade
  top3: iade-politikasi#1, iade-politikasi#2, iade-politikasi#0

The category guess is right for four of five, and in those the filter gathered the top three into a single document. The fifth shows self-query's real risk: the model put "if our school buys 120 books, what discount do we get?" into the Sadakat (loyalty) category. Because the filter narrows the search space, the corporate sales contract never entered the search; all three results come from the points programme. In hybrid search this question was already first. A wrong filter is worse than a wrong ranking: a reranker can fix ranking errors, but no later step can bring back a document that was filtered out. Latency isn't negligible either: these structured-output calls took 10–16 seconds per question while other jobs shared the same Ollama. Use self-query only where the category is genuinely clear, and so that a low-confidence plan doesn't apply a filter at all (e.g. by allowing a null category and falling back to an unfiltered search when the filtered result is empty).

Production Notes

Every technique is a separate LLM call, and they add up. In this post's measurements rewriting added about 0.5–0.8 seconds, HyDE 3.1 seconds, multi-query 2.6 seconds and the LLM reranker 3.2 seconds. Chain them all and the user waits 10 seconds before generation even starts. First measure which failure class a technique fixes, then pay only for that: if you have a chat interface, rewriting is close to mandatory; if not, it's unnecessary.

Store reranker scores with your logs. In production the answer to "why did it answer wrong?" is usually "the right chunk was in the pool but the reranker pushed it to fifth" or "it was never in the pool". The fixes differ: the first concerns the reranker, the second retrieval. Pool size (10 here) is the reranker's ceiling; q10 and q11 weren't in the pool, so no reranker can save them.

Don't forget caching for the latency budget. The score of a question-chunk pair doesn't change as long as the chunk doesn't; a cache keyed on (model, question hash, chunk content_hash) can remove a large share of reranker calls. For frequently asked questions the difference is big.

Common Pitfalls

Adding an extra LLM call to every request and doubling latency. Stacking the techniques above because "they're all good" is the most common mistake. Don't add a call to the pipeline without a gain you've measured.

Using the reranker score as a threshold without calibrating it. Our LLM reranker gave 0 to everything, including the right chunk, in q12, and a 3 to the right chunk in q16. A rule like "say I don't know below 5" would have misrouted both. If you want a threshold, first look at the score distribution on your own question set; cross-encoders' sigmoid scores are also calibrated differently from model to model.

Treating the HyDE passage as real information. The passage exists only to produce a search vector. If it reaches the prompt, the model will read its own made-up "500 TL" as a source.

Not checking for query drift. Multi-query and rewriting can change intent ("repair" instead of "return"). Always include the original question in the search, as the first list in multi-query.

A Small Experiment

★☆☆ Run rerank-eval with 5 and 20 candidates instead of 10. How do recall@1 and per-question time change? After how many candidates does the gain stop?

★★☆ Make LlmReranker listwise instead of pointwise: pass the 10 candidates numbered in a single call and ask for an ordered list of ids with structured output. Compare recall@1 and total time with the pointwise version on the 30 questions.

★★☆ Merge the HyDE vector list with the original question's vector list using RRF (as an extra list, not on its own). Does the effect of the made-up 30-day passage in q01 shrink?

★★★ If you have a machine that can run a cross-encoder with TEI in Docker, plug in TeiReranker and compare recall@1 and latency with the LLM reranker on the same 30 questions. Plot the score distribution and try picking a threshold for "I don't know".

The chunks going into the prompt are now fewer and more accurate. But how we put them into the prompt also changes the answer: in what order, with which ids, within how many tokens, and not at all when the score is low. In the next post we pack chunks into id-tagged blocks, get an answer plus a source list with structured output, catch made-up citation ids in code, and measure how hard it is to say "I don't know" before calling the model.