Reranking with a classifier instead of a chat model

29 September 2026 · The Stuga team

TL;DR

A search in Stuga returns 24 candidate passages, and a judge reorders them so that the model reads the best eight. We compared 23 judges on two public benchmarks whose questions and answers were marked by people. Every judge reordered the same 24 candidates and saw each one’s title, heading path and 1,200 characters of its text. The main measure is Hit@1: how often the first passage is one that answers the question.

Questions that ask for facts. MIRACL has questions over Wikipedia in six languages; we took 240 and reordered the 24 candidates MMTEB publishes for each. Without a judge, a relevant passage came first for 36% of questions. TypeSafe’s Jev, a System One classifier, raised that to 77%, in 131 ms for $0.37 per 1,000 searches, and put a relevant passage among the eight the model reads for 99% of questions, against 80%. The best LLM judge, Claude Opus 5.5, reached 81%. The 4.2 points between them are within this test’s margin of error (−2.9 to 10.8 points, adjusted for the 23 comparisons with Jev), so the data say neither that one is better nor that they are equal. Claude Opus 5.5 took 27 times as long and cost 100 times as much. Among the newest models tested, Claude Sonnet 5.5 reached 80% in 2.0 s, at $20 per 1,000 searches; GPT-6.1 Sol reached 77% in 2.2 s, at $14.

Questions that need reasoning. BRIGHT’s 117 StackOverflow questions are answered by documentation pages that may share few words with the question. Search found such a page among the 24 candidates for only 52% of questions, and its own order put one first for 19% of all 117. The passages are long, typically 4,000 characters, and of the judges reading the first 1,200 of each, as Stuga’s did until this study, none clearly beat search’s order. Reading instead the 1,200 characters that best match the question, 21 of 24 judges rose, 7 of them clearly: over all 117 questions, Claude Opus 5.5 then put a gold passage first for 32% and Jev for 25%. We designed that reading after seeing the first results and measured it on the same questions, so it is exploratory; Stuga’s judges now read that part (Section 4.3).

In this setup, Jev was a fast, cheap judge for questions that ask for facts, and no judge we tested clearly beat it. On long passages, what a judge reads of each mattered as much as which judge it is.

We ran this test to decide what Stuga’s Ask and its tools for agents show an AI before it answers: which judge orders the candidates, and which part of each passage it reads. Section 6 shows the result in a workspace.

1. Introduction

Stuga answers questions from a workspace’s documents (Ask) and serves the same retrieval to agents over MCP. Retrieval returns 24 candidate passages; a judge reorders them, and the model reads the first eight. The judge decides what the model sees, so we measured the options on public benchmarks, in the shape Stuga runs them. The primary comparison is Jev, which Stuga’s Reranking setting uses, against the LLM judges Stuga falls back to. The contribution is an empirical comparison through the interfaces a product calls, not a new ranker:

• quality, latency and cost for 23 rerankers of three kinds, hosted and local, on the same candidates;
• a fact-seeking benchmark in six languages and a reasoning-heavy one, the second under both of Stuga’s first stages;
• on long passages, what the judge should read: each passage’s first 1,200 characters or the 1,200 that best match the question;
• every call’s result and the code to reproduce or extend it.

Five questions guided the study:

1. How often does each judge put a relevant passage first, and among the eight?
2. At what latency and cost?
3. How do LLM judges fail?
4. Can the judge run on the machine that runs Stuga?
5. What should a judge read of a long passage?

2. Background

2.1 Retrieval in Stuga

Stuga splits each document at its headings: every section is a passage, split further only past 12,000 characters, and a document without headings is cut into pieces of about 6,000. Each passage gets one vector, from its heading path and text. A question runs two searches in one SQL statement over these passages: BM25 [5] through pg_search and vector similarity through pgvector, 96 passages each. Reciprocal rank fusion [6] (k = 60) merges the lists and keeps 24 (Figure 2).

Three terms recur below. The first stage is the search that picks the 24 candidates; its order, with no judge, is the baseline every judge is measured against. A relevant passage is one the benchmark’s annotators marked as answering the question; BRIGHT calls it gold. Hit@1 is the share of questions whose first passage is relevant.

A first stage finds the neighbourhood, not the house: a passage about the question’s subject can outrank the one that answers it. On MIRACL’s candidate lists a relevant passage came first for 36% of questions and was among the first eight for 80%. Figure 3 follows three of them.

2.2 Three kinds of judge

Table 1. The judges compared.

An LLM judge scores passages by generating text, the approach RankGPT [4] popularised. A reranker is a cross-encoder in the tradition of monoBERT [3]: the question and one passage pass through the model together and a head outputs a score. Cohere and Amazon do not publish their architectures. A System One classifier answers typed questions about a state with probabilities in a single call. The name follows Kahneman’s fast, intuitive System 1 [1]; a model that reasons before answering is the deliberate System 2.

Stuga’s LLM judge sends this system prompt, with each candidate’s title, heading path and the 1,200 characters of its text that best match the question (Section 4.3):

You are a search relevance judge. Given a user query and numbered
document snippets, rate how well EACH snippet helps answer or act on the query.
Score each 0-10 (10 = directly answers it; 0 = irrelevant). Judge only relevance,
not writing quality. Respond with ONLY a JSON array of {"i":<number>,"score":<0-10>}
for every snippet, no prose.

Its System One judge sends the same 24 passages to Jev [8] in one request:

{
  "model": "jev-latest",
  "state": { "query": "…", "passages": { "p0": { "title": "…", "section": "…", "text": "…" }, "…": {} } },
  "questions": {
    "p0": {
      "type": "noul",
      "instructions": "Does passage p0 help answer the query?",
      "criteria": {
        "true": "The passage states the information the query asks for, or information needed to answer it",
        "false": "The passage is only on a related topic, or is irrelevant to the query"
      }
    }
  }
}

3. Method

3.1 Test sets

We used two public benchmarks whose questions and relevance judgments were made by people, so that neither the questions nor the labels came from us or from a model under test. Both are openly licensed and published with the benchmark.

• MIRACL [13], in MMTEB’s reranking version [14]: questions over Wikipedia written by native speakers, who also judged which passages answer them. We drew 40 questions in each of English, Chinese, Japanese, Korean, Arabic and Thai, the languages of the privacy laws in Stuga’s sample workspaces, with a fixed seed from those with a relevant passage among their first 24 candidates. Passages are CC BY-SA 4.0; MIRACL’s judgments are Apache-2.0.
• BRIGHT [15], StackOverflow subset: 117 real StackOverflow questions, often with code. The gold passages are the documentation pages that accepted answers link to, confirmed by annotators. Finding them takes reasoning: the question describes a problem, the documentation describes a function. CC BY 4.0.

3.2 Candidates

Every ranker reordered the same 24 candidates per question. For MIRACL they are the first 24 of MMTEB’s published list for the question, in its order. Those lists hold a relevant passage in their first 24 for 79% of the six languages’ dev questions, and we drew ours from those.

For BRIGHT we retrieved the candidates as Stuga does, from the question alone, over the subset’s 107,081 passages, as BRIGHT splits them, in Stuga’s two configurations. With embeddings, BM25 and vector search each return 96 passages, fused by reciprocal rank (k = 60), with one vector per passage from OpenAI’s text-embedding-3-large at 1,024 dimensions; without them, BM25’s first 24 stand. A gold passage was among the 24 for 61 of the 117 questions (52%) with embeddings and for 47 (40%) with BM25 alone. The rest are retrieval misses that no judge can fix, so BRIGHT’s scores below count only the questions with a gold passage among the candidates, unless they say all 117. Over all 117, no ranker put a gold passage first for more than 22%.

3.3 Judges

We compared current models across the listed families. OpenAI judges used OpenAI’s API or Bedrock’s OpenAI-compatible endpoint, TypeSafe its own API; the other hosted models ran on Bedrock in us-west-2, on US or global inference profiles. LLM judges received Stuga’s prompt verbatim, at the lowest reasoning each model offers (low for GPT-6.1 Sol, none for GPT-6 Sol) and with up to 8,192 output tokens, as Stuga’s judge sends them. Jev received Stuga’s single request and, separately, one request per passage. Kev and Laya, which serve the same System One API on local hardware, received one passage per request: Kev’s README notes training on states of up to 384 tokens, and Laya, an encoder, reads its state as text, so it received plain text and chose its English or multilingual checkpoint itself. Rerankers scored each question and passage pair.

Every judge saw each candidate’s title, heading path and 1,200 characters of its text: the first 1,200, as Stuga’s judge read them until this study, and on BRIGHT also the 1,200 that best match the question, as it reads them now (Section 4.3). On MIRACL’s short passages the two differ for 1% of candidates, so its results stand for both.

3.4 Measures

• Hit@1, the share of questions with a relevant passage first, the passage the model reads first. Recall@8, the share with one among the eight passages Ask reads. 95% confidence intervals are percentile bootstrap over questions [9]. Rankers are compared on the same questions, by the paired difference and its bootstrap interval; where a ranker is compared with many others, the interval is also Bonferroni-adjusted for their number. A difference whose interval includes zero is within the margin of error: the test cannot tell the two rankers apart, which does not make them equal.
• Latency, the wall-clock time of each search from the client, retries included, reported as median and 95th percentile (for 28 retried GPT-6 Luna calls on MIRACL, the answering attempt only). Each ranker sent one request at a time.
• Cost per 1,000 searches, from measured tokens at list price: the Pi library’s model catalog, with vendor-price overrides for hosted profiles recorded in meta.json, TypeSafe’s announcement for Jev ($0.042 per million input tokens, output free) [8], and AWS’s pricing page for Cohere Rerank 3.5 ($2 per 1,000 queries) [10]. AWS lists no price for Amazon Rerank 1.0. An unusable answer counts toward latency and cost, because Stuga waits for it and pays for it.
• Release date: the vendor’s own, otherwise the model’s first listing on OpenRouter or Hugging Face, each with its source in the repository.

3.5 Environment

The benchmark client and every local model ran on one Mac: Apple M6, 32 GB, macOS 27. Qwen3-Reranker ran as Q8_0 GGUF in llama.cpp 0.5.0, Kev-4B on MLX 0.32 in bfloat16, and Laya 0.3.20 on PyTorch 2.14, one model at a time.

4. Results

4.1 MIRACL: questions that ask for facts

The candidate lists here are MMTEB’s, so this section measures reranking a fixed list; it says nothing about how Stuga’s own retrieval would have ranked MIRACL. Reranking lifted Hit@1 from 36% to at most 81%, Claude Opus 5.5’s. Table 2 gives each ranker’s difference from Jev on the same 240 questions. Claude Opus 5.5 was 4.2 points ahead (0.0 to 8.8, or −2.9 to 10.8 adjusted for the 23 comparisons with Jev). Another 17 rankers, of all three kinds, scored between 73% and 81%, each within the margin of error of Jev even before adjustment. Adjusted, Mistral Large 3, Amazon Nova 2 Lite, Kev-4B and Laya remain clearly below Jev. What separates the leaders is latency and cost (Figure 1): Jev and Cohere Rerank 3.5 answer in 131 ms and 163 ms, the LLM judges among them 945 ms to 12 s. With any of these rankers a relevant passage is among the eight the model reads for 93%–100% of questions, against 80% without a judge.

4.2 By language

With 40 questions per language, a difference of one question is 2.5 points, and no ranker is best in every language.

4.3 BRIGHT: questions that need reasoning

Reading each passage’s first 1,200 characters. With embeddings, search’s own order put a gold passage first for 36% of the 61 questions it found one for (Figure 4, Table 4). No ranker clearly beat that. The best LLM judge, GPT-6 Luna, was 6.6 points ahead (−6.6 to 19.7), and Jev 4.9 points behind (−21.3 to 11.5). Laya, Cohere Rerank 3.5 and Amazon Rerank 1.0 fell below it, though within the margin of error once adjusted for the 24 comparisons.

A judge that does not beat the first stage still changes its order, gaining some questions and losing others. GPT-6 Luna put a gold passage first for 11 questions where search had not, and moved one out of first place for 7; Jev did so for 12 and 15, Cohere Rerank 3.5 for 5 and 19.

Why search’s order held up. BRIGHT’s passages are long. The median is 4,000 characters, against 291 on MIRACL, and 99% of the gold passages run past the 1,200 characters a judge sees. Search scores each passage whole. Ordering the same 24 candidates by BM25 on whole passages put a gold passage first for 34% of questions, close to search’s own 36%; on their first 1,200 characters alone, for 13%. Most of the words that led BM25 to the gold passage lie beyond what a judge reads. Stuga’s own passages, whole sections of up to 12,000 characters, can be longer still.

Reading the 1,200 characters that best match. We designed this after the results above and measured it on the same questions, so it is exploratory until tested on others. We gave every judge, for each passage longer than 1,200 characters, the 1,200 that hold the most of the question’s words, the rarer ones counting for more, instead of the first 1,200, within the same 1,200-character limit. 21 of 24 judges rose, 7 of them clearly once adjusted for the 24 comparisons (Figure 4). Claude Opus 5.5 reached 61%, from 39%, and Jev 48%, from 31%; only Claude Opus 5.5 clearly beat search’s own order. Cohere Rerank 3.5 and Amazon Rerank 1.0 stayed below it, at 23% and 20%. Kev-4B, Qwen3-Reranker 4B and Laya, all on this machine, did not rise. Over all 117 questions, a retrieval miss counting as a miss, Claude Opus 5.5 put a gold passage first for 32% and Jev for 25%, against search’s 19%. With Jev a gold passage was among the eight the model reads for 93% of the 61 questions, against 72% without a judge. Stuga’s judges now read the best-matching 1,200 characters.

With and without embeddings. Table 4’s two Hit@1 columns count different questions: 36% of the 61 that search with embeddings found a gold passage for, and 40% of the 47 that BM25 alone found one for. With embeddings, search found a gold passage for every question BM25 alone did and for 14 more, but put it first for only 2 of those 14: they raise the count and lower the share. Over all 117 questions, search put a gold passage first for 19% with embeddings and 16% without. On BM25’s candidates the judges gained more. Reading the first 1,200 characters, Claude Opus 5.5 was 17.0 points ahead of search (−2.1 to 36.2); reading the best-matching 1,200, Claude Opus 5.5, Claude Sonnet 5.5 and MiniMax M2.5 clearly beat search’s 40%, at 72%, 70% and 68%.

4.4 How LLM judges fail

Stuga reads the scores wherever an answer states them unambiguously: as the JSON array it asks for, as the same objects one per line or followed by a note, or as [n] score lines. Four failure modes remained, all from producing the ranking as text. An unusable answer leaves the first-stage order.

• The answer ignores the format. GPT-6 Luna sometimes returned a bare list of numbers, only the best passage, or [index, score] pairs: 11% of its MIRACL answers were unusable.
• The answer is corrupted. DeepSeek V3.2 sometimes wrote a stray token where a score belongs, or text unrelated to the task, in 3% of its MIRACL answers.
• The answer does not end. Mistral Large 3 repeated its list until the output limit, and MiniMax M2.5’s reasoning ran into it: 9% and 10% of their MIRACL answers were unusable.
• Scores tie. Two passages shared the top score in 38% of GPT-6 Sol’s MIRACL answers and 24% of Claude Opus 5.5’s; the first-stage order broke the tie. Jev’s probabilities tied in 13%.

Reasoning costs time. Grok 4.6 wrote 1,623 output tokens per search on average for a list of 24 scores, where GPT-6 Sol wrote 198, and took 12 s.

4.5 Cost

Jev counted 8,801 input tokens per MIRACL search and costs input only. Claude Opus 5.5 read 7,093 and wrote 273. Cohere Rerank 3.5 is billed per query, whatever its length.

4.6 Local judges

Qwen3-Reranker 4B reached 81% on MIRACL, level with the hosted leaders, in 5.3 s: a cross-encoder reads the question with each of the 24 passages, and Thai, Japanese and Korean text is long in tokens. The 0.6B model reached 75% in 987 ms.

Two open projects serve the System One API: Kev [11] and Laya [12]. Kev-4B reached 64% in 5.4 s. Laya, whose English checkpoint is ModernBERT-large and whose multilingual one is mmBERT-base, reached 49% in 646 ms; its README places further accuracy in fine-tuning.

5. Choosing a first stage and a judge

In this study’s setup, the results suggest the following defaults. None was tested on a team’s own documents, with other chunking or with real users’ questions, and each holds only as far as two benchmarks of this size allow (Section 7).

Table 5. Choosing a first stage and a judge, from the results above.

For Stuga’s defaults this means Semantic search on and Reranking with Jev, reading the part of each passage that best matches the question. A team whose questions are mostly troubleshooting may gain more from an LLM judge, at its latency and cost; rewriting the question before retrieving helped on BRIGHT in its authors’ tests [15], and we did not test it in Stuga.

6. In Stuga

Reranking is set up under Settings → This node → AI providers, beside Built-in AI and Semantic search, with a TypeSafe key, directly or through OpenRouter. Every search from Ask and from agents then goes through it. Without it, the chat model judges while Built-in AI is on; with neither, fusion order stands.

A hosted judge sees what it ranks: the question and up to 24 passages, 1,200 characters of each, go to TypeSafe or OpenRouter, as they go to the chat provider when it judges. Reranking accepts any System One endpoint, so a server such as Kev on the same machine keeps everything local, at 64% on MIRACL and 5.4 s per search.

Ranking decides what an agent reads. What it writes back to a document or table waits, by default, for a person to accept or reject: how that review works.

7. Limitations

• MIRACL’s candidates are MMTEB’s lists, not Stuga’s retrieval, and they count every candidate no annotator marked relevant as not relevant, so a relevant passage nobody saw counts against whichever ranker puts it first.
• Both benchmarks are public and older than every model tested, so their questions may be in the models’ training data, which would favour the LLM judges.
• With 61 rankable BRIGHT questions and 40 per MIRACL language, differences of a few points are within noise. A paired interval that includes zero means this test cannot tell two rankers apart, not that they are equivalent.
• We measured the ranking, not the answers Ask writes from it: a better first passage need not mean a better answer or citation.
• BRIGHT’s authors show that reasoning about a query before retrieving changes what is found [15]; Stuga retrieves from the question as asked, and so did we.
• Wikipedia articles and StackOverflow documentation are not a team’s own documents; other collections may rank differently.
• Latency depends on vendor load and the network; it was measured from one client.
• Reading the part of each passage that best matches the question was designed after the first BRIGHT results and measured on the same questions; it needs testing on other data.
• We did not vary the number of candidates, how many characters a judge sees of each, or the prompt, and varied which characters on BRIGHT only; Section 4.3 shows that choice moves the results.
• Every LLM received the same prompt, untuned. Tuning prompts or reasoning per model would move some results.

8. Reproducibility

Code, the two test sets, every candidate list and every call’s result are at stuga-dev/rerank-bench (code MIT; data under each source’s licence). The test sets are built from pinned revisions of the public datasets.

git clone https://github.com/stuga-dev/rerank-bench && cd rerank-bench && pnpm install
BENCH_KEYS=keys.json node src/run.ts --set miracl --only jev,gpt-6-sol --limit 5
node src/report.ts results/miracl/<date>

A ranker maps a query and 24 passages to 24 scores; pull requests with new rankers are welcome.

Stuga has no business relationship with TypeSafe or any other vendor named here; all calls were paid at list price.

References

1. D. Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011.
2. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL, 2019.
3. R. Nogueira, K. Cho. Passage Re-ranking with BERT. arXiv:1901.04085, 2019.
4. W. Sun et al. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. EMNLP, 2023.
5. S. Robertson, H. Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3(4), 2009.
6. G. V. Cormack, C. L. A. Clarke, S. Büttcher. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR, 2009.
7. B. Warner et al. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663, 2024.
8. TypeSafe. Introducing System One models and Jev. 15 September 2026.
9. B. Efron, R. J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993.
10. Amazon Web Services. Amazon Bedrock pricing, retrieved 28 September 2026.
11. J. Palmer. Kev and kev-4b, 2026.
12. Convai Innovations. Laya and Laya multilingual (models); laya 0.3.20 (code), 2026.
13. X. Zhang, N. Thakur, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu, M. Rezagholizadeh, J. Lin. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. TACL 11, 2023.
14. K. Enevoldsen et al. MMTEB: Massive Multilingual Text Embedding Benchmark. arXiv:2502.13595, 2025.
15. H. Su et al. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval. arXiv:2407.12883, 2024.