EmbeddingGemma v1 vs v2: v1 ranked better in our text-search test

9 October 2026 · The Stuga team

TL;DR

If you’re choosing an embedding model for text search, the older of Google’s two EmbeddingGemma models ranked the relevant passages higher in our test. We call them EmbeddingGemma v1, Google’s EmbeddingGemma (embeddinggemma:300m in Ollama), and EmbeddingGemma v2, the new EmbeddingGemma 2 (embeddinggemma-2:270m).

Ask both “What is the longest snake in the world?” over English Wikipedia. v1 puts first a passage from List of largest snakes that names the reticulated python. v2 puts first a line about a sea snake, from Hydrophis spiralis, and ranks a passage that MIRACL’s annotators judged relevant 4th (Example 1).

Averaged over MIRACL’s Korean, Chinese and English sets, 1,405 questions in all, the scores point the same way. Searching by vectors alone, with both models at bf16 (16-bit weights), v1 scored 61.1 nDCG@10 and v2 55.5. nDCG@10 runs from 0 to 100, and higher is better: the relevant passages sit nearer the top of the first ten results. v1’s lead was 5.6 points, with a 95% interval of 4.2 to 7.0. Hybrid search, which adds keyword search to the vectors, brought the two within about a point.

Google reports v2 ahead on code and adds images, audio and video [1, 2]; we tested none of those. For text in these languages, v1 is the better pick on our sets. If you run v2 for text, pull its bf16 or 8-bit build rather than Ollama’s 4-bit default, send its prompts, and set any distance cutoff for it again.

6:18 Prefer watching? The benchmark walkthrough Real search examples, precision checks, and what changes when you add keyword search. Watch on YouTube

1. What Google says, and what we tested

Google released EmbeddingGemma 2, our v2, on 6 October 2026 under Apache 2.0. It maps text, images, audio and video into one 768-dimension space and reads up to 8,192 tokens [1, 2]. Google compares it with the first EmbeddingGemma, our v1: 78.68 against 68.76 on MTEB Code, about 14% better, and 61.36 against 61.15 on MTEB Multilingual, which the launch post calls matching its predecessor’s multilingual text performance [1, 2].

An embedding model turns a text into a list of numbers, a vector, so that texts with similar meanings land close together; search returns the passages closest to the question. MTEB Multilingual averages many kinds of task, from classification and clustering to sentence similarity and retrieval [4]. We measured one of them, retrieval: given a question, put the passages that answer it at the top. That’s the job of a search box, and the first step when an LLM answers from your documents, often called RAG. Our result covers that job on these sets, so it doesn’t contradict Google’s average.

2. What we ran

We searched the way Stuga, our open-source document workspace, searches: its prompts, vector width, keyword search and fusion. We have no relationship with Google, Ollama or any other vendor whose model we tested, and we paid list price for every hosted call.

Sets. Two public benchmarks whose questions and relevant passages were marked by people:

• MIRACL [3]: native speakers wrote questions over Wikipedia and judged the passages. We used MMTEB’s hard-negative version [4], which keeps, per language, every judged passage and the top results that several retrievers, BM25 among them, returned for each question: passages that rank high for some question without being marked relevant to it. A passage is embedded the way Stuga embeds a document’s first section: its title, a blank line, its text.
• BRIGHT StackOverflow [5]: technical questions, each with the documentation and forum passages BRIGHT judged relevant to it, often worded differently from the question. With 117 questions, a difference of a few points may be noise, so we call these results exploratory. It’s a set of technical questions, not a code-retrieval set.

A check on the Chinese result, on four C-MTEB sets, is in section 3.

Models. Ten models: nine on MIRACL, six of them local through Ollama and three hosted on Amazon Bedrock, as served on 7 October 2026, and OpenAI’s text-embedding-3-large on BRIGHT only.

How each model was called. Each model got what a Stuga node sends it:

• Prompts. Many embedding models were trained to see a short instruction before a question and another before a passage. Each model got the ones Stuga sends it, copied verbatim from Stuga’s code: for v1 and v2, task: search result | query: before a question and title: none | text: before a passage. bge-m3 and Titan Text Embeddings V2 take none. On Bedrock, both Cohere models got the input types search_query and search_document; Cohere Embed Multilingual v3 refuses a text over 2,048 characters, so texts were cut to that first.
• Width. A Stuga node stores 1,024-dimension vectors by default. A narrower model returned its own width, 768 for v1 and v2; hosted models were asked for 1,024.
• Ollama. Version 0.40.0 on one Mac (Apple M6, 32 GB), 64 texts a request, each tag at the quantization it ships with. Of the tags we ran, only v2’s default is 4-bit (nvfp4), so we ran its bf16 and mxfp8 tags too.
• A check outside Ollama. On Korean, we also ran v2 through sentence-transformers [10], the library Google’s model card uses, at bf16 with the same prompts, cut at 2,048 tokens as Ollama cut it.

Scoring.

• Vectors alone: every passage scored by cosine similarity, how closely its vector points the same way as the question’s, with no index in between. nDCG@10 as MMTEB scores it [4, 6], and Recall@100, the share of relevant passages in the first hundred.
• Hybrid: Stuga’s search. The 96 nearest passages and the 96 best by BM25 [7], through pg_search with Stuga’s tokenizers for Korean and Chinese, are merged by reciprocal rank fusion [8] (k = 60) and 24 kept: what Stuga hands its reranker. The reranker is off, and no distance cutoff applies, as for Stuga’s Ask and agents’ retrieval.
• Differences: paired on the same questions, 2,000 bootstrap resamples within each language, 95% interval [9]. An interval that includes zero means the test can’t tell the two apart. The intervals are exploratory: they aren’t adjusted for multiple comparisons. MIRACL figures average the three languages.
• Examples: each is one MIRACL English question. For each pattern we took the short, plain questions where it happened, ordered them by how near their ranks sat to the middle of that list, read them in that order, and kept the first whose difference shows in the excerpts. The ones we passed over, some because an unjudged passage also answered the question, are in the results with the reason. Beside each example is how many questions fit its rule, and how many fit the reverse.

3. MIRACL: Korean, Chinese and English

3.1 One question

Example 1 shows the first three passages each model returned for one question, both at bf16 with vectors alone.

• This is one of 70 English questions where v1 put a relevant passage first and v2’s first relevant passage came 3rd to 10th. The reverse, v2 first and v1 3rd to 10th, happened on 26.
• Over all 799 English questions, the first passage was one MIRACL judged relevant for 404 with v1 and 362 with v2.
• Exploratory: v2 put a very short passage first, like the sea-snake line, more often than v1. A passage under 80 characters, such as a heading, a list’s lead-in or a sentence fragment, came first on 58 of the 799 English questions for v2 and on 14 for v1. About 15% of the passages are that short.

3.2 Every question

• At the same precision, v1 scored higher in each language. With vectors alone, 61.1 against 55.5 for v2 at bf16, +5.6 [4.2, 7.0]. The gap was largest on MIRACL Chinese, 63.0 against 54.7, 8.3 points; English 55.8 against 50.1, 5.7; Korean 64.4 against 61.7, 2.8. Differences come from the unrounded scores.
• v2 wasn’t behind on every measure. On Korean, its bf16 tag found 97.4% of relevant passages in the first hundred, against 96.9% for v1 (Table 5). Over the three languages, v1’s Recall@100 was higher, +0.8 [0.2, 1.6].
• With hybrid search, the two were about a point apart: 54.4 against 53.4, +1.0 [0.1, 2.0]. Between the best and the worst model, the gap shrank from 12.6 points with vectors alone to 4.6 in hybrid search.
• Against v2 at bf16, with vectors alone, bge-m3, Cohere Embed Multilingual v3, nomic-embed-text-v2-moe, EmbeddingGemma v1, Snowflake Arctic Embed 2 and Cohere Embed v4 scored higher with intervals above zero; the test couldn’t tell Qwen3-Embedding 0.6B apart from it; Amazon Titan Text Embeddings V2 scored lower.
• bge-m3 scored highest with vectors alone, 63.9, and in hybrid search, 56.3.

3.3 Where keywords help

On MIRACL, adding keyword search lowered most scores (Table 3), yet it can still find what vectors miss, such as a name. Example 2 is one question, with v2 at bf16.

By meaning alone, v2 put two other men called Christian first and the relevant passage 16th. Keyword search ranked it 1st, on the words Christian, von and Ehrenfels, and the fused list put it 2nd. This is one of 20 English questions where hybrid search lifted v2’s first relevant passage from below 10th into the top 3. The reverse, a relevant passage in the top 3 by vectors pushed below 10th by hybrid search, happened on 36. v1 had this passage first with vectors alone.

3.4 A second Chinese check

To test the Chinese result on text other than Wikipedia, and on passages that weren’t pooled from BM25, we ran four C-MTEB retrieval sets [11], about 100,000 passages each. v2 ran there only as Ollama’s default tag, not its bf16 build, so the comparison is with the 4-bit build. With vectors alone, v1 scored higher on three of them and the test couldn’t tell the two apart on e-commerce. Each of those gaps was smaller than MIRACL Chinese’s against the same tag (Table 4).

4. Precision and prompts

4.1 Ollama’s 4-bit default

Through sentence-transformers, v2 scored 62.4 on Korean, +0.7 [0.2, 1.3] against Ollama’s bf16 tag, with the same Recall@100: Ollama’s bf16 build came close to it. We checked Korean only, and the check doesn’t tell us where the gap to v1 comes from.

Over the three languages, with vectors alone:

• Ollama’s default 4-bit (nvfp4) tag scored −1.4 [−2.1, −0.6] against the bf16 tag.
• The mxfp8 tag scored −0.1 [−0.5, 0.3]: the test can’t tell it from bf16.
• In hybrid search the 4-bit difference was −0.4 [−1.0, 0.2], which the test can’t tell from zero either.

If you run v2 in Ollama for text search, pull embeddinggemma-2:270m-bf16-text or embeddinggemma-2:270m-mxfp8-text.

4.2 Send the prompts

Google’s model card says omitting the prefix “still works but reduces precision” [2]. Example 3 is one question, on Ollama’s default tag of v2, sent with its prompts and as raw text.

Sent raw, a list’s lead-in came first, and the definition MIRACL judged relevant fell to 4th, though an unjudged one-line definition stayed 2nd in both lists. This is one of 53 English questions where the prompts put a relevant passage first and raw text put its first relevant passage 3rd to 10th; the reverse happened on 13. Exploratory: sent raw, a passage under 80 characters came first on 173 questions, against 60 with the prompts.

Over the three languages, raw text scored 50.5 against 54.1 with the prompts, −3.6 [−4.7, −2.4]. English fell most, from 49.0 to 42.8; MIRACL Chinese from 53.5 to 49.3; Korean from 59.7 to 59.3. In hybrid search the difference was −1.4 [−2.1, −0.7]. If your code embeds raw text, add the prefixes yourself.

5. BRIGHT StackOverflow (exploratory)

• v1 scored highest on BRIGHT: 16.4 with vectors alone and 19.2 in hybrid search, against 10.4 and 17.2 for v2 at bf16.
• v2’s vectors alone scored below BM25 alone, 14.0. Hybrid search scored above vectors alone for every model.
• bge-m3, first on MIRACL, scored lowest here: 10.2 and 14.3.
• OpenAI’s text-embedding-3-large ran here only, from the vectors our reranker benchmark made of the same texts.
• These are technical questions answered by documentation and forum pages, not code search, so they neither confirm nor contradict Google’s code results.

6. Switching models moves the distances (exploratory)

Many apps drop a match by meaning when it’s too far from the question. Cosine distance is one minus the cosine similarity of two vectors: 0 when they point the same way, larger as texts grow less alike, so lower is closer. How far is too far depends on the model.

• On MIRACL, v1 put a question’s relevant passages at a median distance of 0.425 (the median over questions) and random passages at 0.964. v2, on Ollama’s default tag, put them at 0.224 and 0.464.
• A fixed cutoff of 0.6 kept 95.6–99.0% of relevant passages for v1 and let 0.08–0.22% of random ones through, depending on the language. For v2 it let 99.98–100% of random passages through.

Keeping that cutoff when you switch from v1 to v2 lets nearly every random passage through. Set a cutoff for each model from that model’s own distances.

7. Choosing a model for text search

• Korean, Chinese or English text: v1; pull embeddinggemma:300m. On these MIRACL sets it scored 5.6 points above v2 at bf16 with vectors alone and 1.0 in hybrid search.
• Images, audio, video or code: of the two, only v2 embeds images, audio and video, and Google reports it ahead on code [1, 2]. We didn’t test these.
• Running v2 for text: pull the bf16 or mxfp8 tag, send its prompts, and set your distance cutoff for it.
• Hybrid search: it narrowed the gaps between models on MIRACL and scored above vectors alone for every model on BRIGHT.
• Your own text: bge-m3 led MIRACL and trailed BRIGHT. Try a few dozen of your own questions with known answers before you choose.

8. In Stuga

Semantic search adds search by meaning to Stuga’s keyword search, and Settings suggests a model for it from this benchmark. In Stuga 0.1.12:

• When a local Ollama has no embedding model yet, Settings says “No embedding model here yet. Pull one, such as embeddinggemma:300m.”
• The node adds the prompts of the models it knows by name, EmbeddingGemma among them, whichever server runs them.
• A model narrower than the node’s default 1,024-dimension column, such as v1 or v2 at 768, is padded with zeros, which leaves every cosine distance unchanged, with no database change.
• Ask and agents’ retrieval run the hybrid search measured here: the 96 nearest passages and the 96 best by words, fused to 24, with no distance cutoff. With Chinese chosen under the search languages, keyword search reads traditional characters as simplified.
• Search strictness sets the search box’s distance cutoff from the model in use. The node measures the model when it’s saved, on fixed texts written for the purpose, separately for short searches and longer questions. Balanced sits where 1 in 100 of the unrelated sample pairs falls. Every MIRACL query counts as a question, and there, in an exploratory check, Balanced kept 99.8–100% of relevant passages for v1 and let 0.37–1.80% of random ones through, depending on the language; for v2’s default tag, 99.3–100% and 0.48–1.35%.

AI is off until a node administrator sets it up, and with Ollama on the same machine the text stays there.

9. Limitations

• The local runs used one Mac and Ollama 0.40.0. Each tag ran at the quantization it ships with; another build of the same model can score differently, as v2’s did.
• OpenAI’s BRIGHT vectors came from our reranker benchmark.
• MIRACL’s hard negatives were picked partly by BM25, which may work against hybrid search there. The Chinese result at bf16 is MIRACL Chinese alone: Wikipedia, with simplified questions. The C-MTEB check compared v1 with v2’s 4-bit default tag only.
• BRIGHT has 117 questions and no paired intervals.
• An example is one question. The counts beside it, and the scores, say how often a pattern happened.
• We tested text retrieval only, not images, audio, video or code search.
• v2 read at most 2,048 tokens of each text, so its 8,192-token context wasn’t tested.
• The sentence-transformers check covers Korean only.
• The reranker was off: these are first-stage results.

10. Reproduce

The code, the commands and every result file are in stuga-dev/experiments. The sets download at pinned revisions. The vectors aren’t committed, so a fresh checkout embeds every set before it scores. scripts/examples.ts picks the examples from the same vectors and writes results/2026-10-07/examples.json, with the questions it passed over.

References

1. Google. EmbeddingGemma 2: an open, lightweight multimodal embedding model. 6 October 2026.
2. Google DeepMind. EmbeddingGemma 2 model card. Hugging Face, 2026.
3. X. Zhang, N. Thakur, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu, M. Rezagholizadeh, J. Lin. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. TACL 11, 2023.
4. K. Enevoldsen et al. MMTEB: Massive Multilingual Text Embedding Benchmark. arXiv:2502.13595, 2025.
5. H. Su et al. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval. arXiv:2407.12883, 2024.
6. K. Järvelin, J. Kekäläinen. Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems 20(4), 2002.
7. S. Robertson, H. Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3(4), 2009.
8. G. V. Cormack, C. L. A. Clarke, S. Büttcher. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR, 2009.
9. B. Efron, R. J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993.
10. N. Reimers, I. Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP, 2019.
11. S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, J.-Y. Nie. C-Pack: Packed Resources For General Chinese Embeddings. SIGIR, 2024.