RAG retrieval benchmark: bge-m3 vs nomic-embed-text for a Chinese knowledge base

This is the English version of the Chinese original (中文原文). Both versions use the same data from the same run.

The most common RAG mistake is picking an embedding model at random, then wondering why answers miss the point. I indexed 27 articles from this site with two popular local embedding models and checked which one retrieves the right article.

1. Result first

Metric	bge-m3	nomic-embed-text
Vector dimensions	1024	768
Time to index 156 chunks	39.9 s (about 256 ms per chunk)	19.8 s (about 127 ms per chunk)
Average query time	50 ms	26 ms
top-1 hits (natural questions)	8 / 10	4 / 10
top-3 hits (natural questions)	9 / 10	5 / 10

Verdict: use bge-m3 for a Chinese knowledge base. nomic is about twice as fast, but with natural Chinese questions it fails to retrieve the right article almost half the time.

2. How it was measured

• Corpus: 27 markdown files from this site (including index pages and the About page), split by second-level headings into 156 chunks
• Questions: 10 natural questions; each one is labelled in advance with the article that should be retrieved
• Retrieval: cosine similarity, top 3; chunk hits are deduplicated to article level before scoring
• Interpretation: a top-1 hit means the expected article was ranked first; a top-3 hit means it appeared anywhere in the first three results
• Machine: Intel i5-12400, 32 GB RAM, no discrete GPU; Ollama 0.40.13; models bge-m3:latest and nomic-embed-text:latest

3. Full results

Condition	bge-m3 top-1 / top-3	nomic top-1 / top-3
Section index, natural question	8 / 9	4 / 5
Section index, keyword query	8 / 10	7 / 9
Whole-document index, natural question	7 / 9	5 / 6

4. Four findings

1. bge-m3 is clearly stronger on natural Chinese questions. 8/10 versus 4/10 is not noise. Asked "what should I solve first on a cube?", nomic returned "common questions" and "timer and records" — it never reached the beginner tutorial.

2. Keyword phrasing rescues nomic — up to a point. Rewriting the question as keywords ("slow turning, getting slower") lifted nomic from 4/10 to 7/10, which suggests it relies heavily on literal overlap. bge-m3's keyword version reached 10/10 on top-3, but it was already retrieving the right article.

3. Whole-document indexing does not help. Both models got worse when each article was embedded as a single vector (bge-m3 8→7, nomic 4→5). Averaging a long article into one vector dilutes the topic; splitting by section works better.

4. Speed and quality pull in opposite directions. nomic indexes twice as fast and queries twice as fast, at the cost of roughly half the hit rate. For a personal knowledge base of a few hundred chunks, spending 20 more seconds on indexing is clearly worth it.

5. Failure case (verbatim)

Question	bge-m3 retrieval	Expected article
Why is my cubing getting slower, and how do I fix it?	Common questions → Buying guide → Common questions	Slow turning

bge-m3's only real miss is a paraphrase: the article talks about "slow turning", the question says "getting slower". When the question vocabulary drifts far from the document vocabulary, retrieval degrades — rewrite the question or add synonyms.

6. How to choose

Situation	Recommendation
Chinese knowledge base, Q&A	bge-m3 (1024 dimensions, recall first)
English corpus, latency-sensitive	nomic-embed-text, but phrase questions as keywords
Mixed Chinese and English	bge-m3
Very weak machine (under 8 GB RAM)	nomic is lighter, accept the lower hit rate

7. Reproduce it

ollama pull bge-m3
ollama pull nomic-embed-text

Embedding endpoint:

curl http://127.0.0.1:11434/api/embed -d "{\"model\":\"bge-m3\",\"input\":\"text to embed\"}"

The benchmark scripts are editor\_rag-eval.mjs and _rag-eval2.mjs; raw results, including which three articles each question retrieved, are in editor\logs\rag-eval2-*.json.

8. Limitations

• 10 questions is a small sample: use this as a direction, not a benchmark score
• The "correct article" labels are my own judgement
• Only retrieval was tested, not whether the LLM answers correctly after retrieval
• Single machine, single run; other hardware or model versions will give different numbers