Domain-Tuned Retrieval Embeddings
A proof of concept: fine-tuning bge-small on synthetic query-passage pairs lifted NDCG@10 by 4.59 points on 710 held-out queries.
Problem
General-purpose embeddings missed relevant passages when people asked in modern words about archaic-language text. The meaning lines up, but the vocabulary doesn't.
This is a proof of concept, not a production system.
What I built
I assembled a corpus of 3,500 passages of public-domain scripture and commentary. Qwen2.5-7B, quantized to 4-bit AWQ and served on local vLLM with structured JSON output, generated 6,274 synthetic query-passage pairs. I dropped queries that copied the passage's wording.
I split at the passage level and held out 12% of passages, which left 710 test queries. Hard negatives mined with the base model gave 13,084 triplets. I fine-tuned bge-small-en-v1.5 with MultipleNegativesRankingLoss at batch size 32, learning rate 2e-5, and 10% warmup. A run takes about 3 to 4 minutes on an RTX 5070 Ti.
3,500 passages
│ Qwen2.5-7B · 4-bit AWQ · local vLLM · JSON output
▼
6,274 query-passage pairs (copied-wording queries dropped)
│ passage-level split: 12% held out → 710 queries
▼
hard negatives mined with the base model → 13,084 triplets
│ MultipleNegativesRankingLoss · batch 32 · lr 2e-5
▼
fine-tuned bge-small-en-v1.5 ──▶ eval on 710 held-out queriesDecisions and tradeoffs
Tested: the mining margin. With no margin, likely false negatives got through. A margin of 0.05 kept only 5,196 triplets. A margin of 0.0 kept 13,084.
Tested: epochs. One epoch beat two and four.
Tested: the base model. all-MiniLM-L6-v2, tuned the same way, landed nearly level with bge-small.
Reasoned: a passage-level split, so no test passage leaks into training through a different query. A full fine-tune rather than LoRA, because at 33M parameters there's little to save. Dropping copied-wording queries, so the model doesn't learn keyword matching.
Results
On the 710 held-out queries, compared with the base model: Accuracy@1 up 4.65 points, NDCG@10 up 4.59, MRR@10 up 4.61, and Recall@10 up 4.51.
More training hurt. Four epochs dropped NDCG@10 from 0.320 to 0.296.
What's next
The evaluation queries are synthetic and there's no confidence interval yet, so these are early numbers. Next: real queries, a bigger batch, and cross-translation pairs.