RAG assistant returns high-similarity but irrelevant passages from vector DB on production queries

Environment

Production RAG service: embedding model OpenAI text-embedding-3-small, vector DB: self-hosted Milvus, chunking pipeline produces 1–2 KB passages with metadata, retriever uses cosine similarity and returns top 10 hits; downstream LLM (gpt-4o-mini) consumes hits and generates answers.

Symptoms

Users report the assistant returns confident paragraphs that do not answer the question: retrieved passages have high similarity scores but are tangential or contain outdated references. This happens more on long, multi-part queries. Local manual searches in the vector DB for the same query find different top hits than the production retrieval returns.

Observations

Query embedding vectors logged show unusually low norm variance compared to document embeddings. Similarity scores for top hits cluster tightly (0.88–0.91) even when relevance judged poor. Milvus reports no errors; index info shows IVF index with nlist=1024 and runtime_search_params = {nprobe: 8}. The chunker produced overlapping chunks but some chunks lack clear context metadata (source document IDs missing). Retriever uses an approximate index + then filters by metadata, but metadata filtering sometimes excludes the best matches.

Already attempted

Re-run retrieval locally using exact cosine on stored vectors — results differ: exact search surfaces more relevant chunks that were missing from approximate index top-10. Rebuilt the Milvus index with higher nlist and increased nprobe to 32 in a test cluster; relevance improved but query latency increased. Examined prompting: LLM prompt contains a strict instruction to prefer text from 'source' metadata, but many chunks lacked complete metadata.

Question

How do I diagnose whether the root cause is embedding quality, vector index approximation parameters, chunking metadata loss, or prompt/selection logic, and what reversible tuning or tests should I run to improve RAG relevance without risking production downtime?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 2d ago• Trouble ID

Most likely causes

Aggressive approximate index params (low nprobe) causing false negatives in ANN search — relevant vectors not retrieved in top-K. 90%

Chunking pipeline dropped or mis-populated metadata, so post-retrieval filters exclude otherwise relevant chunks. 80%

Embedding drift or normalization problem producing low-variance query embeddings that don't separate queries well from corpus. 70%

Summary

The discrepancy between approximate index results and exact search indicates the vector index configuration (IVF nlist + nprobe) or stale/incorrect index may be returning suboptimal approximate nearest neighbors. Contributing factors: low-variance query embeddings (embedding model mismatch or normalization issues), incomplete chunk metadata causing filtering to exclude good results, and retrieval pipeline ordering (approximate search then metadata filter) causing relevant items to be dropped. Each of these is independently reversible and should be tested systematically.

Suggested Fixes

Diagnostic step 1

Run an A/B retrieval test: compare exact cosine similarity (brute-force) vs ANN results for a representative query set to quantify recall loss; do this on a copy of the DB (non-production) to avoid impacting latency.

Diagnostic step 2

Temporarily increase nprobe or re-create the ANN index with a larger nlist on a staging index to measure relevance improvements; monitor query latency and resource use before promoting to production.

Diagnostic step 3

Validate embedding pipeline: compute per-dimension and norm statistics for a sample of query and document embeddings to detect model drift or missing normalization (e.g., ensure consistent use of L2 normalization if expected).