Dense vector index returns stale document IDs and wrong passages after incremental update job — RAG answers become semantically irrelevant

Environment

Self-hosted vector DB (FAISS on disk with custom metadata mapping layer) behind a microservice written in Python 3.10; nightly incremental indexer runs: re-embed changed documents and upsert to FAISS; embedding model is served via local transformer container; RAG pipeline uses top-10 nearest neighbors with cosine similarity and a re-ranking step.

Symptoms

After last incremental update, user queries return passages that are high-similarity but clearly unrelated to the query; some returned passages correspond to old document versions that were deleted weeks ago. The issue is intermittent: some queries return correct results for a few minutes after reindex, then degrade.

Observations

Indexer logs show upsert operations completed without error; store mapping file (doc_id -> vector_offset) timestamps updated; search returns vectors with very high cosine similarity scores but metadata lookup yields different document text than expected for that vector offset; inspection of FAISS index file shows size unchanged and mtime older than mapping file; reloading the index into process memory causes immediate changes in returned results (sometimes more correct); embedding service logs show occasional timeouts during index job but overall success.

Already attempted

Restarted vector DB service and reloaded index into memory; re-ran incremental indexer with force-upsert and verified no errors; ran a full reindex to a new index file (takes hours) which restored correct search results; implemented a stop-the-world pause for query traffic during incremental run as a temporary mitigation.

Question

Why does the incremental update leave stale/incorrect mappings between vectors and metadata, returning unrelated high-similarity passages, and how can I safely run incremental updates without full reindexing?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1d ago• Trouble ID

Most likely causes

Non-atomic incremental upserts to FAISS combined with separate metadata store updates: vectors and metadata get out of sync if index updates fail mid-flight or queries read partially-updated state. 85%

Indexer process experienced intermittent timeouts from embedding service causing it to insert placeholder vectors or reuse offsets incorrectly, corrupting vector->doc_id mapping. 60%

In-memory index cache in the search service wasn't hot-reloaded properly after on-disk index file replaced, causing queries to hit stale in-memory structure referencing old offsets. 45%

Summary

The symptoms point to a metadata-vector mapping mismatch: the vector index (FAISS) on-disk structure and the separate metadata store got out of sync during incremental updates. Likely causes include concurrent reads/writes to the index without atomic swap, partial upserts when embedding service timeouts occurred, or a bug in the mapping layer where upserts update metadata but fail to atomically update the corresponding vector slot. The fact that a full reindex fixes results and that reloading the index into memory changes behavior indicates a race between in-memory and on-disk representations or non-atomic index updates. Safe incremental updates require atomic replacement of the searchable index or a write-ahead staging approach.

Suggested Fixes

Diagnostic step 1

Stop serving queries or divert them to a fallback when running incremental indexing until you confirm an atomic swap mechanism is in place; avoid running index update and traffic on the same index without staging.

Diagnostic step 2

Implement atomic index replace: write updated FAISS index to a new file, validate it, then atomically rename/swap pointer (symlink) used by the search service and trigger in-process reload. Ensure metadata store swap occurs in the same transaction or in a coordinated manner.

Diagnostic step 3

Add checks in the indexer to verify that every vector upsert has a corresponding metadata commit; if embedding timeouts occur, retry embedding and avoid inserting placeholder vectors. Log and alert on partial failures.