1
Diagnostic step 1
Stop serving queries or divert them to a fallback when running incremental indexing until you confirm an atomic swap mechanism is in place; avoid running index update and traffic on the same index without staging.
FULL AI ANALYSIS
Dense vector index returns stale document IDs and wrong passages after incremental update job — RAG answers become semantically irrelevant
Generated Oct 14, 2026, 9:39:02 AM
Analysis summary
The symptoms point to a metadata-vector mapping mismatch: the vector index (FAISS) on-disk structure and the separate metadata store got out of sync during incremental updates. Likely causes include concurrent reads/writes to the index without atomic swap, partial upserts when embedding service timeouts occurred, or a bug in the mapping layer where upserts update metadata but fail to atomically update the corresponding vector slot. The fact that a full reindex fixes results and that reloading the index into memory changes behavior indicates a race between in-memory and on-disk representations or non-atomic index updates. Safe incremental updates require atomic replacement of the searchable index or a write-ahead staging approach.
Possible causes
Non-atomic incremental upserts to FAISS combined with separate metadata store updates: vectors and metadata get out of sync if index updates fail mid-flight or queries read partially-updated state.
85%Indexer process experienced intermittent timeouts from embedding service causing it to insert placeholder vectors or reuse offsets incorrectly, corrupting vector->doc_id mapping.
60%In-memory index cache in the search service wasn't hot-reloaded properly after on-disk index file replaced, causing queries to hit stale in-memory structure referencing old offsets.
45%FAISS file corruption or race-condition bug in the version used that manifests under concurrent writer/readers during incremental updates.
30%Suggested fixes
2
Diagnostic step 2
Implement atomic index replace: write updated FAISS index to a new file, validate it, then atomically rename/swap pointer (symlink) used by the search service and trigger in-process reload. Ensure metadata store swap occurs in the same transaction or in a coordinated manner.
3
Diagnostic step 3
Add checks in the indexer to verify that every vector upsert has a corresponding metadata commit; if embedding timeouts occur, retry embedding and avoid inserting placeholder vectors. Log and alert on partial failures.
4
Diagnostic step 4
Enable strong consistency mode during updates: quiesce writers, run a validated incremental commit, then resume queries. For immediate mitigation, run a full reindex to known-good state and schedule maintenance window for incremental workflow fixes.
5
Diagnostic step 5
If you suspect FAISS corruption or a library bug, upgrade FAISS to a maintained release and test incremental updates in a staging environment; collect indexer and search service logs and stack traces and escalate to the vector DB/library maintainers if index corruption recurs.