Vector search returns semantically unrelated high-similarity passages after incremental index update

Environment

Self-hosted vector index (open-source vector DB v0.9) with 1.4M documents, embeddings produced by local sentence-transformer model v1.2. Incremental index updates performed nightly using a Python ETL that upserts new embeddings and periodically re-serializes index segments.

Symptoms

After a recent incremental index update, RAG system returns passages with very high vector similarity (reported similarity scores near recent top threshold) that are nonetheless semantically unrelated to the query; users see confidently wrong context inserted into answers. Prior nightly full rebuilds did not exhibit the issue.

Observations

Reproduced locally: same query before update returned correct top-k; after incremental update, top-k includes unrelated documents that have very similar embedding vectors (near-duplicates by cosine). Inspection of affected vectors shows multiple documents sharing identical or near-identical embedding vectors after upsert. ETL logs show occasional retry/partial-write events during overnight job. No hardware errors reported. Incremental indexing uses append-only segment files and a merge job.

Already attempted

Validated embedding generation pipeline separately — embedding model run over suspect documents produced distinct vectors locally. Re-ran upsert for a subset of documents and observed their stored vectors in DB did not match freshly computed vectors (stored vectors appear duplicated across many docIDs). Performed a full rebuild in a test cluster and the issue disappeared.

Question

What are the probable root causes that lead to stored vectors becoming duplicated or mismatched during incremental upserts, and what safe remediation and prevention steps should be taken for production RAG systems?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1d ago• Trouble ID

Most likely causes

Vector DB incremental upsert/merge bug causing pointers to wrong vector payloads or duplicate vector entries during segment merger 82%

ETL job retry/partial-write behavior writing malformed placeholder vectors that were not overwritten during merge 60%

Serialization/version mismatch between embedding encoder format and vector DB expected format after recent model version bump 35%

Summary

Key evidence: stored vectors diverge from freshly computed embeddings; issue appears only after incremental upserts and disappears after full rebuild. This suggests a bug in the index write/merge code path: either upsert writes wrong offsets (indexing bug), a serialization/deserialization mismatch during partial writes, or ETL job retries wrote placeholder/empty vectors that were later merged without replacement. Less likely are model drift or embedding collisions because local re-embedding produced distinct vectors. The most probable root is index storage bug or upsert race/partial-write during incremental merge.

Suggested Fixes

Diagnostic step 1

Take a read-only snapshot of the current index files and application logs for vendor/maintainer analysis; do not delete index files.

Diagnostic step 2

In a staging environment, reproduce the incremental upsert and enable index debug logging (upsert/merge trace) to capture sequence of writes and merges. Compare stored vector bytes to freshly computed embedding bytes per document to identify where mismatch occurs.

Diagnostic step 3

If ETL shows retries/partial writes, add atomic upsert semantics or write-ahead/temp-file pattern: write new segment to temp location then atomically swap so partial writes cannot be merged.