Incremental vector index job leaves stale vectors and duplicate document IDs — RAG answers cite deleted passages

Environment

Self-hosted vector database cluster (disk-backed segment storage), production RAG pipeline uses periodic incremental update jobs that embed new documents and merge segments in background. Incremental job was interrupted by an OOM in indexer process, then the controller marked job as succeeded after partial recovery.

Symptoms

Search queries return passages that were deleted weeks ago; top-K results include duplicate document IDs with slightly different embedding scores. Downstream RAG responses quote old text that should no longer exist. Users report inconsistent answers between identical queries seconds apart.

Observations

Index metadata (manifest) reports a larger document count than the authoritative document store. Segment files on disk show overlapping doc ID ranges and multiple segment manifests referencing the same doc ID. Compaction logs indicate 'partial segment write' at the time the indexer OOMed. Service restart shows the search API still uses the stale segments until a full reindex is done. Attempts to re-run the incremental job reported 'no updates' for the same time window.

Already attempted

Restarted vector service and re-ran incremental update; service returned success but results unchanged. Restored a snapshot from 24 hours earlier to a staging cluster and re-ran the job — staging showed correct results after targeted repair. Full reindex on prod estimated to take multiple days and was deferred.

Question

What are the most likely root causes for stale vectors and duplicate IDs after an interrupted incremental update/compaction job, and what reversible integrity checks or remediation can I run to avoid a full reindex immediately?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 2h ago• Trouble ID

Most likely causes

Interrupted incremental indexing/compaction (process OOM or crash) left partially written segments and the manifest in an inconsistent state, exposing stale segments to queries. 85%

Index manifest update race: system updated segment listings before durable rename/fsync of new segment files, so catalog points to inconsistent file versions. 60%

Garbage collection or compaction bug that failed to delete old segments after merging, leaving duplicate doc IDs across segments. 45%

Summary

Evidence strongly suggests the interrupted indexing/compaction created inconsistent state between on-disk segments and the index manifest: partial writes left older segments still referenced, while new segment files may have incomplete metadata. The controller marking the job as successful despite partial completion points to an instrumentation or error-handling gap. Targeted remediation is possible: isolate and mark suspect segments read-only, rebuild or merge individual segments, or point queries to the authoritative segment set after a manifest repair. Before any write, snapshot storage and take cluster read-only to prevent further divergence.

Suggested Fixes

Diagnostic step 1

Immediately put the vector service into read-only mode (if supported) to prevent further writes that could complicate recovery. Take a full storage and metadata snapshot (filesystem-level snapshot) before any repairs.

Diagnostic step 2

Run the vector DB's integrity/consistency check tool (or vendor-provided validation script) to list segments whose manifest entries don't match file checksums or doc counts.

Diagnostic step 3

If the DB supports it, mark suspect segments as offline and re-run search to confirm whether results change; identify the minimum set of segments causing stale results.