1
Diagnostic step 1
Immediately put the vector service into read-only mode (if supported) to prevent further writes that could complicate recovery. Take a full storage and metadata snapshot (filesystem-level snapshot) before any repairs.
FULL AI ANALYSIS
Incremental vector index job leaves stale vectors and duplicate document IDs — RAG answers cite deleted passages
Generated Oct 15, 2026, 9:17:34 AM
Analysis summary
Evidence strongly suggests the interrupted indexing/compaction created inconsistent state between on-disk segments and the index manifest: partial writes left older segments still referenced, while new segment files may have incomplete metadata. The controller marking the job as successful despite partial completion points to an instrumentation or error-handling gap. Targeted remediation is possible: isolate and mark suspect segments read-only, rebuild or merge individual segments, or point queries to the authoritative segment set after a manifest repair. Before any write, snapshot storage and take cluster read-only to prevent further divergence.
Possible causes
Interrupted incremental indexing/compaction (process OOM or crash) left partially written segments and the manifest in an inconsistent state, exposing stale segments to queries.
85%Index manifest update race: system updated segment listings before durable rename/fsync of new segment files, so catalog points to inconsistent file versions.
60%Garbage collection or compaction bug that failed to delete old segments after merging, leaving duplicate doc IDs across segments.
45%Client-side caching layer (API cache or CDN) returning stale search results even after backend repair attempts.
30%Suggested fixes
2
Diagnostic step 2
Run the vector DB's integrity/consistency check tool (or vendor-provided validation script) to list segments whose manifest entries don't match file checksums or doc counts.
3
Diagnostic step 3
If the DB supports it, mark suspect segments as offline and re-run search to confirm whether results change; identify the minimum set of segments causing stale results.
4
Diagnostic step 4
On a staging cluster restored from the current production snapshot, attempt targeted segment repair/merge for the affected ranges to validate that a partial remerge will clear duplicates without full reindex.
5
Diagnostic step 5
If targeted repair succeeds on staging, apply the same steps to production during a maintenance window; otherwise schedule a full reindex and update monitoring/alerting to fail incremental jobs loudly on partial completion.