Artificial Intelligence
Local LLM slows down dramatically after a long conversation
troublesync_editorial
2w ago
Environment
Environment
Quantized 13B model running on a 12 GB GPU with a growing chat context.
What happens
What happens
Generation starts near 25 tokens per second and falls below 3 after many messages.
What I have tried
What I have tried
Restarted the UI and changed temperature without affecting the slowdown.
I would appreciate advice on how to identify the root cause without risking existing data or configuration.
COMMUNITY RESPONSES
Verified community troubleshooting
0 replies
No answers yet..
Showing 0 of 0 community responses
AI DIAGNOSTICS
Incident analysis
Most likely causes
The key-value cache outgrows available VRAM and layers spill to system memory. 87%
A recent software, firmware or configuration change is exposing an existing compatibility issue. 39%
Summary
The reported behaviour is consistent with the key-value cache outgrows available VRAM and layers spill to system memory. The safest next step is to verify that hypothesis with a reversible test before making permanent configuration changes.
Suggested Fixes
Verify the likely cause
Reduce context length, summarize older turns, use a smaller quantization and monitor VRAM allocation while reproducing the slowdown.
Validate and monitor
Reproduce the issue once after the change, review relevant logs or metrics, and keep a rollback path until the system has remained stable.