Artificial Intelligence
Fine-tuned classifier performs well offline but fails in production
troublesync_editorial
2w ago
Environment
Environment
Text classifier trained on historical support tickets and deployed behind an API.
What happens
What happens
Validation accuracy is 94%, but recent tickets are frequently assigned to the wrong queue.
What I have tried
What I have tried
Retrained with a different random split and increased training epochs.
I would appreciate advice on how to identify the root cause without risking existing data or configuration.
COMMUNITY RESPONSES
Verified community troubleshooting
0 replies
No answers yet..
Showing 0 of 0 community responses
AI DIAGNOSTICS
Incident analysis
Most likely causes
The random split leaks historical patterns and production language has drifted over time. 72%
A recent software, firmware or configuration change is exposing an existing compatibility issue. 40%
Summary
The reported behaviour is consistent with the random split leaks historical patterns and production language has drifted over time. The safest next step is to verify that hypothesis with a reversible test before making permanent configuration changes.
Suggested Fixes
Verify the likely cause
Evaluate with a time-based holdout, inspect errors by product version, add recent labeled data and monitor class-level drift.
Validate and monitor
Reproduce the issue once after the change, review relevant logs or metrics, and keep a rollback path until the system has remained stable.