A decision
Name the action a prediction could change and the person who owns it.
Predict for a decision. Measure the cost of being wrong.
Evaluate whether a model can improve a real choice, with the data, capacity and error trade-offs made visible.
Illustrative support decision
Waiting timeRepeat contactsIssue history
Risk ranking
Review queueCapacity + cost of missed cases
Tools & technologies
- Python
- scikit-learn
- PyTorch
- TensorFlow
- SQL
- Model evaluation and monitoring
First, establish whether a model is worth using.
A baseline
Compare the current rule or manual approach before adding model complexity.
An outcome
Check that historical labels reflect what will be known at the actual decision time.
More cases reviewed. Fewer cases missed?
Move the review threshold on a small fictional evaluation set and see the trade-off.
Interactive exampleSupport review threshold
Set the review threshold
Eight fictional historical cases with fixed ranking scores and known outcomes. No model runs here.
- Sent to review
- 4
- Escalations found
- 2
- Escalations missed
- 2
- Unneeded reviews
- 2
Case / scoreObserved outcome
- T-010.92EscalatedReview
- T-020.80Did not escalateReview
- T-030.74EscalatedReview
- T-040.65Did not escalateReview
- T-050.52EscalatedNot reviewed
- T-060.30Did not escalateNot reviewed
- T-070.20EscalatedNot reviewed
- T-080.10Did not escalateNot reviewed
4 cases sent to review: 2 escalations found, 2 missed and 2 unneeded reviews.
A useful evaluation needs an operating plan.
Before release
Test against a held-out period, relevant groups and the agreed baseline.
During use
Track data changes, outcomes and review capacity alongside prediction quality.
When it degrades
Define a fallback and a decision owner before changing or retraining the model.
Evidence and the means to reproduce it.
Decision & baseline definition
Dataset and feature assessment
Reproducible evaluation
Threshold & review policy
Model integration package
Monitoring and retraining plan
Customer data and AI analysis
A few practical questions.
Can you guarantee model accuracy?
No. Feasibility depends on the data, outcome definition and changing conditions. Evaluation should compare an agreed baseline and report relevant errors, not promise a universal accuracy figure.
How much historical data do we need?
There is no useful universal minimum. We assess coverage, outcome labels, missing cases, leakage and how closely the history matches the intended use.
Does a prediction automatically trigger action?
Only if that is explicitly designed and accepted. A score can instead prioritise a review queue, with human decisions and outcomes recorded for evaluation.
What happens when performance changes?
We define monitoring, review ownership and fallback conditions. Retraining, threshold changes and release approval are governed by the agreed operating plan.
What would a prediction change?
Bring the decision, examples of past outcomes and the cost of a missed or unnecessary action.
