Evaluating Models
How do we know if a model is any good? Test it fairly, count its mistakes and choose the right score.
1Why evaluate a model?
Evaluation checks how well a model's predictions match reality, using data the model has never seen. It tells us whether we can trust the model before we put it to real use.
● training days (the model learned from these) ◆ test days (kept hidden from the model)
Predicting ice-cream sales from temperature. Make the model more flexible and watch its error on the days it learned from. Then reveal days it has never seen.
2Train-test split
Before training, we split the data into a training set, which the model learns from, and a test set, which we hide and use only to evaluate it. A common split is 80:20.
■ passed ■ failed Each square is one student's record: hours studied and result.
Will a student pass? The model learns one rule, "pass if hours studied ≥ some number", from the training set. Then we check it on the test set.
3Accuracy and error
Accuracy tells us how close the model's predictions are to the truth. Error tells us how far off a prediction is. For number predictions, error = |actual − predicted|, error rate = error ÷ actual, and accuracy = 1 − error rate.
4The confusion matrix
For classification, we compare predictions and actual answers in a 2 × 2 table called a confusion matrix. Each test case falls into one box: True Positive, False Positive, False Negative or True Negative.
A rain predictor was tested for 10 days. Each card shows what the model predicted and what actually happened. Tap a card, then tap its box in the confusion matrix.
5Precision, recall and F1 score
Accuracy is not always enough. Precision asks: of everything the model flagged, how much was right? Recall asks: of everything that should have been flagged, how much did the model find? The F1 score balances the two.
A heart disease check. The model gives each person a risk score from 0 to 1. Everyone to the right of the threshold is predicted to have heart disease. Drag the threshold.
● really has heart disease ● really healthy ringed = wrong prediction
6Which metric should we use?
The right metric depends on which mistake costs more. If false positives are costly, focus on precision. If false negatives are costly, focus on recall. Use accuracy for balanced data where both errors matter equally, and F1 for unbalanced data when you need both.
7Ethical concerns in model evaluation
A model can score well overall and still be unfair to some people. While evaluating, we must watch for bias, be transparent about how the model was tested, and be clear about who is accountable for its decisions.
■ speaks with Accent A ● speaks with Accent B green = understood red = misunderstood
A voice assistant is tested on 100 people. 80 speak with Accent A and 20 with Accent B.