Confusion Matrix, ROC Curves, and AUC for Model Evaluation
Master classification model evaluation metrics for Exam SRM.
Confusion Matrix
For a binary classifier, the confusion matrix tabulates predictions against actual outcomes: True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN). Key metrics derived from it: Accuracy = (TP + TN) / (TP + TN + FP + FN). Sensitivity (Recall, True Positive Rate) = TP / (TP + FN). Specificity (True Negative Rate) = TN / (TN + FP). Precision = TP / (TP + FP). F1 Score = 2 * Precision * Recall / (Precision + Recall).
In insurance, false negatives (missed fraudulent claims) and false positives (wrongly flagged legitimate claims) have different costs, so accuracy alone is often insufficient.
ROC Curve
The Receiver Operating Characteristic (ROC) curve plots Sensitivity (True Positive Rate) on the y-axis against 1 - Specificity (False Positive Rate) on the x-axis for all possible classification thresholds. A perfect classifier hugs the top-left corner. A random classifier follows the diagonal. The ROC curve is threshold-independent, making it useful for comparing models regardless of the chosen cutoff.
The Area Under the ROC Curve (AUC) summarizes discrimination ability. AUC = 1 is perfect; AUC = 0.5 is no better than random. AUC between 0.7 and 0.8 is considered acceptable; above 0.8 is good; above 0.9 is excellent.
Practical Considerations
AUC measures ranking ability (whether the model assigns higher probabilities to positive cases), not calibration (whether predicted probabilities match actual frequencies). A well-calibrated model with AUC 0.75 may be more useful than a poorly calibrated model with AUC 0.80. For Exam SRM, understand how to construct and interpret ROC curves, calculate AUC, and recognize the limitations of each metric, particularly in imbalanced datasets where accuracy can be misleading.