Bias-Variance Tradeoff in Actuarial Predictive Models

Decomposition of Prediction Error

For a given observation, the expected prediction error (mean squared error) decomposes as: E[(Y - f_hat(X))^2] = Var(f_hat(X)) + [Bias(f_hat(X))]^2 + Var(epsilon). The irreducible error Var(epsilon) cannot be reduced regardless of the model. Bias measures how far the average prediction is from the truth. Variance measures how much predictions vary across different training sets.

Simple models (e.g., linear regression with few predictors) have high bias and low variance. Complex models (e.g., deep decision trees, high-degree polynomials) have low bias but high variance.

The Tradeoff

As model complexity increases, bias decreases (the model can capture more complex patterns) but variance increases (the model fits noise in the training data). The optimal model minimizes the total test error, which is the sum of squared bias, variance, and irreducible error. The test error curve is U-shaped as a function of complexity.

Overfitting occurs when the model is too complex: it fits the training data well but generalizes poorly. Underfitting occurs when the model is too simple: it misses important patterns.

Managing the Tradeoff

Techniques to manage the tradeoff include: cross-validation (directly estimates test error to guide complexity choice), regularization (penalizes complexity to reduce variance at the cost of a small increase in bias), ensemble methods like bagging (reduces variance without much bias increase) and boosting (reduces bias with controlled variance), and early stopping in iterative algorithms. For Exam SRM, the bias-variance tradeoff is the unifying concept behind model selection. Every method for choosing among models attempts to find the sweet spot that minimizes the total prediction error.