Cross-Validation Techniques: k-Fold and Leave-One-Out
Master cross-validation for model selection and assessment on Exam SRM.
Why Cross-Validation?
Cross-validation estimates a model's prediction error on new data without requiring a separate test set. Training error is typically too optimistic (it decreases as model complexity increases), while test error follows a U-shape. Cross-validation approximates the test error to guide model selection. It is the standard approach for comparing models, tuning hyperparameters, and assessing predictive performance.
k-Fold Cross-Validation
In k-fold CV, the data is randomly divided into k roughly equal folds. For each fold i = 1, ..., k: train the model on the other k-1 folds and compute the prediction error on fold i. The CV estimate of test error is the average of the k fold errors. Common choices: k = 5 or k = 10. Larger k gives less bias but more variance in the error estimate and is computationally more expensive.
The CV error for each candidate model (or hyperparameter setting) is computed, and the model with the lowest CV error is selected. The "one standard error rule" selects the simplest model within one standard error of the minimum CV error.
Leave-One-Out Cross-Validation
Leave-one-out CV (LOOCV) is k-fold CV with k = n: each observation is held out once. It is approximately unbiased but has high variance because the n training sets are nearly identical. For linear models, LOOCV can be computed efficiently using the hat matrix: CV = (1/n) * sum of (e_i / (1 - h_ii))^2. For Exam SRM, understand the bias-variance tradeoff in choosing k: LOOCV has low bias but high variance; small k (e.g., 5) has higher bias but lower variance.