Variable Selection Methods: Forward, Backward, Stepwise for Exam SRM

Forward Selection

Forward selection starts with no predictors and adds them one at a time. At each step, it tests all predictors not yet in the model and adds the one with the smallest p-value (most significant), provided that p-value is below a threshold (e.g., alpha = 0.05 or 0.15). The process stops when no remaining predictor meets the entry criterion.

Advantages: computationally efficient, works well when the true model is sparse. Disadvantage: once a variable enters, it stays, even if it becomes insignificant after other variables are added.

Backward Elimination

Backward elimination starts with all predictors and removes them one at a time. At each step, it identifies the predictor with the largest p-value (least significant) and removes it if the p-value exceeds a threshold. The process stops when all remaining predictors are significant.

Advantages: considers all predictors simultaneously from the start, can detect suppressor effects. Disadvantage: requires n > p (more observations than predictors), computationally intensive for large p.

Stepwise Selection and Alternatives

Stepwise selection combines forward and backward: after adding a variable, it checks whether any previously entered variable should be removed. This addresses the main weakness of pure forward selection. However, all three methods share limitations: they do not guarantee finding the best subset, they inflate Type I error rates due to multiple testing, and the selected model depends on arbitrary p-value thresholds. For Exam SRM, understand these methods and their limitations. Modern alternatives include information criteria (AIC, BIC), cross-validation, and regularization (LASSO), which are covered in other topics.