Random Forests and Ensemble Methods for Exam SRM

Bagging

Bootstrap aggregating (bagging) reduces variance by training multiple models on bootstrap samples and averaging their predictions. For B bootstrap samples, train B trees and take the average (regression) or majority vote (classification). Bagging works because averaging reduces variance while keeping bias roughly the same. It is most effective for high-variance, low-bias models like deep decision trees.

Out-of-bag (OOB) observations (those not in a bootstrap sample, about 1/3 of data) provide a built-in estimate of test error, reducing the need for a separate validation set.

Random Forests

Random forests improve on bagging by decorrelating the trees. At each split, only a random subset of m predictors (out of p total) is considered. Typical choices: m = sqrt(p) for classification, m = p/3 for regression. By forcing each tree to consider different predictors, the trees become less correlated, and averaging them produces a larger variance reduction.

Random forests generally outperform single trees and bagging. They are robust to hyperparameter choices and require minimal tuning. The main downside is loss of interpretability compared to a single tree.

Variable Importance

Random forests measure variable importance in two ways: (1) the total decrease in impurity (Gini or RSS) attributed to splits on each variable, averaged across all trees, and (2) the permutation importance, which measures the increase in OOB error when a variable's values are randomly permuted. For Exam SRM, understand that random forests sacrifice interpretability for prediction accuracy and that variable importance measures help identify the most influential predictors.