Decision Trees: Classification and Regression Trees for Actuaries

How Decision Trees Work

A decision tree recursively partitions the predictor space into regions and assigns a prediction to each region. For regression trees, the prediction is the mean of Y in each region. For classification trees, it is the most common class. At each step, the algorithm selects the predictor and split point that best improves a criterion: reduction in RSS (sum of squared residuals) for regression, or reduction in impurity for classification.

The result is a tree where internal nodes represent split conditions (e.g., "Age > 35") and leaf nodes represent predictions. Trees are intuitive and easy to interpret, making them popular in insurance underwriting.

Impurity Measures

For classification, common impurity measures include the Gini index: G = sum of p_k * (1 - p_k) over all classes k, and cross-entropy (deviance): D = -sum of p_k * ln(p_k). Both equal zero when a node is pure (all observations in one class) and are maximized when classes are equally represented. The misclassification rate is not recommended for tree growing because it is less sensitive to changes in class probabilities, though it may be used for pruning.

Pruning and Overfitting

A fully grown tree (splitting until each leaf has very few observations) overfits the training data. Pruning reduces the tree by removing splits that do not improve predictive accuracy on new data. Cost-complexity pruning adds a penalty alpha * |T| (where |T| is the number of terminal nodes) to the training error. The optimal alpha is chosen by cross-validation. The result is a simpler tree that balances bias and variance. For Exam SRM, understand how pruning controls overfitting and how cross-validation selects the optimal tree size.