Gradient Boosting and XGBoost for Actuarial Modeling

Boosting Concept

Boosting builds an ensemble of weak learners (typically shallow trees) sequentially. Each new tree is trained on the residuals (or a pseudo-residual) of the current ensemble, focusing on observations that previous trees predicted poorly. The final prediction is a weighted sum of all trees. Unlike bagging (which reduces variance), boosting primarily reduces bias by gradually "learning" the patterns that earlier models missed.

The learning rate (shrinkage parameter, typically 0.01 to 0.3) scales each tree's contribution. Smaller learning rates require more trees but generally produce better results.

Gradient Boosting

Gradient boosting generalizes boosting by fitting each new tree to the negative gradient of the loss function. For squared error loss, the negative gradient is the residual. For other loss functions (absolute error, deviance for classification), it is a different quantity. This framework unifies boosting for regression, classification, and other tasks.

Key hyperparameters: number of trees (B), learning rate (lambda), tree depth (d, often 1 to 6), and minimum samples per leaf. These interact: more trees with a smaller learning rate is generally better, but slower to train.

XGBoost and Actuarial Use

XGBoost (Extreme Gradient Boosting) adds regularization penalties on tree complexity and uses second-order gradient information for faster optimization. It handles missing values natively, supports parallel processing, and includes built-in cross-validation. In actuarial science, gradient boosting models are widely used for claim frequency and severity modeling, competing with GLMs. For Exam SRM, understand the conceptual differences between bagging and boosting, the role of hyperparameters, and why gradient boosting is prone to overfitting without proper tuning (controlled by the learning rate, tree depth, and number of trees).