Bias-Variance Tradeoff

The principle that reducing model bias (by adding complexity) tends to increase variance, and vice versa. The optimal model minimizes total prediction error, which is bias squared plus variance plus irreducible error.

Generalized Linear Model (GLM)

A regression model where the response follows a distribution from the exponential family and a link function connects the mean to a linear predictor. Includes logistic regression, Poisson regression, and Gamma regression.

Link Function

In a GLM, the function g that relates the expected response to the linear predictor: g(mu) = X*beta. Common link functions include identity (Gaussian), log (Poisson), and logit (Binomial).

AIC (Akaike Information Criterion)

AIC = -2*log-likelihood + 2p, where p is the number of parameters. Lower AIC indicates a better balance between fit and complexity. Used for model selection among non-nested models.

BIC (Bayesian Information Criterion)

BIC = -2*log-likelihood + p*ln(n), where p is the number of parameters and n is the sample size. Penalizes complexity more heavily than AIC, favoring simpler models in larger samples.

Cross-Validation

A model evaluation technique that partitions data into training and validation sets. K-fold CV divides data into k groups, trains on k-1 folds, and validates on the held-out fold, rotating through all k folds.

Principal Component Analysis (PCA)

A dimension reduction technique that finds orthogonal linear combinations of the original variables that capture maximum variance. The first principal component explains the most variance, the second the next most, and so on.

Decision Tree

A non-parametric model that recursively splits the predictor space into regions, predicting a constant value in each region. Classification trees predict class labels; regression trees predict continuous values.

Random Forest

An ensemble method that fits many decision trees on bootstrapped samples, using a random subset of predictors at each split. The final prediction averages across all trees, reducing variance compared to a single tree.

ARIMA Model

AutoRegressive Integrated Moving Average. A time series model that combines autoregressive terms (past values), differencing (to achieve stationarity), and moving average terms (past errors). Specified as ARIMA(p,d,q).

Deviance

A measure of goodness of fit in GLMs. It compares the fitted model to the saturated model (a model with as many parameters as observations). Analogous to the residual sum of squares in linear regression.

Gini Impurity

A measure of node impurity in classification trees. For a node with class proportions p_1, p_2, ..., p_K, the Gini impurity is 1 - sum(p_k^2). A pure node has Gini = 0.

SRM Glossary