Regularization: Ridge, Lasso, and Elastic Net for Exam SRM

Ridge Regression

Ridge regression adds an L2 penalty to the OLS objective: minimize sum(Yi - X_i*beta)^2 + lambda * sum(beta_j^2). The penalty shrinks coefficients toward zero but never sets them exactly to zero. The solution is beta_ridge = (X'X + lambda*I)^(-1) * X'Y. As lambda increases, coefficients shrink more, reducing variance but increasing bias. lambda = 0 gives OLS; lambda = infinity gives all coefficients equal to zero.

Ridge is especially useful when there are many correlated predictors (multicollinearity), as it stabilizes the coefficient estimates.

Lasso Regression

The Lasso (Least Absolute Shrinkage and Selection Operator) uses an L1 penalty: minimize sum(Yi - X_i*beta)^2 + lambda * sum(|beta_j|). The L1 penalty can shrink coefficients exactly to zero, performing automatic variable selection. This makes the Lasso produce sparse models that are easier to interpret. However, when predictors are highly correlated, the Lasso tends to select one and zero out the others, which may be undesirable.

There is no closed-form solution; the Lasso is solved using coordinate descent or other optimization algorithms.

Elastic Net

Elastic net combines the L1 and L2 penalties: minimize sum(Yi - X_i*beta)^2 + lambda_1 * sum(|beta_j|) + lambda_2 * sum(beta_j^2). It balances the variable selection property of the Lasso with the stability of ridge regression. The mixing parameter alpha controls the proportion: alpha = 1 gives Lasso, alpha = 0 gives ridge. For Exam SRM, understand the differences in behavior: ridge shrinks but does not select; Lasso selects but may be unstable with correlated predictors; elastic net offers a compromise. The tuning parameters (lambda and alpha) are chosen by cross-validation.