Linear Regression: Assumptions, Estimation, and Inference for Exam SRM

The Simple Linear Regression Model

Simple linear regression models the relationship Y = beta_0 + beta_1*X + epsilon, where epsilon ~ N(0, sigma^2) independently. The four key assumptions are: linearity (the true relationship is linear), independence (errors are independent), normality (errors are normally distributed), and constant variance (homoscedasticity: Var(epsilon) = sigma^2 for all X). These are often abbreviated as LINE.

The least squares estimators minimize the sum of squared residuals: beta_1_hat = sum((Xi - X_bar)(Yi - Y_bar)) / sum((Xi - X_bar)^2) = S_xy / S_xx, and beta_0_hat = Y_bar - beta_1_hat * X_bar.

Properties of Estimators

Under the model assumptions, the OLS estimators are unbiased: E[beta_1_hat] = beta_1 and E[beta_0_hat] = beta_0. They are the Best Linear Unbiased Estimators (BLUE) by the Gauss-Markov theorem. Var(beta_1_hat) = sigma^2 / S_xx and Var(beta_0_hat) = sigma^2 * (1/n + X_bar^2/S_xx). The unbiased estimator of sigma^2 is MSE = SSE/(n-2), where SSE = sum of (Yi - Y_hat_i)^2.

Inference

To test H0: beta_1 = 0, use the t-statistic t = beta_1_hat / SE(beta_1_hat), which follows a t-distribution with n-2 degrees of freedom under H0. Reject H0 if |t| > t_{alpha/2, n-2}. The confidence interval for beta_1 is beta_1_hat +/- t_{alpha/2, n-2} * SE(beta_1_hat). The coefficient of determination R^2 = 1 - SSE/SST = (correlation)^2 measures the proportion of variance in Y explained by X. For Exam SRM, understand both the computation and the interpretation of these quantities.