Hypothesis Testing Review for Exam SRM

Framework

A hypothesis test evaluates evidence against a null hypothesis H0 in favor of an alternative H1. The test statistic measures how far the observed data deviates from what H0 predicts. The p-value is the probability, under H0, of observing a test statistic at least as extreme as the one obtained. Reject H0 if the p-value < alpha (the significance level, typically 0.05).

Type I error: rejecting H0 when it is true (probability = alpha). Type II error: failing to reject H0 when it is false (probability = beta). Power = 1 - beta = P(reject H0 | H0 is false). Power increases with sample size, effect size, and alpha.

Common Tests for Exam SRM

t-test for a regression coefficient: t = beta_hat / SE(beta_hat), with n-p-1 degrees of freedom. Tests whether a predictor has a significant linear relationship with Y. F-test for overall regression significance or comparing nested models. Chi-squared test for goodness of fit or comparing categorical variables. Likelihood ratio test: G^2 = -2(L_0 - L_1) ~ chi-squared with df = difference in parameters. Used extensively with GLMs.

Multiple Testing Issues

When conducting many tests simultaneously (e.g., testing each coefficient in a regression), the probability of at least one false positive increases. With m independent tests at level alpha, the family-wise error rate is 1 - (1-alpha)^m. The Bonferroni correction adjusts by testing each at alpha/m, controlling the family-wise error rate at alpha. However, Bonferroni is conservative (low power). For Exam SRM, be aware that stepwise variable selection involves multiple tests and inflates Type I error rates, which is one motivation for using AIC/BIC or cross-validation instead.