Cluster Analysis: K-Means and Hierarchical Clustering

K-Means Clustering

K-means partitions n observations into K clusters by minimizing the within-cluster sum of squares: sum over all clusters of sum over observations in that cluster of ||x_i - mu_k||^2, where mu_k is the centroid of cluster k. The algorithm alternates between assigning each observation to the nearest centroid and recomputing centroids. It converges to a local minimum (not necessarily the global minimum), so multiple random starts are recommended.

Choosing K: the elbow method plots the within-cluster SS against K and looks for a bend. The silhouette score measures how well each observation fits its cluster versus the next-best cluster.

Hierarchical Clustering

Hierarchical clustering builds a tree (dendrogram) of nested clusters without specifying K in advance. Agglomerative (bottom-up) clustering starts with each observation as its own cluster and merges the closest pair at each step. Divisive (top-down) starts with one cluster and splits. The linkage method determines how inter-cluster distance is measured: single linkage (minimum distance), complete linkage (maximum distance), average linkage, or Ward's method (minimum increase in total within-cluster variance).

Cut the dendrogram at a chosen height to obtain K clusters. Ward's method tends to produce compact, equally sized clusters and is commonly used.

Actuarial Applications

Clustering is used in insurance for market segmentation, risk classification, identifying groups of similar policyholders, and detecting fraud. For Exam SRM, understand the algorithms, how to choose K, the impact of scaling variables (always standardize before clustering), and the strengths and weaknesses of each method. K-means is efficient but assumes spherical clusters; hierarchical clustering is more flexible but slower for large datasets.