Unsupervised Learning Algorithms

Unsupervised learning finds structure in data without any labels. There is no "right answer" column. The model must infer groups, compressed representations, or anomalous patterns from the distribution of the input data alone.

Shared Setup (Run Once)

1import numpy as np
2import matplotlib.pyplot as plt
3from sklearn.datasets import make_blobs, make_classification
4from sklearn.preprocessing import StandardScaler
5from sklearn.metrics import silhouette_score
6
7# Clustering dataset
8X_cluster, _ = make_blobs(
9    n_samples=1500, centers=4,
10    cluster_std=1.2, random_state=42
11)
12X_cluster = StandardScaler().fit_transform(X_cluster)
13
14# High-dimensional dataset for reduction
15X_high, y_high = make_classification(
16    n_samples=2000, n_features=30,
17    n_informative=10, n_redundant=10,
18    n_classes=3, random_state=42
19)
20X_high = StandardScaler().fit_transform(X_high)
21
22# Anomaly detection dataset
23rng = np.random.RandomState(42)
24inliers  = rng.normal(0.0, 1.0, size=(1000, 2))
25outliers = rng.uniform(-8, 8, size=(50, 2))
26X_anom   = np.vstack([inliers, outliers])
27y_anom   = np.hstack([np.ones(1000), -np.ones(50)])

1. K-Means Clustering

Imagine dropping 4 magnets on a map of cities. Each city jumps to the nearest magnet. Then each magnet moves to the centre of all its attracted cities. You repeat this until no city switches magnets. K-Means does exactly this — the magnets are the cluster centres and the cities are your data points.

What it does: Partitions data into K clusters by alternating between two steps: (1) assigning each point to the nearest cluster centroid, (2) recomputing centroids as the mean of all assigned points. Repeats until assignments stop changing.

When to use it:

• Customer segmentation on behavioral or demographic data.
• Vector quantization for signal compression.
• Fast initialization for more complex clustering methods.

When it fails:

• Assumes spherical clusters of roughly equal size — fails on elongated or irregular cluster shapes.
• Sensitive to initialization (use k-means++ via init="k-means++", the default).
• K must be specified in advance.
• Trapped in local minima — run with n_init=20.

Math — Objective (Within-Cluster Inertia): J = Σ_k Σ_{x∈C_k} ||x - μ_k||²

Alternating steps:

• Assignment: C_k = {x_i : k = argmin_j ||x_i - μ_j||}
• Update: μ_k = (1/|C_k|) Σ_{x∈C_k} x

Choosing K: Use the Elbow method (inertia vs K) or Silhouette score.

1from sklearn.cluster import KMeans
2
3# Elbow curve
4inertias = []
5K_range = range(2, 10)
6for k in K_range:
7    km = KMeans(n_clusters=k, n_init=20, random_state=42)
8    km.fit(X_cluster)
9    inertias.append(km.inertia_)
10
11plt.plot(K_range, inertias, marker="o")
12plt.xlabel("Number of clusters K")
13plt.ylabel("Inertia")
14plt.title("Elbow Method for Optimal K")
15plt.show()
16
17# Fit with optimal K
18kmeans = KMeans(n_clusters=4, n_init=20, random_state=42)
19labels = kmeans.fit_predict(X_cluster)
20print("K-Means silhouette:", round(silhouette_score(X_cluster, labels), 4))

2. Hierarchical (Agglomerative) Clustering

Picture every student in a school standing alone. The two who are most similar shake hands and form a pair. Then the two closest pairs (or individuals) merge into a group. Keep merging until everyone is in one big circle. The entire history of merges is recorded as a family tree — the dendrogram. You can cut that tree at any height to get however many clusters you need.

What it does: Starts with every data point as its own cluster. Iteratively merges the two closest clusters until the desired number of clusters remains. The merge history is stored as a dendrogram — a tree structure you can cut at any level.

When to use it:

• When you do not know K in advance (cut the dendrogram at the desired level).
• Exploring cluster hierarchy (e.g., biological taxonomy, product category trees).
• Smaller datasets where the O(n³) time complexity is acceptable.

When it fails: Merges cannot be undone — early mistakes cascade. Very slow on large datasets.

Linkage Criteria:

1from sklearn.cluster import AgglomerativeClustering
2
3agg = AgglomerativeClustering(n_clusters=4, linkage="ward")
4labels_agg = agg.fit_predict(X_cluster)
5print("Agglomerative silhouette:", round(silhouette_score(X_cluster, labels_agg), 4))

3. DBSCAN (Density-Based Spatial Clustering)

At a crowded school fair, students gather in tight groups around different stalls. A few loners wander in the gaps between groups. DBSCAN identifies the groups by looking at how many people are close together — dense regions become clusters, loners are labelled noise. Unlike K-Means, you never have to guess the number of groups in advance.

What it does: Groups points that are closely packed together and marks isolated sparse points as noise. Does not require you to specify K.

When to use it:

• When clusters have irregular, non-convex shapes.
• Geospatial hotspot detection.
• When identifying and removing noise/outliers during preprocessing.

When it fails: Struggles with clusters of very different densities (same eps doesn't fit all). Performance is sensitive to the eps and min_samples settings.

Core Concepts:

• Core point: has ≥ min_samples neighbors within radius eps
• Border point: within eps of a core point, but not a core point itself
• Noise point: not within eps of any core point — labeled -1

1from sklearn.cluster import DBSCAN
2
3dbscan = DBSCAN(eps=0.5, min_samples=8)
4labels_db = dbscan.fit_predict(X_cluster)
5
6valid = labels_db != -1
7n_clusters = len(set(labels_db[valid]))
8n_noise    = (labels_db == -1).sum()
9print(f"Clusters found: {n_clusters}")
10print(f"Noise points:   {n_noise}")
11if n_clusters > 1:
12    print("DBSCAN silhouette:", round(silhouette_score(X_cluster[valid], labels_db[valid]), 4))

4. Gaussian Mixture Model (GMM)

K-Means forces every student into exactly one study group. GMM is more realistic — it says "This student is 70% Science-type and 30% Math-type." It models each cluster as a probability cloud (a Gaussian bell) and gives soft, overlapping membership. Great when the real world doesn't have hard boundaries between categories.

What it does: Models the data as a mixture of K Gaussian distributions. Unlike K-Means, GMM gives a soft, probabilistic cluster membership — a point can belong to multiple clusters with different probabilities.

When to use it:

• When clusters overlap and hard assignment is inappropriate.
• Density estimation and generative modeling.
• Soft persona modeling (e.g., a user is 70% Cluster A, 30% Cluster B).

Math — Mixture Density: p(x) = Σ_{k=1..K} π_k N(x | μ_k, Σ_k)

Fit via Expectation-Maximization (EM):

• E-step: compute posterior responsibilities γ_{ik} = P(z_k | x_i)
• M-step: update π_k, μ_k, Σ_k to maximize expected log-likelihood

1from sklearn.mixture import GaussianMixture
2
3gmm = GaussianMixture(n_components=4, covariance_type="full", random_state=42)
4labels_gmm  = gmm.fit_predict(X_cluster)
5proba_gmm   = gmm.predict_proba(X_cluster)
6
7print("GMM silhouette:", round(silhouette_score(X_cluster, labels_gmm), 4))
8print("Point 0 soft membership:", np.round(proba_gmm[0], 3))

5. Principal Component Analysis (PCA)

You have a 3D sculpture and need to photograph it in 2D. PCA figures out the best angle to stand at so the photo captures as much of the sculpture's shape as possible — minimising the detail you lose in the flattening. In data terms, PCA projects your many features onto fewer axes while preserving the maximum amount of variation.

What it does: Finds a set of orthogonal axes (principal components) that capture the maximum variance in the data. Projecting onto the top d components reduces dimensionality while preserving as much information as possible.

When to use it:

• Reducing feature count before training a slow model (e.g., SVM).
• Visualizing high-dimensional data in 2D or 3D.
• Removing correlated features and noise.

When it fails: Only captures linear structure. Cannot represent non-linear manifolds. Explained variance is not the same as predictive usefulness.

Math: Given centered data X:

• Compute covariance C = XᵀX/N
• Eigendecompose: C = VΛVᵀ
• Top d eigenvectors of V are the principal components
• Equivalent to the SVD of X: X = UΣVᵀ

1from sklearn.decomposition import PCA
2
3pca = PCA(n_components=0.95, random_state=42)  # Keep 95% variance
4X_pca_95 = pca.fit_transform(X_high)
5print(f"Original: {X_high.shape[1]} features → Reduced: {X_pca_95.shape[1]} components")
6print("Cumulative variance explained:", np.cumsum(pca.explained_variance_ratio_)[-5:].round(3))
7
8# 2D visualization
9pca2d = PCA(n_components=2)
10X_2d  = pca2d.fit_transform(X_high)
11plt.scatter(X_2d[:, 0], X_2d[:, 1], c=y_high, s=8, cmap="viridis")
12plt.title("PCA — 2D Projection")
13plt.show()

6. t-SNE

Imagine folding a complex origami crane flat on a table. You want the parts that were close in the 3D shape to still be close on the flat table. t-SNE does this for high-dimensional data — it finds a 2D layout where points that were neighbours in the original high-dimensional space remain neighbours in the map. Useful for visualising clusters, but the map cannot be used for prediction.

What it does: A non-linear dimensionality reduction method primarily designed for 2D/3D visualization. It converts pairwise distances between points into probabilities and places points in 2D such that similar points stay close.

When to use it:

• Visualizing cluster structure in embedding spaces (word vectors, image encodings, sensor feature vectors).
• Exploring class separability before building a classifier.

When it fails:

• Cannot be used for inference on new data (must refit from scratch on every new dataset).
• Distances in the 2D output do not represent true inter-cluster distances.
• Results depend heavily on perplexity (typically 5–50).
• Very slow on large datasets.

Math: High-dimensional similarity: p_{j|i} = exp(-||x_i-x_j||²/2σ²) / Σ_k exp(-||x_i-x_k||²/2σ²) Low-dimensional similarity (Student-t with 1 df): q_{ij} = (1+||y_i-y_j||²)⁻¹ / Z Objective: minimize KL(P || Q) = Σ_{i,j} p_{ij} log(p_{ij}/q_{ij})

1from sklearn.manifold import TSNE
2
3tsne = TSNE(
4    n_components=2, perplexity=30,
5    learning_rate="auto", init="pca", random_state=42
6)
7X_tsne = tsne.fit_transform(X_high[:1200])
8
9plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y_high[:1200], s=8, cmap="plasma")
10plt.title("t-SNE — 2D Embedding")
11plt.show()

7. UMAP

Same goal as t-SNE — make a 2D map of complex data — but UMAP is like a better cartographer. It draws a map that is both locally accurate (nearby points stay nearby) and globally accurate (the relative positions of different clusters are also preserved). It runs much faster and the map can be used to project new points without refitting from scratch.

What it does: A faster alternative to t-SNE that often better preserves both local neighborhood structure and global inter-cluster relationships. Based on topological graph construction in high and low dimensions.

When to use it:

• Interactive data exploration dashboards with large datasets (t-SNE too slow).
• Pre-processing step before clustering (UMAP + K-Means is a powerful combination).
• Can be used for inference on new points (unlike t-SNE).

Key Parameters:

• n_neighbors — controls local/global balance (low → local, high → global)
• min_dist — controls how tightly embedded points are clustered

1# pip install umap-learn
2import umap
3
4reducer = umap.UMAP(n_components=2, n_neighbors=20, min_dist=0.1, random_state=42)
5X_umap  = reducer.fit_transform(X_high)
6
7plt.scatter(X_umap[:, 0], X_umap[:, 1], c=y_high, s=8, cmap="coolwarm")
8plt.title("UMAP — 2D Embedding")
9plt.show()

8. Isolation Forest

In a dense forest, a normal tree has many neighbours. To isolate it, you need many cuts. But a lone tree standing alone in a clearing is isolated with just one or two cuts. Isolation Forest detects anomalies using exactly this logic — unusual points in sparse regions are isolated very quickly, while normal points buried in dense regions take much longer.

What it does: Detects anomalies by randomly partitioning the feature space into trees. Points that are isolated in very few splits (short path length) are statistically unusual and scored as anomalies.

When to use it:

• Fraud detection, manufacturing defect detection, network intrusion.
• Sensor anomaly alerting (predictive maintenance).
• When you have only normal (inlier) examples for training.

Why it works: Normal points are dense and require many splits to isolate. Anomalous points are in sparse regions and get isolated in just a few splits.

1from sklearn.ensemble import IsolationForest
2from sklearn.metrics import classification_report
3
4iso = IsolationForest(n_estimators=300, contamination=0.05, random_state=42)
5pred_iso = iso.fit_predict(X_anom)  # Returns +1 (inlier) or -1 (outlier)
6print(classification_report(y_anom, pred_iso, digits=3))

9. One-Class SVM

Imagine drawing a tight fence around all the normal animals in a zoo. Every animal that was present during the fence-drawing fits inside it. Anything arriving later that cannot fit inside the fence is flagged as unusual — perhaps an escaped exotic species. One-Class SVM builds this fence using only normal (inlier) training data.

What it does: Trains only on normal (inlier) data and learns a tight boundary around it. At inference, points that fall outside this boundary are flagged as anomalies (novelties).

When to use it:

• Novelty detection when you have no labeled anomaly examples.
• Machine health monitoring where "failure" data is rare or unavailable.
• Systems that must detect unknown failure modes.

When it fails: Slow on large datasets. Sensitive to nu (expected fraction of outliers) and kernel parameters. Does not work well when normal data is multimodal.

1from sklearn.svm import OneClassSVM
2from sklearn.pipeline import Pipeline
3
4ocsvm = Pipeline([
5    ("scaler", StandardScaler()),
6    ("model",  OneClassSVM(kernel="rbf", gamma="scale", nu=0.05)),
7])
8pred_oc = ocsvm.fit_predict(X_anom)
9print(classification_report(y_anom, pred_oc, digits=3))

Practical Selection Guide

Always validate clustering results using both the Silhouette Score (quantitative) and domain expert review (qualitative). A high silhouette score with meaningless business segments is worse than a slightly lower score that produces actionable groups.