Classification Algorithms

Classification is the task of predicting a discrete label from input features — spam vs not-spam, fault vs normal, or which of ten categories an image belongs to. Each algorithm makes different assumptions about data geometry, which determines where it works and where it fails.

Shared Setup (Run Once)

1import numpy as np
2from sklearn.datasets import make_classification
3from sklearn.model_selection import train_test_split
4from sklearn.metrics import accuracy_score, f1_score, classification_report
5from sklearn.preprocessing import StandardScaler
6from sklearn.pipeline import Pipeline
7
8X, y = make_classification(
9    n_samples=5000, n_features=20,
10    n_informative=12, n_redundant=4,
11    n_classes=2, weights=[0.65, 0.35],
12    class_sep=1.2, random_state=42,
13)
14X_train, X_test, y_train, y_test = train_test_split(
15    X, y, test_size=0.2, random_state=42, stratify=y
16)
17
18def evaluate(name, model):
19    model.fit(X_train, y_train)
20    pred = model.predict(X_test)
21    print(f"\n=== {name} ===")
22    print("Accuracy:", round(accuracy_score(y_test, pred), 4))
23    print("F1:      ", round(f1_score(y_test, pred), 4))
24    print(classification_report(y_test, pred, digits=3))

1. Logistic Regression

Think of a doctor scoring a patient's health test. Each result (blood pressure, sugar level, heart rate) adds or subtracts points from a total score. If the final score crosses a threshold — the patient is flagged at risk. Logistic Regression does exactly this with numbers: weigh the inputs, add them up, and decide yes or no.

What it does: Assigns a weight to each feature, sums the weighted inputs, and pushes the result through a sigmoid function to produce a probability between 0 and 1.

When to use it:

• First baseline for any binary or multi-class problem.
• When you need probability estimates that are well-calibrated.
• When interpretability of weights matters.

When it fails: When the relationship between features and target is strongly non-linear. A decision boundary is always a hyperplane — it cannot capture curves.

Math: z = wᵀx + b p(y=1|x) = σ(z) = 1 / (1 + e^{-z}) Training minimizes binary cross-entropy: L = -(1/N) Σ [y_i log(p_i) + (1-y_i) log(1-p_i)] + λΩ(w)

1from sklearn.linear_model import LogisticRegression
2
3log_reg = Pipeline([
4    ("scaler", StandardScaler()),
5    ("model", LogisticRegression(max_iter=3000, C=1.0, class_weight="balanced")),
6])
7evaluate("Logistic Regression", log_reg)

2. K-Nearest Neighbors (KNN)

You just moved to a new city and want to know if a neighbourhood is safe. You ask the 9 nearest residents for their opinion and go with whatever the majority says. KNN does the same thing with data points — it asks its closest neighbours and goes with the majority label.

What it does: Stores all training examples. For a new input, it finds the k most similar training examples by distance and assigns the majority class among them.

When to use it:

• When you have no prior knowledge of the data distribution.
• Useful as a sanity-check baseline on small datasets.
• When decision boundaries are locally irregular.

When it fails: Slow at inference on large datasets (must compute distances to all points). Sensitive to feature scaling and irrelevant features. Degrades with high dimensionality (curse of dimensionality).

Math: ŷ = mode({ y_i | x_i ∈ N_k(x) })

With distance weighting: votes scaled by 1/d(x, x_i).

1from sklearn.neighbors import KNeighborsClassifier
2
3knn = Pipeline([
4    ("scaler", StandardScaler()),
5    ("model", KNeighborsClassifier(n_neighbors=9, weights="distance", metric="euclidean")),
6])
7evaluate("K-Nearest Neighbors", knn)

3. Naive Bayes

Imagine a detective collecting clues at a crime scene. He considers each clue separately — a muddy boot, a broken window, a dropped receipt — and multiplies their individual probabilities to reach the most likely suspect. Naive Bayes is that detective: it treats every feature as an independent clue and picks the most probable class.

What it does: Applies Bayes' theorem, assuming all features are conditionally independent given the class label. Despite this strong "naive" assumption, it often performs well on text classification.

When to use it:

• Spam detection, document classification, sentiment analysis.
• When you have very limited training data.
• When training speed is critical.

When it fails: When features are highly correlated (violates the independence assumption). Not reliable for continuous features without Gaussian assumption.

Math: P(y|x) ∝ P(y) * Π_j P(x_j|y) For Gaussian NB: P(x_j|y=c) ~ N(μ_{cj}, σ²_{cj})

1from sklearn.naive_bayes import GaussianNB
2
3nb = GaussianNB()
4evaluate("Naive Bayes (Gaussian)", nb)

4. Decision Tree

Remember the game "20 Questions"? You ask yes/no questions one at a time — "Is it an animal? Does it have four legs? Does it bark?" — until you narrow it down to the answer. A Decision Tree plays this exact game with your dataset, asking threshold questions on features until it reaches a final label.

What it does: Recursively partitions the feature space by asking threshold questions on one feature at a time — building a tree of if/else rules. Each leaf node holds a predicted class.

When to use it:

• When explainability is required (rules can be visualized and printed).
• Rule-based approval/rejection systems.
• When categorical and numerical features are mixed.

When it fails: Single trees overfit easily without depth limits. Not competitive with ensembles on complex data.

Math — Split criterion (Gini Impurity): Gini(t) = 1 - Σ_k p_k² At each node, choose the feature and threshold maximizing impurity reduction.

1from sklearn.tree import DecisionTreeClassifier, export_text
2
3dt = DecisionTreeClassifier(max_depth=6, min_samples_leaf=5, random_state=42)
4evaluate("Decision Tree", dt)
5
6# Print the learned rules
7dt.fit(X_train, y_train)
8print(export_text(dt, max_depth=3))

5. Random Forest

Instead of trusting one teacher's opinion on your essay, you hand it to 400 different teachers and take a majority vote. No single teacher can mislead the result. Even if some teachers are wrong, the crowd is usually right. Random Forest does exactly this — hundreds of decision trees vote together.

What it does: Builds many decision trees, each on a bootstrap sample of the data and a random subset of features. Final prediction is a majority vote over all trees.

When to use it:

• Strong general-purpose classifier for tabular data.
• When you need feature importance scores.
• When interpretability of individual trees is less important than overall accuracy.

When it fails: Slower to predict than linear models. Can overfit on very small datasets. Consumes more memory than single trees.

Math: Each tree f_t is trained on bootstrap sample D_t. At each split, only √p features are considered. ŷ(x) = mode({ f_t(x) | t = 1..T })

1from sklearn.ensemble import RandomForestClassifier
2
3rf = RandomForestClassifier(
4    n_estimators=400, max_features="sqrt",
5    random_state=42, n_jobs=-1
6)
7evaluate("Random Forest", rf)

6. Support Vector Machine (SVM)

Two groups of students are sitting in a school cafeteria. You want to draw a line on the floor between them that leaves the widest possible gap — so even if one person shifts their chair a little, they still won't cross into the other group's side. SVM finds that widest-margin dividing line.

What it does: Finds the hyperplane that maximizes the margin between two classes. Points near the boundary (support vectors) define the margin. Kernel functions allow non-linear separation in higher-dimensional spaces.

When to use it:

• High-dimensional data (text, genes, image descriptors).
• When the dataset is not too large (SVMs scale poorly beyond ~50k samples).
• When a clear margin boundary exists in the data.

When it fails: Very slow to train on large datasets. Sensitive to feature scaling. Choosing the right kernel and C requires careful tuning.

Math — Soft-Margin Primal: min_{w,b,ξ} (1/2)||w||² + C Σ ξ_i subject to: y_i(wᵀx_i + b) ≥ 1 - ξ_i, ξ_i ≥ 0

Kernel trick: K(x_i, x_j) = φ(x_i)ᵀ φ(x_j) without explicitly computing φ.

1from sklearn.svm import SVC
2
3svm = Pipeline([
4    ("scaler", StandardScaler()),
5    ("model", SVC(C=2.0, kernel="rbf", gamma="scale", probability=True)),
6])
7evaluate("SVM (RBF Kernel)", svm)

7. AdaBoost

Imagine a class of struggling students taking a math test in rounds. After each round, the teacher pays extra attention to the questions most students got wrong. Over time, the class improves dramatically because every round focuses on the existing weaknesses. AdaBoost trains weak models the same way — each round targets what the previous ones got wrong.

What it does: Trains a sequence of weak learners (typically shallow trees). Each round reweights training samples to focus more on examples the previous round got wrong. Final prediction is a weighted vote over all learners.

When to use it:

• When you need a boosting baseline with fewer hyperparameters than XGBoost.
• Binary classification with simple features.

When it fails: Very sensitive to noisy data and outliers — mislabeled examples get amplified. Slower than bagging.

Math: At iteration t:

• Fit weak learner h_t on weighted data
• ε_t = Σ w_i · 𝟙[h_t(x_i) ≠ y_i]
• Learner weight: α_t = 0.5 · ln((1-ε_t)/ε_t)
• Update sample weights: increase for misclassified, decrease for correct
• Final: H(x) = sign(Σ_t α_t h_t(x))

1from sklearn.ensemble import AdaBoostClassifier
2from sklearn.tree import DecisionTreeClassifier
3
4ada = AdaBoostClassifier(
5    estimator=DecisionTreeClassifier(max_depth=1),
6    n_estimators=300, learning_rate=0.05, random_state=42,
7)
8evaluate("AdaBoost", ada)

8. Gradient Boosting Classifier

Think of a sculptor carving a statue. In the first round, they rough out the shape. In the second round, they only fix what was uneven. In the third, they smooth even finer details. Each round corrects the mistakes left by the previous one. Gradient Boosting builds its model the same way — each tree fixes the residual errors of the ensemble so far.

What it does: Builds trees sequentially where each tree fits the residual errors (negative gradients) of the previous ensemble. The key difference from AdaBoost: it works in function space, fitting pseudo-residuals rather than reweighting samples.

When to use it:

• High-quality tabular classification tasks.
• Churn modeling, risk scoring, medical diagnosis.

Math: F_m(x) = F_{m-1}(x) + η · h_m(x) where h_m fits the negative gradient of the loss: r_i = -[∂L(y_i, F(x_i))/∂F(x_i)]

1from sklearn.ensemble import GradientBoostingClassifier
2
3gbc = GradientBoostingClassifier(
4    n_estimators=250, learning_rate=0.05,
5    max_depth=4, subsample=0.8, random_state=42
6)
7evaluate("Gradient Boosting", gbc)

9. XGBoost

Same sculptor from above, but now equipped with a precision laser chisel and safety guardrails. They work significantly faster, make fewer accidental cuts, and the final statue is cleaner. XGBoost is Gradient Boosting with hardware-level optimizations — regularization, second-order gradients, and parallel processing built in.

What it does: An optimized, regularized implementation of gradient boosting. It adds L1/L2 regularization on leaf weights, uses second-order gradient approximations for faster splits, and handles sparse data natively.

When to use it:

• The default go-to for tabular data competitions and production systems.
• Fraud detection, financial risk, click prediction.
• When training speed on large datasets matters.

Math — Objective: Obj = Σ_i l(y_i, ŷ_i) + Σ_k Ω(f_k) where Ω(f) = γT + (1/2)λ||w||²

Uses Taylor expansion to second order for each split gain calculation.

1# pip install xgboost
2from xgboost import XGBClassifier
3
4xgb = XGBClassifier(
5    n_estimators=500, learning_rate=0.05,
6    max_depth=6, subsample=0.9,
7    colsample_bytree=0.8, reg_alpha=0.1,
8    eval_metric="logloss", random_state=42, n_jobs=-1,
9)
10evaluate("XGBoost", xgb)

10. LightGBM

Same precision sculptor, but now with a team of assistants who split the workload intelligently — some work on the left side while others work on the right simultaneously. LightGBM is XGBoost with a smarter work distribution: it processes large datasets dramatically faster by being selective about which data points and features to focus on.

What it does: A gradient boosting framework optimized for speed and memory efficiency. Uses histogram binning to find splits, leaf-wise (best-first) tree growth instead of level-wise, and gradient-based sampling to skip less informative training examples.

When to use it:

• Very large datasets (millions of rows) where XGBoost is too slow.
• High-cardinality categorical features (has native categorical support).

Math — Key Differences from XGBoost:

• Leaf-wise growth: grows the leaf with maximum loss reduction at each step (faster convergence, but may overfit on small data).
• EFB (Exclusive Feature Bundling): merges mutually exclusive sparse features to reduce dimensionality.
• GOSS (Gradient-based One-Side Sampling): keeps high-gradient samples, randomly samples low-gradient ones.

1# pip install lightgbm
2from lightgbm import LGBMClassifier
3
4lgbm = LGBMClassifier(
5    n_estimators=500, learning_rate=0.05,
6    num_leaves=63, min_child_samples=20,
7    random_state=42, n_jobs=-1,
8)
9evaluate("LightGBM", lgbm)

11. Linear Discriminant Analysis (LDA)

Two school clubs — Science Club and Art Club — both took the same 30-question exam. LDA finds the single exam question (or combination of questions) whose answers best separate the two clubs' scores — so Science students score high on it and Art students score low. It projects everyone onto that most-separating axis.

What it does: Finds a projection of the data that maximizes class separability while keeping within-class variance minimal. Unlike logistic regression, LDA explicitly models the class-conditional Gaussian distributions.

When to use it:

• Small, clean datasets.
• As a preprocessing step to project data into a lower-dimensional, class-discriminative space before another classifier.
• Multi-class problems (naturally extends to K-1 discriminant axes).

When it fails: Assumes features are normally distributed with equal class covariance. Real-world data rarely satisfies this perfectly.

Math: Assumes x | y=k ~ N(μ_k, Σ) with shared covariance. Discriminant score: δ_k(x) = xᵀΣ⁻¹μ_k - (1/2)μ_kᵀΣ⁻¹μ_k + log π_k Predict: argmax_k δ_k(x)

1from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
2
3lda = Pipeline([
4    ("scaler", StandardScaler()),
5    ("model", LinearDiscriminantAnalysis()),
6])
7evaluate("LDA", lda)

Practical Selection Guide

Always compare using F1 or PR-AUC — not raw accuracy — especially on imbalanced datasets.