Regression Algorithms

Regression predicts a continuous numeric value — temperature, price, energy consumption, signal strength, remaining battery life. The algorithm you choose determines what assumptions you make about the underlying relationship between inputs and output.

Shared Setup (Run Once)

1import numpy as np
2from sklearn.datasets import make_regression
3from sklearn.model_selection import train_test_split
4from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
5from sklearn.preprocessing import StandardScaler
6from sklearn.pipeline import Pipeline
7
8X, y = make_regression(
9    n_samples=5000, n_features=20,
10    n_informative=12, noise=15.0, random_state=42,
11)
12X_train, X_test, y_train, y_test = train_test_split(
13    X, y, test_size=0.2, random_state=42
14)
15
16def evaluate(name, model):
17    model.fit(X_train, y_train)
18    pred = model.predict(X_test)
19    mae  = mean_absolute_error(y_test, pred)
20    rmse = mean_squared_error(y_test, pred) ** 0.5
21    r2   = r2_score(y_test, pred)
22    print(f"\n=== {name} ===")
23    print(f"MAE : {mae:.4f}")
24    print(f"RMSE: {rmse:.4f}")
25    print(f"R²  : {r2:.4f}")

1. Linear Regression

If you plot study hours against exam scores on graph paper, you'll notice the points roughly follow a diagonal line going upward. Linear Regression finds the single best straight line that runs through all those scattered points — so you can read off a predicted score for any number of study hours.

What it does: Fits a linear function y = wᵀx + b by minimizing the sum of squared differences between predictions and actual values.

When to use it:

• First baseline for any numeric prediction task.
• When you need interpretable coefficients (each weight = marginal effect of that feature).
• When the relationship between features and target is approximately linear.

When it fails: Cannot capture curves or interaction effects. Very sensitive to outliers (squared errors give them enormous influence). Breaks down with multicollinear features.

Math: ŷ = Xw + b Minimize: ||y - Xw||² Closed-form solution: w* = (XᵀX)⁻¹ Xᵀy

1from sklearn.linear_model import LinearRegression
2
3lin = LinearRegression()
4evaluate("Linear Regression", lin)
5
6# Inspect coefficients
7lin.fit(X_train, y_train)
8print("Coefficients:", lin.coef_[:5])

2. Polynomial Regression

Same graph as before, but the points don't follow a straight line — they follow a curve, like a skateboard ramp. A straight line can't capture a curve, so Polynomial Regression bends the line to match the actual shape of the data.

What it does: Extends linear regression by adding polynomial terms (x², x³, x·x₂) to the feature set, enabling it to fit curved relationships.

When to use it:

• When scatter plots clearly show a curved, not straight, trend.
• Sensor calibration curves, physical equations with known polynomial form.

When it fails: High degrees overfit rapidly. Always pair with regularization (Ridge or Lasso) or cross-validation for degree selection.

Math: Transform x → Φ(x) = [1, x, x², ..., x^d] Then fit: ŷ = βᵀΦ(x)

1from sklearn.preprocessing import PolynomialFeatures
2from sklearn.linear_model import Ridge
3
4poly_pipe = Pipeline([
5    ("poly",   PolynomialFeatures(degree=2, include_bias=False)),
6    ("scaler", StandardScaler()),
7    ("model",  Ridge(alpha=1.0)),
8])
9evaluate("Polynomial Regression (deg=2, Ridge)", poly_pipe)

3. Ridge Regression (L2 Regularization)

Imagine a football team where one player is so dominant, the whole strategy depends entirely on him. If he gets injured, the team collapses. Ridge Regression adds a rule: no single player (feature) can carry too much weight. Everyone contributes a little, and no one dominates completely — making the team (model) more resilient.

What it does: Adds an L2 penalty on the coefficients to prevent any single weight from growing too large. This stabilizes estimation when features are highly correlated (multicollinear).

When to use it:

• When you have many correlated features.
• When Linear Regression's coefficients are unstable across cross-validation folds.
• Economic forecasting, sensor fusion problems.

When it fails: Does not perform feature selection — all features remain with small non-zero weights.

Math: min_w ||y - Xw||² + α||w||² Solution: w* = (XᵀX + αI)⁻¹ Xᵀy The αI term makes the matrix invertible even when features are correlated.

1from sklearn.linear_model import Ridge, RidgeCV
2
3# RidgeCV automatically finds best alpha via cross-validation
4ridge_cv = Pipeline([
5    ("scaler", StandardScaler()),
6    ("model",  RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0, 100.0], cv=5)),
7])
8evaluate("Ridge (RidgeCV)", ridge_cv)

4. Lasso Regression (L1 Regularization)

Same football team, but this time the coach goes further — he doesn't just limit the star player, he removes the weakest players from the squad entirely. Only the genuinely useful players survive. Lasso automatically zeros out irrelevant features, giving you a leaner, simpler model.

What it does: Adds an L1 penalty on coefficients. Unlike Ridge, L1 creates sparsity — it drives unimportant feature coefficients exactly to zero, performing automatic feature selection.

When to use it:

• When you have more features than you need and want the model to select relevant ones.
• Genomics, text regression, any domain with sparse true signals.

When it fails: When groups of correlated features are all informative — Lasso arbitrarily zeroes all but one. Use Elastic Net instead.

Math: min_w ||y - Xw||² + α||w||₁ The L1 penalty creates a non-differentiable corner at zero, which is what produces exact sparsity.

1from sklearn.linear_model import LassoCV
2
3lasso_cv = Pipeline([
4    ("scaler", StandardScaler()),
5    ("model",  LassoCV(cv=5, max_iter=10000, random_state=42)),
6])
7evaluate("Lasso (LassoCV)", lasso_cv)
8
9lasso_cv.fit(X_train, y_train)
10import numpy as np
11n_zero = np.sum(lasso_cv.named_steps["model"].coef_ == 0)
12print(f"Features zeroed out: {n_zero}/{X_train.shape[1]}")

5. Elastic Net

You want the best of both coaches — Ridge's discipline (limit dominance) and Lasso's boldness (cut weak links). Elastic Net is the compromise: it limits overly powerful features AND removes irrelevant ones. A balanced team strategy when you have many correlated players on the field.

What it does: Linearly combines Ridge and Lasso penalties. The l1_ratio parameter controls the mix. This combines Ridge's stability under correlation with Lasso's ability to zero out features.

When to use it:

• High-dimensional data with groups of correlated features that are all informative.
• When neither pure Ridge nor Lasso gives stable results.

Math: min_w ||y - Xw||² + α[ρ||w||₁ + (1-ρ)||w||²] where ρ = l1_ratio.

1from sklearn.linear_model import ElasticNetCV
2
3elastic_cv = Pipeline([
4    ("scaler", StandardScaler()),
5    ("model",  ElasticNetCV(l1_ratio=[0.1, 0.5, 0.7, 0.9, 1.0], cv=5, max_iter=10000)),
6])
7evaluate("Elastic Net (CV)", elastic_cv)

6. Decision Tree Regressor

Instead of drawing a formula, imagine dividing a class of students into buckets by asking questions: "Did you study more than 5 hours? Yes → did you sleep more than 7 hours?" You keep splitting until each bucket is small enough to have a reliable average score. The predicted score for a new student is simply the average of their matching bucket.

What it does: Partitions the feature space into rectangular regions by asking binary threshold questions. Predicts the mean y value for all training samples that fall in each region (leaf).

When to use it:

• When you need an interpretable, explainable prediction rule.
• When data has discontinuous jumps or step-function relationships.

When it fails: Single trees overfit unless depth is constrained. Prediction is constant within each leaf — extrapolation beyond training range fails completely.

Math: At each split, minimize residual MSE: I(t) = (1/|t|) Σ_{i∈t} (y_i - ȳ_t)² Choose feature and threshold maximizing I(parent) - p_L·I(left) - p_R·I(right).

1from sklearn.tree import DecisionTreeRegressor
2
3dtr = DecisionTreeRegressor(max_depth=8, min_samples_leaf=5, random_state=42)
4evaluate("Decision Tree Regressor", dtr)

7. Random Forest Regressor

Ask 500 people to estimate how long a road trip will take. Each person uses their own method and makes slightly different errors. When you average all 500 estimates, random individual errors cancel out and you get a much more accurate answer than any single person gave. Random Forest does this with 500 decision trees.

What it does: Builds many decision tree regressors on bootstrap samples with random feature subsets. Predictions are averaged across all trees to reduce variance.

When to use it:

• General-purpose non-linear regression baseline.
• When you want robust out-of-the-box performance.
• When you need feature importance scores.

Math: ŷ(x) = (1/T) Σ_{t=1..T} f_t(x) Averaging over T decorrelated trees reduces prediction variance by a factor of T.

1from sklearn.ensemble import RandomForestRegressor
2
3rfr = RandomForestRegressor(
4    n_estimators=500, max_features=0.5,
5    random_state=42, n_jobs=-1
6)
7evaluate("Random Forest Regressor", rfr)

8. Gradient Boosting Regressor & XGBoost

Think of a relay race where each runner doesn't run the full track — they only run the section where the previous runner dropped the baton. Each tree in this ensemble picks up exactly where the last one made an error, correcting it a little more each round until predictions converge to the true answer.

What it does: Builds trees sequentially where each new tree fits the pseudo-residuals — the negative gradient of the loss with respect to current predictions. Final output is the sum of all tree contributions.

When to use it:

• Production-level demand forecasting, energy prediction, cost estimation.
• When you need the best possible accuracy on tabular regression.

Math: Stage-wise additive model: F_m(x) = F_{m-1}(x) + η · h_m(x) With squared error: r_i = y_i - F_{m-1}(x_i) (literally the residuals). With MAE: residuals are replaced by the sign of the errors.

1from sklearn.ensemble import GradientBoostingRegressor
2
3gbr = GradientBoostingRegressor(
4    n_estimators=400, learning_rate=0.05,
5    max_depth=4, subsample=0.8, random_state=42
6)
7evaluate("Gradient Boosting Regressor", gbr)

1# pip install xgboost
2from xgboost import XGBRegressor
3
4xgb_reg = XGBRegressor(
5    n_estimators=500, learning_rate=0.05,
6    max_depth=6, subsample=0.9,
7    colsample_bytree=0.8, reg_alpha=0.1,
8    random_state=42, n_jobs=-1,
9)
10evaluate("XGBoost Regressor", xgb_reg)

9. Support Vector Regression (SVR)

Stretch a rubber tube around a curve drawn through your data points. SVR ignores any point that falls inside the tube — small wiggles don't matter. Only points outside the tube get pulled back in. This gives you a smooth, noise-resistant prediction curve that doesn't overreact to minor fluctuations.

What it does: Fits a function within an ε-insensitive tube around the data — errors smaller than ε are completely ignored. Outside that tube, a margin-based penalty applies. The result is a smooth function robust to small noise.

When to use it:

• Signal prediction where small fluctuations are measurement noise (not real signal).
• Small-to-medium datasets where training time is not a bottleneck.

When it fails: Very slow on large datasets (quadratic scaling). Requires careful tuning of C, ε, and kernel parameters. Does not extrapolate well beyond training range.

Math: ε-insensitive loss: L_ε(y, f(x)) = max(0, |y - f(x)| - ε) Objective: minimize ||w||² + C Σ(ξ_i + ξ_i*) subject to |y_i - f(x_i)| ≤ ε + ξ_i.

1from sklearn.svm import SVR
2
3svr_pipe = Pipeline([
4    ("scaler", StandardScaler()),
5    ("model",  SVR(C=10.0, epsilon=0.1, kernel="rbf", gamma="scale")),
6])
7evaluate("SVR", svr_pipe)

Practical Selection Guide

Compare both MAE and RMSE — RMSE penalizes large errors more heavily and exposes outlier sensitivity. Use R² alongside these, but do not use it alone (it can be misleading on non-linear data splits).