Machine Learning Foundations

Machine learning builds systems that improve at a task by learning patterns from data, not by following hand-coded rules. Understanding these foundations before touching a library separates engineers who can debug and extend systems from those who can only copy examples.

What Machine Learning Actually Does

Traditional programming: developer writes rules → computer applies rules → output.

Machine learning inverts this. You provide (input, output) examples and an algorithm discovers the function mapping inputs to outputs.

Formally, ML seeks a function f(x; θ) parameterized by θ:

θ* = argmin_θ (1/N) Σ L(f(x_i; θ), y_i) + λΩ(θ)

• L = loss function measuring prediction error
• Ω(θ) = regularization term penalizing complexity
• λ = regularization strength
• N = training sample count

This is Empirical Risk Minimization — minimize average loss over training data while controlling complexity.

Three Types of Machine Learning

Supervised Learning — model learns from labeled pairs (x, y). Goal: predict y for unseen x. Use when: spam detection, price forecasting, image classification, fault prediction.

Unsupervised Learning — no labels provided. Model discovers hidden structure: groups, compressed representations, density regions. Use when: customer segmentation, anomaly detection, dimensionality reduction.

Reinforcement Learning — an agent takes actions and learns from reward signals. Objective: J(π) = E_{τ~π}[ Σ_t γ^t r_t ] where π = policy, γ = discount factor. Use when: robotic control, game agents, adaptive recommendation engines.

Problem Framing

The most common ML failure is poor problem framing — not model choice. Before writing any code:

1. Prediction target — exactly what are you predicting? A number, a label, a probability?
2. Unit of prediction — per user? per device? per transaction?
3. Prediction horizon — now, next hour, next week?
4. Success metric — MAE? F1? Recall at threshold? Business revenue?
5. Cost asymmetry — is a false positive more expensive than a false negative?

Misalignment between your training objective and business objective is the most common reason a model that scores well fails in production.

Data Quality and Distribution Shift

Your model learns the patterns in your training data. If training data does not represent real-world deployment, performance degrades immediately.

Data checks to run before every training run:

• Missing value profiles per column
• Outlier and range diagnostics
• Duplicate rows that cause leakage
• Temporal drift analysis for time-dependent data

Splitting Data Correctly

1from sklearn.model_selection import train_test_split
2
3X_train, X_test, y_train, y_test = train_test_split(
4    X, y, test_size=0.2, random_state=42,
5    stratify=y  # Preserve class ratios
6)

Common leakage patterns:

• Fitting a StandardScaler on the full dataset before splitting
• Using a feature derived from the target variable
• Target encoding computed with test rows included

Preprocessing Pipelines

Always wrap preprocessing inside a Scikit-Learn Pipeline. This ensures transformations fitted on training data are applied correctly at inference — preventing the most common source of train-serve skew.

1import numpy as np
2import pandas as pd
3from sklearn.compose import ColumnTransformer
4from sklearn.pipeline import Pipeline
5from sklearn.impute import SimpleImputer
6from sklearn.preprocessing import OneHotEncoder, StandardScaler
7from sklearn.ensemble import RandomForestClassifier
8
9numeric_pipe = Pipeline([
10    ("imputer", SimpleImputer(strategy="median")),
11    ("scaler", StandardScaler()),
12])
13
14categorical_pipe = Pipeline([
15    ("imputer", SimpleImputer(strategy="most_frequent")),
16    ("onehot", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
17])
18
19preprocess = ColumnTransformer([
20    ("num", numeric_pipe, ["age", "salary"]),
21    ("cat", categorical_pipe, ["city"]),
22])
23
24model_pipeline = Pipeline([
25    ("prep", preprocess),
26    ("clf", RandomForestClassifier(n_estimators=200, random_state=42)),
27])
28
29model_pipeline.fit(X_train, y_train)

Acronyms & Abbreviations Quick Reference

Machine Learning is full of abbreviations. This table covers every acronym you will encounter on this page and across the algorithm pages.

Evaluation Metrics

Classification

Regression

Bias-Variance Tradeoff

Total Error ≈ Bias² + Variance + Irreducible Noise

• High Bias (Underfitting): Model too simple, misses signal. Fix: increase model capacity, add features, reduce regularization.
• High Variance (Overfitting): Model memorizes noise. Fix: add more data, increase regularization, reduce model complexity.

1from sklearn.model_selection import learning_curve
2import matplotlib.pyplot as plt
3import numpy as np
4
5train_sizes, train_scores, val_scores = learning_curve(
6    model_pipeline, X_train, y_train,
7    train_sizes=np.linspace(0.1, 1.0, 8),
8    cv=5, scoring="f1"
9)
10plt.plot(train_sizes, train_scores.mean(axis=1), label="Train F1")
11plt.plot(train_sizes, val_scores.mean(axis=1), label="Validation F1")
12plt.xlabel("Training samples")
13plt.ylabel("F1 Score")
14plt.legend()
15plt.show()

Cross-Validation

Single train-test splits can have high variance. K-Fold CV averages performance over multiple held-out folds.

1from sklearn.model_selection import StratifiedKFold, cross_val_score
2from sklearn.svm import SVC
3
4cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
5pipe = Pipeline([
6    ("scaler", StandardScaler()),
7    ("svc", SVC(kernel="rbf", C=1.0, gamma="scale")),
8])
9
10scores = cross_val_score(pipe, X, y, cv=cv, scoring="f1")
11print(f"CV F1: {scores.mean():.4f} ± {scores.std():.4f}")

Use TimeSeriesSplit for temporal data — never StratifiedKFold.

Hyperparameter Tuning

Hyperparameters control how a model learns. They cannot be learned from data and must be chosen before training.

1from sklearn.model_selection import GridSearchCV
2from sklearn.ensemble import RandomForestClassifier
3
4param_grid = {
5    "n_estimators":      [100, 200, 400],
6    "max_depth":         [None, 5, 10],
7    "min_samples_split": [2, 5, 10],
8}
9
10grid = GridSearchCV(
11    RandomForestClassifier(random_state=42),
12    param_grid, scoring="f1", cv=5, n_jobs=-1
13)
14grid.fit(X_train, y_train)
15print("Best params:", grid.best_params_)
16print("Best CV F1:", round(grid.best_score_, 4))

Class Imbalance

When one class is rare (e.g., 98% normal, 2% fraud), a naive model achieves 98% accuracy by always predicting "normal" — and is completely useless.

Strategies:

• Class-weighted loss: penalize minority misclassification more
• SMOTE: synthesize new minority examples
• Undersampling: reduce majority class size
• Threshold tuning: choose cutoff from PR-curve, not default 0.5

1from sklearn.linear_model import LogisticRegression
2clf = LogisticRegression(max_iter=2000, class_weight="balanced")
3clf.fit(X_train, y_train)

Model Explainability

1import pandas as pd
2from sklearn.ensemble import RandomForestClassifier
3
4rf = RandomForestClassifier(n_estimators=300, random_state=42)
5rf.fit(X_train, y_train)
6
7importance = pd.Series(rf.feature_importances_, index=X_train.columns)
8print(importance.sort_values(ascending=False).head(10))

For local explanations, use SHAP values:

1# pip install shap
2import shap
3explainer = shap.TreeExplainer(rf)
4shap_values = explainer.shap_values(X_test)
5shap.summary_plot(shap_values[1], X_test)

Saving Models

Always serialize the full pipeline — not just the estimator — to guarantee consistent preprocessing at inference time.

1import joblib
2joblib.dump(model_pipeline, "full_pipeline.joblib")
3
4loaded = joblib.load("full_pipeline.joblib")
5predictions = loaded.predict(X_test)

Production Checklist

Before shipping any model to production, verify:

• Input schema validation at the inference endpoint
• Drift detection: compare P_prod(X) against P_train(X)
• Prediction distribution monitoring over time
• Latency and memory profiling under expected load
• Fallback behavior on malformed or unexpected inputs
• Retraining triggers: schedule-based or degradation-triggered
• Model versioning, rollback mechanism, and audit trail