ML Glossary

This glossary covers every term you will encounter when learning or working in Machine Learning, Data Science, and Deep Learning. Use it as a quick reference when a word on any of the algorithm pages or interview questions is unfamiliar.

A

Accuracy The fraction of predictions that were correct. (TP + TN) / Total. Only reliable when classes are balanced.

Activation Function A mathematical function applied to a neuron's output to introduce non-linearity. Common choices: ReLU, sigmoid, softmax, tanh.

AdaBoost An ensemble method that trains weak classifiers sequentially, increasing focus on misclassified examples each round.

Anomaly Detection The task of identifying data points that deviate significantly from the norm (e.g., fraud, sensor faults).

AUC (Area Under the Curve) A scalar summary of a ROC or Precision-Recall curve. Higher is better. A perfect model scores 1.0.

Autoencoder A neural network trained to compress input into a compact latent vector (encoder) and reconstruct it (decoder). Used for dimensionality reduction and anomaly detection.

B

Backpropagation The algorithm used to compute gradients of the loss with respect to every weight in a neural network using the chain rule. It drives gradient-based optimization.

Bagging (Bootstrap Aggregating) An ensemble technique that trains multiple models on different bootstrap (random-with-replacement) samples of the training data and aggregates their predictions.

Batch Size The number of training examples processed in one forward-backward pass during neural network training.

Bias (in ML context)

1. The systematic error in a model's predictions — difference between average prediction and true value.
2. A learned offset term b added to a neuron's weighted sum.

Bias-Variance Tradeoff The fundamental tension between model complexity (variance) and prediction error (bias). High complexity → low bias, high variance. Low complexity → high bias, low variance.

Boosting A sequential ensemble strategy where each model tries to correct the errors made by the previous one.

C

Calibration A measure of how well a model's predicted probabilities match actual observed frequencies. A perfectly calibrated model predicting 0.8 probability should be correct ~80% of the time.

Classification A supervised learning task where the goal is to predict a discrete label or category.

CNN (Convolutional Neural Network) A neural network architecture designed for spatial data (images, signals). Uses convolutional filters that slide across the input to detect local patterns.

Confusion Matrix A table showing TP, TN, FP, FN to give a complete picture of classifier performance across all classes.

Concept Drift When the relationship P(Y|X) changes over time — the same input features now map to different output values. Requires model retraining.

Cross-Entropy Loss The most common loss function for classification problems. Measures the difference between true and predicted probability distributions.

Cross-Validation (CV) A technique for evaluating model performance by training and testing on multiple non-overlapping subsets of the data (folds).

Curse of Dimensionality As the number of features grows, the volume of the feature space grows exponentially, making data increasingly sparse and distance-based methods unreliable.

D

Data Augmentation Artificially expanding a training dataset by applying transformations (flipping, rotating, adding noise) to existing samples.

Data Leakage When information from outside the training set (including the test set or future data) influences the training process, producing unrealistically optimistic metrics.

Decision Boundary The boundary in feature space that separates predicted classes. Linear models produce flat hyperplane boundaries; non-linear models produce curved ones.

Decision Tree A model that partitions the feature space using a series of binary threshold questions, forming a tree structure.

Dimensionality Reduction The process of reducing the number of features while retaining the most important information. Methods: PCA, t-SNE, UMAP, Autoencoders.

Dropout A regularization technique where a random fraction of neurons is deactivated during each training step to prevent co-adaptation.

E

Early Stopping Halting training when validation loss stops improving, preventing overfitting without explicit regularization.

Embedding A dense, lower-dimensional representation of a high-dimensional object (e.g., a word, image, user) that captures semantic meaning as a vector.

Ensemble A collection of multiple models whose predictions are combined (by averaging, voting, or stacking) to produce a stronger final prediction.

Epoch One complete pass through the entire training dataset during neural network training.

Explainability The degree to which a model's predictions can be understood and interpreted by humans.

F

F1 Score The harmonic mean of Precision and Recall: 2 × (P × R) / (P + R). Balances both metrics and is appropriate for imbalanced classes.

Feature An individual measurable input variable used by a model (e.g., temperature, age, pixel intensity).

Feature Engineering The process of transforming raw data into informative features that improve model performance.

Feature Importance A ranking of features by their contribution to a model's predictions. Available in tree-based models and computed generally via permutation or SHAP.

Fine-Tuning Adapting a pre-trained model's weights to a new target task, usually by continuing training with a smaller learning rate.

FN (False Negative) A positive example that the model incorrectly classified as negative. Also called a missed detection.

FP (False Positive) A negative example that the model incorrectly classified as positive. Also called a false alarm.

G

Generalization A model's ability to perform well on new, unseen data — not just on the training set.

GBM (Gradient Boosting Machine) An ensemble method that builds models sequentially, where each new model fits the pseudo-residuals (negative gradients) of the loss from the current ensemble.

GRU (Gated Recurrent Unit) A recurrent neural network variant with fewer parameters than LSTM, often achieving comparable sequence modeling performance.

Gradient The vector of partial derivatives of the loss with respect to model parameters, indicating the direction and magnitude of steepest loss increase.

Grid Search An exhaustive search over a specified hyperparameter grid, evaluating all combinations via cross-validation.

H

Hyperparameter A parameter that controls the learning process but is not learned from data (e.g., learning rate, number of trees, depth limit, regularization strength).

Hyperparameter Tuning The process of finding the best hyperparameter values using search strategies (Grid Search, Random Search, Bayesian Optimization).

I

Imputation Filling in missing values using a strategy such as mean, median, most frequent, or model-based prediction.

Inertia (K-Means) The sum of squared distances from each point to its assigned cluster centroid. Used as the K-Means optimization objective.

Isolation Forest An anomaly detection algorithm that isolates points by randomly partitioning the feature space. Anomalies are isolated quickly (short path lengths).

Iteration One forward-backward pass through a single mini-batch during training. One epoch = N/batch_size iterations.

K

K-Fold Cross-Validation Splits data into K equal parts (folds), trains on K-1 folds, tests on the remaining fold, and repeats K times. Averages the K scores.

Kernel (SVM) A function that computes the dot product of two points in a higher-dimensional feature space without explicitly computing the transformation. Enables non-linear decision boundaries.

KNN (K-Nearest Neighbors) A lazy learning algorithm that classifies or regresses a new point using the majority label or average value of its K nearest training examples.

L

L1 Regularization (Lasso) Adds the sum of absolute weight values to the loss. Promotes sparsity — drives unimportant feature weights to exactly zero.

L2 Regularization (Ridge) Adds the sum of squared weight values to the loss. Shrinks all weights continuously without zeroing them out.

Label The target output value for a supervised learning example. Also called the ground truth or target.

Latent Space The compressed, lower-dimensional representation learned by an encoder (e.g., in autoencoders or variational autoencoders).

Learning Rate (η) A hyperparameter that controls the step size during gradient-based optimization. Too large → unstable; too small → slow convergence.

LightGBM A gradient boosting framework using leaf-wise tree growth, histogram binning, and gradient-based sampling for fast training on large datasets.

Loss Function A function that measures the discrepancy between a model's prediction and the true label. The model is trained to minimize this.

LSTM (Long Short-Term Memory) A recurrent network cell with three gates (forget, input, output) and a cell state that allows learning long-range dependencies without vanishing gradients.

M

MAE (Mean Absolute Error) (1/N) Σ |y - ŷ|. Average absolute difference between predictions and true values. Robust to outliers.

MSE (Mean Squared Error) (1/N) Σ (y - ŷ)². Average squared difference. Penalizes large errors more heavily than MAE.

Mini-Batch A small subset of the training data processed together in one gradient update step. Balances the stability of batch gradient descent with the speed of stochastic gradient descent.

Model Complexity How flexible or expressive a model is. Higher complexity risks overfitting; lower complexity risks underfitting.

N

NaN (Not a Number) A floating-point representation of a missing or undefined value. Must be handled before training any ML model.

Neural Network A computational model loosely inspired by biological neurons. Composed of layers of interconnected nodes (neurons) that learn transformations from input to output.

Normalization Scaling features to a [0,1] range: (x - min) / (max - min). Sensitive to outliers. See also: Standardization.

O

Objective Function The function being optimized during training. In ML, this is typically the loss plus any regularization terms.

One-Hot Encoding Converting a categorical variable with K categories into K binary (0/1) columns. Required by most ML algorithms that don't natively handle categories.

Overfitting When a model performs very well on training data but poorly on new data because it has memorized noise instead of learning generalizable patterns.

P

PCA (Principal Component Analysis) A linear dimensionality reduction method that finds orthogonal axes of maximum variance and projects data onto them.

Pipeline A chained sequence of preprocessing and modeling steps that are applied in order. Scikit-Learn Pipeline ensures transformations are fitted only on training data.

Precision TP / (TP + FP). Of all predictions labelled positive, what fraction were actually positive?

PR-AUC Area Under the Precision-Recall Curve. More informative than ROC-AUC on imbalanced datasets.

Pseudo-Residuals The negative gradient of the loss with respect to current model predictions. Gradient Boosting fits each new tree to these pseudo-residuals.

R

R² (R-Squared) 1 - SS_res/SS_tot. Fraction of variance in the target explained by the model. 1.0 = perfect; 0.0 = model no better than predicting the mean; negative = worse than mean.

Random Forest An ensemble of decision trees where each tree is trained on a bootstrap sample and uses a random feature subset at each split. Predictions are aggregated by majority vote (classification) or mean (regression).

Recall TP / (TP + FN). Of all actual positives, what fraction did the model catch? Also called Sensitivity or True Positive Rate.

Regularization Techniques that add a penalty for model complexity to the loss function, reducing overfitting (e.g., L1, L2, Dropout, Early Stopping).

RMSE (Root Mean Squared Error) √MSE. Expressed in the same unit as the target. More interpretable than raw MSE.

ROC Curve Receiver Operating Characteristic curve. Plots True Positive Rate vs. False Positive Rate at all classification thresholds.

S

Scikit-Learn The standard Python ML library for classical algorithms (classification, regression, clustering, preprocessing, cross-validation).

Silhouette Score A metric for clustering quality ranging from -1 (wrong cluster) to +1 (well-separated). Near 0 = overlapping clusters.

SMOTE (Synthetic Minority Over-sampling Technique) A technique for handling class imbalance by generating synthetic minority class examples through interpolation between existing examples.

Standardization Scaling features to zero mean and unit variance: (x - μ) / σ. More robust than normalization when outliers are present.

SVM (Support Vector Machine) A model that finds the maximum-margin hyperplane separating two classes. Supports non-linear boundaries via kernel functions.

T

TN (True Negative) A negative example correctly classified as negative.

TP (True Positive) A positive example correctly classified as positive.

t-SNE A non-linear dimensionality reduction technique for 2D/3D visualization. Preserves local neighborhood structure. Cannot generalize to new data.

Transfer Learning Reusing weights from a model pre-trained on a large dataset as the starting point for a new, related task.

U

Underfitting When a model is too simple to capture the true relationship in the data. High bias, high training error.

UMAP Uniform Manifold Approximation and Projection. A faster, often better alternative to t-SNE for dimensionality reduction and visualization. Can generalize to new data.

V

Validation Set A held-out portion of data used to tune hyperparameters and monitor training. Distinct from the test set, which must remain untouched until final evaluation.

Variance (in ML context) The amount by which a model's predictions change across different training datasets. High variance = overfitting.

W

Weight A learned scalar parameter in a model (e.g., coefficient in linear regression, connection strength in a neural network).

X

XGBoost Extreme Gradient Boosting. An optimized, regularized gradient boosting implementation with second-order gradient approximations, parallel column subsampling, and built-in regularization.