Deep Learning Foundations

Deep learning uses layered neural networks to learn hierarchical representations from data. Each layer transforms its input into a slightly more abstract representation, allowing the network to learn complex patterns that flat feature engineering cannot capture.

What Makes Deep Learning Different

Classical ML algorithms (Random Forest, SVMs) operate on hand-crafted feature vectors. Deep learning learns the feature representation itself — from raw pixels, raw audio, or raw sensor sequences — through end-to-end gradient-based optimization.

This is powerful when:

• Data is high-dimensional (images, audio, text, time-series)
• Human-engineered features miss important structural patterns
• Very large labeled datasets are available

This is unnecessary when:

• Dataset is small (< a few thousand labeled examples)
• Tabular data with clean, interpretable features
• Interpretability of decision process is required
• A tree ensemble already achieves target performance

Building Blocks

Activation Functions

The Training Loop

Every deep learning training run executes this cycle:

1. Forward pass — data flows through layers, producing a prediction
2. Loss computation — loss function compares prediction to ground truth
3. Backward pass (backpropagation) — computes gradient ∂L/∂w for every weight using the chain rule
4. Optimizer step — updates each weight: w ← w - η·∂L/∂w
5. Repeat for many mini-batches (iterations) across multiple epochs

Key Hyperparameters

Architecture 1: Dense Network (MLP)

Think of a game of telephone where each person in a line transforms a message slightly before passing it to the next person. Raw, complex input goes in at one end. By the time it reaches the last person, it has been condensed into a clear, simplified answer — a class label or a number. A Dense network does this with numbers, layer by layer, until the final layer gives the prediction.

A Multi-Layer Perceptron consists of fully connected (Dense) layers. Every neuron in one layer connects to every neuron in the next.

Use when: Tabular data, structured feature vectors, small classification/regression tasks.

Forward equations for a 3-layer MLP: h¹ = ReLU(W¹x + b¹) h² = ReLU(W²h¹ + b²) ŷ = softmax(W³h² + b³) (for multi-class)

1import tensorflow as tf
2from tensorflow import keras
3from sklearn.model_selection import train_test_split
4from sklearn.datasets import make_classification
5from sklearn.preprocessing import StandardScaler
6import numpy as np
7
8X, y = make_classification(n_samples=4000, n_features=20, n_classes=3,
9                           n_informative=12, random_state=42)
10X = StandardScaler().fit_transform(X)
11X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
12
13model = keras.Sequential([
14    keras.layers.Input(shape=(20,)),
15    keras.layers.Dense(128, activation="relu"),
16    keras.layers.BatchNormalization(),
17    keras.layers.Dropout(0.3),
18    keras.layers.Dense(64, activation="relu"),
19    keras.layers.BatchNormalization(),
20    keras.layers.Dropout(0.3),
21    keras.layers.Dense(3, activation="softmax"),
22])
23
24model.compile(
25    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
26    loss="sparse_categorical_crossentropy",
27    metrics=["accuracy"]
28)
29model.summary()
30
31history = model.fit(
32    X_train, y_train,
33    validation_data=(X_val, y_val),
34    epochs=50, batch_size=64,
35    callbacks=[keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)]
36)

Architecture 2: Convolutional Neural Network (CNN)

Imagine a team of inspectors scanning a newspaper photo with magnifying glasses. The first team looks only for straight edges. The second team combines those edges into corners and curves. The third team assembles the corners into faces and objects. Each layer reports its findings to the next, and the deeper you go, the more abstract and meaningful the pattern becomes.

CNNs apply learned filters that slide across the spatial dimensions of an input (image, 1D signal). Each filter learns to detect a specific local pattern — edges, textures, frequency components.

Use when: Image classification, object detection, signal pattern recognition, time-series with local structure.

Key Layers:

• Conv2D(filters, kernel_size) — applies F filters over local regions
• MaxPooling2D — downsamples, reducing spatial dimensions and computation
• GlobalAveragePooling2D — collapses spatial dimensions before Dense layers
• BatchNormalization — stabilizes training by normalizing layer inputs

Math — Convolution: (f * x)[i, j] = Σ_m Σ_n f[m, n] · x[i+m, j+n]

1# Example: image classification (e.g., 32x32 grayscale sensor images)
2cnn = keras.Sequential([
3    keras.layers.Input(shape=(32, 32, 1)),
4    keras.layers.Conv2D(32, (3,3), activation="relu", padding="same"),
5    keras.layers.BatchNormalization(),
6    keras.layers.MaxPooling2D((2,2)),
7
8    keras.layers.Conv2D(64, (3,3), activation="relu", padding="same"),
9    keras.layers.BatchNormalization(),
10    keras.layers.MaxPooling2D((2,2)),
11
12    keras.layers.Conv2D(128, (3,3), activation="relu", padding="same"),
13    keras.layers.GlobalAveragePooling2D(),
14
15    keras.layers.Dense(64, activation="relu"),
16    keras.layers.Dropout(0.4),
17    keras.layers.Dense(10, activation="softmax"),
18])
19
20cnn.compile(
21    optimizer="adam",
22    loss="sparse_categorical_crossentropy",
23    metrics=["accuracy"]
24)

Architecture 3: Recurrent Networks (RNN, LSTM, GRU)

Imagine reading a mystery novel. What the detective discovers in Chapter 10 only makes sense because of clues from Chapters 1 through 9. A plain RNN tries to remember past chapters but forgets them quickly as the novel gets longer. An LSTM keeps a dedicated notebook — it selectively writes down what matters, crosses out what doesn’t, and holds onto critical plot points even 100 chapters later.

Recurrent networks maintain a hidden state across timesteps, allowing them to model sequential dependencies.

Use when: Time-series prediction, sensor data streams, text sequences, anomaly detection on sequential signals.

Why vanilla RNNs fail (Vanishing Gradient): During backpropagation through time, gradients are multiplied by weight matrices at every timestep. For long sequences, these multiplications shrink gradients exponentially → weights at early timesteps receive almost no learning signal.

LSTM gates (solution to vanishing gradient): The Long Short-Term Memory cell uses three learned gates:

• Forget gate f_t = σ(W_f[h_{t-1}, x_t] + b_f) — what to discard from memory
• Input gate i_t = σ(W_i[h_{t-1}, x_t] + b_i) — what new info to write
• Output gate o_t = σ(W_o[h_{t-1}, x_t] + b_o) — what to expose to next layer

Cell state update: c_t = f_t ⊙ c_{t-1} + i_t ⊙ tanh(W_c[h_{t-1}, x_t] + b_c)

1# Time-series classification: sensor window → fault label
2timesteps  = 100
3n_features = 6
4n_classes  = 4
5
6lstm_model = keras.Sequential([
7    keras.layers.Input(shape=(timesteps, n_features)),
8    keras.layers.LSTM(128, return_sequences=True),
9    keras.layers.LSTM(64, return_sequences=False),
10    keras.layers.Dense(32, activation="relu"),
11    keras.layers.Dropout(0.3),
12    keras.layers.Dense(n_classes, activation="softmax"),
13])
14
15lstm_model.compile(
16    optimizer="adam",
17    loss="sparse_categorical_crossentropy",
18    metrics=["accuracy"]
19)
20
21# GRU — Fewer parameters than LSTM, often comparable quality
22gru_model = keras.Sequential([
23    keras.layers.Input(shape=(timesteps, n_features)),
24    keras.layers.GRU(128, return_sequences=True),
25    keras.layers.GRU(64),
26    keras.layers.Dense(n_classes, activation="softmax"),
27])

Architecture 4: Autoencoders

A student must summarise a 500-page textbook into a single A4 page — capturing only the most essential ideas. Then, using only that summary, they try to reconstruct the full textbook. If the reconstruction is accurate, the summary preserved what mattered. If it fails badly, the summary lost critical information. In machine learning, data that fails to reconstruct accurately is flagged as anomalous.

An autoencoder consists of an Encoder that compresses input into a compact latent vector z, and a Decoder that reconstructs the original input from z. Trained by minimizing reconstruction error.

Use when:

• Dimensionality reduction — latent vector is the compressed representation
• Anomaly detection — normal data reconstructs well; anomalies have high reconstruction error
• Denoising — train on (noisy input, clean output) pairs
• Pre-training for downstream tasks

1# Autoencoder for anomaly detection on tabular sensor data
2input_dim  = 20
3latent_dim = 4
4
5# Encoder
6encoder_input = keras.Input(shape=(input_dim,))
7x = keras.layers.Dense(64, activation="relu")(encoder_input)
8x = keras.layers.Dense(32, activation="relu")(x)
9latent = keras.layers.Dense(latent_dim, activation="relu", name="latent")(x)
10
11# Decoder
12x = keras.layers.Dense(32, activation="relu")(latent)
13x = keras.layers.Dense(64, activation="relu")(x)
14output = keras.layers.Dense(input_dim, activation="linear")(x)
15
16autoencoder = keras.Model(encoder_input, output)
17autoencoder.compile(optimizer="adam", loss="mse")
18
19# Train on NORMAL data only
20# autoencoder.fit(X_normal, X_normal, epochs=50, validation_split=0.1)
21
22# Detect anomalies: high reconstruction error = anomaly
23# recon_error = np.mean(np.abs(X_test - autoencoder.predict(X_test)), axis=1)
24# anomalies = recon_error > threshold

Regularization Techniques

1# Using all three regularization techniques together
2model = keras.Sequential([
3    keras.layers.Dense(256, activation="relu",
4                       kernel_regularizer=keras.regularizers.l2(1e-4)),
5    keras.layers.BatchNormalization(),
6    keras.layers.Dropout(0.4),
7    keras.layers.Dense(128, activation="relu"),
8    keras.layers.Dropout(0.3),
9    keras.layers.Dense(3, activation="softmax"),
10])

Diagnosing Training Problems

1# Gradient clipping — essential for RNNs
2optimizer = keras.optimizers.Adam(learning_rate=1e-3, clipnorm=1.0)

Learning Rate Scheduling

The learning rate is the most important hyperparameter. Constant learning rates are rarely optimal — starting high and decaying helps convergence.

1# Reduce learning rate when validation loss plateaus
2lr_scheduler = keras.callbacks.ReduceLROnPlateau(
3    monitor="val_loss", factor=0.5,
4    patience=5, min_lr=1e-6, verbose=1
5)
6
7# Cosine decay schedule
8cosine_decay = keras.optimizers.schedules.CosineDecay(
9    initial_learning_rate=1e-3,
10    decay_steps=1000
11)
12optimizer = keras.optimizers.Adam(learning_rate=cosine_decay)

Deploying to Edge Devices (TensorFlow Lite)

Embedded devices (ESP32-S3, STM32H7, Raspberry Pi) cannot run full TensorFlow models. TensorFlow Lite converts and compresses models for constrained hardware.

1import tensorflow as tf
2
3# Step 1: Convert to TFLite
4converter = tf.lite.TFLiteConverter.from_keras_model(model)
5tflite_model = converter.convert()
6
7# Step 2: Save
8with open("model.tflite", "wb") as f:
9    f.write(tflite_model)
10
11# Step 3: Quantize (float32 → int8) — reduces model size 4x, speeds up inference
12converter.optimizations = [tf.lite.Optimize.DEFAULT]
13quantized_model = converter.convert()
14
15with open("model_int8.tflite", "wb") as f:
16    f.write(quantized_model)
17
18# Step 4: Run inference using the TFLite interpreter
19interpreter = tf.lite.Interpreter(model_path="model_int8.tflite")
20interpreter.allocate_tensors()
21
22input_details  = interpreter.get_input_details()
23output_details = interpreter.get_output_details()
24
25import numpy as np
26test_input = np.array([X_val[0]], dtype=np.float32)
27interpreter.set_tensor(input_details[0]["index"], test_input)
28interpreter.invoke()
29output = interpreter.get_tensor(output_details[0]["index"])
30print("Prediction:", output)

Checklist Before Training

• [ ] Problem is clearly defined with a measurable target metric
• [ ] Dataset is split correctly (no leakage from validation/test into training)
• [ ] Data is normalized/scaled before feeding into the network
• [ ] Baseline ML model performance is recorded for comparison
• [ ] Architecture starts small — grow only if needed
• [ ] Validation curves are being monitored per epoch
• [ ] Early stopping callback is active
• [ ] Final evaluation is on a held-out test set that was never touched