Fundamentals of Artificial Neural Networks

Other popular activations include ReLU (Rectified Linear Unit) and tanh, each with distinct properties. While ReLU is preferred in hidden layers for its simplicity and gradient‑preserving behavior, sigmoid remains the go‑to choice for binary outputs.

2. The Perceptron Model: Simple Decision Units

A perceptron is the most elementary neural unit. It receives binary inputs x_i ∈ {0,1} and associated weights w_i ∈ [0,1]. The perceptron computes a weighted sum plus a bias term b:

• z = Σ w_i x_i + b

If z exceeds a predefined threshold (often set to 0), the perceptron fires, producing an output y = 1. Otherwise, the output is y = 0. This binary decision rule forms the basis for more complex multilayer networks.

3. Forward Propagation in Multilayer Networks

Forward propagation is the process of passing input data through successive layers to obtain predictions. For a network with layers indexed by k, the activation of layer k is computed as:

• Formula: A[k] = sigmoid(W[k]·A[k‑1] + B[k])
• Components:
  • W[k]: Weight matrix connecting layer k‑1 to layer k.
  • A[k‑1]: Activation vector from the previous layer.
  • B[k]: Bias vector for layer k.
• Why sigmoid? Using the sigmoid activation ensures the output stays in the (0,1) interval, which is especially useful for probability interpretation in classification tasks.

Alternative activations (ReLU, tanh) can replace sigmoid in hidden layers, but the mathematical pattern of W·A + B remains consistent across architectures.

4. Loss Functions and Gradient Computation

Training a neural network involves minimizing a loss function that measures the discrepancy between predicted outputs a and true targets y. For regression or binary classification, the Mean Squared Error (MSE) is frequently used:

• L = (1/m) Σ (y - a)², where m is the batch size.

When differentiating the MSE with respect to a weight w_i, the gradient term that appears is:

• Gradient term: (1/m) Σ (y - a)·a·(1 - a)·x_i
• This expression combines three key factors:
• The error (y - a).
• The derivative of the sigmoid activation a·(1 - a).
• The input feature x_i that contributed to the weight.

Understanding this gradient is crucial for implementing backpropagation correctly.

5. Historical Breakthroughs: Why Deep Learning Took Off After 2006

Although the backpropagation algorithm was introduced decades earlier, deep neural networks became practically trainable around 2006 due to two synergistic advances:

• GPU acceleration: Graphics Processing Units (GPUs) provided massive parallel computation, increasing processing speed by up to 10⁶ times compared to traditional CPUs.
• Large labeled datasets: Datasets such as ImageNet offered millions of annotated examples, enabling networks to learn rich representations.

These factors, together with improved initialization techniques and regularization methods, sparked the deep learning revolution.

6. The Universal Approximation Theorem

The Universal Approximation Theorem guarantees that a feed‑forward network with at least one hidden layer can approximate any continuous bounded function to an arbitrary degree of accuracy, provided it has enough neurons. Key points include:

• It does **not** restrict the network to piecewise linear functions.
• A single hidden layer suffices; depth is not required for universal approximation, though deeper architectures often learn more efficiently.
• The theorem does **not** specify the exact number of layers or neurons needed for a given error tolerance.

7. Backpropagation: Computing Gradients in Hidden Layers

During the backward pass, the gradient for a weight w_ij in a hidden layer is calculated using the chain rule. The gradient depends on:

• The gradient of the subsequent layer (often denoted as δ_{next}).
• The activation of the current neuron (a_i).
• Mathematically: ∂L/∂w_ij = δ_{next} · a_i.

This relationship highlights why information must flow backward through the network to update earlier weights.

8. Stochastic Gradient Descent (SGD) and Error Definition

When the batch size m = 1, the training algorithm becomes Stochastic Gradient Descent. The error for a single data point X is defined as:

• Loss for one sample: E = ½ (a(X) - y)²
• The factor ½ simplifies the derivative, yielding (a - y) during backpropagation.

SGD updates weights after each sample, offering faster convergence on large datasets despite higher variance in the gradient estimate.

9. Putting It All Together: A Simple Example

Consider a binary classification task with a single hidden layer:

1. Input layer: Features x₁, x₂ (binary).
2. Hidden layer: Two neurons with weights w₁₁, w₁₂ and w₂₁, w₂₂, bias b_h, and sigmoid activation.
3. Output layer: One neuron with weights v₁, v₂, bias b_o, and sigmoid activation producing a.

During forward propagation, compute:

• h₁ = σ(w₁₁·x₁ + w₁₂·x₂ + b_h)
• h₂ = σ(w₂₁·x₁ + w₂₂·x₂ + b_h)
• a = σ(v₁·h₁ + v₂·h₂ + b_o)

Then calculate the loss E = ½ (a - y)² and backpropagate the error to update all weights using the gradients described earlier.

10. Key Takeaways

• Sigmoid activation maps real numbers to (0,1) and is ideal for binary outputs.
• A perceptron outputs 1 when its weighted sum plus bias exceeds a threshold.
• Forward propagation follows the pattern A[k] = sigmoid(W[k]·A[k‑1] + B[k]).
• The gradient of MSE with respect to a weight includes the term (y - a)·a·(1 - a)·x_i.
• GPU acceleration and large datasets made deep learning feasible after 2006.
• The Universal Approximation Theorem assures that a single hidden layer can approximate any continuous bounded function.
• Backpropagation gradients in hidden layers depend on the next layer’s gradient and the current neuron’s activation.
• For SGD (batch size 1), the error is ½ (a - y)².

By mastering these foundational concepts, you are well‑prepared to explore more advanced topics such as convolutional networks, recurrent architectures, and modern optimization techniques.