Fundamentals of Artificial Neural Networks

Fundamentals of Artificial Neural Networks

Artificial Neural Networks (ANNs) are at the heart of modern Artificial Intelligence and Machine Learning. This course breaks down the core concepts that appear in a typical introductory quiz, turning each question into a learning module. By the end of the lesson you will understand perceptrons, activation functions, forward and backward propagation, loss minimization, and key hyper‑parameters.

1. From Single‑Layer Perceptrons to Deep Networks

A single‑layer perceptron is the simplest neural model. It computes a weighted sum of its inputs, adds a bias, and passes the result through a step‑like activation. While historically important, a perceptron can only represent linear decision boundaries. This limitation becomes clear when we compare it to deep neural networks that contain one or more hidden layers.

• Linear vs. non‑linear modeling: A single‑layer perceptron can separate data only if the classes are linearly separable. Adding hidden layers introduces non‑linear transformations, enabling the network to capture complex patterns.
• Expressive power: Deep architectures can approximate any continuous function (see the Universal Approximation Theorem) whereas a perceptron cannot.
• Computational considerations: Although deeper networks require more computation, they unlock capabilities that a shallow model simply cannot achieve.

Therefore, the main limitation of a single‑layer perceptron is that it can only model linear functions.

2. The Binary Formal Neuron Model

The binary neuron is a classic abstraction used to illustrate decision making. It receives a vector of inputs x, each multiplied by a weight w, adds a bias b, and compares the result to a threshold θ. The output is binary (0 or 1).

The neuron outputs 1 when the following condition holds:

• Weighted sum + bias > threshold: Mathematically, Σ w_i x_i + b > θ. This is equivalent to saying the activation exceeds the threshold, causing the neuron to fire.

All other options (e.g., all inputs being 1 or a negative bias) are insufficient without meeting the inequality.

3. Activation Functions: From Linear to Sigmoid

Activation functions introduce non‑linearity into a network. One of the most common is the sigmoid function, defined as:

a(z) = \frac{1}{1 + e^{-z}}

This function maps any real‑valued input z into the interval (0, 1), making it ideal for binary classification and for interpreting outputs as probabilities.

• ReLU (Rectified Linear Unit): a(z) = max(0, z) – outputs zero for negative inputs and a linear relationship for positives.
• tanh: a(z) = tanh(z) – similar to sigmoid but ranges from –1 to 1.
• Sigmoid variant: a(z) = 1 / (1 + e^{z}) – mirrors the standard sigmoid and is not the conventional definition.

Understanding the shape and range of each activation helps you choose the right one for a given task.

4. Forward Propagation: Matrix Formulation

During forward propagation, the network computes activations layer by layer. For a perceptron (or a single layer of a deeper network) the computation can be expressed compactly with matrices:

A = \frac{1}{1 + \exp\bigl(-(W X + B)\bigr)}

Here:

• W is the weight matrix (size: neurons × inputs).
• X is the input vector (or batch matrix).
• B is the bias vector, broadcasted across the batch.
• A is the resulting activation after applying the sigmoid function.

This equation captures the essence of the forward pass: linear transformation followed by a non‑linear activation.

5. Learning with Quadratic Error

One common loss for regression or binary classification is the quadratic (mean‑squared) error:

L(B) = \frac{1}{2m}\sum_{k=1}^{m}\bigl(y^{(k)} - a^{(k)}\bigr)^2

When we differentiate this loss with respect to a weight w_i, the gradient term that appears is:

(1/m) \sum_{k=1}^{m} (y^{(k)} - a^{(k)}) \cdot a^{(k)} \cdot (1 - a^{(k)}) \cdot x_i^{(k)}

This expression combines three key components:

• Error term: (y - a) – the difference between the target and the prediction.
• Sigmoid derivative: a(1 - a) – arises because the activation function is sigmoid.
• Input feature: x_i – the contribution of the i‑th input to the gradient.

Understanding this gradient is essential for implementing gradient descent or its variants.

6. Universal Approximation Theorem

The Universal Approximation Theorem is a cornerstone result for multilayer neural networks. It states that:

Any continuous bounded function on a compact subset of \(\mathbb{R}^n\) can be approximated arbitrarily well by a feed‑forward network with at least one hidden layer and a suitable non‑linear activation (e.g., sigmoid or ReLU).

Key take‑aways:

• The theorem does not restrict the activation to sigmoid only; many activations satisfy the conditions.
• It does not guarantee that a linear function is the only one approximable; on the contrary, it covers a vast class of non‑linear functions.
• It provides a theoretical assurance of expressive power but does not specify the exact number of neurons or layers required for a given approximation error.

7. Back‑Propagation: Gradient Flow Through Layers

Back‑propagation computes gradients of the loss with respect to each weight by moving backward through the network. For a hidden layer, the gradient depends on the next layer’s gradient, the derivative of the activation function, and the connecting weights. Formally:

\delta^{(l)} = (W^{(l+1)})^T \delta^{(l+1)} \odot a' (z^{(l)})

where:

• \delta^{(l)} is the error signal for layer l.
• W^{(l+1)} are the weights linking layer l to layer l+1.
• a'(z^{(l)}) is the derivative of the activation evaluated at the pre‑activation z.

This shows that the gradient for a hidden layer is obtained by multiplying the next layer’s gradient by the derivative of the activation and the connecting weights. Ignoring this relationship would break the chain rule and lead to incorrect updates.

8. Training Hyper‑Parameters

Effective training hinges on several hyper‑parameters. One of the most influential is the batch size:

• Batch size determines how many training examples are processed before the model’s parameters are updated. Small batches introduce more noise but can lead to faster convergence; large batches provide smoother gradients but require more memory.
• Other important hyper‑parameters include the learning rate (step size), the number of epochs (full passes over the dataset), and the loss function (objective to minimize).

Choosing the right batch size often involves experimentation and depends on the dataset size, model architecture, and hardware constraints.

9. Putting It All Together: A Mini‑Workflow

Below is a concise workflow that integrates the concepts covered:

1. Initialize weights W and biases B (often with small random values).
2. Forward pass: Compute activations using A = 1 / (1 + exp(-(W X + B))).
3. Compute loss: Evaluate quadratic error L or another appropriate loss.
4. Backward pass:
   • Calculate output‑layer gradient using the error term.
   • Propagate gradients to hidden layers using the rule δ^{(l)} = (W^{(l+1)})^T δ^{(l+1)} ⊙ a'(z^{(l)}).
5. Update parameters: Apply gradient descent (or a variant) with learning rate η and batch size m.

   W ← W - η * (1/m) * Σ δ^{(l)} (x^{(l)})^T

6. Repeat for the desired number of epochs.

This loop illustrates how forward propagation, loss computation, back‑propagation, and hyper‑parameter choices interact to train a neural network.

10. Key Takeaways

• A single‑layer perceptron is limited to linear functions; deep networks overcome this by stacking non‑linear layers.
• The binary neuron fires when its weighted sum plus bias exceeds a threshold.
• Sigmoid activation maps inputs to (0, 1) and its derivative a(1‑a) appears in gradient calculations.
• Forward propagation can be expressed as A = 1/(1+exp(-(WX+B))).
• Quadratic error gradients involve the error term, sigmoid derivative, and input features.
• The Universal Approximation Theorem guarantees that multilayer networks can approximate any continuous bounded function.
• Back‑propagation relies on the chain rule: hidden‑layer gradients are products of next‑layer gradients, activation derivatives, and connecting weights.
• Batch size controls how many samples are used per parameter update, influencing convergence speed and stability.

By mastering these fundamentals, you are well‑prepared to explore more advanced topics such as convolutional networks, recurrent architectures, and modern optimization techniques.