Fundamentals of Artificial Neural Networks

Fundamentals of Artificial Neural Networks

Artificial Neural Networks (ANNs) are at the heart of modern artificial intelligence and machine learning. This course unpacks the core concepts tested in a typical quiz on neural networks, turning each question into a learning module. By the end, you will understand perceptrons, activation functions, the universal approximation theorem, gradient descent, back‑propagation, and key hyper‑parameters such as batch size.

1. Single‑Layer Perceptron vs. Deep Neural Networks

A single‑layer perceptron is the simplest form of a neural network: it computes a weighted sum of inputs and passes the result through an activation function. While historically important, it has a critical limitation:

• Only linear functions can be modeled. The perceptron can separate data that is linearly separable, but it cannot capture complex, non‑linear relationships that deep networks can learn through multiple hidden layers.

Deep neural networks overcome this restriction by stacking layers, each applying non‑linear transformations, enabling the model to approximate highly intricate functions.

2. Computing the Activation in a Perceptron

During forward propagation, the perceptron first calculates the weighted sum z(k) for the k‑th example:

z(k) = \sum_{i} w_i \cdot x_i(k) + b

It then applies the sigmoid activation function to obtain the activation a(k):

a(k) = \frac{1}{1 + e^{-z(k)}}

This S‑shaped curve maps any real‑valued input to the interval (0,1), making it ideal for binary classification.

3. Universal Approximation Theorem

The universal approximation theorem is a cornerstone of neural network theory. It states that:

• Any continuous bounded function on a compact domain can be approximated arbitrarily well by a multilayer feed‑forward network with at least one hidden layer, provided the hidden layer has enough neurons.

Key takeaways:

• The theorem does not guarantee perfect approximation with a single hidden layer; it only assures the existence of a sufficiently large network.
• It does not specify the exact number of layers or neurons required; that depends on the function’s complexity.

4. Gradient Descent Update for the Bias Term

When training a perceptron with the cross‑entropy loss, the gradient of the loss L with respect to the bias b is:

\frac{\partial L}{\partial b} = \frac{1}{m}\sum_{k=1}^{m} (y(k) - a(k)) \cdot a(k) \cdot (1 - a(k))

Here, m is the number of training examples, y(k) the true label, and a(k) the predicted activation. This term appears in the numerator of the bias update rule:

b \leftarrow b - \eta \frac{\partial L}{\partial b}

where \eta is the learning rate.

5. Derivative of the Sigmoid Activation

The sigmoid function a = 1/(1+e^{-z}) has a convenient derivative expressed directly in terms of the activation itself:

\frac{da}{dz} = a \cdot (1 - a)

This property simplifies back‑propagation because the gradient can be computed without re‑evaluating the exponential term.

6. Why Back‑Propagation Becomes More Complex with Depth

Back‑propagation relies on the chain rule to propagate error gradients from the output layer back to earlier layers. As the number of layers increases:

• Each layer’s gradient depends on the gradient of the subsequent layer, creating a cascade of multiplications.
• These repeated multiplications can lead to vanishing or exploding gradients, making training deeper networks more challenging.

Understanding this dependency is essential for designing architectures, choosing appropriate activation functions, and applying techniques such as gradient clipping or residual connections.

7. Controlling Training Frequency: Batch Size

The batch size hyper‑parameter determines how many training examples are processed before the model’s parameters are updated. Common settings include:

• Mini‑batch (e.g., 32, 64, 128 samples) – balances computational efficiency and gradient noise.
• Full‑batch – uses the entire dataset for each update, often slower but yields stable gradients.
• Stochastic – batch size of 1, leading to noisy but frequent updates.

Choosing the right batch size influences convergence speed, memory usage, and generalization performance.

8. Matrix Operations in Forward Passes

In a multilayer network, each layer k computes a pre‑activation matrix Z[k] using the weight matrix W[k], the activation from the previous layer A[k‑1], and the bias vector B[k]:

Z[k] = W[k] \cdot A[k-1] + B[k]

This operation follows the standard linear algebra convention where W[k] (shape: neurons_k × neurons_{k-1}) multiplies A[k-1] (shape: neurons_{k-1} × batch_size). Adding the bias B[k] (broadcasted across the batch) yields the raw scores before applying the activation function.

9. Putting It All Together: A Mini‑Course Summary

Below is a concise checklist to reinforce the concepts covered:

• Perceptron limitation: can only model linear functions.
• Sigmoid activation: a = 1/(1+e^{-z}) with derivative a(1-a).
• Universal approximation: multilayer networks can approximate any continuous bounded function given enough neurons.
• Bias gradient: includes the term (y - a)·a·(1-a) scaled by 1/m.
• Back‑propagation: complexity grows with depth due to the chain rule.
• Batch size: controls how many examples are processed before each parameter update.
• Forward matrix formula: Z[k] = W[k]·A[k-1] + B[k].

Mastering these fundamentals equips you to design, train, and troubleshoot neural networks across a wide range of AI applications.