Fundamentals of Artificial Neural Networks

Fundamentals of Artificial Neural Networks

Welcome to this comprehensive module on the Fundamentals of Artificial Neural Networks (ANNs). Designed for learners interested in computer science and artificial intelligence, this course breaks down the core concepts that appear in typical quiz questions, providing clear explanations, illustrative examples, and SEO‑friendly language.

1. From Single‑Layer Perceptrons to Deep Networks

A single‑layer perceptron is the earliest form of a neural model. It computes a weighted sum of its inputs and passes the result through a step‑function activation. While historically important, it has a crucial limitation:

• Linear separability: The perceptron can only represent linear functions. If the data cannot be separated by a straight line (or hyperplane in higher dimensions), the perceptron fails.

Deep neural networks overcome this restriction by stacking multiple layers, each with non‑linear activation functions. The composition of layers enables the network to approximate highly complex, non‑linear mappings.

2. The Binary McCulloch‑Pitts Neuron

The McCulloch‑Pitts model is a classic binary neuron. Its output y becomes 1 when the following condition holds:

• Weighted sum plus bias exceeds the threshold: Σ w_i x_i + b > θ. In other words, the neuron fires only if the combined influence of inputs and bias surpasses a predefined threshold.

This simple rule forms the basis for more sophisticated activation functions used today.

3. Sigmoid Activation and Its Derivative

The sigmoid function is defined as a(z) = 1 / (1 + e^{-z}). Its smooth, S‑shaped curve makes it suitable for binary classification. A key property for training via gradient descent is its derivative, which can be expressed compactly in terms of the activation itself:

• Derivative: a'(z) = a(z)·(1 – a(z)). This form simplifies back‑propagation because the derivative can be computed directly from the forward‑pass activation.

4. Forward Propagation: Computing the Linear Combination

During forward propagation, each layer k receives an activation matrix A^{[k‑1]} from the previous layer. The linear combination matrix Z^{[k]} is calculated as:

• Formula: Z^{[k]} = W^{[k]}·A^{[k‑1]} + B^{[k]}, where W^{[k]} is the weight matrix and B^{[k]} is the bias vector (broadcasted across columns).

After obtaining Z^{[k]}, an activation function (e.g., ReLU, sigmoid) is applied to produce A^{[k]} for the next layer.

5. Storing Information for Gradient Computation

When training a perceptron with a quadratic loss, the gradient with respect to the weights requires a specific term from the forward pass:

• Stored term: the activation a^{(k)} for each training example. This value is needed to compute the error (a^{(k)} – y^{(k)}) and subsequently update the weights.

Other quantities such as the raw linear combination z^{(k)} or the bias are not directly required for the weight gradient in this simple case.

6. Universal Approximation Theorem

The Universal Approximation Theorem is a cornerstone of neural network theory. It states that:

• Statement: Any continuous bounded function on a compact domain can be approximated arbitrarily well by a multilayer perceptron (MLP) with at least one hidden layer, provided the hidden layer has a sufficient number of neurons.

This theorem does not guarantee perfect representation of every discrete function, nor does it prescribe the exact number of layers or neurons needed. It simply assures the existence of a network capable of achieving any desired level of approximation accuracy.

7. Log‑Likelihood Loss for Binary Classification

When modeling binary outcomes with a Bernoulli distribution, the appropriate loss function is the negative log‑likelihood (also called binary cross‑entropy). For a dataset of m examples, the loss L(B) is:

• Expression: L(B) = (1/m) Σ_{k=1}^{m} [ y_k·ln(a_k) + (1‑y_k)·ln(1‑a_k) ], where y_k is the true label and a_k is the predicted probability.

This formulation penalizes confident but incorrect predictions, encouraging the network to output probabilities that reflect true class frequencies.

8. Back‑Propagation: Gradient Flow Through Hidden Layers

Back‑propagation computes gradients by moving from the output layer toward the input layer. For a weight w_j in a hidden layer, the gradient depends on the downstream error term and the derivative of the downstream activation:

• Gradient relationship: ∂L/∂w_j = δ^{[l+1]}·a^{[l]}, where δ^{[l+1]} is the error signal from the next layer (downstream) and a^{[l]} is the activation from the current layer (upstream). The error signal itself incorporates the downstream activation derivative, linking the two layers.

This dependency ensures that weight updates in earlier layers are informed by how errors propagate through the entire network.

9. Putting It All Together: A Mini‑Project Walkthrough

To solidify your understanding, consider building a simple binary classifier using a two‑layer MLP:

1. Data preparation: Generate a synthetic dataset with two classes that are not linearly separable (e.g., concentric circles).
2. Network architecture: Input layer → hidden layer (ReLU activation) → output layer (sigmoid activation).
3. Forward pass: Compute Z^{[1]} = W^{[1]}·X + B^{[1]}, apply ReLU to obtain A^{[1]}, then compute Z^{[2]} = W^{[2]}·A^{[1]} + B^{[2]} and apply sigmoid to get predictions a.
4. Loss calculation: Use the binary cross‑entropy loss defined above.
5. Backward pass: Calculate the error term for the output layer, propagate it back to the hidden layer using the derivative of ReLU, and update all weights with gradient descent.
6. Training loop: Iterate over epochs, monitoring loss reduction and classification accuracy.

By following these steps, you will experience firsthand how each concept—from the linear combination Z to the universal approximation property—plays a role in building functional neural networks.

10. Key Takeaways

• Single‑layer perceptrons are limited to linear functions; deep networks introduce non‑linearity.
• The McCulloch‑Pitts neuron fires when the weighted sum plus bias exceeds a threshold.
• Sigmoid derivative can be expressed as a·(1‑a), simplifying back‑propagation.
• Forward propagation uses Z = W·A_{prev} + B for each layer.
• During training, store the activation values needed for gradient calculations.
• The universal approximation theorem guarantees that MLPs can approximate any continuous function given enough hidden units.
• Binary classification loss is best expressed with the log‑likelihood (cross‑entropy) formula.
• Back‑propagation gradients in hidden layers depend on downstream error signals and activation derivatives.

Mastering these fundamentals equips you with the knowledge to explore more advanced topics such as convolutional networks, recurrent architectures, and modern optimization techniques.