LLM Architecture, Training and Evaluation

Understanding Large Language Model (LLM) Architecture

Large language models have transformed the field of artificial intelligence by mastering natural language tasks. This course breaks down the core components that make LLMs work, from the foundational Transformer architecture to advanced techniques such as Mixture‑of‑Experts (MoE) and Retrieval‑Augmented Generation. Each section is designed to be SEO‑friendly, using clear headings, keyword‑rich paragraphs, and semantic HTML for optimal discoverability.

1. The Transformer Backbone: Self‑Attention

The self‑attention mechanism is the heart of modern Transformers. Unlike convolutional pooling or recurrent gating, self‑attention allows every token in a sequence to weigh the relevance of every other token, creating a dynamic, context‑aware representation.

• Key idea: For each token, compute query (Q), key (K), and value (V) vectors.
• Computation: Attention scores are derived from the dot‑product of Q and K, scaled, and passed through a softmax to obtain weights.
• Result: The weighted sum of V vectors produces the attended representation.

Because self‑attention processes the entire sequence in parallel, it scales efficiently on modern hardware and captures long‑range dependencies that traditional RNNs struggle with.

2. Causal Language Modeling (CLM) Loss

In a causal language model, the objective is to predict the next token given all previous tokens. The loss function is expressed as:

− Σₜ log P(tₜ | t₁…tₜ₋₁)

This formulation ensures that during training the model only attends to past tokens, preserving the autoregressive property required for generation.

3. Mixture‑of‑Experts (MoE) for Efficient Scaling

MoE architectures introduce a set of specialized "experts"—typically feed‑forward sub‑networks—that are sparsely activated. The primary advantage is that only a small subset of experts is activated per token, keeping compute roughly constant while the model size grows. This sparsity enables training models with billions of parameters without a proportional increase in inference cost.

• Routing networks decide which experts to use for each token.
• During back‑propagation, only the selected experts receive gradients, reducing memory usage.
• MoE has been a cornerstone of models like GLaM and Switch‑Transformer.

4. Positional Encodings that Generalise

Transformers need a way to incorporate token order. While learned absolute embeddings work well within the training length, they struggle to extrapolate. Rotary Position Embedding (RoPE) solves this by rotating query and key vectors based on token positions, allowing the model to generalise to longer sequences than seen during training.

RoPE integrates seamlessly with self‑attention, preserving the dot‑product structure while encoding relative positions.

5. Reinforcement Learning from Human Feedback (RLHF)

RLHF aligns LLM outputs with human preferences. The Reward Model (RM) predicts a scalar score that reflects how well a response matches human judgments. This score guides policy optimisation, typically via Proximal Policy Optimization (PPO), to fine‑tune the language model toward desirable behaviour.

• Data collection: Human annotators rank model outputs.
• Reward model training: A regression head learns to predict these rankings.
• Policy update: The LLM is updated to maximise the predicted reward.

6. Scaling Laws: The Chinchilla Exponent

Research on scaling laws, notably the Chinchilla study, shows that performance scales with model size raised to a small exponent. The exponent αₙ ≈ 0.076 indicates that while larger models improve performance, the gains diminish rapidly compared to increasing data volume.

Think of it as adding a few drops of water to an already full bucket—each drop helps, but the impact is modest.

7. Gemini 1.5 Architecture: KV‑Cache Compression

Gemini 1.5 introduces a novel component called Multi‑Head Latent Attention (MLA) that compresses the key‑value (KV) cache. By projecting the KV pairs into a latent space before storage, MLA reduces memory footprint while preserving essential contextual information, enabling longer context windows without prohibitive hardware costs.

8. Decoding Strategies: Balancing Diversity and Coherence

When generating text, the choice of decoding strategy dramatically affects output quality. Nucleus (Top‑p) sampling selects tokens from the smallest set whose cumulative probability exceeds a threshold p (e.g., 0.9). This method maintains diversity by allowing low‑probability tokens while ensuring coherence by discarding the tail of the distribution.

• Greedy decoding: picks the highest‑probability token; can be repetitive.
• Top‑k sampling: limits choices to the k most likely tokens; may miss rare but appropriate words.
• Beam search: explores multiple hypotheses; computationally expensive.

Putting It All Together: A Mini‑Project

To solidify your understanding, build a small Transformer‑based language model using PyTorch or TensorFlow. Follow these steps:

1. Implement self‑attention with query, key, and value projections.
2. Add RoPE positional encodings to enable length extrapolation.
3. Define a causal mask and train with the CLM loss − Σₜ log P(tₜ | t₁…tₜ₋₁).
4. Introduce a simple MoE layer where a gating network selects one of three experts per token.
5. After pre‑training, fine‑tune using a synthetic reward model that scores outputs based on a predefined rubric.
6. Generate text with nucleus sampling (p = 0.9) and compare it to greedy decoding.

Document your observations: note how MoE affects training speed, how RoPE influences generation length, and how nucleus sampling improves fluency.

Key Takeaways

• Self‑attention enables flexible context handling.
• CLM loss predicts the next token using only past information.
• MoE provides compute‑efficient scaling by activating a subset of experts.
• Rotary Position Embedding (RoPE) supports extrapolation to longer sequences.
• The Reward Model in RLHF aligns LLMs with human preferences.
• Chinchilla’s exponent αₙ ≈ 0.076 highlights diminishing returns from model size alone.
• Gemini 1.5’s MLA compresses KV‑cache for longer contexts.
• Nucleus (top‑p) sampling balances diversity and coherence.

By mastering these concepts, you’ll be equipped to design, train, and evaluate state‑of‑the‑art language models that are both powerful and efficient.