Deep Reinforcement Learning with DQN

Deep Reinforcement Learning with DQN: Core Concepts and Practical Insights

Introduction

Deep Q‑Network (DQN) is a landmark algorithm that combines the classic Q‑learning paradigm with deep neural networks. It enables agents to learn policies directly from high‑dimensional observations such as images or raw state vectors. This course unpacks the most important ideas behind DQN, explains why certain design choices are essential, and illustrates common pitfalls using the FrozenLake environment as a running example.

1. The Instability Problem of a Single Neural Network

When Q‑learning is implemented with a single deep network, the training process can become highly unstable. The core reason is that the target values used in the loss function shift at every weight update. This creates a moving target that the network is simultaneously trying to chase, much like a dog that keeps running after its own tail.

• Moving targets: In standard Q‑learning the target for a state‑action pair is r + γ max_a' Q(s', a'; θ). If the same parameters θ are used to compute both the prediction and the target, any change in θ instantly changes the target, causing a feedback loop.
• Non‑stationarity: The distribution of training samples changes as the policy improves, which further destabilises gradient descent.
• Consequences: Divergence, oscillating Q‑values, and failure to converge to the optimal policy.

Understanding this instability motivates the introduction of two key mechanisms: the Target Network and Experience Replay.

2. The Role of the Target Network

The Target Network is a second neural network that holds a copy of the main Q‑network’s weights, but it is updated only periodically (e.g., every 10,000 steps). Its primary purpose is to provide fixed Q‑targets for several learning updates before being refreshed.

• Stabilisation: By keeping the target network static for a short window, the loss function becomes more stationary, allowing gradient descent to make meaningful progress.
• Implementation tip: Use a soft update (τ‑weighted average) to slowly blend the main network into the target network, which reduces abrupt changes.

Without a target network, the agent would suffer from the moving‑target problem described earlier, leading to poor performance especially on environments with sparse rewards like FrozenLake.

3. Experience Replay: Breaking Temporal Correlations

Reinforcement learning data is inherently sequential; consecutive transitions are highly correlated. Training on such correlated samples violates the i.i.d. assumption of most stochastic gradient methods. Experience Replay solves this by storing transitions in a buffer and sampling them uniformly at random.

• Uniform random sampling: Each minibatch contains a mixture of experiences from many different time steps, effectively decorrelating the data.
• Sample efficiency: The same transition can be reused many times, improving data efficiency.
• Practical note: A buffer size of 10⁵–10⁶ transitions is typical; too small a buffer re‑introduces correlation, while too large a buffer may contain outdated dynamics.

In the FrozenLake example, replay ensures that the agent does not over‑fit to the most recent trajectory, which would otherwise cause divergence.

4. Failure Mode Without Experience Replay

If a DQN agent is trained on FrozenLake without Experience Replay, the most likely failure is that the network overfits to the most recent trajectory and diverges. Because each update uses highly correlated samples, the Q‑values can explode or collapse, preventing the agent from learning the true value function.

To avoid this, always enable a replay buffer when using DQN, especially in environments where episodes are short and the state space is limited.

5. State Representation: One‑Hot Encoding for FrozenLake

FrozenLake’s 4×4 grid contains 16 distinct states. A common way to feed these states into a neural network is one‑hot encoding, where each state is represented by a binary vector with a single 1 at the index corresponding to the state and 0s elsewhere.

• Number of input neurons = number of possible states = sixteen.
• Each episode begins with the agent at a particular state; the one‑hot vector tells the network exactly which tile the agent occupies.

Using one‑hot encoding simplifies the learning problem because the network does not need to infer spatial relationships; it can focus on learning the Q‑values for each state‑action pair directly.

6. Off‑Policy Learning with Function Approximation

Among the classic reinforcement‑learning algorithms, DQN uniquely combines two important properties:

• Off‑policy: The learning target uses the greedy action (max_a' Q) regardless of the behavior policy that generated the data. This allows the replay buffer to contain experiences from any policy, including older, exploratory policies.
• Function approximation: A deep neural network approximates the Q‑function, enabling the algorithm to scale to large or continuous state spaces.

Other algorithms like SARSA are on‑policy, while Monte Carlo and TD(0) typically use tabular representations unless combined with separate function approximators.

7. Double Q‑Learning: Reducing Overestimation Bias

Standard DQN suffers from an overestimation bias because the same network selects the action (max_a') and evaluates its value. Double Q‑Learning mitigates this by decoupling selection and evaluation:

• The main network selects the best action for the next state.
• The target network evaluates the value of that selected action.

This separation reduces the systematic upward bias in Q‑value estimates, leading to more accurate policies, especially in environments with noisy rewards.

8. From Monte Carlo to Temporal‑Difference (TD) Learning

Monte Carlo methods wait until the end of an episode to compute the full return before updating the value estimate. TD‑learning, on the other hand, introduces bootstrapping:

• Updates are performed after each step using the immediate reward plus the estimated value of the next state.
• This enables learning from incomplete episodes and reduces variance compared to Monte Carlo returns.

Bootstrapping is the cornerstone of DQN, allowing the algorithm to learn online and efficiently in environments where episodes can be long or infinite.

9. Putting It All Together: A Mini‑Guide to Building a DQN Agent

Below is a concise checklist that captures the essential components discussed:

• Network architecture: Input layer sized to the state representation (e.g., 16 for FrozenLake), a few hidden layers (e.g., 2×64 ReLU units), and an output layer with one neuron per action.
• Target network: Clone of the main network, updated every C steps or via soft updates.
• Replay buffer: Store tuples (s, a, r, s', done) and sample minibatches uniformly.
• Loss function: Mean‑squared error between predicted Q(s,a) and target r + γ max_a' Q_target(s', a').
• Optimization: Adam or RMSprop with a learning rate around 1e‑4.
• Exploration strategy: ε‑greedy policy with decay (e.g., start ε=1.0, decay to 0.01).
• Double DQN extension: Use the main network for action selection and the target network for evaluation.

Following this structure ensures that the agent avoids the instability pitfalls and leverages the strengths of DQN.

10. Frequently Asked Questions (FAQ)

• Q: Can I use a single network without a target network if I train very slowly?
  A: In theory, very small learning rates might reduce instability, but in practice the moving‑target problem persists and training becomes impractically slow.
• Q: Is experience replay necessary for small state spaces?
  A: Even in small discrete environments, replay improves sample efficiency and prevents over‑fitting to recent trajectories.
• Q: How does Double Q‑Learning differ from Double DQN?
  A: Double Q‑Learning is the original tabular algorithm; Double DQN adapts the same idea to deep networks by using separate target and online networks.
• Q: What is the impact of the replay buffer size?
  A: A larger buffer provides more diverse samples but may contain outdated dynamics; a typical size balances diversity and relevance.

Conclusion

Deep Q‑Networks revolutionised reinforcement learning by enabling agents to learn directly from raw observations. The key to their success lies in two stabilising mechanisms: the Target Network that supplies fixed Q‑targets, and Experience Replay that decorrelates training data. Understanding why a single network is unstable, how Double Q‑Learning reduces overestimation, and the role of bootstrapping versus Monte Carlo methods equips you to design robust agents for a wide range of tasks, from simple grid worlds like FrozenLake to complex video‑game environments.