Foundations of Generative Modeling

Foundations of Generative Modeling

Generative modeling lies at the heart of modern artificial intelligence research. It enables machines to create new data—images, text, audio—that resemble the patterns observed in training sets. This course unpacks the core concepts behind popular generative frameworks such as autoencoders, variational autoencoders (VAEs), and generative adversarial networks (GANs). By the end of the lesson, you will understand why certain models struggle as generators, how latent variable priors are chosen, and the mathematical foundations that make variational inference possible.

Why a Standard Autoencoder Is Not a Generative Model

A classic autoencoder (AE) consists of an encoder that compresses input x into a latent code z, and a decoder that reconstructs x from z. The training objective minimizes a reconstruction loss, typically the mean‑squared error or binary cross‑entropy. However, an AE fails to serve as a generative model because:

• It lacks a defined prior distribution for the latent space. Without a probabilistic model p(z), we cannot sample new latent vectors and decode them into realistic data.
• The mapping from x to z is deterministic, meaning the encoder does not capture the uncertainty needed for sampling.

Consequently, while an AE can reconstruct training examples well, it cannot generate novel, diverse samples—a key requirement for generative modeling.

Choosing a Prior in Latent Variable Models

In the latent variable framework, we introduce a hidden variable z and define a joint distribution p(x, z) = p(x|z) p(z). For tractability, the prior p(z) is usually selected as a simple distribution. The most common choice is a multivariate Gaussian with zero mean and identity covariance (i.e., z ~ N(0, I)). This Gaussian prior offers several advantages:

• It is analytically easy to sample from.
• The density is known, which simplifies the computation of the Kullback‑Leibler (KL) divergence term in variational objectives.
• Its isotropic nature encourages the latent space to be organized without bias toward any particular direction.

Alternative priors (Bernoulli, uniform, Laplace) are possible but typically introduce additional computational complexity without clear benefits for most deep generative models.

Why the Marginal Likelihood Is Intractable

The marginal likelihood p(x) = ∫ p(x|z) p(z) dz measures how well a model explains observed data. For most latent variable models, evaluating this integral is intractable because:

• The latent space is high‑dimensional, requiring integration over many variables.
• The likelihood p(x|z) is often parameterized by a deep neural network, making the integrand a highly non‑linear function.

Exact computation would demand enumerating every possible latent configuration, which is computationally impossible. This challenge motivates the use of approximate inference techniques such as variational inference and Monte‑Carlo methods.

Kullback‑Leibler Divergence in Variational Inference

Variational inference replaces the true posterior p(z|x) with a tractable approximation q(z|x). The quality of this approximation is measured by the Kullback‑Leibler (KL) divergence. A crucial property that makes KL suitable for variational objectives is its non‑negativity:

• KL(q‖p) ≥ 0, with equality only when q = p.

This property guarantees that the evidence lower bound (ELBO) is indeed a lower bound on the log‑likelihood log p(x). Because the KL term cannot be negative, any improvement in the ELBO directly translates into a tighter bound on the true likelihood.

Understanding the ELBO as a Lower Bound

When training a VAE, we maximize the ELBO:

ELBO = E_{q(z|x)}[log p(x|z)] - KL(q(z|x) ‖ p(z))

The second term, the KL divergence, is always non‑negative. By subtracting this term from the expected log‑likelihood, we obtain a quantity that is guaranteed to be less than or equal to the true log‑likelihood:

log p(x) = ELBO + KL(q(z|x) ‖ p(z|x))

Since KL(q(z|x) ‖ p(z|x)) ≥ 0, the ELBO is a valid lower bound. Maximizing the ELBO therefore pushes the approximate posterior q(z|x) closer to the true posterior and improves the generative performance.

Concentration of Measure in High‑Dimensional Spaces

The "concentration of measure" phenomenon describes how probability mass behaves in high‑dimensional spaces. Contrary to intuition, most of the mass does not concentrate near the origin; instead, it resides near the "outer shells" of the space. This has two practical implications for generative modeling:

• Random samples drawn from a naïve prior may fall in low‑density regions, making training unstable.
• Designing priors and latent representations that respect this geometry (e.g., using spherical Gaussian priors) helps models capture the true data distribution more efficiently.

Understanding this phenomenon guides the choice of latent dimensionality and regularization strategies.

Explicit vs. Implicit Density‑Estimation Models

Generative models can be categorized as explicit or implicit based on how they handle probability densities:

• Explicit models define a tractable density function p(x). Examples include VAEs and normalizing flows, where the likelihood can be evaluated directly.
• Implicit models provide only a sampling mechanism without an explicit density. Generative adversarial networks (GANs) fall into this category; they learn to generate realistic samples but do not compute exact likelihoods.

This distinction matters when choosing evaluation metrics. Explicit models can be assessed with log‑likelihood scores, while implicit models rely on proxy metrics such as the Inception Score or Fréchet Inception Distance.

Adversarial Training and the GAN Equilibrium

GANs formulate a two‑player minimax game between a generator G and a discriminator D. The objective is:

min_G max_D  E_{x~p_data}[log D(x)] + E_{z~p_z}[log(1 - D(G(z)))]

The equilibrium point of this game occurs when the generator reproduces the true data distribution, i.e., the distribution of G(z) matches p_data. At this point, the discriminator cannot distinguish real from fake samples and outputs a probability of 0.5 for any input. This equilibrium reflects the ideal scenario where the generator has learned to model the data perfectly.

Key Takeaways

• Standard autoencoders lack a prior over latent variables, preventing them from generating new data.
• Choosing a simple Gaussian prior simplifies sampling and KL‑divergence computation.
• The marginal likelihood is intractable due to high‑dimensional integration over neural network likelihoods.
• KL divergence’s non‑negativity ensures the ELBO is a valid lower bound on log‑likelihood.
• Concentration of measure informs the design of latent spaces for high‑dimensional data.
• Explicit models provide tractable densities; implicit models focus on sample quality.
• GAN training reaches equilibrium when the generator’s distribution equals the real data distribution.

Further Reading and Resources

To deepen your understanding, explore the following resources:

• Auto-Encoding Variational Bayes by Kingma & Welling – foundational VAE paper.
• Generative Adversarial Nets by Goodfellow et al. – original GAN introduction.
• Distill’s interactive GAN tutorial – visual intuition for adversarial training.
• Understanding the difficulty of training deep generative models – discusses concentration of measure.

By mastering these concepts, you will be equipped to design, train, and evaluate modern generative models across a variety of AI applications.