Chuanhui Liu

Email: chuanhui.liu1994@gmail.com

   6 minute read

1. The Bayesian computing problem

In Bayesian statistics, we start with a prior distribution over unknown parameters $\theta$, combine it with the likelihood from observed data $x$, and obtain the posterior distribution:

\[p(\theta \mid x) = \frac{p(x \mid \theta)p(\theta)}{p(x)}.\]

The hard part is the denominator

\[p(x) = \int p(x \mid \theta)p(\theta)\,d\theta,\]

also called the marginal likelihood or evidence. For advance probalistic models, especially hierarchical models, latent-variable models, and deep generative models, this integral is usually impossible to compute exactly. Bayesian computing is largely about approximating this posterior.1

Two major strategies are:

  1. MCMC: approximate the posterior by drawing samples from it. 2
  2. Variational inference (VI): approximate the posterior by solving an optimization problem.3

2. What is variational inference?

Variational inference turns posterior inference into optimization. Instead of trying to sample from the exact posterior $p(\theta \mid x)$, we choose a simpler family of distributions

\[\mathcal{Q} = \{q_\lambda(\theta): \lambda \in \Lambda\},\]

and look for the member of that family closest to the true posterior:

\[q_{\lambda^*}(\theta) \approx p(\theta \mid x).\]

The optimization problem is usually written as

\[\lambda^* = \arg\min_\lambda \mathrm{KL}\big(q_\lambda(\theta) \Vert p(\theta \mid x)\big).\]

Because the true posterior contains the intractable evidence $p(x)$, we usually optimize an equivalent objective called the evidence lower bound, or ELBO:

\[\mathcal{L}(\lambda) = \mathbb{E}_{q_\lambda(\theta)}[\log p(x,\theta)] - \mathbb{E}_{q_\lambda(\theta)}[\log q_\lambda(\theta)].\]

Equivalently,

\[\log p(x) = \mathcal{L}(\lambda) + \mathrm{KL}\big(q_\lambda(\theta) \Vert p(\theta \mid x)\big).\]

Since the KL divergence is nonnegative, $\mathcal{L}(\lambda)$ is a lower bound on $\log p(x)$. Maximizing the ELBO makes $q_\lambda$ closer to the posterior.4

A useful intuition is:

VI replaces posterior sampling with posterior approximation by optimization.

This is why VI naturally connects to a lot of Machine Learning research topics in ELBOs, Gibbs variational principles, Fenchel duality, and EM-style algorithms: many modern inference objectives can be viewed as optimizing over distributions rather than only over point parameters.

3. VI versus MCMC

MCMC and VI are both tools for approximate Bayesian inference, but they have different computational philosophies.

Dimension MCMC Variational Inference
Core idea Draw samples from the posterior Fit a simpler distribution to the posterior
Computational form Simulation / sampling Optimization
Output Posterior samples Approximate distribution $q_\lambda$
Accuracy Asymptotically exact under convergence conditions Biased by the choice of variational family and objective
Speed Often slower, especially for large data or complex latent structures Often faster and easier to scale with stochastic optimization
Diagnostics Mixing, convergence, effective sample size, $\hat{R}$ ELBO monitoring, held-out likelihood, posterior predictive checks, sensitivity to family choice
Common weakness Can be computationally expensive and hard to diagnose in high dimensions May underestimate uncertainty, especially with mean-field reverse-KL VI

MCMC methods construct a Markov chain whose stationary distribution is the target posterior. If the chain mixes well and is run long enough, posterior averages can be estimated from samples. Modern systems such as Stan use Hamiltonian Monte Carlo and the No-U-Turn Sampler to make this sampling more efficient in continuous parameter spaces.2

VI takes a different route. It chooses an approximating family $\mathcal{Q}$, such as a mean-field family

\[q(\theta) = \prod_j q_j(\theta_j),\]

and searches for the best approximation inside that family. This makes VI highly scalable, but also introduces approximation bias. In particular, minimizing $\mathrm{KL}(q \Vert p)$ often produces mode-seeking approximations, which can miss posterior mass and understate uncertainty.35

A practical summary is:

MCMC asks: “Can I generate representative samples from the true posterior? VI asks: “Can I find a tractable distribution that is close enough to the posterior for my inferential or predictive goal?

4. Why VI became important for deep learning

VI became especially important in deep learning because modern deep probabilistic models often contain latent variables and intractable posteriors. For example, in a latent-variable generative model, we may assume $z \sim p(z), x \sim p_\theta(x \mid z)$, where $z$ is a latent representation and $p_\theta(x \mid z)$ is parameterized by a neural network. The posterior$p_\theta(z \mid x)$ is usually intractable. MCMC could be used in principle, but running a separate sampling procedure for every data point is usually too expensive for large-scale neural network training.

The key breakthrough was amortized variational inference. Instead of optimizing a separate variational distribution for each observation, we train an inference network $q_\phi(z \mid x)$, which maps data $x$ directly to an approximate posterior over $z$. In variational autoencoders (VAEs), this creates an encoder-decoder structure:67

  • the encoder $q_\phi(z \mid x)$ approximates the posterior;
  • the decoder $p_\theta(x \mid z)$ defines the generative model;
  • the model is trained by maximizing the ELBO.

The VAE objective is commonly written as

\[\mathcal{L}(\theta,\phi; x) = \mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - KL[q_\phi(z \mid x) \Vert p(z)].\]

This objective has two parts:

  1. Reconstruction term: make generated samples explain the data.
  2. Regularization term: keep the approximate posterior close to the prior.

The reparameterization trick made this objective practical for neural networks. For example, instead of sampling $z \sim q_\phi(z \mid x)$, we write

\[z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon, \qquad \epsilon \sim \mathcal{N}(0,I),\]

which allows gradients to flow through stochastic latent variables during backpropagation.67

This is why VI became a central paradigm in deep learning: it translated Bayesian latent-variable inference into a differentiable, minibatch-compatible, GPU-friendly optimization problem.

5. Why VI is more than an approximation trick

At an entry level, VI is often introduced as a faster alternative to MCMC. But conceptually, it is deeper than that.

VI provides a general language for turning hard probabilistic problems into optimization problems over distributions. This viewpoint appears in many areas:

  • Bayesian posterior approximation, through ELBO maximization.
  • EM algorithms, where latent-variable posteriors are approximated or optimized in an E-step.
  • Deep generative models, where inference networks amortize posterior approximation.
  • Information-theoretic learning, where KL, entropy, and mutual information terms regularize objectives.
  • Optimal transport and Sinkhorn methods, where entropic regularization creates Gibbs-like couplings and tractable variational objectives.

This is also why VI connects naturally to the Gibbs variational principle and Fenchel-style representations. A broad template is:

\[\sup_{q \in \mathcal{P}} \{ \mathbb{E}_q[f] - R(q) \}\]

, where $q$ is a distribution being optimized, $f$ is a score or energy function, and $R(q)$ is a regularizer such as KL divergence or entropy. The ELBO is one important instance of this general idea.

6. Main limitations

VI is powerful, but it is not automatically reliable. Important limitations include:

  1. Approximation bias: the result depends on the variational family $\mathcal{Q}$.
  2. Underestimated uncertainty: mean-field VI with reverse KL often produces overly concentrated approximations.
  3. Optimization difficulty: the ELBO can be nonconvex for deep models.
  4. Amortization gap: in VAEs, the inference network may not match the best per-data-point variational approximation.
  5. Posterior collapse: in powerful decoder models, the latent variable may be ignored, causing $q_\phi(z \mid x)$ to collapse toward the prior.

So VI should be seen as a scalable inference framework, not a magic substitute for exact Bayesian computation.

7. One-paragraph takeaway

Variational inference is a framework for approximate Bayesian computation that replaces posterior sampling with optimization over a tractable family of distributions. Compared with MCMC, VI is usually faster and more scalable, but it introduces approximation bias and can understate uncertainty. Its importance grew dramatically in deep learning because the ELBO, stochastic gradients, reparameterization trick, and amortized inference made latent-variable generative models trainable with neural networks. In modern machine learning, VI is not only a Bayesian computing method; it is also a general design principle for learning probabilistic models, optimizing over distributions, and building scalable uncertainty-aware AI systems.

References

  1. Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999). An Introduction to Variational Methods for Graphical Models. Machine Learning, 37, 183–233. ↩

  2. Stan Development Team. MCMC Sampling. Stan Reference Manual. ↩ ↩2

  3. Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017). Variational Inference: A Review for Statisticians. Journal of the American Statistical Association. ↩ ↩2

  4. Wainwright, M. J., & Jordan, M. I. (2008). Graphical Models, Exponential Families, and Variational Inference. Foundations and Trends in Machine Learning. ↩

  5. Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. (2013). Stochastic Variational Inference. Journal of Machine Learning Research, 14, 1303–1347. ↩

  6. Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes. ICLR. ↩ ↩2

  7. Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. ICML. ↩ ↩2

Tags: ,

Categories:

Updated: