Knockoffs: A framework of Control Variable Selection
In many fields of science, we observe a response variable together with a large number of potential explanatory variables, but only a small subset of them are truely related (i.e. sparsity assumptions). We would like to be able to discover which predictors are important for the response. At the same time, we need to assure that most of the discoveries are indeed true and replicable.
Knockoffs are a framework for variable selection and error-controlled discovery. The goal is not only to find variables that are important, but to control the expected fraction of false discoveries among all discoveries where selected variables that are actually null. The statistical target is the FDR of selected variable, which is defined as below:
Let $X = [X_1,\ldots,X_p]$ be the covariates and $Y$ be the response. A variable $X_j$ is null if it carries no additional information about $Y$ once the other variables are known:
\[H_{0j}: Y \perp X_j \mid X_{-j}.\]If $\hat{S}$ is the selected set and $S_0$ is the set of null variables, the false discovery rate is
\[\operatorname{FDR} = E\left[\frac{|\hat{S} \cap S_0|}{|\hat{S}| \vee 1}\right].\]The knockoff idea is that under exchangeability, null predictors behave symmetrically and independently of the non-null signal. Techinically, by creating a synthetic copy $\widetilde X_j$ for every original variable $X_j$, the knockoff variable should look like $X_j$ in its dependence with the other covariates, but it should not have its own independent relationship with $Y$. In this sense, knockoffs act as negative controls: if $X_j$ is truly important, it should beat its knockoff; if $X_j$ is null, the original and knockoff should be statistically indistinguishable.1
Fixed-X and Model-X Knockoffs
1. The Core Idea: A Feature Competes Against Its Knockoff
For each original feature $X_j$, construct a knockoff copy $\widetilde X_j$.
The knockoff must satisfy two principles:
- It should look like $X_j$ in its dependence structure with the other covariates.
- It must be constructed without looking at $Y$.
Thus $\widetilde X_j$ is designed to be a fake competitor: it resembles $X_j$ statistically, but it has no independent relationship with the response beyond what is already contained in $X$.2
After constructing knockoffs, fit a model on the augmented design
\[[X,\widetilde X].\]For each original and knockoff variable, compute importance scores
\[Z_j = \operatorname{Importance}(X_j)\]and
\[\widetilde Z_j = \operatorname{Importance}(\widetilde X_j).\]A simple knockoff statistic is
\[W_j = Z_j - \widetilde Z_j.\]More generally, one may use any antisymmetric contrast
\[W_j = h(Z_j,\widetilde Z_j),\]where
\[h(Z_j,\widetilde Z_j) = -h(\widetilde Z_j,Z_j).\]Then:
- $W_j > 0$ means the original feature beats its knockoff;
- $W_j < 0$ means the knockoff beats the original;
- for null variables, positive and negative signs should be symmetric.
This is the central inferential trick:
The knockoff variables create an observable control group for false discoveries.
2. Sign-Flip Symmetry and the Knockoff Threshold
The knockoff statistic must satisfy a flip-sign property. If we swap $X_j$ and $\widetilde X_j$, the statistic changes sign:
\[W_j([X,\widetilde X]_{\operatorname{swap}(j)},Y) = -W_j([X,\widetilde X],Y).\]| For null variables, the original and knockoff are exchangeable relative to $Y$. Therefore, conditional on the magnitudes $ | W_j | $, the signs of null $W_j$ values behave like fair coin flips: |
So large negative $W_j$ values estimate the number of false positive large positive $W_j$ values.
The knockoff+ threshold is
\[T_+ = \min \left\{ t>0: \frac{ 1+\#\{j: W_j \le -t\} }{ \#\{j: W_j \ge t\} \vee 1 } \le q \right\}.\]Then select
\[\hat S = \{j: W_j \ge T_+\}.\]The ratio
\[\frac{ 1+\#\{j: W_j \le -t\} }{ \#\{j: W_j \ge t\} \vee 1 }\]is a data-adaptive estimate of the false discovery proportion. The numerator counts knockoff wins; the denominator counts original wins. The Stanford methodology page describes this threshold as the step that converts knockoff competition into FDR-controlled variable selection.2
3. Fixed-X Knockoffs: Geometry for Linear Models
The original knockoff filter was developed for the fixed-design Gaussian linear model:1
\[Y = X\beta + \varepsilon, \qquad \varepsilon \sim N(0,\sigma^2 I_n),\]where $X \in \mathbb R^{n \times p}$ is treated as fixed.
The null hypothesis is
\[H_{0j}: \beta_j=0.\]The knockoff matrix $\widetilde X$ is constructed so that the augmented Gram matrix is
\[[X,\widetilde X]^T[X,\widetilde X] = \begin{bmatrix} \Sigma & \Sigma-\operatorname{diag}(s) \\ \Sigma-\operatorname{diag}(s) & \Sigma \end{bmatrix},\]where
\[\Sigma = X^T X.\]Equivalently,
\[\widetilde X^T\widetilde X = X^T X\]and
\[X^T\widetilde X = X^T X-\operatorname{diag}(s).\]The vector $s$ controls how different each feature is from its knockoff. The feasibility condition is
\[2\Sigma-\operatorname{diag}(s) \succeq 0.\]A standard construction is
\[\widetilde X = X[I_p-\Sigma^{-1}\operatorname{diag}(s)] + UC,\]where $U$ has orthonormal columns orthogonal to the column space of $X$, and
\[C^TC = 2\operatorname{diag}(s) - \operatorname{diag}(s)\Sigma^{-1}\operatorname{diag}(s).\]Fixed-X knockoffs are geometric objects. They are artificial columns whose inner products with the original features mimic the original design geometry.
The strength is that no distributional model for $X$ is needed. The limitation is that the response model is restrictive: the Stanford modeling page states that Fixed-X knockoffs assume a homoscedastic linear model and low-dimensional conditions, with $n \ge p$.3
There is also a sufficiency requirement. The statistic $W_j$ must depend on the data only through the augmented Gram matrix and feature-response inner products:
\[W_j = W_j \left( [X,\widetilde X]^T[X,\widetilde X], [X,\widetilde X]^TY \right).\]This is why some adaptive ML procedures are not automatically valid in the Fixed-X setting. The statistic must respect the geometry on which the proof is built.
4. Model-X Knockoffs: Exchangeability for Random Covariates
Model-X knockoffs change the source of validity. Instead of assuming a linear model for $Y \mid X$, they assume knowledge of the covariate distribution $P_X$.4
The data are viewed as
\[(X_i,Y_i) \overset{i.i.d.}{\sim} P_{XY}, \qquad P_{XY}=P_{Y \mid X}P_X.\]The response model $P_{Y \mid X}$ may be arbitrary and unknown. This is the key advantage.
A valid Model-X knockoff $\widetilde X$ must satisfy pairwise exchangeability:
\[(X,\widetilde X)_{\operatorname{swap}(S)} \overset{d}{=} (X,\widetilde X)\]for every subset $S \subseteq {1,\ldots,p}$.
It also must satisfy
\[\widetilde X \perp Y \mid X.\]The Stanford modeling page summarizes the paradigm clearly: Model-X treats predictors as random, requires the covariate distribution $F_X$ to be known, allows $F_{Y \mid X}$ to be arbitrary, and permits powerful machine-learning tools for the importance statistics as long as the fairness condition is respected.3
The conceptual shift is:
\[\text{Fixed-X: assume structure on } Y \mid X.\] \[\text{Model-X: assume structure on } X.\]This is why Model-X knockoffs are more natural for modern ML variable selection. We can use random forests, boosting, neural networks, lasso paths, or other black-box learners to compute $Z_j$ and $\widetilde Z_j$, because the inferential validity comes from exchangeability of $(X,\widetilde X)$, not from the predictive model being correctly specified.
5. Gaussian Model-X Knockoffs
The most common explicit Model-X construction is the Gaussian second-order knockoff.
Assume
\[X \sim N(\mu,\Sigma).\]Construct
\[\begin{bmatrix} X \\ \widetilde X \end{bmatrix} \sim N \left( \begin{bmatrix} \mu \\ \mu \end{bmatrix}, \begin{bmatrix} \Sigma & \Sigma-\operatorname{diag}(s) \\ \Sigma-\operatorname{diag}(s) & \Sigma \end{bmatrix} \right),\]where
\[2\Sigma-\operatorname{diag}(s) \succeq 0.\]Then
\[\widetilde X \mid X \sim N(m_K(X),V_K),\]where
\[m_K(X) = \mu + [I_p-\operatorname{diag}(s)\Sigma^{-1}](X-\mu)\]and
\[V_K = 2\operatorname{diag}(s) - \operatorname{diag}(s)\Sigma^{-1}\operatorname{diag}(s).\]This construction works because the joint covariance matrix is invariant to swapping original and knockoff coordinates. In the Gaussian setting, second-order matching is enough to imply the required exchangeability.
The practical difficulty is now visible:
Model-X validity depends on how well we can model and sample from $P_X$.
If the knockoff generator fails to reproduce the dependence structure of $X$, the sign-flip symmetry for nulls may be distorted, and FDR control can degrade.
6. Fixed-X versus Model-X
| Aspect | Fixed-X Knockoffs | Model-X Knockoffs |
|---|---|---|
| View of covariates | Fixed design matrix | Random covariates |
| Source of validity | Linear response model and Gram geometry | Covariate distribution and exchangeability |
| Null hypothesis | $\beta_j=0$ | $Y \perp X_j \mid X_{-j}$ |
| Response model | Homoscedastic linear model | Arbitrary and unknown |
| Covariate model | Not needed | Must know or estimate $P_X$ |
| Dimension | Typically requires $n \ge p$ | Can be high-dimensional in principle |
| Statistic requirement | Sufficiency plus antisymmetry | Fairness / flip-sign condition |
| ML compatibility | Limited | Strong |
The key tradeoff is:
\[\text{Fixed-X avoids modeling } X \text{ but restricts } Y \mid X.\] \[\text{Model-X frees } Y \mid X \text{ but requires modeling } X.\]This distinction is exactly why Model-X became important for modern ML. It allows flexible prediction models while maintaining a rigorous inferential layer.
7. Why Knockoffs Matter in the Age of AI
Modern AI systems are good at ranking features. They are much weaker at saying which selected features are false discoveries.
Deep Knockoffs provide an inferential wrapper:
\[\text{black-box model} \quad \Rightarrow \quad (Z_j,\widetilde Z_j) \quad \Rightarrow \quad W_j \quad \Rightarrow \quad \hat S \text{ with FDR control}.\]The predictive model decides what looks important. The knockoff construction calibrates how much more important the real feature must be than a synthetic null competitor.
8. Main Takeaway
Knockoffs turn variable selection into a calibrated competition between real variables and fake negative controls.
Fixed-X knockoffs create fake competitors geometrically:
\[[X,\widetilde X]^T[X,\widetilde X] = \begin{bmatrix} \Sigma & \Sigma-\operatorname{diag}(s) \\ \Sigma-\operatorname{diag}(s) & \Sigma \end{bmatrix}.\]Model-X knockoffs create fake competitors probabilistically:
\[(X,\widetilde X)_{\operatorname{swap}(S)} \overset{d}{=} (X,\widetilde X).\]The common principle is:
\[\text{null features should not systematically beat their knockoffs}.\]Therefore, knockoff wins estimate false discoveries, and original wins beyond an adaptive threshold define the selected set.
In one sentence:
Knockoffs convert black-box variable importance into finite-sample, FDR-controlled discovery by building feature-level negative controls.
References
-
Barber, R. F., & Candès, E. J. (2015). “Controlling the False Discovery Rate via Knockoffs.” The Annals of Statistics, 43(5), 2055–2085. ↩ ↩2
-
Candès Lab. “The Methodology of Knockoffs — Outline.” Stanford University Knockoffs website. https://web.stanford.edu/group/candes/knockoffs/outline.html ↩ ↩2
-
Candès Lab. “The Methodology of Knockoffs — Modeling.” Stanford University Knockoffs website. https://web.stanford.edu/group/candes/knockoffs/modeling.html ↩ ↩2
-
Candès, E., Fan, Y., Janson, L., & Lv, J. (2018). “Panning for Gold: ‘Model-X’ Knockoffs for High Dimensional Controlled Variable Selection.” Journal of the Royal Statistical Society: Series B, 80(3), 551–577. ↩