Conformal Prediction 如何在不假設特定分佈下提供預測覆蓋保證

Most machine learning models give you a point prediction and stop there. A house price, a risk score, a class label. The number looks precise, but precision is not the same as reliability. If a model says a patient will recover in five days, a decision maker does not only need the five. They need to know how wrong that number could plausibly be, and how often the stated range can be trusted.

Conformal prediction answers that question with unusual honesty. It turns almost any predictor into a prediction set or interval with a finite-sample coverage guarantee, and it does so without assuming that residuals are Gaussian, that the model is correctly specified, or that you know the true data generating process. The price is modest: you need a held-out calibration set, and you need exchangeability. The reward is a guarantee you can state plainly.

What "valid" means here

Suppose you choose a target miscoverage level, say α=0.1 . A conformal prediction set C(X) aims to satisfy:

P(Yn+1∈C(Xn+1))≥1−α

That is marginal coverage. Over many fresh test points drawn in the same way as the calibration data, at least 90 percent of the true outcomes should fall inside their sets. It is not a promise for one specific input, and it is not a promise for every subgroup. That distinction matters, and we will return to it.

The guarantee is finite-sample, not asymptotic. It does not only hold when the calibration set grows infinitely large. It holds for the calibration set you actually have, provided the exchangeability assumption holds. Exchangeability is weaker than independence plus identical distribution in the formal sense, but for practical purposes you can read it as: the calibration points and the future test point are drawn from the same process, with no ordering, drift, or selection effect that makes the test point systematically different.

That is the whole trick, and also the whole vulnerability. Conformal prediction does not fix distribution shift. It converts a calibration sample into a calibrated statement about a future sample from the same distribution.

Split conformal prediction in four steps

Split conformal prediction is the simplest and most widely used variant. You split your data into a training set and a calibration set, fit the model once, and never refit it during calibration.

Step 1: Fit a predictor f^ on the training set. It can be a linear model, a gradient boosting model, a neural network, or anything else. Conformal prediction is model agnostic.

Step 2: Choose a nonconformity score. This score measures how unusual an observed outcome is, given the prediction. For regression, the classic choice is the absolute residual:

si=∣Yi−f^(Xi)∣

A large score means the model was badly wrong on that calibration point. A small score means it was close.

Step 3: Compute the scores on the calibration set, then take a suitable quantile. If you have n calibration points and want coverage 1−α , you do not take the plain 1−α quantile. You take the adjusted level:

q^=Quantile(s1,…,sn;⌈(n+1)(1−α)⌉n)

The ceiling and the n+1 correction are not pedantry. They are exactly what turns an approximate empirical quantile into a finite-sample guarantee. Intuitively, the new test point is equally likely to rank anywhere among the n+1 points (the calibration points plus itself) under exchangeability, so you must leave room for that extra rank.

Step 4: For a new input Xn+1 , return every outcome whose score would be at most q^ . With the absolute residual score, that gives the familiar symmetric interval:

C(Xn+1)=[f^(Xn+1)−q^,  f^(Xn+1)+q^]

That is it. No distributional assumption about the residuals was used. No second model was fitted. The calibration scores did all the work.

Why the guarantee works, in one paragraph

Under exchangeability, the scores of the calibration points and the future point are exchangeable too. Rank the n+1 scores. The future score is equally likely to occupy any rank. The adjusted quantile is chosen so that the future score falls at or below it with probability at least 1−α . Falling at or below it is precisely the event that the true outcome lies inside the prediction set. This rank argument is why the method is so general: it never asks what distribution produced the scores, only that their ordering is not rigged by the split.

A worked example in code

The code below implements split conformal prediction for regression with NumPy. It is deliberately minimal so the mechanics are visible: fit, score, take the adjusted quantile, build intervals.

import numpy as np
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split

# X, y are your features and targets. Split into train, calibration, and test.
X_train, X_rest, y_train, y_rest = train_test_split(X, y, test_size=0.4, random_state=0)
X_cal, X_test, y_cal, y_test = train_test_split(X_rest, y_rest, test_size=0.5, random_state=1)

model = HistGradientBoostingRegressor(random_state=0)
model.fit(X_train, y_train)

# Step 2 and 3: absolute residual scores, then the adjusted quantile.
cal_pred = model.predict(X_cal)
scores = np.abs(y_cal - cal_pred)

alpha = 0.1
n = len(scores)
level = np.ceil((n + 1) * (1 - alpha)) / n
q_hat = np.quantile(scores, level, method="higher")

# Step 4: intervals for new points.
test_pred = model.predict(X_test)
lower = test_pred - q_hat
upper = test_pred + q_hat

coverage = np.mean((y_test >= lower) & (y_test <= upper))
width = np.mean(upper - lower)
print(f"Empirical coverage: {coverage:.3f} (target at least {1 - alpha})")
print(f"Mean interval width: {width:.3f}")

Two details deserve attention. First, the calibration set must not have been used to fit the model, or the scores will be optimistically small and the intervals too narrow. Second, the quantile method matters at small n . The "higher" method used above is a safe choice for the finite-sample argument; other interpolation rules can break the exact guarantee even if they look similar.

If your calibration set is small, the intervals will still be valid on average, but their width will be noisy. Validity is about coverage over repeated draws of calibration and test data, not about one lucky or unlucky calibration sample giving a stable width. In practice, a few hundred calibration points are a reasonable starting point for a 90 percent target, and a thousand or more gives much more stable widths.

Constant width is a limitation, not a feature

The absolute residual version gives every test point an interval of the same width: 2q^ . That is honest on average, but it ignores heteroscedasticity. A region where the model is accurate gets the same width as a region where it is erratic. For many applications, that is wasteful and hard to explain.

A natural fix is a normalised score. Suppose you also have a difficulty estimate σ^(X) , for example from a model trained to predict absolute residuals, or from the spread of an ensemble. Then define:

si=∣Yi−f^(Xi)∣σ^(Xi)

The prediction interval becomes:

C(Xn+1)=[f^(Xn+1)−q^ σ^(Xn+1),  f^(Xn+1)+q^ σ^(Xn+1)]

The coverage guarantee is unchanged, because the rank argument does not care which score you chose. What changes is adaptivity: intervals widen where the model is expected to struggle and narrow where it is expected to be sharp. The catch is that σ^ must be fitted on training data, not on the calibration set, or you again leak calibration information into the scores.

Conformalised quantile regression goes one step further. Instead of starting from a point predictor, you fit lower and upper quantile models, say at levels α/2 and 1−α/2 , then conformalise their residuals. The score becomes the distance outside the initial quantile interval, and the final interval corrects that interval just enough to restore coverage. When the quantile models capture heteroscedasticity well, this often gives sharper, more adaptive intervals than a normalised residual score.

The lesson generalises: conformal prediction is a wrapper, not a model. A better base model or a better score usually gives narrower sets at the same coverage. The wrapper guarantees coverage; the score decides efficiency.

Classification: prediction sets, not just labels

In classification, the output is a set of plausible labels rather than an interval. A simple score is one minus the predicted probability of the true class:

si=1−p^Yi(Xi)

At test time, you include every class whose score is at most q^ , equivalently every class with predicted probability at least 1−q^ . Easy inputs get singleton sets. Hard or ambiguous inputs get larger sets. That behaviour is exactly what you want from an abstention or triage system: the size of the set is itself a signal.

More refined scores, such as adaptive prediction sets, accumulate sorted class probabilities until the true class is reached, with a randomisation step to avoid over-conservatism from discrete scores. The details matter for efficiency, but the principle is the same: define a score, calibrate its quantile, include everything that scores well enough.

A common mistake is to threshold softmax probabilities directly, for example by keeping classes above 0.5, and to call the result calibrated. Softmax probabilities from a modern neural network are often poorly calibrated, and an arbitrary threshold has no coverage guarantee. Conformal prediction replaces the arbitrary threshold with a quantile chosen from held-out data.

What conformal prediction does not promise

Marginal coverage is weaker than most people first assume. A method with 90 percent marginal coverage can still have 70 percent coverage for one subgroup and 98 percent for another, as long as the average works out. If you need coverage within each group, you need conditional or group-conditional methods. Mondrian conformal prediction is the classic fix: split the calibration set by group and calibrate a separate quantile per group. You pay for that guarantee with smaller calibration samples per group and wider intervals for small groups, which is an honest reflection of how much you actually know about them.

Full conditional coverage, meaning 90 percent coverage for every single x , is impossible in general without further assumptions. This is a known impossibility result, not a gap in implementation. Any method claiming exact conditional coverage for arbitrary distributions should be treated with scepticism.

Distribution shift is the other hard boundary. If the deployment distribution differs from the calibration distribution, the guarantee does not transfer. Weighted conformal prediction can handle known or estimable shifts, such as covariate shift with a known likelihood ratio, but it cannot conjure validity from a calibration set that no longer resembles deployment. In practice, this means you should monitor coverage on fresh labelled data after deployment, refresh the calibration set when the distribution moves, and treat a conformal interval built on stale data as a historical artefact, not a live guarantee.

Finally, conformal prediction does not make a bad model good. If the base predictor is poor, the intervals will be wide, because the calibration scores will be large. Wide intervals are the method telling the truth about the model. Shrinking them by hand, by tuning α after looking at test results, or by reusing calibration data for training, destroys the guarantee and replaces it with wishful thinking.

Conformal prediction versus Bayesian and ensemble uncertainty

Bayesian methods give posterior predictive intervals, but only under the prior and likelihood you chose. If those are misspecified, the intervals can be systematically wrong, and there is no finite-sample correction waiting to rescue them. Ensembles and dropout-based uncertainty give useful heuristics, but their spread is not calibrated by construction either.

Conformal prediction sits at a different level. It does not compete with a Bayesian model or an ensemble; it can wrap either of them. Use the Bayesian posterior mean as f^ and the posterior standard deviation as σ^ , then conformalise. You keep the structure of the Bayesian model and gain a distribution-free coverage statement on top. That combination is often the most defensible choice in a high-stakes setting: a sensible model, plus a wrapper that does not trust the model.

Practical checklist

Before you ship conformal intervals, check these points:

• The calibration set was not used for training, tuning, or early stopping. Any reuse leaks optimism into the scores.
• The calibration set is large enough that the quantile is stable, and the split was random with respect to the deployment process.
• The score matches the task: absolute residuals for a first pass, normalised or quantile-based scores when heteroscedasticity matters, set-based scores for classification.
• You report both coverage and mean width. Coverage alone is easy to game by returning huge sets. Width alone is meaningless without coverage.
• You monitor coverage after deployment and have a plan to recalibrate when drift appears.
• You state the guarantee correctly: marginal coverage at the chosen level, under exchangeability, for the calibration distribution. No more, no less.

Takeaway

Conformal prediction is rare in machine learning: a method whose main claim is modest, precise, and actually true. It does not promise that your model is right. It promises that, if tomorrow looks like your calibration data, your stated uncertainty will be right at least as often as you claimed. For decisions where overconfidence is expensive, that is a much more useful promise than a sharper-looking point prediction.

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial