Uncertainty

Conformal prediction, ensemble and Gaussian-process uncertainty, and the calibration diagnostics that tell you whether an error bar means anything.

An applicability domain answers “should I trust this prediction at all”; uncertainty answers “how wrong is it likely to be”. They are different questions and a serious model reports both.

Conformal prediction

Conformal prediction is the only method here with a distribution-free coverage guarantee: set alpha=0.2 and, given exchangeable data, 80% of intervals contain the truth — regardless of the underlying model.

>>> from qsarkit.models import QSARRegressor
>>> from qsarkit.model_selection import RandomSplitter
>>> from qsarkit.uncertainty import ConformalRegressor
>>> X, y = demo_fingerprints(256), DEMO_Y
>>> train, test = next(RandomSplitter(test_size=0.25, random_state=0).split(X, y))
>>> conformal = ConformalRegressor(
...     QSARRegressor("rf", random_state=0), alpha=0.2, random_state=0
... ).fit(X[train], y[train])
>>> lower, upper = conformal.predict_interval(X[test])
>>> lower.shape, upper.shape
((6,), (6,))

evaluate checks the guarantee held:

>>> result = conformal.evaluate(X[test], y[test])
>>> result["coverage"], result["expected_coverage"]
(1.0, 0.8)
>>> round(result["mean_width"], 1)
4.2

Coverage of 1.0 against an expected 0.8 is not a bug — with six test compounds the empirical coverage can only take seven values, and the guarantee is marginal, not conditional. The width is the number that should worry you here: an interval 4.2 log units wide on data spanning 3.5 log units is honest about the model knowing very little, which is the correct conclusion from 18 training compounds.

The split-conformal construction spends part of the training set on calibration, which is what calibration_size controls. Pay it: an uncalibrated interval is a decoration.

Ensemble and Gaussian-process uncertainty

>>> from qsarkit.uncertainty import EnsembleUncertainty
>>> ensemble = EnsembleUncertainty(
...     QSARRegressor("rf", random_state=0), random_state=0
... ).fit(X[train], y[train])
>>> mean, sigma = ensemble.predict_uncertainty(X[test])
>>> mean.shape, sigma.shape
((6,), (6,))

These have no coverage guarantee — the spread of an ensemble is a proxy for uncertainty, not a measurement of it — which is exactly why the calibration diagnostics below matter.

Is the error bar meaningful?

>>> from qsarkit.uncertainty import UncertaintyCalibration
>>> calibration = UncertaintyCalibration(n_bins=3)
>>> report = calibration.report(y[test], mean, sigma)
>>> round(report["ence"], 1)
2.9
>>> round(report["spearman_error_correlation"], 2)
-0.54

Both numbers say the ensemble spread is not a usable error bar here. ENCE (expected normalized calibration error) should be near 0; 2.9 means the predicted σ is badly mis-scaled. The Spearman correlation between σ and actual error should be positive — a model should be least certain where it is most wrong. At −0.54 it is anti-correlated: this ensemble is most confident precisely where it errs.

That is a useful thing to discover before shipping predictions, and it is invisible if you only look at RMSE:

>>> round(report["rmse"], 3)
0.496

API

Uncertainty quantification for QSAR predictions.

Two complementary families:

UncertaintyCalibration checks whether either kind actually means what it claims.

Examples

>>> import numpy as np
>>> from sklearn.ensemble import RandomForestRegressor
>>> from qsarkit.uncertainty import ConformalRegressor
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(200, 4))
>>> y = X[:, 0] * 2 + rng.normal(scale=0.3, size=200)
>>> cp = ConformalRegressor(
...     RandomForestRegressor(n_estimators=20, random_state=0),
...     alpha=0.1, random_state=0,
... ).fit(X, y)
>>> lower, upper = cp.predict_interval(X)
>>> bool(np.all(upper >= lower))
True

References

class qsarkit.uncertainty.ConformalRegressor(estimator, alpha=0.1, normalized=False, difficulty_estimator=None, beta=0.1, calibration_size=0.3, random_state=None)[source]

Bases: BaseEstimator

Inductive (split) conformal prediction intervals for regression.

Conformal prediction turns any point predictor into an interval predictor with a guaranteed marginal coverage: at significance alpha, at least 1 - alpha of future predictions contain the true value. The guarantee needs only that the data be exchangeable — no distributional assumption, no assumption that the model is correct.

The split (inductive) variant fits the model on a proper training subset, computes nonconformity scores on a held-out calibration subset, and takes the empirical 1 - alpha quantile of those scores as the interval half-width.

Normalized conformal prediction scales each nonconformity score by a difficulty estimate, so easy molecules get tighter intervals than hard ones. Without it every compound receives the same width, which satisfies the coverage guarantee but says nothing useful about any individual prediction.

Parameters:

  • estimator (Any) – The underlying point predictor. Cloned, not modified.

  • alpha (float) – Significance level; intervals target 1 - alpha coverage.

  • normalized (bool) – Scale intervals by a per-sample difficulty estimate.

  • difficulty_estimator (Optional[Any]) – Model predicting the log absolute residual, used when normalized=True. Defaults to a k-NN regressor.

  • beta (float) – Stabilizer added to the difficulty estimate, preventing near-zero denominators from producing absurdly tight intervals.

  • calibration_size (float) – Fraction of fit data held out for calibration.

  • random_state (Optional[int]) – Seed for the calibration split.

Variables:

  • calibration_scores (ndarray) – Nonconformity scores on the calibration set.

  • quantile (float) – The 1 - alpha quantile used as the interval half-width.

Examples

>>> import numpy as np
>>> from sklearn.ensemble import RandomForestRegressor
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(200, 4))
>>> y = X[:, 0] * 2 + rng.normal(scale=0.3, size=200)
>>> cp = ConformalRegressor(RandomForestRegressor(n_estimators=20,
...                                              random_state=0),
...                         alpha=0.1, random_state=0).fit(X, y)
>>> lower, upper = cp.predict_interval(X)
>>> bool(np.all(upper >= lower))
True

References

calibration_scores_: ndarray[tuple[Any, ...], dtype[float64]]
quantile_: float
fit(X, y)[source]

Fit the model and calibrate nonconformity scores.

Parameters:
Return type:

ConformalRegressor

predict(X)[source]

Point predictions from the underlying model.

Return type:

ndarray[tuple[Any, ...], dtype[double]]

predict_interval(X, alpha=None)[source]

Prediction intervals at significance alpha.

Parameters:
Returns:

lower, upper

Return type:

Tuple[ndarray[tuple[Any, ...], dtype[double]], ndarray[tuple[Any, ...], dtype[double]]]

interval_width(X, alpha=None)[source]

Width of each prediction interval — the uncertainty estimate.

Return type:

ndarray[tuple[Any, ...], dtype[double]]

evaluate(X, y, alpha=None)[source]

Empirical coverage and efficiency on a held-out set.

Parameters:
Returns:

coverage (fraction of true values inside the interval), expected_coverage (1 - alpha), mean_width and median_width. A valid conformal predictor has coverage at or just above the expected value; among predictors that achieve it, narrower is better.

Return type:

dict

class qsarkit.uncertainty.ConformalClassifier(estimator, alpha=0.1, mondrian=False, calibration_size=0.3, random_state=None)[source]

Bases: BaseEstimator

Inductive conformal prediction sets for classification.

Instead of one label, returns the set of labels that cannot be rejected at significance alpha. The set size is the honest expression of uncertainty: a singleton means a confident call, two or more labels means the model genuinely cannot distinguish them, and an empty set means the compound resembles no training class at all — which a plain predict_proba can never tell you, since it always sums to one no matter how unfamiliar the input.

Parameters:

  • estimator (Any) – Must expose predict_proba. Cloned, not modified.

  • alpha (float) – Significance level.

  • mondrian (bool) – Calibrate per class rather than globally, which gives per-class rather than only marginal validity — important on imbalanced datasets, where global calibration lets the majority class absorb the error budget.

  • calibration_size (float) – Fraction of fit data held out for calibration.

  • random_state (Optional[int]) – Seed for the calibration split.

Variables:

  • classes (ndarray) – Class labels.

  • calibration_scores (ndarray) – Nonconformity scores on the calibration set.

Examples

>>> import numpy as np
>>> from sklearn.ensemble import RandomForestClassifier
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(200, 4))
>>> y = (X[:, 0] > 0).astype(int)
>>> cp = ConformalClassifier(RandomForestClassifier(n_estimators=20,
...                                                 random_state=0),
...                          alpha=0.1, random_state=0).fit(X, y)
>>> sets = cp.predict_set(X[:5])
>>> all(isinstance(s, list) for s in sets)
True

References

classes_: ndarray[tuple[Any, ...], dtype[Any]]
calibration_scores_: ndarray[tuple[Any, ...], dtype[float64]]
fit(X, y)[source]

Fit the classifier and calibrate nonconformity scores.

Parameters:
Return type:

ConformalClassifier

p_values(X)[source]

Conformal p-value for each (sample, class) pair.

Parameters:

X (Union[Buffer, _SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], complex, bytes, str, _NestedSequence[complex | bytes | str]])

Returns:

The fraction of calibration scores at least as nonconforming as this sample would be if it belonged to that class.

Return type:

ndarray[tuple[Any, ...], dtype[double]]

predict_set(X, alpha=None)[source]

Prediction sets: every label not rejected at alpha.

Parameters:
Returns:

One label list per sample. May be empty (nothing conforms) or hold several labels (genuinely ambiguous).

Return type:

List[List[Any]]

predict(X)[source]

Point predictions from the underlying classifier.

Return type:

ndarray[tuple[Any, ...], dtype[Any]]

evaluate(X, y, alpha=None)[source]

Coverage and set-size statistics on a held-out set.

Parameters:
Returns:

coverage, expected_coverage, mean_set_size, singleton_fraction (confident calls) and empty_fraction (compounds resembling no training class).

Return type:

dict

qsarkit.uncertainty.ConformalPredictor(estimator, task='auto', **kwargs)[source]

Build the right conformal predictor for an estimator.

Parameters:
Return type:

Any

Examples

>>> from sklearn.ensemble import RandomForestRegressor
>>> type(ConformalPredictor(RandomForestRegressor())).__name__
'ConformalRegressor'

References

class qsarkit.uncertainty.BaseUncertaintyEstimator[source]

Bases: BaseEstimator, ABC

Common interface: fit, predict, and predict_uncertainty.

Every estimator here returns a point prediction and a per-sample standard deviation, so they are interchangeable wherever a model’s confidence is needed — active-learning acquisition functions, applicability-domain scoring, or simply reporting error bars.

References

  • Hirschfeld, L. et al. (2020). “Uncertainty Quantification Using Neural Networks for Molecular Property Prediction.” J. Chem. Inf. Model., 60(8), 3770-3780. https://doi.org/10.1021/acs.jcim.0c00502

  • Scalia, G. et al. (2020). “Evaluating Scalable Uncertainty Estimation Methods for Deep Learning-Based Molecular Property Prediction.” J. Chem. Inf. Model., 60(6), 2697-2717. https://doi.org/10.1021/acs.jcim.9b00975

abstractmethod fit(X, y)[source]

Fit the underlying model(s).

Return type:

BaseUncertaintyEstimator

abstractmethod predict_uncertainty(X)[source]

Return (mean, std) per sample.

Return type:

Tuple[ndarray[tuple[Any, ...], dtype[double]], ndarray[tuple[Any, ...], dtype[double]]]

predict(X)[source]

Point predictions (the mean of predict_uncertainty()).

Return type:

ndarray[tuple[Any, ...], dtype[double]]

predict_interval(X, n_std=1.96)[source]

Gaussian prediction interval at n_std standard deviations.

Parameters:

  • X (Union[Buffer, _SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], complex, bytes, str, _NestedSequence[complex | bytes | str]])

  • n_std (float) – Multiplier; 1.96 gives a nominal 95% interval if the errors are Gaussian. When that assumption is doubtful — which for QSAR it usually is — prefer ConformalRegressor, whose coverage guarantee is distribution-free.

Returns:

lower, upper

Return type:

Tuple[ndarray[tuple[Any, ...], dtype[double]], ndarray[tuple[Any, ...], dtype[double]]]

class qsarkit.uncertainty.EnsembleUncertainty(estimator, n_estimators=10, bootstrap=True, use_native_ensemble=True, random_state=None)[source]

Bases: BaseUncertaintyEstimator

Uncertainty from the disagreement among an ensemble.

Trains several models — on bootstrap resamples, or with different seeds — and reports the spread of their predictions. Where the members agree the prediction is well determined by the data; where they diverge it is not. For a random forest the trees already form an ensemble, so their per-tree predictions are used directly rather than refitting.

Parameters:

  • estimator (Any) – Base model. Cloned, not modified.

  • n_estimators (int) – Number of ensemble members (ignored when reusing a forest’s own trees).

  • bootstrap (bool) – Resample the training data for each member. Without it, members differ only through their own randomness, which understates uncertainty for deterministic learners.

  • use_native_ensemble (bool) – For estimators exposing estimators_ (forests, bagging), use the existing members instead of training new ones.

  • random_state (Optional[int]) – Seed.

Variables:

estimators (list) – The fitted members.

Examples

>>> import numpy as np
>>> from sklearn.tree import DecisionTreeRegressor
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(60, 3)); y = X[:, 0] * 2
>>> est = EnsembleUncertainty(DecisionTreeRegressor(), n_estimators=5,
...                           random_state=0).fit(X, y)
>>> mean, std = est.predict_uncertainty(X)
>>> bool(np.all(std >= 0))
True

References

estimators_: List[Any]
fit(X, y)[source]

Fit the ensemble members.

Parameters:
Return type:

EnsembleUncertainty

predict_uncertainty(X)[source]

Mean and standard deviation across the ensemble members.

Return type:

Tuple[ndarray[tuple[Any, ...], dtype[double]], ndarray[tuple[Any, ...], dtype[double]]]

class qsarkit.uncertainty.MCDropoutUncertainty(model, n_samples=50, device=None)[source]

Bases: BaseUncertaintyEstimator

Monte-Carlo dropout uncertainty for a PyTorch network.

Keeps dropout active at prediction time and samples the network several times. Gal and Ghahramani showed this approximates variational inference in a deep Gaussian process, so the sample spread is a principled posterior estimate rather than just noise — at the cost of one forward pass per sample.

Parameters:

  • model (Any) – A network containing at least one dropout layer. Without one, every pass is identical and the reported uncertainty is zero.

  • n_samples (int) – Forward passes per prediction.

  • device (Optional[str]) – Torch device; defaults to the model’s own.

Examples

>>> import pytest
>>> torch = pytest.importorskip("torch")

References

fit(X, y)[source]

No-op: the wrapped network is expected to be trained already.

Parameters:
Return type:

MCDropoutUncertainty

predict_uncertainty(X)[source]

Mean and standard deviation over n_samples stochastic passes.

Return type:

Tuple[ndarray[tuple[Any, ...], dtype[double]], ndarray[tuple[Any, ...], dtype[double]]]

class qsarkit.uncertainty.GaussianProcessUncertainty(kernel=None, alpha=1e-10, normalize_y=True, random_state=None)[source]

Bases: BaseUncertaintyEstimator

Posterior standard deviation of a Gaussian process.

The only method here whose uncertainty is exact rather than approximate: a GP returns a full posterior, so the standard deviation is the model’s own belief, not a sample statistic. The cost is cubic in training-set size, which caps it at a few thousand compounds.

Parameters:

  • kernel (Optional[Any]) – Defaults to an RBF with a white-noise term. For fingerprints, pass TanimotoKernel.

  • alpha (float) – Value added to the diagonal for numerical stability.

  • normalize_y (bool) – Standardize the target before fitting.

  • random_state (Optional[int]) – Seed.

Examples

>>> import numpy as np
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(40, 2)); y = X[:, 0]
>>> gp = GaussianProcessUncertainty(random_state=0).fit(X, y)
>>> mean, std = gp.predict_uncertainty(X)
>>> bool(np.all(std >= 0))
True

References

fit(X, y)[source]

Fit the Gaussian process.

Parameters:
Return type:

GaussianProcessUncertainty

predict_uncertainty(X)[source]

Posterior mean and standard deviation.

Return type:

Tuple[ndarray[tuple[Any, ...], dtype[double]], ndarray[tuple[Any, ...], dtype[double]]]

class qsarkit.uncertainty.QuantileRegressionUncertainty(estimator=None, quantiles=(0.05, 0.95), random_state=None)[source]

Bases: BaseUncertaintyEstimator

Uncertainty from quantile regression.

Fits separate models for a lower quantile, the median and an upper quantile. Unlike every other estimator here it does not assume the error is symmetric or constant, so it captures heteroscedasticity — the common situation where potent compounds are measured more precisely than weak ones.

Parameters:

  • estimator (Optional[Any]) – Must accept a quantile/alpha parameter. Defaults to GradientBoostingRegressor(loss="quantile").

  • quantiles (Tuple[float, float]) – Lower and upper quantiles to fit.

  • random_state (Optional[int]) – Seed.

Examples

>>> import numpy as np
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(120, 3)); y = X[:, 0] * 2 + rng.normal(size=120)
>>> q = QuantileRegressionUncertainty(random_state=0).fit(X, y)
>>> lower, upper = q.predict_interval(X)
>>> bool(np.all(upper >= lower))
True

References

fit(X, y)[source]

Fit lower, median and upper quantile models.

Parameters:
Return type:

QuantileRegressionUncertainty

predict_uncertainty(X)[source]

Median prediction and a standard deviation implied by the quantile span.

Return type:

Tuple[ndarray[tuple[Any, ...], dtype[double]], ndarray[tuple[Any, ...], dtype[double]]]

predict_interval(X, n_std=1.96)[source]

The fitted quantiles directly — no Gaussian assumption needed.

Parameters:
Returns:

lower, upper

Return type:

Tuple[ndarray[tuple[Any, ...], dtype[double]], ndarray[tuple[Any, ...], dtype[double]]]

class qsarkit.uncertainty.UncertaintyCalibration(n_bins=10)[source]

Bases: object

Assess whether predicted uncertainties mean what they claim.

An uncertainty estimate is only useful if it is calibrated: when a model says +/-0.5, the true value should land inside that interval about as often as the nominal level promises. Models routinely fail this — deep ensembles are typically overconfident — and a well-ranked but miscalibrated uncertainty will silently break any downstream decision rule with an absolute threshold.

Two distinct properties are measured here, and a good estimator needs both:

  • Calibration (ENCE, miscalibration area, coverage curve): are the magnitudes right?

  • Ranking (Spearman correlation of uncertainty with absolute error): do higher-uncertainty predictions actually err more?

Parameters:

n_bins (int) – Number of equal-count bins used for the binned statistics.

Examples

>>> import numpy as np
>>> rng = np.random.RandomState(0)
>>> y_true = rng.normal(size=500)
>>> sigma = np.full(500, 1.0)
>>> y_pred = y_true + rng.normal(scale=1.0, size=500)
>>> cal = UncertaintyCalibration()
>>> report = cal.report(y_true, y_pred, sigma)
>>> 0.0 <= report["ence"] < 1.0
True

References

  • Levi, D. et al. (2022). “Evaluating and Calibrating Uncertainty Prediction in Regression Tasks.” Sensors, 22(15), 5540. https://doi.org/10.3390/s22155540

  • Kuleshov, V., Fenner, N. & Ermon, S. (2018). “Accurate Uncertainties for Deep Learning Using Calibrated Regression.” ICML 2018. https://arxiv.org/abs/1807.00263

  • Scalia, G. et al. (2020). “Evaluating Scalable Uncertainty Estimation Methods for Deep Learning-Based Molecular Property Prediction.” J. Chem. Inf. Model., 60(6), 2697-2717. https://doi.org/10.1021/acs.jcim.9b00975

  • Tran, K. et al. (2020). “Methods for Comparing Uncertainty Quantifications for Material Property Predictions.” Mach. Learn.: Sci. Technol., 1, 025006. https://doi.org/10.1088/2632-2153/ab7e1a

ence(y_true, y_pred, sigma)[source]

Expected Normalized Calibration Error.

Samples are binned by predicted uncertainty; within each bin the root-mean-square error is compared to the mean predicted sigma. ENCE is the mean relative discrepancy — 0 is perfect.

Parameters:
Returns:

Non-negative; smaller is better.

Return type:

float

References

coverage_curve(y_true, y_pred, sigma, n_points=20)[source]

Observed coverage against nominal confidence level.

For each nominal level, the fraction of true values falling inside the corresponding Gaussian interval. A perfectly calibrated model traces the diagonal; below it is overconfident.

Parameters:
Returns:

nominal, observed

Return type:

Tuple[ndarray[tuple[Any, ...], dtype[double]], ndarray[tuple[Any, ...], dtype[double]]]

miscalibration_area(y_true, y_pred, sigma, n_points=20)[source]

Area between the coverage curve and the ideal diagonal.

Parameters:
Returns:

0 is perfect calibration; the maximum is about 0.5.

Return type:

float

References

spearman_error_correlation(y_true, y_pred, sigma)[source]

Rank correlation between predicted uncertainty and absolute error.

Measures ranking quality rather than calibration: whether the model knows which predictions are worse, regardless of whether the magnitudes are right. An estimator can score well here and still be badly calibrated (and vice versa), which is why both are reported.

Parameters:
Returns:

Spearman rho in [-1, 1]; higher is better.

Return type:

float

report(y_true, y_pred, sigma)[source]

Full calibration report.

Parameters:
Returns:

ence, miscalibration_area, spearman_error_correlation, coverage_68, coverage_95 (observed coverage at the nominal 1- and 2-sigma levels), mean_sigma and rmse.

Return type:

Dict[str, float]

plot_calibration(y_true, y_pred, sigma, n_points=20)[source]

Plot the coverage curve against the ideal diagonal.

Parameters:
Return type:

Figure

References

  • Vovk, V., Gammerman, A. & Shafer, G. (2005). “Algorithmic Learning in a Random World.” Springer. doi:10.1007/b106715

  • Papadopoulos, H. et al. (2002). “Inductive Confidence Machines for Regression.” ECML 2002, 345-356. doi:10.1007/3-540-36755-1_29

  • Norinder, U. et al. (2014). “Introducing Conformal Prediction in Predictive Modeling.” J. Chem. Inf. Model., 54(6), 1596-1603. doi:10.1021/ci5001168

  • Levi, D. et al. (2022). “Evaluating and Calibrating Uncertainty Prediction in Regression Tasks.” Sensors, 22(15), 5540. doi:10.3390/s22155540

  • Hirschfeld, L. et al. (2020). “Uncertainty Quantification Using Neural Networks for Molecular Property Prediction.” J. Chem. Inf. Model., 60(8), 3770-3780. doi:10.1021/acs.jcim.0c00502