Model selection

QSAR-aware data splitting, nested cross-validation and hyperparameter search.

How you split is a bigger decision than which model you fit. A random split of a QSAR dataset measures interpolation: public sets are dense with near-duplicate analogues, so random assignment scatters a congeneric series across both sides and scores the model on compounds whose close relatives it has memorized.

The splitters

Every splitter yields (train_idx, test_idx) and follows the scikit-learn protocol, so they drop into cross_val_score unchanged:

>>> from qsarkit.model_selection import ScaffoldSplitter
>>> X, y = demo_fingerprints(256), DEMO_Y
>>> train, test = next(ScaffoldSplitter(test_size=0.25).split_mols(demo_mols, y))
>>> len(train), len(test)
(18, 6)

The guarantee is that no scaffold appears on both sides:

>>> from qsarkit.chemspace import bemis_murcko_smiles
>>> train_scaffolds = {bemis_murcko_smiles(demo_mols[i]) for i in train}
>>> test_scaffolds = {bemis_murcko_smiles(demo_mols[i]) for i in test}
>>> train_scaffolds & test_scaffolds
set()

split_mols takes molecules; split takes the matrix and accepts molecules as groups for the chemistry-aware splitters:

>>> from qsarkit.model_selection import (
...     ButinaClusterSplitter, KennardStoneSplitter, RandomSplitter)
>>> len(next(RandomSplitter(test_size=0.25, random_state=0).split(X, y))[1])
6
>>> len(next(ButinaClusterSplitter(test_size=0.25).split_mols(demo_mols, y))[1])
6
>>> len(next(KennardStoneSplitter(test_size=0.25).split(X, y))[1])
6

Which to use:

ScaffoldSplitter

The default hard split. Keeps whole chemotypes together, so the test set is chemistry the model has never seen. Deterministic — no seed.

StratifiedScaffoldSplitter

The same, preserving class balance. Use for classification when the actives are scarce.

ButinaClusterSplitter / SphereExclusionSplitter

Split by similarity cluster rather than scaffold. Catches analogue series that share no Bemis-Murcko framework.

KennardStoneSplitter / PerimeterSplitter

Deterministic coverage-driven selection. Kennard-Stone puts the most spread-out compounds in training, which is what you want when training data is precious and the test set need only be representative.

TimeSplitter

Train on early compounds, test on later ones. The only split that simulates prospective use, and reliably the most pessimistic.

RandomSplitter

Included for baselines and for the comparison that shows how much the others cost you.

API

QSAR-aware data splitting and hyperparameter search.

A random split flatters a QSAR model: molecular datasets are dense with near-duplicate analogues, so random assignment puts close relatives on both sides and measures interpolation rather than generalization to new chemistry. The splitters here make the evaluation harder in specific, defensible ways – and the gap between a random and a scaffold split is the size of the illusion.

Examples

>>> from rdkit import Chem
>>> from qsarkit.model_selection import ScaffoldSplitter
>>> mols = [Chem.MolFromSmiles(s) for s in
...         ("c1ccccc1C", "c1ccccc1CC", "c1ccncc1C", "CCO", "CCN")]
>>> train, test = next(ScaffoldSplitter(test_size=0.4).split_mols(mols))
>>> set(train) & set(test)
set()

References

class qsarkit.model_selection.BaseSplitter(test_size=0.2, random_state=None)[source]

Bases: ABC

Common machinery: one train/test split, in scikit-learn’s shape.

Parameters:

  • test_size (float) – Fraction of the dataset assigned to the test set.

  • random_state (Optional[int]) – Seed, where the splitter has a stochastic component.

References

split(X, y=None, groups=None)[source]

Yield one (train_idx, test_idx) pair.

Parameters:
Yields:

train_idx, test_idx (ndarray of int)

Return type:

Iterator[Tuple[ndarray[tuple[Any, …], dtype[int64]], ndarray[tuple[Any, …], dtype[int64]]]]

split_mols(mols, y=None)[source]

Yield one split, taking RDKit molecules directly.

Parameters:
Yields:

train_idx, test_idx (ndarray of int)

Return type:

Iterator[Tuple[ndarray[tuple[Any, …], dtype[int64]], ndarray[tuple[Any, …], dtype[int64]]]]

get_n_splits(X=None, y=None, groups=None)[source]

Number of splits produced (always 1 for these splitters).

Return type:

int

class qsarkit.model_selection.RandomSplitter(test_size=0.2, random_state=None)[source]

Bases: BaseSplitter

Uniformly random split.

The baseline every other splitter should be compared against — and, on molecular data, almost always the optimistic one.

Parameters:

Examples

>>> import numpy as np
>>> X = np.arange(20).reshape(10, 2)
>>> train, test = next(RandomSplitter(random_state=0).split(X))
>>> len(train), len(test)
(8, 2)

References

class qsarkit.model_selection.ScaffoldSplitter(test_size=0.2, include_chirality=False, random_state=None)[source]

Bases: BaseSplitter

Split by Bemis-Murcko scaffold, largest scaffold group first.

Guarantees that no scaffold appears on both sides, so the test set contains only chemistry the model has never seen. This is the standard hard split in molecular machine learning, and typically drops reported performance substantially relative to a random split — which is the point: the gap is the size of the illusion.

Assigning the largest scaffold groups to training first is deterministic (no seed needed) and keeps the rarest, most distinct chemotypes in the test set.

Parameters:

  • test_size (float)

  • include_chirality (bool) – Treat enantiomers as different scaffolds.

  • random_state (Optional[int]) – Unused; accepted for interface symmetry.

Examples

>>> from rdkit import Chem
>>> mols = [Chem.MolFromSmiles(s) for s in
...         ("c1ccccc1C", "c1ccccc1CC", "c1ccncc1C", "CCO", "CCN")]
>>> train, test = next(ScaffoldSplitter(test_size=0.4).split_mols(mols))
>>> set(train) & set(test)
set()

References

class qsarkit.model_selection.StratifiedScaffoldSplitter(test_size=0.2, include_chirality=False, random_state=None)[source]

Bases: ScaffoldSplitter

Scaffold split that also balances the label distribution.

Assigns scaffold groups greedily to whichever side is currently furthest from its target label mean (regression) or class balance (classification). Keeps the scaffold-disjointness guarantee while avoiding the common failure where the test set ends up composed entirely of inactives.

Parameters:

Examples

>>> from rdkit import Chem
>>> mols = [Chem.MolFromSmiles(s) for s in
...         ("c1ccccc1C", "c1ccncc1C", "CCO", "CCN", "c1ccccc1CC")]
>>> y = [1, 0, 1, 0, 1]
>>> train, test = next(
...     StratifiedScaffoldSplitter(test_size=0.4).split_mols(mols, y)
... )
>>> set(train) & set(test)
set()

References

class qsarkit.model_selection.ButinaClusterSplitter(test_size=0.2, cutoff=0.35, n_bits=2048, random_state=None)[source]

Bases: BaseSplitter

Split by Taylor-Butina cluster, keeping whole clusters together.

A softer alternative to a scaffold split: it groups by overall fingerprint similarity rather than exact scaffold identity, so it also separates molecules that share no scaffold but are still very similar — which a scaffold split happily puts on opposite sides.

Parameters:

  • test_size (float)

  • cutoff (float) – Butina distance cutoff (Tanimoto similarity 1 - cutoff).

  • n_bits (int) – Fingerprint length.

  • random_state (Optional[int])

Examples

>>> from rdkit import Chem
>>> mols = [Chem.MolFromSmiles(s) for s in
...         ("CCO", "CCN", "c1ccccc1", "c1ccccc1C", "CCCCCC")]
>>> train, test = next(ButinaClusterSplitter(test_size=0.4).split_mols(mols))
>>> set(train) & set(test)
set()

References

  • Butina, D. (1999). “Unsupervised Data Base Clustering Based on Daylight’s Fingerprint and Tanimoto Similarity.” J. Chem. Inf. Comput. Sci., 39(4), 747-750. https://doi.org/10.1021/ci9803381

class qsarkit.model_selection.SphereExclusionSplitter(test_size=0.2, cutoff=0.35, n_bits=2048, random_state=None)[source]

Bases: ButinaClusterSplitter

Split by sphere-exclusion cluster, keeping whole clusters together.

Like ButinaClusterSplitter but using leader-based sphere exclusion, which guarantees a minimum distance between cluster centres and scales to much larger libraries.

Parameters:

Examples

>>> from rdkit import Chem
>>> mols = [Chem.MolFromSmiles(s) for s in ("CCO", "CCN", "c1ccccc1", "CCCC")]
>>> train, test = next(SphereExclusionSplitter(test_size=0.5).split_mols(mols))
>>> set(train) & set(test)
set()

References

class qsarkit.model_selection.MaxMinSplitter(test_size=0.2, n_bits=2048, random_state=None)[source]

Bases: BaseSplitter

Put a maximally diverse subset in the training set.

Uses MaxMin picking to choose training compounds that span the chemical space as widely as possible, leaving the denser regions for testing. This is the split to use when the question is “how few compounds do I need to measure?” rather than “how well does this extrapolate?” — it is deliberately the optimistic structure-aware split, and pairs well with a scaffold split as the pessimistic bound.

Parameters:

Examples

>>> from rdkit import Chem
>>> mols = [Chem.MolFromSmiles(s) for s in
...         ("CCO", "CCN", "c1ccccc1", "c1ccccc1C", "CCCCCC")]
>>> train, test = next(MaxMinSplitter(test_size=0.4).split_mols(mols))
>>> len(train) + len(test)
5

References

class qsarkit.model_selection.TimeSplitter(test_size=0.2, random_state=None)[source]

Bases: BaseSplitter

Split chronologically: earliest compounds train, latest test.

The most honest evaluation available, because it reproduces the real prospective task — predicting compounds that had not been made yet. Sheridan showed time-split validation gives markedly lower, and much more realistic, performance estimates than random or even scaffold-based splits.

Parameters:

  • test_size (float)

  • random_state (Optional[int]) – Unused; the split is fully determined by the dates.

Examples

>>> import numpy as np
>>> X = np.arange(20).reshape(10, 2)
>>> dates = np.arange(10)
>>> train, test = next(TimeSplitter(test_size=0.3).split(X, groups=dates))
>>> bool(dates[train].max() <= dates[test].min())
True

References

  • Sheridan, R. P. (2013). “Time-Split Cross-Validation as a Method for Estimating the Goodness of Prospective Prediction.” J. Chem. Inf. Model., 53(4), 783-790. https://doi.org/10.1021/ci400084k

class qsarkit.model_selection.KennardStoneSplitter(test_size=0.2, metric='euclidean', random_state=None)[source]

Bases: BaseSplitter

Kennard-Stone: training set covers the descriptor space uniformly.

Selects training points to be maximally far apart, starting from the two most distant compounds. Deterministic, and produces a training set whose convex hull encloses most of the test set — which makes it the natural companion to a leverage-based applicability domain, since almost every test compound ends up inside it.

Parameters:

  • test_size (float)

  • metric (str) – Any metric accepted by scipy.spatial.distance.cdist.

  • random_state (Optional[int]) – Unused; the algorithm is deterministic.

Examples

>>> import numpy as np
>>> X = np.random.RandomState(0).normal(size=(20, 3))
>>> train, test = next(KennardStoneSplitter(test_size=0.25).split(X))
>>> len(train), len(test)
(15, 5)

References

class qsarkit.model_selection.PerimeterSplitter(test_size=0.2, random_state=None)[source]

Bases: BaseSplitter

Put the outermost compounds in the training set.

Selects the points furthest from the dataset centroid for training, leaving the interior for testing. The mirror image of a scaffold split: it makes every prediction an interpolation, giving the most favourable honest estimate of a model’s performance inside its own domain.

Parameters:

Examples

>>> import numpy as np
>>> X = np.random.RandomState(0).normal(size=(20, 3))
>>> train, test = next(PerimeterSplitter(test_size=0.25).split(X))
>>> len(train), len(test)
(15, 5)

References

  • Martin, T. M. et al. (2012). “Does Rational Selection of Training and Test Sets Improve the Outcome of QSAR Modeling?” J. Chem. Inf. Model., 52(10), 2570-2578. https://doi.org/10.1021/ci300338w

class qsarkit.model_selection.NestedCV(estimator, param_grid, inner_cv=3, outer_cv=5, scoring=None, n_jobs=None)[source]

Bases: object

Nested cross-validation: unbiased performance for a tuned model.

Tuning hyperparameters and reporting the best cross-validated score from that same search is one of the most common ways QSAR papers overstate performance — the score is optimistically biased because the test folds were used to choose the model. Nested CV fixes it by tuning inside an inner loop and scoring on outer folds the tuning never saw.

Parameters:

  • estimator (BaseEstimator) – Model to tune and evaluate.

  • param_grid (Dict[str, Sequence[Any]]) – Search space for the inner loop.

  • inner_cv (Any) – Cross-validation used for tuning.

  • outer_cv (Any) – Cross-validation used for scoring.

  • scoring (Optional[str]) – scikit-learn scorer name.

  • n_jobs (Optional[int]) – Parallel jobs.

Variables:

  • scores (ndarray) – Outer-fold scores.

  • best_params (list of dict) – The parameters chosen in each outer fold. Disagreement between folds is itself informative: it means the tuning is unstable and the “best” parameters from a single search are noise.

Examples

>>> from sklearn.datasets import make_regression
>>> from sklearn.linear_model import Ridge
>>> X, y = make_regression(n_samples=40, n_features=5, random_state=0)
>>> ncv = NestedCV(Ridge(), {"alpha": [0.1, 1.0]}, inner_cv=2, outer_cv=2)
>>> _ = ncv.run(X, y)
>>> ncv.scores_.shape
(2,)

References

  • Varma, S. & Simon, R. (2006). “Bias in Error Estimation when Using Cross-Validation for Model Selection.” BMC Bioinformatics, 7, 91. https://doi.org/10.1186/1471-2105-7-91

  • Cawley, G. C. & Talbot, N. L. C. (2010). “On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation.” J. Mach. Learn. Res., 11, 2079-2107. https://jmlr.org/papers/v11/cawley10a.html

  • Baumann, D. & Baumann, K. (2014). “Reliable Estimation of Prediction Errors for QSAR Models under Model Uncertainty Using Double Cross-Validation.” J. Cheminform., 6, 47. https://doi.org/10.1186/s13321-014-0047-1

scores_: ndarray[tuple[Any, ...], dtype[float64]]
best_params_: List[Dict[str, Any]]
run(X, y)[source]

Run the nested loop.

Parameters:
Returns:

mean_score, std_score, scores (per outer fold), and best_params (the winner in each outer fold).

Return type:

Dict[str, Any]

Tune hyperparameters by grid, random or Bayesian search.

Parameters:

  • estimator (BaseEstimator) – The model to tune. Cloned, not modified.

  • param_grid (Union[Dict[str, Sequence[Any]], List[Dict[str, Sequence[Any]]]]) – Parameter name -> values to try, in scikit-learn’s format.

  • X (Union[Buffer, _SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], complex, bytes, str, _NestedSequence[complex | bytes | str]])

  • y (Union[Buffer, _SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], complex, bytes, str, _NestedSequence[complex | bytes | str]])

  • method (Literal['grid', 'random', 'optuna']) – Exhaustive grid, random sampling, or Optuna’s TPE sampler (requires the optional optuna package). Random search beats grid search on the same budget whenever only a few parameters actually matter, which is the usual case.

  • cv (Any) – Pass a qsarkit.model_selection splitter to tune under a scaffold or time split rather than a random one.

  • scoring (Optional[str]) – scikit-learn scorer name. Defaults to the estimator’s own score.

  • n_iter (int) – Number of parameter settings sampled by the random and Optuna methods.

  • n_jobs (Optional[int]) – Parallel jobs.

  • random_state (Optional[int]) – Seed.

  • **kwargs (Any) – Forwarded to the underlying search object.

Returns:

Fitted, exposing best_estimator_, best_params_ and best_score_.

Return type:

Any

Examples

>>> from sklearn.datasets import make_regression
>>> from sklearn.ensemble import RandomForestRegressor
>>> X, y = make_regression(n_samples=40, n_features=5, random_state=0)
>>> search = hyperparameter_search(
...     RandomForestRegressor(random_state=0),
...     {"n_estimators": [5, 10]}, X, y, cv=2,
... )
>>> "n_estimators" in search.best_params_
True

References

References

  • Bemis, G. W. & Murcko, M. A. (1996). “The Properties of Known Drugs. 1. Molecular Frameworks.” J. Med. Chem., 39(15), 2887-2893. doi:10.1021/jm9602928

  • Sheridan, R. P. (2013). “Time-Split Cross-Validation as a Method for Estimating the Goodness of Prospective Prediction.” J. Chem. Inf. Model., 53(4), 783-790. doi:10.1021/ci400084k

  • Kennard, R. W. & Stone, L. A. (1969). “Computer Aided Design of Experiments.” Technometrics, 11(1), 137-148. doi:10.1080/00401706.1969.10490666

  • Wu, Z. et al. (2018). “MoleculeNet: A Benchmark for Molecular Machine Learning.” Chem. Sci., 9, 513-530. doi:10.1039/C7SC02664A

  • Cawley, G. C. & Talbot, N. L. C. (2010). “On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation.” J. Mach. Learn. Res., 11, 2079-2107. https://jmlr.org/papers/v11/cawley10a.html