Feature selection

Descriptor selection: variance and correlation filters, mutual information, recursive feature elimination and Boruta. All are scikit-learn selectors, so they compose in a pipeline and expose get_support().

Warning

Select features inside the cross-validation loop, not before it. Choosing columns using all the labels and then splitting is selection bias: the choice has already seen the test set, and the held-out score is optimistic by an amount you cannot estimate afterwards.

Filters

The cheapest and most defensible reductions, because they use no labels at all.

>>> from qsarkit.feature_selection import VarianceFilter
>>> X, y = demo_fingerprints(256), DEMO_Y
>>> VarianceFilter().fit_transform(X).shape
(24, 107)

More than half the bits never fire on 24 compounds. They cannot contribute and only slow things down.

>>> from qsarkit.feature_selection import CorrelationFilter
>>> selector = CorrelationFilter(threshold=0.9).fit(X, y)
>>> selector.transform(X).shape
(24, 61)

Constant columns have undefined correlation, so they are identified and dropped explicitly rather than falling out of NaN comparisons:

>>> int(selector.constant_.sum())
149

When y is supplied, the filter keeps whichever member of a correlated pair correlates better with the target — a better choice than keeping whichever happened to come first.

Supervised selection

>>> from qsarkit.feature_selection import MutualInformationSelector, RFESelector
>>> MutualInformationSelector(k=8).fit_transform(X, y).shape
(24, 8)
>>> RFESelector(n_features_to_select=8).fit_transform(X, y).shape
(24, 8)

Mutual information catches non-linear dependence that a correlation filter misses, and needs no model. RFE is more powerful and far more expensive, and its answer is specific to the estimator it wrapped.

Boruta

Boruta asks a different question: not “which are the best k features” but “which features carry more signal than random noise”. It answers with a set of whatever size the data supports, rather than a number you chose:

>>> from qsarkit.feature_selection import BorutaSelector
>>> boruta = BorutaSelector(n_iterations=30, random_state=0).fit(X, y)
>>> int(boruta.get_support().sum()) >= 0
True

Note

Boruta needs enough iterations for its statistical test to reach significance. With too few it confidently selects nothing — the selector warns when the iteration count cannot support a decision at the chosen alpha, rather than silently returning an empty set.

API

Descriptor selection for QSAR models.

Molecular descriptor sets routinely contain hundreds of correlated, constant or uninformative columns. Selecting among them improves both generalization and the mechanistic interpretability OECD principle 5 asks for. Every selector is scikit-learn compatible with get_support().

Examples

>>> from sklearn.datasets import make_regression
>>> from qsarkit.feature_selection import VarianceFilter
>>> X, y = make_regression(n_samples=50, n_features=8, random_state=0)
>>> VarianceFilter().fit(X, y).get_support().shape
(8,)

References

class qsarkit.feature_selection.VarianceFilter(threshold=0.0)[source]

Bases: SelectorMixin, BaseEstimator

Drop near-constant descriptors.

Constant or near-constant descriptor columns carry no discriminating information and can break downstream scalers (division by a near-zero standard deviation) or destabilize linear-model fits. This is the standard first curation step applied to any QSAR descriptor matrix before feature selection or modelling proper begins.

The algorithm is the two-line computation variances_ = X.var(axis=0), support_ = variances_ > threshold — exactly what sklearn.feature_selection.VarianceThreshold implements, reproduced directly here so this estimator shares the qsarkit selector contract (support_, get_support(), transform() via sklearn.feature_selection.SelectorMixin).

Parameters:

threshold (float) – Descriptors with a training-set variance at or below this value are dropped. The default removes only exactly-constant columns.

Variables:

  • variances (ndarray of shape (n_features,)) – Per-descriptor variance computed on the training data.

  • support (ndarray of bool of shape (n_features,)) – True for descriptors kept (variances_ > threshold).

  • n_features_in (int) – Number of descriptors seen during fit.

Examples

>>> import numpy as np
>>> from qsarkit.feature_selection import VarianceFilter
>>> X = np.array([[1.0, 5.0], [2.0, 5.0], [3.0, 5.0]])
>>> vf = VarianceFilter().fit(X)
>>> vf.support_.tolist()
[True, False]

References

variances_: ndarray[tuple[Any, ...], dtype[float64]]
support_: ndarray[tuple[Any, ...], dtype[bool]]
n_features_in_: int
fit(X, y=None)[source]

Learn per-descriptor variances and the resulting support mask.

Parameters:
Returns:

The fitted selector.

Return type:

VarianceFilter

class qsarkit.feature_selection.CorrelationFilter(threshold=0.95, method='pearson')[source]

Bases: SelectorMixin, BaseEstimator

Drop one descriptor from every pair whose correlation exceeds a threshold.

Collinear descriptors add noise to linear models, inflate coefficient variance and make coefficient interpretation unreliable — a standard QSAR descriptor-curation concern. This selector removes redundant descriptors while trying to keep the more informative member of each correlated pair.

Notes

The algorithm is:

  1. Compute the descriptor-descriptor correlation matrix (Pearson or Spearman).

  2. Determine a processing order: if y is supplied, features are visited in descending order of |corr(feature, y)| so the more target-relevant member of a correlated pair is considered first; otherwise features are visited in their natural column order.

  3. Walk the ordered features, keeping a running set of accepted indices. A candidate feature is kept if its absolute correlation with every already-kept feature is at or below threshold; otherwise it is redundant with something already kept, and is dropped.

Because the target-relevant feature is visited first when y is given, a duplicated/near-duplicated copy of an informative descriptor is reliably the one dropped, not the informative descriptor itself.

Parameters:

  • threshold (float) – Absolute correlation above which a later feature is considered redundant with an earlier, already-kept one.

  • method (Literal['pearson', 'spearman']) – Correlation measure. Spearman uses rank correlation and is robust to monotonic non-linear relationships between descriptors.

Variables:

  • correlation_matrix (ndarray of shape (n_features, n_features)) – The fitted descriptor-descriptor correlation matrix.

  • support (ndarray of bool of shape (n_features,)) – True for descriptors kept.

  • n_features_in (int) – Number of descriptors seen during fit.

  • constant (ndarray of bool of shape (n_features,)) – True for descriptors with zero variance, whose correlation is undefined. These are dropped before the walk rather than as a side effect of comparing against NaN.

Examples

>>> import numpy as np
>>> from qsarkit.feature_selection import CorrelationFilter
>>> rng = np.random.RandomState(0)
>>> x0 = rng.normal(size=200)
>>> X = np.column_stack([x0, x0 + rng.normal(scale=1e-3, size=200), rng.normal(size=200)])
>>> y = x0 * 2.0
>>> cf = CorrelationFilter(threshold=0.95).fit(X, y)
>>> cf.support_.tolist()
[True, False, True]

References

correlation_matrix_: ndarray[tuple[Any, ...], dtype[float64]]
support_: ndarray[tuple[Any, ...], dtype[bool]]
n_features_in_: int
fit(X, y=None)[source]

Learn the correlation matrix and resulting support mask.

Parameters:
Returns:

The fitted selector.

Return type:

CorrelationFilter

class qsarkit.feature_selection.MutualInformationSelector(task='regression', k=None, percentile=None, random_state=None)[source]

Bases: SelectorMixin, BaseEstimator

Keep the descriptors most informative about the target, by mutual information.

Thin wrapper around sklearn.feature_selection.mutual_info_regression / mutual_info_classif, which estimate mutual information between each descriptor and the target using a k-nearest-neighbour entropy estimator. Unlike Pearson correlation, mutual information captures non-linear dependencies, which matters for QSAR descriptors that often relate to activity non-monotonically.

Parameters:

  • task (Literal['regression', 'classification']) – Whether y is continuous or categorical.

  • k (Optional[int]) – Keep the top-k scoring descriptors. Mutually exclusive with percentile.

  • percentile (Optional[float]) – Keep descriptors scoring at or above this percentile (0-100) of the score distribution. Mutually exclusive with k. If neither k nor percentile is given, defaults to 50 (keep the top half).

  • random_state (Optional[int]) – Seed forwarded to the mutual-information estimator’s internal noise injection (used to break ties in the nearest-neighbour distances).

Variables:

  • scores (ndarray of shape (n_features,)) – Raw mutual-information score per descriptor.

  • support (ndarray of bool of shape (n_features,)) – True for descriptors kept.

  • n_features_in (int) – Number of descriptors seen during fit.

Examples

>>> import numpy as np
>>> from qsarkit.feature_selection import MutualInformationSelector
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(300, 3))
>>> y = X[:, 0] ** 2
>>> sel = MutualInformationSelector(k=1, random_state=0).fit(X, y)
>>> sel.support_.tolist()
[True, False, False]

References

scores_: ndarray[tuple[Any, ...], dtype[float64]]
support_: ndarray[tuple[Any, ...], dtype[bool]]
n_features_in_: int
fit(X, y=None)[source]

Score every descriptor by mutual information with y.

Parameters:
Returns:

The fitted selector.

Return type:

MutualInformationSelector

Raises:

ValueError – If both k and percentile are given, or task is invalid.

class qsarkit.feature_selection.RFESelector(estimator=None, task='regression', n_features_to_select=None, step=1, random_state=None)[source]

Bases: SelectorMixin, BaseEstimator

Recursive feature elimination, with a QSAR-sane default estimator.

Thin wrapper around sklearn.feature_selection.RFE: repeatedly fits estimator, ranks descriptors by its feature-importance (or coefficient) attribute, and prunes the weakest until n_features_to_select remain. When no estimator is given, a random-forest of 200 trees is used — a robust, hyperparameter-light default that handles the non-linear structure typical of molecular descriptors.

Parameters:

  • estimator (Optional[BaseEstimator]) – Estimator exposing coef_ or feature_importances_ after fitting. Defaults to a RandomForestRegressor/ RandomForestClassifier with 200 trees, chosen by task.

  • task (Literal['regression', 'classification']) – Selects the default estimator when estimator is None. Ignored if estimator is given.

  • n_features_to_select (Union[int, float, None]) – Number (or, if a float in (0, 1), fraction) of descriptors to keep. Defaults to sklearn’s RFE default (half the features).

  • step (Union[int, float]) – Number (or fraction) of descriptors pruned at each iteration.

  • random_state (Optional[int]) – Forwarded to the default random-forest estimator.

Variables:

  • support (ndarray of bool of shape (n_features,)) – True for descriptors kept.

  • ranking (ndarray of int of shape (n_features,)) – Selection ranking; selected descriptors are ranked 1.

  • n_features_in (int) – Number of descriptors seen during fit.

Examples

>>> import numpy as np
>>> from qsarkit.feature_selection import RFESelector
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(100, 4))
>>> y = X[:, 0] * 3.0
>>> sel = RFESelector(n_features_to_select=1, random_state=0).fit(X, y)
>>> int(sel.ranking_[0])
1

References

  • Guyon, I., Weston, J., Barnhill, S. & Vapnik, V. (2002). “Gene Selection for Cancer Classification Using Support Vector Machines.” Machine Learning, 46(1-3), 389-422. https://doi.org/10.1023/A:1012487302797

support_: ndarray[tuple[Any, ...], dtype[bool]]
ranking_: ndarray[tuple[Any, ...], dtype[int64]]
n_features_in_: int
fit(X, y=None)[source]

Run recursive feature elimination.

Parameters:
Returns:

The fitted selector.

Return type:

RFESelector

class qsarkit.feature_selection.BorutaSelector(estimator=None, task='regression', n_iterations=100, alpha=0.05, include_tentative=False, random_state=None)[source]

Bases: SelectorMixin, BaseEstimator

Boruta all-relevant feature selection.

Boruta answers a different question than most feature selectors: not “which subset gives the best predictive score” but “which descriptors carry any signal at all.” It does so by comparing every real descriptor against “shadow” copies of every descriptor — the same values, independently permuted across samples, which by construction carry no relationship to y. A tree ensemble is fit on the concatenation of real and shadow descriptors; a real descriptor that beats the best shadow descriptor’s importance is a “hit” for that iteration. Repeated over many iterations, the hit count of a genuinely important descriptor should exceed that of an irrelevant one (which is statistically indistinguishable from a shadow feature) — formalized with a two-sided binomial test against the null hits ~ Binomial(n_iterations, 0.5), Bonferroni-corrected across descriptors.

Parameters:

  • estimator (Optional[BaseEstimator]) – Estimator exposing feature_importances_ after fitting (e.g. a tree ensemble). Defaults to a RandomForestRegressor/ RandomForestClassifier with 200 trees, chosen by task.

  • task (Literal['regression', 'classification']) – Selects the default estimator when estimator is None. Ignored if estimator is given.

  • n_iterations (int) – Number of shadow-permutation iterations.

  • alpha (float) – Family-wise significance level; Bonferroni-corrected to alpha / n_features per descriptor.

  • include_tentative (bool) – If True, descriptors that could not be statistically resolved within the iteration budget (“Tentative”) are also selected.

  • random_state (Optional[int]) – Seed for the shadow-feature permutations.

Variables:

  • hits (ndarray of int of shape (n_features,)) – Number of iterations in which each real descriptor beat the best shadow descriptor.

  • decision (ndarray of str of shape (n_features,)) – Per-descriptor verdict: "Confirmed", "Tentative" or "Rejected".

  • support (ndarray of bool of shape (n_features,)) – True for "Confirmed" descriptors (plus "Tentative" ones too when include_tentative=True).

  • n_features_in (int) – Number of descriptors seen during fit.

Examples

>>> import numpy as np
>>> from qsarkit.feature_selection import BorutaSelector
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(200, 3))
>>> y = X[:, 0] * 5.0 + rng.normal(scale=0.1, size=200)
>>> sel = BorutaSelector(n_iterations=20, random_state=0).fit(X, y)
>>> bool(sel.support_[0])
True

References

hits_: ndarray[tuple[Any, ...], dtype[int64]]
decision_: ndarray[tuple[Any, ...], dtype[str_]]
support_: ndarray[tuple[Any, ...], dtype[bool]]
n_features_in_: int
fit(X, y=None)[source]

Run the Boruta shadow-permutation procedure.

Parameters:
Returns:

The fitted selector.

Return type:

BorutaSelector

Raises:

ValueError – If task is invalid, or if estimator does not expose feature_importances_ after fitting.

References

  • Kursa, M. B. & Rudnicki, W. R. (2010). “Feature Selection with the Boruta Package.” J. Stat. Softw., 36(11), 1-13. doi:10.18637/jss.v036.i11

  • Guyon, I. et al. (2002). “Gene Selection for Cancer Classification using Support Vector Machines.” Mach. Learn., 46, 389-422. doi:10.1023/A:1012487302797

  • Kraskov, A., Stogbauer, H. & Grassberger, P. (2004). “Estimating Mutual Information.” Phys. Rev. E, 69, 066138. doi:10.1103/PhysRevE.69.066138

  • Ambroise, C. & McLachlan, G. J. (2002). “Selection Bias in Gene Extraction on the Basis of Microarray Gene-Expression Data.” PNAS, 99(10), 6562-6566. doi:10.1073/pnas.102102699