Data quality

Dataset curation following Fourches, Muratov and Tropsha: duplicate detection, activity outliers, structure validation, and a pipeline that reports what it removed and why.

A QSAR model is bounded by the quality of its activity data. Curation is not tidying — it is the difference between a model of the chemistry and a model of the database’s accumulated errors.

Duplicates

Duplicates are detected by structural identity, not by string equality: the same compound registered as a salt, a tautomer or with different stereo annotation is one record.

>>> from qsarkit.data_quality import DuplicateDetector
>>> groups = DuplicateDetector().find_duplicates(
...     [demo_mols[0], demo_mols[0], demo_mols[1]])
>>> len(groups), groups[0].indices
(1, [0, 1])

With activities attached, a group also reports whether its measurements agree. A duplicate pair two log units apart is not a duplicate to be merged — it is a data problem to be investigated:

>>> groups = DuplicateDetector(activity_tolerance=0.5).find_duplicates(
...     [demo_mols[0], demo_mols[0]], [5.1, 8.4])
>>> round(groups[0].spread, 2), groups[0].consistent
(3.3, False)

Outliers

>>> import numpy as np
>>> from qsarkit.data_quality import ActivityOutlierDetector
>>> ActivityOutlierDetector().detect(np.array([5.0, 5.1, 5.2, 9.9])).tolist()
[False, False, False, True]

The default is the modified z-score, which uses the median and MAD rather than the mean and standard deviation. That matters here: a single extreme value inflates the standard deviation enough to mask itself, so a plain z-score is least reliable exactly when you need it.

Structure validation

>>> from rdkit import Chem
>>> from qsarkit.data_quality import StructureValidator
>>> mols = [Chem.MolFromSmiles(s) for s in ("CCO", "[Na+].[Cl-]", "O")]
>>> sorted({issue.code for issue in StructureValidator().validate(mols)})
['inorganic', 'mixture', 'no_carbon', 'too_small']

The charged check looks at net charge, so a zwitterion is not mistaken for a record that escaped neutralization:

>>> validator = StructureValidator()
>>> [i.code for i in validator.validate([Chem.MolFromSmiles("[NH3+]CC(=O)[O-]")])]
[]
>>> [i.code for i in validator.validate([Chem.MolFromSmiles("CC(=O)[O-]")])]
['charged']

The whole pipeline

>>> from qsarkit.data_quality import DataCurationPipeline
>>> mols_in = demo_mols[:6] + [demo_mols[0]]      # one deliberate duplicate
>>> activities = list(DEMO_Y[:6]) + [5.1]
>>> mols, y, report = DataCurationPipeline().run(mols_in, activities)
>>> len(mols), len(y)
(6, 6)
>>> print(report.summary())
Curation: 7 -> 6 records (85.7% retained)
  standardize                 7 -> 7      (0 removed)
  validate                    7 -> 7      (0 removed)
  deduplicate                 7 -> 6      (1 removed)

The report is the audit trail OECD Principle 2 asks for — every removal, attributed to the stage that made it:

>>> report.n_input, report.n_output, round(report.retention, 3)
(7, 6, 0.857)
>>> frame = report.to_dataframe()
>>> len(frame) >= 1
True

Unit checking

>>> from qsarkit.data_quality import check_activity_units
>>> report = check_activity_units([5.1, 6.2, 7.3], endpoint="IC50")
>>> report["looks_logarithmic"], report["warnings"]
(True, [])

A raw nanomolar column spanning six decades is caught:

>>> report = check_activity_units([1.0, 10.0, 1000.0, 1e6], unit="nM")
>>> report["looks_logarithmic"], round(report["log_range"], 1)
(False, 6.0)
>>> print(report["warnings"][0])
Values span 6.0 orders of magnitude, which suggests a raw concentration scale. Convert to pActivity (-log10 molar) before modeling.

So is the subtler case: molar values spanning little numeric range. They look narrow, but a pActivity of 1e-9 would mean an IC50 near 1 M:

>>> report = check_activity_units([1e-9, 5e-8], endpoint="IC50")
>>> report["looks_logarithmic"]
False

Mixing molar and p-scale values in one column produces a model that learns nothing, and nothing in the numbers themselves announces the mistake — which is why this check exists.

API

Dataset curation: duplicates, structural validity and activity sanity.

Curation changes QSAR model performance more than the choice of algorithm does. This module implements the protocol of Fourches, Muratov and Tropsha, and records what it removed so the result is auditable.

Examples

>>> from rdkit import Chem
>>> from qsarkit.data_quality import DataCurationPipeline
>>> mols = [Chem.MolFromSmiles(s) for s in ("CCO", "OCC", "[Na+].[Cl-]")]
>>> curated, y, report = DataCurationPipeline().run(mols, [5.0, 5.2, 1.0])
>>> report.n_output < report.n_input
True

References

  • Fourches, D., Muratov, E. & Tropsha, A. (2010). “Trust, But Verify: On the Importance of Chemical Structure Curation in Cheminformatics and QSAR Modeling Research.” J. Chem. Inf. Model., 50(7), 1189-1204. https://doi.org/10.1021/ci100176x

  • Fourches, D., Muratov, E. & Tropsha, A. (2016). “Trust, but Verify II.” J. Chem. Inf. Model., 56(7), 1243-1252. https://doi.org/10.1021/acs.jcim.6b00129

  • Tropsha, A. (2010). “Best Practices for QSAR Model Development, Validation, and Exploitation.” Mol. Inform., 29(6-7), 476-488. https://doi.org/10.1002/minf.201000061

class qsarkit.data_quality.DuplicateDetector(level='inchikey', activity_tolerance=1.0)[source]

Bases: object

Find duplicate structures and check whether their activities agree.

Duplicates are endemic in public activity data: the same compound is re-measured across papers, deposited under different salt forms, or drawn with different tautomers. Removing them blindly discards replicate information; keeping them inflates cross-validated performance, because the same structure lands in both the training and test folds.

The useful question is not “are these duplicates?” but “do the duplicates agree?” A pair reported as 5 nM and 50 uM is not a replicate — it is a data error, an assay difference, or a wrong structure, and averaging the two produces a value that describes neither.

Parameters:

  • level (Literal['inchikey', 'smiles', 'connectivity', 'scaffold']) – What counts as the same structure. "connectivity" ignores stereochemistry and charge; "scaffold" collapses whole Bemis-Murcko series and is a deliberately blunt instrument.

  • activity_tolerance (float) – Maximum activity spread, in the units supplied, for a duplicate group to be called consistent. On a log scale 1.0 means a ten-fold disagreement.

Examples

>>> from rdkit import Chem
>>> mols = [Chem.MolFromSmiles(s) for s in ("CCO", "OCC", "c1ccccc1")]
>>> groups = DuplicateDetector().find_duplicates(mols, [5.0, 5.2, 7.0])
>>> len(groups)
1
>>> groups[0].indices
[0, 1]

References

  • Fourches, D., Muratov, E. & Tropsha, A. (2010). “Trust, But Verify: On the Importance of Chemical Structure Curation in Cheminformatics and QSAR Modeling Research.” J. Chem. Inf. Model., 50(7), 1189-1204. https://doi.org/10.1021/ci100176x

  • Fourches, D., Muratov, E. & Tropsha, A. (2016). “Trust, but Verify II: A Practical Guide to Chemogenomics Data Curation.” J. Chem. Inf. Model., 56(7), 1243-1252. https://doi.org/10.1021/acs.jcim.6b00129

  • Heller, S. R. et al. (2015). “InChI, the IUPAC International Chemical Identifier.” J. Cheminform., 7, 23. https://doi.org/10.1186/s13321-015-0068-4

  • Kramer, C. et al. (2012). “The Experimental Uncertainty of Heterogeneous Public Ki Data.” J. Med. Chem., 55(11), 5165-5173. https://doi.org/10.1021/jm300131x

find_duplicates(mols, activities=None)[source]

Group records that share a structure.

Parameters:

  • mols (Sequence[Any]) – Molecules to check. None entries are skipped.

  • activities (Union[Buffer, _SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], complex, bytes, str, _NestedSequence[complex | bytes | str], None]) – Parallel activity values. When given, each group reports its spread and whether it is consistent.

Returns:

Only groups with two or more members, ordered by first appearance.

Return type:

List[DuplicateGroup]

report(mols, activities=None)[source]

Summarize the duplicate content of a dataset.

Parameters:
Returns:

n_records, n_unique, n_duplicate_groups, n_duplicate_records (records that are not the first of their group), duplicate_fraction, n_inconsistent (groups whose activities disagree beyond tolerance) and max_spread.

Return type:

Dict[str, Any]

to_dataframe(groups)[source]

Render duplicate groups as a table.

Parameters:

groups (Sequence[DuplicateGroup])

Returns:

Columns key, n_records, indices, activities, spread, consistent.

Return type:

DataFrame

class qsarkit.data_quality.DuplicateGroup(key, indices, activities=<factory>, spread=0.0, consistent=True)[source]

Bases: object

One set of records judged to be the same structure.

Variables:

  • key (str) – The identity key the members share.

  • indices (list of int) – Positions of the members in the input sequence.

  • activities (list of float) – Their activity values, where supplied.

  • spread (float) – max - min of the activities; 0.0 when fewer than two are known.

  • consistent (bool) – Whether spread is within the detector’s tolerance.

key: str
indices: List[int]
activities: List[float]
spread: float
consistent: bool
qsarkit.data_quality.merge_replicates(mols, activities, level='inchikey', method='median', max_spread=1.0, log_scale=True)[source]

Collapse replicate measurements into one value per structure.

Parameters:

  • mols (Sequence[Any]) – Molecules, one per record.

  • activities (Union[Buffer, _SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], complex, bytes, str, _NestedSequence[complex | bytes | str]]) – Activity values, one per record.

  • level (Literal['inchikey', 'smiles', 'connectivity', 'scaffold']) – Identity level used to group replicates.

  • method (Literal['mean', 'median', 'geometric_mean', 'min', 'max']) – How to combine agreeing replicates. The median is the safer default: activity data carries occasional order-of-magnitude transcription errors, and one of those moves a mean far more than it moves a median.

  • max_spread (Optional[float]) – Discard groups disagreeing by more than this. None keeps all.

  • log_scale (bool) – Whether activities are already logarithmic (pIC50 and the like). geometric_mean requires linear, positive values, so it is refused when this is True — averaging log values arithmetically already is the geometric mean.

Return type:

Tuple[List[Any], ndarray[tuple[Any, ...], dtype[double]], Dict[str, Any]]

Returns:

  • mols (list of Mol) – One representative molecule per retained group, in order of first appearance.

  • activities (ndarray) – The merged values.

  • report (dict) – n_input, n_output, n_merged, n_discarded and discarded_keys.

Examples

>>> from rdkit import Chem
>>> mols = [Chem.MolFromSmiles(s) for s in ("CCO", "OCC", "c1ccccc1")]
>>> merged, y, report = merge_replicates(mols, [5.0, 5.4, 7.0])
>>> len(merged), report["n_merged"]
(2, 1)

References

class qsarkit.data_quality.StructureValidator(allow_inorganic=False, allow_mixtures=False, allow_isotopes=False, min_heavy_atoms=3, max_heavy_atoms=150, require_carbon=True)[source]

Bases: object

Flag records that are not usable molecules for QSAR modeling.

Runs after standardization and before modeling. Each check corresponds to a category of record that routinely appears in public datasets and quietly degrades a model: unparseable structures, mixtures whose activity cannot be attributed to one component, inorganics and organometallics outside the applicability of organic descriptors, isotopically labelled tracers, and molecules far outside the size range the descriptors were designed for.

Parameters:

  • allow_inorganic (bool) – Keep molecules containing elements outside common organic chemistry.

  • allow_mixtures (bool) – Keep multi-component records. These are usually salts that survived desalting, or genuine mixtures whose activity belongs to no single structure.

  • allow_isotopes (bool) – Keep isotopically labelled molecules.

  • min_heavy_atoms (int) – Smallest acceptable molecule. Fragments below this carry almost no descriptor signal.

  • max_heavy_atoms (int) – Largest acceptable molecule, excluding peptides and polymers that most descriptor sets were never calibrated on.

  • require_carbon (bool) – Require at least one carbon atom.

Examples

>>> from rdkit import Chem
>>> mols = [Chem.MolFromSmiles(s) for s in ("CCO", "[Na+].[Cl-]", "O")]
>>> validator = StructureValidator()
>>> issues = validator.validate(mols)
>>> sorted({i.code for i in issues})
['inorganic', 'mixture', 'no_carbon', 'too_small']

Ethanol passes; sodium chloride is a mixture of inorganic ions and water is too small to carry descriptor signal.

The charged check looks at net charge, so a zwitterion is not mistaken for a record that escaped neutralization:

>>> glycine = Chem.MolFromSmiles("[NH3+]CC(=O)[O-]")
>>> [i.code for i in validator.validate([glycine])]
[]
>>> acetate = Chem.MolFromSmiles("CC(=O)[O-]")
>>> [i.code for i in validator.validate([acetate])]
['charged']

References

validate(mols)[source]

Check every molecule and return all issues found.

Parameters:

mols (Sequence[Any])

Returns:

Ordered by record index; a record may raise several issues.

Return type:

List[ValidationIssue]

valid_mask(mols)[source]

Boolean mask of records with no fatal issue.

Parameters:

mols (Sequence[Any])

Return type:

ndarray[tuple[Any, ...], dtype[bool]]

to_dataframe(issues)[source]

Render issues as a table.

Parameters:

issues (Sequence[ValidationIssue])

Returns:

Columns index, code, message, fatal.

Return type:

DataFrame

class qsarkit.data_quality.ValidationIssue(index, code, message, fatal=True)[source]

Bases: object

One problem found with one record.

Variables:

  • index (int) – Position of the offending record.

  • code (str) – Machine-readable issue code, e.g. "inorganic".

  • message (str) – Human-readable explanation.

  • fatal (bool) – Whether the record should be dropped rather than merely flagged.

index: int
code: str
message: str
fatal: bool
class qsarkit.data_quality.ActivityOutlierDetector(method='modified_zscore', threshold=3.5, n_neighbors=5)[source]

Bases: object

Flag activity values that look like errors rather than chemistry.

Distinguishes two different things that both get called “outliers”:

  • Distributional outliers — values far from the rest of the dataset, found by z-score, modified z-score or the IQR rule.

  • Structure-activity outliers — values far from what the compound’s nearest structural neighbours would predict. These are the interesting ones, but note that a genuine activity cliff looks exactly like a data error from this angle, so use qsarkit.sar to tell them apart before deleting anything.

Parameters:

  • method (Literal['zscore', 'modified_zscore', 'iqr', 'neighbor']) – Detection rule. The modified z-score uses the median and MAD, so a few extreme values cannot inflate the scale and hide themselves — which is exactly what happens with a plain z-score.

  • threshold (float) – Cutoff. 3.5 is the conventional modified-z-score limit; use ~3 for "zscore" and 1.5 for "iqr".

  • n_neighbors (int) – Neighbours used by method="neighbor".

Examples

>>> import numpy as np
>>> y = np.concatenate([np.full(20, 5.0), [50.0]])
>>> detector = ActivityOutlierDetector()
>>> bool(detector.detect(y)[-1])
True

References

  • Iglewicz, B. & Hoaglin, D. C. (1993). “How to Detect and Handle Outliers.” ASQC Quality Press. (modified z-score, MAD-based)

  • Tukey, J. W. (1977). “Exploratory Data Analysis.” Addison-Wesley. (the IQR rule)

  • Fourches, D., Muratov, E. & Tropsha, A. (2010). J. Chem. Inf. Model., 50(7), 1189-1204. https://doi.org/10.1021/ci100176x

  • Maggiora, G. M. (2006). “On Outliers and Activity Cliffs.” J. Chem. Inf. Model., 46(4), 1535. https://doi.org/10.1021/ci060117s

scores(activities, mols=None)[source]

Per-record outlier score (larger = more anomalous).

Parameters:
Return type:

ndarray[tuple[Any, ...], dtype[double]]

detect(activities, mols=None)[source]

Boolean mask of records flagged as outliers.

Parameters:
Return type:

ndarray[tuple[Any, ...], dtype[bool]]

qsarkit.data_quality.check_activity_units(activities, unit=None, endpoint=None)[source]

Sanity-check an activity column before modeling.

Catches the most common and most damaging data error in QSAR: a column that mixes units, or is on a linear concentration scale when the modeling assumes a logarithmic one. A dataset spanning six orders of magnitude in raw nM will be dominated by its largest values, and the resulting model is fitted almost entirely to the inactives.

Parameters:
Returns:

n, n_missing, n_non_positive, min, max, median, log_range (orders of magnitude spanned), looks_logarithmic and warnings (a list of plain-language problems found).

Return type:

Dict[str, Any]

Examples

A raw nanomolar column spanning six decades is flagged:

>>> report = check_activity_units([1.0, 10.0, 1000.0, 1e6], unit="nM")
>>> report["looks_logarithmic"]
False
>>> bool(report["warnings"])
True

A pActivity column is recognised and passes clean:

>>> report = check_activity_units([5.1, 6.2, 7.3], endpoint="IC50")
>>> report["looks_logarithmic"], report["warnings"]
(True, [])

Molar values spanning little range are not mistaken for a p-scale column, even though their numeric range is narrow – a pActivity of 1e-9 would mean an IC50 near 1 M:

>>> report = check_activity_units([1e-9, 5e-8], endpoint="IC50")
>>> report["looks_logarithmic"]
False
>>> report["warnings"][0].startswith("All values are below 1")
True

References

class qsarkit.data_quality.DataCurationPipeline(standardize=True, validate_structures=True, remove_duplicates=True, remove_outliers=False, duplicate_level='inchikey', merge_method='median', max_spread=1.0, outlier_method='modified_zscore', outlier_threshold=3.5, validator=None)[source]

Bases: object

Standardize, validate, de-duplicate and screen a QSAR dataset.

Runs the curation protocol of Fourches, Muratov and Tropsha in the order that matters: standardization first (so that duplicate detection sees comparable structures), then structural validation, then replicate merging, then activity outliers. Running de-duplication before standardization is the classic mistake — the same compound stored as a salt and as a free base will not be recognised as a duplicate, and both copies survive into the model.

Every stage records what it removed, so the output is auditable.

Parameters:

Examples

>>> from rdkit import Chem
>>> mols = [Chem.MolFromSmiles(s) for s in
...         ("CC(=O)Oc1ccccc1C(=O)[O-].[Na+]", "CCO", "OCC", "[Na+].[Cl-]")]
>>> curated, y, report = DataCurationPipeline().run(mols, [5.0, 6.0, 6.2, 1.0])
>>> report.n_input
4
>>> report.n_output < report.n_input
True

References

run(mols, activities=None, unit=None, endpoint=None)[source]

Curate a dataset.

Parameters:
Return type:

Tuple[List[Any], Optional[ndarray[tuple[Any, ...], dtype[double]]], CurationReport]

Returns:

  • mols (list of Mol) – The curated molecules.

  • activities (ndarray or None) – The curated activities.

  • report (CurationReport) – What happened at each stage.

class qsarkit.data_quality.CurationReport(n_input=0, n_output=0, stages=<factory>, removed=<factory>, activity_check=<factory>, warnings=<factory>)[source]

Bases: object

What curation did, and why.

A curated dataset is only trustworthy if the curation is auditable — OECD principle 1 asks for a defined endpoint and a documented dataset, and “we cleaned it” is not documentation.

Variables:

  • n_input (int) – Records supplied.

  • n_output (int) – Records surviving.

  • stages (list of dict) – One entry per stage: its name, the counts before and after, and the indices it removed.

  • removed (dict) – Original index -> reason it was removed.

  • activity_check (dict) – Output of check_activity_units().

  • warnings (list of str) – Problems worth a human’s attention.

n_input: int
n_output: int
stages: List[Dict[str, Any]]
removed: Dict[int, str]
activity_check: Dict[str, Any]
warnings: List[str]
property n_removed: int

Number of records removed.

property retention: float

Fraction of records surviving curation.

to_dataframe()[source]

Per-stage summary table.

Returns:

Columns stage, n_before, n_after, n_removed.

Return type:

DataFrame

summary()[source]

Human-readable report.

Return type:

str

References

  • Fourches, D., Muratov, E. & Tropsha, A. (2010). “Trust, But Verify: On the Importance of Chemical Structure Curation in Cheminformatics and QSAR Modeling Research.” J. Chem. Inf. Model., 50(7), 1189-1204. doi:10.1021/ci100176x

  • Fourches, D., Muratov, E. & Tropsha, A. (2016). “Trust, but Verify II.” J. Chem. Inf. Model., 56(7), 1243-1252. doi:10.1021/acs.jcim.6b00129

  • Kalliokoski, T. et al. (2013). “Comparability of Mixed IC50 Data.” PLoS ONE, 8(4), e61007. doi:10.1371/journal.pone.0061007