The functional pipe API

A second way to write the workflow, reading in the order the work happens — in the spirit of R’s %>%. It reaches the whole package, not just curation: featurize() takes any transformer and fit() any estimator, so the pipe is a way of writing the workflow rather than a reimplementation of part of it.

molecules() accepts RDKit molecules, SMILES and InChI in any mixture:

>>> from qsarkit.functional import (
...     drop_invalid, fingerprint, fit, molecules, remove_duplicates,
...     split, standardize)
>>> mols, y = (
...     molecules(DEMO_SMILES, DEMO_Y)
...     >> standardize()
...     >> drop_invalid()
...     >> remove_duplicates(agg="mean")
... )
>>> len(mols), len(y)
(24, 24)

A pipeline crosses from molecules into a feature matrix at featurize(), and can run all the way to a fitted model:

>>> train, test = (
...     molecules(DEMO_SMILES, DEMO_Y)
...     >> standardize()
...     >> drop_invalid()
...     >> fingerprint("morgan", n_bits=512)
...     >> split("scaffold", test_size=0.25)
... )
>>> model = train >> fit("rf", random_state=0)
>>> model.predict(test.X).shape
(6,)

Drawing it is the quickest way to confirm the stages are in the order you meant:

>>> pipe = standardize() >> drop_invalid() >> fingerprint() >> fit("rf")
>>> type(pipe.plot()).__name__
'Figure'
>>> print(pipe.to_dot().splitlines()[0])
digraph qsarkit_pipeline {

pipe.render("workflow.pdf") writes PNG, PDF or SVG.

molecules() returns a MoleculeSet carrying the molecules, the optional labels, and a provenance log. Each step consumes one and returns a new one — nothing is mutated in place — and every step keeps y index-aligned with the molecules. Dropping a molecule drops its label with it, which is the bookkeeping that makes most hand-written curation scripts subtly wrong.

The result unpacks straight into X, y.

Warning

Use >>, not >.

> cannot work as a pipe operator in Python. The language parses a > b > c as the chained comparison (a > b) and (b > c), so a >-based pipe would evaluate the first stage, throw the result away, and hand back the comparison of the last two stages. There is no way to intercept that from __gt__, and the failure is silent — you get a result, just not the one you asked for.

>> and | are ordinary left-associative binary operators, so they chain correctly. Piping with > raises a TypeError explaining this rather than misbehaving quietly.

Inspecting the pipeline

>>> from qsarkit.functional import desalt, drop_invalid, molecules
>>> ms = molecules(DEMO_SMILES[:6], DEMO_Y[:6]) >> desalt() >> drop_invalid()
>>> ms
<MoleculeSet 6 molecules, y shape (6,), 3 steps>
>>> ms.history
['molecules(n=6)', 'desalt()', 'drop_invalid()']
>>> ms.smiles[0]                 # canonical SMILES, None where invalid
'O=C(O)c1ccccc1'
>>> list(ms.to_frame().columns)  # DataFrame with smiles + y
['smiles', 'y']
>>> len(ms)
6

history records every step with its arguments, so a curated dataset carries its own provenance — which is what OECD principle 1 asks you to document.

Reusable pipelines

Steps compose with each other, so a curation protocol can be defined once and applied to several datasets:

>>> from qsarkit.functional import remove_duplicates, standardize
>>> curate = standardize() >> drop_invalid() >> remove_duplicates(max_spread=1.0)
>>> train = molecules(DEMO_SMILES[:12], DEMO_Y[:12]) >> curate
>>> test = molecules(DEMO_SMILES[12:], DEMO_Y[12:]) >> curate
>>> len(train), len(test)
(12, 12)

The composed protocol is itself a step, and prints as one:

>>> curate
<Pipeline standardize() >> drop_invalid() >> remove_duplicates(max_spread=1.0)>

pipeline() does the same thing from a list, for when the steps are assembled programmatically.

Dual-mode steps

Every step works two ways, so the same function serves the pipe API and ordinary imperative code:

>>> from rdkit import Chem
>>> mols, y = molecules(["CC(=O)[O-].[Na+]"]) >> desalt()      # deferred
>>> Chem.MolToSmiles(mols[0])
'CC(=O)[O-]'
>>> mols, y = desalt([Chem.MolFromSmiles("CC(=O)[O-].[Na+]")])  # immediate
>>> Chem.MolToSmiles(mols[0])
'CC(=O)[O-]'

Calling a step with no molecules returns a deferred Step; calling it with molecules runs it now.

Available steps

Step

Does

standardize()

The full standardization pipeline

desalt()

Keep the largest organic fragment

neutralize()

Neutralize charges where a neutral form exists

canonicalize_tautomers()

Map to the canonical tautomer

deglycate()

Remove sugars, keeping the aglycone

remove_protecting_groups()

Strip Boc, Cbz, Fmoc, linkers, tags

drop_invalid()

Drop None entries and their labels

remove_duplicates()

Collapse duplicates, aggregating labels

balance()

Balance classes by under/oversampling

keep_if() / drop_if()

Filter on an arbitrary predicate

filter_by_property()

Filter on MW, logP, heavy atoms, rotatable bonds

to_pactivity()

Convert concentrations to -log10 molar

sample() / shuffle()

Subsample or reorder

apply()

Apply an arbitrary per-molecule function

Writing your own step

Decorate a function with the (X, y=None, **params) -> (X, y) signature:

>>> from qsarkit.functional import step
>>> @step
... def keep_heaviest(X, y=None, n=100):
...     from rdkit.Chem import Descriptors
...     order = sorted(range(len(X)), key=lambda i: -Descriptors.MolWt(X[i]))
...     keep = sorted(order[:n])
...     return [X[i] for i in keep], (None if y is None else y[keep])
>>> mols, y = molecules(DEMO_SMILES, DEMO_Y) >> keep_heaviest(n=5)
>>> len(mols), len(y)
(5, 5)

The decorator gives it both modes and the alignment guarantee for free — returning the subset of y alongside the molecules is the whole contract. feature_step() does the same for a step that operates on a feature matrix.

Relationship to the scikit-learn API

The two APIs are alternatives, not layers — both call the same underlying classes in qsarkit.chemistry. Use the pipe API for exploratory curation, where reading order and provenance matter; use sklearn.pipeline.Pipeline when the preprocessing has to be fitted on training data and reapplied at prediction time, since only that form participates in cross-validation and model persistence.

References