Honest QSAR Models Start With the Assay, Not the Algorithm

A QSAR number is a decision about a compound that has not been made yet. The useful part of Optibrium’s webinar is the order that keeps that number honest: curate the assay, choose a regression or a class the noise can support, hold out a test set that is scored once, and look at the plot rather than at $R^2$. Clearance is the property where that order can still be the wrong model, because one measured number is several biological mechanisms.

Tamsin Mansley and Mario Öeren gave the webinar on 18 February 2025 [1]. Mansley is President of Optibrium Inc. and Global Head of Application Science. Öeren is a Principal Scientist there. The demonstration is StarDrop’s Auto-Modeller on a public ChEMBL set. The order does not belong to the software.

The recording is on YouTube: How to build a better QSAR model (opens youtube.com).

What is worth predicting

In a multi-parameter optimization the potency model is one column. Solubility, $\log P$ or $\log D$, permeability, clearance, and an off-target liability are the others, and a compound fails the project by failing one of them. A model is there to remove compounds before they are made, not to replace the assay that will be run on the ones that remain.

Three situations are where Mansley says the model stops being the right tool [1].

Fig. 1 · Three failures

In vivo clearance is the first. The number can move because of hepatic blood flow, plasma protein binding, uptake, several cytochrome P450 isoforms, or a conjugation. A fingerprint model is being asked to learn all of those from the structure of the parent compound. A promiscuous target is the second: if nearly every tested compound is active, there is no inactive class, and a classifier that says “active” for everything looks accurate. The third is distance. A model fit on one series degrades on chemistry it has not seen. That limit is the applicability domain, and Optibrium treats it as one of the checks a model has to pass, not as a footnote [2].

For a cytochrome P450 project those three boxes are easy to confuse. An assay of one isoform, a $K_i$ or an $\mathrm{IC}_{50}$ against CYP3A4 or CYP2D6, is a single biochemical question. Human clearance, or even a microsomal clearance that several enzymes can produce, is not. The webinar’s advice is to notice which label is in the column before trusting the fit. StarDrop’s own ADME module keeps them separate: isoform affinities on one line, a human-liver-microsome clearance category on another [3].

Four pillars

Öeren’s half of the webinar is the build, in an order [1]. Optibrium has also written the same discipline as four pillars: clean data, an appropriate method, validation with the right metrics, and the domain of applicability [2]. The domain is the bar under the other three. A careful split of the wrong chemical space is still the wrong chemical space.

Fig. 2 · Four steps

Curate the column

A smaller clean set beats a larger noisy one. The work is resolving rows that disagree: different units, a $K_i$ and an $\mathrm{IC}_{50}$ in the same column, duplicates with different values, a qualifier (<, >) treated as a point. ChEMBL will hand you the rows. It will not decide which of two conflicting $K_i$ values is the one to fit. The ChEMBL curation note is that sorting, for bioactivity tables in general.

Dataset size chooses the kind of model, not just the confidence interval. A large set can support a global model on fast 2D descriptors or fingerprints, meant to cover many series. A small set, one series, is a local model. If the local set is too small for a statistical fit, Öeren’s alternative is a physics-based descriptor, such as a docking score, rather than a fingerprint asked to invent a trend from twenty compounds [1]. No cutoff for “large” is a law. The question is whether the table contains enough distinct chemistry for the pattern you want to claim.

Match the method to the noise

If the assay is too noisy for a continuous fit, binning into active and inactive can still produce a filter. The filter answers “worth making?”, not “what is the $K_i$?”. That is a different model, and it should be reported as one.

Auto-Modeller fits several methods and keeps the one that wins on the validation set. The webinar discusses decision trees, random forests, and Gaussian processes [1]. The module also offers partial least squares and radial basis functions [4]. The point of trying more than one is that the validation score, not habit, picks the method. A random forest is not the right answer because it was the right answer on the demonstration set.

Split, then do not look

The split they recommend is 70% training, 15% validation, 15% test [1]. Training fits the weights. Validation is where descriptors are dropped and the method is chosen. The test set is scored once, at the end. Using it to pick a model makes the test number another validation number, and the optimism is baked in. The glossary calls that leakage. An 80/10/10 cut is a common default, not this webinar’s, and not a reason to skip the single look.

Fig. 3 · Split, then do not look

On a small table, a 15% test set may be a handful of compounds. The webinar’s alternative is $k$-fold cross-validation, 5-fold or 10-fold, and a consensus of models rather than one fit [1]. Consensus matters because a single fold can flatter a method. The validation note is the next layer: the $R^2$, $Q^2$, and concordance cuts used when a paper or a filing has to defend the model. Those cuts assume the split was honest. They do not create the honesty.

Class balance is part of the split. A screen that is mostly inactive will reward a model that predicts inactive. The remedies named in the webinar are oversampling the rare class, or a loss that weights it, and then a metric that does not ignore the imbalance [1].

$R^2$ is not the plot

Anscombe built four tiny datasets with the same means, the same variances, the same correlation, and the same straight-line fit, $R^2$ about 0.67 [5]. One is a noisy line. One is a curve. One is a line plus an outlier. One is a vertical stack whose entire slope comes from a single point. The regression summary cannot tell them apart. The scatter plot can. Mansley uses that quartet so a QSAR report does not stop at $R^2$ [1].

Fig. 4 · Anscombe's quartet

For regression, report RMSE and MAE next to the plot. RMSE is in the units of the modeled number, so an RMSE of 0.5 on a $\mathrm{p}K_i$ is half a log unit, about threefold in $K_i$, and it is readable. $R^2$ has no units and no picture. The glossary separates the loss the model minimizes from the metric you use to keep it. This is that distinction on an assay table.

For classification, accuracy is the metric that flatters the majority class. Balanced accuracy is the average of the true-positive rate and the true-negative rate:

$$\mathrm{BA} = \frac{\mathrm{TPR} + \mathrm{TNR}}{2}.$$

Cohen’s $\kappa$ subtracts the agreement you would get from the class frequencies alone:

$$\kappa = \frac{p_o - p_e}{1 - p_e}.$$

$p_o$ is the observed agreement. $p_e$ is the agreement expected by chance. A model that calls everything inactive can have a high accuracy and a $\kappa$ near zero. The webinar names both metrics for that reason [1].

The M5 demonstration

The live session uses Optibrium’s public muscarinic acetylcholine receptor $M_5$ example: 129 compounds with ChEMBL $K_i$ values [1], [6]. The raw $K_i$ distribution is skewed. In that worked example the column is micromolar, and the modeled number is

$$\mathrm{p}K_i = 6 - \log_{10}(K_i / \mu\mathrm{M}),$$

which is $-\log_{10}(K_i)$ with $K_i$ in molar. The $\mathrm{p}K_i$ values sit in a roughly bell-shaped range that a continuous model can fit. The same transform is the one a P450 inhibition table usually needs. An $\mathrm{IC}_{50}$ in micromolar becomes a $\mathrm{pIC}_{50}$ by the same shift of 6. Leaving the column in micromolar asks the model to treat a change from 0.01 to 0.1 as a different kind of error from a change from 10 to 100.

Fig. 5 · Ki to pKi

Auto-Modeller then splits the set. In the demonstration the split is a Tanimoto clustering, so near analogues are not free to land in both training and test by chance [1]. Correlated and redundant descriptors are dropped before the fit. On this set a random forest was the method the validation score preferred [1]. That is a result about these 129 compounds. It is not a standing recommendation to reach for a forest.

The Glowing Molecule view is how StarDrop draws the model back onto the structure. Red marks a region that raises the predicted number. Blue marks a region that lowers it [7]. On a $\mathrm{p}K_i$ model, red is higher predicted affinity. On a clearance model, red is higher predicted clearance, which is often the liability you are trying to remove. The color is about the direction of the number, not about whether the change is good.

Fig. 6 · The colour rule

What to take back to a P450 table

Start from the assay identity. One isoform and one endpoint, a $K_i$ or a categorical call, can be a QSAR. A clearance number that several enzymes can produce is a weaker thing to ask of a fingerprint, and the webinar says so directly [1]. If the column is a concentration, model its negative log, and write the unit shift down. Split so that close analogues are not the hidden contents of the test set, tune only on the validation portion, and score the test portion once. Publish the plot, RMSE or MAE, and for a classifier the balanced accuracy or $\kappa$, not accuracy and not $R^2$ alone. Then ask whether the next compound is inside the chemistry the model saw. If it is not, the pillars were followed and the prediction is still outside their reach.

References

1. Mansley T, Öeren M. How to build a better QSAR model. Optibrium webinar, 18 February 2025. optibrium.com/webinars/how-to-build-a-better-qsar-model. Recording: youtube.com/watch?v=TGXPwi-Z5sY.
2. Optibrium. How do I know if Optibrium’s predictive models work? The four pillars named there are clean data, an appropriate method, validation with appropriate metrics, and the domain of applicability. optibrium.com/knowledge-base/do-optibriums-predictive-models-work.
3. Optibrium. ADME QSAR models in StarDrop, including CYP2C9 and CYP2D6 affinities and a human-liver-microsome intrinsic-clearance category. optibrium.com/products/stardrop/modules/adme-qsar.
4. Optibrium. StarDrop Auto-Modeller. Methods listed include Gaussian processes, random forests, radial basis functions, partial least squares, and decision trees. optibrium.com/products/stardrop/modules/auto-modeller.
5. Anscombe FJ. Graphs in statistical analysis. Am Stat 1973;27:17–21. doi:10.1080/00031305.1973.10478966. The four datasets share a linear $R^2$ of about 0.67.
6. Optibrium. Automatic QSAR model building and validation in StarDrop. Worked example: 129 ChEMBL compounds, muscarinic acetylcholine receptor $M_5$, $K_i$ in micromolar converted to $\mathrm{p}K_i$. youtube.com/watch?v=QqEWf1AB2vk.
7. Optibrium. Glowing Molecule in StarDrop. Regions that increase the predicted value are shown in red. Regions that decrease it are shown in blue. optibrium.com/products/stardrop/features/glowing-molecule-in-stardrop. The color rule is also stated in StarDrop’s worked example on guided multi-parameter optimization.