We changed one line and the benchmark score moved 0.21 AUROC

We spent the last few weeks building a benchmark for drug property prediction tools. It went live this week: LEADBOARD, 21 boards, 18,382 held-out compounds, labels we don't hand out.

This post is mostly about two numbers we ran into while building it, because they changed what we thought the thing should be.

The first number

hERG was the board we built first. It's the potassium channel that, when a drug blocks it, gives you a QT interval problem and a dead clinical program. Everybody screens for it early, so there's a lot of public data, which made it a good place to shake out the pipeline.

We pulled ChEMBL 37 (CHEMBL240, IC50, relation =). 12,021 records with a document year attached, 11,972 after unit checks and RDKit parsing, collapsed by InChIKey to 9,788 unique compounds with a median pIC50 each.

Then we split it by year. Train on everything first reported before 2022, test on what showed up after. Morgan fingerprints into LightGBM, library defaults, no tuning. AUROC came back at 0.606.

That felt low, so we did the obvious sanity check and split the same data at random instead. Five seeds:

Same molecules, same fingerprints, same learner, same hyperparameters, same test fraction. AUROC moves 0.211. That's a bigger gap than you'll find between most competing methods in the literature.

The mechanism isn't mysterious. Medicinal chemists don't generate molecules independently. Somebody finds a scaffold and then makes forty analogues of it, and those forty sit on top of each other in structure space. Split at random and the cluster gets sprayed across both sides of the line, so your test set is full of near-twins of things the model already saw. Nearest-neighbour lookup will carry you a long way under those conditions.

But that isn't how anyone uses a prediction tool. You use it to rank compounds nobody has assayed yet, and you have no promise that anything similar is in the training data. A time split reproduces that honestly, and it has a nice property: you can't look the answer up, because when the model was trained the answer hadn't been published. Structural overlap between train and test is zero by construction.

One caveat we want to state clearly, because it would be easy to over-read this table. We compared time split against random split, on our data. We are not saying published benchmarks use random splits; plenty use scaffold splits, which land somewhere in between. The claim we'll defend is narrower: on identical data, the split choice moves AUROC by 0.21, so a leaderboard number quoted without its split isn't comparable to anything.

Every board on LEADBOARD uses a time or scaffold split. We don't open random ones.

The second number

The other thing we hadn't taken seriously enough was how imprecise the labels are.

Two labs measuring the same compound don't get the same answer. Different cell line, different buffer, different day. You can actually quantify this, because ChEMBL tracks which publication every value came from, so you can find compounds that appear in two or more papers and look at how far apart they land.

We only use pairs that cross documents. Repeats inside a single paper are the same batch and the same hands, and they make the spread look far smaller than it is.

hERG again:

Cross-document pairs      5,185   (857 compounds appearing in 2+ papers)

median |difference|       0.148 log
mean   |difference|       0.475 log
75th percentile           0.669 log
90th percentile           1.338 log
standard deviation        0.736 log

single-measurement SD     0.421 log     (mean|diff| / 1.128)

Look at the 90th percentile. One compound in ten differs by more than 20-fold between two published measurements.

We call 0.421 the board's noise floor and print it next to every score, because it tells you how to read the ranking. If first and second place are 0.02 apart and the floor is 0.42, that ordering is a coin flip wearing a rank badge.

It also tells you something about the test set that we'd rather say out loud than bury. 696 of the 1,338 hERG test compounds, 52% of them, sit within one noise floor of the pIC50 = 5.0 classification cutoff. Half the test set could change class if you measured it again. So this board reports regression as the primary metric and classification as secondary, and the board page says why.

What this looks like across all the boards

Divide the best untrained baseline's MAE by the noise floor and you get one number per board: how far the field is from the ceiling.

Median 1.45. That chart is the first thing on the front page of the Space, with a line at 1.0.

Two boards are already under the line, and that needs a caveat, because the intuitive reading is wrong. A ratio below 1.0 does not mean a model got more accurate than an experiment. It means the disagreement between labs on that endpoint has grown to roughly the size of the spread you're trying to predict. The assay disagrees with itself about as much as the compounds differ from each other. CYP2D6 and CYP3A4 inhibition are both there. Those boards are approaching the point where they stop separating tools at all, and we'd rather write that on the board page than keep ranking noise.

The baselines that made us uncomfortable

Three untrained references run on every board before it opens.

Constant, which predicts the training mean for everything. Nearest neighbour, which copies the label of the most Tanimoto-similar training molecule. And Morgan + LightGBM at defaults.

Across our 19 regression boards, constant prediction has the lowest MAE on 7 of them. Morgan+LightGBM takes the best AUROC on 13.

Go back to the hERG table. Under the time split, LightGBM's MAE is 0.599 and the constant's is 0.589. The trained model loses. Under a random split on exactly the same data it wins by a mile, 0.457 against 0.671.

Our reading is that fingerprint models are genuinely good at ordering compounds and much weaker at calibrating absolute potency for chemistry they haven't seen. The rank information survives the harder split; the numbers don't. Both are worth knowing, and you lose one of them if you report a single headline metric.

A benchmark that skips the trivial baselines can't tell you which of those regimes you're in. So we run them everywhere and publish them before anyone submits.

Two letters on every board

Boards on the same leaderboard aren't all asking the same kind of question, and stacking them into one ranking would be dishonest. So each one carries a tag like [T/P2].

The first letter is how we cut the data. T is a time split by first-report year, S is a scaffold split on Murcko cores, R is random, and we don't open those.

The second letter is whether you could go look the answer up. P1 means the source is public and we curated it. P2 means the source is public but our unit conversion and selection define this particular revision. P3 means we hold the labels. P4 is prospective, where the answer doesn't exist anywhere yet.

[T/P1] and [T/P3] are different experiments. Better to say so on the tin.

The boards

Seven disciplines, 212,670 training compounds, 18,382 held out.

Two of them are unusual enough to describe properly.

Cell / Phenotype

This one sits on the JUMP Cell Painting Consortium release, which is CC0. Compounds go onto cells, six stains go on, a microscope takes pictures, and you come out with 737 morphological features per well.

The task is to predict, from structure alone, how the cell changes shape. The label isn't a number. It's a 16-axis profile, the first 16 principal components, holding 61.0% of the variance. We score the per-compound Pearson correlation between the predicted and observed profile. Scaffold split, 113,688 train and 2,001 test, and we only evaluate on Murcko cores that never appear in training.

There's no cross-lab repeat structure to derive a noise floor from here, so we bootstrapped the metric instead and got an SD of 0.0088. The best baseline sits about 18 of those above the constant, which is enough for the board to separate tools, and 0.0088 becomes the threshold for the ladder rule described below.

Post-marketing withdrawal

Predict whether an approved drug later got pulled from the market. This board has a trap in it that took us a while to see.

Withdrawal rate tracks the decade of approval. Roughly 7.8% for 1990s approvals against 1.2% for the 2010s. Which means a predictor given nothing but the approval year scores AUROC 0.636. No structure, no mechanism, no name. Just the year.

Ship that and you've built a leaderboard for reading dates.

So we matched the controls by approval era, pairing each withdrawn drug with a non-withdrawn drug approved around the same time, and ran the check again. The year-only predictor now gets 0.504. Chance. That's the version we opened.

Scaffold split, 408 train, 184 test, 27.7% positive, bootstrap metric SD 0.0444.

We run some version of that check on every board before opening it, and boards that fail don't open. Same for the noise floor: if we can't measure one for an endpoint, there's no board, because without a floor there's no threshold to enforce.

Submitting more than once

There's an old problem with held-out leaderboards. You never see the labels, but if you submit fifty times and keep your best result, you've been pulling information out of the test set through the score channel the whole time. Enough tries and you climb without generalising at all.

We use the ladder mechanism from Blum and Hardt (ICML 2015). A new score is revealed only if it beats your own previous best by more than the noise floor. Otherwise we show you your previous best again.

An improvement smaller than experimental error isn't an improvement, so we don't hand that information back. You can't climb by submitting more often. You climb by actually getting better than the measurement precision of the field.

This is the other reason the noise floor has to be real rather than decorative. It's the threshold the whole mechanism runs on.

How to enter

Pick a board, download its test set, which contains structures and nothing else. Predict with whatever you like. A trained model, a physics engine, an LLM, a rule of thumb you trust. We don't care what's inside, we measure the output.

Upload a two-column CSV of compound_id,prediction and sign in with your Hugging Face account. Scoring runs off-platform on hardware that holds the labels; the Space never has them.

Each board page ships a filled-in prompt and a runnable skeleton so you don't have to build anything from scratch to make a first submission. The interface picks up Korean or English automatically.

Every score carries the SHA of the exact test file and scoring code behind it, so a number from six months ago is still traceable.

Licensing

ChEMBL 37 from EMBL-EBI, CC BY-SA 3.0, re-curated, with the attribution travelling on every board card. JUMP Cell Painting, CC0. The withdrawal board is assembled from public regulatory records. Non-commercial research benchmark, structures only, no labels distributed.

What we're not claiming

Three limits, since a benchmark that oversells itself is worse than no benchmark.

A time split is not prospective validation. The molecules exist and the values are published somewhere; what the split buys you is that the structural shortcut is closed and the training set is a real snapshot of what was known. Actually prospective boards get graded P4 and are a separate thing.

A noise floor built from cross-publication pairs is an estimate. It leans on a normality assumption and on ChEMBL's document attribution being right. We think it's much better than no reference at all, and we publish the pair counts so you can argue with the estimate.

And we're not saying other benchmarks are wrong. We're saying a score reported without its split, its baselines, and its label precision can't be compared to another score. Publishing all three, on every board, before anyone submits, is the whole contribution.

Why we bothered

There is no shortage of prediction tools, and every one of them reports a best-in-class number on its own benchmark. Put those numbers side by side and you learn nothing, because the rulers are different lengths.

We wanted one ruler. And it seemed only fair that if you hand somebody a ruler, you tell them how thick the markings are.

Open LEADBOARD

Specification v1.1, 8 clauses. FINAL-Bench / VIDRAFT.