# Embracing assay heterogeneity with neural processes
for markedly improved bioactivity predictions

Lucian Chan<EMAIL_ADDRESS>Astex Pharmaceuticals, Cambridge, UK  Marcel
Verdonk  Astex Pharmaceuticals, Cambridge, UK  Carl Poelking
<EMAIL_ADDRESS>Astex Pharmaceuticals, Cambridge, UK

###### Abstract

Predicting the bioactivity of a ligand is one of the hardest and most
important challenges in computer-aided drug discovery. Despite years of data
collection and curation efforts by research organizations worldwide,
bioactivity data remains sparse and heterogeneous, thus hampering efforts to
build predictive models that are accurate, transferable and robust. The
intrinsic variability of the experimental data is further compounded by data
aggregation practices that neglect heterogeneity to overcome sparsity. Here we
discuss the limitations of these practices and present a hierarchical meta-
learning framework that exploits the information synergy across disparate
assays by successfully accounting for assay heterogeneity. We show that the
model achieves a drastic improvement in affinity prediction across diverse
protein targets and assay types compared to conventional baselines. It can
quickly adapt to new target contexts using very few observations, thus
enabling large-scale virtual screening in early-phase drug discovery.

The primary aim of drug discovery is to design compounds that are safe for the
patient and efficacious against the disease. During preclinical development,
efficacy is usually equated with potency: A drug that is a potent inhibitor of
a target protein is assumed to be effective against the associated disease.
Hence building potency while maintaining a safe chemical profile is the key
objective in early-phase drug discovery. Models that can predict the binding
affinity as a measure of potency of a small molecule against a protein target
are therefore the subject of intense research efforts. Accurate and
transferable affinity models could help escalate the drug discovery process by
identifying relevant candidate molecules across ultra-large chemical spaces
[vsynthes_sadybekov_2022, ultralarge_lyu_2019, exploration_warr_2022], such as
Enamine REAL [enamine] or GalaXi [galaxi]. Large-scale, standardized and open-
access data collection, curation and storage, such as provided by ChEMBL
[mendez19] and the Protein Data Bank (PDB) [berman00], are pivotal as they
open up the possibility of data-driven, low-compute, deep-learning-based
alternatives to experimental or physics-based computational screening
techniques.

Among these data-driven techniques, we conventionally distinguish between
ligand-based [deepdta_ozturk_2018, nguyen20_dta, Daga23, metadta_2022] and
structure-based approaches [frustration_volkov_2022, mbp_yan_2023,
jones_improved_2021, hacnet_2023] that typically produce models that are
either local (i.e., specific to a certain protein system) or global in scope.
Here a model is said to be structure-based if it makes use of the three-
dimensional conformation of an experimental or predicted protein-ligand
complex, instead of just the topological structure of the ligand and the
protein identity or sequence. Despite significant interest from the community,
existing structure-based models still fall significantly short of medicinal-
chemistry requirements, with typical root-mean-square errors of around 1.2-1.3
$\text{p}K$ units [frustration_volkov_2022] when evaluated on the PDBBind core
set [Wang04]. Ligand-based approaches, by contrast, benefit from a much larger
data volume than structural datasets currently provide. They have shown
adequate or even superior performance in identifying hits in a binary
classification setting, but are susceptible to structural bias and
memorization [memorization_wallach_2018]. Many attempts have been made to
develop ligand-based regressors [deepdta_ozturk_2018, nguyen20_dta, Daga23],
with model performance varying widely depending on the data scope, locality,
quality and volume. In practice, such ligand-based models are frequently used
to filter out inactive compounds from large molecular libraries, followed by
an evaluation with more accurate physics-based simulation methods, such as
free-energy perturbation. These physics-based approaches can, however, suffer
from significant systematic errors, generally require long simulation times,
and are with current computing capabilities challenging to deploy on a scale
beyond a few hundred to thousand compounds.

Figure 1: Illustration of the MetaBind approach. The model constructs an
assay-specific SAR using a local support set of per-assay observations to
predict affinities for unobserved protein-ligand pairs. In meta-learning
terms, each assay is thus treated as a task, and assays are clustered into a
small number of $K$ covariance groups (with, e.g.,, K=4). The subset of
observed activities and the associated protein-ligand pairs are used to
construct the SAR representation. At training time, this subset is randomly
sampled from the available data on a per-assay basis, with the model trying to
reconstruct the covariance structure associated with the full set of
observations (dashed lines up top) from only a partial set sampled at random,
as guided by a Kullback-Leibler loss term. The aggregation function, which is
here based on a cross-covariance operator, as well as the protein and ligand
embeddings are implemented using convolutional or (for ligands) graph-
convolutional neural networks. The bioactivities of each protein-ligand pair
and for each cluster are obtained via a multi-output neural network. Note that
the assay-specific SAR is based, first and foremost, on the covariance
structure inferred from the observed bioactivities of the context set. The
assay metadata itself incorporates only the assay type (biochemical vs cell-
based), in line with the notion that the formalism should be agnostic with
respect to the source of the data heterogeneity, and that all condition
variables are implicitly encoded into the bioactivities themselves.

So why are data-driven models not performing better? The key issue is that the
available bioactivity data are globally sparse and locally heterogeneous. This
means that the models will struggle to identify relevant patterns across
apparently unrelated assays (unrelated because they might target very
different proteins); and, also, that they struggle to reconcile apparent
inconsistencies among supposedly related assays (related because they might
target similar proteins or assay similar compounds). For the latter, we use
the term heterogeneous to describe the case where two seemingly equivalent
assays measure systematically different structure-activity relationships
(SARs). This heterogeneity – or between-assay variability – is therefore not
caused by experimental uncertainty – the within-assay variance. Instead, it is
the result of differences in the experimental approach, assay type, assay
conditions, and unobserved or undeclared hidden variables that affect the
assay outcome: Consider, for example, cellular vs non-cellular and direct-
binding vs functional assays; variations in pH, temperature, buffer
composition or protein concentration; batch effects, solubility issues and
non-specific binding; differences in the sensitivities and dynamic ranges of
the experimental setups, or differences in the data analysis. There are thus a
multitude of reasons why point-wise aggregation (i.e., the act of collating
the results from different assays to create a single dataset per protein)
introduces inconsistencies into the data, thus causing issues downstream.

Previous analyses of mixed $\text{K}_{i}$ and $\text{IC}_{50}$ [kramer12,
kalliokoski13] data have attempted to estimate the experimental uncertainty
from pairwise protein-ligand bioactivity relationships independently of the
experimental approach. This line of analysis thus assumes that the variability
is caused exclusively by uncertainty in the measurement itself, and does not
separate the between-assay variability from the within-assay variance. As a
rule of thumb, point-wise aggregation is meaningful only if the systematic
variability between assays is far smaller than the statistical variability
within. This implies that point-wise aggregation is particularly problematic
when aggregating assays with different endpoints ($\text{IC}_{50}$ or
$\text{K}_{i}$, for example), as this would require a conversion rule for the
assay output [cheng73] that in turn relies on knowledge of either the
bimolecular interaction kinetics or of any relative offsets – neither of which
are typically available.

Due to these sometimes hidden complexities of bioactivity data, we argue that
the data aggregation process can be learnt implicitly by the predictive model
itself, and that the model should be agnostic of the exact source of the
heterogeneity. Here we show that this objective can be met with a hierarchical
meta-learning model that incorporates assay heterogeneity while improving data
efficiency. The model design reflects the fact that in a drug-discovery
setting, we routinely find ourselves faced with a few-shot learning problem,
as at best a few hundred to thousand molecules are made and tested during the
typical life cycle of a discovery project. Importantly, the model is able to
infer an assay-specific SAR from only a small number of observed data points
in a few-shot manner, and is therefore able to ‘translate’ results from one
assay into the context of another. This assay-specific approach differs
conceptually from previous meta-learning attempts and produces, as we show, a
drastic improvement in predictive performance.

##  I Framework

In this section we describe a meta-learning approach that models assay
heterogeneity by exploiting the underlying correlation structure across
diverse sets of assays. The resulting model, dubbed MetaBind, applies meta-
learning at the assay level (each experiment thus constituting a separate
task) to learn a local assay-specific SAR; it is a clustered multi-task neural
process that can be viewed as an extension of standard neural processes
[garnelo18b].

Conventionally, to construct a neural process, task-specific samples, denoted
as context data $\mathcal{C}$, are collected in order to infer the statistics
of the associated domain. Based on these domain-wide statistics learned from
the context, the model predicts the distribution $p(y|x,\mathcal{C})$ of a
target output $y$ for a given input $x$. In our case, the context data
consists of the ligand structures and read-outs from a given assay. The
learned statistics not only capture information on the underlying SAR, but
also implicitly the experimental conditions. Using this framework, ligand
affinities and uncertainties in the predictions can therefore be inferred
independently for each assay, in contrast to existing models where the assay
heterogeneity is neglected during training. The statistics generated from the
context data is critical as it has to encapsulate the functional form of the
SAR and be rich enough to identify variability across assays. Nevertheless,
despite their flexibility, scalability and well calibrated uncertainty
estimation, standard neural processes are designed to model data from only a
single stochastic process, which render them unsuitable for bioactivity data
collected using various techniques (e.g., biophysical, biochemical or cell-
based assays), with various endpoints (e.g., $\text{K}_{i}$, $\text{K}_{d}$,
$\text{IC}_{50}$ and $\text{EC}_{50}$) and variable experimental conditions.

A simple way of incorporating heterogeneity among related or equivalent assays
would be to estimate an affinity offset from protein-ligand pairs that are
shared across experiments (i.e., calibration or reference compounds). Such
pairs are, however, few and far between within mixed-origin databases such as
ChEMBL. Therefore, instead of estimating the offset directly (and thus
limiting the model to constant-offset heterogeneities), the idea is to model
the relative change within assays while incorporating information from related
experiments in a mechanistically agnostic and functionally flexible manner.
This is why here we introduce a clustered multi-task neural process (Fig. 1)
that represents the bioactivity as a linear combination of a learnt reference
affinity value $\tilde{y}^{\text{ref}}$, and $K$ independent stochastic
processes $f_{k}$ with scaling factor $\gamma$ that model the relative SAR
within the assay. I.e., we use the ansatz

$\begin{split}y_{ia}=&\ \
y_{a}^{\text{ref}}+\sum_{k=1}^{K}w_{ka}\big{(}\gamma_{a}f_{k}(x_{ia}-x_{a}^{\text{ref}})\big{)}+\epsilon_{ia},\\\
\text{s.t.}&\sum_{k=1}^{K}w_{ka}=1,\end{split}$ (1)

where $x_{ia}$ is the representation of protein-ligand pair $i$, and $y_{ia}$
is the associated bioactivity relationship for assay $a$; $\epsilon_{ia}$ is
an error term that follows a Gaussian distribution with zero mean and constant
variance. The scales $\gamma_{a}$ and weights $w_{ka}$ are learnable functions
(see Methods for details).

This formulation assumes that the assays can be clustered into a small number
of covariance groups, and that the SAR within each group is defined by a mean
function $f_{k}$ and assay-dependent scaling factor $\gamma_{a}$, as well as
weight functions $(w_{1a},...,w_{ka})$ that quantify the relevance of each
assay in the respective clusters. This functional structure makes the key
difference to existing models, as assays are not aggregated in a point-wise
fashion before the model is constructed, but instead aggregated implicitly on
the fly in a manner learnt by the model itself.

Note that the cluster functions $f_{k}$ operate on a relative embedding
$\tilde{x}=x_{ia}-x_{a}^{\text{ref}}$ to emphasise the variability within an
assay relative to a reference calculated over the context $\mathcal{C}$,

$\displaystyle(x_{a}^{\text{ref}},y_{a}^{\text{ref}})$
$\displaystyle=\left(\frac{\sum_{i}^{N_{a}^{\mathcal{C}}}x_{i}}{N_{a}^{\mathcal{C}}},\frac{\sum_{i}^{N_{a}^{\mathcal{C}}}y_{ia}}{N_{a}^{\mathcal{C}}}\right).$
(2)

To effectively aggregate functionally related assays, one could model the
joint distribution of $(\tilde{x},\tilde{y})$, i.e., $P_{\tilde{x}\tilde{y}}$,
through mean embedding of concatenated samples, $\phi(\tilde{x},\tilde{y})$.
Here we instead represent this joint distribution using the cross-covariance
operator $C_{XY}$[gretton20] over the context $\mathcal{C}$:

$\begin{split}C_{XY}^{a}&=\frac{1}{||\mathcal{C}_{a}||}\sum_{i\in\mathcal{C}_{a}}\phi_{x}(\tilde{x}_{ia})\otimes\phi_{y}(\tilde{y}_{ia})\\\
&=\frac{1}{||\mathcal{C}_{a}||}\Phi_{x}^{T}\Phi_{y}^{a}\in\mathbb{R}^{h\times
d},\end{split}$ (3)

where $\otimes$ denotes the outer product, $\phi_{\tilde{x}}$ and
$\phi_{\tilde{y}}$ are feature maps for covariates and dependent variables,
respectively; $h$ and $d$ are the output dimensions of $\phi_{\tilde{x}}$ and
$\phi_{\tilde{y}}$. This cross-covariance is then flattened to parameterize
the learned statistics via the weights $w_{ka}=w_{k}(C^{a}_{XY})$ and scale
parameter $\gamma_{a}=\gamma(C^{a}_{XY})$. Conceptually, similarity of the
SARs of two assays $a$ and $b$ implies similar statistics,
$P_{\tilde{x}\tilde{y}}^{a}\simeq P_{\tilde{x}\tilde{y}^{b}}$, and thus
similar functional parameters $w_{k}$ and $\gamma$.

This multi-modal approach with relative embeddings ensures that the neural
process can capture non-trivial data heterogeneities. In the simple case that
two assays differ by a mean shift caused by, for example, an annotation error
that substituted nanomolar with micromolar units, the change is fully
encapsulated in the reference affinity $y^{\text{ref}}$, whereas the induced
shape parameters are unaffected, as the covariance structure $C_{XY}$ remains
unchanged. Concept shifts in the SAR, by contrast, would trigger adjustments
in $(\mathbf{w},\gamma)$, in addition to potential changes in the reference
itself.

In this study, we use bioactivity data from ChEMBL30 to train this neural
process. Proteins and ligands are featurized via their sequence and molecular
graphs, respectively. Please refer to the Methods section at the end as well
as the Supplementary Information for further details on data processing, the
featurization and encoding of protein-ligand pairs, the neural networks used
to represent the shape parameters and cluster functions, the different loss
terms or training protocol.

##  II Results and Discussion

Figure 2: Observed and predicted heterogeneities in structure-activity
relationships. (a-c) Correlation of the bioactivity read-out for congeneric
series of three different assay pairs. The pairs (A,B) and (C,D) share the
same protein variants, but differ in assay type and conditions. The assays of
pair (E,F), on the other hand, differ in protein variant, with assay E
investigating the wild type, and F a singly mutated variant thereof. The first
assay pair (a) exhibits a strong correlation between the read-outs with,
however, a noticeable systematic affinity offset. The second pair (b) does not
show any significant correlation. The third pair (c) exhibits both an offset
and rescaling of the bioactivities across the partners. (d-f) Baseline
predictions for the series of assays A-F obtained with a standard graph-
convolutional model. The baseline reproduces neither the activity trends
within the assays, nor between the partners. (g-i) Affinity predictions
obtained with MetaBind, demonstrating improved performance and at least
partial reconstruction of the heterogeneity for all three pairs.

| Paired (biochemical) | Chronological (biochemical)
---|---|---
|  Task RMSE |  Task MAE |  Task RMSE |  Task MAE
Baseline |  1.125 (0.477) |  0.981 (0.465) |  1.150 (0.477) |  1.008 (0.480)
Baseline (local) |  1.102 (0.462) |  0.960 (0.450) |  1.129 (0.464) |  0.988 (0.466)
MetaBind |  0.794 (0.251) |  0.663 (0.231) |  0.828 (0.270) |  0.689 (0.243)

| Paired (cell-based) | Chronological (cell-based)
---|---|---
|  Task RMSE |  Task MAE |  Task RMSE |  Task MAE
Baseline |  1.160 (0.493) |  1.023 (0.500) |  1.171 (0.497) |  1.030 (0.498)
Baseline (local) |  1.137 (0.484) |  1.001 (0.487) |  1.152 (0.485) |  1.012 (0.487)
MetaBind |  0.810 (0.236) |  0.672 (0.210) |  0.854 (0.282) |  0.712 (0.257)

| Paired (all) | Chronological (all)
---|---|---
|  Task RMSE |  Task MAE |  Task RMSE |  Task MAE
Baseline |  1.110 (0.443) |  0.974 (0.436) |  1.131 (0.468) |  0.992 (0.469)
Baseline (local) |  1.090 (0.436) |  0.955 (0.430) |  1.113 (0.457) |  0.974 (0.459)
MetaBind |  0.763 (0.234) |  0.632 (0.208) |  0.803 (0.252) |  0.668 (0.227)

Table 1: MetaBind vs baseline performance in paired-assay and chronological
test settings. The baseline models are constructed using pre-aggregated
training data. Baseline does not use any additional test data during
predictions, as opposed to Baseline (local), which performs a local update of
the model given the assay context data. For each test setting (paired and
chronological), we furthermore distinguish between biochemical and cell-based
data partitions. MetaBind outperforms both baselines by a sizeable margin
across all splits and test scenarios. Note the value in brackets, which
indicates the standard deviation of the per-assay RMSEs and MAEs (not the
error of the mean, which is far smaller). The MetaBind approach thus improves
not only the predictions themselves, but also reduces (super-proportionally)
the variability in performance across assays. Furthermore, it achieves the
largest improvement in performance when aggregating biochemical and cell-based
assays, despite the increased heterogeneity of the combined data.

The MetaBind formalism proposed in this work is designed to handle assay
heterogeneity in a functionally agnostic manner. This means that the approach
does not make any specific assumptions how the SARs of two related assays
differ. We illustrate this aspect first by showcasing how the model behaves
for three examples that each reflect a different heterogeneity class. We will
then proceed with a broader, statistical validation of the approach designed
to indicate performance levels in a semi-prospective or prospective setting
relative to conventional baselines.

In qualitative terms, we can distinguish between three different grades of
heterogeneity among related assays: Simple offsets, offsets combined with
rescaling, and fundamental conceptual shifts in the SAR. Fig. 2 shows data for
three different assay pairs that fall within one of these heterogeneity
classes. Two assay pairs (Fig. 2a-b) display distinct SARs despite the
partners sharing the same protein-ligand series; the third pair (Fig. 2c)
compares experiments for the wild-type versus a mutated variant of the same
target protein. The SARs for the first assay pair (panel a) are strongly
correlated with a Pearson correlation coefficient $r=0.93$, but a sizeable
systematic offset of approximately 0.78 log units that might be the effect of
human plasma that is present in one assay (assay A) but not the other (assay
B).

The second pair (panel b) compares a cell-based (C) and biochemical assay (D)
for the same protein. The SARs are essentially uncorrelated with $r=0.26$,
raising the question whether there is any meaningful information exchange
possible between these assays. It is pairs like these that can cause severe
issues during point-wise data aggregation, as the data samples are clearly
drawn from two different distributions, and any agreement along the diagonal
must be interpreted as coincidental.

The third pair (panel c) compares the SARs for two assays of different
variants of the same protein: Assay E is a cell-based assay of wild-type
B-Raf, whereas assay F is a biochemical assay of a singly mutated variant
(V600E, which is a key mutation in various cancer types). Although the
measured bioactivities are clearly correlated with $r=0.63$, the SARs differ
in terms of both scaling and offset. It is then impossible to determine
without in-depth analysis what role the mutation plays in causing this
difference, compared to the role of the variability in assay type, or the
effect of a random heterogeneity due to unobserved or undeclared condition
variables.

So how do machine-learning models fare at predicting this variability in SAR?
First, consider a standard graph-convolutional neural network (GCNN) that is
trained on ChEMBL data preprocessed using point-wise aggregation rules.
Expectedly, this simple baseline performs extremely poorly on this type of
problem (see Fig. 2d-f): Not only does the model ignore differences in SAR,
but it also fails to produce any discernable correlation between its
predictions and the experimental affinities per assay. Furthermore, its
predictions tend to fall into a very narrow affinity range. Such behaviour can
be caused by the learning dynamics of neural networks, which tend to produce
smooth functions over the input space. As a result, small perturbations due
to, e.g., minor changes in a lead series or single-point mutations of a
protein, fail to produce a significant change in predicted SAR. This behaviour
is aggravated further if the functional form of the predictor remains static,
which is the case for conventional GCNNs.

Figure 3: Global correlation between predicted and observed bioactivities.
Tests based on the (a) paired-assay and (b) chronological split produce a
strong Pearson correlation $r$ of $0.72$ and $0.82$ with the experimental
affinities, respectively. The same holds for the (c) ligand heterogeneities
extracted from the test set of the paired-assay split. These heterogeneities
$\Delta$ (see their definition in eq. 4) are not typically known, but here
available by construction because of the congeneric series underlying the test
set of the paired split. Note that panels a and b compare centred affinities
to aid interpretation, with the same centring constant applied to all axes.

The MetaBind model proves to be significantly more flexible in this regard
(Fig. 2g-i): Even though the intra-assay correlation of its predictions with
the experimental observations is rather modest, the mean absolute errors for
the different assays range between 0.41 and 0.85 $\text{p}K$ units.
Importantly, the ligand heterogeneity, i.e., the difference in affinity for
the same ligand $i$ but different assays $(a,b)$,

$\displaystyle\Delta_{ia,ib}=y_{ib}-y_{ia},$ (4)

is reasonably well predicted with mean absolute errors (MAEs) of $0.35$,
$0.48$ and $0.89$ $\text{p}K$ units for assay pairs (A,B), (C,D) and (E,F),
respectively. These values should be compared to the mean absolute deviation
of the measured ligand heterogeneities (which are observable here because of
the joint ligand set) of $0.78$, $0.68$ and $2.33$. This suggests that the
model is able to capture the heterogeneity at least partially. Analysis of the
functional parameters (cluster weights $w$ and scales $\gamma$) gives further
insight into the type of heterogeneity predicted by the model. E.g., for assay
pair (A,B), the difference in predicted scaling is insignificant,
$|\gamma_{A}-\gamma_{B}|\approx 0.005$, as is expected for a constant-offset
heterogeneity. Meanwhile, the ranks of the cluster weights $w$ are identical
within each pair, illustrating that the partners are predicted to fall within
a similar SAR covariance class, despite perhaps what the experimental read-out
from assay pair (C,D) suggests.

These examples underline why ‘neural’ data aggregation as performed by
MetaBind is expedient. Still, the analysis so far was mostly anecdotal, and we
will now pursue a broader validation of the approach. To this end, we again
use ChEMBL30 data, this time to set up two different benchmark settings: A
paired assay split and a chronological split. The former defines a test set of
190 assay pairs where the assays of each pair target the same protein, share
at least 15 ligands, and have an intra-assay mean absolute deviation of at
least $0.5$ $\text{p}K$ units. The latter uses a time split to divide the
ChEMBL data into a training set (pre-2016) and a test set (post-2016), where
the latter is made up of 178 unseen protein targets. Among the paired assays
of the test set, only 34 exhibit a correlation $r>0.7$, despite the Pearson
correlation coefficient being invariant to constant offsets or rescaling of
the SAR. For further details on the splitting procedure and dataset
properties, see Methods and the Supplementary Information.

The paired-assay split is designed to test the model’s capability to infer the
SAR heterogeneity, whereas the chronological split measures how well the model
generalizes to new assay contexts. We include two baseline models as
reference: First, a conventional GCNN – denoted Baseline – trained on pre-
aggregated data, and a variant thereof that uses gradient updates on context
data from the test set – denoted Baseline (local). The latter enables a fair
comparison with MetaBind, as additional context data is used in an
optimization-based meta-learning-style approach. Because MetaBind infers an
assay-specific SAR from a context support set, a direct comparison with
existing bioactivity and meta-learning models is not possible for conceptual
reasons.

The results of the comparison and MetaBind’s test performance are summarised
in Table 1 and Fig. 3a-b. We find that MetaBind outperforms the two baselines
by a sizeable margin across all splits and metrics. The metrics we use are
assay-level task RMSEs and MAEs (see Methods for their definition). For the
test settings explored here, the mean task MAEs measured for the baselines
range between $0.955$ to $1.171$, compared to $0.632$ and $0.854$ for
MetaBind. For each of the two test scenarios (paired and chronological split)
we additionally investigate different partitions of the ChEMBL data according
to assay type (biochemical, cell-based, any): Cell-based assays capture the
biological context more faithfully, but provide a less direct measure of
ligand activity against a protein target, are less reproducible, and more
susceptible to off-target effects. It is nevertheless reassuring that models
built exclusively on cell-based data perform almost on par with those trained
exclusively on biochemical data. Furthermore, despite significant differences
in chemical- and protein-space coverage of the biochemical and cell-based data
partitions, MetaBind achieves a noticeable gain in performance when trained on
a combined dataset. This is likely due to its enhanced ability to model the
additional heterogeneity introduced into the data upon aggregation of diverse
assay types. Overall, the good metrics and healthy correlation plots (Fig.
3a-b) obtained with MetaBind indicate that the framework successfully
constructs expressive SARs from local context data. The locally optimized
baseline, by contrast, achieves only a very minor improvement over the
standard GCNN.

Finally, we stress that estimating offsets, scalings or other heterogeneities
across two assays without duplicate compounds is typically an intractable
problem. For the paired-test setting, however, we have the experimental ground
truths for the heterogeneities available as by construction. This means that
we can directly compare the predicted per-ligand shifts $\Delta_{ia,ib}$ to
their experimental values. The correlation (Fig. 3c) is excellent, with a
Pearson $r$ of $0.84$ and a mean absolute deviation of $0.56$ $\text{p}K$
units. This is particularly encouraging as only five context observations were
randomly selected from each assay to be supplied to the model, and not
necessarily the same five within each pair. This indicates (as can be verified
using dedicated tests, see the Supplementary Information) that the covariance
structure $C_{\tilde{X}\tilde{Y}}$ induced by the context set is robust.
Still, if the signal-to-noise ratio is extremely low, the cluster assignment
should be expected to become unstable, resulting in conservative predictions
biased towards the sample mean. Potential solutions for this noisy regime
could be to either increase the size of the context set, or provide additional
metadata, for instance on binding sites or assay conditions.

##  III Conclusions

Here we proposed a meta-learning formalism, MetaBind, that infers local,
assay-specific SARs for protein-ligand affinity modelling by harnessing both
local prior knowledge and global, diverse bioactivity data. The model enables
a form of neural data aggregation to address the issue of assay heterogeneity:
systematic differences in SAR produced by equivalent assays which, if
unaccounted for, can have a severe and deleterious effect on model
performance. We show that MetaBind produces exceptional results in different
benchmark settings and in a few-shot manner, far outperforming conventional
baseline models trained on pre-aggregated data, even when these are optimized
for a specific assay context. MetaBind is thus able to adapt quickly to new
target proteins, ligands, and assays, rendering it easily applicable to a
drug-discovery setting.

Testing on paired assays highlights that the formalism is able to reconstruct
in a robust way the heterogeneity even when presented with only minimal
support data from each assay partner. Importantly, MetaBind does not assume a
fixed functional form for the heterogeneity (such as constant offsets or
rescalings), does not rely on knowledge of its causal origin, or on duplicate
compounds shared across assays.

Data heterogeneity is generally considered a key issue that affects all major
(bio-)chemical databases due to their mixed origin, diverse content and almost
necessarily incomplete annotation. We therefore expect that formalisms such as
MetaBind, which are based on an implicitly learnt, ‘smart’ data aggregation
function, will prove increasingly important in constructing transferable,
performant models using community-driven, public-domain databases not just for
bioactivity modelling, but also chemical reactivity, ADME-Tox predictions and
beyond.

## Methods

#### Framework

MetaBind architecture. We employed a clustered multitask neural process
(CMTNP, eq. 1) to model the protein-ligand bioactivity because of its
flexibility, scalability and capacity to produce well calibrated predictions.
Derived from context data, the assay representation plays a crucial role in
our framework as it provides the basis for how the model learns to aggregate
data while constructing a local SAR. We use the cross-covariance operator in
eq. 3 to model the functional dependency of the SAR as probed by the different
protein-ligand pairs of an assay, i.e., $P_{\tilde{X}\tilde{Y}}$. To infer
appropriate scaling factors, the assay representation is normalised by the
variance of the covariates. The full representation thus assumes the form

$x_{a}=\frac{\text{Cov}(\tilde{X}^{a},\tilde{Y}^{a})}{\text{Var}(\tilde{X}^{a})},$
(5)

where $\text{Var}(\tilde{X}^{a})$ is a diagonal matrix of variances
($\sigma^{2}_{\tilde{x}_{1}}\dots,\sigma^{2}_{\tilde{x}_{h}}$) estimated from
context samples, with $h$ the latent dimension of the sample representation
$x_{ia}$. Note that we also apply a global translation to the bioactivity
values to ensure they are centered at zero.

Given the context data and the assay representation, the model jointly
predicts shape parameters $(\alpha,\beta)$ and the cluster assignment
$\mathbf{w}$ through a multi-output multilayer perceptron (MLP). This is
followed by soft-plus and softmax activation functions over ($\alpha,\beta$)
and w, respectively. Here we use $K=4$ covariance clusters. The scale
parameter $\gamma=\frac{\alpha}{\beta}$ is calculated from the shape
parameters of the associated gamma distribution $\text{Gamma}(\alpha,\beta)$;
$w$ is modelled with a categorical distribution.

The model input, $x_{ia}$, captures information on the protein sequence and
ligand structure of the protein-ligand pair $i$ within an assay $a$. Sequences
are encoded using a convolutional neural network with a sliding window size of
20. Ligands are encoded using a graph-convolutional neural network operating
on an atom-based molecular graph. The bioactivity predictions are formed by a
multi-output MLP decoder, with $2K$ output channels, with the first $K$
channels corresponding to the mean function $f_{k}$, the second $K$ channels
to the predicted variances $\sigma^{2}_{k}$. See the Supplementary Information
for details on the input featurization and the network architectures of the
encoders and decoder.

Training protocol. When training the model, we need to simulate an appropriate
context set $\mathcal{C}$ using the available assay data. Chronological
sampling would be ideal as this would most closely mimic the situation during
hit-to-lead and lead optimization. Unfortunately, however, chronological
rankings of the compounds are not usually available. Hence we resorted to on-
the-fly sampling of random subsets of varying size to construct the context
set, which trades in realism for improved data augmentation.

We train the model in an end-to-end fashion using the variational lower bound

$\displaystyle\text{log}\,p_{\theta}$
$\displaystyle(Y_{D}^{1:T}|X_{D}^{1:T},C)\geq\mathbb{E}_{q_{\phi}}\text{log}\,p_{\theta}(Y_{D}^{t}|X^{t}_{D},w^{t},\gamma^{t})$
$\displaystyle-
D_{KL}(q_{\phi}(\gamma^{D},w^{D}|D)||q_{\phi}(\gamma^{C},w^{C}|C))$ (6)

where $D_{KL}(\cdot,\cdot)$ is the Kullback–Leibler (KL) divergence; $q$ and
$p$ are the encoder and decoder networks, respectively. As in standard NPs,
the KL divergence term encourages the network to predict scalings and weights
from the context data, ($\gamma^{\mathcal{C}},w^{\mathcal{C}}$) that are
consistent with those from the full assay data
($\gamma^{\mathcal{D}},w^{\mathcal{D}}$).

Model implementation. The model was implemented in PyTorch [torch19] and
optimized using ADAM [kingma17] with a learning rate of $1e^{-5}$. The graph-
convolutional network was implemented using the Deep Graph Library (DGL) [dgl]
package. RDKit [rdkit] was used to read, write and manipulate ligands.
BioPython [biopython] was used to read, write and manipulate protein
structures and sequences.

Baseline models. In contrast to traditional QSAR models, MetaBind learns to
aggregate relevant assays instead of relying on hard-coded point-wise
aggregation rules. Furthermore, its predictions are based on local support
data (the context set) and hence, by design, assay-dependent. To fairly
compare MetaBind with traditional QSAR approaches, we constructed two
bioactivity models, Baseline and Baseline (local) with two distinct learning
protocols. Like MetaBind, both models use convolutional and graph-
convolutional neural networks to encode protein sequences and ligand
structures, respectively. Furthermore, the encoder architectures are identical
for both the baselines and the meta-learning model. The differences in
architecture are therefore on the decoder side: Specifically, in the baseline
models, the protein and ligand features are concatenated and fed into an MLP
to predict the ligand bioactivity (see the Supplementary Information for
details).

Importantly, the baselines are trained using preprocessed training data.
Duplicate observations are aggregated using their geometric mean if the
deviation of the minimum and maximum observed values is less than 0.3
$\text{p}K$ units (corresponding to a threefold change in affinity) under the
assumption that the endpoints are exchangeable; otherwise the observations are
discarded. The two baseline models use identical network architectures, data
pre-aggregation rules, and a mean-squared-error loss function for training,
but differ in how they are applied to test data, as Baseline (local) uses a
local training update over the context set to tune its weights to each test
assay independently. This local version thus enables a fair comparison to
MetaBind in that it produces an assay-specific SAR informed by local data.

#### Related Work

Meta-learning. Meta-learning is an umbrella term for machine-learning
approaches that aim to help a model adapt to new tasks using information on
previous tasks. Various studies have applied meta-learning principles to
molecular property prediction [han17, nguyen20, pappu20] and molecular
optimisation [wang21] in low-data regimes. These meta-learning techniques can
be broadly classified into model-based, optimisation-based [finn17] and
metric-based [snell17] approaches. Model-based techniques aggregate
information from previous tasks to extrapolate to new tasks and contexts.
Optimisation-based approaches specialise a meta-learner on a new task using,
e.g., gradient-based optimization on a support set. Metric-based approaches
construct a similarity metric across tasks and use embedding-based queries to
contextualize a support set.

Multi-task learning. Multi-task models have a long-standing tradition in
molecular-property prediction, and have also been developed for bioactivity
modelling [martin19, whitehead19, pentina22]. These models generally treat
each assay as a task and model all tasks jointly. This formulation avoids the
assay heterogeneity effect introduced in point-wise aggregation, and
explicitly exploits the correlation between assays to predict the bioactivity
of new protein-ligand pairs. These approaches, however, typically require a
moderate number of observations per task ($>50$) and some degree of compound
overlap among the different assays, which renders them impractical to apply to
low-data protein targets.

Neural processes. Neural Processes (NPs) [garnelo18b] learn a stochastic
process over functions as solution to a supervised learning problem. They thus
combine the adaptability of neural networks with the characteristics of
Gaussian Processes. The neural process is constructed in two stages. First, it
learns the statistics of a generic domain from a large sample set without
committing to a specific learning task. Second, using these domain-wide
statistics, it constructs a distribution over functions for a specific task
from a small support set. An NP variant also includes a global latent
variable, $z$, to account for the uncertainty in the predictions of the output
for given input pairs ($x$, $y$) [kim2018]. Its application to molecular-
property prediction has been explored recently [lee22, miguel22]. We note that
standard NPs only model data from a single stochastic process and are thus
designed to infer each task independently. There have been various efforts to
extend NPs to a multi-task setting [kim2022].

#### Data

Data filters. ChEMBL 30 [mendez19] was used throughout the study. We
considered binding assays that are assigned to a single protein target with a
ChEMBL confidence score of at least 9 (the maximum score). We included four
endpoints: $\text{K}_{i}$, $\text{K}_{d}$, $\text{IC}_{50}$ and
$\text{EC}_{50}$. The target species is homo sapiens. Here we do not consider
censored activity data, i.e., we remove data samples with inequality labels
$<$ or $>$. Moreover, we exclude assays that report identical activity values
for more than five compounds, or those with an activity range of less than
$0.5$ $\text{p}K$ units. Among the remaining set, we include only assays with
more than ten exact measurements, and consider only ligands with heavy-atom
counts $\leq 50$ and the following heavy-atom types: C, N, O, F, P, S, Cl, Br,
I. The resulting datasets consists of 6968 assays, with a total of 150,159
measured affinities across 100,279 unique ligands and 830 distinct target
proteins. Note that we only aggregate duplicates within assays, but keep
duplicates across assays separate. See the Supplementary Information for a
more detailed data summary.

Metrics and data splits. We evaluate the model performance at the task (i.e.,
assay) level using a task mean squared error (T-RMSE) and mean absolute error
(T-MAE):

$\displaystyle\text{T-RMSE}_{a}$
$\displaystyle=\sqrt{\frac{1}{N_{a}}\sum_{i=1}^{N_{a}}(\hat{y_{ia}}-y_{ia})^{2}},$
(7) $\displaystyle\text{T-MAE}_{a}$
$\displaystyle=\frac{1}{N_{a}}\sum_{i=1}^{N_{a}}|\hat{y}_{ia}-y_{ia}|,$ (8)

with task $a\in\\{1,...,n\\}$, observations $i\in\\{1,...,N_{a}\\}$, and
$N_{a}$ the number of observations for task $a$; $\hat{y}$ is the predicted
binding affinity (or bioactivity), $y$ is the observed binding affinity (or
bioactivity); $|\cdot|$ indicates the absolute value. We additionally use the
Pearson’s correlation coefficient $r$ to assess the (global) correlation
between the predicted and observed affinities.

To assess the degree to which a model is able to capture assay heterogeneity,
we constructed a test set of assays pairs where the partners share identical
target proteins together with a series of at least fifteen ligands.
Additionally, we ensured that the mean absolute deviation of the bioactivities
within the individual assays is $0.5$ log units or larger. We thus identified
100 assay pairs formed from among 190 unique assays as test set, and denoted
the corresponding split as the paired-assay split. The affinities of the assay
partners are not generally strongly correlated, see the Supplementary
Information for details.

To assess how the models adapt to new assays and targets, we split the assays
chronologically, with assays prior to 2016 entering the training set, assays
of 2016 or later the test set. This results in a test set of 178 unseen target
proteins. The median ligand similarity between training and testing set is
0.32. See further details in the Supplementary Information.

Data availability. The datasets used in this study are available at
https://github.com/lucianlschan/metabind

Code availability. The source code and pre-trained models can be accessed at
https://github.com/lucianlschan/metabind.

Acknowledgements. LC acknowledges funding from Astex through the Sustaining
Innovation Postdoctoral Programme. We thank Chris Murray, Davide Branduardi,
Lisa Ronan, Tugce Oruc, Rudolfs Petrovs and Caroline Richardson for thoughtful
comments on the manuscript.

Author contributions. LC conceived the project. LC and CP designed the model.
LC implemented the model, ran the experiments and performed the data analysis.
LC, MV and CP devised the analysis and validation. LC and CP wrote the
manuscript. CP and MV supervised the research. All authors reviewed the
manuscript.

Competing interests. The authors declare no competing interests.

## References

  * [1] Sadybekov, A. A. _et al._ Synthon-based ligand discovery in virtual libraries of over 11 billion compounds.  _Nature_ 601, 452–459 (2022).  URL https://www.nature.com/articles/s41586-021-04220-9.
  * [2] Lyu, J. _et al._ Ultra-large library docking for discovering new chemotypes.  _Nature_ 566, 224–229 (2019).  URL https://www.nature.com/articles/s41586-019-0917-9.
  * [3] Warr, W. A., Nicklaus, M. C., Nicolaou, C. A. & Rarey, M.  Exploration of ultralarge compound collections for drug discovery.  _Journal of Chemical Information and Modeling_ 62, 2021–2034 (2022).  URL https://pubs.acs.org/doi/10.1021/acs.jcim.2c00224.
  * [4] Enamine REAL Space.  https://enamine.net/library-synthesis/real-compounds/real-space-navigator (2022).
  * [5] GalaXi Space.  https://www.labnetwork.com/frontend-app/p/##!/library/virtual (2022).
  * [6] Mendez, D. _et al._ ChEMBL: towards direct deposition of bioassay data.  _Nucleic Acids Res._ 47, D930—D940 (2019).
  * [7] Berman, H. M. _et al._ The Protein Data Bank.  _Nucleic Acids Res._ 28, 235–242 (2000).
  * [8] Öztürk, H., Özgür, A. & Ozkirimli, E.  Deepdta: deep drug–target binding affinity prediction.  _Bioinformatics_ 34, i821–i829 (2018).  URL https://academic.oup.com/bioinformatics/article/34/17/i821/5093245.
  * [9] Nguyen, T. _et al._ GraphDTA: predicting drug–target binding affinity with graph neural networks.  _Bioinformatics_ 37, 1140–1147 (2020).
  * [10] Daga, A. _et al._ LEP-AD: Language Embedding of Proteins and Attention to Drugs predicts drug target interactions.  In _ICLR MLDD workshop_ (2023).
  * [11] Lee, E., Yoo, J., Lee, H. & Hong, S.  Metadta: Meta-learning-based drug-target binding affinity prediction (2022).  URL https://openreview.net/forum?id=yzlif16IASM.
  * [12] Volkov, M. _et al._ On the frustration to predict binding affinities from protein–ligand structures with deep neural networks.  _Journal of Medicinal Chemistry_ 65, 7946–7958 (2022).  URL https://pubs.acs.org/doi/10.1021/acs.jmedchem.2c00487.
  * [13] Yan, J. _et al._ Multi-task bioassay pre-training for protein-ligand binding affinity prediction (2023).  URL http://arxiv.org/abs/2306.04886.  ArXiv:2306.04886 [cs, q-bio].
  * [14] Jones, D. _et al._ Improved protein–ligand binding affinity prediction with structure-based deep fusion inference.  _Journal of Chemical Information and Modeling_ 61, 1583–1592 (2021).  URL https://pubs.acs.org/doi/10.1021/acs.jcim.0c01306.
  * [15] Kyro, G. W., Brent, R. I. & Batista, V. S.  Hac-net: A hybrid attention-based convolutional neural network for highly accurate protein–ligand binding affinity prediction.  _Journal of Chemical Information and Modeling_ 63, 1947–1960 (2023).  URL https://pubs.acs.org/doi/10.1021/acs.jcim.3c00251.
  * [16] Wang, R., Fang, X., Lu, Y. & Wang, S.  The PDBbind Database: Collection of Binding Affinities for Protein-Ligand Complexes with Known Three-Dimensional Structures.  _J. Med. Chem._ 47, 2977–2980 (2004).
  * [17] Wallach, I. & Heifets, A.  Most ligand-based classification benchmarks reward memorization rather than generalization.  _Journal of Chemical Information and Modeling_ 58, 916–932 (2018).
  * [18] Kramer, C., Kalliokoski, T., Gedeck, P. & Vulpetti, A.  The experimental uncertainty of heterogeneous public ki data.  _J. Med. Chem._ 55, 5165–5173 (2012).
  * [19] Kalliokoski, T., Kramer, C., Vulpetti, A. & Gedeck, P.  Comparability of mixed ic50 data – a statistical analysis.  _PLOS ONE_ 8, 1–12 (2013).
  * [20] Cheng, Y.-C. & Prusoff, W. H.  Relationship between the inhibition constant (${K}_{I}$) and the concentration of inhibitor which causes 50 percent inhibition (${I}_{50}$) of an enzymatic reaction.  _Biochem. Pharmacol._ 22, 3099–3108 (1973).
  * [21] Garnelo, M. _et al._ Neural processes.  _ArXiv_ abs/1807.01622 (2018).
  * [22] Gretton, A.  Notes on mean embeddings and covariance operators (2020).
  * [23] Paszke, A. _et al._ Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , 8024–8035 (Curran Associates, Inc., 2019).
  * [24] Kingma, D. P. & Ba, J.  Adam: A method for stochastic optimization (2017).
  * [25] Wang, M. _et al._ Deep graph library: A graph-centric, highly-performant package for graph neural networks.  _arXiv preprint arXiv:1909.01315_ (2019).
  * [26] RDKit: Open-source cheminformatics.  http://www.rdkit.org.
  * [27] Cock, P. J. A. _et al._ Biopython: freely available Python tools for computational molecular biology and bioinformatics.  _Bioinformatics_ 25, 1422–1423 (2009).
  * [28] Altae-Tran, H., Ramsundar, B., Pappu, A. S. & Pande, V.  Low data drug discovery with one-shot learning.  _ACS Central Science_ 3, 283–293 (2017).
  * [29] Nguyen, C. Q., Kreatsoulas, C. & Branson, K. M.  Meta-learning initializations for low-resource drug discovery.  In _ICML Workshop on Graph Representation Learning and Beyond (GRL+)_ (2020).
  * [30] Pappu, A. & Paige, B.  Making graph neural networks worth it for low-data molecular machine learning.  In _NeurIPS Machine Learning for Molecules Workshop_ (2020).
  * [31] Wang, J., Zheng, S., Chen, J. & Yang, Y.  Meta learning for low-resource molecular optimization.  _Journal of Chemical Information and Modeling_ 61, 1627–1636 (2021).
  * [32] Finn, C., Abbeel, P. & Levine, S.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _ICML_ , 1126–1135 (2017).
  * [33] Snell, J., Swersky, K. & Zemel, R.  Prototypical networks for few-shot learning.  In Guyon, I. _et al._ (eds.) _NIPS_ , vol. 30 (2017).
  * [34] Martin, E. J. _et al._ All-assay-max2 pqsar: Activity predictions as accurate as four-concentration ic50s for 8558 novartis assays.  _Journal of Chemical Information and Modeling_ 59, 4450–4459 (2019).
  * [35] Whitehead, T. M., Irwin, B. W. J., Hunt, P., Segall, M. D. & Conduit, G. J.  Imputation of assay bioactivity data using deep learning.  _Journal of Chemical Information and Modeling_ 59, 1197–1204 (2019).
  * [36] Pentina, A. & Clevert, D.-A.  Multi-task proteochemometric modelling.  _ChemRxiv_ (2022).
  * [37] Kim, H. _et al._ Attentive neural processes.  In _International Conference on Learning Representations_ (2019).
  * [38] Lee, E., Yoo, J., Lee, H. & Hong, S.  MetaDTA: Meta-learning-based drug-target binding affinity prediction.  In _ICLR Machine Learning for Drug Discovery_ (2022).
  * [39] Garcia-Ortegon, M., Bender, A. & Bacallado, S.  Conditional neural processes for molecules.  In _Machine Learning in Structural Biology_ (2022).
  * [40] Kim, D., Cho, S., Lee, W. & Hong, S.  Multi-task processes.  In _International Conference on Learning Representations_ (2022).

SUPPLEMENTARY INFORMATION
Embracing assay heterogeneity with neural processes
for markedly improved bioactivity predictions

###  A Data

Summary statistics for the assays selected from ChEMBL 30 for this study are
provided in Fig. S1. The dataset includes 6968 assays, covering 830 target
proteins with measured bioactivities for 100,279 molecules. The molecular
weight ranges from 107 Da to 857 Da, with a median of 415 Da. The heavy-atom
counts per ligand range from 8 to a maximum of 50 (as by construction).
Enzymes are the predominant protein target class, followed by membrane
receptors, as based on ChEMBL’s classification schema. The median target
sequence length is 471. The half maximal inhibition concentration,
$\text{IC}_{50}$, and inhibition constant, $\text{K}_{i}$, are the most common
endpoints, accounting for ca 63% and 31% of the data, respectively. The p$K$
values range from $-6.7$ (millimolar, mM) to $2$ (picomolar, pM) across all
assays, with an average of $-2.21$.

Figure S1: Dataset summary. (a) Composition by assay type, with biochemical
and cell-based assays being the predominant types. Half maximal inhibitory
concentration ($\text{IC}_{50}$) and inhibition constant ($\text{K}_{i}$)
together make up more than 90% of the data volume. (b) Composition by protein
target family. Enzymes are the predominant target class, followed by membrane
receptors. (c) Composition by protein-sequence length. The median sequence
length is 491. (d) Distribution of ligand molecular weights, ranging from
107Da to 857Da, with a median of 415 Da. (e) Distribution of per-ligand heavy-
atom counts. The median is 29. (f) Distribution of log-affinity values. The
median is $6.8$.

###  B Notation and definitions

We denote by $\mathcal{C}^{a}$ and $\mathcal{D}^{a}$, with $a=1,...,K$ , the
context and the target data from assay $a$, respectively, where $K$ is the
number of assays in the dataset. Further, we define
$\mathcal{C}=\cup_{a=1}^{K}\mathcal{C}^{a}$ and
$\mathcal{D}=\cup_{a=1}^{K}\mathcal{D}^{a}$.

The neural network components used for the convolutional and graph-
convolutional neural networks include activation functions, layer
normalisation, convolution and pooling operations as defined below.

#### Activation functions

Rectifying Linear Unit (ReLU)

$\displaystyle\text{ReLU}(x)$ $\displaystyle=\begin{cases}0&\text{if $x\leq
0$}\\\ x&\text{if $x>0$}\end{cases}$

Softplus ($\beta$ = 1)

$\displaystyle\text{Softplus}(x)$
$\displaystyle=\frac{1}{\beta}\log\big{(}1+\exp(\beta x)\big{)}$

Tanh

$\displaystyle\text{Tanh}(x)$
$\displaystyle=\frac{\exp(x)-\exp(-x)}{\exp(x)+\exp(-x)}$

Softmax

$\displaystyle\text{Softmax}(x)$
$\displaystyle=\frac{\exp(x_{i})}{\sum_{i=1}^{N}\exp(x_{i})}$

#### Convolutions

Affine transformation

$\displaystyle\text{Linear}(x)=Wx+b$

Molecular GCNN layer

$\displaystyle h_{i}^{l+1}=\sigma\left(b^{l}+\sum_{j\in
N(i)}\frac{1}{c_{ij}}h^{l-1}W^{l}\right)$

Here, $N(i)$ is the set of neighbors of node $i$, $c_{ij}$ is the square root
of the product of the node degrees, $\sigma$ is an activation function, $l$ is
the layer index, $W^{l}$ and $b^{l}$ are learnable weights and biases,
respectively.

#### Pooling operators

Max Pooling

$\displaystyle\text{maxpool}(x^{i})=\max_{k=1}^{N_{i}}\,(x_{k}^{i})$

Sum Pooling

$\displaystyle\text{sumpool}(x^{i})=\sum_{k=1}^{N_{i}}(x_{k}^{i})$

###  C Architectures

Ligand input features. Atoms (nodes) and bonds (edges) of the molecular graph
are featurized using a set of chemical properties as summarized in Tab. S3 and
S4. Categorical features are encoded in a one-hot format, resulting in an
overall input dimension of 36 for the atom and 6 for the bond feature vectors.
RDKit was used to compute all atom and bond features.

Feature |  Description | Dimension
---|---|---
Atom type |  Element (one-hot) | 12
Atom degree |  Neighbour count (one-hot) | 7
Radical electrons |  Radical-electron count | 1
Formal charge |  Charge in $e$ | 1
Hybridization |  $s$, $sp^{1}$-$sp^{3}$, $sp^{3}d$, $sp^{3}d^{2}$, other | 7
Aromaticity |  Aromatic indicator | 1
Hydrogen count |  Explicit + implicit | 5
Chirality |  CIP R and S indicators | 2
Table 2: Atomic input features. Atom node attributes used by all 2D molecular embedding networks. Feature |  Description | Dimension
---|---|---
Bond type |  Single – triple, aromatic, conjugated | 5
Ring bond |  Ring indicator | 1
Table 3: Bond input features. Covalent edge attributes used by all 2D
molecular embedding networks.

Protein input features. The residues of the protein sequence are encoded in a
one-hot fashion using the amino-acid type (AA), the sidechain topology and its
physicochemical properties. The four side-chain topologies are: (i) no
sidechain, (ii) linear side-chain, (iii) branched side-chain, and (iv) cyclic
side-chain. The physicochemical properties capture the electrostatic and
hydrophobic characteristics of the different AAs: positively charge (H, K, R),
negatively charged (D, E); polar (N, Q, S, T); aromatic (F, W, Y); hydrophobic
(A, I, L, M, V).

Feature |  Description | Dimension
---|---|---
Amino Acid |  20 AA and 5 special characters | 25
Side Chain Topology |  AA side-chain topology | 4
Properties |  AA physicochemical properties | 6
Table 4: Protein input features. Amino-acid (AA) features used as input for
the 1D protein-sequence embedding networks.

Ligand representation. The ligand is represented as a graph, featurized as
described above, and encoded using a 3-layer graph-convolutional neural
network, with residual connections between each layer. To generate the ligand
features, we apply max and sum pooling over the atom encodings, and
concatenate the respective outputs to obtain the ligand embedding.

Protein representation. The protein is represented as a sequence, featurized
as described above, and encoded using a discrete cosine transform applied to
the feature matrix, followed by a 1D convolution operation with kernel size 10
and stride 3. Note that we set a maximum sequence length of 1500 residues.

Protein-ligand representation. The joint protein-ligand embedding vector is
generated from the individual ligand and protein embeddings via concatenation
and subsequent linear transformation.

Model |  Input |  Architecture |  Dimensions
---|---|---|---
Ligand Encoder |  L |  GCN |  (34,128,128)
Protein Encoder |  P |  CN |  (32 ,32, 1)
P-L Encoder |  P, L |  Linear |  (419,128)
Cluster Decoder |  P-L,y |  MLP |  (128,128,6)
Bioactivity Decoder |  P-L |  MLP |  (128,128,$h$)
Table 5: Model architectures. L, P, P-L, y denote the ligand, protein,
protein-ligand and bioactivity, respectively. GCN, CN, Linear and MLP are the
graph-convolutional and convolutional network, linear map and multi-layer
perceptron, respectively. The values in brackets indicate the input,
intermediate and output dimension, with (see bottom row) $h=8$ for MetaBind,
$h=1$ for the baseline models.

###  D Data splits

Paired-assay split. The paired-assay test set contains 100 assay pairs
generated from 190 assays. The distribution of the number of observations per
assay pair is shown in Fig. S2a. The distribution of the mean absolute
deviation within a pair (i.e., the mean of the absolute ligand hererogeneity
$\Delta_{ia,ib}$) is shown in Fig. S2b, and has a median of $1.05$ log units.
The majority of the assay pairs are positively correlated, as shown in Fig.
S2c-d using the Pearson correlation $r$ and Kendall rank correlation
coefficient $\tau$. There are multiple assay pairs, however, for which the
correlation is negligible or even negative, see the tails of the distributions
over $r$ and $\tau$.

The correlation between observed and predicted values achieved by the standard
and local baseline models on the paired-assay test set is shown in Fig. S4a
and b, respectively. The correlation is weak, with a Pearson $r$ of $0.29$ and
$0.32$, respectively. The local update on context data thus improved the model
performance only slightly, limited presumably by the Lipschitz
continuity/constant inherent to the neural network.

Chronological split. The test set of the chronological split contains 539
proteins, with 178 unseen targets. The distribution of the normalised sequence
similarity between the unseen target proteins and observed target proteins is
shown in Fig. S3a. Here the normalised sequence similarity is defined as the
product of the identity score calculated from BLAST and the min-max ratio
$\rho$ of the sequence lengths,

$\displaystyle\rho_{AB}=\frac{\min(N^{A}_{AA},N^{B}_{AA})}{\max(N^{A}_{AA},N^{B}_{AA})},$

where $N^{A}_{AA},N^{B}_{AA}$ are the lengths of sequences A and B,
respectively. We used default parameters for the BLAST. For assays with
targets already seen during training, we calculated the maximum ligand
similarity between the training set and test set of those targets, with the
resulting distribution of similarities shown in Fig. S3b. Even though some
assays share identical ligands, most of the ligand sets differ considerably,
as indicated by a relatively low median ligand similarity of $0.32$.

As for the paired-assay test set, both the standard and local baseline give
poor fits with a limited dynamic range and Pearson $r$ of $0.36$ and $0.39$,
respectively (see Fig. S4c-d).

Assay | ChEMBL ID |  Description
---|---|---
A | CHEMBL1039372 |  | Inhibition of human recombinant renin assessed as decrease in plasma renin activity
---
by competitive radioimmunoassay in presence of human plasma
B | CHEMBL1039373 |  Inhibition of trypsin-activated human recombinant renin
C | CHEMBL2073500 |  Inhibition of human KDR autophosphorylation expressed in mouse NIH/3T3 cells
D | CHEMBL2073498 |  Inhibition of KDR by HTRF analysis in presence of 1 mM ATP
E | CHEMBL3804128 |  | Inhibition of wild type B-Raf in human MIAPaCa2 cells assessed as reduction in ERK
---
phosphorylation preincubated for 1 hr by Western blot method
F | CHEMBL3804127 |  | Inhibition of B-Raf V600E mutant (unknown origin) assessed as MEK1 phosphorylation
---
using MEK1-Avitag as substrate after 1 hr by HTRF assay
Table 6: ChEMBL ID and assay description of assays A-F used to illustrate the
different heterogeneity classes associated with the pairs (A,B) – constant
offset; (C,D) – uncorrelated; (E,F) – offset and rescaling. Figure S2: Paired-
assay test set. (a) Distribution of the size of the shared ligand set (number
of observations $N_{\text{obs}}$ per assay pair. (b) Distribution of the mean
absolute deviation $\Delta$ measured for the assay partners. (c) Distribution
of the Pearson correlation coefficient and (d) Kendal tau of the pair read-
outs. Note that some assays pairs show a negative or statistically negligible
correlation. Figure S3: Chronological test set. (a) Distribution of the
maximum sequence similarity between unobserved protein targets in the test set
with protein targets of the training set. (b) Distribution of the maximum
ligand similarity between ligands of the training set and ligands of the test
set for all observed protein targets. The median maximum similarity is $0.32$.
Note that we used Tanimoto similarity over 2048-bit Morgan fingerprints of
radius 2 to measure the ligand-ligand similarity. Figure S4: Baseline model
performance. Correlation plots comparing observed and predicted affinity
values for the (a) standard baseline and (b) local baseline under the paired-
assay split; (c) the standard baseline and (d) local baseline under the
chronological split. To aid interpretation, all panels compare centred
affinities, with the same centring constant applied to all axes.


Jaemin<EMAIL_ADDRESS>Minseok<EMAIL_ADDRESS>Jongchan
<EMAIL_ADDRESS>Dong-Geol<EMAIL_ADDRESS>Hanbat National
University,
Daejeon, Korea Lunit Inc.
Seoul, Korea Exploiting Features with Split-and-Share Module

# Exploiting Features
with Split-and-Share Module

###### Abstract

Deep convolutional neural networks (CNNs) have shown state-of-the-art
performances in various computer vision tasks. Advances on CNN architectures
have focused mainly on designing convolutional blocks of the feature
extractors, but less on the classifiers that exploit extracted features. In
this work, we propose Split-and-Share Module (SSM), a classifier that splits a
given feature into parts, which are partially shared by multiple sub-
classifiers. Our intuition is that the more the features are shared, the more
common they will become, and SSM can encourage such structural characteristics
in the split features. SSM can be easily integrated into any architecture
without bells and whistles. We have extensively validated the efficacy of SSM
on ImageNet-1K classification task, and SSM has shown consistent and
significant improvements over baseline architectures. In addition, we analyze
the effect of SSM using the Grad-CAM visualization.

##  1 Introduction

Deep convolutional neural networks (CNNs) achieve high performance in various
computer vision tasks [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-
Fei, Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and
Zitnick, Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler,
Benenson, Franke, Roth, and Schiele, Kuehne et al.(2011)Kuehne, Jhuang,
Garrote, Poggio, and Serre]. A general anatomy of CNN splits the architecture
into two parts: a feature extractor and a classifier [Bengio et
al.(2013)Bengio, Courville, and Vincent]. A feature extractor consists of
conv-blocks which are made of normalization layers, convolutional layers, non-
linear activations [Nair and Hinton(2010)], and pooling layers. To design CNN
architecture is to find a good conv-block and stack it repetitively. ResNet
[He et al.(2016)He, Zhang, Ren, and Sun] added identity-based skip connections
to the Conv-block to enable stable training even when the Conv-block is
repeatedly stacked deeply. In addition, the Xception [Chollet(2017)] structure
is a network structure developed from the Inception [Szegedy et
al.(2015)Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, and
Rabinovich] structure. Xception utilizes Depth-wise-separable convolution
using 1x1Conv to significantly lower the computation of the network and even
improve its performance. Accordingly, the recent trend on neural architecture
search [Zoph et al.(2018)Zoph, Vasudevan, Shlens, and Le, Real et
al.(2019)Real, Aggarwal, Huang, and Le, Tan and Le(2019)] focuses on designing
better conv-blocks in a data-driven way. While the classifier is also a
crucial part of a CNN, less attention has been paid on designing better
classifiers. In this study, we focus on designing a classifier that further
exploits a given feature vector. To the best of our knowledge, most of the CNN
architectures simply adopt single or multiple linear combinations as the
classifier. In this work, we propose a novel classifier, named Split-and-Share
Module (SSM). SSM divides the given feature into several groups of channels,
and the groups are partially shared among sub-classifiers. Each group of
channels has different degree of sharing; and our intuition is that the mostly
shared group will contain general features, and vice versa. This feature
split-and-share method can encourage the diversity of the features by
structure, and thus the diversity of the sub-classifiers, leading to higher
performances when ensembled.

Figure 1: An overview of SSM. The illustrated example has 2048 channels in the
final feature vector, and the output is 1000-way classification.

Figure 1 shows the structure of the proposed SSM.  Given a feature vector
extracted from the backbone network (feature extractor), SSM splits the
feature into 4 groups and each group is fed into the designated sub-
classifier. The final output is the sum of outputs from each sub-classifier.
The smallest group, illustrated as the bottom group in Fig. 2, is shared by
all other sub-classifiers, and should contribute to the final prediction
alone. It is encouraged to learn more common and general features in the
limited number of channels. On the other hand, the least shared channels,
illustrated as the top group in Fig. 2, will learn additional features such as
contextual information.  The Grad-CAM [Selvaraju et al.(2017)Selvaraju,
Cogswell, Das, Vedantam, Parikh, and Batra] visualization in Fig. 2
qualitatively supports our intuition.  As shown in Fig. 2, the first column
shows the acoustic guitar taken by Grad-CAM for each channel group. We can see
that going down from the first row to the bottom row, starting with the
additional characteristics of the acoustic guitar and gradually visualizing it
as the core characteristic of the acoustic guitar. SSM shows stable
performance improvement in architectures such as ResNet [He et al.(2016)He,
Zhang, Ren, and Sun] and ResNeXt [Xie et al.(2017)Xie, Girshick, Dollár, Tu,
and He], and is a simple structure consisting of BatchNorm [Ioffe and
Szegedy(2015)] and ReLU, easy to attach to any CNN architecture.  While the
sub-classifiers may resemble the ensemble technique, which may lead to
concerns on less improvements with ensemble. In our experiments, we show that
a SSM-augmented network can further be improved with ensemble without any
compromises.

Figure 2: Grad-CAM visualization of channels with respect to sub-classifiers.
ResNet-50+SSM is used for visualization, and the final feature size is 2048.
Each column visualizes 512 channels. The first column is visualized with
respect to FC1, the second column is visualized w.r.t. FC2, and so on. More
details will be described in Sec. 5.

##  2 Related works

###  2.1 Deeper architectures

Starting with AlexNet [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and
Hinton], many CNN structures have been proposed. VGGNet [Simonyan and
Zisserman(2014)] showed significant CNN performance improvement by increasing
network depth. Another study made network learning stable by normalizing the
input to each layer in batch units. Based on these developments, ResNet was
proposed. ResNet proposed identity-based skip connections to deepen the
network, greatly improving the performance of CNN. Since then, studies have
been proposed to discover CNN structures through architecture search such as
NASNet [Zoph et al.(2018)Zoph, Vasudevan, Shlens, and Le] and EfficientNet
[Tan and Le(2019)]. Also, architecture search methods based on Evolutionary
Algorithms such as AmoebaNet [Real et al.(2019)Real, Aggarwal, Huang, and Le]
have been proposed. Studies of these CNN structures have been continuously
proposed and attracted great attention. But in the progress of the structure
of these CNN classifier has been excluded. We conducted a study using features
that were already well extracted from the feature extractor, and can be used
in the architecture search later.

###  2.2 Feature analysis

Various analyses of features have been proposed. Ilyas et al. [Ilyas et
al.(2019)Ilyas, Santurkar, Tsipras, Engstrom, Tran, and Madry] was
experimental in that it exists as robust and non-robust features and can be
classified, rather than simply dividing features into useful and useless
features. From the perspective of Ilyas et al. [Ilyas et al.(2019)Ilyas,
Santurkar, Tsipras, Engstrom, Tran, and Madry], there are robust features and
non-features among features used for prediction, both of which are useful for
prediction but have different meanings and have room to utilize these
characteristics.

Afalo et al. [Aflalo et al.(2020)Aflalo, Noy, Lin, Friedman, and Zelnik]
showed that using all the features that enter the classifier does not improve
the accuracy of the CNN, but removing unnecessary features through pruning can
improve the performance and the computational speed of the CNN.

In this respect, our SSM is also a new analysis and utilization of features.
Our SSM forcibly assigns the role of features used for prediction through
backprop, and qualitatively and quantitatively analyzes the effect on the
network according to the location and number of such features.

##  3 Split-and-Share Module

In this section, we describe how SSM is formulated. SSM is a simple classifier
that splits and share features with multiple sub-classifiers. The overall
architecture of SSM is illustrated in Fig. 1, and the pseudo code algorithm is
described in Algorithm 1.

First, SSM equally divides the input feature in 4 splits, and sequentially
append the splits one-by-one to formulate 4 features with different numbers of
channels. For example, given the feature $F\in R^{2048}$, the first feature
$F_{1}$ contains the first 1/4 channels, i.e. $F_{1}=F[0:512]$. Accordingly,
$F_{2}$ contains the first 1/2 channels, and so on. In order to diversify the
4 features while keeping the feature domain with minimum overheads, we apply
BatchNorm with ReLU to the first 3 features for simple scaling and non-linear
activation. The resulting 4 features will have the same semantic meaning with
different scales for the shared channels. Channels in the 4 features can be
zeroed out by ReLU. BatchNorm and ReLU are essential in SSM, as they add extra
non-linearity to the overall process. Without BatchNorm and ReLU, SSM can be
reduced to a simple linear combination (fully-connected) layer. After
splitting, recombining and re-scaling, the 4 features are feed-forwarded to 4
sub-classifiers. Each sub-classifier is a simple fully-connected layer, where
the output dimension is the number of classes. The final output of SSM is the
average of the 4 outputs from the sub-classifiers.

The key intuition of our design is to partially share the given feature. The
first 1/4 channels are shared among all sub-classifiers. These channels are
forwarded 4 times and back-propagated 4 times. As they are most frequently
used channels, we expect these channels are trained to be the most important
key features. In contrast, the last 1/4 channels are used only by the last
sub-classifier, so they are expected to contain some additional features, such
as context information on the surrounding environments. We visualized the 4
splits of channels with the Grad-CAM visualization technique in Fig. 2 and
Fig. 3, and more analysis will be discussed in Sec. 5.

Algorithm 1 Split-and-Share Module

1:procedure SSM($features,num\\_channels$=2048, $num\\_heads$=4)

2: $n\leftarrow$INT$(num\\_channels$ / $num\\_heads$)

3: $v\leftarrow[(empty\\_list)$]

4: for $i$=1 to $num\\_heads$ do

5: $out\leftarrow features[:i*n]$

6: $out\leftarrow$BatchNorm$(out)$

7: $out\leftarrow$ReLU$(out)$

8: $out\leftarrow$FC$(out)$

9: $v$.append($out$)

10: end for

11: $result\leftarrow v$.sum() / $num\\_heads$

12: return $result$

13:end procedure

##  4 Experiments

In this section, we validate the efficacy of the proposed SSM on various
architectures, and analyze the effect of SSM in several aspects. First, we use
SSM upon ResNet and ResNeXt architectures in ImageNet-1K classification
dataset [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei]. SSM has
shown performance improvements in most cases, and details will be described in
Sec. 4.1. In Sec. 4.2, we describe the ablation studies of SSM.

###  4.1 ImageNet-1K classification

The ImageNet-1K dataset [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-
Fei] consists of 1.28 million training images and 50k validation datasets. We
follow the training details in ResNet [He et al.(2016)He, Zhang, Ren, and
Sun]. During training, the images are resized to 256x256 shape, and randomly
cropped to 224x224 patches with random horizontal flipping. During testing,
the images are also resized to 256x256 shape, and a single 224x224 patch is
cropped at the center. For both training and testing, images are normalized
with the mean and standard deviation of all pixels in the dataset. We adopt
He’s method [He et al.(2015)He, Zhang, Ren, and Sun] for network random
initialization. We use SGD optimizer with base learning rate 0.1 and batch
size of 256. The running rate is reduced by one-tenth at epoch 30 and 60, and
the total number of epochs is 90. The weight decay value is set to 0.0001 and
the momentum value is set to 0.9.

Architecture | Dataset | Epoch | Top-1 Acc
---|---|---|---
ResNet-18 [He et al.(2016)He, Zhang, Ren, and Sun] | ImageNet-1K | 90 | 70.04%
ResNet-18 [He et al.(2016)He, Zhang, Ren, and Sun] \+ SSM | ImageNet-1K | 90 |  70.05%(+0.01%)
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] | ImageNet-1K | 90 | 75.65%
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] \+ SSM | ImageNet-1K | 90 |  76.68%(+1.03%)
ResNet-101 [He et al.(2016)He, Zhang, Ren, and Sun] | ImageNet-1K | 90 | 76.62%
ResNet-101 [He et al.(2016)He, Zhang, Ren, and Sun] \+ SSM | ImageNet-1K | 90 |  77.93%(+1.31%)
ResNeXt50 [Xie et al.(2017)Xie, Girshick, Dollár, Tu, and He] | ImageNet-1K | 90 | 77.19%
ResNeXt50 [Xie et al.(2017)Xie, Girshick, Dollár, Tu, and He] \+ SSM | ImageNet-1K | 90 |  77.96%(+0.77%)
ResNeXt101 [Xie et al.(2017)Xie, Girshick, Dollár, Tu, and He] | ImageNet-1K | 90 | 78.46%
ResNeXt101 [Xie et al.(2017)Xie, Girshick, Dollár, Tu, and He] \+ SSM | ImageNet-1K | 90 |  79.68%(+1.22%)

Table 1: Classification results on ImageNet-1K. Single-crop validation errors
are reported.

The experiment result is summarized in Table 1. SSM has consistently improved
performance in all the architectures, except ResNet-18 that does not improve.
The distinctive difference between ResNet-18 and other architectures is that
the final feature of ResNet-18 has 512 channels, while others have 2048
channels. Therefore, we assume that the number of channels in the final
feature should be large enough for SSM to be effective.  In all architectures
except ResNet-18, the performance improvement is significant.  Furthermore,
the absolute improvements in larger architectures are greater than the smaller
ones. ResNet-101 improves 1.31% in the top-1 accuracy, while ResNet-50
improves 1.03%; ResNeXt-101 improves 1.22%, while ResNeXt-50 improves 0.77%.

###  4.2 Ablation studies and analysis

####  4.2.1 Training scheme for sub-classifiers

There are 2 simple ways to train the 4 sub-classifiers: apply the
classification loss to individual sub-classifier outputs, or apply the loss to
the average of the outputs. The former one requires each sub-classifier to
independently learn to classify, and then ensemble the 4 sub-classifiers; the
latter one allows the sub-classifiers to jointly learn to classify. The
results are summarized in Table 2. When individually trained, the sub-
classifiers’ performances are much higher than the jointly trained ones.
Interestingly, the final ensemble performance is significantly higher in the
jointly trained one. The result indicates that jointly training the sub-
classifiers will encourage the sub-classifiers to have different roles to
create synergy, and thus the final ensemble performance is higher than the
independently trained one.

Architecture | Dataset | FC1 Acc | FC2 Acc | FC3 Acc | FC4 Acc | Averaging Acc
---|---|---|---|---|---|---
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] \+ SSM | ImageNet-1K | 65.24% | 73.24% | 75.09% | 1.02% |  76.68%
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] \+ SSM-individual | ImageNet-1K | 75.60% | 75.11% | 76.18% | 74.77% | 75.38%

Table 2: Results of ImageNet-1K classification according to two different
training schemes. SSM is the result of training with the loss given to the
averaged output; SSM-individual is the result of training each output
independently.

####  4.2.2 Is SSM a new way of ensemble?

Ensemble is a simple technique to further boost performance by combining
multiple models that have different random initializations. The sub-
classifiers of SSM may resemble the ensemble technique, and there may be
concerns that SSM benefits from the ensemble-like effect and thus may not
benefit from ensemble. However, we argue that SSM is not simply an ensemble
method, and we validate that SSM-augmented models can further benefit from
ensemble.  We train two ResNet-50 models and two ResNet-50 + SSM models with
different initializations, and test if SSM can further benefit from ensembles.
The results are summarized in Table 3. The two ResNet-50 + SSM models’
accuracies are 76.37% and 76.68%, and the ensembled accuracy is 78.04%, which
is 1.35% higher. The improvement is a little less than the ResNet-50 ensemble,
but it may be simply due to the performance saturation, and the 1.35% is still
a significant improvement by ensemble. Therefore, through this experiment we
show that SSM-augmented models can further benefit from ensemble.

Architecture | Dataset | Epoch | Top-1 Accs | Ensembled Acc
---|---|---|---|---
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] | ImageNet-1K | 90 | 75.60%, 75.65% | 77.25% (+1.60%)
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] \+ SSM | ImageNet-1K | 90 | 76.37%, 76.68% | 78.04% (+1.35%)

Table 3: Results of ensemble classification in ImageNet-1K. All experiments
are conducted in the same environment. We separately train the two models for
two times each.

####  4.2.3 Is the improvement simply due to parameter increases?

Finally, we show that the efficacy of SSM is not simply due to parameter
increase. To verify this, we further train two models with more parameters by
adding more parallel classifiers. As shown in Table 4, the base ResNet-50 has
25.55M parameters, and ResNet-50 + SSM has 28.58M parameter, so the parameter
overhead is 3.03M. One fully connected layer has 2.05M parameters, so we add
one or two parallel fully connected layers to the baseline ResNet-50.
ResNet-50 (2FC) and ResNet-50 (3FC) are the new comparison methods that brings
additional parameters in the classifier part, like SSM. The result is
summarized in Table 4. A simple increase in parameters, like ResNet-50 (2FC)
and (3FC), does not improves the performance much, but SSM does bring a
significant improvement. Therefore, we argue that the performance improvement
is not simply due to parameter increase, but due to the feature exploiting
characteristics of SSM.

Architecture | Dataset | Epoch | Top-1 Acc | Parameters
---|---|---|---|---
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] | ImageNet-1K | 90 | 75.65% | 25.55M
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] \+ SSM | ImageNet-1K | 90 | 76.68% | 28.63M
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] (2FC) | ImageNet-1K | 90 | 75.91% | 27.60M
ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] (3FC) | ImageNet-1K | 90 | 75.67% | 29.65M

Table 4: The result of the parameter increase in ImageNet-1K. In all
experiments, FC was added vertically, and all were ensembled using the
averaging method.

##  5 Qualitative analysis

The key intuition of SSM is to partially share the features among different
sub-classifiers. As described in Sec. 3, the first 1/4 channels are shared
among all sub-classifiers, and the last 1/4 channels are used only by the last
sub-classifier. The first 1/4 channels are the most frequently feed-forwarded
and back-propagated, and are expected to contributes mostly to the final
prediction of SSM. In short, our hypothesis is that the degree of sharing is
positively correlated to the importance of the feature. Therefore, the first
1/4 channels are expected to contain the key features to classify among the
target classes, and the last 1/4 channels are expected to contain additional
features such as contextual information.

Figure 3: Additional Grad-CAM visualization results.

We qualitatively analyze the channels with the Grad-CAM visualization. Fig. 2
and Fig. 3 shows input images from the validation set and the overlaid Grad-
CAM heatmaps with respect to the ground-truth labels. ResNet-50+SSM is the
visualization target. To analyze whether the feature splits have learned
differently, the visualizations are generated for each 1/4 split of channels,
instead of the full feature. The column ‘Channel 0~511’ denotes the Grad-CAM
of the first 1/4 channels with respect to the first sub-classifier. The column
‘Channel 512~1023’ denotes the visualization of the second 1/4 channels w.r.t.
the second sub-classifier, and so on. While the input to the second sub-
classifier is the first 1/2 channels, we visualized the second 1/4 to
explicitly compare the semantics learned in each 1/4 channels.

The samples in Fig. 2 and Fig. 3 supports our intuition. The Grad-CAMs for the
first sample in Fig. 2 demonstrate that the first 1/4 channels focus on the
ground-truth ‘guitar’ location, and the last 1/4 channels focus on the
corresponding context, which in this case is the guitar player. The two
intermediate Grad-CAMs gradually changes from the key feature of the guitar to
the corresponding context of the guitar player. The Grad-CAMs of the second
sample in Fig. 2 also show that the first 1/4 channels focus on the fish, and
the last 1/4 channels focus on the corresponding context of the river. We
demonstrate more samples in Fig. 3. In summary, the Grad-CAM visualizations
for each split channels show that the most shared channels focus on the target
object, and the least shared channels focus on the corresponding context
information.

##  6 Discussion

We experimented with how much performance improvement would be achieved by
selecting the best inferred output of each output and using it for prediction
in the ResNet-50+SSM structure learned from the training set of ImageNet-1K.
We have not yet developed an algorithm to select the optimal FC, so we have
selected the optimal FC by ourselves, utilizing the label data of ImageNet-1K.
The result was that if FC could be ideally chosen, an additional 6%
performance improvement could be seen with up to 82.9% performance. Further
research on this part is believed to be possible.

##  7 Conclusion

We propose Split-and-Share Module (SSM). SSM is a classifier that improves the
performance of CNN networks. We apply the BatchNorm and ReLU to the shared
features extracted from the feature extractor, limiting the number of commonly
used and non-featured backprops, and have an effect of learning by placing
weights on important features. Through this process, features learned
according to importance have a classifier suitable for their capacity, and
averaging multiple outputs from the classifier for use in training and
testing. We verified SSM by applying CNNs of various structures in
ImageNet-1K, and showed significant performance improvement in all the
experiments. We also adopted Grad-CAM for qualitative analysis of SSM. Grad-
CAM results showed qualitatively that our SSM could learn according to the
importance of features as we intended. In addition, SSM divides features into
four groups and learns features that are common or non-common features, and
these characteristics are thought to be available in many areas of research.

## References

  * [Aflalo et al.(2020)Aflalo, Noy, Lin, Friedman, and Zelnik] Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik.  Knapsack pruning with inner distillation.  _arXiv preprint arXiv:2002.08258_ , 2020.
  * [Bengio et al.(2013)Bengio, Courville, and Vincent] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1798–1828, 2013.
  * [Chollet(2017)] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1251–1258, 2017.
  * [Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and Schiele] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3223, 2016.
  * [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * [He et al.(2015)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1026–1034, 2015.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * [Ilyas et al.(2019)Ilyas, Santurkar, Tsipras, Engstrom, Tran, and Madry] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry.  Adversarial examples are not bugs, they are features.  In _Advances in Neural Information Processing Systems_ , pages 125–136, 2019.
  * [Ioffe and Szegedy(2015)] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  _arXiv preprint arXiv:1502.03167_ , 2015.
  * [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , pages 1097–1105, 2012.
  * [Kuehne et al.(2011)Kuehne, Jhuang, Garrote, Poggio, and Serre] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.  Hmdb: a large video database for human motion recognition.  In _2011 International Conference on Computer Vision_ , pages 2556–2563. IEEE, 2011.
  * [Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer, 2014.
  * [Nair and Hinton(2010)] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Proceedings of the 27th international conference on machine learning (ICML-10)_ , pages 807–814, 2010.
  * [Real et al.(2019)Real, Aggarwal, Huang, and Le] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  In _Proceedings of the aaai conference on artificial intelligence_ , volume 33, pages 4780–4789, 2019.
  * [Selvaraju et al.(2017)Selvaraju, Cogswell, Das, Vedantam, Parikh, and Batra] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626, 2017.
  * [Simonyan and Zisserman(2014)] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * [Szegedy et al.(2015)Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, and Rabinovich] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1–9, 2015.
  * [Tan and Le(2019)] Mingxing Tan and Quoc V Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  _arXiv preprint arXiv:1905.11946_ , 2019.
  * [Xie et al.(2017)Xie, Girshick, Dollár, Tu, and He] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1492–1500, 2017.
  * [Zoph et al.(2018)Zoph, Vasudevan, Shlens, and Le] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8697–8710, 2018.