$(\boldsymbol{x}_{i},\boldsymbol{x}_{j})\in E$ if and only if
$\boldsymbol{x}_{i}$ and $\boldsymbol{x}_{j}$ share the same topic label.

Plane, Möbius, Sphere, & Torus Graphs. For these graphs, we take a more
involved practice to localize sentences onto specifiable simplicial manifolds,
then construct pre-training graphs via radius nearest neighbor graphs on those
manifolds. This involves several steps:

Localizing Sentences on Simplices

We can localize any sentence in our overall dataset onto a 2-simplex by
mapping them onto the (re-normalized) probabilities associated with their
top-3 topics. Doing this in this way means that the simplex on which they are
localized has vertices that correspond to possible topics among our set of 100
total topics.

Stitching Topic-simplices Into Manifolds

Given these topic-simplex localized sentences, we need to construct our
manifolds. To do so, we first produce any arbitrary simplicial tiling of a
2-manifold. With this tiling, all that remains to localize sentences onto the
manifold is to find a self-consistent mapping of topics to simplex vertices
(in the tiling) such that all topic-simplices induced by this mapping have
sufficiently many associated samples to enable roughly uniform sampling.

Sampling points

After finding a self-consistent mapping of topics to simplicial tiling
vertices that satisfies density requirements, we can then sample sentences
onto the manifold. To make this process more uniform, we also calculate the
relative entropy of each sentence (over the re-normalized probabilities of the
top-3 topics), bin those entropies into buckets, then sample first what
entropy bucket we wish to draw from such that the induced distribution of
sentence entropies is approximately uniform, then sample within that entropy
bucket.

Calculating on-Manifold Distances

Finally, with sentences sampled and localized onto a simplicial manifold, we
then need to compute approximate geodesic distances to enable building radius-
nearest-neighbor graphs over these sentences. To do so, we use an approximate
algorithm that considers only on-simplex distance (e.g., it does not consider
any curvature penalties) which is equivalent to calculating the distance
between any pair of points over the simplices presuming they were flattened
onto a plane (this flattening naturally does not preserve manifold topology,
but along only the shortest path between any particular set of two points it
is always possible to do so with a 2-manifold).

The above process describes how to produce a radius-nearest-neighbor graph for
any specifiable manifold using our topic-model outputs. We do this for
simplicial manifolds that correspond topologically to a simple plane (Plane),
a möbius strip (Möbius), a sphere (Sphere), and a torus (Torus).

Structural, Neighborhood, & Motifs Graphs. In order to form these examples, we
must (1) define our overall graphs, (2) featurize these graphs in a manner
that is reflective of different forms of graph structure, then (3) use these
featurizations to assign sentences to graph nodes to form our pre-training
dataset.

Graph Construction

We sample graphs as described in the main body, with a base cycle and motifs
distributed along that cycle evenly.

Node Featurization

Nodes in this graph are then assigned internal features based on three notions
of graph topology. For the “Homophily” label, a node $n$ is identified
according to an index-vector indicating which nodes in the graph are within
shortest-path distance 3 of $n$. For the “Motif” label, $n$ is identified
based on its membership either in the base cycle or any of the attached random
subgraphs. For the “Structural” label, $n$ is identified based on its graphlet
degree vector (of order 4). For structural and homophily features, categorical
labels are then produced by feeding these raw representations through a
$k$-means clustering algorithm.

Sentence Assignment

We assign sentences to nodes in multiple ways, so that we can produce datasets
that reflect each of the notions of graph structure discussed previously. In
particular, for either the homophily, motif, or structural labels, each
sentence topic is matched to a node label, then sentences are assigned
randomly to nodes in the graph with a matching topic label. Note that this
produces a dataset where the graph structure is only partially reflected by
the node’s features, which is itself another useful test of the SIPT system,
as it would not be useful if SIPT could only capture data in contexts where
the graph was perfectly reflected by the node features themselves.

####  E.1.2 Network Architecture & Hyperparameters

The Cliques and Mechanistic experiments in this domain use a shallow
Transformer model with 2 layers and 10 hidden units. The Manifold experiments
use a 3-layer Transformer model with 256 hidden units. Hyperparameters were
not tuned, but were chosen by-hand to produce as small a network as possible
while still permitting reasonable learning dynamics.

###  E.2 Full Details for Real-world Experiments

####  E.2.1 PT Datasets

The Proteins PT data (the Stanford tree-of-life dataset) lists no license on
their download page (https://snap.stanford.edu/tree-of-life/data.html), though
the associated github repository lists an MIT license. The Abstracts PT data
(the Microsoft Academic Graph dataset) is licensed with a Open Data Commons
Attribution License (ODC-By) v1.0 license. The Networks PT dataset, released
in the relevant source article Hu et al. (2020b) lists no specific data
license but releases the code (which contains the dataset files) under an MIT
license.

####  E.2.2 Proteins

PT Dataset.. We use a dataset of $\sim$1.5M protein sequences from the
Stanford Tree-of-life dataset (Zitnik et al., 2019).

PT Graph.. Two proteins are linked in $G_{\mathrm{PT}}$ if and only if they
are documented in the scientific literature to interact, according to the
tree-of-life dataset. This is an external knowledge graph.

FT Dataset/Tasks.. We use the TAPE FT benchmark tasks (Rao et al., 2019),
including Remote homology (RH), a per-sequence classification task to predict
protein fold category (metric: accuracy); Secondary structure (SS), a per-
token classification task to predict amino acid structural properties (metric:
accuracy); Stability (ST) & Fluorescence (FL), per-sequence, regression tasks
to predict a protein’s stability and fluorescence, respectively (metric:
Spearman’s $\rho$); and Contact prediction (CP), an intra-sequence
classification task to predict which pairs of amino acids are in contact in
the protein’s 3D conformation (metric: Precision at $L/5$).

Baselines.. We compare against the published TAPE model (Rao et al., 2019),
which uses a LM task alone, as our per-token comparison point and the PLUS
(Min et al., 2020) model, which optimizes for LM and supervised classification
jointly, for our per-sample comparison point.

####  E.2.3 Abstracts

PT Dataset.. We use a dataset of $\sim$650K free-text scientific article
abstracts from the Microsoft Academic Graph (MAG) dataset (Wang et al., 2019a;
Hu et al., 2020a).

PT Graph.. Two abstracts are linked in $G_{\mathrm{PT}}$ if and only if their
corresponding papers cite one another. This is a self-supervised graph.

FT Dataset/Tasks.. We use the SciBERT benchmark tasks (Beltagy et al., 2019),
including Paper field (PF), SciCite (SC), ACL-ARC (AA), and SciERC Relation
Extraction (SRE), all of which are per-sentence classification problems
(metric: Macro-F1). PF tasks models to predict a paper’s area of study from
its title, SC & AA tasks both are to predict an “intent” label for citations,
and SRE is a relation extraction task.

Baselines.. We compare against the published SciBERT model (Beltagy et al.,
2019) as our per-token comparison, and lack an associated per-sample
comparison as we don’t know of any published per-sample models in the academic
papers modality.

####  E.2.4 Networks

PT Dataset.. We use a dataset of $\sim$70K protein-protein interaction (PPI)
ego-networks here, sourced from (Hu et al., 2020b). Each individual sample
here describes a single protein, realized as a biological network (i.e., an
attributed graph) corresponding to the ego-network about that protein (i.e., a
small subgraph containing all nodes within the target protein) in a broader
PPI graph. Unlike our other domains, this domain does not contain sequences.

PT Graph.. The dataset from (Hu et al., 2020b) is labeled with the presence or
absence of any of 4000 protein gene ontology terms associated with each the
central protein in each PPI ego-network. Leveraging these labels, two PPI ego-
networks are linked in $G_{\mathrm{PT}}$ if and only if the hamming distance
between their observed label vectors is no more than $9$. This is an
alternate-representation nearest-neighbor graph.

FT Dataset/Tasks.. Our FT task is the multi-label binary classification of the
40 gene-ontology term annotations (metric: macro-AUROC) used in (Hu et al.,
2020b). We use the PT set for FT training, and evaluate the model on a held-
out random 10% split.

Baselines.. We compare against both attribute-masking (Hu et al., 2020b) and
multi-task PT.

####  E.2.5 Fine-tuning Task Descriptions

Proteins. The tasks in the TAPE benchmark Rao et al. (2019) on which we test
are described more fully below. All these datasets are publicly available. All
datasets can be obtained directly on TAPE’s github
(https://github.com/songlab-cal/tape#data) which lists no licenses for these
datasets, though the overall github is released under a BSD 3-Clause ”New” or
”Revised” License.

Remote Homology

This is a per-sequence, multi-class classification problem, evaluated using
accuracy, which tasks a model to predict a protein fold category at a per-
sequence level. This task’s dataset contains 12,312/736/718 train/val/test
proteins, and is originally sourced from Hou et al. (2018).

Secondary Structure

This is a per-token, multi-class classification problem, evaluated using
accuracy, which tasks a model to predict the structural properties of each
amino acid in the final, folded protein. This task’s dataset contains
8,678/2,170/513 train/val/test proteins, and is originally sourced from
Klausen et al. (2019).

Stability

This is a per-sequence, continuous regression problem, evaluated using
Spearman correlation coefficient, which tasks a model to predict the protein’s
stability in response to environmental conditions. This task’s dataset
contains 53,679/2,447/12,839 train/val/test proteins, and is originally
sourced from Rocklin et al. (2017).

Fluorescence

This is a per-sequence, continuous regression problem, evaluated using
Spearman correlation coefficient, which tasks a model to predict how brightly
a protein will fluoresce. This task’s dataset contains 21,446/5,362/27,217
train/val/test proteins, and is originally sourced from Sarkisyan et al.
(2016).

Abstracts. The tasks in the SciBERT benchmark Beltagy et al. (2019) on which
we test are described more fully below. All tasks here are per-sentence,
multi-class classification problems (i.e., we do not study any per-token
tasks), and all are evaluated in Macro-F1 (out of 1). All FT datasets can be
obtained from the SciBERT github (https://github.com/allenai/scibert), which
lists no dataset specific licenses but is itself released with an Apache-2.0
license.

Paper Field

This problem tasks models to predict a paper’s area of study given its title.
This task’s dataset contains 84,000/5,599/22,399 train/val/test sentences.
Though original derived from the MAG Wang et al. (2019a), it was to the best
of our knowledge formulated into this task format by SciBERT directly Beltagy
et al. (2019).

SciCite

This problem tasks models to predict an “intent” label for sentences that cite
other scientific works within academic articles. This task’s dataset contains
7,320/916/1,861 train/val/test sentences, and is originally sourced from Cohan
et al. (2019).

ACL-ARC

This problem tasks models to predict an “intent” label for sentences that cite
other scientific works within academic articles. This task’s dataset contains
1,688/114/139 train/val/test sentences, and is originally sourced from Jurgens
et al. (2018).

Networks. The Networks FT task is a multi-task, binary classification task
that is defined as follows. Recall that the dataset here consists of PPI ego-
networks, which means more concretely that an individual sample input to the
model is an attributed graph $\boldsymbol{x}$ which contains a central node,
corresponding to a protein, along with the ego-graph surrounding that node in
a larger PPI graph. This ego-graph can thus be seen to correspond to the
central protein, and the FT and PT tasks leverage this association, as both of
which flag whether or not that central protein is associated with particular
gene-ontology (GO) terms (annotations relating to protein properties or
function applied in the literature). The PT tasks contain 4000 possible GO
annotations, but the FT tasks correspond to a smaller set of only 40 GO terms,
chosen as they were of greater interest than the full set. See the original
source (Hu et al. (2020b)) for more information and full details.

####  E.2.6 Architecture & Hyperparameters

The architectures of our encoders for the Proteins and Abstracts domains are
fully determined from our source models in TAPE Rao et al. (2019) and SciBERT
Beltagy et al. (2019). In particular, for proteins and scientific articles, we
use a 12-layer Transformer with a hidden size of 768, intermediate size of
3072, and 12 attention heads. Provided TAPE and SciBERT tokenizers are also
used. A single linear layer to the output dimensionality of each task is used
s the prediction head, taking as input the output of the final layer’s [CLS]
token as a whole-sequence embedding. We also tested either pre-training for a
single or for four additional epochs, based on validation set performance, and
ultimately used a single epoch for proteins and four for scientific articles.

For the Networks domain, we match the architecture used in the original source
Hu et al. (2020b) for the mask model runs, save that for computational
efficiency we scale the batch-size up as high as it can go, then
proportionally scale up the learning rate to account for the larger batch
size. This corresponds to a batch size of 1024, learning rate of 0.01, a GCNN
encoder type of GIN, embedding dimensions of 300, 5 layers, 10% dropout, mean
pooling, and a JK strategy of “last”.

Fine-tuning hyperparameters (learning rate, batch size, and number of epochs)
were determined based on a combination of existing results, hyperparameter
tuning, and machine limitations. On proteins, most hyperparameters were set to
follow those reported for a LM PT model in McDermott et al. (2021), though
additional limited hyperparameter searches were performed to validate that
these choices were adequate. As the original source for these hyperparameters
was an LM PT model, any bias here should be _against_ SIPT, meaning this is a
conservative choice. Early stopping (based on the number of epochs without
observing improvement in the validation set performance) was employed and
batch size was set as large as possible given the limitations of the
underlying machine. For the PLUS reproduction, we additionally compared
hyperparameters analogous to the reported PLUS hyperparameters for other tasks
as well as analogous to our hyperparameters for other tasks and used those
that performed best on the validation set. For scientific articles, we
performed grid search to optimize downstream task performance on the
validation set, with learning rate varying between 5e-6 and 5e-5 and number of
epochs between 2 and 5. The same grid search was used in original SciBERT
system. We additionally match the SciBERT benchmark by applying dropout of
0.1, using the Adam optimizer with linear warm-up and decay, a batch size of
32, and no early stopping. For the Networks, FT hyperparameters were again
chosen to match the original source model Hu et al. (2020b) save the increase
in batch size and learning rate. No additional hyperparameter search was
performed.

Final hyperparameters for each downstream task are shown in Tables 5 for
proteins and 6 for scientific articles.

Table 5: Final hyperparameters for our Proteins domain. All tasks used 200
total epochs and performed early stopping after 25 epochs of no validation set
improvement. LR, learning rate.
Task | Batch Size | LR
---|---|---
Remote Homology | 16 | 1e-5
Fluorescence | 128 | 5e-5
Stability | 512 | 1e-4
Secondary Structure | 16 | 1e-5
Table 6: Final hyperparameters for our Abstracts dataset. All models used a
batch size of 32 and no early stopping, to match the original SciBERT paper
Beltagy et al. (2019). LR, learning rate. A / B = [LM PT Hyperparameter] /
[SIPT Hyperparameter].
Task | # Epochs | LR
---|---|---
Paper Field | 2 | 5e-5
ACL-ARC | 4/5 | 5e-5
SciCite | 3/2 | 1e-5

###  E.3 Implementation and Compute Environment

We leverage PyTorch for our codebase. FT Experiments and Networks PT were run
over various ubuntu machines (versions ranged from 16.04 to 20.04) with a
variety of NVIDIA GPUs. Proteins and Abstracts PT runs were performed on a
Power 9 system, each run using 4 NVIDIA 32 GB V100 GPUs with InfiniBand at
half precision.

##  F Further Empirical Results

###  F.1 Raw Results

In Tables 7 and 8, we show the raw FT results for all tasks in the Proteins
and Abstracts domains, respectively.

Model | RH | FL | ST | SS | CP
---|---|---|---|---|---
TAPE | 21% | 0.68 | 0.73 | 73% | 0.32
PLUS | 19.8%$\pm$1.7∗ | 0.63 | 0.76 | 73% | N/A
LM PT | 23.8%$\pm$1.1 | 0.67$\pm$0.00 | 0.76$\pm$0.02 | 73.9%$\pm$0.0 | 0.38
SIPT-C | 25.1%$\pm$0.6 | 0.68$\pm$0.00 | 0.77$\pm$0.01 | 73.9%$\pm$0.0 | 0.38
SIPT-M | 26.6%$\pm$1.0 | 0.68$\pm$0.00 | 0.76$\pm$0.01 | 74.2%$\pm$0.1 | 0.39
Table 7:  Results of the TAPE Transformer Rao et al. (2019), the PLUS Transformer Min et al. (2020) (∗: our measurements), our LM PT baseline, and two SIPT variants (“-C” indicates the contrastive loss, “-M” the multisimilarity loss). Higher is better.  Model | PF | SC | AA | SRE
---|---|---|---|---
SciBERT | 0.66 | 0.85 | 0.71 | 0.80
LM PT | 0.66$\pm$0.0 | 0.85$\pm$0.01 | 0.70$\pm$0.05 | 0.80$\pm$0.01
SIPT-C | 0.66$\pm$0.0 | 0.86$\pm$0.01 | 0.76$\pm$0.02 | 0.81$\pm$0.00
SIPT-M | 0.66$\pm$0.0 | 0.85$\pm$0.00 | 0.73$\pm$0.05 | N/A
Table 8:  Results of the original SciBERT Beltagy et al. (2019) model, our own
LM PT baseline, and two SIPT variants (“-C” indicates the contrastive loss,
“-M” the multisimilarity loss). Higher is better.

###  F.2 A deeper dive into Proteins vs. Abstracts

For the Proteins and Abstracts dataset, to choose the optimal value of
$\lambda_{\mathrm{SI}}$ for use at PT time, we pre-trained several models and
evaluated their efficacy in a link retrieval task on $G_{\mathrm{PT}}=(V,E)$.
In particular, we score a node embedder $f$ by embedding all nodes $n\in V$ as
$f(n)$, then rank all other nodes $n^{\prime}$ by the euclidean distance
between $f(n)$ and $f(n^{\prime})$, and assess this ranked list via IR metrics
including label ranking average precision (LRAP), normalized discounted
cumulative gain (nDCG), average precision (AP), and mean reciprocal rank
(MRR), where a node $n^{\prime}$ is deemed to be a “successful” retrieval for
$n$ if $(n,n^{\prime})\in E$. In this way, note that we choose
$\lambda_{\mathrm{SI}}$ in a manner that is independent of the fine-tuning
task, and can be determined solely based on the PT data. Final results for
these experiments are shown in Table 9 for the proteins dataset and Table 10
for scientific articles.

In both settings, we compare the following models.

Random

Nodes are embedded with random vectors, to assess chance performance.

Initial Model

Nodes are embedded with the base pre-trained model we build on in our
experiments without further modifications. This model is TAPE Rao et al.
(2019) for proteins and SciBERT Beltagy et al. (2019) for scientific articles.

LM PT

Nodes are embedded with the final encoder after additional pre-training on our
graph-augmented datasets, but without any SIPT (i.e.,
$\lambda_{\mathrm{SI}}=0$).

CS RoBERTa

(for scientific articles only) Nodes are embedded via Gururangan et al.
(2020)’s DAPT CS RoBERTa model, which is another LM PT model over scientific
abstracts which performed very well on ACL-ARC, the task on which SIPT does
best in scientific articles.

SIPT

(for various values of $\lambda_{\mathrm{SI}}$). Nodes are represented via
SIPT PT models at the specified weighting. For proteins, all SIPT models are
initialized from TAPE, but for scientific articles, we test against both
initializing from SciBERT and from CS RoBERTa (as both are just different,
domain-specific LM PT models).

Note that in addition to the discrepancy in the magnitude of improvement (over
scientific articles, average precision goes from 12.9% to 14.2%, vs. 2.4% to
3.5% on proteins, which is proportionally much more significant), we can also
see that SIPT improves retrieval performance over the baselines for proteins
much more than it does for scientific articles. This is, admittedly, largely
due to Gururangan et al. (2020)’s CS RoBERTa model’s surprisingly good
performance without any modifications, however as we also compare SIPT pre-
trained from a CS RoBERTa model and it does not demonstrate significant
improvements, we still feel this is a fair comparison. These findings are
consistent with our hypothesis that SIPT will offer more significant
advantages on non-natural language domains.

Table 9: PT set link-retrieval performance for a random baseline, the raw TAPE
model, and SIPT for various weighting parameters $\lambda_{\mathrm{SI}}$ on
the dataset of protein sequences. LRAP, label ranking average precision; nDCG,
normalized discounted cumulative gain; AP, average precision; MRR, mean
reciprocal rank. Higher values indicate better performance. Highlighted in
grey are realizations of SIPT framework that yield better results than the
strongest baseline, providing evidence that incorporating sequence-level
relational information into PT (i.e., $\lambda_{\mathrm{SI}}>0$) leads to
improved performance.
Method | $\lambda_{\mathrm{SI}}$ | LRAP | nDCG | AP | MRR
---|---|---|---|---|---
Random Baseline | N/A | 0.88% | 27.1% | 0.88% | 0.003
TAPE Rao et al. (2019) | N/A | 8.50% | 34.9% | 2.41% | 0.226
LM PT Baseline | 0 | 8.92% | 38.0% | 2.33% | 0.238
SIPT (TAPE Initialized) | 0.01 | 9.69% | 39.1% | 2.56% | 0.254
0.10 | 10.95% | 39.4% | 3.46% | 0.260
0.50 | 10.54% | 40.3% | 3.43% | 0.246
0.90 | 10.12% | 39.0% | 3.16% | 0.237
0.99 | 14.50% | 37.5% | 3.13% | 0.236
Table 10: PT set link-retrieval performance for a random baseline, the raw
SciBERT model, and SIPT for various weighting parameters
$\lambda_{\mathrm{SI}}$ on the scientific articles dataset. LRAP, label
ranking average precision; nDCG, normalized discounted cumulative gain; AP,
average precision; MRR, mean reciprocal rank. Higher values indicate better
performance. Highlighted in grey are realizations of SIPT framework that yield
better results than the strongest baseline, providing evidence that
incorporating sequence-level relational information into PT (i.e.,
$\lambda_{\mathrm{SI}}>0$) leads to improved performance.
Method | $\lambda_{\mathrm{SI}}$ | LRAP | nDCG | AP | MRR
---|---|---|---|---|---
Random Baseline | N/A | 0.89% | 26.0% | 0.27% | 0.016
SciBERT Beltagy et al. (2019) | N/A | 17.22% | 52.8% | 5.16% | 0.272
LM PT Baseline (SciBERT initialized) | 0 | 16.79% | 35.4% | 5.00% | 0.271
DAPT CS RoBERTa Gururangan et al. (2020) | N/A | 32.56% | 50.3% | 12.86% | 0.459
LM PT Baseline (CS RoBERTa initialized) | 0 | 30.58% | 48.3% | 12.36% | 0.438
SIPT (SciBERT initialized) | 0.01 | 42.26% | 58.7% | 14.23% | 0.536
0.10 | 34.73% | 52.5% | 9.39% | 0.457
0.50 | 32.85% | 50.8% | 8.37% | 0.438
0.90 | 31.61% | 49.8% | 7.82% | 0.426
0.99 | 30.72% | 49.0% | 6.80% | 0.415
SIPT (CS RoBERTa initialized) | 0.01 | 33.32% | 51.2% | 8.61% | 0.448
0.10 | 25.46% | 44.4% | 5.88% | 0.359
0.50 | 25.08% | 44.0% | 6.08% | 0.355
0.90 | 22.43% | 41.6% | 4.27% | 0.317
0.99 | 22.38% | 41.5% | 4.68% | 0.316