# Graph Barlow Twins: A self-supervised representation learning framework for
graphs

Piotr Bielak
Department of Computational Intelligence
Wroclaw University of Science and Technology
Wrocław, Poland
<EMAIL_ADDRESS>
&Tomasz Kajdanowicz
Department of Computational Intelligence
Wroclaw University of Science and Technology
Wrocław, Poland
<EMAIL_ADDRESS>
&Nitesh V. Chawla
Department of Computer Science and Engineering
University of Notre Dame
Notre Dame, IN, USA
<EMAIL_ADDRESS>

###### Abstract

The self-supervised learning (SSL) paradigm is an essential exploration area,
which tries to eliminate the need for expensive data labeling. Despite the
great success of SSL methods in computer vision and natural language
processing, most of them employ contrastive learning objectives that require
negative samples, which are hard to define. This becomes even more challenging
in the case of graphs and is a bottleneck for achieving robust
representations. To overcome such limitations, we propose a framework for
self-supervised graph representation learning – Graph Barlow Twins, which
utilizes a cross-correlation-based loss function instead of negative samples.
Moreover, it does not rely on non-symmetric neural network architectures – in
contrast to state-of-the-art self-supervised graph representation learning
method BGRL. We show that our method achieves as competitive results as BGRL,
best self-supervised methods, and fully supervised ones while requiring
substantially fewer hyperparameters and converging in an order of magnitude
training steps earlier.

##  1 Introduction

Graph representation learning has been intensively studied for the last few
years, having proposed various architectures and layers, like GCN (13), GAT
(27), GraphSAGE (10) etc. A substantial part of these methods was introduced
in the semi-supervised learning paradigm, which requires the existence of
expensive labeled data (e.g. node labels or whole graph labels). To overcome
this, the research community has been exploring unsupervised learning methods
for graphs. This resulted in a variety of different approaches including:
shallow ones (DeepWalk (18), Node2vec (8), LINE (23)) that often ignore the
feature attribute richness, focusing only on the structural graph information;
and graph neural network based methods (DGI (28), GAE, VGAE (12)) that build
the representation upon node or graph features, achieving state-of-the-art
performance in those days.

Recently self-supervised paradigm is the most emerging branch of unsupervised
graph representation learning and gathers current interest and strenuous
research effort towards better results. The most prominent methods were
developed around the contrastive learning approach, such as GCA (32), GraphCL
(29), GRACE (31) or DGI (28). Although contrastive methods are popular in many
machine learning areas, including computer vision and natural language
processing, their fundamental limitation is the need for negative samples.
Consequently, the sampling procedure for negative examples highly affects the
overall quality of the embeddings. In terms of images or texts, the definition
of negative samples might seem not that problematic, but in the case of graphs
there is no clear intuition. For instance, what is the negative counterpart
for a particular node in the graph, should it be a node that is not a direct
neighbor, or a node that is in a different graph component? There are multiple
options available, but the right choice strictly dependent on the downstream
task.

Researchers have already tackled the problem of building so-called negative-
sample-free methods. In research being conducted in computer vision they
obtained successful results with methods such as BYOL (7), SimSiam (3) or
Barlow Twins (30). These models utilize siamese network architectures with
various techniques, like gradient stopping, asymmetry or batch and layer
normalizations, to prevent collapsing to trivial solutions. Based on BYOL,
(24) proposed the Bootstrapped Representation Learning on Graphs (BGRL)
framework. It utilizes two encoders: an online and a target network. The
former one passes the embedding vectors to a predictor network, which tries to
predict the embeddings from the target encoder. The loss is measured as the
cosine similarity and the gradient is backpropagated only through the online
network (gradient stopping mechanism). The target encoder is updated using an
exponential moving average of the online encoder weights. Such setup has been
shown to produce graph representation vectors that achieve state-of-the-art
performance in node classification using various benchmark datasets.
Notwithstanding, assuming asymmetry between the network twins (such as the
predictor network, gradient stopping, and a moving average on the weight
updates) the method is conceptually complex.

Employing a symmetric network architecture would seem more intuitive and
reasonable, hence in this paper, we propose Graph Barlow Twins (G-BT), a self-
supervised graph representation learning framework, which computes the
embeddings cross-correlation matrix of two distorted views of a single graph.
Such approach was initially introduced for image representation learning as
the Barlow Twins model (30). The utilized network architecture is fully
symmetric and does not need any special techniques to build non trivial
embedding vectors. The distorted graph views are passed through the same
encoder which is trained using the backpropagated gradients (in a symmetrical
manner).

Our main contributions can be summarized as follows:

  1. 1.

We propose a self-supervised graph representation learning framework Graph
Barlow Twins. It is built upon the recently proposed Barlow Twins loss, which
utilizes the embedding cross-correlation matrix of two distorted views of a
graph to optimize the representation vectors. Our framework neither requires
using negative samples (opposed to most other self-supervised approaches) nor
it introduces any kind of asymmetry in the network architecture (like state-
of-the-art BGRL). Moreover, our architecture is converges an order of
magnitude faster than all other state-of-the-art methods. We also we relate
the method to Hilbert-Schmidt Independence Criterion maximization (26), which
achieved comparable performance to the original Barlow Twins method.

  2. 2.

We evaluate our framework (with both loss functions) in node classification
tasks: (1) for 5 smaller benchmark datasets in a transductive setting, (2)
using the ogbn-arxiv dataset from the Open Graph Benchmark (also in the
transductive setting), and finally (3) for multiple graphs in the inductive
setting using the PPI (Protein-Protein Interaction) dataset. We use both GCN-
based encoders as well as a GAT-based one. We observe that our method achieves
analogous results compared to state-of-the-art methods.

  3. 3.

We make the code of both our models as well as experimental pipeline publicly
available at: https://github.com/pbielak/graph-barlow-twins.

##  2 Related works

##### Self-supervised learning

The idea of self-supervised learning has a long history. Introduced in the
early work of Schmidhuber (20) has more than 30 years of exploration and
research now. Recently self-supervised learning was again rediscovered and
found a broad interest, especially in computer vision and natural language
processing. One of the most prominent SSL methods for image representation
learning, Bootstrap Your Own Latent, BYOL (7), performs on par or better than
the current state of the art on both transfer and semi-supervised benchmarks.
It relies on two neural networks that interact and learn from each other. From
an augmented view of an image, it trains the first one to predict the target
network representation of the same image under a different view. At the same
time, it updates the second network with a slow-moving average of the first
network. Another approach to image representation SSL implements simple
siamese networks, namely SimSiam (3). It achieves comparative results while
not demanding negative samples, large batches, nor momentum encoders. Authors
emphasize collapsing solutions for the loss and structure but show how a stop-
gradient operation plays an essential role in preventing it. Recent method,
Barlow Twins (30), advances the SSL field with a new objective function that
naturally avoids collapses by measuring the cross-correlation matrix between
the outputs of two twin, identical networks fed with distorted versions of a
sample, and makes it as close to the identity matrix as possible.
Representations of distorted versions of samples are then expected to be
similar, reducing the redundancy between them. What differentiates the method
is that it does not require large batches or asymmetry between the network
twins. It outperforms previous methods on ImageNet for semi-supervised
classification.

##### Graph representation learning

Learning the representation also spreads to other domains. The graph embedding
problem has also attracted much attention from the research community
worldwide in recent years. Plenty of methods have been developed, each focused
on a different aspect of network embeddings, such as proximity, structure,
attributes, learning paradigm or scalability. There exist plenty of shallow
methods, among others DeepWalk (18), Node2vec (8) or LINE (23), that use a
simple notion of graph coding through random walks or on encoder-decoder
objectives that optimize first and second-order node similarity. More complex
graph neural networks, such as GCN(13) or GraphSAGE (10) implements the basic
encoder algorithm with various neighborhood aggregation. Following the
extension, graph attention network GAT (27) leverages masked self-attentional
layers to address the shortcomings of graph convolutions and their troublesome
approximations.

##### Self-supervised graph representation learning

Inspired by the success of contrastive methods in vision and NLP, the
procedures were also adapted to graphs. Early DGI (28) employs GNN to learn
node embeddings and obtains the graph embedding via a readout function and
maximizes the mutual information between node embeddings and the graph
embedding by discriminating nodes in the original graph from nodes in a
corrupted graph. GCA (32) studied various augmentation procedures. GRACE (31)
creates two augmented versions of a graph, pulls together the representation
of the same node in both graphs, and pushes apart representations of every
other node. Recent GraphCL (29) method is another example of representative
approach using contrastive learning. All the previous methods use negative
sampling approaches for the embedding optimization, yet such setting has a
high complexity. To overcome this, BGRL (24) proposed to use an approach that
does not rely on negative samples. It uses two kinds of encoder networks
(online and target), introducing a non-intuitive asymmetric pipeline
architecture, but provides state-of-the-art SSL results. Moreover, it relies
on several techniques to prevent trivial solutions (gradient stopping,
momentum encoder). A concurrent approach to BGRL is DGB (2).

##  3 Proposed framework

Figure 1: The processing pipeline of our proposed Graph Barlow Twins
framework. A given input graph $\mathcal{G}$ is transformed using an
augmentation function, resulting in two views: $\mathcal{G}^{(1)}$ and
$\mathcal{G}^{(2)}$. Both views are passed through the same GNN encoder
$f_{\theta}$, resulting in two embedding matrices $Z^{(1)}$, $Z^{(2)}$. The
cross-correlation matrix $\mathcal{C}$ is computed based on those embeddings.
The loss function compares $\mathcal{C}$ to the identity matrix.

Motivated by the emerging self-supervised learning paradigm and its recent
applications in graph representation learning (BGRL (24)), we propose Graph
Barlow Twins – a framework that builds node embeddings using a symmetric
network architecture and an empirical cross-correlation based loss function.
The overall pipeline of our framework is shown in Figure 1. The consecutive
processing steps can be described as follows:

##### Graph data

We represent a graph $\mathcal{G}$ with nodes $\mathcal{V}$ and edges
$\mathcal{E}$ as the tuple: $(X,A)$, where
$X\in\mathbb{R}^{|\mathcal{V}|\times k}$ is the node feature matrix and $k$ is
feature dimensionality; $A\in\\{0,1\\}^{|\mathcal{V}|\times|\mathcal{V}|}$ is
the adjacency matrix, such that $A_{i,j}=1\;\text{iff}\;(i,j)\in\mathcal{E}$.
In the general case, a graph could also have associated edge features or graph
level features, but for simplicity we omit those here. Nevertheless, these
could also be used in our framework, as long as the encoder can make use of
such features (see below).

##### Generating graph views via augmentation

Following other works (24; 32; 29; 31), we select two kinds of augmentations –
edge dropping and node feature masking – and generate two views of the input
graph $\mathcal{G}^{(1)}$ and $\mathcal{G}^{(2)}$. In the edge dropping case,
we remove edges according to a generated mask of size $|\mathcal{E}|$ (number
of edges in the graph) with elements sampled from the Bernoulli distribution
$\mathcal{B}(1-p_{A})$. When it comes to masking node features, we employ a
similar scheme and generate a mask of size $k$ also sampled from the Bernoulli
distribution $\mathcal{B}(1-p_{X})$. Note that we mask node features at the
scale of the whole graph, i.e. the same features are masked for each node.
Other works apply different augmentation parameters $p_{X},p_{A}$ for each
generated view, but as our framework is fully symmetrical, we postulate that
it is enough to use the same parameters to generate both augmentations.

##### Encoder network

The main component of the proposed framework is the encoder network
$f_{\theta}(\cdot)$. It takes an augmented graph as the input and computes (in
our case) a representation vector for each node in the graph. Note that we do
not specify any particular encoder network and one may use even encoders that
construct embeddings for edges, whole graphs or their attributed version. In
our experiments, we will show the application of GCN (13) and GAT (27) based
encoder networks. Both augmented graph views $\mathcal{G}^{(1)}$,
$\mathcal{G}^{(2)}$ are passed through the same encoder, resulting in two
embedding matrices $Z^{(1)}$ and $Z^{(2)}$, respectively.

##### Loss function

In our work, we propose to use a negative-sample-free loss function to train
the encoder network. We first normalize the embedding matrices along the batch
dimension, and then we compute the empirical cross-correlation matrix
$\mathcal{C}\in\mathbb{R}^{d\times d}$, which is a square matrix in the size
of the embedding dimension. We optimize it to be as close as possible to the
identity matrix. Such was originally proposed under the name Barlow Twins
(30). Moreover, we investigate the connection of Barlow Twins loss function to
the Hilbert-Schmidt Independence Criterion (26) and employ it as an
alternative for our framework’s loss function (further we will refer to this
as Graph HSIC or shorter G-HSIC). The gradient is symmetrically backpropagated
through the encoder network. We do not rely on any special techniques, like
momentum encoders, gradient stopping, or predictor networks.

###  3.1 Barlow Twins

Neuroscientist H. Barlow’s redundancy-reduction principle has motivated many
methods both in supervised and unsupervised learning (4; 21; 1). Recently,
(30) has employed this principle to build a self-supervised image
representation learning algorithm (we extend this idea to the domain of graph-
structured data). The main idea is to compute a cross-correlation matrix
$\mathcal{C}\in\mathcal{R}^{d\times d}$ of the embeddings $Z^{(1)}$ and
$Z^{(2)}$, instead of relying on negative sampling and contrastive loss
functions. A single element $\mathcal{C}_{ij}$ of this matrix can be computed
as follows:

$\mathcal{C}_{ij}=\frac{\sum_{b}z^{(1)}_{b,i}z^{(2)}_{b,j}}{\sqrt{\sum_{b}(z^{(1)}_{b,i})^{2}}\sqrt{\sum_{b}(z^{(2)}_{b,j})^{2}}},$
(1)

where $b$ are the batch indexes and $i,j$ are the indexes of embeddings. We
assume that both embeddings $z^{(1)}$, $z^{(2)}$ are normalized along the
batch dimension (a mean of zero and a standard deviation equal to one,
respective to the batch dimension).

The Barlow Twins loss function $\mathcal{L}_{\text{BT}}$ (see Equation 2)
encourages the cross-correlation matrix to be equal to the identity matrix. It
is composed of two parts: (I) invariance term and (II) redundancy reduction
term. The first one forces the on diagonal elements $\mathcal{C}_{ii}$ to be
equal to one, hence making the embeddings invariant to the applied
augmentations. The second term optimizes the off-diagonal elements
$\mathcal{C}_{ij}$ to be equal to zero – this results in decorrelated
components of the embedding vectors.

$\mathcal{L}_{\text{BT}}=\sum_{i}(1-\mathcal{C}_{ii})^{2}+\lambda\sum_{i}\sum_{j\neq
i}{\mathcal{C}_{ij}}^{2}$ (2)

The $\lambda>0$ parameter defines the trade-off between the invariance and
redundancy reduction terms when optimizing the overall loss function. In (26),
the authors proposed to use $\lambda=\frac{1}{d}$, which we employ in our
experimental setting. Otherwise, one can perform a simple grid search to find
the best $\lambda$ value in a particular experiment.

###  3.2 Hilbert-Schmidt Independence Criterion

The Barlow Twins method was derived based on the Information Bottleneck (IB)
theory (25). The representation vectors are assumed to be drawn from a
Gaussian distribution. In practice, this is a quite strong assumption and not
fulfilled by many encoders. However, the Barlow Twins method can be
interpreted in another way, as showed by (26). The authors view it as an
instance of the so-called negative-sample-free contrastive learning. This
allows to connect the method to the Hilbert-Schmidt Independence Criterion
(HSIC) (6). It does not assume the Gaussianity of the embedding vectors. Hence
it is more consistent with the practical usage of the Barlow Twins method.
Representations are learned by maximizing HSIC for augmented views of data
points.

Using the same empirical cross-correlation matrix $\mathcal{C}$ as in the
Barlow Twins method, we can maximize the Hilbert-Schmidt Independence
Criterion by minimizing the following loss function
$\mathcal{L}_{\text{HSIC}}$:

$\mathcal{L}_{\text{HSIC}}=\sum_{i}(1-\mathcal{C}_{ii})^{2}+\lambda\sum_{i}\sum_{j\neq
i}{(1+\mathcal{C}_{ij})}^{2}$ (3)

Notice that both $\mathcal{C}_{ij}=-1$ and $\mathcal{C}_{ij}=+1$ maximize
$\mathcal{C}_{ij}$. The $\lambda$ term has the same role as in the Barlow
Twins loss and we choose to use $\lambda=\frac{1}{d}$. For a detailed
derivation of the HSIC loss, please refer to (26).

##  4 Experiments

We evaluate the performance of our model using a variety of popular benchmark
datasets, including smaller ones, such as WikiCS, Amazon-Photo or Coauthor-CS,
as well as larger ones, such as ogbn-arxiv, provided by the Open Graph
Benchmark (11). In this section, we will provide an overview of the dataset
statistics, experimental scenario details, and the discussion of the results.
Overall, we use a similar experimental setup, as the state-of-the-art self-
supervised graph representation learning method BGRL(24), so we can perform a
fair comparison to this method. To track our experiments and provide a simple
way for reproduction, we employ the Data Version Control tool (DVC) (14). We
perform all experiments on a TITAN RTX GPU with 24GB RAM. The total running
time of all experiments (including hyperparameter searches and training
repetitions over 20 splits) is equal to approximately 50 hours.

###  4.1 Evaluation protocol

##### Self-supervised framework training

We start the evaluation procedure by training the encoder networks using our
proposed Graph Barlow Twins framework. In all scenarios, we use the AdamW
optimizer (9) with weight decay equal to $10^{-5}$. The learning rate is
updated using a cosine annealing strategy with a linear warmup period. Our
framework uses a single set of augmentation parameters for both graph views.
Therefore we do not use reported values of these parameters from other
publications that use two different sets. Instead we perform a grid search
over the range: $p_{A},p_{X}:\\{0,0.1,\ldots,0.5\\}$ for 500 epochs with a
warmup time of 50 epochs. We implement our experiment using the PyTorch
Geometric (5) library. All datasets are available in this library as well. The
details about the used augmentation hyperparameters, node embedding dimensions
and the encoder architecture are given in Appendix A. All dataset statistics
are summarized in Table 1.

Table 1: Dataset statistics. We use small to medium sized standard datasets together with the larger ogbn-arxiv dataset in the transductive setting. We also utilize the PPI dataset for inductive learning over multiple graphs. Name | Nodes | Edges | Features | Classes
---|---|---|---|---
WikiCS | 11,701 | 216,123 | 300 | 10
Amazon Computers | 13,752 | 245,861 | 767 | 10
Amazon Photos | 7,650 | 119,081 | 745 | 8
Coauthor CS | 18,333 | 81,894 | 6,805 | 15
Coauthor Physics | 34,493 | 247,962 | 8,415 | 5
ogbn-arxiv | 169,343 | 1,166,243 | 128 | 40
PPI (24 graphs) | 56,944 | 818,716 | 50 | 121 (multilabel)

##### Node embedding evaluation

We follow the linear evaluation protocol proposed in (28). We use the trained
encoder network, freeze the weights and extract the node embeddings for the
original graph data without any augmentations. Next, we train a $l_{2}$
regularized logistic regression classifier from the Scikit learn (17) library.
We also perform a grid search over the regularization strength using following
values: $\\{2^{-10},2^{-9},\ldots,2^{9},2^{10}\\}$. In the case of the larger
ogbn-arxiv and the PPI dataset, the Scikit implementation takes too long to
converge. Hence, we implement the logistic regression classifier in PyTorch
and optimize it for 1000 steps using the AdamW optimizer. We check various
weight decay values using a smaller grid search:
$\\{2^{-10},2^{-8},\ldots,2^{8},2^{10}\\}$. We use these classifiers to
compute the classification accuracy and report mean and standard deviations
over 20 model initializations and splits, except for the ogbn-arxiv and PPI
datasets, where we there is only one data split provided – we only re-
initialize the model weights 20 times.

###  4.2 Transductive experiments

We evaluate and compare our framework to other graph representation learning
approaches on 6 real-world datasets using the transductive setting. The whole
graph including all the node features is observed during the encoder training.
The node labels (classes) are hidden at that moment (unsupervised learning).
Next, we use the frozen embeddings and labels of training nodes to train the
logistic regression classifier.

####  4.2.1 Small and medium sizes benchmark datasets

Our first experiment uses 5 small and medium sized popular benchmark datasets,
namely:

  * •

WikiCS (16) is a network of Computer Science related Wikipedia articles with
edges denoting references between those articles. Each article belongs to one
of 10 subfields (classes) and has features computed as averaged GloVe
embeddings of the article content. We use the provided 20 train/val/test data
splits without any modifications.

  * •

Amazon Computers, Amazon Photos (15) are two networks extracted from Amazon’s
co-purchase data. Nodes are products and edges denote that these products were
often bought together. Each product is described using a Bag-of-Words
representation (node features) based on the reviews. There are 10 and 8
product categories (node classes), respectively. For these datasets there are
no data splits available, so similar to BGRL, we generate 20 random
train/val/test splits (10%/10%/80%).

  * •

Coauthor CS, Coauthor Physics are two networks extracted from the Microsoft
Academic Graph (22). Node are authors and edges denote a collaboration of two
authors. Each author is described by the keywords used in their articles (Bag-
of-Words representation; node features). There are 15 and 5 author research
fields (node classes), respectively. Similarly to the Amazon datasets there
are no data splits provided, so we generate 20 random train/val/test splits
(10%/10%/80%).

##### Encoder model

Similarly to (24),4 we build our encoder $f_{\theta}$ as a 2-layer GCN (13)
network. After the first GCN layer we apply a batch normalization layer (with
momentum equal to $0.01$) and the PReLU activation function. Accordingly to
the original Barlow Twins method, we do not apply any normalization or
activation to the final layer. A graph convolution layer (GCN) uses the
diagonal degree matrix $\mathbf{D}$ to apply a symmetrical normalization to
the adjacency matrix with added self-loops
$\mathbf{\hat{A}}=\mathbf{A}+\mathbf{I}$. Hence the propagation rule of such
layer is defined as follows:

$\text{GCN}(X,A)=\mathbf{\hat{D}}^{-\frac{1}{2}}\mathbf{\hat{A}}\mathbf{\hat{D}}^{-\frac{1}{2}}X\mathbf{W}$
(4)

Note that we do not include the activation $\sigma(\cdot)$ in this definition,
as we first apply the batch normalization and then the activation function.

##### Results and discussion

We train our framework for a total of 4000 epochs, but we observe that our
model converges much earlier at about 1000 epochs. This is an order of
magnitude faster than the state-of-the-art method BGRL, which reports results
for 10 000 epochs. In Table 2 we report the mean node classification accuracy
along with the standard deviations. As our experimental scenario was aligned
with BGRL, we re-use their reported values and compare them to our results. We
observe that our proposed method outperforms other baselines and achieves
comparable results to state-of-the-art methods.

Table 2: Results of transductive node classification. The reported values are the mean accuracy with the standard deviation. Our experiments are entries marked as $\ast$ and computed over 20 data splits and model initializations. Other results are taken from the BGRL paper (24). These are obtained in the same way as ours. OOM denotes running out of memory on a 16GB V100 GPU. | WikiCS | Am.CS | Am.Photo | CoauthorCS | CoauthorPhy
---|---|---|---|---|---
Raw features | 71.98 $\pm$ 0.00 | 73.81 $\pm$ 0.00 | 78.53 $\pm$ 0.00 | 90.37 $\pm$ 0.00 | 93.58 $\pm$ 0.00
DeepWalk | 74.35 $\pm$ 0.06 | 85.68 $\pm$ 0.06 | 89.44 $\pm$ 0.11 | 84.61 $\pm$ 0.22 | 91.77 $\pm$ 0.15
DeepWalk + features | 77.21 $\pm$ 0.03 | 86.28 $\pm$ 0.07 | 90.05 $\pm$ 0.08 | 87.70 $\pm$ 0.04 | 94.90 $\pm$ 0.09
DGI | 75.35 $\pm$ 0.14 | 83.95 $\pm$ 0.47 | 91.61 $\pm$ 0.22 | 92.15 $\pm$ 0.63 | 94.51 $\pm$ 0.52
GMI | 74.85 $\pm$ 0.08 | 82.21 $\pm$ 0.31 | 90.68 $\pm$ 0.17 | OOM | OOM
MVGRL | 77.52 $\pm$ 0.08 | 87.52 $\pm$ 0.11 | 91.74 $\pm$ 0.07 | 92.11 $\pm$ 0.12 | 95.33 $\pm$ 0.03
G-BT$\ast$ (1k epochs) | 77.30 $\pm$ 0.62 | 88.02 $\pm$ 0.32 | 92.23 $\pm$ 0.35 | 92.85 $\pm$ 0.31 | 95.23 $\pm$ 0.11
G-HSIC$\ast$ (1k epochs) | 76.83 $\pm$ 0.73 | 87.93 $\pm$ 0.36 | 92.46 $\pm$ 0.35 | 92.91 $\pm$ 0.25 | 95.25 $\pm$ 0.13
GRACE (10k epochs) | 80.14 $\pm$ 0.48 | 89.53 $\pm$ 0.35 | 92.78 $\pm$ 0.45 | 91.12 $\pm$ 0.20 | OOM
BGRL (10k epochs) | 79.36 $\pm$ 0.53 | 89.68 $\pm$ 0.31 | 92.87 $\pm$ 0.27 | 93.21 $\pm$ 0.18 | 95.56 $\pm$ 0.12
GCA | 78.35 $\pm$ 0.05 | 88.94 $\pm$ 0.15 | 92.53 $\pm$ 0.16 | 93.10 $\pm$ 0.01 | 95.73 $\pm$ 0.03
Supervised GCN | 77.19 $\pm$ 0.12 | 86.51 $\pm$ 0.54 | 92.42 $\pm$ 0.22 | 93.03 $\pm$ 0.31 | 95.65 $\pm$ 0.16

####  4.2.2 ogbn-arxiv dataset

In the next experiment, we use a large graph from the Open Graph Benchmark
(11), namely ogbn-arxiv. As presented in Table 1, this graph consists of about
170 thousand nodes and about 1.1 million edges. Each node represents a
Computer Science article on the arXiv platform, whereas the graph was
extracted from the Microsoft Academic Graph (22). Edges denote citations
across papers. The node features are build as word2vec embeddings of the whole
article content. There are 40 subject areas a node can be classified into
(node label/class). The ogbn-arxiv dataset provides a single train/val/test
split, so we use it without any modifications, but we retrain the whole
framework 20 times, similarly to previous experiments.

##### Encoder model

Due to the larger size of the network, we extend the encoder $f_{\theta}$ to a
3-layer GCN model. We employ batch normalization and PReLU activations after
the first and second layer, leaving the final layer as is (i.e. without any
activation of normalization). In the BGRL paper, the authors suggested to use
layer normalization together with weight standardization (19), yet we did not
observe any performance gain, but more importantly the training procedure was
unstable, with many peaks in the loss function. Although, these techniques
helped in the BGRL method, in our case we suspect that our loss function might
not be compatible to work with these techniques.

##### Results and discussion

In Table 3 we report the mean classification accuracy along with the standard
deviations. Note that we provide values for both validation and test splits,
as the provided data splits are build according to chronological order. Hence,
any model will be more affected with the out-of-distribution error on further
(in time) away data samples. We evaluate our model for 1000 epochs but it
converges as fast as about 300-400 epochs (further training did not give any
improvements). Both model variants achieve similar results, which are only 1.5
pp off to the state-of-the-art method BGRL, which takes 10 000 epochs to
converge to such solution.

Table 3: Results of transductive node classification on the ogbn-arxiv dataset. The reported values are the mean accuracy with the standard deviation. We provide both results on the validation and test splits due to the chronological order of the split. Our experiments are entries marked as $\ast$ and computed over 20 model initializations. Other results are taken from the BGRL paper (24). These are obtained in the same way as ours. | Validation | Test
---|---|---
MLP | 57.65 $\pm$ 0.12 | 55.50 $\pm$ 0.23
node2vec | 71.29 $\pm$ 0.13 | 70.07 $\pm$ 0.13
DGI | 71.26 $\pm$ 0.11 | 70.34 $\pm$ 0.16
GRACE (10k epochs) | 72.61 $\pm$ 0.15 | 71.51 $\pm$ 0.11
BGRL (10k epochs) | 72.53 $\pm$ 0.09 | 71.64 $\pm$ 0.12
G-BT$\ast$ (300 epochs) | 70.95 $\pm$ 0.12 | 70.07 $\pm$ 0.16
G-HSIC$\ast$ (400 epochs) | 70.94 $\pm$ 0.17 | 70.01 $\pm$ 0.17
Supervised GCN | 73.00 $\pm$ 0.17 | 71.74 $\pm$ 0.29

###  4.3 Inductive experiments

Our last experimental scenario evaluates our proposed framework on an
inductive task. We employ the Protein-Protein Interaction (PPI) dataset (33).
Each node in the graph represents a single protein, described by 50 biological
features, and edges denote interactions among those proteins. There are 121
labels which can be assigned to every node. Yet these labels are not exclusive
– this is a multilabel classification problem. Instead of reporting the
accuracy, we provide results in terms of the Micro-F1 score. PPI consists of
24 separate graphs and there exists a predefined data splits, where 20 graphs
are used for training, 2 graphs for validation and 2 graphs for testing. Note
that the validation and test graphs are completely unobserved during training
time, hence the model is more challenged during inference time.

##### Encoder model

Here we employ a Graph Attention (GAT) (27) based encoder model, as previous
works have shown better results of such network compared to standard GCN
layers on PPI. Specifically, we build our encoder $f_{\theta}$ as a 3-layer
GAT network with skip connections. The first and second layer uses 4 attention
heads of size 256 which are concatenated, and the final layer uses 6 attention
heads of size 512, whose outputs are averaged instead of applying
concatenation. In the GAT model an attention mechanism learns the weights that
are used to aggregate information from neighboring nodes. The attention
weights $\alpha_{ij}$ are computed according to the following equation:

$\alpha_{ij}=\frac{\text{exp}\left(\text{LeakyReLU}\left(\mathbf{a}^{T}\left[\mathbf{W}h_{i}||\mathbf{W}h_{j}\right]\right)\right)}{\sum_{k\in\mathcal{N}_{i}}\text{exp}\left(\text{LeakyReLU}\left(\mathbf{a}^{T}\left[\mathbf{W}h_{i}||\mathbf{W}h_{k}\right]\right)\right)}$
(5)

where $\mathcal{N}_{i}$ are the neighbors of node $i$, $\mathbf{W}$ is a
trainable matrix to transform node attributes, $\mathbf{a}$ is the trainable
attention matrix, and $||$ denotes the contatenation operation.

We use the ELU activation for the first and second layer, leaving the last
layer without any activation function. We do not apply any normalization
techniques in the model as preliminary experiments showed no performance
improvement.

Table 4: Results of multilabel node classification on PPI dataset. We report mean Micro-F1 score along with the standard deviation. Our experiments are entries marked as $\ast$ and computed over 20 model initializations. Other results are taken from the BGRL paper (24). These are obtained in the same way as ours. | PPI (test set)
---|---
Raw features | 42.20
DGI | 63.80 $\pm$ 0.20
GMI | 65.00 $\pm$ 0.02
GRACE | 66.20 $\pm$ 0.10
GRACE (10k epochs) | 69.66 $\pm$ 0.15
BGRL MLP predictor | 68.90 $\pm$ 0.21
BGRL GCN predictor | 69.55 $\pm$ 0.21
GRACE GAT encoder (20k epochs) | 69.71 $\pm$ 0.17
BGRL GAT encoder (20k epochs) | 70.49 $\pm$ 0.05
G-BT$\ast$ (500 epochs) | 70.61 $\pm$ 0.19
G-HSIC$\ast$ (500 epochs) | 70.15 $\pm$ 0.24
Supervised MeanPooling | 96.90 $\pm$ 0.20
Supervised GAT | 97.30 $\pm$ 0.20

##### Results and discussion

We train our framework using a batch size of $1$ graph for a total of 500
epochs, which turned out to be enough for the model to converge (we conducted
some preliminary experiments). In Table 4 we report the mean Micro-F1 score
along with the standard deviations over 20 model initialization, as this
dataset provided only one data split. Training for only 500 epochs provided
results on par and even better than SOTA method – BGRL – we achieve $70.61$
Micro-F1 score, whereas BGRL achieves $70.49$ also using a GAT encoder.

##  5 Conclusions

In this work we presented Graph Barlow Twins, a self-supervised graph
representation learning framework, which utilizes the embeddings’ cross-
correlation matrix computed from two distorted views of a particular graph.
The framework is fully symmetric and does not need any special techniques to
build non trivial embedding vectors. It builds representations that are
invariant to the applied augmentations and reduces the redundancy in the
representation vectors by enforcing the cross-correlation matrix to be equal
to the identity matrix (Barlow Twins loss). Using 7 real-world datasets we
evaluate our model in node classification tasks, both transductive and
inductive, and achieve results that are on par or better than SOTA methods in
SSL graph representation learning. We also show that our model converges an
order of magnitude faster than other approaches.

## Broader impact

Our method allows to reduce the computation cost (faster convergence) keeping
a decent performance in downstream tasks. Consequently, it can be used to
process larger graph datasets and efficiently perform tasks such as node
classification, link prediction or graph classification. These tasks have
crucial impact on various machine learning areas where graph structured data
is used, e.g. detection of bots or hate speech in social networks, or building
graph based recommendation engines.

## Limitations

We need to emphasize that we do not examine and implement any GNN encoder
capable of handling graph data in a batched manner, i.e. using a subsample of
nodes. We focus only on a full-batch scenario, where the whole graph is
processed at once. Hence, currently we are limited to use graphs that fit
completely into memory. We also focus only on node classification tasks as the
most popular in self-supervised graph representation learning. We leave both
points as future work, i.e. we will examine batch sampling GNN encoders as
well as check other kinds of GNNs that are capable computing edge-level or
graph-level representations and evaluate them in other tasks.

## References

  * Ballé et al. [2017] J Ballé, V Laparra, and E P Simoncelli.  End-to-end optimized image compression.  In _Int’l Conf on Learning Representations (ICLR)_ , Toulon, France, April 2017.  URL https://arxiv.org/abs/1611.01704.  Available at http://arxiv.org/abs/1611.01704.
  * Che et al. [2020] Feihu Che, Guohua Yang, Dawei Zhang, Jianhua Tao, Pengpeng Shao, and Tong Liu.  Self-supervised graph representation learning via bootstrapping.  _arXiv preprint arXiv:2011.05126_ , 2020.
  * Chen and He [2020] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning, 2020.
  * Deco and Parra [1997] G. Deco and L. Parra.  Non-linear feature extraction by redundancy reduction in an unsupervised stochastic neural network.  _Neural Netw._ , 10(4):683–691, June 1997.  ISSN 0893-6080.  doi: 10.1016/S0893-6080(96)00110-4.  URL https://doi.org/10.1016/S0893-6080(96)00110-4.
  * Fey and Lenssen [2019] Matthias Fey and Jan E. Lenssen.  Fast graph representation learning with PyTorch Geometric.  In _ICLR Workshop on Representation Learning on Graphs and Manifolds_ , 2019.
  * Gretton et al. [2012] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  _J. Mach. Learn. Res._ , 13(null):723–773, March 2012.  ISSN 1532-4435.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko.  Bootstrap your own latent - a new approach to self-supervised learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 21271–21284. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf.
  * Grover and Leskovec [2016] Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In _KDD_ , 2016.
  * Gugger and Howard [2018] S. Gugger and J. Howard.  Adamw and super-convergence is now the fastest way to train neural nets., 2018.  URL https://www.fast.ai/2018/07/02/adam-weight-decay/.
  * Hamilton et al. [2017] William L. Hamilton, Rex Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In _NIPS_ , 2017.
  * Hu et al. [2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 22118–22133. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/fb60d411a5c5b72b2e7d3527cfc84fd0-Paper.pdf.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Variational graph auto-encoders.  _NIPS Workshop on Bayesian Deep Learning_ , 2016.
  * Kipf and Welling [2017] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Kuprieiev et al. [2021] Ruslan Kuprieiev, Dmitry Petrov, Saugat Pachhai, Paweł Redzyński, Casper da Costa-Luis, Alexander Schepanovski, Peter Rowlands, Ivan Shcheklein, Jorge Orpinel, Fábio Santos, Batuhan Taskaya, Aman Sharma, Zhanibek, Gao, Dani Hodovic, Andrew Grigorev, Earl, nik123, Nabanita Dash, George Vyshnya, maykulkarni, Max Hora, Vera, Sanidhya Mangal, Wojciech Baranowski, Clemens Wolff, Alex Maslakov, Alex Khamutov, and Kurian Benoy.  Dvc: Data version control - git for data & models, May 2021.  URL https://doi.org/10.5281/zenodo.4733984.
  * McAuley et al. [2015] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel.  Image-based recommendations on styles and substitutes.  In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’15, page 43–52, New York, NY, USA, 2015. Association for Computing Machinery.  ISBN 9781450336215.  doi: 10.1145/2766462.2767755.  URL https://doi.org/10.1145/2766462.2767755.
  * Mernyei and Cangea [2020] Péter Mernyei and Cătălina Cangea.  Wiki-cs: A wikipedia-based benchmark for graph neural networks.  _arXiv preprint arXiv:2007.02901_ , 2020.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Perozzi et al. [2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena.  Deepwalk: Online learning of social representations.  In _Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’14, pages 701–710, New York, NY, USA, 2014. ACM.  ISBN 978-1-4503-2956-9.  doi: 10.1145/2623330.2623732.  URL http://doi.acm.org/10.1145/2623330.2623732.
  * Qiao et al. [2019] Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille.  Weight standardization.  _arXiv preprint arXiv:1903.10520_ , 2019.
  * Schmidhuber [1990] J. Schmidhuber.  Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments.  Technical Report FKI-126-90 (revised), Institut für Informatik, Technische Universität München, November 1990.
  * Schmidhuber et al. [1996] Jürgen Schmidhuber, Martin Eldracher, and Bernhard Foltin.  Semilinear predictability minimization produces well-known feature detectors.  _Neural Computation_ , 8(4):773–786, 1996.  doi: 10.1162/neco.1996.8.4.773.
  * Sinha et al. [2015] Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, and Kuansan Wang.  An overview of microsoft academic service (mas) and applications.  In _Proceedings of the 24th International Conference on World Wide Web_ , WWW ’15 Companion, page 243–246, New York, NY, USA, 2015. Association for Computing Machinery.  ISBN 9781450334730.  doi: 10.1145/2740908.2742839.  URL https://doi.org/10.1145/2740908.2742839.
  * Tang et al. [2015] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei.  Line: Large-scale information network embedding.  In _WWW_. ACM, 2015.
  * Thakoor et al. [2021] Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Remi Munos, Petar Veličković, and Michal Valko.  Bootstrapped representation learning on graphs.  In _ICLR 2021 Workshop on Geometrical and Topological Representation Learning_ , 2021.  URL https://openreview.net/forum?id=QrzVRAA49Ud.
  * Tishby et al. [1999] Naftali Tishby, Fernando C. Pereira, and William Bialek.  The information bottleneck method.  In _Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing_ , pages 368–377, 1999.  URL https://arxiv.org/abs/physics/0004057.
  * Tsai et al. [2021] Yao-Hung Hubert Tsai, Shaojie Bai, Louis-Philippe Morency, and Ruslan Salakhutdinov.  A note on connecting barlow twins with negative-sample-free contrastive learning, 2021.
  * Veličković et al. [2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph Attention Networks.  _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rJXMpikCZ.  accepted as poster.
  * Veličković et al. [2019] Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm.  Deep Graph Infomax.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=rklz9iAcKQ.
  * You et al. [2020] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen.  Graph contrastive learning with augmentations.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 5812–5823. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/3fe230348e9a12c13120749e3f9fa4cd-Paper.pdf.
  * Zbontar et al. [2021] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction, 2021\.
  * Zhu et al. [2020a] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang.  Deep Graph Contrastive Representation Learning.  In _ICML Workshop on Graph Representation Learning and Beyond_ , 2020a.  URL http://arxiv.org/abs/2006.04131.
  * Zhu et al. [2020b] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang.  Graph contrastive learning with adaptive augmentation.  _CoRR_ , abs/2010.14945, 2020b.  URL https://arxiv.org/abs/2010.14945.
  * Zitnik and Leskovec [2017] Marinka Zitnik and Jure Leskovec.  Predicting multicellular function through multi-layer tissue networks.  _Bioinformatics_ , 33(14):i190–i198, 07 2017\.  ISSN 1367-4803.  doi: 10.1093/bioinformatics/btx252.  URL https://doi.org/10.1093/bioinformatics/btx252.

##  Appendix A Experimental details

##### Augmentation hyperparameters

Our proposed framework uses a single pair of augmentation hyperparameters
$p_{A}\in\mathbb{R},p_{X}\in\mathbb{R}$ compared to other methods that use
different values to generate both graph views. We claim that a single set is
enough to achieve a decent performance in a symmetrical network architecture
like ours. Therefore, we cannot use the reported values of other works. We
instead perform a grid search over these hyperparameters and use those where
the model performs the best (in terms of classification accuracy or Micro-F1
score, for PPI). We do not evaluate the model during training and just use the
final version after training. We use the following setting:

  * •

the framework is trained to 500 epochs,

  * •

we set the learning rate warmup time to 50 epochs,

  * •

for both hyperparameters $p_{A}$ and $p_{X}$ we check following values:
$\\{0,0.1,\ldots,0.5\\}$.

For values greater than $0.5$ the augmentation removes too much information
from the graph. We summarize the augmentation hyperparameters of the best
performing models in Table 5.

Table 5: Augmentation hyperparameters | G-BT | G-HSIC
---|---|---
| $p_{A}$ | $p_{X}$ | $p_{A}$ | $p_{X}$
WikiCS | 0.5 | 0.1 | 0.2 | 0.1
Amazon-CS | 0.4 | 0.1 | 0.4 | 0.1
Amazon-Photo | 0.1 | 0.3 | 0.4 | 0.3
Coauthor-CS | 0.5 | 0.2 | 0.4 | 0.1
Coauthor-Physics | 0.5 | 0.2 | 0.5 | 0.2
ogbn-arxiv | 0.4 | 0.0 | 0.3 | 0.0
PPI | 0.1 | 0.1 | 0.1 | 0.1

##### Training setup

For all datasets, we train our framework using the AdamW [Gugger and Howard,
2018] optimizer with a weight decay of $10^{-5}$. The learning rate is
adjusted using a cosine annealing strategy with a linear warmup period up to
the base learning rate. During training we set a total number of epochs and an
evaluation interval, after which the frozen embeddings are evaluated in
downstream tasks (using either the $l_{2}$ regularized logistic regression
from Scikit learn [Pedregosa et al., 2011] with liblinear solver, or the
custom PyTorch version with AdamW for ogbn-arxiv and PPI). For instance, if we
set the total number of epochs to 1000 and the evaluation interval to 500, the
model will be evaluated at epochs: 0, 500 and 1000 (three times in total). We
report the values for the best performing model found during those
evaluations. We summarize these training statistics in Table 6.

Table 6: Training hyperparameters. We use the same settings for both G-BT and G-HSIC. |  G-BT / G-HSIC
---|---
| total | warmup | evaluation | base | best model
| epochs |  | interval | learning rate | found at
WikiCS | 4 000 | 400 | 1 000 | $5*10^{-4}$ | 1 000
Amazon-CS | 4 000 | 400 | 1 000 | $5*10^{-4}$ | 1 000
Amazon-Photo | 4 000 | 400 | 1 000 | $1*10^{-4}$ | 1 000
Coauthor-CS | 4 000 | 400 | 1 000 | $1*10^{-5}$ | 1 000
Coauthor-Physics | 4 000 | 400 | 1 000 | $1*10^{-5}$ | 1 000
ogbn-arxiv | 700 | 100 | 100 | $1*10^{-3}$ | 300 (G-BT)
|  |  |  |  | 400 (G-HSIC)
PPI | 500 | 50 | 100 | $5*10^{-3}$ | 500

##### Encoder architecture

We compare our framework against the state-of-the-art self-supervised graph
representation learning method BGRL[Thakoor et al., 2021]. To provide a fair
comparison, we use similar encoder architectures to the ones presented in
their paper. We do not use any predictor networks in our framework, so we need
to slightly modify the encoders to be better suited for the loss function (as
given in the Barlow Twins paper[Zbontar et al., 2021]), i.e. we do not apply
any normalization (like batch or layer normalization) or activation function
in the final layers of the encoder. Note that the lack of predictor network
and batch normalization in the final layer, reduces the overall number of
trainable network parameters. In all cases, we use a batch normalization with
the momentum equal to $0.01$ (as in BGRL[Thakoor et al., 2021], where they use
the equivalent weight decay equal to $0.99$).

For the small up to medium sized datasets, i.e. WikiCS, Amazon-CS, Amazon-
Photo, Coauthor-CS, Coauthor-Physics, we use a 2-layer GCN[Kipf and Welling,
2017] based encoder with the following architecture:

  * •

GCN($k,2d$),

  * •

BatchNorm($2d$),

  * •

PReLU(),

  * •

GCN($2d,d$),

where $k$ is the number of node features and $d$ is the embedding vector size.

For the ogbn-arxiv dataset, we use a slightly larger model – a 3-layer
GCN[Kipf and Welling, 2017] based encoder. We tried to utilize weight
standarization[Qiao et al., 2019] and layer normalization, but our model did
not benefit from those techniques (as it helped in BGRL[Thakoor et al.,
2021]). The training procedure under this setting was unstable with various
fluctuations and peaking of the loss function. The final architecture is
summarized as follows:

  * •

GCN($k,d$),

  * •

BatchNorm($d$),

  * •

PReLU(),

  * •

GCN($d,d$),

  * •

BatchNorm($d$),

  * •

PReLU(),

  * •

GCN($d,d$).

In the inductive experiment with the PPI dataset, we use a 3-layer
GAT[Veličković et al., 2018] based encoder. Graph Attention network are known
to perform better on this dataset compared to GCNs. This was also showed in
BGRL[Thakoor et al., 2021], where their approach with GAT layers provided
state-of-the-art performance in self-supervised graph representation learning
for PPI. Our architecture can be summarized as follows:

  * •

GAT($k,256,$ heads=$4$) $+$ Linear($k,4*256$)

  * •

ELU(),

  * •

GAT($4*256,256,$ heads=$4$) $+$ Linear($4*256,4*256$)

  * •

ELU(),

  * •

GAT($4*256,d,$ heads=$6$) $+$ Linear($4*256,d$)

The outputs of the attention heads in the first and second layer are
concatenated and for the last GAT layer, the attention heads outputs are
averaged. In every layer, we utilize skip connections using linear layers to
project the outputs of the previous layer (features in the case of the first
layer) to the desired dimensionality.

The exact values for the input feature dimension $k$ and the embedding
dimension $d$ are given in Table 7.

Table 7: Encoder layer size parameters. We use the same settings for both G-BT and G-HSIC. |  G-BT / G-HSIC
---|---
| number of | embedding
| node features | dimensionality
| $k$ | $d$
WikiCS | 300 | 256
Amazon-CS | 767 | 128
Amazon-Photo | 745 | 256
Coauthor-CS | 6 805 | 256
Coauthor-Physics | 8 415 | 128
ogbn-arxiv | 128 | 256
PPI | 50 | 512

##### Code and reproducibility

We implement all our models using the PyTorch-Geometric library [Fey and
Lenssen, 2019]. The experimental pipeline is built using the Data Version
Control tool (DVC [Kuprieiev et al., 2021]). It enables to run all experiments
in a single command and ensure better reproducibility. We publish the code at
https://github.com/pbielak/graph-barlow-twins. To reproduce the whole pipeline
run: dvc repro and to execute a single stage use: dvc repro <stage name>.
There are following stages:

  * •

preprocess_dataset@<dataset_name> – downloads the <dataset_name> dataset; if
applicable, generates the node splits for train/val/test,

  * •

hps_transductive_barlow_twins@<dataset_name>,
hps_transductive_hsic@<dataset_name> – runs the augmentation hyperparameter
search for a given dataset (in the transductive scenario, i.e. WikiCS, Amazon-
CS, Amazon-Photo, Coauthor-CS, Coauthor-Physics, ogbn-arxiv), using the Barlow
Twins and Hilbert-Schmidt Independence Criterion loss, respectively,

  * •

train_transductive_barlow_twins@<dataset_name>,
train_transductive_hsic@<dataset_name> – trains and evaluates the G-BT and
G-HSIC model, respectively, for a given dataset (in the transductive
scenario),

  * •

hps_inductive_barlow_twins, hps_inductive_hsic – runs the augmentation
hyperparameter search for the PPI dataset, using the Barlow Twins and Hilbert-
Schmidt Independence Criterion loss, respectively,

  * •

train_inductive_ppi_barlow_twins, train_inductive_ppi_hsic – trains and
evaluates the G-BT and G-HSIC model, respectively, for the PPI dataset.

All hyperparameters described in this Appendix are stored in configuration
files in the experiments/configs/ directory, whereas the experimental Python
scripts are placed in the experiments/scripts/ directory.