# MRC-LSTM: A Hybrid Approach of Multi-scale Residual CNN and LSTM to Predict
Bitcoin Price
††thanks: $\ast$ indicates the corresponding author.

Qiutong Guo  School of Computer Science
Sichuan University
Chengdu, China  Shun Lei  School of Computer Science
Sichuan University
Chengdu, China  Qing Ye∗ School of Computer Science
Sichuan University
Chengdu, China  Zhiyang Fang  School of Computer Science
Sichuan University
Chengdu, China

###### Abstract

Bitcoin, one of the major cryptocurrencies, presents great opportunities and
challenges with its tremendous potential returns accompanying high risks. The
high volatility of Bitcoin and the complex factors affecting them make the
study of effective price forecasting methods of great practical importance to
financial investors and researchers worldwide. In this paper, we propose a
novel approach called MRC-LSTM, which combines a Multi-scale Residual
Convolutional neural network (MRC) and a Long Short-Term Memory (LSTM) to
implement Bitcoin closing price prediction. Specifically, the Multi-scale
residual module is based on one-dimensional convolution, which is not only
capable of adaptive detecting features of different time scales in
multivariate time series, but also enables the fusion of these features. LSTM
has the ability to learn long-term dependencies in series, which is widely
used in financial time series forecasting. By mixing these two methods, the
model is able to obtain highly expressive features and efficiently learn
trends and interactions of multivariate time series. In the study, the impact
of external factors such as macroeconomic variables and investor attention on
the Bitcoin price is considered in addition to the trading information of the
Bitcoin market. We performed experiments to predict the daily closing price of
Bitcoin (USD), and the experimental results show that MRC-LSTM significantly
outperforms a variety of other network structures. Furthermore, we conduct
additional experiments on two other cryptocurrencies, Ethereum and Litecoin,
to further confirm the effectiveness of the MRC-LSTM in short-term forecasting
for multivariate time series of cryptocurrencies.

##  I Introduction

Bitcoin is the world’s first distributed super sovereign digital currency,
proposed and established by Satoshi Nakamoto [1] in 2009. It relies on an
electronic payment system based on cryptography and P2P (Point to Point)
technology. By using encryption algorithms and automatic authentication
mechanisms, it is difficult to crack or forge, making its transactions more
secure and transparent. Since the birth of Bitcoin, it has quickly gained
widespread attention.

As a new type of cryptocurrency, Bitcoin conducts 24/7 transactions and is
able to exchange for many major currencies at a low cost of foreign exchange.
Compared to other traditional financial assets, Bitcoin provides investors a
new type of portfolio management. Existing research [2, 3] indicated that
Bitcoin has an apparent role in the portfolio management market. Empirical
analysis by Anne H et al. [4] also corroborates the investability of Bitcoin.
In recent years, its rich portfolio and the potential for high returns have
attracted the attention of an increasing number of financial investors.
However, the Bitcoin market is highly volatile and subject to frequent
speculative bubbles [5, 6], thus its trading risks are enormous. Consequently,
as discussed above, the study of effective price forecasting methods is of
great practical importance to investors, researchers, and policymakers around
the world due to the tremendous opportunities and challenges posed by Bitcoin.

In recent years increasingly machine learning methods, especially deep neural
networks(DNNs), have been applied to market forecasting in cryptocurrencies.
In terms of the Bitcoin market, it is highly volatile and generates a large
amount of highly non-linear transaction data. In order to effectively explore
these dynamic data, we need models which are able to analyze the internal
interactions and hidden patterns in the data. Several researchers [7, 8] have
shown that DNN models are well suited for learning lagged correlations between
step-wise trends in large financial time series. In a literature review [9] of
comparative studies between artificial neural networks and traditional
statistical models, it was found that in 72% of 96 cases, artificial neural
network models were shown to have better predictive performance. Consequently,
there is no doubt that due to the highly nonlinear and volatile nature of
financial markets, DNN models are increasingly applied in the field of
finance, especially for financial time series forecasting.

In this paper, a novel DNN model consisting of a Multi-scale Residual
Convolutional Neural Network with Long Short-Term Memory (LSTM) is proposed
for Bitcoin price prediction. In the study, not only the influence of
transaction information such as Bitcoin historical price is considered on the
closing price of Bitcoin, but also external influences such as macroeconomic
factors and investors’ attentions are introduced. The proposed model consists
of two main parts. First, is the proposed multi-scale residual module based on
one-dimensional convolution. It contains an identity mapping and three
branching networks, where the size of the convolutional kernel is different
for each branching network. Then, the second part is the LSTM network, which
is able to further learn the trends of multivariate time series and the
interactions between the series, and output the final predicted values.
Different from previous financial multivariate time series forecasting, we
construct a multi-scale residual module, which can also be called a three-
bypass residual module, in which information from these bypasses can be shared
with each other. Moreover, this structure can extract potential features that
have a high impact on bitcoin price in multiple time scales and integrate them
into highly expressive feature vectors after the concatenate operation. In
addition to the Bitcoin price prediction experiments, we also conducted
extensive experiments on datasets of two different cryptocurrencies (Litecoin
and Ethereum) to confirm the strong ability of the proposed MRC-LSTM model for
short-term price prediction of cryptocurrencies.

The remainder of the paper is organized as follows. Section II reviews the
relevant literature. Section III introduces the methodology, including the
proposed model. Section IV presents our experiments, while our empirical
results and discussion are shown in Section V. Finally, Section VI is the
conclusion of this paper.

##  II Related literature

In this section, the related work about price prediction of the cryptocurrency
and the basis knowledge to the MRC-LSTM are provided.

###  II-A Neural Network Approaches

Salim Lahmiri et al. [10] first applied the deep neural networks (DNNs) to
cryptocurrency price prediction, elucidated the short-term predictability of
cryptocurrencies, and found that the predictive accuracy of Long Short-term
Memory (LSTM) neural networks is higher than that of Generalized Regression
Neural Networks (GRNN)[11]. Subsequently, a growing body of research has
emerged in this area [12, 13]. LSTM neural networks [14] have been shown to be
significantly effective in forecasting bitcoin prices due to their ability to
identify long-term dependencies and store both long-term and short-term
temporal information. Researchers have obtained better results with the LSTM
network, in their works to predict bitcoin prices and compare them with the
performance of different models [15, 16].

Further, Convolutional Neural Networks (CNNs) have also been applied to
cryptocurrency market forecasting. In a study [17], researchers combined CNN
with LSTM neural network for high-frequency market trend prediction for a
variety of cryptocurrencies. Their empirical analysis shows that the addition
of convolutional layers improves the prediction performance and that the
hybrid network structure provides the best prediction results in the
experiment. Yan Li et al. [18] propose a hybrid neural network model based on
CNN and LSTM neural networks, and experimental results show that CNN-LSTM
hybrid neural network can effectively improve the accuracy of value prediction
and direction prediction compared to single-structure neural network. The
hybrid neural network model combining CNN with LSTM has also been used by many
researchers for time series prediction of financial data such as Gold
volatility prediction[19], stock prices, etc. Its excellent performance has
also been demonstrated in many other fields [20, 21, 22].

###  II-B Time Series Forecasting

Time series research has always been an important field of machine learning.
By building up neural networks based on deep learning, we are able to extract
and exploit the hidden information represented by digital currency raw data of
digital currencies in order to make accurate and efficient price predictions
[10]. In recent years, researchers have continuously made progress in the
field of financial time series forecasting [23, 24, 25]. Many traditional
research approaches focus on learning internal patterns, such as
autocorrelation, in a particular time series. However, for reality scenarios,
especially in cryptocurrencies, stock markets and other financial domains, in
most cases we are dealing with multivariate time series. Changwei Hu et al.
[26] developed a deep learning structural time series model to handle
correlated multivariate time series inputs. Their model is able to leverage
dependencies among multiple correlated time series and extract weighted
differencing features for better trend learning. The existence of close
correlations among many multivariate time series motivates us to consider not
only intra-series pattern learning but also inter-series pattern learning when
dealing with such tasks.

###  II-C Residual Network

As one of the milestones in the evolution of CNN, Residual Network (ResNet)
[27] has achieved impressive, record-breaking performance on many challenging
tasks. ResNet share some similarities with Highway networks, such as residual
blocks and shortcut connections. ResNet simplifies the training of very deep
networks by bypassing signals from one layer to the next through identity
connections. The basic idea underlying residual learning is the branching of
gradient propagation paths. For CNNs, this idea was first introduced in the
form of parallel paths in the inception models of [28]. Many research works
[29, 30] have exploited the multi-level features in CNNs by skip-connections
and found them to be effective for a variety of visual tasks. GoogLeNet [28,
31] proposed an “Inception module” that connects feature mappings generated by
filters of different sizes to increase the diversity of feature extraction.
Meanwhile, GoogLeNet increased the network width by skipping connections to
improve the robustness and expressiveness of the network. Gao Huang et al.
[32] proposed the Dense Convolutional Network (DenseNet) to exploit the
potential of the network through feature reuse. In contrast to ResNet, it
introduces a direct connection from any layer to all subsequent layers. In
addition, DenseNet combines features by concatenating them, rather than
combining features through summation. These works show us the utility of skip
connections, and DenseNet also shows us how to connect feature maps via
concatenation. Our work is partly inspired by these two ideas and explores
their application to model building for short-term price forecasting of
cryptocurrencies.

##  III Methodologies

In this section, the details of our proposed method MRC-LSTM and basic blocks
are described.

###  III-A ResNet

In order to solve the problem of gradient disappearance and gradient explosion
due to network deepening, He et al. proposed a new DNN——ResNet[27]. ResNet
consists of many residual units, shown in Fig. 1, and each residual unit can
be represented in the following Equations 1 and 2.

$y_{t}=h(x_{t})+F(x_{t},w_{t})$ (1) $x_{t+1}=f(y_{i})$ (2)

Where $F$ is a residual function, $f$ is a ReLU function, $w_{t}$ is the
weight matrix, and $x_{t}$ and $y_{t}$ are the inputs and outputs of the
$t$-th layer. The function $h$ is an identity mapping given by Equation 3:

$h(x_{t})=x_{t}$ (3)

In a residual block, skip connections can effectively aggregate historical
information, reduce the loss of features and information to some extent, and
enable the network to learn richer content. Motivated by the idea of Residual
units, our proposed model applies the method of skip connections.

Figure 1: the residual block

###  III-B LSTM

LSTM is based on the Recurrent Neural Network (RNN) model, which can
effectively solve the gradient disappearance and gradient explosion problems
in the RNN model. The addition of a special gate control mechanism makes it
possible to solve the problem of long-term dependence, and it is suitable for
time series processing and prediction, natural language generation [33, 34,
35]. Fig. 2 gives the basic unit of the LSTM neural network. Its basic unit is
the memory block, which contains the memory cell and three gates that control
the memory cell, namely, the input gate, the output gate, and the forget gate.

Figure 2: Basic unit of LSTM network

Forget Gate is mainly used to calculate the degree of information retention
and discarding. The output $f_{t}$ represents the probability of forgetting
the state of the underlying cell layer, and is calculated as follows:

$f_{t}=\sigma(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})$ (4)

In the Equation (4), the $x_{t}$ indicates the input of the current cell,
$h_{t-1}$ presents the output of the previous cell, $\sigma$ is the sigmoid
function.

In the input gate, the sigmoid activation function is used to calculate which
information needs to be updated. Then, use the tanh activation function to get
a vector of candidate values $\tilde{C}_{t}$, after which updates the previous
state $C_{t-1}$ to $C_{t}$. The formulas are Equations 5, 6 and 7.

$i_{t}=(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})$ (5)
$\tilde{C}_{t}=tanh(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})$ (6) $C_{t}=f_{t}\cdot
C_{t-1}+i_{t}\cdot\tilde{C}_{t}$ (7)

The output gate is used to calculate the extent of the information output at
the current moment. The information is filtered by the sigmoid activation
function to obtain $o_{t}$. Then the tanh activation function is used to
obtain the desired information $h_{t}$:

$o_{t}=\sigma(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})$ (8) $h_{t}=o_{t}\cdot
tanh(C_{t})$ (9)

###  III-C The Proposed Network Architecture.

Inspired by ResNet, and based on the need to deal with time series problems,
we developed a novel building block, Multi-scale Residual Convolutional block
(MRC) based on one-dimensional time convolution. It is combined with an LSTM
neural network to form a new hybrid network architecture (MRC-LSTM) to perform
cryptocurrency time series prediction. In the following, we will first
introduce how to design the multi-scale residual block, followed by the
proposed MRC-LSTM model, i.e., how the new building block can be combined with
LSTM to predict the bitcoin price.

Figure 3: The Structure of Multi-scale Residual Block

The Multi-scale Residual Block. This is the first part of MRC-LSTM which is
utilized to extract potential features with high expressiveness at different
time scales from the dataset. We constructed a three-bypass convolutional
layer with convolutional kernels of different sizes, and joined a jump
connection to further aggregate historical information efficiently. Due to the
high volatility of Bitcoin and the short-term predictability of
cryptocurrencies [10], the Bitcoin market is suitable for short-term
forecasting, and the longer the forecast period, the worse the forecast
performance. Therefore, we select 1D convolutional kernels of size 1, 2, and 3
to slide the sequences in the temporal domain, which means that such multi-
scale residual module can simultaneously extract the trends and the hidden
interactions of the data within 1, 2, and 3 adjacent days in the sequences. In
addition, it is known that the window size of the kernel in a one-dimensional
convolution will affect the network learning effect. It is likely to miss
local feature information when the window size is too large; When the window
size is small, local features are easy to extract, but may reduce the
correlation of local features. Hence, the ”Multi-scale” design of the proposed
model can also take into account the advantages of both large and small
windows. Fig. 3 shows the design of the residual module.

When it comes to the way to combine the extracted features of three bypasses,
here we draw on the idea of DenseNet, which differs from ResNet in that we
never combine features by summing them before they are passed to the next
layer; instead, we combine features by concatenating them. We know that the
concatenation operation is not viable when the feature-map size changes.
Therefore, in order to keep the feature-maps size uniform, we need to perform
a zero-padding when performing 1D convolution. Then, the identity mapping is
concatenated in the depth direction with the feature maps of the three paths
and fed into the subsequent layer.

At the end of the module, the use of 1 × 1 2D convolutional kernels enables
cross-channel information interaction and expansion. In this way, the feature-
maps extracted by the three prediction cycles will be fused and the network
will adaptive extract useful information from these hierarchical features.
Meanwhile, non-linearity can be added while keeping the feature map scale
constant.

Figure 4: The overall architecture of MRC-LSTM model. Conv 1D, One-dimensional
convolutional layer; MRC, multi-scale residual block; LSTM, long short-term
memory; FC, fully connected layer.

Hybrid MRC-LSTM model. The network consists of two main parts, the first is
the multi-scale residual module for extracting features in the multivariate
time series, and the second is the LSTM layer for learning pattern changes and
predicting prices. Fig. 4 shows the structure of the MRC-LSTM neural network.
The network contains an input layer, a 1D convolutional layer, the multi-scale
residual module, an LSTM layer, a fully connected layer, and an output layer.

During the construction of the MRC-LSTM, we first use a set of 1D
convolutional kernels with a window size of 1 to slide in the time direction
for initial feature extraction and feature augmentation of the sequence. Then,
the multi-scale residual module is utilized to extract and integrate local
features across multiple time scales. Finally, an LSTM network is used to
learn relationships in time steps for price forecasting.

$f(x)=\begin{cases}x,(x>0)\\\ 0,(x\leq 0)\end{cases}$ (10)

We introduce activation functions to incorporate nonlinearities in the
network. The common activation functions include Sigmoid, Tanh, ReLU, and
SELU. In this paper, the rectified linear unit (ReLU) is selected as the
nonlinear activation function, which is expressed as the Equation (10).

##  IV Experiments

###  IV-A Data Collection and pre-processing

The dataset used in this experiment includes the daily closing price of
Bitcoin and 10 types of internal (Bitcoin trading datas) and external
(macroeconomic variables and investor attention) information that have an
impact on the price of Bitcoin from October 25, 2015 to October 17, 2020, for
a total of 1820 records. The dataset is divided into 80% as training set and
20% as test set. The training set is further divided into a training set (80%)
and a validation set (20%) for evaluating performance and avoiding
overfitting. The Table I shows all the attributes that compose the dataset.

The internal information in the dataset refers to the daily transaction datas
of Bitcoin, including the open price, the closing price, the highest price,
the lowest price, the weighted price, the trade volume in Bitcoins and the
market’s currency. They are collected from the offical website 111
https://bitcoincharts.com.

As for external factors, we take the macroeconomic factors and investor
attention into consideration. As a new type of investment instrument, the
price of bitcoin is considered to be related to several macroeconomic
variables. The study by Leon Li considered four representative volatility
indices published by the Chicago Board Options Exchange(CBOE), including the
Volatility Index(VIX) and the Gold Price Volatility Index(GVZ). He believed
that volatility indices can be used to speculate and trade on market sentiment
regarding future volatility [36]. Some researchers have found that there is a
relationship between the price of bitcoin and gold, crude oil, and stock
market indices [37]. Therefore, this paper introduces macroeconomic variables
that have been proven to have predictive power for the bitcoin market to
predict bitcoin prices. Ultimately, the following macroeconomic factors were
chosen for the experiment: S&P 500 Index, GVZ, VIX. They are all daily data
from the Wind Database.

Moreover, in recent years, a number of researchers have found that Google
trends, which reflect investors’ attention, play a significant role in
predicting the Bitcoin market [38, 39]. Da et al. [40] pointed out that search
activity can reflect investors’ attention. Therefore, we employ the Google
Trends as one of the predictors, which is derived from Google Trends 222
https://trends.google.com.

TABLE I: List of input attributes Transaction Information | Macroeconomic Variables | Investor Attention
---|---|---
the Open price | S&P 500 Index | Google Trends
the Close price | GVZ | -
the Highest price | VIX | -
the Lowest price | - | -
the Weighted price | - | -
Volume(BTC) | - | -
Volume(Currency) | - | -

The prediction period used in this experiment is five days, which means the
closing price of bitcoin on the sixth day is predicted using the
characteristic parameter data from the previous five days. In the meantime,
our proposed residual module is capable of using three convolutional kernels
of different scales to extract the temporal characteristics of a multivariate
sequence between one, two, and three adjacent days in a 5-day period. The
richer potential information contained in the sequences is extracted by
getting different sized receptive fields.

The original data is usually normalized to eliminate scale effects between
indicators before modeling. The common normalization methods are min-max
normalization and Z-Score. In the experiment, the min-max normalization is
used. This approach, as shown in Equation (11), is capable of varying the
original data linearly and mapping the values of the data between 0 and 1.

$x_{norm}=\frac{x_{t}-x_{min}}{x_{max}-x_{min}}$ (11)

where $x_{t}$, $x_{norm}$ are the input sample data and the data after
normalization, $x_{min}$,$x_{max}$ are the minimum and maximum value of the
samples. After modeling, the target outputs need to be anti-normalized.

$\hat{y}_{t}=y_{norm}(y_{max}-y_{min})+y_{min}$ (12)

where $y_{t}$ is the output of the prediction value, $y_{min}$, $y{max}$ are
the minimum and maximum value of the target data, and $y_{norm}$ is the
predicted value derived directly from the building model.

(a) MLP, MRC-LSTM and Actual curves

(b) LSTM, MRC-LSTM and Actual curves

(c) CNN-LSTM, MRC-LSTM and Actual curves

Figure 5: Comparison diagrams of prediction and actual curves

###  IV-B Process and Parameters Settings

The proposed neural network structure in this paper consists of two main
parts. Through repeated experiments, the parameters of the proposed network
are determined as follows.

  * •

In the first part, a 1D convolutional layer is used for initial feature
extraction and expansion. There are 16 convolutional kernels of size 1 in this
layer. After that, the features obtained from the upper layer are input into
the proposed residual module for feature extraction and integration at
multiple time scales. The parameters of the convolutional layer in the
residuals module are shown in Fig. 3.

  * •

In the second part of the network structure, the features extracted by the
residual module are input into the LSTM layer, which has a layer number of 1
and a number of neurons of 50. Since the features extracted by the 1D
convolution-based residual module are along the temporal dimension, they can
be entered into the LSTM to learn the long-range time pattern changes of the
trends in the sequence. Finally, the full connection layer is used and the
predicted value is output.

In the experiment, the loss function is the MSE, the optimization function is
the Adam optimization algorithm, the batch normalization number Batch size is
50. In addition, we use the Piece wise decay method for learning rate decay,
with an initial learning rate of 0.001, and the learning rate decreases to 0.3
times the original rate every 500 epochs, for a total of 2000 epochs. Each
time the neural network outputs a value, a sliding window is used for
prediction. The experiments are compared and evaluated with actual values to
obtain the final predictive performance. All algorithms are implemented using
Pytorch 333https://pytorch.org/.

###  IV-C Performance Evaluation Criteria

In this work, four error functions, Mean Absolute Error (MAE)(i.e., Equation
(13)), Root Mean Square Error (RMSE)(i.e., Equation (14)), Mean Absolute
Percentage Error (MAPE)(i.e., Equation (15)) and the coefficient of
determination($R^{2}$), are introduced as performance metrics to quantify the
ability of the DNN to forecast prices.

$MAE=\frac{1}{N}\sum_{i=1}^{N}\left|(y_{i}-f_{i})\right|$ (13)
$RMSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_{i}-f_{i})^{2}}$ (14)
$MAPE=\frac{100}{N}\sum_{i=1}^{N}\left|\frac{(y_{i}-f_{i})}{y_{i}}\right|$
(15)

where $y_{i}$ and $f_{i}$ are the $i$th actual value and predicted value, and
$N$ is the number of test samples. They reflect the deviation between the
predicted and actual values. The larger their value, the greater the error in
the forecast.

The $R^{2}$ value reflects the accuracy of the model which ranges from 0 to 1
with 1 denotes perfect match:

$R^{2}=\frac{\begin{matrix}\sum_{n=1}^{n}(\hat{y}_{i}-\bar{y})^{2}\end{matrix}}{\begin{matrix}\sum_{n=1}^{n}(y_{i}-\bar{y})^{2}\end{matrix}}$
(16)

where $\hat{y}_{i}$ represents the predicted value, $\bar{y}$ is the average
value, and $y_{i}$ is the observed value.

TABLE II: Bitcoin: The errors of different deep neural networks Architecture | MAE | RMSE | MAPE |  $R^{2}$(%)
---|---|---|---|---
MLP | 246.55 | 345.50 | 2.31 | 87.95
LSTM | 212.71 | 317.24 | 1.99 | 89.83
CNN | 191.92 | 268.26 | 1.77 | 92.73
CNN-LSTM | 176.79 | 270.66 | 1.66 | 92.60
MRC-LSTM | 166.52 | 261.44 | 1.56 | 93.10
TABLE III: ETH and LTC: The errors of different deep neural networks Architecture | Ethereum(ETH) | Litecoin(LTC)
---|---|---
MAE | RMSE | MAPE | MAE | RMSE | MAPE
MLP | 1.04 | 1.53 | 7.83 | 0.42 | 0.70 | 9.67
LSTM | 0.76 | 1.17 | 5.74 | 0.32 | 0.38 | 7.65
CNN | 0.81 | 1.23 | 5.92 | 0.39 | 0.44 | 9.37
CNN-LSTM | 0.77 | 1.24 | 6.20 | 0.27 | 0.37 | 6.43
MRC-LSTM | 0.70 | 1.13 | 5.43 | 0.14 | 0.26 | 3.17

##  V Results and discussion

The algorithms used for comparison in the experiments are Multilayer
Perceptron (MLP), Two-dimensional Convolutional Neural Network (CNN), Long
short-term memory (LSTM), and CNN-LSTM. Table II shows the accuracy metrics of
the proposed model compared to the benchmarks using different architectures.

From Table II, it is apparent that the MLP has the largest prediction error,
while the LSTM outperforms the MLP. The LSTM can solve the long-term
dependence problem in time series samples, making it more suitable for solving
time series prediction problems than the MLP.

The CNN-LSTM model consists of a two-dimensional convolutional layer for
feature extraction first, and then the extracted features are fed into the
LSTM for price prediction. The prediction accuracy of the LSTM model is just
lower than that of the CNN-LSTM, which illustrates that when there are many
features and few samples, the LSTM cannot fully extract the important
potential information from the data, which will affect the training effect of
the neural network to some extent.

The error of MRC-LSTM prediction results is smaller than that of CNN-LSTM,
which is the best prediction performance algorithm in this experiment. It
indicates that the feature extraction strategy of Multi-scale residual block
is more efficient. The main reason is that, on the one hand, in the one-
dimensional convolutional layer, the convolutional kernel slides only along
the time dimension and performs the convolution operation on all feature
values at the time step, which reduces the problem of multiple covariance in
the features. On the other hand, extracting features in multivariate sequences
from three different scales improves the expression of local features and
facilitates LSTM to make more accurate and fast predictions.

Besides, Fig. 5 shows the effect of our proposed network by extending the
prediction map to several sample points, and three sets of comparisons are
made. It can be directly reflected from the figure that the predicted values
derived from the MRC-LSTM model are overall closer to the actual values
compared to several other models. The main reason is that the proposed
residual module enables the model to combine the features extracted from the
underlying convolution into the upper convolution and learn the trends
embedded in the multivariate time series from different time scales.
Consequently, it can make efficient use of the features in the series and
improve the prediction performance of the model.

It is evident from the Fig. 5 that the predictions of both the MRC-LSTM and
several other models have some degree of lag with the actual values. This is
probably because despite the outstanding performance of the Multi-scale
residual convolution module in feature extraction as described above, we still
cannot ignore the working principle of the LSTM, which uses information from
previous lags to predict future instances. Moreover, as the Bitcoin market is
a highly dynamic system, the patterns and dynamics present in that system are
not always the same. Therefore, when the LSTM module misses a large leap, it
makes a quick correction in time with new information, which in turn creates
the curves in the Fig. 5.

To further evaluate the performance of our proposed model for short-term
forecasting of cryptocurrencies, we also use the same sets of models for price
forecasting of two other major cryptocurrencies, ETH and LTC. In this
experiment, both the ETH and LTC data sets contain daily transaction data from
April 15, 2016 to November 20, 2020. This includes the highest price, the
lowest price, the bid price, the ask price and the volume, and all these data
are from Quandl444https://www.quandl.com/. Table III shows the prediction
errors under different DNN architectures. As this table shows, the results
indicate that the LSTM outperforms the MLP in the time series task. The CNN-
LSTM model uses the CNN for feature extraction before using the LSTM for
prediction, which results in a smaller error than the LSTM. Finally, it is
apparent from the table that the MRC-LSTM has the best performance. Our
proposed MRC-LSTM model improves the feature extraction method by using a
multi-scale residual convolutional layer for feature extraction, which gives
the best prediction results in the experiments.

The empirical analysis shows that the proposed model also performs well in
predicting the financial time series of the other two cryptocurrencies.

##  VI Conclusion

In this study, a hybrid method consisting of a multi-scale residual block and
an LSTM network is proposed to predict the bitcoin price. Specifically, multi-
scale residual block in this hybrid model is able to extract rich features at
different time scales and also strengthen the representational ability of
these features. Besides, the utilization of local residual learning leads to a
reduction in computational complexity as well as improves the performance of
the DNN. Different from some traditional studies, we introduce external
influences such as macroeconomic variables and investor attention when
performing price forecasting to combine various factors that may affect
bitcoin prices. Furthermore, sufficient experimental results confirm that our
proposed model has better prediction results than other single-structured
models. The efficient feature extraction and integration capability of the
proposed residual block is also demonstrated in the experimental comparison.
In addition, we perform price prediction with the MRC-LSTM for two other major
cryptocurrencies, ETH and LTC, further demonstrating the effectiveness of our
model for multivariate time series forecasting of cryptocurrencies. In
summary, compared with several state-of-art works, the proposed DNN model
achieves better prediction performance, but it also has some limitations,
since the Bitcoin market is particularly sensitive to some national policies,
regulatory and market events, etc. More future work could be focused on
comprehensive metrics which measure the investor’s attention to more timely
detection of bitcoin market volatility and thus more accurate price
prediction.

## References

  * [1] S. Nakamoto and A. Bitcoin, “A peer-to-peer electronic cash system,” _Bitcoin.–URL: https://bitcoin. org/bitcoin. pdf_ , vol. 4, 2008.
  * [2] A. Dyhrberg, “Bitcoin, gold and the dollar – a garch volatility analysis,” _Finance Research Letters_ , vol. 16, 11 2015.
  * [3] A. H. Dyhrberg, “Hedging capabilities of bitcoin. is it the virtual gold?” _Finance Research Letters_ , vol. 16, pp. 139 – 144, 2016.
  * [4] A. H. Dyhrberg, S. Foley, and J. Svec, “How investible is bitcoin? analyzing the liquidity and transaction costs of bitcoin markets,” _Economics Letters_ , vol. 171, pp. 140 – 143, 2018.
  * [5] J. E.-T. Cheah and J. Fry, “Speculative bubbles in bitcoin markets? an empirical investigation into the fundamental value of bitcoin,” _Economics Letters_ , vol. 130, 02 2015.
  * [6] S. Corbet, B. Lucey, and L. Yarovaya, “Datestamping the bitcoin and ethereum bubbles,” _Finance Research Letters_ , vol. 26, pp. 81 – 88, 2018.
  * [7] R. Adcock and N. Gradojevic, “Non-fundamental, non-parametric bitcoin forecasting,” _Physica A: Statistical Mechanics and its Applications_ , vol. 531, p. 121727, 2019.
  * [8] M. Nakano, A. Takahashi, and S. Takahashi, “Bitcoin technical trading with artificial neural network,” _Physica A: Statistical Mechanics and its Applications_ , vol. 510, pp. 587 – 609, 2018.
  * [9] M. Khashei and M. Bijari, “An artificial neural network (p,d,q) model for timeseries forecasting,” _Expert Systems with Applications_ , vol. 37, no. 1, pp. 479 – 489, 2010.
  * [10] S. Lahmiri and S. Bekiros, “Cryptocurrency forecasting with deep learning chaotic neural networks,” _Chaos, Solitons & Fractals_, vol. 118, pp. 35–40, 2019.
  * [11] D. F. Specht _et al._ , “A general regression neural network,” _IEEE transactions on neural networks_ , vol. 2, no. 6, pp. 568–576, 1991.
  * [12] H. Liang, W. Lei, P. Y. Chan, Z. Yang, M. Sun, and T.-S. Chua, “Pirhdy: Learning pitch-, rhythm-, and dynamics-aware embeddings for symbolic music,” _Proceedings of the 28th ACM International Conference on Multimedia_ , Oct 2020. [Online]. Available: http://dx.doi.org/10.1145/3394171.3414032
  * [13] F. Zhu, W. Lei, C. Wang, J. Zheng, S. Poria, and T.-S. Chua, “Retrieving and reading: A comprehensive survey on open-domain question answering,” 2021.
  * [14] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [15] N. Uras, L. Marchesi, M. Marchesi, and R. Tonelli, “Forecasting bitcoin closing price series using linear regression and neural networks models,” _arXiv preprint arXiv:2001.01127_ , 2020.
  * [16] P. Linardatos and S. Kotsiantis, “Bitcoin price prediction combining data and text mining,” in _Advances in Integrations of Intelligent Methods_. Springer, 2020, pp. 49–63.
  * [17] S. Alonso-Monsalve, A. L. Suárez-Cetrulo, A. Cervantes, and D. Quintana, “Convolution on neural networks for high-frequency trend prediction of cryptocurrency exchange rates using technical indicators,” _Expert Systems with Applications_ , vol. 149, p. 113250, 2020.
  * [18] Y. Li and W. Dai, “Bitcoin price forecasting method based on cnn-lstm hybrid neural network model,” _The Journal of Engineering_ , vol. 2020, 01 2020\.
  * [19] A. Vidal and W. Kristjanpoller, “Gold volatility prediction using a cnn-lstm approach,” _Expert Systems with Applications_ , vol. 157, p. 113481, 2020\.
  * [20] C. Tian, J. Ma, C. Zhang, and P. Zhan, “A deep neural network model for short-term load forecast based on long short-term memory network and convolutional neural network,” _Energies_ , vol. 11, no. 12, p. 3493, 2018\.
  * [21] W. Lei, X. Wang, M. Liu, I. Ilievski, X. He, and M.-Y. Kan, “Swim: A simple word interaction model for implicit discourse relation recognition,” in _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17_ , 2017, pp. 4026–4032. [Online]. Available: https://doi.org/10.24963/ijcai.2017/562
  * [22] W. Lei, X. Jin, M.-Y. Kan, Z. Ren, X. He, and D. Yin, “Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 1437–1447. [Online]. Available: https://www.aclweb.org/anthology/P18-1133
  * [23] A. A. Ariyo, A. O. Adewumi, and C. K. Ayo, “Stock price prediction using the arima model,” in _2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation_. IEEE, 2014, pp. 106–112.
  * [24] R. Adhikari and R. Agrawal, “A combination of artificial neural network and random walk models for financial time series forecasting,” _Neural Computing and Applications_ , vol. 24, no. 6, pp. 1441–1449, 2014.
  * [25] J. Cao, Z. Li, and J. Li, “Financial time series forecasting model based on ceemdan and lstm,” _Physica A: Statistical Mechanics and its Applications_ , vol. 519, pp. 127–139, 2019.
  * [26] C. Hu, Y. Hu, and S. Seo, “A deep structural model for analyzing correlated multivariate time series,” in _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_. IEEE, 2019, pp. 69–74.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [28] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [29] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [30] S. Yang and D. Ramanan, “Multi-scale recognition with dag-cnns,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1215–1223.
  * [31] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [32] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [33] W. Lei, X. Jin, M.-Y. Kan, Z. Ren, X. He, and D. Yin, “Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 1437–1447.
  * [34] X. Jin, W. Lei, Z. Ren, H. Chen, S. Liang, Y. Zhao, and D. Yin, “Explicit state tracking with semi-supervisionfor neural dialogue generation,” in _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , 2018, pp. 1403–1412.
  * [35] L. Pan, W. Lei, T.-S. Chua, and M.-Y. Kan, “Recent advances in neural question generation,” _arXiv preprint arXiv:1905.08949_ , 2019.
  * [36] L. Li, “Risk of investing in volatility products: A regime-switching approach,” _Investment Analysts Journal_ , pp. 1–16, 2020.
  * [37] S. Vassiliadis, P. Papadopoulos, M. Rangoussi, T. Konieczny, and J. Gralewski, “Bitcoin value analysis based on cross-correlations,” _Journal of Internet Banking and Commerce_ , vol. 22, no. S7, p. 1, 2017.
  * [38] A. Urquhart, “What causes the attention of bitcoin?” _Economics Letters_ , vol. 166, pp. 40–44, 2018.
  * [39] A. Yelowitz and M. Wilson, “Characteristics of bitcoin users: an analysis of google search data,” _Applied Economics Letters_ , vol. 22, no. 13, pp. 1030–1036, 2015.
  * [40] Z. Da, J. Engelberg, and P. Gao, “In search of attention,” _The Journal of Finance_ , vol. 66, no. 5, pp. 1461–1499, 2011.


How does GPT-2 Predict Acronyms? Extracting and Understanding a Circuit via
Mechanistic Interpretability

Jorge García-Carrasco Alejandro Maté Juan Trujillo

Lucentia Research, Department of Software and Computing Systems, University of
Alicante

###### Abstract

Transformer-based language models are treated as black-boxes because of their
large number of parameters and complex internal interactions, which is a
serious safety concern. Mechanistic Interpretability (MI) intends to reverse-
engineer neural network behaviors in terms of human-understandable components.
In this work, we focus on understanding how GPT-2 Small performs the task of
predicting three-letter acronyms. Previous works in the MI field have focused
so far on tasks that predict a single token. To the best of our knowledge,
this is the first work that tries to mechanistically understand a behavior
involving the prediction of multiple consecutive tokens. We discover that the
prediction is performed by a circuit composed of 8 attention heads ($\sim 5\%$
of the total heads) which we classified in three groups according to their
role. We also demonstrate that these heads concentrate the acronym prediction
functionality. In addition, we mechanistically interpret the most relevant
heads of the circuit and find out that they use positional information which
is propagated via the causal mask mechanism. We expect this work to lay the
foundation for understanding more complex behaviors involving multiple-token
predictions.

##  1 INTRODUCTION

Scaling up the size of Language Models based on the Transformer architecture
(Vaswani et al., 2017; Brown et al., 2020) has been shown to greatly improve
its performance on a wide range of tasks. Because of this, the use of Large
Language Models (LLMs) on high-impact fields such as in medicine
(Thirunavukarasu et al., 2023; Zhang et al., 2023) is increasingly growing and
is expected to keep growing even larger. However, these models are treated as
black-boxes due to the fact that they have a large number of parameters and
complex internal interactions, hampering our ability to understand its
behavior. This is a considerable concern with regards to the safety of
Artificial Intelligence (AI) systems, as using a model without knowing its
internal heuristics or algorithms can derive in unexpected outcomes such as
privacy issues (Li et al., 2023) or harmful behavior (Wei et al., 2023) among
others.

Mechanistic Interpretability (MI) aims to tackle this issue by interpreting
behaviors in terms of human-understandable algorithms or concepts (Elhage et
al., 2021; Olsson et al., 2022; Elhage et al., 2022; Nanda et al., 2023). In
other words, it tries to reverse-engineer the large amount of parameters that
compose the model into understandable components, which essentially increases
the trustworthiness of the model. MI has been successfully applied to explain
different tasks on transformer-based models. For example, Wang et al. (2023)
discover the circuit responsible for the Indirect Object Identification task
(IOI) in GPT-2 Small, composed by 26 attention heads grouped in 7 different
classes. Similarly, Hanna et al. (2023) use MI techniques to explain how GPT-2
Small performs the greater-than operation on a single task and test if the
discovered circuit generalizes to other contexts. Likewise, Heimersheim and
Janiak (2023) discovered how a smaller 4-layer transformer model predicted
argument names on a docstring.

In summary, works like the ones mentioned above are proof that mechanistic
analysis can be used to shine light into the inner workings of language
models. As MI is a young field, the current focus is on developing methods and
understanding behaviors on relatively smaller models to lay a solid foundation
that will be used to interpret increasingly larger models. In fact,
preliminary studies have already appeared discussing whether current MI
techniques are scalable to larger models, with optimistic results (Lieberum et
al., 2023).

In this work, we contribute to the growing body of MI works by focusing on
understanding how GPT-2 Small performs the task of predicting three-letter
acronyms (e.g. "The Chief Executive Officer" $\rightarrow$ "CEO"). We have
mainly chosen this task because it consists on predicting three consecutive
tokens, in contrast to previous existing work which focused on single-token
prediction. To the best of our knowledge, this is the first work that applies
MI to understand a behavior involving the prediction of multiple tokens.
Hence, we expect that the work presented here serves as a starting point for
understanding more complex behaviors that involve predicting multiple tokens.

More specifically, we will adopt a _circuits_ perspective (Elhage et al.,
2021; Olah et al., 2020) and identify the components of the model that are
responsible for the behavior under study via a series of systematic
_activation patching_ (Meng et al., 2022) experiments. Our contributions can
be summarized as follows:

  * •

We discover the circuit responsible for three-letter acronym prediction on
GPT-2 Small. The circuit is composed by 8 attention heads ($\sim 5\%$ of
GPT-2’s heads) which we classified on three groups according to their role.

  * •

We evaluate the circuit by ablating the rest of components of the model and
show that the performance is preserved and even slightly improved when
isolating the discovered 8-head circuit.

  * •

We interpret the main components of the circuit, which we term _letter mover
heads_ by reverse-engineering their parameters.

  * •

We also found that _letter mover heads_ make use of positional information,
mainly derived from the attention probabilities due to the causal mask
mechanism instead of the positional embeddings.

The remainder of this paper is structured as follows. In Section 2, the
required background and the problem statement are presented. Section 3
describes the procedure used to discover the circuit responsible for three-
letter acronym prediction as well as the role of each component, followed by
an evaluation of the circuit. Section 4 delves into mechanistically
interpreting _letter mover heads_ as well as studying how these heads use
positional information. Finally, the conclusions about the work are presented
in Section 5.

##  2 BACKGROUND

In this section we briefly present the transformer and the used notation, the
task of study and how to evaluate the performance of the model on that task.

###  2.1 Model and Notation

GPT-2 Small (Radford et al., 2019) is a 117M parameter decoder-only
transformer architecture composed by 12 transformer blocks containing 12
attention heads followed by an MLP, each component preceded by Layer
Normalization (Ba et al., 2016). The input to the model is a sequence of $N$
consecutive tokens which are embedded into $x_{0}\in\mathbb{R}^{N\times d}$
via a learned embedding matrix $W_{E}\in\mathbb{R}^{V\times d}$, where $V$ is
the size of the vocabulary. Similarly, positional embeddings are added to
$x_{0}$.

Following the notation presented in Elhage et al. (2021), $x_{0}$ is the
initial value of the _residual stream_ , where all the components of the model
read from and write to. Specifically, if $h_{ij}$ denotes the $j$th attention
head at layer $i$, the $i$th attention layer will update the residual stream
as $x_{i+1}=x_{i}+\sum_{j}h_{ij}(x_{i})$ (omitting layer normalization). Each
attention head is parameterized by the matrices
$W_{Q}^{ij},W_{K}^{ij},W_{V}^{ij}\in\mathbb{R}^{d\times d/H}$ and
$W_{O}^{ij}\in\mathbb{R}^{d/H\times d}$, where $H$ is the number of heads in a
single layer, which can be arranged into the QK and OV matrices
$W_{QK}^{ij}=W_{Q}^{ij}W_{K}^{ij}$, $W_{OV}^{ij}=W_{V}^{ij}W_{O}^{ij}$. The QK
matrix contains information about which tokens the head attends to, whereas
the OV matrix is related to what the head writes into the residual stream.

Finally, the resulting vector is unembedded via a unembedding matrix, which in
the case of GPT-2 is tied to the embedding matrix (i.e. $W_{U}=W_{E}^{T}$) to
obtain a vector $y\in\mathbb{R}^{N\times V}$ where $y_{ij}$ represents the
logits of the $j$th token of the vocabulary for the prediction following the
$i$th token of the sequence.

###  2.2 Task Description

We will focus on the task of predicting three-letter acronyms. To evaluate
whether GPT-2 is able to properly perform this task or not, we curated a
dataset of 800 acronyms. It is important to remark that this dataset will not
be used to re-train the model, but to perform experiments and identify the
underlying circuit associated to the task of study. In other words, our aim is
to detect a circuit responsible for a concrete task on an LLM that has already
been trained in a general, self-supervised way. Hence, in order to isolate the
behavior of study and reduce the amount of noise, we made each acronym to meet
the following characteristics:

  * •

Each word must be composed by two tokens, the first being only composed by the
capital letter and its preceding space (e.g. "| C|ane|")

  * •

The acronym must be tokenized by exactly three tokens, each for one letter of
the acronym (e.g. "|C|K|L|")

In order to build the dataset, we took a public list of the most frequently-
used common nouns in English (Quintans, 2023) containing a total of $6775$
nouns. However, building the dataset is not as easy as choosing three random
words and tokenizing them according to our imposed characteristics: words have
to be tokenized as GPT-2 naturally expects to stay in-distribution. GPT-2 uses
byte-pair encoding (BPE) tokenization (Sennrich et al., 2016), a technique
that tokenizes according to the most frequent substring. This means that
common substrings/words such as `"ABC"` or `" Name"` are encoded as a single
token, hence reducing the amount of possible nouns and acronyms to use on our
dataset. Taking this into account, the building procedure was the following:

  1. 1.

Nouns: We took the list of 6775 nouns and filtered out the words that did not
meet the characteristics (i.e. each word of the acronym must be composed by
two tokens, the first being only composed by the capital letter and its
preceding space), leaving us with 381 nouns.

  2. 2.

Acronyms: We tokenized the $P^{R}(26,3)=26^{3}=17576$ possible 3-letter
acronyms and checked which were naturally tokenized as three separate tokens,
which reduced the amount of possible acronyms to 2740. As common nouns
beginning with the letter X, Q or U are rare, we also excluded acronyms
containing that letter, resulting in a total of 1154 possible combinations.

  3. 3.

Dataset: Finally, we built the dataset by (i) sampling one of the 152 possible
acronyms (e.g. WVZ) and randomly sampling three of the 381 nouns, one for
every letter of the acronym (e.g. Wreck, Vibe and Zipper). Notice that we can
build much more than 800 samples in this way, but we chose this size because
of computational constraints. As a reference, Hanna et al. (2023) curated a
dataset of 490 datapoints to properly identify a circuit.

In summary, this results in a dataset composed by prompts with the structure:

"|The|C1|T1|C2|T2|C3|T3| (|A1|A2|A3|"

where Ci is the token encoding the capital letter of the $i$th word (together
with its preceding space), Ti is the remainder of the word, and Ai is the
$i$th letter of the acronym. Therefore, the task consists on predicting A1, A2
and A3 given the previous context.

The reason for choosing a list of nouns was because (i) it is the common way
to build acronyms and (ii) the type of word (noun, adjective, etc.) does not
affect the result obtained. We can confirm these results since we have also
experimented with synthetic words by taking a random token "| A|" where A can
be any capital letter, followed by one to three random tokens containing just
lowercase letters and found the same results that will be presented in this
work.

Also, one important concern is that the model could have memorized popular
acronyms (e.g. The Central Processing Unit (CPU)). However, the acronyms built
with our procedure are rare (e.g. The Wreck Vibe Zipper (WVZ)). We took this
decision to ensure that the model has not been trained with these samples,
implying that the discovered circuit generalizes to samples outside of the
training dataset and does not just memorize common acronyms.

###  2.3 Evaluation

In order to quantitatively evaluate the ability of GPT-2 on the task under
study, we will compute the _logit difference_ between the correct letter and
the incorrect letter with the highest logits, for each of the three letters of
the acronym:

$\text{logit\\_diff}_{i}=\text{logits}_{a_{i}}-\underset{l\in\mathcal{L}\backslash\\{a_{i}\\}}{\text{max}}\text{logits}_{l}$
(1)

where $a_{i}$ is the correct prediction for the $i$th letter of the acronym
and $\mathcal{L}$ is the set of possible predictions, which in the case of
acronym prediction, is the set of capital letters. GPT-2 has an average logit
difference across every letter and sentence of the dataset of $2.22$, which
translates to an average $\sim 90.2\%$ probability difference. Overall, this
result provides quantitative evidence supporting that GPT-2 is indeed able to
perform three-letter acronym prediction.

##  3 A CIRCUIT FOR 3-LETTER ACRONYM PREDICTION

Now that the task has been clearly defined and checked that GPT-2 is indeed
able to perform it, we will discover the circuit responsible for this
behavior, evaluate it and understand the components that compose such circuit.
The following experiments were performed by using both _PyTorch_ (Paszke et
al., 2019) and _TransformerLens_ (Nanda and Bloom, 2022) with a 40GB A100 GPU.
A repository containing the code required to reproduce the experiments and
figures can be found in https://github.com/jgcarrasco/acronyms_paper.

###  3.1 Discovering the circuit

In order to discover which components form the circuit responsible for three-
letter acronym prediction, we will perform a systematic series of _activation
patching_ experiments, first presented in Meng et al. (2022). The idea of
activation patching is to patch (i.e. replace) the activations of a given
component with the activations obtained by running the model on a _corrupted
prompt_. If the metric degrades when patching a component, it means that it is
relevant to the task under study, therefore enabling us to locate the circuit.

In this case, we have carefully performed activation patching with three
different types of corruption. For each of the $i$th letter prediction, we (i)
randomly resample the $i$th word, (ii) randomly resample the words previous to
the $i$th word and (iii) randomly resample the acronym letters previous to the
$i$th letter. This will allow us to better track the flow of information and
the role of each component. Table 1 shows the different types of corruption
for the prediction of the third letter on a sample prompt. We will perform the
corresponding activation patching experiments for each of the three letters on
parallel.

Table 1: Example of prompt corruption for the third letter prediction $i=3$ (this is also performed for $i=1,2$) Corruption Type | Prompt
---|---
Original | The Cane Knee Lender (CK
Current Word | The Cane Knee Tandem (CK
Previous Words | The Ego Icy Lender (CK
Previous Letters | The Cane Knee Lender (BJ
All corruptions | The Ego Icy Tandem (BJ

####  3.1.1 Corrupting the Current Word

Fig. 1 shows the change in logit difference when patching the residual stream
before each layer at every position, for each of the three letters of the
acronym to predict. If patching the residual stream at a given position and
layer considerably degrades the performance, then it implies that the
activations stored at that specific step are important for the acronym
prediction task. In this case, we can see that patching Ci at earlier layers
does indeed drastically degrade the performance, with the logit difference
dropping up to -5 units. We can also notice a shift from Ci to A(i-1)
beginning at layer 8. This implies that there are some components that read
information about Ci, move it to A(i-1) and then use it to perform the
prediction of the next letter Ai.

Figure 1: Patching the residual stream at every position and before every
layer (corrupting the current word).

Once that we have tracked the flow of information, we can have a more fine-
grained view by patching at the level of individual components, i.e. attention
heads or MLPs. We performed activation patching experiments on MLPs and found
that they were not relevant for acronym prediction. This was expected, as this
task mostly requires moving information between token positions, which can
only be performed by attention heads. Fig. 2 shows the result of patching the
output of attention heads with the activations obtained by corrupting the
current word. We are able to localize four heads that are relevant across the
three predictions: 8.11, 10.10, 9.9 and 11.4. It is also interesting to notice
that the drop on performance is larger on the last letter than on the first.
This is due to the fact that the model has more context (i.e. the two previous
letters of the acronym) when predicting the last letter, which translates to
the model being more confident on its prediction, which corresponds to a
larger drop when patching.

Figure 2: Patching the output of attention heads for every iteration
(corrupting the current word).

Fig. 3 shows the distribution of the average attention paid from A(i-1) to the
previous token positions for head 8.11. It can clearly be seen that it mostly
attends from A(i-1) to Ci, strongly suggesting that these heads copy the
information of the corresponding letter and use it to perform the prediction
of the next letter of the acronym, so we term this heads as _letter mover
heads_. The behavior of these heads will be extensively discussed on Section
4, after performing the remaining activation patching experiments. The
attention patterns for the rest of letter mover heads can be seen in the
Supplementary Materials.

Figure 3: Average probability paid from A(i-1) to the previous token positions
for head 8.11.

####  3.1.2 Corrupting the Previous Words

Fig. 4 shows the results of performing activation patching on the residual
stream by corrupting the previous words. As expected, there is no effect on
the prediction of the first letter because there are no previous words to
corrupt. However, on the remaining letters, patching C(i-1) at earlier layers
has a significant (although quite smaller than the previous) effect on
predicting Ai, indicating that the circuit uses information about the previous
words to perform the task. Concretely, it seems that the information is moved
from C(i-1) to Ci on layers 1-2 and from there to A(i-1) at layer 5 and below.
Interestingly, patching Ti around layers 5-11 slightly improves the
performance of the circuit. We hypothesize that patching only this position
may cause the model to become less confident on previous letters, essentially
increasing the logit difference. As it is not the focus of the paper and the
effect is minimal, we will not delve deeper into this fact.

Figure 4: Patching the residual stream (corrupting previous words).

In order to check which attention heads were responsible for this movement of
information, we patched the output of attention heads at positions Ci and
A(i-1). Fig. 5 shows that there are a diffuse set of attention heads
responsible for moving information from C(i-1) to Ci, such as 4.11, 1.0 and
2.2. Further inspection of the attention patterns show that they are _previous
token heads_ (or fuzzy versions of it), i.e. heads that attend to the previous
token position w.r.t. the current token and move information. Visualizations
of the attention patterns of these heads can be found on the Supplementary
Materials.

Figure 5: Patching the output of attention heads for every iteration at
position Ci (corrupting the previous words).

On the other hand, Fig. 6 shows that heads 5.8, 8.11 and 10.10 are the most
relevant in this patching experiment. Further inspection of the attention
patterns reveals that they mostly attend to the T(i-1) and Ci tokens and move
the information that was propagated to these positions via the previous token
heads.

Figure 6: Patching the output of attention heads for every iteration at
position A(i-1) (corrupting the previous words).

There are a few important aspects to remark from this patching experiment. The
first is that the performance drop when patching individual components is
significantly lower than on the previous experiment. Secondly, the computation
is more diffuse, i.e. it is distributed across many components, specially when
looking at the Ci position. As we will see in the next section, this is due to
the fact that the model is able to obtain the exact same information (i.e. the
capital letter of the previous word) by just attending to the previous
predicted letter, which is considerably easier. Another interesting fact is
that some letter mover heads are also present in this part of the computation,
i.e. some heads have multiple roles or behaviors, which is a motif that has
also been discovered on other works Heimersheim and Janiak (2023).

####  3.1.3 Corrupting Previous Predicted Letters

The results presented in Fig. 7 clearly show that the model uses information
about the previous predicted letter to predict the next one, as patching the
A(i-1) position causes a considerable performance drop (larger than the
previous corruption method) across every layer. This provides even more
evidence in favor of the previously presented hypothesis that letter mover
heads obtain the same information via two paths: from C(i-1) via the
combination of previous token heads and heads that move information to Ci and
then to A(i-1), and directly from A(i-1).

Figure 7: Patching the residual stream at every position and before every
layer (corrupting the previous predicted letters).

To summarize, we have been able to discover the following circuit via a series
of activation patching experiments:

  * •

Heads 8.11, 10.10, 9.9 and 11.4, termed Letter Mover Heads, attend mostly to
the Ci token position from the A(i-1) token position and are the main
responsible for acronym prediction on GPT-2.

  * •

Letter Mover Heads use the previous predicted letter to attend to the correct
token position and predict the next letter of the acronym.

  * •

This information, although more faintly, is also propagated from C(i-1) to Ci
via a set of fuzzy previous heads such as 4.11, 1.0 and 2.2, which is then
moved from Ci to A(i-1) via heads 5.8, 8.11 and 10.10.

###  3.2 Circuit Evaluation

Now that we have defined a circuit, it is necessary to evaluate whether it is
sufficient to effectively perform acronym prediction. In order to evaluate it,
we will ablate every other component that is not part of the circuit.
Specifically, we will perform mean-ablation, which consists on replacing the
activation of a component with the mean activation obtained across all samples
of the dataset. In this way, we only discard the information related to the
task of study while keeping the rest. Fig. 8 shows the logit difference
obtained when progressively adding heads to the circuit. We start with an
empty circuit (i.e. ablating every head) to check that the model is unable to
perform the task, obtaining negative values of the logit difference, as
expected. Then, progressively adding Letter Mover Heads greatly improves
performance, the most significant increase being on the third letter
prediction, where adding just head 8.11 increases the average logit difference
from -1 to 2 approximately. The logit difference keeps increasing by
progressively adding the rest of components until we reach the baseline
performance with just the 8 discovered heads.

Figure 8: Logit Difference obtained by ablating everything and progressively
adding components to the circuit. The dashed horizontal line represents the
logit difference obtained with the complete model.

##  4 UNDERSTANDING LETTER MOVER HEADS

Now that we have discovered and evaluated the main circuit responsible for the
task of three-letter acronym prediction on GPT-2, we will provide further
evidence on how Letter Mover Heads work, which are the main components of the
circuit.

###  4.1 What do Letter Mover Heads Copy?

We discovered that Letter Mover Heads mostly attend from A(i-1) to Ci and were
the main responsible for the acronym prediction task. Because of this, we
hypothesize that these heads directly increase the logits of the correct
letter to predict. In order to give evidence about this, we will take a look
at the weights of Letter Mover Heads and try to reverse-engineer their
behavior.

Specifically, we will inspect the full OV circuit, obtained by retrieving the
embeddings corresponding to the capital letter tokens and the capital letter
tokens preceded by a space, passing them through the OV circuit of a Letter
Mover Head and unembedding the resulting vector. This essentially tells us
what would the head write into the residual stream if it attended perfectly to
that token. Fig. 9 shows the full OV circuit for Letter Mover Head 8.11,
rearranging it in four different ways to check what it writes when fully
attending to capital letters, with or without a preceding space. At first
sight, one cannot draw any conclusion except that there is a slight diagonal
when attending to capital letters preceded with a space (two rightmost plots.

Figure 9: Full OV circuit for head 8.11, for all capital letter tokens
with/without a preceding space.

However, the pattern becomes much more clear when we plot the full OV circuit
taking into account all Letter Mover Heads. As it can be seen in Fig. 10,
there is now a clear diagonal pattern on the two rightmost plots. In other
words, this implies that when Letter Mover Heads attend to a capital letter
preceded with a space (which is exactly what Ci are), they translate it to the
token corresponding to the same capital letter without a space (i.e. Ai) and
write it into the residual stream. It is important to remark that during
analysis we did not use any information from the dataset, i.e. it was purely
performed by inspecting the weights.

Figure 10: Sum of every Letter Mover Head OV circuit, for all capital letter
tokens with/without a preceding space.

We also studied the copying behavior by analyzing the relationship between the
attention paid to Ci and the increase of the logits of Ai for different heads.
Specifically, it can be seen on Fig. 11 that it pays more attention to Ci when
predicting the $i$th digit, and that the value written along the logits of Ai
increases with such attention.

Figure 11: Projection of the output of head 8.11 along the logits correct
letter vs. the attention probability paid to Ci.

###  4.2 Positional Information Experiments

We also hypothesized that Letter Mover Heads should use positional information
to perform the final prediction (i.e. to attend to the first token of the
_first/second/third_ word), specially when predicting the first letter of the
acronym, as there is no available information regarding the previously
predicted letters of the acronym. In order to test this hypothesis, we first
study the positional embeddings, as it is the most evident source of
positional information of the model. Specifically, we swapped the positional
embeddings of different pairs of Ci and checked if it had an effect on the
attention pattern of Letter Mover Heads. Ideally, if a head relied in
positional embeddings, swapping them should force them to attend a different
letter. Fig. 12 shows the effect of swapping the positional embeddings of C1
and C3 on the attention probabilities of head 8.11.

Figure 12: Attention probabilities on head 8.11 when swapping the positional
embeddings of C1 and C3.

As it can be seen, the change in attention probabilities is negligible. We
performed all the possible swapping combinations for every letter mover head
and found similar results, implying that positional embeddings are not the
main source of positional information for Letter Mover Heads. However, the
model has to use some source of positional information to predict the first
letter, so we looked for another possible source. It has been recently
hypothesized (Heimersheim and Janiak, 2023) that models are able to derive
positional information from attention probabilities. Specially, due to causal
masking, the attention pattern paid to the Beggining of Sequence (BOS) token
position generally decreases with the destination token position. Therefore,
the model could infer the position of a certain token Ci by looking at the
attention paid to the BOS token: a lower attention paid to this token position
implies that the destination token is further from the start of the sentence,
and vice versa.

Therefore, we patched the activations of each head by swapping their attention
paid to the BOS token from tokens Ci and Cj for all possible combinations and
measured the change in logit difference. Fig. 13 shows the results for $i=1$,
$j=3$. Indeed, swapping the attention values does have an impact on the
performance, meaning that letter mover heads are likely to use positional
information derived from this mechanism, specially when predicting the first
letter. There are also other heads that contribute positively. We hypothesize
that these heads are writing on the opposite direction to avoid the model
becoming overconfident (similar to negative name mover heads on Wang et al.
(2023)). However, we leave this aspect as part of a future study, as this
requires an extensive analysis.

Figure 13: Change in logit difference obtained by swapping the attention paid
to BOS from the C1 and C3 for every head in the model.

In order to provide further evidence, we swapped the BOS attentions for those
heads that had a negative impact of at least $1\%$ in the previous experiment
and visualized the change of attention pattern on letter mover heads.
Specifically, Fig. 14 shows the attention probabilities paid to the Ci tokens
on head 8.11 on the clean run, swapping the positional embeddings, swapping
the BOS tokens and applying both swapping techniques. In general, swapping the
BOS tokens has the most impact across all predictions, meaning that head 8.11
does indeed use positional information. As expected, the greatest difference
can be found on predicting the first letter: swapping the positional
embeddings and the BOS tokens of C1 and C3 changes the average prediction from
A1 to A3. It is also important to remark that, most of the change on the
attention pattern, is caused by simply swapping two scalars on each of the
patched heads, i.e. a slight change in the attention pattern of the patched
heads causes a large impact on the attention probabilities of head 8.11.

Figure 14: Change in attention when performing the BOS attention swapping
experiment.

We also performed this experiment with every possible swapping combination and
found two important aspects. First, the attention probabilities are mainly
affected on the first letter prediction, aligned with our hypothesis that the
model relies mainly on positional information (i.e. it has to look for the
_first_ capital letter). Comparatively, the second and third letter prediction
rely on the previous predicted letter/s to gain some context. Second, we found
that only swapping C1 and C3 had a considerable impact, probably due to the
fact that the other two possible swaps are performed between tokens that are
closer together, hence the degree of corruption is smaller. This phenomena
also occured on the rest of letter mover heads. The experiment involving the
rest of letter mover heads and swapping combinations are presented in the
Supplementary Materials.

##  5 CONCLUSIONS

In this work, we identified the circuit responsible for the task of predicting
three-letter acronyms on GPT-2 Small via a series of activation patching
experiments. The discovered circuit was composed by 8 attention heads which we
classified into three different groups according to their role. We showed that
ablating every other head did preserve the performance, meaning that the task
of acronym prediction does indeed rely on the discovered circuit. We also paid
special attention to the most important heads of the circuit, which we termed
_letter mover heads_ , whose role is to attend to the capital letter of the
$i$th word and copy its content for the $i$th letter prediction. We provided
evidence of this behavior by studying their attention patterns, OV matrices
and output activations. We also show that these heads use positional
information and that this information is received not only by the positional
embeddings, but from the attention patterns. Our experiments show that the
positional information is derived from the attention paid to the BOS token, in
accordance to what it was discovered in simpler models (Heimersheim and
Janiak, 2023).

In summary, this is the first work that tries to mechanistically interpret a
task involving multiple consecutive token using MI, laying the foundation for
understanding more complex behaviors. Moreover, we strongly believe that MI
will enable us to understand larger models, increasing the safety and
trustworthiness of AI systems.

#### Acknowledgements

This work has been co-funded by the BALLADEER (PROMETEO/2021/088) project, a
Big Data analytical platform for the diagnosis and treatment of Attention
Deficit Hyperactivity Disorder (ADHD) featuring extended reality, funded by
the _Conselleria de Innovación, Universidades, Ciencia y Sociedad Digital
(Generalitat Valenciana)_ and the AETHER-UA project (PID2020-112540RB-C43), a
smart data holistic approach for context-aware data analytics: smarter machine
learning for business modelling and analytics, funded by the _Spanish Ministry
of Science and Innovation_. Jorge García-Carrasco holds a predoctoral contract
(CIACIF/2021/454) granted by the _Conselleria de Innovación, Universidades,
Ciencia y Sociedad Digital (Generalitat Valenciana)_.

## References

  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_ , 2021.  https://transformer-circuits.pub/2021/framework/index.html.
  * Elhage et al. (2022) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah.  Toy models of superposition.  _Transformer Circuits Thread_ , 2022.  https://transformer-circuits.pub/2022/toy_model/index.html.
  * Hanna et al. (2023) Michael Hanna, Ollie Liu, and Alexandre Variengien.  How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model.  In _Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.  https://openreview.net/forum?id=p4PckNQR8k.
  * Heimersheim and Janiak (2023) Stefan Heimersheim and Jett Janiak.  A circuit for Python docstrings in a 4-layer attention-only transformer.  https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/a-circuit-for-python-docstrings-in-a-4-layer-attention-only, 2023.
  * Li et al. (2023) Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song.  Multi-step jailbreaking privacy attacks on chatGPT.  In _The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2023.  https://openreview.net/forum?id=ls4Pfsl2jZ.
  * Lieberum et al. (2023) Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik.  Does circuit analysis interpretability scale? Evidence from multiple choice capabilities in Chinchilla.  _arXiv preprint arXiv:2307.09458_ , 2023.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov.  Locating and editing factual associations in GPT.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.  https://openreview.net/forum?id=-h6WAS6eE4.
  * Nanda and Bloom (2022) Neel Nanda and Joseph Bloom.  TransformerLens, 2022.  https://github.com/neelnanda-io/TransformerLens.
  * Nanda et al. (2023) Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt.  Progress measures for grokking via mechanistic interpretability.  In _The Eleventh International Conference on Learning Representations (ICLR)_ , 2023.  https://openreview.net/forum?id=9XFSbDPmdW.
  * Olah et al. (2020) Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.  Zoom in: An introduction to circuits.  _Distill_ , 2020.  doi: 10.23915/distill.00024.001.  https://distill.pub/2020/circuits/zoom-in.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah.  In-context learning and induction heads.  _Transformer Circuits Thread_ , 2022.  https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf.
  * Quintans (2023) Desi Quintans.  The Great Noun List.  https://www.desiquintans.com/nounlist, 2023.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_ , pages 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1162.  URL https://aclanthology.org/P16-1162.
  * Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting.  Large language models in medicine.  _Nature Medicine_ , 29(8):1930–1940, Aug 2023.  ISSN 1546-170X.  doi: 10.1038/s41591-023-02448-8.  URL https://doi.org/10.1038/s41591-023-02448-8.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Wang et al. (2023) Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.  Interpretability in the wild: a circuit for indirect object identification in GPT-2 small.  In _The Eleventh International Conference on Learning Representations (ICLR)_ , 2023.  URL https://openreview.net/forum?id=NpsVSN6o4ul.
  * Wei et al. (2023) Alexander Wei, Nika Haghtalab, and Jacob Steinhardt.  Jailbroken: How does LLM safety training fail?  In _Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.  URL https://openreview.net/forum?id=jA235JGM09.
  * Zhang et al. (2023) Zhiyue Zhang, Hongyuan Mei, and Yanxun Xu.  Continuous-time decision transformer for healthcare applications.  In _Proceedings of The 26th International Conference on Artificial Intelligence and Statistics (AISTATS)_ , volume 206 of _Proceedings of Machine Learning Research_ , pages 6245–6262. PMLR, 25–27 Apr 2023.  URL https://proceedings.mlr.press/v206/zhang23i.html.

## Checklist

  1. 1.

For all models and algorithms presented, check if you include:

    1. (a)

A clear description of the mathematical setting, assumptions, algorithm,
and/or model. [Yes/No/Not Applicable] Yes, at Section 2.

    2. (b)

An analysis of the properties and complexity (time, space, sample size) of any
algorithm. [Yes/No/Not Applicable] Not applicable. The focus of this work is
not on any algorithm but on discovering a circuit by using an already existing
technique. However, we specify the sample size used on Section 3.

    3. (c)

(Optional) Anonymized source code, with specification of all dependencies,
including external libraries. [Yes/No/Not Applicable] No, but it will be made
public upon acceptance.

  2. 2.

For any theoretical claim, check if you include:

    1. (a)

Statements of the full set of assumptions of all theoretical results.
[Yes/No/Not Applicable] Not applicable. The work presented here is mainly
empirical evidence.

    2. (b)

Complete proofs of all theoretical results. [Yes/No/Not Applicable] Not
applicable.

    3. (c)

Clear explanations of any assumptions. [Yes/No/Not Applicable] Not applicable.

  3. 3.

For all figures and tables that present empirical results, check if you
include:

    1. (a)

The code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL). [Yes/No/Not
Applicable] No, but it will be made publicly available upon acceptance.

    2. (b)

All the training details (e.g., data splits, hyperparameters, how they were
chosen). [Yes/No/Not Applicable] Not applicable, as no training is performed.
We study a pretrained model.

    3. (c)

A clear definition of the specific measure or statistics and error bars (e.g.,
with respect to the random seed after running experiments multiple times).
[Yes/No/Not Applicable] Yes

    4. (d)

A description of the computing infrastructure used. (e.g., type of GPUs,
internal cluster, or cloud provider). [Yes/No/Not Applicable] Yes, on the
start of Section 3.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets, check if you include:

    1. (a)

Citations of the creator If your work uses existing assets. [Yes/No/Not
Applicable] Yes

    2. (b)

The license information of the assets, if applicable. [Yes/No/Not Applicable]
Not applicable

    3. (c)

New assets either in the supplemental material or as a URL, if applicable.
[Yes/No/Not Applicable] Yes

    4. (d)

Information about consent from data providers/curators. [Yes/No/Not
Applicable] Not applicable

    5. (e)

Discussion of sensible content if applicable, e.g., personally identifiable
information or offensive content. [Yes/No/Not Applicable] Not applicable

  5. 5.

If you used crowdsourcing or conducted research with human subjects, check if
you include:

    1. (a)

The full text of instructions given to participants and screenshots.
[Yes/No/Not Applicable] Not applicable

    2. (b)

Descriptions of potential participant risks, with links to Institutional
Review Board (IRB) approvals if applicable. [Yes/No/Not Applicable] Not
applicable

    3. (c)

The estimated hourly wage paid to participants and the total amount spent on
participant compensation. [Yes/No/Not Applicable] Not applicable

##  Appendix A ATTENTION PATTERNS

This section contains additional attention pattern visualizations to support
the findings described in Section 3. Fig. 15 shows the mean attention patterns
for heads 1.0, 2.2 and 4.11, which were the main responsible of moving
information from previous words. As it can be seen, these heads have the
characteristic offset diagonal pattern of previous token heads, meaning that
these heads attend to the previous token w.r.t. the current one and copy their
information. On the other hand, heads 5.8, 8.11 and 10.10 move the previous
information into A(i-1) by attending to the tokens T(i-1) and Ci, as it can be
seen on Figs. 16-17.

The attention probability histograms of the rest of letter mover heads 10.10,
9.9 and 11.4 are shown on Figs. 17-19. Even though the histograms are noisier
than the one associated to the main letter mover head 8.11, it can clearly be
seen that these heads generally pay more attention to the proper capital
letter Ci compared with the other capital letters. We also see a high
attention paid to T(i-1), in particular on heads 9.9 and 10.10. This is likely
due to the fact that these heads (specially 10.10) also perform the role of
copying information about the previous capital letter, as previously-
mentioned.

Figure 15: Mean attention patterns for the 3 heads on the circuit that move
information from C(i-1) to Ci. Figure 16: Average probability paid from A(i-1)
to the previous token positions for head 5.8. Figure 17: Average probability
paid from A(i-1) to the previous token positions for head 10.10. Figure 18:
Average probability paid from A(i-1) to the previous token positions for head
9.9. Figure 19: Average probability paid from A(i-1) to the previous token
positions for head 11.4.

##  Appendix B POSITIONAL INFORMATION EXPERIMENTS

This section contains the remaining positional experiments (presented in
Section 4.2) regarding the rest of possible swapping combinations and letter
mover heads. Figs. 20 and 21 show the result of swapping the attention paid to
BOS from the C1 and C2 tokens, and the C2 and C3 tokens respectively.

Figs. 22-24 show the difference in attention paid to the Ci tokens on the
clean run, when swapping the positional embeddings, swapping the attention
paid to the BOS token, and performing both swaps at the same time. This is
visualized for every possible swap and letter mover head. As mentioned in the
paper, it can be seen that the largest impact happens when performing the
swapping operation on tokens C1 and C3, specially on the first letter
prediction, whereas the changes on the other swapping experiments are almost
negligible.

Figure 20: Change in logit difference obtained by swapping the attention paid to BOS from the C1 and C2 tokens for every head in the model. Figure 21: Change in logit difference obtained by swapping the attention paid to BOS from the C2 and C3 tokens for every head in the model. |
---|---
|
Figure 22: Effect of swapping the positional embeddings and/or attention to BOS of C1 and C2 on the attention paid to the capital letter tokens for each letter mover head. |
---|---
|
Figure 23: Effect of swapping the positional embeddings and/or attention to BOS of C1 and C3 on the attention paid to the capital letter tokens for each letter mover head. |
---|---
|
Figure 24: Effect of swapping the positional embeddings and/or attention to
BOS of C2 and C3 on the attention paid to the capital letter tokens for each
letter mover head.