# PIM-GPT: A Hybrid Process-in-Memory Accelerator for Autoregressive
Transformers

Yuting Wu2, Ziyu Wang2, Wei D. Lu  2 Equal Contribution; listing order
determined by random dice rolling.  Department of Electrical Engineering and
Computer Science, the University of Michigan, Ann Arbor, MI, USA
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Decoder-only Transformer models such as GPT have demonstrated superior
performance in text generation, by autoregressively predicting the next token.
However, the performance of GPT is bounded by low compute-to-memory-ratio and
high memory access. Throughput-oriented architectures such as GPUs target
parallel processing rather than sequential token generation, and are not
efficient for GPT acceleration, particularly on-device inference applications.
Process-in-memory (PIM) architectures can significantly reduce data movement
and provide high computation parallelism, and are promising candidates to
accelerate GPT inference.

In this work, we propose PIM-GPT that aims to achieve high throughput, high
energy efficiency and end-to-end acceleration of GPT inference. PIM-GPT
leverages DRAM-based PIM solutions to perform multiply-accumulate (MAC)
operations on the DRAM chips, greatly reducing data movement. A compact
application-specific integrated chip (ASIC) is designed and synthesized to
initiate instructions to PIM chips and support data communication along with
necessary arithmetic computations. At the software level, the mapping scheme
is designed to maximize data locality and computation parallelism by
partitioning a matrix among DRAM channels and banks to utilize all in-bank
computation resources concurrently. We develop an event-driven clock-cycle
accurate simulator to validate the efficacy of the proposed PIM-GPT
architecture. Overall, PIM-GPT achieves 41$-$137$\times$, 631$-$1074$\times$
speedup and 339$-$1085$\times$, 890$-$1632$\times$ energy efficiency over GPU
and CPU baseline, respectively, on 8 GPT models with up to 1.4 billion
parameters.

###### Index Terms:

Process-in-memory, GPT, Transformer, DRAM, Accelerator

##  I Introduction

Attention-based Transformer models have revolutionized natural language
processing (NLP) by capturing long-term dependencies in the input data [1].
Transformer models including GPT and BERT have demonstrated superior
performance in many NLP tasks such as text generation[2][3], text
classification[4][5][6], and machine translation[7][8] compared to convolution
neural networks (CNNs) or recurrent neural networks (RNNs) [2] [9] [10]. GPT
in particular has attracted widespread public interest. It is a decoder-only
Transformer model that generates context in an autoregressive manner by
producing a single token at one time [3]. However, the sequential processing
feature of GPT will induce severe resource under-utilization of GPU, which is
optimized for massively parallel processing. Additionally, GPT has two main
features compared to CNNs: (1) extremely large model size and (2) low compute-
to-memory-ratio. As shown in Figure 1, the GPT3-XL model consists of 1.15
billion parameters [10], more than a hundred times higher than common CNNs
such as ResNet-18 [11], while the arithmetic intensity per parameter
(ops/parameter) is 2.1, much lower than 48.3 in ResNet-18 .

Figure 1: (a) Parameter and computation cost comparison of GPTs and Resnet-18.
(b) Operation/parameter ratios for CNN and GPT models.

These properties make it difficult to accelerate GPT models efficiently using
architectures optimized for throughput such as GPUs. Several Transformer
accelerators have been recently proposed to improve the performance of GPT
inference [12][13][14]. However, these designs generally suffer from the
following drawbacks: (1) expensive hardware overhead such as the usage of
high-bandwidth memory (HBM) and dense in-memory logic; (2) model customization
such as model and token pruning, which will make the architecture less
flexible and cause accuracy loss; (3) incapability of long token length
generation due to inefficient intermediate data management; (4) lack of end-
to-end acceleration as most studies only focus on attention computation and
feed-forward layers. Moreover, many of the existing Transformer accelerators
are only designed for encoder-only model like BERT, instead of decoder-only
models like GPT [15][16][17][18][19][20][21].

DRAM-based Process-in-memory (PIM) is a promising architecture to accelerate
memory-bounded tasks [22][23]. The high storage capacity of DRAM allows all
model parameters to be stored. By adding compute elements to the DRAM chip,
PIM can consume data locally and minimize external DRAM data access. Recent
PIM developments show efficient acceleration of MAC operation on DRAM
chips[24] [25][26][27]. In comparison to conventional processor-centric
architectures, implementing compute-logic near memory banks is a data-centric
solution. However, only limited, low-density logic circuits can be fabricated
on DRAM chips [28][29]. Developments to date are limited to generic MAC
operations, and efficient end-to-end PIM acceleration of GPT inference still
needs to be developed.

In this work, we propose PIM-GPT, a complete solution for GPT inference
acceleration. At the hardware level, PIM-GPT is a hybrid system that includes
DRAM-based PIM chips to accelerate vector-matrix multiplication (VMM) near
data and an application-specific integrated circuit (ASIC) to support other
functions that are too expensive for PIM including necessary arithmetic
computations, data communication and initiating instructions to the PIM chips.
At the software level, mapping scheme is optimized to efficiently support the
GPT dataflow. To accommodate the large model size and improve performance, the
computation workloads are evenly distributed across PIM channels and banks to
maximize the utilization of available computation resources and on-chip
bandwidth. Multi-attention heads are concatenated to maximize data locality.
Compared to existing Transformer accelerators[12][13][14], the proposed PIM-
GPT supports large GPT models end-to-end without the need of expensive HBM,
making it a practical solution for GPT acceleration on the edge. Benchmarking
analysis shows the proposed PIM-GPT achieves state-of-the-art speedup and
energy efficiency for GPT inference tasks.

In summary, the main contributions of our work are as follows:

  * •

We design a hybrid architecture to accelerate the memory-bounded GPT inference
end-to-end using GDDR6-based PIM with minimal changes to DRAM.

  * •

We propose an optimized mapping scheme that maximizes data locality and
distributes workloads across PIM channels and an ASIC to achieve high
computation parallelism and efficiency.

  * •

We analyze performance, efficiency, scalability and sensitivity of the
proposed system through circuit synthesis and a clock-cycle accurate
simulator.

  * •

PIM-GPT achieves the state-of art performance 41$-$137$\times$,
631$-$1074$\times$ speedup and 339$-$1085$\times$, 890$-$1632$\times$ energy
efficiency over GPU and CPU baseline, respectively, for 8 GPT models with up
to 1.4 billion parameters.

##  II Background and Motivation

###  II-A Transformer Models

Figure 2 illustrates the typical structure of a Transformer model. Different
from CNNs and RNNs, the Transformer model uses a self-attention mechanism that
captures the relationship between different words in the whole sentence[1].

Figure 2: Transformer model architectures of BERT and GPT.

The original Transformer model consists of an encoder and a decoder, both
containing _N_ number of identical transformer blocks. Each block includes a
self-attention module and a feed-forward network (FFN). Among them, BERT and
GPT are the two most popular language models. BERT is an encoder-only model,
which processes all input tokens at once and attends to content in both
directions [2]. In contrast, GPT is a decoder-only model, which typically
handles a single token at one time and generates the next token in a
sequential manner by attending to all previous tokens [10].

Despite the differences, these two models share similar blocks, as shown in
Figure 2. For GPT, the input token is first transformed into a vector of
dimension $d_{m}$ by the input embedding layer, where $d_{m}$ is the feature
dimension of the model. Positional information of the token with respect to
the sentence is also added to the input embeddings. The processed token is
then fed into the Transformer blocks.

The input token is first multiplied with three linear transformation matrices
($W_{K,Q,V}$) to obtain Query (q) , Key (k) and Value (v) vectors, where
$W_{K}\in\mathbb{R}^{d_{m}\times d_{k}}$, $W_{Q}\in\mathbb{R}^{d_{m}\times
d_{k}}$, $W_{V}\in\mathbb{R}^{d_{m}\times d_{v}}$. The current Key and Value
vectors k, v are then concatenated to the key, value matrices computed from
the previous inputs to form the updated Key, Value matrices. The q vector and
Key, Value matrices are then passed to the self-attention head to capture the
dependencies between tokens with the following equation:

$\text{Attention}\left(q,K,V\right)=\text{softmax}(\frac{qK^{T}}{\sqrt{d_{k}}})V$
(1)

The scaled dot product between q and $K^{T}$ is first computed to obtain the
attention score, which measures the relation between the current token and all
previous tokens. Then a softmax operation is applied to normalize the
attention score between 0 and 1. In the next step, the attention score is
multiplied with the value matrix to produce the head output. To allow the
model to learn different relationships for each token, multi-head attention
technique is adopted. The input vectors are split across attention heads, and
each chunk goes through a separate head in parallel. All head outputs are
combined by a linear projection layer to produce the final attention output.

Following the multi-head attention, the attention output is fed into a FFN
network, which consists of two fully-connected networks with Gaussian Error
Linear Unit (GELU) activation function[30] in between. The attention and FFN
layers both contain a layer normalization and a residual connection, as shown
in Figure 2. The output from the attention block is then applied as inputs to
the next attention block for subsequent processing, and the process is
repeated through N attention blocks. Afterwards, a final output layer is used
to predict the next token. The content is generated autoregressively by
repeating this process until it reaches the required token length.

Unlike GPT, BERT processes all input tokens in parallel and produces the
outputs at once. Therefore, the core computation is matrix-matrix
multiplication, and the performance is computation-bounded. In contrast, for
GPT the core computation is VMM and the arithmetic intensity is relatively low
but the required memory access is high. As a result, throughput optimized
architectures such as GPUs are not efficient for GPT inference, while PIM
techniques that leverage compute capabilities on DRAM chips are promising for
GPT inference hardware acceleration.

###  II-B DRAM-based PIM

The conventional von Neumann architecture is a processor-centric design. Data
need to be transferred across the memory hierarchy to the processor, which
consumes about two orders of magnitude more energy than performing floating-
point operation [25][31]. Given these limitations, memory-centric computing
architectures have been widely studied to accelerate memory-bounded tasks
[22], such as genome sequence analysis [32][33], graph processing[29][34] and
machine learning[35]. By moving processing units closer to data, PIM
architectures enable data-centric, data-driven and data-aware designs. Amongst
all PIM technologies, DRAM-based PIM architectures are promising candidates
since DRAM is the mainstream memory in today’s computing systems.

The DRAM bank is a two-dimensional array of memory cells with a 1T1C
structure, as shown in the inset of Figure 3(c). The cell represents the
binary values of “1” or “0” by the presence or absence of charge. When reading
data from a DRAM bank, the wordline will be activated and data stream from an
entire row will be offloaded to the row-buffer. Accessing and consuming
existing data in the row-buffer is significantly efficient than from the array
to the processor. However, the DRAM fabrication process is highly constrained
for logic integration. It only contains three metal layers that severely limit
the complexity of the circuit, and the transistors are 3$\times$ slower than
those in logic chips at the same node[36][37]. Only until recent, advances in
DRAM technology has made PIM design applicable. For example, Samsung proposed
a function-in-memory DRAM (FIMDRAM) by adding programmable computing units
between two banks of HBM[24]. Generally, HBM has been used in existing PIM
architecture designs [14] [29][38][39][40] [41]. However, the high-cost
associated with HBM makes it more suitable for servers instead of edge
applications. Recently, SK Hynix announced GDDR6-based Accelerator-in-Memory
(AiM) by implementing logic at the bank level to accelerate MAC operations and
activation functions [26].

DRAM-based PIM can be divided into (1) process-using-memory (PUM) and (2)
process-near-memory (PNM). The first approach supports computing in DRAM banks
or subarrays. It can yield higher level of parallelism and less data movement.
However, to enable in-bank computation, extensive modifications to the
subarray architecture are needed[42][43][44]. Additionally, only a limited
instruction set is supported. The second approach implements computation logic
near DRAM banks and can take advantage of the large internal
bandwidth[24][26]. By balancing the computation logic design and bank
organization, PNM based PIM can achieve both high-performance and energy
efficiency.

###  II-C Motivation

As shown in Figure 1, the computation primitive of GPT has low ops/parameter
ratio and is memory-bounded since it processes and generates a single token at
one time. Compared to computation-bounded models, the sequential feature of
decoder cannot utilize parallel processing cores in GPU efficiently,
particularly for inference tasks without batching. Recently, several
Transformer accelerators have been proposed. Most of them mainly focus on the
acceleration of matrix-matrix multiplication in encoder rather than the end-
to-end model acceleration [15][16][17][18][19][20][21]. Only few considers GPT
inference [12][13][14]. However, the use of HBM to increase memory bandwidth
in these studies is expensive for edge inference applications.

PIM is a promising approach to relieve the memory bottleneck by storing matrix
and performing computation in memory. However, at the hardware level, non-VMM
operations are better performed in another chip to achieve efficient end-to-
end GPT acceleration. At the software level, efficient workload distribution
and dataflow management are required to fully exploit the advantage of PIM.
The proposed PIM-GPT is such an end-to-end GPT accelerator with practical
considerations in hardware implementation.

Figure 3: PIM-GPT system overview. (a) Hardware-aware GPT model partition. (b)
Compilation of computation stream to command stream. (c) PIM-GPT hardware
architecture.

##  III PIM-GPT Architecture

###  III-A System Overview

PIM-GPT is a memory-centric accelerator system aimed to support Transformer-
based autoregressive token generation models including GPT. The PIM-GPT
architecture is shown in Figure 3. The proposed system includes 8 GDDR6-based
PIM channels with a custom designed ASIC. Each proposed PIM channel contains
16 memory banks, and only requires the integration of 16 on-chip MAC units and
a shared global buffer (GB). Compared to HBM-PIM systems, GDDR6 DRAM does not
rely on expensive 2.5D multi-die stack and through-silicon vias (TSVs), and is
a low cost option for edge application. Our PIM design follows JEDEC
standard[45] and operates at 1 GHz.

The design principle of PIM-GPT is to maximally leverage data locality and
parallelism to achieve high system performance and energy efficiency. PIM-GPT
supports end-to-end acceleration of large GPT models with long token length
generation through: (1) hardware-aware model partition, (2) parallelism-aware
instruction compilation and (3) throughput-aware dataflow between PIM and
ASIC. As shown in Figure 3(a), weight values from multiple heads will be
concatenated and distributed to multiple channels for parallel operation,
which will be elaborated in Section IV. PIM-GPT is integrated with a data-
triggered instruction scheduler, which compiles high-level computation streams
of the model to DRAM/ASIC command streams. Figure 3(b) illustrates the
compilation of ➍ and ➎ to command streams for PIM and ASIC, respectively. The
hardware architecture is shown in Figure 3(c). In the hybrid architecture, the
PIM chips execute VMM locally by broadcasting the vector stored in the GB and
loading matrices from DRAM bank arrays to the MAC units. Only input/output
vectors need to be transferred between PIM and ASIC instead of whole matrices.
The ASIC computes non-VMM arithmetic computations, as well as manages data
communication and intermediate data storage. All data in PIM-GPT are in
bfloat16 (BF16) format, which preserves the approximate dynamic range of
32-bit floating point number to balance performance and accuracy.

###  III-B GDDR6-based PIM Architecture

The GDDR6-based PIM architecture is shown in Figure 4. Unlike existing PIM
designs [14][29], our design only requires lightweight modifications to the
conventional GDDR6 architecture. Integrating large SRAM buffer into DRAM is
expensive and impractical. Therefore, only a 2 KB GB is added to each channel
to store the input vector. The vector from GB can be broadcast to MAC units in
all 16 banks to perform MAC simultaneously, as shown in Figure 4(a). When the
input length exceeds the GB size, partial MAC results will be computed and
forwarded to the SRAM on ASIC, followed by downstream partial sum execution on
the ASIC. Compared to writing back to DRAM, forwarding the subportion of VMM
to ASIC reduces overall latency. Eliminating writing back also frees DRAM
banks to perform subsequent parallel VMM. The data transmission rate of GDDR6
is 16 Gb/s/pin, and each channel has 16 pins leading to a total memory
interface bandwidth of 32 GB/s.

The bank organization is identical to conventional DRAM architectures. as
shown in Figure 4(b). Once a row address is decoded, the entire corresponding
row will be activated and all stored data will be forwarded to the row buffer.
Herein, if the bank is not closed (precharged), data will be preserved in the
row buffer. Reading data from the row buffer is much faster than from the bank
array, since it skips the long latency row activation step. Whether or not to
precharge a row immediately after accessing the data is determined by the
scheduling policy: open-row or close-row. Hence, using open-row policy can let
the MAC unit consuming data much faster from DRAM if the data are properly
mapped. In our optimized mapping scheme, the corresponding weights from DRAM
banks share the same row addresses and the row hit rate is maximized, which
will be discussed in Section IV.

Inside each bank, only multipliers and an adder tree are implemented for MAC
operation. Similar architecture has been proven to be effective in prior
designs [26][46][47]. During MAC operation, 16 vector values and corresponding
weights are fetched from the GB and banks in the PIM channel, respectively.
The 16 multipliers multiply the vector data with weights. The adder tree
accumulates the multiplication results for downstream computation, as shown in
Figure 4(c). The MAC units are operated in a pipelined fashion to maximize the
throughput of the system, i.e. once the multiplication is done, the
multipliers fetch the next chunk of vector and weight in the next clock cycle.
The same principle is used for the adder stages. To minimize hardware cost and
improve efficiency, PIM-GPT only performs VMM operations in the PIM while
assigning all other computations, such as data bypassing, division and
activation functions, to the ASIC. By doing so, the design allows the
integration of lightweight MAC units to the DRAM chip to consume data locally
without significantly sacrificing the memory capacity.

Figure 4: DRAM PIM organization. (a) A channel is composed of a global buffer
and 16 banks. A bank contains (b) a conventional DRAM bank and (c) a MAC unit
with multipliers and an adder tree.

###  III-C ASIC Architecture

The ASIC in PIM-GPT is used to manage data communication and support non-VMM
arithmetic computations and intermediate data storage. The ASIC architecture
is shown in Figure 5. Since VMM operations are performed in the PIM channels,
data will only be read from DRAM PIM when a VMM operation is done and requires
downstream computation or communication. The PIM channels communicate with the
ASIC through memory bus and crossbar interconnects. The interconnects support
data fetching from any DRAM channel and sending memory requests to a single
channel or broadcasting data to all channels after packeting data with
address. Data read from PIM have two possible paths on the ASIC: (1) writing
back to banks in other PIM channels, such as Key, Value matrices for
subsequent VMM operations, and (2) going through computation blocks in the
ASIC, such as layer normalization, softmax, etc.

If the data require downstream computation, they will be temporarily stored in
the on-chip SRAM buffer. To further improve performance, PIM-GPT leverages
pipelining between data transmission and ASIC computation to hide the data
transmission latency caused by the limited GDDR6 bandwidth. The computation
engine in ASIC can start execution when partial data are ready. The SRAM
buffers and computation engines in the ASIC of the proposed PIM-GPT are
designed for billion-parameter level models such as GPT3-XL. For smaller
models or instructions that only utilize portions of the computation
resources, power gating schemes will be applied to SRAM arrays or unused
computation blocks to lower the ASIC power consumption.

Figure 5: ASIC architecture of the PIM-GPT system.

PIM-GPT is designed for full stack autoregressive model acceleration. The
computation engines on ASIC are responsible for computations that cannot be
run as MACs, such as activation functions, layer normalization, softmax and
partial sum. Details of the computation engine implementation will be
discussed in the next sub-section.

###  III-D ASIC Computation Block Design

The adders and multipliers in the computation engines follow the standard
floating-point unit design to support summation and multiplication. For design
reuse and performance considerations, other computation tasks are all
implemented with approximation algorithms using only addition and
multiplication to achieve the required precision.

Three functions that require approximation are shown in Equation 2$-$4:
softmax, layer normalization and activation function GELU further approximated
using Equation 4.

$s(x_{i})=\frac{e^{x_{i}}}{\Sigma_{j=1}^{N}e^{j}}$ (2)

$y=\frac{x-E[x]}{\sqrt{Var[x]+\epsilon}}\times\gamma+\beta$ (3)

$\begin{split}&\text{GELU}(x)=\frac{x}{2}\times[1+erf(\frac{x}{\sqrt{2}})]\\\
=\frac{x}{2}\times&[1+\tanh{\sqrt{2/\pi}(x+0.044715\times x^{3})}]\end{split}$
(4)

The nonlinear function $e^{x}$, $\tanh{x}$, division and square root in these
functions cannot be naively computed using addition and multiplication. Under
given precision and data range, they can be efficiently approximated and
converge in rapid iterations. Here $e^{x}$ and $\tanh{x}$ are computed using
Taylor series approximation with the first six items, which can be computed
with addition and multiplication.

Data: BFloat16 $D=(S)M\times 2^{E}$

Result: BFloat16 $\frac{1}{D}$

/* Scale by exponent subtraction */

1 $D^{\prime}=D/2^{(E+1)}$

2 $X=\frac{48}{17}-\frac{32}{17}\times D^{\prime}$

3 for iter in $\lceil\log_{2}\frac{P+1}{\log_{2}17}\rceil$ do

4 $X=X+X\times(1-D^{\prime}\times X)$

5

6 end for

/* Scale the result */

7 $X^{\prime}=X/2^{(E+1)}$

Algorithm 1 Newton-Raphson Division

The division operation is computed by multiplying the numerator with the
inverse of the denominator. Both the reciprocal and inverse square root
operations can be calculated with addition and multiplication following
Newton’s method. A proper initial value is required to ensure the fast
convergence. For reciprocal, we take advantage of Newton-Raphson division in
Algorithm 1. The fast inverse square root algorithm is adopted from Quake III
Arena’s source code [48], as shown in Algorithm 2.

The Newton-Raphson Division algorithm shown in Algorithm 1 is friendly for
floating point numbers since it requires the input D to scale to a value close
to 0, which can be easily done with exponent subtraction and mantissa shift in
floating point data format. The P in line 3 is the precision of P binary
places. Hence, for a 16-bit floating point number, it will take three
iterations to get an accurate result. The fast inverse square root algorithm
in Algorithm 2 unpacks the BF16 data into 16-bit integer data and padding
16-bit zeros to get an accurate approximation, followed by shift and
subtraction from a constant. The pack step utilizes the 16 high bits of INT32
$L’$ to assemble a BF16 data’s sign bit, exponent and mantissa. In the fast
square root algorithm, it can converge in a single step iteration. Here we
take a conservative two step iteration.

##  IV PIM-GPT Dataflow

###  IV-A Overview

PIM-GPT distributes workloads among all PIM banks and ASIC efficiently. For
VMM operation, the input vector is stored in the ASIC SRAM buffer and
broadcast to the GB of all PIM channels. All MAC units will execute the same
MAC instruction on different matrix partitions concurrently to fully utilize
the PIM computation resources. The subvectors from channels will be sent back
to ASIC for concatenation and downstream communication and computation. PIM-
GPT implements the following techniques to coordinate workload between the PIM
channels and the ASIC: (1) The partial outputs of VMM can be forwarded to the
ASIC before the whole computation is completed, which effectively eliminates
the data write back to DRAM banks; (2) When input vector length exceeds the GB
size, SRAM buffer on the ASIC are reserved to store intermediate data and the
ASIC will accumulate partial VMM results from DRAM; (3) Pipelining between
data transmission and computation, i.e. the ASIC will start operations on
partially received vector while the rest are in transmission.

Data: BFloat16 $D=(S)M\times 2^{E}$

Result: BFloat16 $\frac{1}{\sqrt{D}}$

1 $D^{\prime}=D\times 0.5$

/* Unpack data and pad with 0 */

2 uint32_t $L$ $\leftarrow$ {unpack($D^{\prime}$), $0$x$0000$}

3 $L^{\prime}=0$x5f3759df $-L>>1$

/* Keep 16 high bits */

4 BFloat16 $X$ $\leftarrow$ pack($L^{\prime}$)[$31:16$]

5 for iter in IterNum do

6 $X=X\times(1.5-D^{\prime}\times X\times X)$

7

8 end for

Algorithm 2 Fast Inverse Square Root

Data: Computation Graph; PIM Configuration

Result: Memory Mapping and Reservation

/* Map weights to PIM banks */

1 for vmmBlock in Computation Graph do

2 if vmmBlock.multiHead then

3 hitScore $\leftarrow$ maxRowHit(nhead, ncol)

4 vmmBlock $\leftarrow$ concat(hitScore, vmmBlock)

5 end if

6 Mapping $\leftarrow$ maxParallel(vmmBlock, nch, nbank)

7

8 end for

/* Reserve PIM bank rows for kv */

9 for wrBlock in Computation Graph do

10 if block == write_k then

11 hitScore $\leftarrow$ maxRowHit(nhead, ncol, ltoken)

12 wrBlock $\leftarrow$ concat(hitScore, wrBlock)

13 end if

14 Mapping $\leftarrow$ Reserve(wrBlock, ltoken, nch, nbank)

15 end for

Algorithm 3 Model Mapping to PIM Banks

The model mapping includes storing the weights to the allocated banks, and
reserving space for the intermediate data (Key, Value matrices) for attention
computation. To enhance the system performance, the mapping scheme is
optimized to: (1) maximize row-hit rate by exploiting data locality; (2)
increase computational parallelism by balancing the workload across DRAM
banks; and (3) reduce latency by minimizing data movement. During runtime, the
system automatically computes the bank address in the reserved space to write
back the generated Key and Value vectors. The high-level description of the
mapping scheme is shown in Algorithm 3.

###  IV-B Mapping Methods

Weight Mapping:

Figure 6: Mapping strategy. (a) Concatenate multi-head to exploit data
locality. (b) Distribute weight matrix (left) evenly across channels and banks
to maximize computation parallelism.

As shown in Figure 6, the mapping scheme leverages data locality and maximizes
computation parallelism. Since commands such as activation (ACT) and precharge
(PRE) are expensive, achieving a high row-hit rate is preferred. To this end,
matrix data used for MAC operations need to be mapped to consecutive physical
DRAM cells. This approach means the corresponding row only needs to be
activated once to transfer all required data to the row buffer, and the MAC
units can keep consuming data already in the opened row to minimize ACT and
PRE operations.

To take advantage of the data locality, it is desired that a row is fully
mapped with data. However, a single attention head can be much smaller than
the DRAM array dimension, e.g. head dimension of GPT2-XL is 64 while a row can
store 1024 16-bit data. To maximize row hit rate, all attention heads in the
same layer are concatenated to fill up the DRAM bank, as shown in Figure 6(a).
To maximize the utilization of MAC units, the matrices are evenly distribute
across PIM channels and banks, as shown in Figure 6(b). Figure 6(b) is a
detailed example showing how $W_{v}$ matrices are mapped through 8 PIM
channels. First, attention heads in a layer are concatenated along column
direction to form a larger matrix. The concatenated matrix is mapped following
the row major approach and evenly distributed to all available channels, as
indicated by the colors in Figure 6(a) and (b). Inside each channel, all 16
banks are mapped with a portion of the matrix and execute MAC operations in
parallel. The computation is similar when it comes to VMM in the attention
projection and FFN layers. This mapping scheme distributes weights evenly to
achieve the highest possible DRAM channel-wise and bank-wise parallelism and
run as many MAC units at the same time as possible.

Intermediate Data Memory Reservation:

Figure 7: (a) Write k row-wise. (b) Write v column-wise.

Key and Value results need to be written back to the PIM banks and append to
the existing Key and Value matrices. In the mapping stage, PIM-GPT reserves
the required space in PIM banks for these intermediate data. The write back
scheme of key and value results are shown in Figure 7(a) and (b),
respectively. Key and Value write back are in row-major and column-major,
respectively, since the transpose of Key matrix is required in Equation 1,
while no transpose of Value matrix is required.

PIM-GPT exploits data locality during write. During runtime, the Key vectors
produced by multi-attention heads will be concatenated together. As shown in
Figure 7(a), Key vectors with length of 64 from 12 heads are concatenated to
form a vector with length of 768, and written to the corresponding PIM bank
row reserved for the current token, as highlighted by magenta in Figure 7(a).
The write command can be executed consecutively after one ACT to store the
whole Key vector, as shown in the timing diagram in Figure 7(a). The
concatenated Key vectors produced by all token generation steps are evenly
stored across all channels, as indicated by the color blocks in Figure 7(a).
Within each channel, they are evenly distributed to all banks.

Value results are stored in column-major fashion. As shown in the timing
diagram of Figure 7(b), in this case we can only write one data in an
activated row. Then we close the row and move to the next row for the next
data. Hence, data locality cannot be leveraged in Value result write back. To
maximize the write throughput and computation parallelism in the subsequent
VMM step, we distribute the Value matrix to all channels and banks, as shown
in Figure 7(b).

##  V System Evaluation

###  V-A Evaluation Method

Hardware Configuration As DRAM fabrication process is highly constrained
compared to the CMOS process, we refer AiM prototype reported from SK Hynix
for area overhead evaluation. The area of one processing unit (PU) in AiM is
0.19 $mm^{2}$ [26]. Apart from MAC operation, the AiM PU includes bias
addition and activation functions. For the proposed PIM-GPT, these additional
functions will be performed on the ASIC, and only MAC units are integrated on
the DRAM chip resulting in less area overhead than the AiM PU design. All
logic components of the ASIC are synthesized with SystemVerilog using Synopsys
Design Compiler at TSMC 28nm HPC+ process node. Area and power of the logic
components are obtained from the synthesis results. The area and power of SRAM
buffer are extracted from the TSMC 28nm datasheet based on Synopsys Memory
Compiler. The ASIC only consumes a core area of 0.64 $mm^{2}$, and the peak
power is 304.59 $mW$.

For DRAM energy benchmark analysis, we synthesized the optimized MAC units at
28 nm technology, followed by scaling the voltage to 1.25V to match the GDDR6
supply voltage [26]. Since routing is more complex in DRAM due to the limited
metal layers compared to CMOS logic process, we conservatively multiply the
power by 1.5, which comes to 149.29 $mW$ for 16 MAC units. Since GDDR6 can be
fabricated in 1ynm technology [26], the actual power consumption is expected
to be lower than this estimate. Table I lists the timing constraints and
current values used to model the PIM behavior for each command. For PIM
related commands, the timing constraints are obtained from [26]. For normal
DRAM commands, we adopt GDDR5 timing constrains in [49] to make a conservative
estimation due to the lack of detailed information of GDDR6. Similarly, the
current values are obtained from DDR5 datasheet [50]. The system performance
in latency and power efficiency is evaluated based on these conservative
assumptions. Detailed hardware configuration is summarized in Table I.

Simulation Configuration To evaluate the PIM-GPT system performance, we
developed an event-driven clock-cycle accurate simulator in C++ that models
the system behavior at token generation runtime. The simulator takes the GPT
model and system configuration as inputs for model mapping. The computation
graph is compiled into an instruction sequence following the hardware
constraints.

The PIM and ASIC behaviors are abstracted as state machines. The PIM hierarchy
is organized as a tree structure at package, channel and bank levels. The PIM
package node refers to the entire PIM portion that contains 8 PIM channels as
child nodes. Each channel consists of 16 banks as child nodes. The state
update will traverse down the tree from the root to the leaves. The same
organization hierarchy can be found in other memory simulators, such as
Ramulator[49].

The transition of states follows the timing constraints. At every clock cycle,
the simulator checks the status of the ASIC and the PIM package. If both ASIC
and PIM are in Idle state, the current instruction is completely consumed. It
then fetches the next instruction, which will be decoded into command
sequences. The ASIC or PIM will be put into Process state after the
instruction is issued. The simulator will compute the future time next_time
that the ASIC or relevant PIM banks will complete the triggered events based
on the latency model. The simulator keeps track of the status of all hardware
components. If the CLK reaches the next_time, the status of the corresponding
node will be changed back to Idle.

TABLE I: PIM-GPT Baseline Hardware Configuration. Timing Constraint |  tRCD=12ns, tRP=12ns, tCCD=1ns, tWR=12ns, tRFC=455ns, tREFI=6825ns
---|---
IDD |  IDD2N=92mA, IDD3N=142mA, IDD0=122mA, IDD4R=530mA, IDD4W=470mA, IDD5B=277mA
GDDR6
Specification |  Channel = 8, Banks/channel = 16, Capacity/channel = 4Gb, Row size = 2KB, Column number = 16k, Frequency = 1GHz, Pins = 16/channel, Data rate = 16Gb/s/pin
PIM |  Buffer = 2KB/channel, MAC unit = 1/bank, Frequency = 1GHz, Power = 149.29mW
AISC |  Technology 28nm, Frequency = 1GHz, SRAM = 128KB, # of Adders = 256, # of Multipliers = 128, Area = 0.64mm2, Power = 304.59mW

Benchmark Analysis The performance and energy efficiency of the proposed PIM-
GPT system are evaluated using the simulator and compared to GPU (NVIDIA T4)
and CPU (Intel Xeon Gold 6154). 4 GPT2 and 4 GPT3 models with up to 1.4
billion parameters are implemented on the proposed PIM-GPT system, and used
for the benchmark analysis. The simulator reports the total latency and the
command sequences that are performed by the PIM channels and the ASIC. For PIM
power, we multiply the IDD values consumed during each command with the
corresponding latency and VDD, following the standard procedure [50][51]. DRAM
refresh operations are also included. The energy consumed by the PIM MAC units
and by the ASIC are computed by multiplying the latency reported by the
simulator with the synthesized power consumption.

We select NVIDIA T4 as the GPU benchmark as it also uses GDDR6 as memory for a
fair comparison. For GPU, latency is recorded using torch.cuda.Event(), and
power is measured with pynvml, which is a wrapper around the NVIDIA management
library. The dynamic power consumption is tracked at each token generation and
multiplied with the corresponding latency to get the total energy. For CPU
characterization, we use python package time.time() for latency measurement
and an open-source terminal tool s-tui for power monitor. In each measurement,
we generate 1024 tokens and repeat 10 times, and report average energy and
latency values.

###  V-B Overall System Performance

Speedup As shown in Figure 8, the PIM-GPT system achieves remarkable
performance improvements over GPU and CPU: 41$\times$ to 137$\times$ speedup
over GPU T4 and 639$\times$ to 1074$\times$ speedup over CPU Xeon for the 8
GPT models. The high speedup originates from three aspects: (1) Memory
bottleneck is effectively removed by performing the memory-intensive VMM
operations inside PIM channel; (2) The mapping strategy maximizes computation
parallelism and data locality; (3) Different workloads are efficiently
distributed between PIM and ASIC. In comparison, GPU is not suitable for
sequential token generation, since the large memory footprint and low data
reuse rate under-utilize the GPU computation resources [13].

We also compare the PIM-GPT performance with previously reported Transformer
accelerators, as shown in Table II. SpAtten[12] accelerates GPT2-medium by
35$\times$ over GPU, and TransPIM[14] obtains similar speedup of 33$\times$.
Both ignore the layer normalization and residual connections in Transformer
models. DFX[13] provides 3.2$\times$ latency reduction on average. PIM-GPT
achieves state-of-the-art performance with 89$\times$ speedup over GPU. We
also want to highlight that the PIM-GPT testing result is based on 1024
tokens, which cannot be supported by these prior prototypes.

Figure 8: Speedup w.r.t GPU and CPU.

Energy Efficiency Figure 9 shows that the PIM-GPT system achieves energy
reduction of 339$\times$ to 1085$\times$ and 890$\times$ to 1632$\times$ over
GPU and CPU, respectively. PIM-GPT effectively eliminates the energy
consumption of DRAM data transmission by using PIM to locally consume data.
Additionally, the mapping method leverages data locality and minimizes the row
ACT and PRE operations that are energy consuming. The ASIC only contributes a
very small fraction of the total system energy, but provides highly efficient
arithmetic computations.

In comparison, DFX only achieves 3.99$\times$ higher energy efficiency
compared to the GPU baseline[13]. TransPIM reports $\sim$250$\times$ energy
reduction[14]. SpAtten reports 382$\times$ over GPU[12], but only the
attention layer is considered for energy consumption while others including
FFN layers are not considered. Speedup and energy efficiency for the above
mentioned accelerators are summarized in Table II.

###  V-C Detailed Performance Analysis

Figure 9: Energy efficiency improvement w.r.t GPU and CPU.

The layerwise latency breakdown of GPT3-small and GPT3-XL in Figure 10 shows
that VMM operations dominate the total execution time for PIM-GPT. All other
arithmetic computations only account for 1.16% of total latency in GPT3-XL.
For larger Transformer models, the improvement of PIM-GPT over GPU is reduced.
This is because larger GPT models allow better utilization of GPU computation
resources, while PIM has limited computation resources and the performance of
PIM-GPT is computation-bounded. As a result, the gain over GPU is reduced when
compared with smaller GPT models, although the performance gain
($>$40$\times$) is still significant.

The data locality is optimized during model mapping to reduce the memory
access time and enhance the computation throughput. Figure 11(a) plots the row
hit rates, achieving $\sim 98\%$ for all tested GPT models. For PIM-GPT, the
VMM computation occurs on the same DRAM chip that stores the required weights
and Key, Value vectors. Therefore, a significant amount of data movement can
be eliminated. The data movement only happens when MAC results are transmitted
to the ASIC for downstream processing or data synchronization. Figure 11(b)
shows the data transfer reduction can be as large as 110$\times$ to
259$\times$. In PIM-GPT, data movement no longer becomes the bottleneck and
consumes a very small proportion of the total latency, as illustrated by the
layerwise breakdown in Figure 10.

TABLE II: Comparison with Other GPT Accelerators.

|  SpAtten
[12] |  TransPIM
[14] |  DFX
[13] |  PIM-GPT
---|---|---|---|---
Memory |  HBM |  HBM |  HBM+DDR |  GDDR6
End-to-end |  ✗ |  ✗ |  ✓ |  ✓
PIM |  ✗ |  ✓ |  ✗ |  ✓
Data Type |  INT |  INT |  FP16 |  BF16
Largest Model |  GPT2-medium |  GPT2-medium |  GPT2-XL |  GPT2/3-XL
Longest Token |  32 |  - |  128 |  8096
Speedup |  35$\times$ |  33$\times$ |  3.2$\times$ |  89$\times$
Energy
Efficiency |  - |  $\sim$260$\times$ |  3.99$\times$ |  618$\times$

Notes: Speedup and energy efficiency are over GPU. PIM-GPT results are based
on 1024 token generation. Other architectures generate the specified token
lengths.

Figure 10: Layer-wise latency breakdown of (a) GPT3-small and (b) GPT3-XL.
Figure 11: (a) Row hit rate. (b) Reduction of data movement.

###  V-D Sensitivity Study

We conduct sensitivity studies on the ASIC clock frequency and data transfer
rate between PIM and ASIC to prove the robustness of the PIM-GPT design.

ASIC Frequency The PIM-GPT ASIC is designed with TSMC 28 nm technology at 1
GHz clock frequency. However, frequency scaling is an important technique to
optimize the power consumption, especially for edge devices. We conduct a
sensitivity study of ASIC frequency by varying the latency setting in our
simulator. Figure 12 shows the latency at different clock frequencies for the
8 GPT models, where the latency is normalized with respect to the 1 GHz
results. Overall there is only a small latency increase when the ASIC
frequency scales down from 1 GHz to 200 MHz for all models. Even when further
scaling down the frequency to 100 MHz, which is 10 times slower than the
baseline, the worst case only incurs a performance slowdown of 20%. Moreover,
the larger models are less sensitive to the ASIC frequency scaling, since
their operations are more dominated by VMM and the proportion of ASIC
arithmetic computation is less than in smaller models. Therefore, the PIM-GPT
design is not sensitive to ASIC clock frequency, which justifies its use for
edge applications where power often needs to be optimized by reducing clock
frequencies.

Figure 12: Sensitivity of performance to ASIC clock frequency.

Data Transmission Rate The memory interface can be a bottleneck for many
computation tasks, even for PIM implementations. In [27], SK Hynix reported
that the performance of the 8-channel GDDR6-AiM system degrades by two orders
of magnitude for matrix-vector multiplication (4096 $\times$ 4096) and
handwritten digit recognition [784 $\times$ (4096 $\times$ 4096 $\times$ 4096)
$\times$ 10, ReLU] tasks when the memory interface bandwidth is reduced from
300 Gb/s to 2 Gb/s [27]. When only accelerating fully connected layers in GPT
models, the system with 4 channels experienced $\sim$3$\times$ slowdown when
memory interface bandwidth changes from 16 Gb/s to 2 Gb/s [52].

We test the PIM-GPT’s sensitivity to the memory interface by changing the
bandwidth configuration in our simulator. Figure 13 shows the latency as a
function of memory interface bandwidth for 8 GPT models. When the memory
interface bandwidth changes from 16 Gb/s to 2 Gb/s, the end-to-end GPT
inference time is increased $\sim$1.5$\times$ on average. That is 2$\times$
better than reported in [52], where only VMM and GELU in GPT are considered.
Even when the data transfer rate is decreased to 1 Gb/s, all models are slowed
down by $\sim$2$\times$ on average. These results show PIM-GPT is not
sensitive to the memory interface bandwidth, since most data are consumed
locally and the data transfer requirements are significantly reduced.

###  V-E Long Token Support

As Transformer model sizes and token lengths increase, the required
computation grows quadratically with the sequence length [53]. A lot of
efforts have been devoted to reduce the computation and memory footprint for
longer sequences by exploiting the sparsity and quantization of Transformer
models [12] [54][55]. Another approach is to approximate the softmax
attention, e.g. linear Transformer [53], Performer [56] and spikeGPT[57],
which reduces the space complexity of attention from quadratic to linear.
However, approximations can impact the accuracy of large Transformer models,
while dynamic pruning at runtime adds hardware design complexity [12][46]. The
proposed PIM-GPT mapping scheme evenly distributes model parameters across PIM
channels, therefore can support more than 8k token generation for GPT3-XL.
Figure 14 shows normalized latency of different token lengths with respect to
1k tokens.

Figure 13: Sensitivity of performance to data transfer rate. Figure 14: PIM-
GPT latency with increased token length.

###  V-F Scalability

Since the performance of PIM-GPT is computation-bounded, the latency can be
reduced by adding more computation resources. Figure 15 evaluates the
scalability of the system. When the computation throughput of the MAC unit is
increased from processing 16 MACs to 64 MACs at a time, the latency is reduced
by 1.8$\times$ and 2.0$\times$ for GPT3-small and GPT3-XL, respectively, as
shown in Figure 15(a). The speedup is sub-linear. This is because when PIM
performs MAC operations on a bank row, the row needs to be activated and
precharged, which consumes a significant amount of time. We also evaluate
performance enhancement by increasing computation parallelism, as shown in
Figure 15(b). More memory channels can be attached to the ASIC with relatively
minor modifications to the ASIC data port. The improvement scales almost
linearly with the number of channels. The speedup slightly reduces for longer
sequences, because longer sequence requires more arithmetic computation for
ASIC to process.

##  VI Related Work

Process-in-Memory DRAM-based PIM is one of the most promising technologies to
accelerate data-intensive computing tasks. UPMEM is the first reported PIM
product using standard DIMM modules, with a large number of processors
combined with DRAM chips [36][37]. However, the throughput per PU is 4 GOPS
and it only supports INT8 data type. DRAM vendors including Samsung [24][25]
and SK Hynix[26][27] have recently announced DRAM-based PIM technologies.
Samsung’s PIM architecture is based on HBM2, which offers high bandwidth of
307.2 GBps to tackle data intensive tasks. The design integrates PIM dies on a
buffer die through TSV. Inside each PIM die, a PU is shared by two banks,
operating at 9.6GFLOPS per PU. However, the high cost of HBM limits the
application to server level. SK Hynix’s GDDR6-based PIM prototype,
Accelerator-in-Memory (AiM), uses the standard GDDR6 interface. AiM supports
VMM with high throughput of 32GFLOPS per PU. However, AiM alone cannot
accelerate end-to-end applications due to the limited logic and arithmetic
units that are fabricated on DRAM chips. For example, AiM cannot support
complex interlayer functions such as softmax, normalization, and residual
connections. Compared to AiM, PIM-GPT reduces PIM cost by limiting PIM only to
VMM operations, and performs the other arithmetic and control functions in a
separate ASIC with optimized mapping and dataflow design to allow efficient
end-to-end GPT acceleration.

Figure 15: Scalability of PIM-GPT with increased number of (a) MAC units and
(b) channels.

GPT Accelerators Recently, several Transformer accelerators have been
proposed. Most of them target computation-intensive encoder
models[15][16][17][18] and are not well-optimized for the memory-bounded GPT
inference. Few works accelerate GPT token generation tasks [12][13][14]. They
all rely on the high cost HBM. These architectures are not well-optimized for
the memory-bounded GPT inference. Hence, the performance improvement is
limited. DFX[13] partitions the GPT models to multiple FPGAs to leverage
parallel computation, but does not reduce the data load/store and requires
both DDR4 and HBM for GPT models. SpAtten[12] involves algorithmic methods
such as pruning and quantization to reduce memory overhead. However, these
methods inevitably reduce the model inference accuracy. SpAtten also only
speeds up the attention mechanism in Transformer. TransPIM[14] takes advantage
of PIM and proposes a token based dataflow requiring a large ring broadcast
buffer (2kb) per bank to store temporary data and direct link between two
adjacent banks. This design requires subarray-level modification that adds
significant overhead to the DRAM chip. Arithmetic computations like layernorm
and residual connections are not considered either. Moreover, these prior
works only demonstrate acceleration for limited token length. The proposed
PIM-GPT optimizes the workload distribution to exploit the benefits of both
PIM and ASIC to achieve fast end-to-end GPT acceleration on the edge.

##  VII Conclusion

In this work, we propose a hybrid hardware system, PIM-GPT, to accelerate the
memory-bounded GPT token generation tasks. PIM-GPT consists of GDDR6-based PIM
chips and a lightweight ASIC chip to support end-to-end GPT acceleration.
Model mapping and workload distribution are optimized to maximize computation
parallelism and data locality. The proposed system achieves 41$-$137$\times$,
631$-$1074$\times$ speedup and 339$-$1085$\times$, 890$-$1632$\times$ energy
efficiency over GPU and CPU on 8 GPT models. We highlight that the design only
requires light modifications to the DRAM architecture, which offers a
practical solution for edge applications. PIM-GPT exhibits other favorable
characteristics, including low sensitivity to ASIC clock frequency and memory
interface specs, capable of long token generation and excellent scalability.

## Acknowledgements

This work was supported in part by the National Science Foundation through
Award CCF-1900675.

## References

  * [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [3] R. OpenAI, “Gpt-4 technical report,” _arXiv_ , pp. 2303–08 774, 2023.
  * [4] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune bert for text classification?” in _Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18_. Springer, 2019, pp. 194–206.
  * [5] W.-C. Chang, H.-F. Yu, K. Zhong, Y. Yang, and I. S. Dhillon, “Taming pretrained transformers for extreme multi-label text classification,” in _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_, 2020, pp. 3163–3171.
  * [6] S. Garg and G. Ramakrishnan, “Bae: Bert-based adversarial examples for text classification,” _arXiv preprint arXiv:2004.01970_ , 2020.
  * [7] Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning deep transformer models for machine translation,” _arXiv preprint arXiv:1906.01787_ , 2019.
  * [8] S. Yao and X. Wan, “Multimodal transformer for multimodal machine translation,” in _Proceedings of the 58th annual meeting of the association for computational linguistics_ , 2020, pp. 4346–4350.
  * [9] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [10] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [12] H. Wang, Z. Zhang, and S. Han, “Spatten: Efficient sparse attention architecture with cascade token and head pruning,” in _2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2021, pp. 97–110.
  * [13] S. Hong, S. Moon, J. Kim, S. Lee, M. Kim, D. Lee, and J.-Y. Kim, “Dfx: A low-latency multi-fpga appliance for accelerating transformer-based text generation,” in _2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 2022, pp. 616–630.
  * [14] M. Zhou, W. Xu, J. Kang, and T. Rosing, “Transpim: A memory-based acceleration via software-hardware co-design for transformer,” in _2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2022, pp. 1071–1085.
  * [15] T. J. Ham, S. J. Jung, S. Kim, Y. H. Oh, Y. Park, Y. Song, J.-H. Park, S. Lee, K. Park, J. W. Lee _et al._ , “A^ 3: Accelerating attention mechanisms in neural networks with approximation,” in _2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2020, pp. 328–341.
  * [16] T. J. Ham, Y. Lee, S. H. Seo, S. Kim, H. Choi, S. J. Jung, and J. W. Lee, “Elsa: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks,” in _2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 2021, pp. 692–705.
  * [17] H. Jang, J. Kim, J.-E. Jo, J. Lee, and J. Kim, “Mnnfast: A fast and scalable system architecture for memory-augmented neural networks,” in _Proceedings of the 46th International Symposium on Computer Architecture_ , 2019, pp. 250–263.
  * [18] A. H. Zadeh, I. Edo, O. M. Awad, and A. Moshovos, “Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference,” in _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 2020, pp. 811–824.
  * [19] H. Wang, H. Xu, Y. Wang, and Y. Han, “Cta: Hardware-software co-design for compressed token attention mechanism,” in _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2023, pp. 429–441.
  * [20] J. Dass, S. Wu, H. Shi, C. Li, Z. Ye, Z. Wang, and Y. Lin, “Vitality: Unifying low-rank and sparse approximation for vision transformer acceleration with a linear taylor attention,” in _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2023, pp. 415–428.
  * [21] H. You, Z. Sun, H. Shi, Z. Yu, Y. Zhao, Y. Zhang, C. Li, B. Li, and Y. Lin, “Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator co-design,” in _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2023, pp. 273–286.
  * [22] O. Mutlu, “Memory-centric computing,” _arXiv preprint arXiv:2305.20000_ , 2023\.
  * [23] S. Ghose, A. Boroumand, J. S. Kim, J. Gómez-Luna, and O. Mutlu, “Processing-in-memory: A workload-driven perspective,” _IBM Journal of Research and Development_ , vol. 63, no. 6, pp. 3–1, 2019.
  * [24] Y.-C. Kwon, S. H. Lee, J. Lee, S.-H. Kwon, J. M. Ryu, J.-P. Son, O. Seongil, H.-S. Yu, H. Lee, S. Y. Kim _et al._ , “25.4 a 20nm 6gb function-in-memory dram, based on hbm2 with a 1.2 tflops programmable computing unit using bank-level parallelism, for machine learning applications,” in _2021 IEEE International Solid-State Circuits Conference (ISSCC)_ , vol. 64. IEEE, 2021, pp. 350–352.
  * [25] S. Lee, S.-h. Kang, J. Lee, H. Kim, E. Lee, S. Seo, H. Yoon, S. Lee, K. Lim, H. Shin _et al._ , “Hardware architecture and software stack for pim based on commercial dram technology: Industrial product,” in _2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 2021, pp. 43–56.
  * [26] S. Lee, K. Kim, S. Oh, J. Park, G. Hong, D. Ka, K. Hwang, J. Park, K. Kang, J. Kim _et al._ , “A 1ynm 1.25 v 8gb, 16gb/s/pin gddr6-based accelerator-in-memory supporting 1tflops mac operation and various activation functions for deep-learning applications,” in _2022 IEEE International Solid-State Circuits Conference (ISSCC)_ , vol. 65. IEEE, 2022, pp. 1–3.
  * [27] D. Kwon, S. Lee, K. Kim, S. Oh, J. Park, G.-M. Hong, D. Ka, K. Hwang, J. Park, K. Kang _et al._ , “A 1ynm 1.25 v 8gb 16gb/s/pin gddr6-based accelerator-in-memory supporting 1tflops mac operation and various activation functions for deep learning application,” _IEEE Journal of Solid-State Circuits_ , vol. 58, no. 1, pp. 291–302, 2022.
  * [28] A. Yazdanbakhsh, C. Song, J. Sacks, P. Lotfi-Kamran, H. Esmaeilzadeh, and N. S. Kim, “In-dram near-data approximate acceleration for gpus,” in _Proceedings of the 27th International Conference on Parallel Architectures and Compilation Techniques_ , 2018, pp. 1–14.
  * [29] X. Xie, Z. Liang, P. Gu, A. Basak, L. Deng, L. Liang, X. Hu, and Y. Xie, “Spacea: Sparse matrix vector multiplication on processing-in-memory accelerator,” in _2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2021, pp. 570–583.
  * [30] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” _arXiv preprint arXiv:1606.08415_ , 2016.
  * [31] S. W. Keckler, W. J. Dally, B. Khailany, M. Garland, and D. Glasco, “Gpus and the future of parallel computing,” _IEEE micro_ , vol. 31, no. 5, pp. 7–17, 2011.
  * [32] H. Mao, M. Alser, M. Sadrosadati, C. Firtina, A. Baranwal, D. S. Cali, A. Manglik, N. A. Alserr, and O. Mutlu, “Genpip: In-memory acceleration of genome analysis via tight integration of basecalling and read mapping,” in _2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 2022, pp. 710–726.
  * [33] N. M. Ghiasi, J. Park, H. Mustafa, J. Kim, A. Olgun, A. Gollwitzer, D. S. Cali, C. Firtina, H. Mao, N. A. Alserr _et al._ , “Genstore: A high-performance and energy-efficient in-storage computing system for genome sequence analysis,” _arXiv preprint arXiv:2202.10400_ , 2022.
  * [34] L. Song, Y. Zhuo, X. Qian, H. Li, and Y. Chen, “Graphr: Accelerating graph processing using reram,” in _2018 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2018, pp. 531–543.
  * [35] J. Gómez-Luna, Y. Guo, S. Brocard, J. Legriel, R. Cimadomo, G. F. Oliveira, G. Singh, and O. Mutlu, “Evaluating machine learningworkloads on memory-centric computing systems,” in _2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_. IEEE, 2023, pp. 35–49.
  * [36] F. Devaux, “The true processing in memory accelerator,” in _2019 IEEE Hot Chips 31 Symposium (HCS)_. IEEE Computer Society, 2019, pp. 1–24.
  * [37] J. Gómez-Luna, I. El Hajj, I. Fernandez, C. Giannoula, G. F. Oliveira, and O. Mutlu, “Benchmarking a new paradigm: Experimental analysis and characterization of a real processing-in-memory system,” _IEEE Access_ , vol. 10, pp. 52 565–52 608, 2022.
  * [38] J. H. Kim, S.-h. Kang, S. Lee, H. Kim, W. Song, Y. Ro, S. Lee, D. Wang, H. Shin, B. Phuah _et al._ , “Aquabolt-xl: Samsung hbm2-pim with in-memory processing for ml accelerators and beyond,” in _2021 IEEE Hot Chips 33 Symposium (HCS)_. IEEE, 2021, pp. 1–26.
  * [39] S. Kang, S. Lee, B. Kim, H. Kim, K. Sohn, N. S. Kim, and E. Lee, “An fpga-based rnn-t inference accelerator with pim-hbm,” in _Proceedings of the 2022 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_ , 2022, pp. 146–152.
  * [40] D. Niu, S. Li, Y. Wang, W. Han, Z. Zhang, Y. Guan, T. Guan, F. Sun, F. Xue, L. Duan _et al._ , “184qps/w 64mb/mm 2 3d logic-to-dram hybrid bonding with process-near-memory engine for recommendation system,” in _2022 IEEE International Solid-State Circuits Conference (ISSCC)_ , vol. 65. IEEE, 2022, pp. 1–3.
  * [41] M. Orenes-Vera, E. Tureci, D. Wentzlaff, and M. Martonosi, “Dalorex: A data-local program execution and architecture for memory-bound applications,” in _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2023, pp. 718–730.
  * [42] S. Li, D. Niu, K. T. Malladi, H. Zheng, B. Brennan, and Y. Xie, “Drisa: A dram-based reconfigurable in-situ accelerator,” in _Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture_ , 2017, pp. 288–301.
  * [43] F. Gao, G. Tziantzioulis, and D. Wentzlaff, “Computedram: In-memory compute using off-the-shelf drams,” in _Proceedings of the 52nd annual IEEE/ACM international symposium on microarchitecture_ , 2019, pp. 100–113.
  * [44] M. Jeong and W. Jung, “Mac-do: Charge based multi-bit analog in-memory accelerator compatible with dram using output stationary mapping,” _arXiv preprint arXiv:2207.07862_ , 2022.
  * [45] JEDEC. Graphics double data rate (gddr6) sgram standard. [Online]. Available: https://www.jedec.org/standards-documents/docs/jesd250d
  * [46] Z. Fan, Q. Zhang, P. Abillama, S. Shoouri, C. Lee, D. Blaauw, H.-S. Kim, and D. Sylvester, “Taskfusion: An efficient transfer learning architecture with dual delta sparsity for multi-task natural language processing,” in _Proceedings of the 50th Annual International Symposium on Computer Architecture_ , 2023, pp. 1–14.
  * [47] M. He, C. Song, I. Kim, C. Jeong, S. Kim, I. Park, M. Thottethodi, and T. Vijaykumar, “Newton: A dram-maker’s accelerator-in-memory (aim) architecture for machine learning,” in _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 2020, pp. 372–385.
  * [48] Fast inverse square root. [Online]. Available: https://en.wikipedia.org/wiki/Fast_inverse_square_root
  * [49] Y. Kim, W. Yang, and O. Mutlu, “Ramulator: A fast and extensible dram simulator,” _IEEE Computer architecture letters_ , vol. 15, no. 1, pp. 45–49, 2015.
  * [50] Micron. 16gb ddr5 sdram addendum. [Online]. Available: https://media-www.micron.com/-/media/client/global/documents/products/data-sheet/dram/ddr5/16gb_ddr5_sdram_diereva.pdf
  * [51] S. Ghose, A. G. Yaglikçi, R. Gupta, D. Lee, K. Kudrolli, W. X. Liu, H. Hassan, K. K. Chang, N. Chatterjee, A. Agrawal _et al._ , “What your dram power models are not telling you: Lessons from a detailed experimental study,” _Proceedings of the ACM on Measurement and Analysis of Computing Systems_ , vol. 2, no. 3, pp. 1–41, 2018.
  * [52] Y. Kwon, K. Vladimir, N. Kim, W. Shin, J. Won, M. Lee, H. Joo, H. Choi, G. Kim, B. An _et al._ , “System architecture and software stack for gddr6-aim,” in _2022 IEEE Hot Chips 34 Symposium (HCS)_. IEEE, 2022, pp. 1–25.
  * [53] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” in _International conference on machine learning_. PMLR, 2020, pp. 5156–5165.
  * [54] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” _arXiv preprint arXiv:2004.05150_ , 2020.
  * [55] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient transformers: A survey. arxiv 2020,” _arXiv preprint arXiv:2009.06732_.
  * [56] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser _et al._ , “Rethinking attention with performers,” _arXiv preprint arXiv:2009.14794_ , 2020.
  * [57] R.-J. Zhu, Q. Zhao, and J. K. Eshraghian, “Spikegpt: Generative pre-trained language model with spiking neural networks,” _arXiv preprint arXiv:2302.13939_ , 2023.