# Data-driven Forecasting of
Deep Learning Performance on GPUs

Seonho Lee<EMAIL_ADDRESS>Georgia Institute of Technology ,  Amar
Phanishayee<EMAIL_ADDRESS>Meta and  Divya Mahajan
<EMAIL_ADDRESS>Georgia Institute of Technology

###### Abstract.

Deep learning kernels exhibit predictable memory accesses and compute
patterns, making GPUs’ parallel architecture well-suited for their execution.
Software and runtime systems for GPUs are optimized to better utilize the
stream multiprocessors, on-chip cache, and off-chip high-bandwidth memory. As
deep learning models and GPUs evolve, access to newer GPUs is often limited,
raising questions about _the performance of new model architectures on
existing GPUs, existing models on new GPUs, and new model architectures on new
GPUs._ To address these questions, we introduce NeuSight, a framework to
predict the performance of various deep learning models, for both training and
inference, on unseen GPUs without requiring actual execution. The framework
leverages both GPU hardware behavior and software library optimizations to
estimate end-to-end performance. Previous work uses regression models (Isaev
et al., 2023; Li et al., 2023) that capture linear trends or multilayer
perceptrons (Yu et al., 2021) to predict the overall latency of deep learning
kernels on GPUs. These approaches suffer from higher error percentages when
forecasting performance on unseen models and new GPUs. Instead, NeuSight
decomposes the prediction problem into smaller problems, bounding the
prediction through fundamental performance laws. NeuSight decomposes a single
deep learning kernel prediction into smaller working sets called tiles, which
are executed independently on the GPU. Tile-granularity predictions are
determined using a machine learning approach and aggregated to estimate end-
to-end latency. NeuSight outperforms prior work across various deep learning
workloads and the latest GPUs. It reduces the percentage error from 198% and
19.7% to 3.8% in predicting the latency of GPT3 model for training and
inference on H100, compared to state-of-the-art prior works (Yu et al., 2021;
Li et al., 2023), where both GPT3 and H100 were not used to train the
framework.

##  1\. Introduction

GPUs are the canonical platform to be deployed across any Artificial
Intelligence (AI) cluster. There are two main reasons for their widespread
adoption for deep learning: (1) the parallel compute engines that mitigate the
Von-Neumann overheads of general purpose CPUs and (2) closely knit software
libraries that exploit the GPU architecture for various Deep Neural Network
(DNN) kernels (NVIDIA, ; NVIDIA, 2017). This trend has been further augmented
by the availability of specialized AI servers, aiming to minimize the
overheads of distributed execution (Nvidia, ; AMD, ).

Figure 1. Growth of AI models and the compute and memory capacity of GPUs.
(NVIDIA, ; AMD, ; Krizhevsky et al., 2012; Simonyan and Zisserman, 2015;
Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2023; Brown et al.,
2020; Smith et al., 2022)

In this rapidly evolving field of AI, constant iterations are continuously
introduced over the years. However, these newer GPUs are limited in access -
either due to the long lead time (of upto 52 weeks for H100 (Forbes, 2024)) or
their prohibitively high cost. Currently, devising new models and the success
of model execution is contingent on the availability of underlying hardware.
As shown in Figure 1, recent times (since 2013) have witnessed exponential
growth in model size, accompanied by a corresponding increase in the peak
performance of GPUs. In summary, the prevailing shift towards the widespread
deployment of GPU-centric workloads raises a significant challenge: how can we
closely estimate the performance of deep learning models on inaccessible GPUs
and upcoming models?

The primary challenge in developing a solution capable of closely estimating
GPU performance using only the abstract hardware architecture features lies in
the fact that the performance of deep learning models on GPUs is intricately
tied to low-level hardware details and software optimizations, such as those
provided by DNN kernel libraries like CuDNN and Cutlass. Unlike hardware
accelerators (Jouppi et al., 2023; Sharma et al., 2016; Mahajan et al., 2016,
2018; Fowers et al., 2018; Chen et al., 2019), GPUs exhibit complex interplay
between micro-architectural interactions and execution characteristics, due L1
and L2 caches, DRAM, memory, I/O configurations and warp scheduling.
Consequently, while there are various established works (Olyaiy et al., 2023;
Kwon et al., 2019) that can closely estimate the performance of DNN kernels on
domain-specific accelerators, they are not suitable for GPU performance
forecasting. Nevertheless, previous works have aimed to leverage cycle-
accurate simulators to provide detailed execution and performance estimates
for GPUs (Bakhoda et al., 2009; Khairy et al., 2020; Avalos Baddouh et al.,
2021). However, these simulators need to be manually updated for each change
in the GPU architecture. As such, developing a cycle accurate simulator is a
time-consuming process and using them to simulate the simplest of the model
can take a long time. The most popular GPU simulator (Khairy et al., 2020;
Avalos Baddouh et al., 2021) can take up to 18 hours to simulate ResNet-50
with a batch size of 256 (Li et al., 2023). ResNet-50 is relatively smaller
compared to new-age complex networks such as the LLMs. Thus, it will take a
prohibitively long time to use this simulator to predict the performance of
future deep learning models.

In this work, we devise a novel prediction mechanism built on top of a machine
learning model, designed with consideration of the tile-based execution
strategy on Stream Multiprocessors (SMs) in GPUs, and the compute and memory
bounds of DNN kernels. Our approach involves utilizing a multi-layer
perceptron to encompass both the hardware features of the GPU and the
execution patterns of deep learning. We train our model using data collected
from a set of readily available older generation GPUs and existing deep
learning kernel information, which is then used to predict performance of
newer models, both training and inference, on unseen GPUs.

Prior works in the domain of modeling GPU performance take an analytical
model, a linear regression-based model, or a neural network approach (Yu et
al., 2021; Li et al., 2023; Isaev et al., 2023). We empirically observe that
these solutions exhibit high percentage errors for newer models and GPUs, for
both inference and training. They are limited for the proposed use case
because they either: (1) use a rough analytical model without information
about detailed GPU execution (hardware or software) (2) require execution on a
specific GPU to extrapolate latency for newer GPUs (Li et al., 2023; Isaev et
al., 2023), or (3) are unable to capture the necessary architectural
information of the GPUs (Yu et al., 2021). Additionally, these prior works
exhibit high percentage error for newer models and unseen GPUs because they
aim to directly model the overall latency of a kernel through machine
learning. However, GPU execution, even of a single kernel, is a complex
conflation of hardware execution across SMs and software optimizations, which
often cannot be captured by a machine learning model trying to determine the
latency of the kernel only using certain high-level features such as effective
memory bandwidth, compute utilization, and peak flop.

Thus, we develop NeuSight which digresses from prior works as it leverages the
insight that popular GPU libraries decompose deep learning kernels into
multiple smaller working sets. These small working sets, or tiles, are then
dispatched and executed on the GPU independently across the SMs. _Based on
this observation we partition the end-to-end latency prediction into regular
and more manageable sub-problems, predicting the latency at tile-granularity._
However, note that the GPU execution is governed by a few performance laws -
such as the performance cannot exceed the peak flops or peak memory bandwidth.
_We capture the above two aspects in our prediction and only use a machine
learning model to determine the utilization of the device based on the DNN
kernel and GPU properties._ The output of the MLP, i.e., utilization of the
device, is subsequently used to predict the tile, operator, and the model
level latency. Overall, the goal of this entire framework is to distill down
to the properties of GPU execution, use machine learning to capture the
runtime non-linear behavior, bound the results by fundamental performance
laws, and gather the predictions for end-to-end latency predictions. This
approach ensures robustness in unseen scenarios, such as new GPUs and models.

To ensure comprehensive prediction for large models that require multiple
GPUs, we extend our approach to predict the latency of distributed execution
within a server. To do so, we predict the latency of communication operators
and integrate these with the per-device execution latencies determined by
NeuSight to calculate the overall distributed performance.

The evaluation on a diverse set of GPUs (Nvidia H100, A100-80GB, V100,
A100-40GB, P100, T4, L4) and deep learning workloads (BERT, GPT2, GPT3, OPT,
Switch Transformer) shows that NeuSight demonstrates a percentage error of
8.9%, compared to 140% by MLP-based (Yu et al., 2021) and 60.8% by linear
regression-based (Li et al., 2023) prior work, without being trained on H100,
L4, and A100-80GB GPUs and trained only on a subset of GPT3 and OPT model
dimensions. When evaluated for distributed training on a 4-GPU server, H100
(DGX Box) and A100 (NVLink), NeuSight shows a modest percentage error of 5.4%
across GPT2 and GPT3 model. The error presented throughout this paper is the
mean absolute percentage error, calculated against actual measured latencies.
For brevity, we henceforth refer to it as percentage error.

##  2\. Background

In this section, we dive into the background of GPU architecture and how DNNs
are executed on it.

###  2.1. GPU Architecture

The availability of parallelism in deep learning workloads and their
suitability to GPU-architectures has led to co-optimization across various
generations of workloads and GPUs. While the focus in this paper is on NVIDIA
GPUs and their terminology, it is important to note most GPUs have similar
architectures(AMD, 2024). A GPU is comprised of a collection of identical
building blocks known as Streaming Multiprocessors (SMs). These SMs are
interconnected with L2 cache through an on-chip interconnection network. The
L2 cache, in turn, is connected to memory through off-chip I/O. Each SM
features multiple processing units and a small L1 cache. The regularity of the
GPU architecture makes it scalable to tens of thousands of processing units,
allowing it to effectively exploit parallelism in deep learning kernels.

###  2.2. Deep Learning Execution

Kernel execution. Modern deep learning workloads consist of multiple layers,
where each layer itself has multiple fine-grained kernels such as matrix
multiplication or element-wise operators. In the context of this paper, DNN
kernel is a tensor operator such as CONV, GEMM, ADD, SOFTMAX, that is executed
entirely on the device once initiated. Although the entire workload comprises
thousands of kernels, due to the repetitive structure of the model, they can
be decomposed into a few types of operators. These types are General Matrix
Multiplication (GEMM), fully-connected layer, and point wise operators (ReLU,
Tanh, Softmax, etc.). Hence, this work focuses on determining the execution of
each kernel type on GPU to predict the latency as detailed in Section 4.1.

Per-device execution. Machine learning practitioners use frameworks such as
PyTorch and TensorFlow (PyTorch, a; Google, ) to represent and deploy a model
as a set of layers comprising their corresponding kernels. The dataflow of the
kernels is managed by the framework and dispatched to the GPU. The
corresponding device library then executes each kernel and returns the results
to the framework. For example, NVIDIA GPUs utilize the CuDNN library for
kernel execution, abstracting the underlying hardware complexity from the
user. In this paper, similar to prior works, we assume kernels execute
sequentially on the device, which is also empirically validated for the
mainstream machine learning frameworks (Zhu et al., 2020). Thus, the predicted
latency of each kernel is aggregated based on the dataflow of the model that
executes on each device to determine the per-GPU execution latency.

Distributed per-server execution. Due to the size of the model, training or
inference is often executed in a distributed fashion. Pipeline parallel and
tensor model parallel are used to split a model across various devices,
whereas data parallel is used to improve throughput (Narayanan et al., 2019;
Huang et al., 2019; Nvidia, ; Krizhevsky et al., 2017). Regardless,
distributed execution warrants communication across devices such as
send/receive for pipeline parallel and collective operations like all-reduce
for tensor model parallel and data parallel. In this work, NeuSight supports
prediction of latency when a model is distributed across GPUs within a single
server. Within this server, GPUs can be connected through NVLink or like a DGX
box to support the aforementioned parallelization techniques (NVIDIA, a;
Nvidia, ). The type of parallelism, its width, and the schedule is provided by
the user as an input to NeuSight.

(a)  Multi-Layer Perceptron Approach (Habitat) (Yu et al., 2021)

(b)  Linear Regression-based Approach (Li et al.) (Li et al., 2023)

Figure 2. Prediction error of prior works on BMM operator, reported in
percentage error. Out of distribution dimensions and GPUs are highlighted. For
these results, we trained Habitat and Li et al. models only up to V100,
excluding any A100s, H100 and L4.

##  3\. Motivation

There is a growing need to predict GPU performance for both existing and
upcoming models for a variety of reasons: (a) comparing performance across
various devices, (b) utilizing estimates to identify GPUs that meet the
performance requirements, and (c) evaluating the feasibility of deploying new
model architectures on either existing or new GPUs. There are certain prior
works that aim to predict the performance of GPUs by employing data-driven
methods, and we empirically validate these works and observe that do not
generalize well to new GPUs or new models. These drive the insights for our
framework, NeuSight.

For motivation, we focus on a single kernel Batched Matrix Multiplication
(BMM), commonly found in DNNs, and present end-to-end comparisons (training
and inference) in Section 6. We observe that even predicting performance of a
single kernel on a GPU requires a careful consideration of hardware
architecture and software optimization. This insight is based on empirical
validation of prior works that employ machine learning to determine the
operator latency.

###  3.1. Predicting Performance of Batched Matrix Multiplication Using Prior
Works

We cover two most closely related prior works, Habitat (Yu et al., 2021) and a
linear regression based approach (Li et al., 2023) to estimate the latency of
GPUs for deep learning.

Habitat classifies operators into two types: kernel-alike which use similar
kernels across different GPU architectures, and kernel-varying which use
different kernels across different GPU architectures. For kernel-alike
operators (such as vector operators), it executes applications on an existing
GPU to measure latency and scales the latency with the ratio of target GPU
hardware configuration to that of existing GPU. As such, Habitat requires a
GPU in-hand to estimate the latency of these operators. For kernel-varying
operators (such as BMM), they train multiple Multi-Layer Perceptron (MLP)
models to predict the latency of different operators. Their MLP trains on the
following GPU features: memory size, memory bandwidth, number of SM cores,
Peak FLOPS, and kernel input and output dimensions as input, and the measured
latency as the output.

Li et al.(Li et al., 2023) proposes a linear regression model to predict
performance based on FLOPs count of target kernel. For each GPU, they collect
multiple data points to construct a linear regression between latency and the
number of FLOPs (determined based on the size of input/output matrices). They
observe that achieved performance across different GPUs is usually linearly
proportional to their memory bandwidth. Thus, they construct a linear
regression between the memory bandwidth of GPUs and achieved FLOPS
performance, using this relationship to infer the performance of GPUs outside
the training set.

Figure 2 reports the prediction errors for these two works. For Habitat, we
utilized the vanilla MLP model provided by the authors. This model was trained
on diverse GPUs predating 2018, including V100, 2080Ti, T4, RTX2070, P100, and
P4000, with matrix dimensions of up to 1024 and a batch size less than 128. We
aim to understand the applicability of this work on newer GPUs and larger
dimensions common in newer models. As Figure 2a shows, Habitat faces
challenges in generalizing to matrix multiplications with dimensions larger
than 1024, resulting in a percentage error of up to 38%. Additionally, Habitat
struggles with unseen GPUs, displaying a percentage error of up to 127% on
A100.

For Li et al., we gather BMM data on P4, T4, P100, and V100. We apply linear
regression to determine the correlation between the FLOP count of the kernel
and latency following the procedures outlined in the paper. For GPUs outside
the training set (A100, L4, and H100), we use this linear regression to
predict the latency. As Figure 2b shows, the linear regression assumption
fails on matrix multiplications with smaller dimensions due to under-
utilization of the GPU, when the latency does not scale linearly. This work
also faces challenges in generalizing to unseen GPUs because of its simplistic
extrapolation method, i.e., using a linear ratio of memory bandwidth to infer
the effective performance of unseen GPUs.

###  3.2. Predicting Performance of Batched Matrix Multiplication with Larger
Predictors

One potential solution to reduce the prediction error could be to use larger
models for the same problem, i.e., predicting the latency of an operator using
a machine learning model, with more layers and features. Thus, as a first
attempt, we leverage a larger and more complex model with an increased number
of layers to capture a broader set of execution patterns. To test the
limitations of relying purely on large machine learning models, we extend
Habitat’s prediction model to include more layers in the MLP or alternatively
use a transformer architecture from prior work, Prime (Kumar et al., 2022). We
train these models using the same method specified in Section 6.1 for the
Habitat model. BMM operations with dimensions between 1 and 4096 are used as
the test set. Table 1 summarizes the prediction error averaged across in-
distribution BMM operations (matrix dimensions less than 1024) and out-of-
distribution BMM operations (some dimensions larger than 1024). Despite
improved performance with larger models, all the evaluated models still show
high percentage error exceeding 70% for out-of-distribution cases.

It is evident from the results that naively using machine learning techniques
to predict kernel latency does not suffice, as they fail to capture the
continuous performance improvements of GPUs resulting from both software and
hardware optimizations. This drives the key insight for NeuSight: unlike prior
work, it does not directly predict the latency of kernels using machine
learning; instead, it distills the kernel execution into tiles, uses a machine
learning model to predict the utilization of the device per tile, and then
determines the latency based on performance bounds defined by GPU
architecture. NeuSight only relies on architectural details readily available
for any new or incoming GPU.

Table 1. Various ML models on predicting latency of BMM. Prediction errors are reported in percentage error. Predictor | Number | In-distribution | Out-of-distribution
---|---|---|---
Architecture | of layers |  Prediction Error (%) |  Prediction Error (%)
MLP | 8 | 28.0 | 70.9
| 16 | 22.3 | 81.4
Transformer | 3 | 22.3 | 126.1
| 6 | 21.0 | 86.4

###  3.3. Other Related Works

Performance modeling using analytical frameworks. Several works (Hong, 2009;
Qi et al., 2017; Isaev et al., 2023) propose analytical models to predict GPU
performance. Paleo (Qi et al., 2017) and Calculon (Isaev et al., 2023)
decompose latency into computation and communication for distributed training.
They estimate latency by measuring effective peak throughput with
representative workloads on the target GPU, which requires access to the
target GPU for estimation.

Cycle accurate simulators. Cycle-accurate simulators (Bakhoda et al., 2009)
require detailed modeling of each GPU component. For every new GPU, the
simulator must be updated to support new architectural features, which might
not be readily available at launch. Proprietary architectural features can
create a gap between real hardware and simulators. Additionally, cycle-
accurate simulators are not suitable for end-to-end ML application execution
due to their long simulation times. While certain works, such as sampled
principal kernels (Avalos Baddouh et al., 2021), reduce simulation time, they
still take at least hours to days to simulate modern DNN due to the need for
modeling every detail of the hardware.

Machine Learning based approaches. Due to the large number of compute lanes in
GPUs, they often observe linear performance trends for compute intensive
kernels, and hence many prior works leverage regression models to predict the
performance of a GPU architecture (Jia et al., 2012; Wu et al., 2015; Li et
al., 2023). Some works require extracting low level hardware counter values to
make predictions, however, these are unavailable unless the device is
accessible (O’Neal et al., 2017). Such approaches cannot capture the complex
intricacies of hardware features and software optimizations such as tiling for
deep learning and do not support cross-generational performance prediction
with GPUs due to the lack of hardware. Other works (Yang et al., 2022) propose
graph neural networks to predict resource consumption (GPU utilization, memory
utilization, and network TX/RX throughput) in distributed deep learning.
However, this approach necessitates target GPUs for transfer training.

CPU executions to predict speed-up on GPU. XAPP (Ardalani et al., 2015)
executes the CPU code of a target algorithm to predict GPU speed-up. However,
this approach assumes the GPU will execute the same algorithm but with higher
parallelizability. In contrast, modern software optimizations for deep
learning kernels are designed specifically for GPUs (Dao et al., 2022; Chen et
al., 2018), diverging from CPU implementations.

(a)  Tiling of output matrix.

(b) Mapping tiles to multiple SMs.

Figure 3. Dataflow of a GEMM on a GPU. We assume multiplication between two
4x4 matrices and tile size of 2x2.

##  4\. NeuSight Forecasting for DNN Kernels

NeuSight forecasts the latency of executing a model on multiple GPUs in a
server in three steps. The first and main step forecasts the latency of a
single kernel that executes on the GPU atomically. The kernel leverages
software and hardware optimizations for GPUs, namely tiling, parallel
execution across SMs, on-chip caching, and HBM bandwidth. NeuSight uses
architectural details such as memory size, memory bandwidth, number of SMs, L2
cache size, and peak FLOPS to predict the kernel’s latency through a machine
learning approach. Our kernel estimation is inspired by how deep learning
kernels are disassembled into multiple smaller tiles and mapped onto GPU
hardware for execution. This approach contrasts with prior works that rely
solely on compute or memory utilization to predict latency. Section 4.1
showcases the execution of a few representative DNN kernels on the GPU.
NeuSight supports tensor operators such as General Matrix Multiplication
(GEMM), element-wise operators such as addition and ReLU, and reduction-based
operators such as Softmax and layer normalization.

In the second step, the per-operator latency is aggregated to predict the
performance of the DNN workload based on its dataflow graph. In the third
step, detailed in Section 5.1, the performance of per-GPU execution is
combined with predicted network latencies to determine the performance of a
DNN executing across multiple GPUs in a server.

###  4.1. Kernel Execution on GPUs

Figure 4. Each tile is distributed evenly across SMs and executed
concurrently, in multiple number of waves. (NVIDIA, b)

Tiled execution. In this section, we focus on GEMM, a commonly found kernel in
DNNs, but NeuSight can predict the latency of various operators to determine
end-to-end performance. GEMM is a core building-block in deep learning layers,
such as fully-connected, convolution, recurrent neural network, or self-
attention found in modern large language models. Figure 3 shows a simplified
view of a typical dataflow of GEMM on a GPU. Modern GPU libraries execute GEMM
by first partitioning the output matrix into multiple identical tiles (Figure
3a). Each tile corresponds to a segment of the output matrix, loads the
necessary input operands from input matrices, and computes the output
elements. These tiles are mapped to each SM of the GPU and executed
concurrently (Figure 3b). Typically the number of tiles which can be executed
concurrently is limited by the number of SMs on the GPU, and the entire GPU
kernel is executed in multiple waves of tile groups as depicted in Figure 4.
Tiling strategy enables scalable execution of GEMM, by decomposing GEMM into
multiple smaller workloads, which executes the identical computation on
different input and output elements.

Fundamental performance laws of GPUs. Runtime of each kernel is bounded by a
few features of a GPU, such as peak memory bandwidth or peak FLOPs. Roofline
anlaysis (Williams et al., 2009) provides a straightforward method to estimate
the approximate performance of a GPU kernel, taking into account the
arithmetic intensity of the kernel. The intensity is computed by dividing
number of flops ($flops_{k}$) by size of memory transactions ($mem_{k}$). The
roofline bandwidth can be computed as follows:

(1) $\footnotesize K=\frac{flops_{k}}{mem_{k}};~{}~{}roofline_{BW}=min(K\times
mem_{p},flops_{p})$

Where $flops_{p}$ is the peak FLOPs and $mem_{p}$ is the peak memory bandwidth
of GPU. Roofline bandwidth represents the maximum achievable throughput of a
kernel on a GPU.

Figure 5. Performance of (256 × 256) × (256 × 256) matrix multiplication with
varying waves on V100; sweeping the number of waves by changing the batch size
from 1 to 300.

Latency hiding in GPU. Another key performance bound in GPU execution is the
use of numerous threads to harness parallelism, achieving significantly higher
throughput compared to traditional CPU architectures. GPUs can achieve the
aforementioned peak hardware performance ($roofline_{BW}$) only when there are
enough available software threads to effectively mitigate latency arising from
data dependency or memory stalls by scheduling independent threads.
Consequently, GPU performance relies heavily on the level of parallelism in
the kernels, which is directly proportional to the number of threads that can
be independently scheduled in each SM. Figure 5 illustrates the correlation
between the number of waves available in each SM and the achievable
throughput. As the number of waves increases, each SM is assigned more waves,
resulting in a greater number of threads per SM. This allows for effective
hiding of stall latency in each thread, thus leading to higher performance.

###  4.2. Kernel-wise Prediction

In predicting the latency of each kernel, we leverage the execution insights
discussed above: tiled execution and fundamental GPU performance laws.

Tile-granularity prediction. For each DNN kernel we break it down into
multiple identical tiles, instead of directly predicting the latency of the
kernel. This insight is motivated from prior works (Yu et al., 2021), where a
machine learning model is unable to capture the necessary hardware level and
software optimizations on out of distribution dimensions and GPUs. This
implies that there are more intricate execution patterns although consistent
across the GPUs, but are not easily captured by machine learning models. Thus,
we use the machine learning model to make a prediction per tile, and assemble
these predictions to determine the latency of the whole operator. This
decomposition simplifies the problem into more manageable sub-problems,
allowing the machine learning model to be trained on smaller and more
straightforward tasks. For example, the typical tile dimensions used by GEMM
library ranges from 32 to 256, which is still much smaller than the dimensions
of GEMMs in modern DNN workloads. To compute the latency of the entire kernel
from the predictions for each tile, we employ the following equations.

(2) $num_{tiles}={\Pi^{N}_{i=1}}\lceil\frac{x_{i}}{t_{i}}\rceil$ (3)
$num_{waves}=\lceil\frac{num_{tiles}}{num_{sm}}\rceil$ (4)
$PerOpLatency=PerTileLatency\times num_{waves}$

Variables $x_{i}$ and $t_{i}$ are the dimensions of the output and the tile,
respectively, $N$ is the number of dimensions of the output, and $num_{sm}$ is
the number of SMs available in GPU. The tile dimensions are determined by
metadata obtained with PyTorch Profiler discussed in Section 6.1. In the next
subsection we discuss how we can obtain the $PerTileLatency$.

Imposing performance laws to tile prediction. $PerTileLatency$ relies on the
compute utilization of the SM and the achieved bandwidth of the kernel
($achieved_{BW}$), and is physically bounded by $roofline_{bw}$. This
relationship is captured by the following equations:

(5) $PerTileLatency=\frac{flops_{tile}}{achieved_{BW}}$ (6)
$achieved_{BW}=roofline_{BW}\times utilization$

Next we need to predict GPU’s utilization. We use the observation that a GPU
approaches the maximum achievable throughput as the number of available
threads in the workload increases (as shown in Figure 5). We formulate the
correlation between number of waves and utilization as a machine learning
model. This is because, an ML model can capture the non-linear behaviour
between kernel latency and GPU characteristics and kernel properties such as
operator type, tile dimensions, and arithmetic intensity. We use MLPs as our
machine learning model, which is tasked with predicting the coefficients of
the following equation:

(7) $utilization=alpha-\frac{beta}{num_{waves}}$ (8)
$alpha,beta=\sigma(MLP(input\\_features))$

This equation models utilization, which increases as the number of waves
increases due to the negative beta factor but is capped by alpha. We apply a
sigmoid function to the outputs of MLPs to bound the predicted utilization
value to below 1. This approach constrains the predictions to adhere to
fundamental performance laws, enhancing its overall predictive capabilities
even when faced with unseen inputs.

###  4.3. Machine Learning Model to Predict Utilization

We use an MLP with multiple fully-connected layers for our machine learning
predictor. We train five distinct MLPs, each tailored for batched matrix
multiplication, fully-connected layers, element-wise operators, softmax, and
layer normalization.

Model architecture. Each MLP comprises 8 hidden layers, each with 512 hidden
units, similar to prior work (Yu et al., 2021). First-layer converts the input
feature vectors into 512-dimensional hidden vector, and the last layer
converts the 512-dimensional hidden vector to single-dimensional output. ReLU
is used as the activation function and applied at the end of every layer. This
simple and small model architecture allows for fast but still accurate
predictions, especially when combined with the GPU execution flow and
performance scaling laws in the NeuSight framework.

We use memory size and bandwidth, peak FLOPS, and L2 cache size as GPU
features for this modeling. We use these features because: (1) they are
publically available even for newer GPUs, (2) bandwidth and peak FLOPs
determine the performance bounds of I/O bound and compute bound operators, and
(3) on-chip memory size (L2 cache) and off-chip memory size (such as High
Bandwidth Memory or DRAM) determine which underlying implementation will be
used by the libraries such as cuDNN and CUTLASS to execute the DNN operators.
For instance, information for the H100 was made available through NVIDIA’s
public documentation after its launch (NVIDIA, 2023a). For the Blackwell
architecture, the latest NVIDIA GPU architecture announced (NVIDIA, 2024),
details on memory size, bandwidth, and peak FLOPs are already available.
Information on the number of SMs and L2 cache size is not availalbe yet, but
should become available closer to the release date or shortly after the GPU is
released.

Instead of directly feeding these input features to the MLP, we pre-process
them through the following steps. First, as we are making predictions on a
tile granularity, with each tile mapped to a single SM, we divide the device
resources by the number of SMs to determine per-SM resources (e.g., per-SM
peak FLOPS). Second, since the MLP predicts achievable throughput, we feed it
with the utilization of each hardware resource. The five features shown in
Table 2 are fed to the MLP as an input.

Table 2. List of input features for predicting the utilization.

Input Features | Unit
---|---
$\frac{FLOPsPerTile}{PeakFLOPSPerSM}$ | $\frac{GFLOPS}{TFLOPS/s}$
$\frac{MemoryPerTile}{MemoryBWPerSM}$ | $\frac{MB}{GB/s}$
$\frac{num_{waves}~{}\times~{}MemoryPerTile}{L2CacheSizePerSM}$ | $\frac{MB}{KB}$
$\frac{num_{waves}~{}\times~{}MemoryPerTile}{MemorySizePerSM}$ | $\frac{MB}{MB}$
$\frac{FLOPsPerTile~{}/~{}MemoryPerTile}{PeakFLOPS~{}/~{}MemoryBW}$ | $\frac{GFLOPS~{}/~{}MB}{(TFLOPS/s)~{}/~{}(GB/s)}$

Figure 6. Overall workflow of NeuSight. The Tile Latency Predictor uses an MLP
to determine the utilization and predicts the kernel latency. This prediction
is aggregated for per-device and distributed execution latency forecasting.

Predictors for various types of kernels. NeuSight includes multiple
specialized predictors to support various types of DNN kernels. There are five
MLPs to predict the utilization for BMM, fully-connected layers, element-wise
operators, softmax, and layer normalization. For unseen operators, we consider
them memory-bound and estimate latency by dividing the memory requirement by
memory bandwidth. These diverse predictors enable NeuSight to handle various
model architectures.

###  4.4. Support for Operator Fusion

Modern ML frameworks (Chen et al., 2018; PyTorch, a) accelerate the execution
of memory-bound kernels by fusing multiple consecutive operators and
eliminating the data movement from external memory. NeuSight supports the
prediction of the fused operators. When multiple consecutive vector kernels
are fused, NeuSight accumulates the FLOPs count of each operator, but discards
the memory size required for the intermediate results between kernels.
NeuSight uses the updated FLOPs count, memory size of the fused operator, and
the metadata including tile size of the first operator to be fused to make
predictions with its MLP. For instance, in the GPT-2 architecture, the
element-wise addition used for residual connections in each layer can be fused
with subsequent layer normalization. NeuSight can predict the latency of this
fused operator accumulating the FLOPs count of both operators and discarding
the memory size required for the output of the element-wise addition and the
input of the layer normalization, and using the predictor for element-wise
addition. Another common form of operator fusion is combining the GEMM
operator with activation functions. NeuSight supports this fusion using the
BMM or fully-connected layer predictor and adjusting the input compute count
and memory size.

##  5\. NeuSight Workflow

NeuSight forecasts the end-to-end latency of a deep learning model executing
on a single GPU or a multi-GPU server in three steps: (1) forecasting the
performance of per-kernel execution on the GPU as detailed above, (2)
combining these kernel-level estimates based on the dataflow graph of the DNN
to determine the per-GPU latency, and (3) estimating collectives and network
operations and integrating them with the per-device execution latency to
determine the performance on a GPU server executing distributed training or
inference. Both training and inference graphs are supported.

Figure 6 shows the overall workflow of NeuSight. NeuSight takes the
description of the target deep learning model in PyTorch and extracts the
operator/kernel graph using the Torch.fx library (PyTorch, 2023a). It records
the metadata of each kernel, including operator type and input/output tensor
dimensions. Based on the metadata and tile size, each operator is annotated
with predictions made by our kernel-level predictor. The per-GPU latency is
computed by accumulating the predicted latency of each kernel executing on a
single device, which aligns with device execution where kernels are executed
sequentially on the GPU (Zhu et al., 2020).

NeuSight also takes as input the distribution strategy, data parallel width,
pipeline parallel depth and schedule, or tensor model parallel width. _It is
important to highlight that the focus of NeuSight is to effectively predict
the performance of DNNs on new or unseen GPUs._ Predicting the performance of
each kernel is the complex and unsolved aspect tackled by NeuSight, as it
involves addressing the non-linearities of GPU software optimizations and
hardware execution.

For completeness, we provide a solution that can forecast the latency of
distributed execution on a multi-GPU server. The goal of this forecasting is
to enable researchers to determine latency of a relatively larger model that
often cannot fit on a single device due to memory constraints. This allows
NeuSight to provide accurate and reliable predictions for the performance of
the compute/memory intensity of the DNN while avoiding the variability
introduced by complex multi-server network operations. There are prior works
(Wang et al., 2023; Rashidi et al., 2020) that already model complex network
topologies, which researchers can leverage with NeuSight.

###  5.1. Forecasting for Distributed Execution

NeuSight supports a single multi-GPU server execution similar to the ones
connected through NVLink or DGX Box configurations. For each form of
parallelism, NeuSight adds the corresponding network operators to the DNN
graph. Pipeline parallelism involves sending and receiving activations between
GPUs in subsequent pipeline stages, while data parallelism and tensor model
parallelism perform allreduce operations on gradients and activations,
respectively.

Estimating performance of network operators. NeuSight estimates the latency of
ring-based allreduce and peer-to-peer send/receive based on the bandwidth of
the network link. We measure the link bandwidth of an existing system and
calculate its network link utilization. Using this data, along with the peak
link bandwidth of the target system, we estimate the latency of the collective
operators on the target system. The estimated latency of network operators is
combined with the predicted latency of per-GPU DNN graph to predict the end-
to-end latency of distributed execution. NeuSight uses the following methods
for each parallelization strategy.

Pipeline parallel. Pipeline parallel requires forward and backward pass micro-
batches to be interleaved to achieve high utilization as per the schedules
proposed by prior works (Huang et al., 2019; Narayanan et al., 2019, 2021).
For NeuSight, the user provides the pipeline parallel schedule and the
framework stitches the execution across various devices to determine the
overall latency of the model executing on a server. Currently, NeuSight
inserts the necessary pipeline bubbles between the forward and backward pass
based on GPipe (Huang et al., 2019) schedule, however can be easily extended
to other schedules. NeuSight estimates the latency of these pipeline bubbles
based on the size of the microbatch, the number of GPUs, and the latency of
send-receive operations.

Tensor model and Data parallel. NeuSight inserts all-reduce operators into the
DNN graph to synchronize the activations (in tensor model parallel) or
gradients (in data parallel) across GPUs. The framework currently supports
Megatron tensor model strategy (Nvidia, ). The latency of ring-based all-
reduce is estimated and aggregated with the compute latency of kernels to
determine the end-to-end latency.

##  6\. Evaluation

###  6.1. Implementation Details

Table 3. List of GPUs used to train and test the frameworks.

| GPU | Year | Peak FLOPS | Memory | Memory | # | L2 Cache
---|---|---|---|---|---|---|---
|  |  | (TFLOPS) | Size (GB) | BW (GB/s) | SMs | (MB)
Training | P4 | 2016 | 5.4 | 8 | 192 | 40 | 2
Set | P100 | 2016 | 9.5 | 16 | 732 | 56 | 4
| V100 | 2017 | 8.1 | 32 | 900 | 80 | 6
| T4 | 2018 | 14.1 | 16 | 320 | 40 | 4
| A100-40GB | 2020 | 19.5 | 40 | 1555 | 108 | 40
Test | A100-80GB | 2020 | 19.5 | 80 | 1935 | 108 | 40
Set | L4 | 2023 | 31.3 | 24 | 300 | 60 | 48
| H100 | 2022 | 66.9 | 80 | 3430 | 132 | 50

Table 4. Workloads evaluated. Table details model complexity through parameter
size and architecture configurations.

Model | Year | Parameter | # of | # Attention | Hidden | Sequence
---|---|---|---|---|---|---
|  | Size | Layers | Heads | Dimensions | Length
BERT Large | 2018 | 340M | 12 | 16 | 760 | 512
GPT2 Large | 2019 | 774M | 36 | 20 | 1280 | 1024
GPT3 XL | 2020 | 1.3B | 24 | 24 | 3072 | 2048
OPT 1.3B | 2022 | 1.3B | 24 | 24 | 2048 | 2048
GPT3 2.7B | 2020 | 2.7B | 32 | 32 | 2560 | 2048
SwitchTrans | 2021 | 5.3B | 24 | 32 | 1024 | 512

Hardware. Table 3 shows the GPUs used to evaluate NeuSight. We use 5 GPUs
released from 2016 to 2020 for training our MLP predictor. Additionally, we
use 2 GPUs released in 2022 and 2023 as out-of-distribution GPUs. For all the
final results, A100-40GB variant was used for training but the 100-80GB
variant with larger memory and memory bandwidth was used as the testset.

Generating the training dataset. We use the 5 training set GPUs shown in Table
3 to collect data to train our MLPs. Latency is measured by running each
operator 25 times and averaging the results. Input tensors are initialized
using a normal distribution. All measurements use PyTorch 2.1.0 compiled with
CUDA Library 12.1. The collected data covers the following operators:

  * •

Batched Matrix Multiplication - 87,627 data points Batch size and dimensions
from 1 to 1024.

  * •

Fully-connected Layer - 32256 data points. Batch size from 1 to 8192, input
and output sizes from 1 to 65,536.

  * •

Element-wise operators - 26,066 data points. Batch size from 512 to 16,384,
vector size from 512 to 4096 for addition, division, multiplication, GELU,
ReLU, and Tanh.

  * •

Softmax - 1,807 data points. Batch size from 4096 to 16,384, vector size from
512 to 4096.

  * •

Layer normalization - 1,501 data points. Batch size from 4096 to 16,384,
vector size from 512 to 4096.

We reserve 20% of this sampled dataset for validation. For other operators,
such as the embedding layer, which do not contribute significantly to the end-
to-end latency (e.g. less than 0.1% for BERT Large), we assume that they are
memory-bound operators and estimate latency by dividing the working memory
size by the memory bandwidth of the GPU.

Baselines. We compare NeuSight with three baselines: (1) the roofline
analysis, most widely used analytical performance estimation method, (2)
Habitat, a recent work that uses an MLP to predict GPU performance, and (3) Li
et al. (Li et al., 2023), the linear regression based method to predict the
latency of deep learning models. For fair comparisons, we train the baselines
with the up-to-date training dataset as NeuSight. For Habitat, we train the
MLP following the methodology presented in their paper, but with the larger
MLP for higher accuracy (Section 3.2). For element-wise operators, Habitat
requires a reference latency of the operator on an existing GPU. We measure
the latency of element-wise operators on V100 and use the methodology
presented in the paper to predict the latency for other GPUs. For V100, we use
the latency measured on P100. For Li et al., following the method described in
the paper, we apply linear regression between GPU memory bandwidth and
achieved FLOPS performance. We then infer the achieved FLOPS performance of
the unseen GPU using its memory bandwidth.

Training the NeuSight predictor. We train the MLPs using the AdamW optimizer
with L2 regularization for 100 epochs, with batch sizes of ranging from 16 to
128. For Habitat predictors, the learning rate is set to $5e^{-5}$ for highest
accuracy. For NeuSight, we use a range of learning rates from $1e^{-6}$ to
$5e^{-3}$, tailored to different predictors within the model. We use mean
absolute percentage error for Habitat and symmetric mean absolute percentage
error (Tofallis, 2015) for NeuSight as the loss functions.

(a)  Inference latency prediction error

(b) Training latency prediction error

Figure 7. The inference and training latency prediction percentage error of
NeuSight and baselines with different batch sizes. Out of distribution GPUs
and models are highlighted. Certain models resulting in OOM are omitted due to
space constraints.

Tile size. We utilize the PyTorch Profiler (PyTorch, 2023b) to obtain the
kernel names and the number of thread blocks. For matrix multiplications, we
deduce tile sizes from the metadata appended to the kernel names. For other
kernels, we use the number of thread blocks to infer tile sizes. We record the
kernel name, input dimensions, GPU features, and tile sizes in a database
during training. For prediction, we estimate tile sizes by finding the closest
match in the database based on the kernel name, input dimensions, and GPU
features.

DNN workloads evaluated. Table 4 shows the 6 commonly used model architectures
released between 2018 and 2022. For the Switch Transformer, which is a mixture
of experts model, a 4-expert configuration is employed, with only one expert
active at a time. For each of these models, we measure the inference latency
for a binary classification task using BERT, and for a text generation task
using GPT-2, GPT-3, OPT, and Switch Transformer. We use the time to generate
the first token as the latency metric for text generation tasks. We also
report and compare the per-iteration training time, including a single forward
and backward pass. Predictions for each model are then compared with the
measured latencies to evaluate our framework. Due to the high memory
requirement of training, we measure training latency only on GPUs with at
least 24 GB of HBM.

###  6.2. Results

End-to-end mean percentage error. Figure 7 shows the inference and training
latency prediction error of NeuSight and the baselines on 6 deep learning
models with varying batch sizes executed on 8 GPUs in percentage error.
Instances that include out-of-distribution GPUs or models are highlighted. For
example, H100 is an out-of-distribution GPU as it is not included in the
training set. Similarly, GPT3-2.7B is an out-of-distribution model as it
includes BMM with one of the operand dimensions equal to 2048, which is larger
than the 1024 used in the training set.

Across all GPUs and models, NeuSight shows a modest average percentage error
compared to all other baselines. Specifically, the roofline analysis shows an
average error of 32.4% and 31.8%, Habitat shows 79% and 259%, and Li et al.
shows 62% and 59%, for inference and training, respectively. In contrast,
NeuSight shows a percentage error of only 9.2% for inference and 8.1% for
training workloads.

The advantage of NeuSight over other baselines, especially Habitat and Li et
al., becomes more apparent when evaluated on out-of-distribution GPUs.
NeuSight can predict the latency of the unseen GPUs, demonstrating an average
error of 8.7% and maximum error of 26.24%. Habitat and Li et al. struggle with
out-of-distribution GPUs, show an average error of 232.8% and 88.7%, and
maximum error of 2067.7% and 425.3% respectively, even though they are trained
on the same data as NeuSight. The main reason for the lower absolute
percentage error of NeuSight is that it does not directly use the MLP to make
latency predictions, instead applies GPU architecture and execution insights
atop the machine learning predictions. In the case of newer generations GPUs
like H100, which exhibit features outside the distribution seen in the
training set, Habitat’s MLP prediction encounters challenges in accurately
predicting GEMM operators in DNN workloads. Li et al. also falls short of
NeuSight due to its limited method of inferring hardware utilization of unseen
GPUs. This superior performance highlights the robustness of NeuSight in
handling out-of-distribution features and making accurate predictions even on
unseen GPUs.

Figure 8. Prediction percentage error for each operator type (BMM, Fully-Connected (FC), Element-Wise (EW), Softmax, Layer Normalization (LN)), averaged over all the evaluated workloads. Table 5. The inference latency prediction percentage error of NeuSight with operator fusion with varying batch sizes. | Batch |  | L4 |  | A100-40GB |  | H100 |
---|---|---|---|---|---|---|---|---
| Size |  | Non-fused | Fused | Non-fused | Fused | Non-fused | Fused
BERT-Large | 8 | Measured Latency (ms) | 409.5 | 344.1 | 212.1 | 195.0 | 69.5 | 65.0
|  | NeuSight Prediction (ms) |  338.4 (17.4%) |  264.3 (23.2%) |  213.9 (0.8%) |  198.3 (1.7%) |  87.8 (26.2%) |  80.7 (24.3%)
| 16 | Measured Latency (ms) | 877.1 | 724.7 | 408.5 | 385.2 | 136.6 | 127.0
|  | NeuSight Prediction (ms) |  645.6 (26.4%) |  519.5 (28.3%) |  405.6 (0.7%) |  378.2 (1.8%) |  170.3 (24.7%) |  157.6 (24.1%)
GPT2-Large | 4 | Measured Latency (ms) | 1242.2 | 886.8 | 550.1 | 484.0 | 213.6 | 174.7
|  | NeuSight Prediction (ms) |  1080.0 (13.1%) |  849.0 (4.3%) |  571.3 (3.9%) |  525.5 (8.6%) |  208.0 (2.6%) |  186.8 (6.9%)
| 8 | Measured Latency (ms) | 2485.4 | 1775.1 | 1110.1 | 959.4 | 411.9 | 343.4
|  | NeuSight Prediction (ms) |  2278.0 (8.3%) |  1816.9 (2.4%) |  1169.2 (5.3%) |  1083.6 (12.9%) |  468.4 (13.7%) |  429.0 (24.9%)
Table 6. The distributed training latency prediction percentage error of NeuSight on H100 and A100 GPU servers training GPT2-Large and GPT3-XL with different global batch sizes. Models resulting in out-of-memory errors are marked as OOM. | Batch |  | A100-40GB x 4 (NVLink) | H100 x 4 (DGX Box)
---|---|---|---|---
| Size |  | Data Parallel | Tensor Parallel | Pipeline Parallel | Data Parallel | Tennsor Parallel | Pipeline Parallel
GPT2-Large | 8 | Measured Latency (ms) | 466.4 | 460.2 | 807.4 | 202.1 | 206.4 | 341.1
|  | NeuSight Prediction (ms) |  461.2 (1.1%) |  461.0 (0.2%) |  784.8 (2.8%) |  195.0 (3.5%) |  198.0 (4.1%) |  330.7 (3.0%)
| 16 | Measured Latency (ms) | OOM | OOM | OOM | 627.8 | 688 | 1146.6
|  | NeuSight Prediction (ms) |  |  |  |  603.9 (3.8%) |  805.3 (17.0%) |  1018.5 (11.2%)
GPT3-XL | 4 | Measured Latency (ms) | OOM | OOM | OOM | OOM | 987.8 | 1715.7
|  | NeuSight Prediction (ms) |  |  |  |  |  1001.7 (1.4%) |  1521.9 (11.3%)

Per-operator prediction error. Figure 8 shows the prediction error for each
kernel type (BMM, fully-connected, element-wise, softmax, layer
normalization), averaged across all workloads evaluated in Figure 7. The
roofline analysis consistently shows a higher percentage error compared to
NeuSight for both in-distribution and out-of-distribution data points, with an
average percentage error of 34%. For, Habitat and Li et al. their accuracy
significantly degrades for out-of-distribution data points, especially for BMM
and fully-connected operators. Habitat and Li et al. show percentage error of
135.9% and 29.6% for BMM, and 264.7% and 148.2% for fully-connected operators,
respectively. In contrast, NeuSight shows better accuracy than roofline
analysis, Habitat and Li et al., with a percentage error of 11.9% for BMM
operators and 15.8% for fully-connected operators.

Operator fusion. Table 5 shows the inference latency prediction error of BERT-
Large and GPT2-Large with operator fusion on L4, A100 (40GB), and H100. Using
torch.compile (PyTorch, b), the built-in operator fusion tool in PyTorch, we
fused the supported operators and measured the latency of the fused models.
NeuSight predicts latency with a 13.6% error across all fused models, 17.2%
for BERT-Large, and 10% for GPT2-Large.

Distributed Execution. Table 6 shows the training latency predictions for
distributed execution on two types of GPU servers: one with 4 A100s-40GB
connected via NVLink and 4 H100s in a DGX Box (NVIDIA, 2023b), training
GPT2-Large and GPT3-XL. The A100 GPUs are connected in a mesh topology with
each GPU using 12 NVLinks, providing a bi-directional bandwidth of 600GB/s. In
the H100 DGX system, each GPU is connected with 18 NVLinks, providing a bi-
directional bandwidth of 900GB/s. Both configurations enable full-bandwidth
communication between any two GPUs. We evaluate NeuSight on data, tensor
model, and pipeline parallel execution, albeit individually. This is because
NeuSight aims to also estimate the latency for a wide range of models,
including the ones which cannot fit on a single device due to memory, thus
require tensor model or pipeline to split the DNN graph. For each method, we
split the execution across the 4 GPUs. For pipeline parallelism, we have 4
micro-batches. Across all the models, NeuSight predicts the latency of
distributed training with an average error of 5.4%, including 6.9% for the
H100 server and 1.4% for the A100 server. This demonstrates that NeuSight
predicts effectively for per-kernel, per-GPU, and distributed execution on
various GPUs and server types. This is mainly due to the robust underlying
mechanism of kernel estimation, that can be extended with the network level
estimates to forecast distributed execution latencies.

##  7\. Conclusion

In this work, we present a novel framework, NeuSight, that forecasts the
performance of deep learning execution on GPU servers. In developing NeuSight,
we leverage the execution deep learning kernel on a GPU architecture such as
tiled execution and use this insight to decompose the prediction problem to
smaller sub-problems, i.e., per-tile prediction. For each tile, we use machine
learning models to capture the complex non-linear relationship between
achieved throughput and input features, and impose performance laws to bound
the predicted latency. Results on a variety of models and GPUs, show that
NeuSight can provide better predictions than prior works in the area. We will
open-source NeuSight to benefit broader research community.

## References

  * Isaev et al. [2023] Mikhail Isaev, Nic Mcdonald, Larry Dennison, and Richard Vuduc.  Calculon: a methodology and tool for high-level co-design of systems and large language models.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC)_ , 2023.
  * Li et al. [2023] Ying Li, Yifan Sun, and Adwait Jog.  Path forward beyond simulators: Fast and accurate GPU execution time prediction for DNN workloads.  In _International Symposium on Microarchitecture (MICRO)_ , 2023.
  * Yu et al. [2021] Geoffrey X. Yu, Yubo Gao, Pavel Golikov, and Gennady Pekhimenko.  Habitat: A Runtime-Based computational performance predictor for deep neural network training.  In _Annual Technical Conference (USENIX ATC)_ , 2021.
  * [4] NVIDIA.  NVIDIA cuDNN.  https://developer.nvidia.com/cudnn.
  * NVIDIA [2017] NVIDIA.  Cutlass: Fast linear algebra in cuda c++.  https://developer.nvidia.com/blog/cutlass-linear-algebra-cuda/, 2017.
  * [6] Nvidia.  Nvidia dgx platform.  https://www.nvidia.com/en-us/data-center/dgx-platform/.
  * [7] AMD.  AMD Instinct HPC Solutions.  https://www.amd.com/en/graphics/servers-instinct-mi-powered-servers.
  * [8] NVIDIA.  NVIDIA Data Center GPUs.  https://www.nvidia.com/en-us/data-center/data-center-gpus/.
  * [9] AMD.  AMD Instinct Accelerators.  https://www.amd.com/en/products/accelerators/instinct.html.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In _International Conference on Neural Information Processing Systems (NeuRIPS)_ , 2012.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very Deep Convolutional Networks for Large-Scale Image Recognition, 2015.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2019.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Raffel et al. [2023] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Smith et al. [2022] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro.  Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model, 2022.
  * Forbes [2024] Forbes.  Meta To Build Open-Source Artificial General Intelligence For All, Zuckerberg Says .  https://www.forbes.com/sites/johnkoetsier/2024/01/18/zuckerberg-on-ai-meta-building-agi-for-everyone-and-open-sourcing-it, 2024.
  * Jouppi et al. [2023] Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson.  Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings, 2023.
  * Sharma et al. [2016] Hardik Sharma, Jongse Park, Divya Mahajan, Emmanuel Amaro, Joon Kyung Kim, Chenkai Shao, Asit Mishra, and Hadi Esmaeilzadeh.  From High-Level Deep Neural Models to FPGAs.  In _International Symposium on Microarchitecture (MICRO)_ , October 2016.
  * Mahajan et al. [2016] Divya Mahajan, Jongse Park, Emmanuel Amaro, Hardik Sharma, Amir Yazdanbakhsh, Joon Kim, and Hadi Esmaeilzadeh.  Tabla: A unified template-based framework for accelerating statistical machine learning.  In _International Symposium on High-Performance Computer Architecture (HPCA)_ , March 2016.
  * Mahajan et al. [2018] Divya Mahajan, Joon Kyung Kim, Jacob Sacks, Adel Ardalan, Arun Kumar, and Hadi Esmaeilzadeh.  In-rdbms hardware acceleration of advanced analytics.  _Proc. VLDB Endow._ , 11(11):1317–1331, July 2018.  ISSN 2150-8097.  doi: 10.14778/3236187.3236188.  URL https://doi.org/10.14778/3236187.3236188.
  * Fowers et al. [2018] Jeremy Fowers, Kalin Ovtcharov, Michael Papamichael, Todd Massengill, Ming Liu, Daniel Lo, Shlomi Alkalay, Michael Haselman, Logan Adams, Mahdi Ghandi, Stephen Heil, Prerak Patel, Adam Sapek, Gabriel Weisz, Lisa Woods, Sitaram Lanka, Steve Reinhardt, Adrian Caulfield, Eric Chung, and Doug Burger.  A configurable cloud-scale dnn processor for real-time ai.  In _Proceedings of the 45th International Symposium on Computer Architecture, 2018_. ACM, June 2018.  URL https://www.microsoft.com/en-us/research/publication/a-configurable-cloud-scale-dnn-processor-for-real-time-ai/.
  * Chen et al. [2019] Yu-Hsin Chen, Tien-Ju Yang, Joel Emer, and Vivienne Sze.  Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices.  _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , 9(2):292–308, 2019.
  * Olyaiy et al. [2023] MohammadHossein Olyaiy, Christopher Ng, Alexandra Sasha Fedorova, and Mieszko Lis.  Sunstone: A Scalable and Versatile Scheduler for Mapping Tensor Algebra on Spatial Accelerators.  In _International Symposium on Performance Analysis of Systems Software (ISPASS)_ , 2023.
  * Kwon et al. [2019] Hyoukjun Kwon, Prasanth Chatarasi, Michael Pellauer, Angshuman Parashar, Vivek Sarkar, and Tushar Krishna.  Understanding reuse, performance, and hardware cost of DNN dataflows: A data-centric approach.  In _International Symposium on Microarchitecture (MICRO)_ , 2019.
  * Bakhoda et al. [2009] Ali Bakhoda, George L. Yuan, Wilson W. L. Fung, Henry Wong, and Tor M. Aamodt.  Analyzing CUDA workloads using a detailed GPU simulator.  In _International Symposium on Performance Analysis of Systems Software (ISPASS)_ , 2009.
  * Khairy et al. [2020] Mahmoud Khairy, Zhesheng Shen, Tor M. Aamodt, and Timothy G. Rogers.  Accel-sim: An extensible simulation framework for validated GPU modeling.  In _International Symposium on Computer Architecture (ISCA)_ , 2020.
  * Avalos Baddouh et al. [2021] Cesar Avalos Baddouh, Mahmoud Khairy, Roland N. Green, Mathias Payer, and Timothy G. Rogers.  Principal kernel analysis: A tractable methodology to simulate scaled GPU workloads.  In _International Symposium on Microarchitecture (MICRO)_ , 2021.
  * AMD [2024] AMD.  AMD Instinct™ MI250X Accelerators, 2024.  URL https://www.amd.com/en/products/accelerators/instinct/mi200/mi250x.html.
  * PyTorch [a] PyTorch.  PyTorch.  https://pytorch.org/, a.
  * [31] Google.  TensorFlow.  https://www.tensorflow.org/.
  * Zhu et al. [2020] Hongyu Zhu, Amar Phanishayee, and Gennady Pekhimenko.  Daydream: Accurately estimating the efficacy of optimizations for DNN training.  In _Annual Technical Conference (USENIX ATC)_ , 2020.
  * Narayanan et al. [2019] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia.  Pipedream: generalized pipeline parallelism for dnn training.  In _Proceedings of the 27th ACM Symposium on Operating Systems Principles_ , SOSP ’19, page 1–15, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450368735.  doi: 10.1145/3341301.3359646.  URL https://doi.org/10.1145/3341301.3359646.
  * Huang et al. [2019] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen.  GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada_ , pages 103–112, 2019.
  * [35] Nvidia.  Megatron-lm.  https://github.com/NVIDIA/Megatron-LM.
  * Krizhevsky et al. [2017] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  _Commun. ACM_ , 60(6):84–90, May 2017.
  * NVIDIA [a] NVIDIA.  Nvlink.  https://developer.nvidia.com/nccl, a.
  * Kumar et al. [2022] Aviral Kumar, Amir Yazdanbakhsh, Milad Hashemi Kevin Swersky, and Sergey Levine.  Data-driven offline optimization for architecting hardware accelerators.  In _International Conference on Neural Information Processing Systems (NeuRIPS)_ , 2022.
  * Hong [2009] Hyesoon Hong, Sunpyo Kim.  An analytical model for a GPU architecture with memory-level and thread-level parallelism awareness.  In _International Symposium on Computer Architecture (ISCA)_ , 2009.
  * Qi et al. [2017] Hang Qi, Evan R Sparks, and Ameet Talwalkar.  PALEO: A PERFORMANCE MODEL FOR DEEP NEURAL NETWORKS.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Jia et al. [2012] Wenhao Jia, Kelly A. Shaw, and Margaret Martonosi.  Stargazer: Automated regression-based GPU design space exploration.  In _International Symposium on Performance Analysis of Systems Software (ISPASS)_ , 2012.
  * Wu et al. [2015] Gene Wu, Joseph L. Greathouse, Alexander Lyashevsky, Nuwan Jayasena, and Derek Chiou.  GPGPU performance and power estimation using machine learning.  In _International Symposium on High-Performance Computer Architecture (HPCA)_ , 2015.
  * O’Neal et al. [2017] Kenneth O’Neal, Philip Brisk, Emily Shriver, and Michael Kishinevsky.  HALWPE: Hardware-assisted light weight performance estimation for GPUs.  In _Design Automation Conference (DAC)_ , 2017.
  * Yang et al. [2022] Gyeongsik Yang, Changyong Shin, Jeunghwan Lee, Yeonho Yoo, and Chuck Yoo.  Prediction of the resource consumption of distributed deep learning systems.  In _Proceedings of the ACM on Measurement and Analysis of Computing Systems (POMACS)_ , 2022.
  * Ardalani et al. [2015] Newsha Ardalani, Clint Lestourgeon, Karthikeyan Sankaralingam, and Xiaojin Zhu.  Cross-architecture performance prediction (XAPP) using CPU code to predict GPU performance.  In _International Symposium on Microarchitecture (MICRO)_ , 2015.
  * Dao et al. [2022] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  FlashAttention: Fast and memory-efficient exact attention with IO-awareness.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Chen et al. [2018] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy.  TVM: An automated end-to-end optimizing compiler for deep learning.  In _USENIX Symposium on Operating Systems Design and Implementation (OSDI)_ , 2018.
  * NVIDIA [b] NVIDIA.  GPU Performance Background.  https://docs.nvidia.com/deeplearning/performance/pdf/GPU-Performance-Background-User-Guide.pdf, b.
  * Williams et al. [2009] Samuel Williams, Andrew Waterman, and David Patterson.  Roofline: an insightful visual performance model for multicore architectures.  _Communications of the ACM_ , 2009.
  * NVIDIA [2023a] NVIDIA.  NVIDIA H100 Tensor Core GPU Architecture.  https://resources.nvidia.com/en-us-tensor-core, 2023a.
  * NVIDIA [2024] NVIDIA.  NVIDIA Blackwell Architecture Technical Brief.  https://resources.nvidia.com/en-us-blackwell-architecture, 2024.
  * PyTorch [2023a] PyTorch.  torch.fx-PyTorch 2.2 documentation.  https://pytorch.org/docs/stable/fx.html, 2023a.
  * Wang et al. [2023] Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch.  TopoOpt: Co-optimizing network topology and parallelization strategy for distributed training jobs.  In _USENIX Symposium on Networked Systems Design and Implementation (NSDI)_ , 2023.
  * Rashidi et al. [2020] Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna.  Astra-sim: Enabling sw/hw co-design exploration for distributed dl training platforms.  In _International Symposium on Performance Analysis of Systems Software (ISPASS)_ , 2020.
  * Narayanan et al. [2021] Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia.  Memory-efficient pipeline-parallel DNN training.  In _International Conference on Machine Learning_ , pages 7937–7947. PMLR, 2021.
  * Tofallis [2015] Chris Tofallis.  A better measure of relative prediction accuracy for model selection and model estimation.  _Journal of the Operational Research Society (JORS)_ , 2015.
  * PyTorch [2023b] PyTorch.  PyTorch Profiler, February 2023b.  URL https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html.
  * PyTorch [b] PyTorch.  Introduction to torch.compile.  https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html, b.
  * NVIDIA [2023b] NVIDIA.  NVIDIA DGX H100.  https://resources.nvidia.com/en-us-dgx-systems/ai-enterprise-dgx, 2023b.