# Determining HEDP Foams’ Quality
with Multi-View Deep Learning Classification

Nadav Schneider$1,2$, Matan Rusanovsky$3,4$, Raz Gvishi$5$ and Gal
Oren$3,6\hskip 2.84544pt$${}^{\textrm{{\char 0\relax}}}$
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>$1$Technology Division, Soreq Nuclear Research
Center - Yavne, Israel  $2$Israel Atomic Energy Commission  $3$Scientific
Computing Center, Nuclear Research Center – Negev, Israel  $4$Department of
Physics, Nuclear Research Center – Negev, Israel  $5$Applied Physics Division,
Soreq Nuclear Research Center - Yavne, Israel  $6$Department of Computer
Science, Technion – Israel Institute of Technology, Israel

###### Abstract

High energy density physics (HEDP) experiments commonly involve a dynamic
wave-front propagating inside a low-density foam. This effect affects its
density and hence, its transparency. A common problem in foam production is
the creation of defective foams. Accurate information on their dimension and
homogeneity is required to classify the foams’ quality. Therefore, those
parameters are being characterized using a 3D-measuring laser confocal
microscope. For each foam, five images are taken: two 2D images representing
the top and bottom surface foam planes and three images of side cross-sections
from 3D scannings. An expert has to do the complicated, harsh, and exhausting
work of manually classifying the foam’s quality through the image set and only
then determine whether the foam can be used in experiments or not. Currently,
quality has two binary levels of normal vs. defective. At the same time,
experts are commonly required to classify a sub-class of normal-defective,
i.e., foams that are defective but might be sufficient for the needed
experiment. This sub-class is problematic due to inconclusive judgment that is
primarily intuitive. In this work, we present a novel state-of-the-art multi-
view deep learning classification model that mimics the physicist’s
perspective by automatically determining the foams’ quality classification and
thus aids the expert. Our model achieved 86% accuracy on upper and lower
surface foam planes and 82% on the entire set, suggesting interesting
heuristics to the problem. A significant added value in this work is the
ability to regress the foam quality instead of binary deduction and even
explain the decision visually.
The source code used in this work, as well as other relevant sources, are
available at: https://github.com/Scientific-Computing-Lab-NRCN/Multi-View-
Foams.git.

###### Index Terms:

HEDP, Low-Density Foams, Aerogel, Multi-View Classification, Deep Learning,
LIME.

##  I Introduction

###  I-A HEDP Foams and Aerogels

High energy density physics (HEDP) experiments [1, 2, 3, 4, 5] commonly
involve a dynamic wave-front propagating inside a low-density foam. This
effect affects its density and hence, its transparency. The analysis of the
experimental measurements required accurate information on the dimension and
homogeny of the foam. Therefore, a dimension and homogeny characterization of
the foam using a 3D-measuring laser confocal microscope is needed. For each
foam, five images were taken: two 2D images representing the upper surface
foam plane (henceforth, ’top’ and ’bottom’ images) and three other images of
side cross-sections from 3D-scanning images (henceforth, ’profiles’ images)
(Fig. 1, 2).

The foam used in the HEDP experiments is commonly aerogel. Aerogels are a
large family of materials, generally defined as extremely low-density solids
(more than 90% porosity, less than 200mg/cm3 density) [6, 7]. Aerogels can be
composed of metals or dielectric materials and can be produced in pure,
hybrid, and doped forms. For these reasons, aerogels can exhibit a diverse
range of properties (chemical and physical) that could also be tailored for a
specific application. Aerogels are signified by their unique properties,
especially low density and high specific surface area (commonly above
$\sim$600m2/g), resulting in an excellent insulating material for heat,
electric and acoustic. Therefore, their applications range from thermal
insulators [8, 9, 10, 11], acoustic insulation [12, 13], catalyst supports,
electrode materials and fuel cell [14, 15, 16], random lasers matrices [17],
space micrometeorites collectors [18], bio-medical [19], drug-delivery [20,
21, 22], cosmetic, lightweight magnetic actuator [23], Cherenkov radiators
[24, 25] and HEDP measurements [2, 4].

(a)

(b)

Figure 1: Illustration of the acquired microscope’s images (left) from a given
foam (right). Figure 2: Five images of an original example from the data set.
From left to right: top and bottom views and three profiles.

(a) (A)

(b) (B)

(c) (C)

Figure 3: (A) The multi-view concept, (B) the chosen multi-view deep learning
implementation approach, and (C) represents the modified Resnet in each CNN
block that was intentionally reduced to prevent overfit due to a lack of data.

###  I-B Determinning HEDP Foams’ Quality: Current Status

Aerogels are commonly prepared from Si-alkoxide precursors through sol-gel
chemistry [26, 27, 28, 29]. The gel formed is made of a few nanometers thick
of fragile walls in a random structure surrounded by meso-sized pores. To dry
these delicate materials, supercritical drying is necessary. Transferring the
pore fluid to the supercritical phase makes it possible to vent it out with no
capillary forces [30]. Thus, supercritical drying is essential to achieve dry
material without a collapse of the fine porous structure. A significant
limitation of dry aerogels is mechanical fragility and tending to suffer from
cracks [31, 32, 33]. Therefore, it is required to characterize each sample for
its quality as a foam. The expert searches for characteristics such as
scratches, dirt, and dark stains in the top and bottom views images, implying
a deep hole inside the foam. As a complementary, the profile images help
confirm or refute the initial assumptions. This complicated, harsh, and
exhausting work of manually classifying the foams’ quality through the image
set is mandatory to decide whether said foams can be used in experiments (such
as HEDP). Currently, quality has binary levels of normal vs. defective.
Experts are also commonly required to classify a sub-class of normal-
defective, i.e., foams that are defective in some way but might be sufficient
for a given experiment. This sub-class is problematic due to inconclusive
judgment that is primarily intuitive. Thus, the need to devise a new, precise,
explainable, consistent, and objective classification of said foams’ quality
is essential.

###  I-C Suggested New Approach: Multi-View Classification

In order to mimic the physicist’s intuitive perspective and automatically
determine the foams’ quality even in borderline cases, there is a need to
devise either an intelligent or even a learned model, which will be able to
self-conclude the desired classification [34, 35]. Moreover, as the image set,
in this case, is diverse – both in 3D perspective (top and bottom views) and
in acquisition technology (optics vs. x-ray), standard classification
methodologies that do not consider perspective and domain heterogeneity will
fail. Thus, the recently novel multi-view approach [36, 37, 38] was chosen to
solve this problem correctly.

Multi-view classification models are designed for cases when the final
decision depends on different features in different images [39]. Multi-view
approach suggests looking over an object from several points of view (both
spatially and visually), which is technically achieved by extracting unique
features of the object from all the viewpoints, concatenating them into one
vector, and making the decision on it. The idea has two implementation
approaches: (1) Classical feature extractions and machine learning algorithms
[40, 41]. (2) Deep multi-view adjusted convolutional neural networks [42, 39].
In this work, we examined both of the methods as our data set is comparably
small for initial learning.

####  I-C1 Features Extraction and Machine Learning

Before the age of deep learning, there was a need to ”manually” extract
features from given images [43]. This technique aims to numerically
distinguish and represent unique features of the image [43]. Several feature
extractor algorithms were devised over the years for different tasks. Some of
the famous ones are SIFT (Scale Invariant Feature Transform) and SURF
(Speeded-Up Robust Features) [44]. The main focus of these feature extractors
is extracting unique features such as scale, illumination, rotation,
distortion, etc. Most feature extractors collect key points of interest in the
image and produce a descriptor for each. The descriptors are usually numerical
vectors with constant size.

In the multi-view approach, the descriptors are collected from the different
images and grouped into centroids [40, 41] using an unsupervised model such as
$K$-Means [45]. These centroids are like a ”vocabulary of visual words” [46].
The next stage is finding visual words for each image and a vector creation
that counts the number of features belonging to each centroid of visual words.
The result is a $(1,K)$ properties vector representing an image, and using
this process on all the images in the data set yields a $(N,K)$ matrix while
$N$ represents the number of images and $K$ represents the vocabulary size.
Afterward, classification is done on the matrix using a machine learning
algorithm such as logistic regression, supported vector machine (SVM) [41],
etc. Finally, solution adaptation into multi-view is done by creating a
feature vector for all images in one example, using a feature extractor, and
counting the visual words for all images (Fig. 3 A).

Nevertheless, in this work, no learning or understanding could be achieved
using this method, suggesting either not enough data was collected for the
data set or no sufficient learning was made. As such, there was a need for a
comprehensive deep-learning approach.

####  I-C2 Deep Multi-View Adjusted Convolutional Neural Networks

Following the success of convolutional neural networks in computer vision
problems [47], there was an attempt to adjust these networks to multi-view
classification [48]. One approach is to take the feature map of every view and
stack them. For example, if every feature map is in size $K\cdot K\cdot C$, a
union of the five maps is executed along the channels’ dimensions, and one
matrix of $K\cdot K\cdot 5C$ of feature maps is created. This approach assumes
there is an order of orientation between the different views. The other
approach is pooling the feature maps (view pooling), which is the chosen
approach for the given problem (although orientation exists, the images do not
naturally precept a single object from different angles). This approach unites
all the views’ features without assuming an order (Fig. 3 B).

In computer vision, when there are merely several examples similar to the
given problem, as in our case, transfer learning, which uses prior knowledge
accumulated from another model trained on another data set, is usually used
[49]. ResNet is the selected pre-trained model for the given problem [50],
trained on the ImageNet data set [51], which contains millions of high-
resolution labeled images and 22 thousand different classes. This model is
designed for object detection tasks using feature extraction, and with an
addition of neuron layers at the end of the network, one can finetune for a
given task.

##  II Data set Creation

While many data sets are created with strict rules and repeatable methods for
an adequate examination, our data set was created solely for an expert’s
classification purpose. In the absence of reproducible settings for the 3D
laser microscope and an arranged process of collecting the data, a
considerable amount of pre-processing is required to fit the data set into a
machine learning model. Furthermore, since only experts can label these
examples, the labeled data is rare, and there is no real option to reach
extensive labeled data. Consequently, our goal in this work was to develop an
end-to-end model with the constraint of a severe shortage in data.
Iteratively, we added batches of new labeled images when they were ready while
designing the model as a few-shot model. This iterative process results in a
data set with 95 labeled examples, each consisting of 5 images, i.e., 475
images in total.

We notice that the normal-defective labels are given when an example does not
have an obvious decision for its quality. Without significant model learning,
which can be achieved using lots of normal and defective examples, one can not
expect to classify samples even an expert could not. Therefore, we refer to
the normal-defective labels as defective labels under the strict assumption of
”if there is a doubt, there is no doubt.” This assumption is necessary for the
given problem because we can not afford to ignore examples due to their tiny
amount. However, the severe assumption above is not always correct, leading to
a trade-off between the measured metrics; accuracy and AUC (Section IV). Thus,
we check our model on different learning configurations (Fig. 6).

##  III Pre-learning Process

An initial running of the multi-view deep convolutional neural networks on the
data set has been done, and a quickly evolved overfit been observed. There are
two common reasons for overfitting: a few training examples and an over-
complicated model [52]. Therefore, solutions such as data augmentation,
reduction, and pre-processing (Table I, Fig. 4) are done to overcome the few
physics-guided examples issue while preserving the original and relevant
properties for decision.

|  Original |  Pre-Proccesed
---|---|---
Top/Bottom |  |
Profiles |  |
TABLE I: Data set images before and after pre-processing.

###  III-A Data Augmentation

Orientation conservation is easily destroyed when creating new examples with
augmentation. Nonetheless, since training deep learning models require lots of
examples, an augmentation approach has been attempted. Classical augmentations
such as brightness, contrast adjustments, and rotations contribute to the
robustness and generalization of the model and, therefore, decrease overfit.
Several rotations permutations have been chosen: -10° to 10° with a 5° jump
for the profiles and 0° to 270° with a 90° jump for the top and bottom views
images. This wide range is possible due to the symmetry of the images
(circles). In addition, gaussian noise, brightness, and contrast adjustments
have been performed (Fig. 4).

Figure 4: Five images of an original example from the data set after pre-
process and data augmentation stages. From left to right: top and bottom views
and three profiles.

###  III-B Data Reduction and Pre-Processing

Pre-process helps focus the model on relevant features for classification. An
automatic physics-guided pre-processing tool has been developed to create a
proper data reduction that preserves the original and relevant properties of
the foam with the following actions:

  1. 1.

RGB to grayscale conversion: classification of the foams is independent of
their color; therefore, the conversion prevents the model from learning
irrelevant features.

  2. 2.

Intensity quantization: a division of the pixels’ values (which range from 0
to 255) to 10 bins. In many images, the black background is not an absolute
black but a collection of values near zero. The model might search for a
reason around these pixels’ variance, and bins division prevents it.

  3. 3.

Circle extraction: only the central circle is relevant for classification.
Extracting this circle111In order to focus the model on relevant features, an
automatic algorithm for circle extraction in the top and bottom views images
is required. Canny edge detection [53] using erosion and dilation is not
enough to effectively mask the circles. Hence, a designated algorithm has been
developed. The algorithm uses permutations of circles in different radii and
locations around the center to find the circle with minimal bright pixels and
maximal dark pixels (Fig. 5). The algorithm assumes that the background will
always be brighter than the circle. Pixel’s definition of bright or dark is
changeable. and masking the ring and other outliers helps the model focus on
relevant features (Fig. 5).

  4. 4.

Circles bounding: minimizing the black background solves the problem of
variance in the circles’ position and conserves homogeneity between the
images.

  5. 5.

Profiles centering and padding: padding black pixels for the background of
profiles if necessary and centering the profiles. These operations conserve
homogeneity and constant size between the profiles. Further manual cropping
has been done to keep only the relevant scope.

Figure 5: Automatic circle extractor algorithm. Choosing the circle with
minimal bright pixels and maximal dark pixels.

##  IV The Learning Process and its Results

ACCURACY | One-view | One-view-ND | Multi-view | Multi-view-ND
---|---|---|---|---
Top | 72 | 78 | – | –
Bottom | 79 | 84 | – | –
Top-Bottom | 77 | 73 | 86 | 78
Profiles | – | – | 79 | 73
Full Group | – | – | 82 | 78
TABLE II: Accuracy for each purposed model.

AUC | One-view | One-view-ND | Multi-view | Multi-view-ND
---|---|---|---|---
Top | 74 | 76 | – | –
Bottom | 76 | 74 | – | –
Top-Bottom | 69 | 78 | 71 | 64
Profiles | – | – | 77 | 81
Full Group | – | – | 84 | 75
TABLE III: AUC for each purposed model.

|  Loss +ND |  Accuracy +ND |  Loss -ND |  Accuracy -ND
---|---|---|---|---
OV Top |  |  |  |
OV Bottom |  |  |  |
OV Top Bottom |  |  |  |
MV Top Bottom |  |  |  |
MV Profiles |  |  |  |
MV Full Group |  |  |  |
TABLE IV: Learning curves (accuracy and loss) of all models. +ND and -ND stand
for normal-defective and represent the models’ training and testing with or
without normal-defective examples. MV stands for a multi-view model and OV
stands for a one-view model. Best models’ cells are shadowed.

In cases of a shortage of data, transfer learning is usually a decent
solution. However, as mentioned above, an over-complicated model prunes to
overfit. Thus, we applied the following methods to the model to overcome the
problem:

  1. 1.

Simplify the model by cutting some of its last layers (Fig. 3 C). The 18 last
convolutional layers have been cut, and the network’s last layer has been
reduced to 128 neurons.

  2. 2.

Changing input dimension for 1x244x244 instead of 3x224x224 – since ResNet has
been trained on a colored data set – differently from our input examples which
are in grayscale – the dimensional change decreases lots of unnecessary
weights.

Overall, there are 1,341,890 learnable parameters instead of 21,799,674 –
about 6% of the original network.

Searching for the optimal performance, we survey 6 model configurations (Fig.
6). Specifically, three one-view models with solely top view, bottom view, and
a combination of both; three multi-view models with both top and bottom views;
all three profiles; and a full group of five images. Each of the 6 models was
trained on the above-stated data set with and without normal-defective
examples (marked in Table III and Table III as ND – normal-defected).

Figure 6: Six learning configurations based on the different types of models
and image sets.

We hereby elaborate on the relevant performance metrics (IV-A), the effect of
data additions on the model performances (IV-B), the yielded experimental
results (IV-C) and the possibility for explainability (IV-D).

###  IV-A Performance Metrics

Two metrics have been used for measuring the models’ performance. The first is
the model’s accuracy in the epoch with the minimum loss (Table III). This
measurement indicates the model’s performance on the particular test set.
Another metric that has been used is the area under curve (AUC) [54] (Table
III). AUC measures the model’s generalization capabilities and, therefore, its
performance not only on a certain test set but on many. An additional
noteworthy parameter is the loss graph, indicating if convergence and real
learning have occurred.

###  IV-B Additional Data Effect

To understand the impact of the data set size on the resulted model
performances, a comparison between two train sets has been performed: The
train set mentioned above was compared to itself with subtraction of 20
examples, while the test set remains similar for both sets. This test has been
performed to determine unequivocally that there is a need for continued
expansion of the data set to achieve better results and that the model
truthfully learns from the data and does not memorize it. The addition of 20
training examples has shown an unambiguous improvement in the models’
performances – an improvement of 8% for every model’s accuracy on average and
a 16.5% improvement for every model’s AUC on average.

###  IV-C Experimental Results

The data set split was a 70:30 train-to-test ratio while maintaining a
balanced ratio between classes to prevent the model from learning from skewed
data. Overall, there are 66 training examples with 23, 21, and 22 normal,
normal-defective, and defective labeling correspondingly and 29 test examples
with 10, 10, and 9 normal, normal-defective, and defective labeling
correspondingly. The models have run on 32GB Tesla V100 GPU, using Pytorch
[55]. Accuracy and AUC measures are shown in Table III and Table III,
respectively. Loss and accuracy trends are presented in Table IV.
Conclusively, we show that the multi-view model with top-bottom views has the
best accuracy – 86%, and the multi-view model with the entire group has the
best AUC – 84%.

###  IV-D Model Explainability

Physics-oriented models generally need to ensure that the outcome predictions
are based on actual, relevant features and not on imaginary correlations.
Thus, LIME [56] algorithm was implemented on the model predictions to study
the connections between inputs and outputs. Specifically on the one-view top
model as it is more intuitive to comprehend. In LIME’s output, contributing
areas for normal or defective decisions are marked as green or red,
respectively. White stain is colored under the red area while the green area
is mostly clear – both truthfully explaining the original rationale (Fig. 7).

##  V Conclusions and Future Work

While there is high accuracy for the multi-view model with top-bottom views
(86%), its AUC is relatively low (71%), meaning the model managed to perform
well on the test set, but its generalization capability is incomplete.
Nevertheless, the loss graph shows a significant learning process without any
overfit. On the other hand, the multi-view model with the entire group (full
group) has the best AUC (84%), and its accuracy is relatively high as well
(82%). However, there is an obvious overfit in the early epochs of the
training.

Most of our multi-view models show better results with the normal-defective
examples in accuracy and AUC. A possible explanation may lie in the nature of
normal-defective examples. Those examples are similar to normal, however,
still have tiny defects. The action of labeling those examples as defective
may contribute to learning these nuances and improve the classification.

For future work, we can name the apparent need for data set expansion while
maintaining class balance. Also, further investigation of different pre-
trained models for better transfer learning can be helpful. For example, the
Inception pre-trained model [57], which has been trained on vast x-rays data
set, may be more fitted for extracting features from the profiles. Finally, we
suggest examining the possibility of concatenating the views’ feature maps
which preserves and assumes order, similarly to the given problem, instead of
max-pooling them.

(a) Original

(b) LIME’s Output

Figure 7: Model’s explainability using LIME. Contributing areas for normal or
defective decision are green and red, respectively.

## Acknowledgment

This work was supported by the Pazy foundation and the Lynn and William
Frankel Center for Computer Science. Computational support was provided by the
NegevHPC project [58]. The authors would like to thank Galit Bar$5$, Guy
Lazovski$5$ and Muriel Tzadka$1$ for foam samples preparation.

## References

  * [1] K. Mussack _et al._ , “Simulating hohlraum dynamics and radiation flow for pleiades experiments on nif,” in _APS Division of Plasma Physics Meeting Abstracts_ , vol. 53, 2011, pp. GP9–124.
  * [2] M. Baer, “A numerical study of shock wave reflections on low density foam,” _Shock Waves_ , vol. 2, no. 2, pp. 121–124, 1992.
  * [3] H. M. Johns _et al._ , “A temperature profile diagnostic for radiation waves on omega-60,” _High Energy Density Physics_ , vol. 39, p. 100939, 2021\.
  * [4] Y. Wu _et al._ , “Dynamics of laser produced plasma from foam targets for future nanolithography devices and x-ray sources,” _Scientific Reports_ , vol. 11, no. 1, pp. 1–14, 2021.
  * [5] R. Harel _et al._ , “Complete deep computer-vision methodology for investigating hydrodynamic instabilities,” in _International Conference on High Performance Computing_. Springer, 2020, pp. 61–80.
  * [6] O. Figovsky _et al._ , _Green nanotechnology_. Jenny Stanford Publishing, 2017.
  * [7] D. W. Schaefer _et al._ , “Structure of random porous materials: silica aerogel,” _Physical review letters_ , vol. 56, no. 20, p. 2199, 1986.
  * [8] G. Wei _et al._ , “Thermal conductivities study on silica aerogel and its composite insulation materials,” _International Journal of Heat and Mass Transfer_ , vol. 54, no. 11-12, pp. 2355–2366, 2011.
  * [9] M. Reim _et al._ , “S., m. arduini-schuster, hp ebert, j. fricke, j,” _Silica aerogel granulate material for thermal insulation and daylighting, Solar Energy_ , vol. 79, pp. 131–139, 2005.
  * [10] B. Yoldas _et al._ , “Chemical engineering of aerogel morphology formed under nonsupercritical conditions for thermal insulation,” _Chemistry of materials_ , vol. 12, no. 8, pp. 2475–2484, 2000.
  * [11] E. Zhang _et al._ , “Insulating and robust ceramic nanorod aerogels with high-temperature resistance over 1400 c,” _ACS Applied Materials & Interfaces_, vol. 13, no. 17, pp. 20 548–20 558, 2021.
  * [12] S. Takeshita _et al._ , “Structural and acoustic properties of transparent chitosan aerogel,” _Materials Letters_ , vol. 254, pp. 258–261, 2019.
  * [13] K. Wu _et al._ , “Lightweight and flexible phenolic aerogels with three-dimensional foam reinforcement for acoustic and thermal insulation,” _Industrial & Engineering Chemistry Research_, vol. 60, no. 3, pp. 1241–1249, 2021.
  * [14] Y. Gu _et al._ , “Electronic structure tuning in ni3fen/r-go aerogel toward bifunctional electrocatalyst for overall water splitting,” _ACS Nano_ , vol. 12, no. 1, pp. 245–253, Jan. 2018.
  * [15] M. A. Worsley _et al._ , “Synthesis of graphene aerogel with high electrical conductivity,” _Journal of the American Chemical Society_ , vol. 132, no. 40, pp. 14 067–14 069, 2010.
  * [16] Peles-Strahl _et al._ , “Bipyridine modified conjugated carbon aerogels as a platform for the electrocatalysis of oxygen reduction reaction,” _Advanced Functional Materials_ , vol. 31, no. 26, p. 2100163, 2021.
  * [17] N. U. Wetter _et al._ , “Dynamic random lasing in silica aerogel doped with rhodamine 6g,” _RSC advances_ , vol. 8, no. 52, pp. 29 678–29 685, 2018.
  * [18] M. J. Burchell _et al._ , “Characteristics of cometary dust tracks in stardust aerogel and laboratory calibrations,” _Meteoritics & Planetary Science_, vol. 43, no. 1-2, pp. 23–40, 2008.
  * [19] N. Ganonyan _et al._ , “Entrapment of enzymes in silica aerogels,” _Materials Today_ , vol. 33, pp. 24–35, 2020.
  * [20] G. Nir _et al._ , “Gradual hydrophobization of silica aerogel for controlled drug release,” _RSC advances_ , vol. 11, no. 14, pp. 7824–7838, 2021.
  * [21] García-González _et al._ , “Polysaccharide-based aerogel microspheres for oral drug delivery,” _Carbohydrate polymers_ , vol. 117, pp. 797–806, 2015.
  * [22] J. Zhao _et al._ , “Polyethylenimine-grafted cellulose nanofibril aerogels as versatile vehicles for drug delivery,” _ACS applied materials & interfaces_, vol. 7, no. 4, pp. 2607–2615, 2015.
  * [23] N. Ganonyan _et al._ , “Ultralight monolithic magnetite aerogel,” _Applied Materials Today_ , vol. 22, p. 100955, 2021.
  * [24] D. Asner _et al._ , “Experimental study of aerogel cherenkov detectors for particle identification,” _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment_ , vol. 374, no. 3, pp. 286–292, 1996.
  * [25] T. Sumiyoshi _et al._ , “Silica aerogel cherenkov counter for the kek b-factory experiment,” _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment_ , vol. 433, no. 1-2, pp. 385–391, 1999.
  * [26] C. J. Brinker _et al._ , _Sol-gel science: the physics and chemistry of sol-gel processing_. Academic press, 2013.
  * [27] R. Gvishi, “Monolith sol-gel materials,” _chapter in [The Sol-Gel Handbook], Eds: Levy D. and Zayat M., in publication by Wiley-VCH_ , 2014.
  * [28] I. Smirnova _et al._ , “Aerogel production: Current status, research directions, and future opportunities,” _The Journal of Supercritical Fluids_ , vol. 134, pp. 228–233, 2018.
  * [29] G. Lazovski _et al._ , “A simple method for preparation of silica aerogels doped with monodispersed nanoparticles in homogeneous concentration,” _The Journal of Supercritical Fluids_ , vol. 159, p. 104496, 2020.
  * [30] L. Guy _et al._ , “A procedure to synthesize silica aerogels in a wide range of densities by a single-step base catalyzed recipe,” _Journal of Porous Materials_ , vol. 28, no. 4, pp. 1227–1236, 2021.
  * [31] A. Sedova _et al._ , “Reinforcing silica aerogels with tungsten disulfide nanotubes,” _The Journal of Supercritical Fluids_ , vol. 106, pp. 9–15, 2015\.
  * [32] R. Haj-Ali _et al._ , “Mechanical characterization of aerogel materials with digital image correlation,” _Microporous and Mesoporous Materials_ , vol. 226, pp. 44–52, 2016.
  * [33] A. Sedova _et al._ , “Silica aerogels as hosting matrices for ws2 nanotubes and their optical characterization,” _Journal of materials science_ , vol. 55, no. 18, pp. 7612–7623, 2020.
  * [34] M. Rusanovsky _et al._ , “An end-to-end computer vision methodology for quantitative metallography,” _Scientific Reports_ , vol. 12, no. 1, pp. 1–27, 2022.
  * [35] R. Matan _et al._ , “Mlography: An automated quantitative metallography model for impurities anomaly detection using novel data mining and deep learning approach,” _arXiv preprint arXiv:2003.04226_ , 2020.
  * [36] C. Xu _et al._ , “A survey on multi-view learning,” _arXiv preprint arXiv:1304.5634_ , 2013.
  * [37] S. Sun, “A survey of multi-view machine learning,” _Neural computing and applications_ , vol. 23, no. 7, pp. 2031–2038, 2013.
  * [38] J. Zhao _et al._ , “Multi-view learning overview: Recent progress and new challenges,” _Information Fusion_ , vol. 38, pp. 43–54, 2017.
  * [39] M. Seeland _et al._ , “Multi-view classification with convolutional neural networks,” _Plos one_ , vol. 16, no. 1, 2021.
  * [40] F. Nie _et al._ , “Multi-view clustering and semi-supervised classification with adaptive neighbours,” in _Thirty-first AAAI conference on artificial intelligence_ , 2017.
  * [41] L. Houthuys _et al._ , “Multi-view least squares support vector machines classification,” _Neurocomputing_ , vol. 282, pp. 78–88, 2018.
  * [42] M. Kan _et al._ , “Multi-view deep network for cross-view classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 4847–4855.
  * [43] G. Kumar _et al._ , “A detailed review of feature extraction in image processing systems,” in _2014 Fourth international conference on advanced computing & communication technologies_. IEEE, 2014, pp. 5–12.
  * [44] L. Juan _et al._ , “A comparison of sift, pca-sift and surf,” _International Journal of Image Processing (IJIP)_ , vol. 3, no. 4, pp. 143–152, 2009.
  * [45] J. A. Hartigan _et al._ , “Algorithm as 136: A k-means clustering algorithm,” _Journal of the royal statistical society. series c (applied statistics)_ , vol. 28, no. 1, pp. 100–108, 1979.
  * [46] S. Tafasca, “Multi-view image classification: From logistic regression to multi-view convolutional neural networks (mvcnn),” https://towardsdatascience.com/multi-view-image-classification-427c69720f30, December 24, 2019, [Online].
  * [47] S. Khan _et al._ , “A guide to convolutional neural networks for computer vision,” _Synthesis lectures on computer vision_ , vol. 8, no. 1, pp. 1–207, 2018.
  * [48] H. Su _et al._ , “Multi-view convolutional neural networks for 3d shape recognition,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 945–953.
  * [49] K. Weiss _et al._ , “A survey of transfer learning,” _Journal of Big data_ , vol. 3, no. 1, pp. 1–40, 2016.
  * [50] K. He _et al._ , “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [51] O. Russakovsky _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [52] S. Kadam _et al._ , “Review and analysis of zero, one and few shot learning approaches,” in _International Conference on Intelligent Systems Design and Applications_. Springer, 2018, pp. 100–112.
  * [53] J. Canny, “A computational approach to edge detection,” _IEEE Transactions on pattern analysis and machine intelligence_ , no. 6, pp. 679–698, 1986.
  * [54] C. Marzban, “The roc curve and the area under it as performance measures,” _Weather and Forecasting_ , vol. 19, no. 6, pp. 1106–1114, 2004.
  * [55] A. Paszke _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [56] M. T. Ribeiro _et al._ , “”why should i trust you?”: Explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA,_ , August 13-17, 2016, pp. 1135–1144.
  * [57] C. Szegedy _et al._ , “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Thirty-first AAAI conference on artificial intelligence_ , 2017.
  * [58] R. I. Park, “NegevHPC Project,” https://www.negevhpc.com, 2019, [Online].


# 10 Years Later: Cloud Computing is Closing the Performance Gap

Giulia Guidi  University of California, BerkeleyLawrence Berkeley National
LaboratoryBerkeleyCaliforniaUSA ,  Marquita Ellis  University of California,
BerkeleyLawrence Berkeley National LaboratoryBerkeleyCaliforniaUSA ,  Aydın
Buluç  University of California, BerkeleyLawrence Berkeley National
LaboratoryBerkeleyCaliforniaUSA ,  Katherine Yelick  University of California,
BerkeleyLawrence Berkeley National LaboratoryBerkeleyCaliforniaUSA and  David
Culler  University of California, BerkeleyGoogle, Inc.BerkeleyCaliforniaUSA

(2021)

###### Abstract.

Can cloud computing infrastructures provide HPC-competitive performance for
scientific applications broadly? Despite prolific related literature, this
question remains open. Answers are crucial for designing future systems and
democratizing high-performance computing. We present a multi-level approach to
investigate the performance gap between HPC and cloud computing, isolating
different variables that contribute to this gap. Our experiments are divided
into (i) hardware and system microbenchmarks and (ii) user application
proxies. The results show that today’s high-end cloud computing can deliver
HPC-competitive performance not only for computationally intensive
applications, but also for memory- and communication-intensive applications –
at least at modest scales – thanks to the high-speed memory systems and
interconnects and dedicated batch scheduling now available on some cloud
platforms.

††copyright: rightsretained††journalyear: 2021††conference: Companion of the
2021 ACM/SPEC International Conference on Performance Engineering; April
19–23, 2021; Virtual Event, France††booktitle: Companion of the 2021 ACM/SPEC
International Conference on Performance Engineering (ICPE ’21 Companion),
April 19–23, 2021, Virtual Event, France††doi: 10.1145/3447545.3451183††isbn:
978-1-4503-8331-8/21/04

##  1\. Introduction

The benefit of high-performance computing for scientific research has grown
rapidly, beyond traditional simulation problems to data analysis in light
sources, cosmology, genomics, particle physics, and more (Yelick et al., 2020;
Alexander et al., 2020). Given the vast amounts of data and/or computation
involved in such applications, they can require the full computing power and
memory of high performance computing (HPC) systems. Cloud computing (Fox et
al., 2009; Buyya et al., 2009; Mell and Grance, 2011) is gaining popularity
among scientists as an alternative to HPC for a wide range of sciences such as
physics, bioinformatics, cosmology, and climate research (Evangelinos and
Hill, 2008; Mushtaq et al., 2017).

There are many efforts in the literature to measure the performance of
scientific applications in the cloud. The lack of a low-latency network has
been consistently identified as the main bottleneck (Gupta et al., 2014; Netto
et al., 2018; Yelick et al., 2011; Ellis et al., 2017, 2019). These studies
have shown that the cloud delivers competitive performance for HPC
applications with minimal communication and I/O, but significantly
underperforms for memory- and communication-intensive workloads.
Virtualization overhead has also been identified as a performance-limiting
factor, but studies do not generally agree on its impact. He et al. (He et
al., 2010) (2010) concluded that virtualization technology has no significant
performance overhead, while the results in the Magellan report (Yelick et al.,
2011) (2011) and by Gupta et al. (Gupta et al., 2014) (2014) show that
virtualization overhead along with slow network is one of the major
limitations of the cloud. Performance variability due to resource sharing and
the lack of tools for using and managing cloud environments–such as batch
scheduling and base images–have also further limited the competitiveness of
the cloud for scientific computing (Yelick et al., 2011; Gupta et al., 2014;
Netto et al., 2018).

Understanding the nature of the gap between HPC and cloud systems and whether
the results from the literature are still valid today is critical for guiding
future system design and running scientific applications efficiently in the
cloud. Furthermore, the number of users that supercomputing facilities can
support is limited, and the deployment of any new supercomputer is a multi-
year multi-million dollar investment. Closing the performance gap between HPC
and the cloud would give many more scientists access to adequate computing
resources.

Our work measures the performance of HPC-oriented codes on both cloud and HPC
platforms. Building on previous literature, we investigate whether the
findings apply to today’s cloud platforms and isolate the contribution of
different variables to the HPC cloud performance gap. Our results show that
cloud platforms with similar processors and networks can achieve HPC-
competitive performance, not only for compute-intensive applications, but also
for communication-intensive applications. At small and medium scales, modern
cloud computing has overcome one of its main limitations by providing higher-
speed memory and interconnects for HPC-oriented instances. Cost models, job
wait times, availability of pre-installed software, and other usability
factors are also relevant to evaluating HPC-cloud competitiveness, but are out
of scope here.

##  2\. Background

High Performance Computing (HPC) and cloud computing differ in their original
purpose as well as their economic objectives and access policies. HPC systems
were designed to deliver high performance for dedicated scientific computing,
while cloud computing made networked hardware and software available for
general use.

The differences in their economic objectives and access policies inevitably
affect scheduling, hardware selection, and software configuration decisions.
HPC systems are typically operated by a non-profit organization (university or
national laboratory), funded by a government agency, and allocated to a
particular research community. These systems have very high utilization (over
90%) with non-trivial wait times for users; they support very large-scale
computations with homogeneous hardware that undergoes major upgrades every few
years. In contrast, cloud systems are built for profit, configured to meet
market demand, and operated at lower utilization rates to ensure little or no
wait time. Cloud resources are upgraded continuously and incrementally,
leading to rapid access to new technologies, but also to heterogeneity within
the cloud.

Researchers running scientific applications in the cloud can access instances
with low-latency networks to achieve performance competitive to HPC (He et
al., 2010). However, cloud heterogeneity can limit what is available within an
HPC cloud offering, e.g., it may be more difficult for a user to obtain a
large number of high-performance instances. In the cloud, users can easily
customize their environment without administrative overhead and quickly
provision additional resources to solve large problems (Yelick et al., 2011).
HPC platforms offer limited support for on-demand self-service (Yelick et al.,
2011), but they do offer important features such as resource pooling and broad
network access.

The basic business model differences between cloud and HPC persist today and
lead to complex cost trade-offs that are beyond the scope of this paper.
However, the growing commercial interest in problems such as large-scale
machine learning training have led to changes in cloud configurations. This
has increased the popularity of HPC-as-a-Service in the cloud and has in turn
resurfaced questions about use of the cloud for modest scale parallel
scientific applications.

###  2.1. Low-Level Benchmark

A common approach to comparing the performance of computer systems is to use
low-level benchmarks (Tsouloupas and Dikaiakos, 2006; Alam et al., 2006).
Here, we focus on the investigation of processor, memory, and network
performance.

Processor. Considering a multi-core processor, we refer to it as a _node_ ,
where a _core_ is the basic execution unit in the system. The number of nodes
is later denoted $P$. Here we use the shared memory version of the LINPACK
benchmark (Dongarra et al., 2003) to compare the floating point performance of
the systems under consideration. LINPACK measures performance as the number of
$64$-bit floating point operations a computer can perform per second (FLOPS).
The performance when running an actual application is likely to be lower than
the performance achieved by the LINPACK benchmark.

Memory Hierarchy. CacheBench (Mucci, [n.d.]) measures the performance of the
local memory hierarchy. It computes a number of operations – _read, write,
read/modify/write_ , _memset_ , and _memcpy_ – varying the underlying array
size, thereby revealing the performance of the cache. Operations run for 2
seconds and the average bandwidth (MB/s) is reported. Here we focus on
_memcpy_.

Memory Bandwidth. To measure the maximum memory bandwidth of our systems, we
use the STREAM benchmark (McCalpin, 1995), which performs four vector
operations: _copy_ , _scale_ , _sum_ , and _triad_. STREAM requires that (a)
each array is at least four times the size of the cache memory, and (b) the
size is such that the “timing calibration” output by the program is at least
20 clock ticks. STREAM provides the best possible memory system bandwidth.

Inter-Node Communication. Following standard practice (He et al., 2010), we
use a subset of MPI operations to measure the inter-node communication
performance of our systems. Specifically, we use MPI_Send- recv and
MPI_Alltoall to measure point-to-point and collective latency and bandwidth
using the OSU microbenchmarks (Panda, 2018).

###  2.2. User Application

Besides comparing HPC and cloud systems on a subset of MPI collectives, we
select two representative user applications from scientific computing as
benchmarks: an N-Body simulation written in C++ and a Fast Fourier Transform
(FFT), written in C. N-Body is a computationally intensive application, while
the FFT is more communication intensive (Woo et al., 1995; Asanovic et al.,
2006; Colella, 2004).

An N-Body simulation models a dynamic system of particles, usually under the
influence of physical forces, such as gravity (Ida and Makino, 1992). It is a
common computation in physics, astronomy, and biology. The naive solution
computes the forces acting on the particles by iterating through each pair of
particles, resulting in a complexity of $O(n^{2})$, where $n$ is the number of
particles. In our implementation we consider the density of the particles to
be sufficiently low so that a linear time solution can be achieved with $n$
particles.

The FFT calculates the discrete Fourier transform (DFT) of a sequence or its
inverse (IDFT). In Fourier analysis, a signal is transformed from its original
domain (often time or space) to a frequency domain representation and vice
versa. As a benchmark for the FFT, we use the implementation of Frigo and
Johnson (Frigo and Johnson, 1997, 1998; Frigo, 1999), Fast Fourier Transform
in the West (FFTW).

##  3\. Empirical Methodology

Processor, memory, network, application and programming model, and system age
are all variables that affect performance. Here, we measure the performance
gap by isolating the contribution of the different variables by dividing our
experiments into two categories: (i) hardware and system and (ii) user
application.

First, we isolate the contribution of processor and memory to identify
similarities or significant differences in in-node performance. Then, we
investigate the contribution of the inter-processor network by measuring the
latency and bandwidth of communication primitives between machines. Finally,
we study performance of HPC and cloud computing from an application
perspective. Unless differently noted, the results reported in this paper
represent the average value across 10 runs.

To this end, we use two metrics to characterize our applications: hardware
events and the communication to computation ratio ($Cm/Cp$). The $Cm/Cp$ ratio
is defined as communication time divided by computation time for a given
execution of a parallel application on a given parallel machine with explicit
communication (Crovella et al., 1992). Both metrics can help interpret the
potential performance gap between HPC and cloud systems.

###  3.1. Experimental Setting

Table 1. Details of the evaluated machines: name, system age in years, number
of cores per node, processor frequency, theoretical peak performance
(GFlops/s) per node, processor, memory, advertised injection bandwidth
(Gigabits/s), and caches sizes. †Custom model for Amazon AWS. KNL’s L2 is
shared between two cores. ∗Advertised user-process injection bandwidth (Cray,
[n.d.]).

Platform Age Core/Node Frequency (GHz) Theoretical Peak (GFlops/Node)
Processor Memory (GiB) Network (Gbps) L1 L2 L3 Cori Haswell 4 32 2.3 1,177
Xeon E5-2698V3 120 ∗82 64KB 256KB 40MB Cori KNL 4 68 1.4 3,046 Xeon Phi 7250
90 ∗82 64KB 1MB - AWS r5dn.16xlarge 1 32 2.5 2,560 Xeon Platinum 8259CL 512 75
64KB 1MB 36MB AWS c5.18xlarge 1 36 3.0 3,456 Xeon Platinum 8124M† 144 25 64KB
1MB 25MB

Our experiments are conducted on the Intel Xeon “Haswell” (Cori Haswell) and
Intel Xeon Phi “Knight’s Landing” (KNL) partitions (Cori KNL) of the Cori Cray
XC40 HPC system at NERSC, an Amazon Web Services (AWS) commodity cluster with
r5dn.16xlarge (R5) instances (optimized for memory-intensive workloads), and
one with AWS c5.18xlarge (C5) instances (optimized for compute- intensive
workloads). Details for each instance are listed in Table 1.

We chose these four platforms because of their easy availability and the
diversity of architectures. In particular, we selected the two AWS instances
to represent two extremes of the AWS catalog (memory-optimized versus compute-
optimized) and selected these two instances because they allowed us to
allocate multiple nodes in the same placement group in a reasonable amount of
time. AWS clusters run as _dedicated instances_ to reduce the potential
performance slowdown from sharing resources, and use Slurm as the workload
manager (Yoo et al., 2003). The need for tools to simplify the use of cloud
environments and better software stacks for clouds has been noted in past
literature (Yelick et al., 2011). We use AWS ParallelCluster to provision and
manage AWS clusters. It automatically sets up the required compute resources
and shared file system in about five to ten minutes in our experience. AWS
also provides a collection of Amazon Machine Images (AMIs) installed with
libraries and software such as MPI, BLAS, and TensorFlow. Notably, AWS
ParallelCluster provides support for several schedulers, such as SGE and
Torque (which will both be discontinued at the end of 2021), as well as Slurm
and the in-house AWS Batch, which currently has limited support for GPU jobs.
We used Slurm for consistency across AWS and Cori systems.

Cori Haswell and KNL also use Slurm as workload manager. Cori has the Cray
Aries “Dragonfly” topology for its interconnect (NERSC, [n.d.]b). AWS does not
disclose details about the underlying interconnect topology, except for an
expected injection bandwidth (Table 1), and that the AWS C5 instances use
Amazon in-house Elastic Fabric Adapter (EFA) (Amazon Web Services, [n.d.]a,
[n.d.]b) as network interface. The AWS cluster instances belong to the same
placement group; the login node and the compute nodes belong to two different
subnets. A subnet is a logically visible subdivision of an IP network. The
subnetwork of compute nodes is private and has no access to the Internet.

###  3.2. A Hardware and System View

Figure 1. LINPACK peak performance compared to the theoretical peak (left)
using one node and all available cores per node (Table 1). CacheBench memcpy()
benchmark (right) using a single core and reporting the median of 10 runs.

Given different in-node configurations, we first investigate performance using
a microbenchmarking approach.

Processor. Figure 1 compares the LINPACK peak performance (left) with the
theoretical peak performance (right) for each platform and also allows cross-
platform comparison. Cori Haswell and AWS R5 achieve peak performance
significantly closer to their theoretical peak than the other two machines.
Closing the gap between theoretical peak and LINPACK peak on Cori KNL is
notoriously difficult; achieving such progress requires a significant
optimization effort for applications in general (Barnes et al., 2016; Doerfler
et al., 2016, 2018). Cori KNL achieves about 350 GFlops/second more in our
benchmark than the number reported in the Top500 (Top500, [n.d.]). This
discrepancy could be due to different implementations of the LINPACK
benchmark, since we use the Intel Math Kernel Library benchmark package.
Further profiling of AWS C5 with VTune (Reinders, 2005) revealed a relatively
low core utilization for this platform, which could explain the large gap
between theoretical and achieved peak.

The cloud instances perform best in absolute terms. AWS R5 and C5 instances
are equipped with newer hardware than Cori systems; this may explain the
greater processing power. It is noteworthy that the elastic nature of cloud
computing $-$ as opposed to multi-year projects to develop and install
supercomputers $-$ offers the potential for rapid hardware turnaround.

Memory Hierarchy. Figure 1 shows the results for the Cache-Bench benchmark (on
the right) and illustrates the performance of the cache hierarchy for our four
machines. For each platform and size, we ran the benchmark 10 times and report
the median; there is little variance among different runs for a given size and
platform.

Table 2. STREAM benchmark: as many OpenMP threads as the number of physical
cores per node (top) and one thread (bottom), 8 bytes per array element, array
size = 120000000 (elements), offset = 0 (elements), memory per array = 915.5
MiB, total memory required = 2746.6 MiB. The best time for each kernel over 10
runs (excluding the first iteration) is used to compute the bandwidth. Results
in GB/s (McCalpin, 1995).

Platform Threads Copy Scale Add Triad Cori Haswell 32 56.6 43.6 49.4 49.7 Cori
KNL 64 247.9 250.3 257.1 260.0 AWS r5dn.16xlarge 32 181.9 127.6 143.9 144.9
AWS c5.18xlarge 36 135.7 106.9 120.4 120.3 Cori Haswell 1 18.0 11.3 12.6 12.6
Cori KNL 1 12.1 6.8 8.4 7.4 AWS r5dn.16xlarge 1 11.1 12.5 13.2 13.1 AWS
c5.18xlarge 1 11.0 12.6 13.5 13.6

Cori Haswell has the best performance for L1 (which is the same size on all
machines). The L2 performance of Cori Haswell and AWS C5 are comparable, while
the performance of Cori Haswell falls below that of the AWS C5 platform below
its second cache level. AWS C5 achieves better performance than Cori Haswell
as long as the data fits into its L2 cache, and its performance falls below
Cori Haswell when it enters the third cache level as expected because Cori
Haswell has a larger L3 cache.

Considering the data in Table 1, one would expect a higher bandwidth for AWS
R5 and Cori KNL given their larger L2. However, the way the caches are shared
between the cores and cache associativity could affect overall memory
throughput. For Cori Haswell, L2 is private to each core, while for Cori KNL
it is shared by two cores. Cori KNL has two cache levels instead of three like
the other machines. Cori Haswell’s cache is $8$-way associative, while Cori
KNL has a direct mapped cache. This direct mapping reduces cache management
complexity, but can significantly increase cache thrashing, resulting in a
high rate of cache misses and main memory accesses (NERSC, [n.d.]a).

Looking only at these single core results, one might suspect that the
virtualization overhead could prevent cloud instances from fully exploiting
the potential of their caches. However, our results, which measure the
performance of the whole memory system, discourage this hypothesis, as shown
in the next microbenchmark.

Memory Bandwidth. To measure memory bandwidth when data does not fit in the
system cache, we run the STREAM benchmark (McCalpin, 1995). The results in
Table 2 show that the performance difference between Cori Haswell and AWS R5
and C5 (Figure 1) is reversed in favor of the cloud clusters when all
available cores are used if the data does not fit in the platforms’ caches.

Cori KNL has the higher memory bandwidth thanks to its on-chip multi-channel
DRAM (MCDRAM) chip of $16$GB. Looking at platforms without on-chip memory,
cloud instances show a significantly higher memory bandwidth than the
corresponding HPC platform. System age and newer cloud hardware can explain
this performance. These results suggest that a faster hardware turnaround time
could benefit not only computationally intensive applications, but also data-
intensive applications. In addition, these results discredit the hypothesis
that virtualization overhead is a major limitation of today’s cloud computing.

Inter-Node Communication. To study network performance, we measure bandwidth
and latency in a multinode setting. In our experiments, we use openmpi-4.0.2
as the MPI implementation. For Cori Haswell and KNL, we ran the benchmark
suite with both openmpi-4.0.2 and the default cray-mpi. They provided similar
performance, and we decided to report only the results for OpenMPI for clarity
and consistency with the cloud instances.

Figure 2 uses one process per node to show point-to-point bandwidth (left) and
latency (right). Our results show peak bandwidth of about $86$ Gbit/s for AWS
R5 and $90$ Gbit/s for AWS C5, while Cori Haswell and Cori KNL show peak
bandwidths of $74$ and $64$ Gbit/s, respectively. Considering that the two
Cori systems share the same network, one would expect the same network
performance, however, their performance in Figure 2 are significantly
different. This difference can be attributed to the overhead of MPI function
calls, which are expensive and penalise lower frequency Cori KNL cores that
cannot match the performance of Cori Haswell nodes. Our results are consistent
with those presented by GASNet (GASNet, [n.d.]).

Figure 2. OSU MPI microbenchmark injection bandwidth (left) and point-to-point
latency (right) in log-log scale. Using two nodes with one process per node on
Cori Haswell, Cori KNL, AWS R5, and AWS C5.  Figure 3. OSU MPI microbenchmark
MPI_Alltoall latency on two nodes (left) and on eight nodes (right) in log-log
scale. On Cori Haswell, AWS R5 and C5 we use 32 processes per node, while on
Cori KNL we use 64 processes per node.

Cloud instances outperform HPC systems in both bandwidth and latency. Until
recently, the lack of a low-latency network has been consistently identified
as the main bottleneck of cloud computing for scientific applications (Yelick
et al., 2011; Gupta et al., 2014; Netto et al., 2018). Our results show that
modern cloud computing has made significant advances in networking technology
that provide cloud instances with HPC-competitive network performance.

Figure 3 shows the MPI_Alltoall latency on $P{=}2,8$. For small message sizes,
Cori Haswell dominates the other platforms on two nodes; the gap decreases as
the number of nodes is increased. The differences between Cori Haswell and KNL
are due to the cost of MPI calls on the two different processors. Also, the
different number of processes per node in this experiment illustrates the
difference between the two Cori systems. On two nodes, the gap decreases as
the message size increases, especially when comparing Cori Haswell and AWS R5,
whose performance almost overlaps at large message sizes. AWS R5 shows similar
performance to Cori Haswell on eight nodes, except for small message sizes.
Looking only at the historical results, one would expect the cloud instances
to lose performance and the gap to grow as the number of nodes increases. On
the contrary, our results show significant improvements, so one can expect
better performance scaling as the number of nodes increases. AWS R5 performs
as we would expect given its performance in the previous microbenchmark, while
AWS C5 is far from Cori Haswell. Its advertised network bandwidth is about
$3\times$ lower than Cori Haswell and since it is a compute-optimized
instance, we suspect it may suffer from network contention.

Table 3.  Characterization of n-body using perf run on a single core. Page
size = $4$KB, problem size: $1$M.

Platform Instruction (G) Page Fault (K) Cache Miss (M) Time (s) Cori Haswell
414.7 367.2 11,347.8 461.7 Cori KNL 415.4 367.4 11,220.1 1,736.5 AWS
r5dn.16xlarge - 367.2 - 486.9 AWS c5.18xlarge 427.2 367.2 21,457.4 480.6

Table 4.  Characterization of FFT using perf run on a single core. Page size =
$4$KB, problem size: $50$K.

Platform Instruction (G) Page Fault (K) Cache Miss (M) Time (s) Cori Haswell
782.1 9,766.8 871.5 312.4 Cori KNL 784.9 9,766.8 20,915.0 2,348.1 AWS
r5dn.16xlarge - 9,767.5 - 303.3 AWS c5.18xlarge 1,097.9 9,766.6 2,953.6 335.8

Our results suggest that the place we would expect HPC to retain an advantage
is in applications with many small messages. Algorithmic techniques, however,
typically try to avoid this situation. These results have important
implications for communication-intensive applications that have not
historically benefited from cloud computing due to their bandwidth
requirements.

###  3.3. A User-Application View

In this section, we first measure and compare the serial runtime of the
applications and analyze the single-core performance of the applications to
better understand the runtime differences and similarities between the
machines. Then, we study the parallel performance of the applications in a
multinode environment.

####  3.3.1. Serial Performance

In Tables 3 and 4, we report the single core performance for the N-Body
simulation and the FFT, respectively. In both applications, Cori KNL has a
significantly higher runtime than the other machines. Its poor performance can
be justified by the lower frequency of its processor and the poor performance
of its memory system. Cori KNL’s clock speed is about half that of the other
cores in the study, and it needs all 68 of them to compete with the
(theoretical) GFlop rate of the other 32-36 core nodes. Recall that the L2
caches on Cori KNL are shared by two cores, while they are private on the
other machines. In fact, the performance for FFT is relatively worse since it
is a more memory intensive application than N-Body. Cori Haswell and the two
cloud instances show similar runtime for both applications. Cloud instances
have lower cache performance than Cori Haswell, while they have higher
bandwidth when data can no longer fit in the cache. Since we study single-core
performance here, the lower half of Table 2 shows that Cori Haswell and the
AWS instances have comparable performance in the single-core STREAM benchmark.

Overall, these results are consistent with the results of our microbenchmarks
and confirm that cloud virtualization overhead has decreased to a point where
application performance is not significantly impacted. As a result, cloud
instances have comparable runtime to a HPC system for both applications.

####  3.3.2. Workload Characterization

Recall, when we measure the runtime of an application, we measure both the
processor and the memory system. Runtime alone is not enough to get a
reasonable understanding of the variables that affect application performance.

Here, we extend our analysis by measuring the number of page faults,
instructions and cache misses for each application on each platform and
comparing the results. A high rate of page swapping-in/out, cache misses, and
a high number of instructions can significantly slow down applications
(Sherwood et al., 1999; Babka et al., 2009; Lam et al., 1991). On all systems,
these metrics are measured for a process on a single node using perf (Weaver,
2013). Cache misses and instructions are not available for AWS R5. In
particular, it is not easy to get access to accurate hardware counters. On HPC
systems they typically require administrative privileges, while on cloud
systems it can be difficult to separate the effects of virtualization and gain
access to accurate metrics.

s

Figure 4. The N-Body strong scaling with $1$M particles (left) and the FFT
strong scaling with $50$K points (right) across the machines. The number next
to the name in the legend indicates the number of processes per node.

Tables 3$-$4 give the number of page faults on the machines for the N-Body
simulation and the FFT. The number of page faults is mostly the same and
confirms the same behavior across the four machines. Cori Haswell and Cori KNL
automatically load a software package to increase the page size from $4$K to
$2$M. This setting was unloaded and disabled to allow a fair comparison
between the four machines. Similarly, the page size could have been increased
on the AWS instances. For simplicity, we chose to reduce the page size on Cori
and do not expect this setting to change the overall trend of our results. The
only significant difference in the number of instructions is between the Cori
systems and AWS C5 for the FFT. This difference could explain the runtime
difference between Cori Haswell and AWS C5, although it is not large.

Cache misses show a more relevant impact on performance than page faults and
instructions. The Cori systems have similar cache misses for N-Body
simulation, while they show a significant gap for FFT. Cori KNL’s direct
mapped cache significantly penalizes its performance for a memory-intensive
application such as FFT. AWS C5 has a larger number of cache misses than Cori
Haswell for both applications. This result, combined with AWS C5’s slower L1
(Figure 1), suggests that cache misses are one of the variables contributing
to the runtime difference between these two machines.

Our workload characterization reveals that cache misses and memory system
performance have the largest impact on single-core performance. Nevertheless,
the resulting runtime differences are small, and our analysis shows comparable
single-core performance between Cori Haswell and the cloud instances.

####  3.3.3. Parallel Performance

In examining parallel performance to highlight the effect of the network, we
report the median of $10$ runs of the application for $P{=}1,2,4,8$. Due to a
limit on the number of instances we can create simultaneously, we were unable
to get more than eight instances in the same placement group, which is
critical for achieving low-latency network performance. AWS support can
increase this limit upon request. Given the varying number of cores per node
of our machines, we normalize our results and specify the configuration that
provides the best performance for each platform.

N-Body Simulation. Figure 4 on the left illustrates the strong scaling
performance across the machine and shows the runtime split in computation and
communication. Our N-Body implementation uses a recursive doubling algorithm
for particle exchange and therefore runs much faster with the power of two
processes. As a result, all machines achieve their best performance with
either $32$ or $64$ processes per node.

The N-Body simulation is computationally intensive and has a low $Cm/Cp$
ratio, suggesting a modest impact of the network on overall runtime. Given the
$Cm/Cp$ ratio and the serial performance of this application, we expect
comparable runtimes between Cori Haswell and the two cloud instances. Cori KNL
also has comparable runtimes, while its $P{=}8$ scaling is significantly worse
than the other three machines. In particular, it uses twice as many processes
per node as the other machines and runs at about half the frequency, with
fewer cache levels and L2 caches shared by two cores.

The MPI_Alltoall microbenchmark shows a significant difference between the two
Cori systems. Remember that the Cori systems use the same network; however,
Cori KNL uses $64$ processes per node instead of $32$ and the MPI function
calls overload the weaker KNL cores. The gap is also significant between Cori
Haswell and AWS C5 at any scale, while the gap between Cori Haswell and AWS R5
is mostly overlapping.

The MPI_Alltoall gap between the two Cori systems is reflected in their
performance in the N-Body simulation. Therefore, one would expect AWS C5 to
have a larger communication time. Nonetheless, the performance of AWS C5 is
consistent with the assumption that when the $Cm/Cp$ ratio is low, the network
has a limited impact on the overall application runtime.

Overall, AWS R5 is the fastest platform, with Cori Haswell and AWS C5 having
the same performance at $P{=}8$. The remarkable comeback of Cori Haswell might
be due to fitting data into the larger L3 cache of Cori Haswell. The N-Body
simulation scales superlinearly $-$ on all machines except Cori KNL. We are
familiar with this implementation and know its superscaling behavior, which
can be briefly explained as the “cache effect”, meaning that as the number of
nodes increases, more data fits into the cache.

Our results confirm that cloud computing can be more suitable than HPC systems
for computationally intensive applications (Yelick et al., 2011) and that
modern cloud computing can provide competitive network performance to HPC.

Fast Fourier Transform. Figure 4 shows the strong scaling performance of the
FFT (right) and splits the runtime into computation and communication. The
library FFTW computes multiple FFTs and measures their execution times to find
the optimal plan that achieves the best performance for each machine. We use
these optimal implementations. Since the optimal plan selected by FFTW is
based on a collection of MPI_Sendrecvs, the results in Figure 2 are relevant
to the following analysis.

FFT has a higher $Cm/Cp$ ratio than the N-Body simulation, and as expected,
Figure 4 shows that the communication overhead is much higher than in the
previous application and can take more than $50\%$ of execution time. There is
a consistent spike in communication at $P{=}4$, which we suspect is due to
implementation details of FFTW. On all machines, the overall scaling of the
FFT is sublinear, mainly due to communication overhead. AWS R5 is the fastest
platform, both in terms of total and communication time. It is followed by
Cori Haswell. The computation times of Cori KNL and AWS C5 are comparable, but
AWS C5 has a higher communication overhead, making it the slowest platform in
this benchmark.

Despite comparable performance for point-to-point communication (Figure 2),
the cloud instances exhibit different performance for all processes on the
node involved in the communication (Figure 3). AWS C5 exhibits significantly
worse performance for the MPI_Alltoall benchmark, which explains the
difference in communication performance between the two AWS instances for the
FFT results (Figure 4). AWS R5 is optimized for memory-intensive workloads,
while AWS C5 is optimized for compute-intensive workloads. Moreover, AWS C5
uses Amazon’s in-house EFA interconnect, whose advertised bandwidth is
$3\times$ lower than R5’s. Our hypothesis is that as the number of processes
increases, the C5 interconnect is more subject to contention than R5’s
network.

AWS R5 is the best performing platform in this benchmark, as one would expect
based on the results of our microbenchmarks and workload characterization. The
communication time on AWS R5 is comparable to or even lower than that on HPC
systems. Thus, it is not only the newer processor that contributes to the high
performance for this application, but also the interconnect speed. Previous
literature has shown that FFTs for cloud instances have significantly lower
performance than for HPC systems. The Magellan report (Yelick et al., 2011)
describes the FFT as $4$ to $20\times$ slower than the HPC systems considered,
running on 8 processes per node and $P{=}8$. Our result is an important
validation of the recent advances that cloud computing has made in networking
technology to close the performance gap with HPC.

##  4\. Conclusions and Future Work

Our work investigated the performance gap between current HPC and cloud
computing systems to understand the nature of their differences and guide the
design of future cloud systems. In this work, we analyzed the cross-stack
performance, from single core compute power, to memory subsystem, inter-node
communication performance, and overall application performance.

In particular, we highlight that cloud computing can offer a greater variety
of hardware configurations and newer technology due to continuous procurement
cycles. If a study requires the latest technology or a particular memory size
and processor type, these are more likely to be available in the cloud, while
a given HPC system may offer only one or a small set of standardized resources
suitable for typical scientific applications. Our results contradict earlier
findings on cloud interconnects, namely that networks for HPC instances within
the cloud have improved to the point of providing competitive performance to
that of HPC systems at modest scales.

On the other hand, cloud policies can limit what is available within an HPC
cloud offering, e.g., one may need to make a request to the vendor to use more
than a few instances, and the latest node architectures may not be available
with the fast network. In contrast, in traditional HPC systems, the entire
system typically has the same network, whose performance is mostly determined
by the age of the system, as the procurement cycles are typically longer.

Our results showed that the compute and memory subsystem performance of cloud
instances is competitive with HPC systems. This is consistent with historical
results demonstrating cloud competitiveness for compute-dominated workloads.

Cloud systems offered higher bandwidth and lower latency than HPC systems for
point-to-point communication. In the FFT benchmark, which is bisection-
bandwidth limited, the performance of the compute-optimized cloud platform
dropped, possibly due to network contention, while the platform optimized for
memory-intensive applications significantly outperformed all other machines.
This represents a significant advance in cloud computing technology, as the
performance of multinode FFT applications on HPC systems has historically been
better than on cloud systems (Yelick et al., 2011). A larger scale performance
study focusing on machine balance would be an interesting future work to
analyze the gap on a larger scale.

Our work shows that today’s cloud computing can provide competitive
performance to HPC, not only for compute-intensive applications, but also on
memory- and communication-intensive workloads. The recent performance
improvements of cloud instances may be due to the increasing demands of deep
learning (Jouppi et al., 2020; Hazelwood et al., 2018), potentially benefiting
seemingly unrelated computational science as a byproduct. It is worth noting
that our study focused on one cloud provider and it would be important to
replicate the study on other providers to draw more generalized conclusions.
Given our results, an important future work would be a comparison focusing on
elasticity and resource management, which together with our results would
allow users to make informed decisions about which system is better suited for
their applications.

###### Acknowledgements.

This work is supported by the Advanced Scientific Computing Research (ASCR)
program within the Office of Science of the DOE under contract number DE-
AC02-05CH11231. We used resources of the NERSC supported by the Office of
Science of the DOE under Contract No. DEAC02-05CH11231. This research was also
supported by the Exascale Computing Project (17-SC-20-SC), a collaborative
effort of the U.S. Department of Energy Office of Science and the National
Nuclear Security Administration. AWS Cloud Credits provided through the AWS
Cloud Credits for Research program. Thanks to Rohan Bavishi, Rohan Padhye,
Neesha Zerin, and James Demmel for useful suggestions and valuable
discussions.

## References

  * (1)
  * Alam et al. (2006) Sadaf R Alam, Richard F Barrett, Jeffery A Kuehn, Philip C Roth, and Jeffrey S Vetter. 2006.  Characterization of scientific workloads on systems with multi-core processors. In _IEEE international symposium on workload characterization_. IEEE, 225–236.
  * Alexander et al. (2020) Francis Alexander, Ann Almgren, John Bell, Amitava Bhattacharjee, Jacqueline Chen, Phil Colella, David Daniel, Jack DeSlippe, Lori Diachin, Erik Draeger, et al. 2020\.  Exascale applications: skin in the game.  _Philosophical Transactions of the Royal Society A_ 378, 2166 (2020), 20190056.
  * Amazon Web Services ([n.d.]a) Amazon Web Services. [n.d.]a.  Amazon EC2 C5 Instances — Amazon Web Services (AWS).  https://aws.amazon.com/ec2/instance-types/c5/.
  * Amazon Web Services ([n.d.]b) Amazon Web Services. [n.d.]b.  Elastic Fabric Adapter — Amazon Web Services.  https://aws.amazon.com/hpc/efa/.
  * Asanovic et al. (2006) Krste Asanovic, Ras Bodik, Bryan Christopher Catanzaro, Joseph James Gebis, Parry Husbands, Kurt Keutzer, David A Patterson, William Lester Plishker, John Shalf, Samuel Webb Williams, and Katherine A. Yelick. 2006.  The landscape of parallel computing research: A view from Berkeley.  (2006).
  * Babka et al. (2009) Vlastimil Babka, Lukáš Marek, and Petr Tuma. 2009\.  When misses differ: Investigating impact of cache misses on observed performance. In _15th International Conference on Parallel and Distributed Systems_. IEEE, 112–119.
  * Barnes et al. (2016) Taylor Barnes, Brandon Cook, Jack Deslippe, Douglas Doerfler, Brian Friesen, Yun He, Thorsten Kurth, Tuomas Koskela, Mathieu Lobet, Tareq Malas, et al. 2016\.  Evaluating and optimizing the nersc workload on knights landing. In _2016 7th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS)_. IEEE, 43–53.
  * Buyya et al. (2009) Rajkumar Buyya, Chee Shin Yeo, Srikumar Venugopal, James Broberg, and Ivona Brandic. 2009.  Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility.  _Future Generation computer systems_ 25, 6 (2009), 599–616.
  * Colella (2004) Phillip Colella. 2004\.  Defining software requirements for scientific computing.
  * Cray ([n.d.]) Cray. [n.d.].  Cray XC Series Network.  https://www.cray.com/sites/default/files/resources/CrayXCNetwork.pdf.
  * Crovella et al. (1992) Mark Crovella, Ricardo Bianchini, Thomas LeBlanc, Evangelos Markatos, and Robert Wisniewski. 1992\.  Using communication-to-computation ratio in parallel program design and performance prediction. In _Proceedings of the Fourth IEEE Symposium on Parallel and Distributed Processing_. IEEE, 238–245.
  * Doerfler et al. (2018) Douglas Doerfler, Brian Austin, Brandon Cook, Jack Deslippe, Krishna Kandalla, and Peter Mendygral. 2018\.  Evaluating the networking characteristics of the Cray XC-40 Intel Knights Landing-based Cori supercomputer at NERSC.  _Concurrency and Computation: Practice and Experience_ 30, 1 (2018), e4297.
  * Doerfler et al. (2016) Douglas Doerfler, Jack Deslippe, Samuel Williams, Leonid Oliker, Brandon Cook, Thorsten Kurth, Mathieu Lobet, Tareq Malas, Jean-Luc Vay, and Henri Vincenti. 2016\.  Applying the Roofline Performance Model to the Intel Xeon Phi Knights Landing Processor. In _High Performance Computing_ , Michela Taufer, Bernd Mohr, and Julian M. Kunkel (Eds.). Springer International Publishing, Cham, 339–353.
  * Dongarra et al. (2003) Jack J Dongarra, Piotr Luszczek, and Antoine Petitet. 2003\.  The LINPACK benchmark: past, present and future.  _Concurrency and Computation: practice and experience_ 15, 9 (2003), 803–820.
  * Ellis et al. (2017) Marquita Ellis, Evangelos Georganas, Rob Egan, Steven Hofmeyr, Aydın Buluç, Brandon Cook, Leonid Oliker, and Katherine Yelick. 2017\.  Performance characterization of de novo genome assembly on leading parallel systems. In _European Conference on Parallel Processing_. Springer, 79–91.
  * Ellis et al. (2019) Marquita Ellis, Giulia Guidi, Aydın Buluç, Leonid Oliker, and Katherine Yelick. 2019\.  diBELLA: Distributed Long Read to Long Read Alignment. In _Proceedings of the 48th International Conference on Parallel Processing_. 1–11.
  * Evangelinos and Hill (2008) Constantinos Evangelinos and Chris Hill. 2008.  Cloud computing for parallel scientific HPC applications: Feasibility of running coupled atmosphere-ocean climate models on Amazon’s EC2.  _CCA-08_ 2, 2.40 (2008), 2–34.
  * Fox et al. (2009) Armando Fox, Rean Griffith, Anthony Joseph, Randy Katz, Andrew Konwinski, Gunho Lee, D Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. 2009\.  Above the clouds: A Berkeley view of cloud computing.  _Dept. Electrical Eng. and Comput. Sciences, University of California, Berkeley, Rep. UCB/EECS_ 28, 13 (2009).
  * Frigo (1999) Matteo Frigo. 1999\.  A fast Fourier transform compiler. In _Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation_. 169–180.
  * Frigo and Johnson (1997) Matteo Frigo and Steven G Johnson. 1997.  _The fastest fourier transform in the west_.  Technical Report. Massachusetts Institute of Technology, Cambridge.
  * Frigo and Johnson (1998) Matteo Frigo and Steven G Johnson. 1998.  FFTW: An adaptive software architecture for the FFT. In _Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP’98 (Cat. No. 98CH36181)_ , Vol. 3. IEEE, 1381–1384.
  * GASNet ([n.d.]) GASNet. [n.d.].  GASNet-EX Performance Examples.  https://gasnet.lbl.gov/performance/.
  * Gupta et al. (2014) Abhishek Gupta, Paolo Faraboschi, Filippo Gioachin, Laxmikant V Kale, Richard Kaufmann, Bu-Sung Lee, Verdi March, Dejan Milojicic, and Chun Hui Suen. 2014.  Evaluating and improving the performance and scheduling of HPC applications in cloud.  _IEEE Transactions on Cloud Computing_ 4, 3 (2014), 307–321.
  * Hazelwood et al. (2018) Kim Hazelwood, Sarah Bird, David Brooks, Soumith Chintala, Utku Diril, Dmytro Dzhulgakov, Mohamed Fawzy, Bill Jia, Yangqing Jia, Aditya Kalro, et al. 2018\.  Applied machine learning at facebook: A datacenter infrastructure perspective. In _2018 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 620–629.
  * He et al. (2010) Qiming He, Shujia Zhou, Ben Kobler, Dan Duffy, and Tom McGlynn. 2010.  Case study for running HPC applications in public clouds. In _Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing_. 395–401.
  * Ida and Makino (1992) Shigeru Ida and Junichiro Makino. 1992.  N-Body simulation of gravitational interaction between planetesimals and a protoplanet: I. velocity distribution of planetesimals.  _Icarus_ 96, 1 (1992), 107–120.
  * Jouppi et al. (2020) Norman P Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. 2020\.  A domain-specific supercomputer for training deep neural networks.  _Commun. ACM_ 63, 7 (2020), 67–78.
  * Lam et al. (1991) Monica D Lam, Edward E Rothberg, and Michael E Wolf. 1991\.  The cache performance and optimizations of blocked algorithms.  _ACM SIGOPS Operating Systems Review_ 25, Special Issue (1991), 63–74.
  * McCalpin (1995) John D McCalpin. 1995\.  STREAM benchmark.  22 (1995).  http://www.cs.virginia.edu/stream/ref.html
  * Mell and Grance (2011) Peter Mell and Tim Grance. 2011.  The NIST definition of cloud computing.  (2011).
  * Mucci ([n.d.]) Philip J. Mucci. [n.d.].  LLCbench Home Page.  http://icl.cs.utk.edu/llcbench/index.htm.
  * Mushtaq et al. (2017) Hamid Mushtaq, Frank Liu, Carlos Costa, Gang Liu, Peter Hofstee, and Zaid Al-Ars. 2017\.  Sparkga: A spark framework for cost effective, fast and accurate dna analysis at scale. In _Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics_. 148–157.
  * NERSC ([n.d.]a) NERSC. [n.d.]a.
  * NERSC ([n.d.]b) NERSC. [n.d.]b.  Interconnect - NERSC Documentation.  https://docs.nersc.gov/systems/cori/interconnect/.
  * Netto et al. (2018) Marco AS Netto, Rodrigo N Calheiros, Eduardo R Rodrigues, Renato LF Cunha, and Rajkumar Buyya. 2018\.  HPC cloud for scientific and business applications: Taxonomy, vision, and research challenges.  _ACM Computing Surveys (CSUR)_ 51, 1 (2018), 1–29.
  * Panda (2018) Dhabaleswar K. Panda. 2018\.  OSU Micro-Benchmarks.
  * Reinders (2005) James Reinders. 2005\.  VTune performance analyzer essentials.  _Intel Press_ (2005).
  * Sherwood et al. (1999) Timothy Sherwood, Brad Calder, and Joel Emer. 1999.  Reducing cache misses using hardware and software page placement. In _Proceedings of the 13th international conference on Supercomputing_. 155–164.
  * Top500 ([n.d.]) Top500. [n.d.].  Cori - Cray XC40, Intel Xeon Phi 7250 68C 1.4GHz, Aries interconnect — TOP500.  https://www.top500.org/system/178924/.
  * Tsouloupas and Dikaiakos (2006) George Tsouloupas and Marios D Dikaiakos. 2006.  Characterization of computational grid resources using low-level benchmarks. In _Second IEEE International Conference on e-Science and Grid Computing_. IEEE, 70–70.
  * Weaver (2013) Vincent M Weaver. 2013\.  Linux perf_event features and overhead. In _The 2nd International Workshop on Performance Analysis of Workload Optimized Systems, FastPath_ , Vol. 13.
  * Woo et al. (1995) Steven Cameron Woo, Moriyoshi Ohara, Evan Torrie, Jaswinder Pal Singh, and Anoop Gupta. 1995\.  The SPLASH-2 programs: Characterization and methodological considerations.  _ACM SIGARCH computer architecture news_ 23, 2 (1995), 24–36.
  * Yelick et al. (2020) Katherine Yelick, Aydın Buluç, Muaaz Awan, Ariful Azad, Benjamin Brock, Rob Egan, Saliya Ekanayake, Marquita Ellis, Evangelos Georganas, Giulia Guidi, Steven Hofmeyr, Oguz Selvitopi, Cristina Teodoropol, and Leonid Oliker. 2020.  The parallelism motifs of genomic data analysis.  _Philosophical Transactions of the Royal Society A_ 378, 2166.
  * Yelick et al. (2011) Katherine Yelick, Susan Coghlan, Brent Draney, and Richard S. Canon. 2011.  The Magellan Report on Cloud Computing for Science.  _US Department of Energy, Office of Science, Office of Advanced Scientific Computing Research (ASCR)_ 3\.
  * Yoo et al. (2003) Andy B Yoo, Morris A Jette, and Mark Grondona. 2003\.  Slurm: Simple Linux utility for resource management. In _Workshop on Job Scheduling Strategies for Parallel Processing_. Springer, 44–60.