# Composable Design of Multiphase Fluid Dynamics Solvers in Flash-X

Akash Dhruv<EMAIL_ADDRESS>Mathematics and Computer Science Division,
Argonne National Laboratory, Lemont, IL, USA

###### Abstract

Multiphysics incompressible fluid dynamics simulations play a crucial role in
understanding intricate behaviors of many complex engineering systems that
involve interactions between solids, fluids, and various phases like liquid
and gas. Numerical modeling of these interactions has generated significant
research interest in recent decades and has led to the development of open
source simulation tools and commercial software products targeting specific
applications or general problem classes in computational fluid dynamics. As
the demand increases for these simulations to adapt to platform heterogeneity,
ensure composability between different physics models, and effectively utilize
inheritance within partial differentiation systems, a fundamental
reconsideration of numerical solver design becomes imperative. The discussion
presented in this paper emphasizes the importance of these considerations and
introduces the Flash-X approach as a potential solution. The software design
strategies outlined in the article serve as a guide for Flash-X developers,
providing insights into complexities associated with performance portability,
composability, and sustainable development. These strategies provide a
foundation for improving design of both new and existing simulation tools
grappling with these challenges. By incorporating the principles outlined in
the Flash-X approach, engineers and researchers can enhance the adaptability,
efficiency, and overall effectiveness of their numerical solvers in the ever-
evolving field of multiphysics simulations.

###### keywords:

Multiphysics Simulations, Software Design and Architecture, Composability,
Reproducibility, High Performance Computing

##  1 Introduction

The recent advancements in High Performance Computing (HPC) have significantly
accelerated progress in multiscale fluid dynamics simulations. Previously,
simulations for phase-change heat transfer, which took a week on leadership
supercomputers, can now be completed within hours, thanks to cutting-edge
libraries developed under the Exascale Computing Project (ECP) [17, 44, 14,
30]. As we approach an era dominated by Artificial Intelligence (AI), Machine
Learning (ML), and remarkable developments in specialized software/hardware
infrastructure, it is imperative that simulation tools transition from a
monolithic software architecture (where all features are packed into one
source code) to a more flexible approach. This involves leveraging and
combining different libraries to address complex and intriguing problems, and
adopting a more strategic approach towards software development that exploits
inheritance and design patterns within the mathematical system.

Computational Fluid Dynamics (CFD), which is at the forefront of computational
mechanics in utilizing large-scale computational resources to tackle problems
of increasing size and complexity, can greatly benefited from this design
philosophy. Direct Numerical Simulation (DNS), for example, which are high-
fidelity, three-dimensional, time-dependent computations resolving all scales
of motion down to the Kolmogorov scale, have provided valuable information on
the fundamental physics of near wall turbulence [25], transition [41], natural
convection [4], and noise reduction in free shear flows [23], just to name a
few. In most cases, to enhance the range of scales that is captured (i.e.
achieve Reynolds Number (Re) that is as close as possible to practical
applications), specialized in-house solvers tuned to the particular
configuration are utilized. These tools cannot be directly extended to more
complex configurations of practical interest. In addition, rapid changes in
hardware architectures require constant development/optimization of these
solvers, which is usually beyond the capabilities of small, single Principal
Investigator (PI) groups.

A common problem with legacy CFD codes passed down from PIs to their students
is the complexity of software design. This complexity is a symptom of inherent
issues associated with waterfall programming model that has largely been the
norm for university research laboratories, focusing on functionality of the
code rather than software design. This typically starts at inception, where a
student or a PI makes software design decisions related to a specific problem
they are trying to simulate and freeze the code structure, not allowing any
flexibility. When it is time to add more functionality to simulate a different
problem, new software developments are implemented in form of patches around
the original design contributing towards unsustainability. This approach is
known as tactical programming and leads to an explosion of complexity. Agile,
or strategic programming, on the other hand, simplifies design complexity by
breaking down the software into small units, commonly referred to as modules.
Each module encompasses/hides a complex software implementation, and can be
used as a building block for a larger application. The focus is on developing
a clean design where new developments can be easily implemented in the form of
new modules, improving scalability and extensibility. Fig 1 shows a comparison
between the strategic and tactical approach that highlights the benefits of
strategic programming over time [31].

Figure 1: Comparison of progress over time using strategic versus tactical
programming approach. Tactical approach improves productivity in the
beginning, but over time it introduces complexity within the software
architecture that slows down progress. Using an agile programming methodology
and following the strategic approach improves productivity over time. The
figure is inspired from [31].

Poorly designed research code can impede productivity for both students and
experienced researchers, consuming valuable time as they navigate the
intricacies of the code rather than efficiently applying or extending it to
different scientific inquiries. The lack of modularity, inadequate
documentation, absence of version control, inconsistent code style, limited
testing, platform dependencies, inefficient algorithms, and inadequate error
handling collectively contribute to this hindrance, underscoring the
importance of prioritizing good software engineering practices to enhance code
adaptability and overall usability across various use cases and hardware
platforms. It is not surprising that research software engineering is
increasingly becoming an important skill for any aspiring research
professional in the field of computational engineering.

The technology industry has long embraced agile software design principles,
drawing on the extensive expertise of software engineering teams. In
scientific computing research, the need for agile design arises as PIs
collaborate and merge their codebases. This collaboration often requires a
complete redesign, and within this context, an open-source philosophy provides
significant advantages, accelerating and scaling research. If a group of PIs,
along with software development experts from national laboratories,
collaborates on the same codebase, they can replicate the productivity evident
in the technology industry.

Scalable, open-source CFD solvers with a broader applicability have been
steadily gaining traction within the fluid dynamics community. Nek5000 [1], a
spectral element solver designed for intricate geometrical configurations,
serves as an exemplary case. It has been extensively applied to address
turbulent and transitional fluid flow and heat transfer problems across
diverse areas, from thermal hydraulics [42] to wind energy [32]. Another
instance is PyFR, an open-source Python-based framework for streaming
architectures applicable to compressible flows [18]. It is crafted for mixed
unstructured grids containing various element types and targets a spectrum of
hardware platforms through the use of an in-built domain-specific language
derived from the Mako templating engine.

The widespread adoption of solvers, including those mentioned above, as well
as others like OpenFOAM [2], Phasta [3, 28, 34], and Fire Dynamics Simulator
(FDS) [38], signifies a paradigm shift within the broader fluid dynamics
community. Despite the growing popularity, this community has been hesitant to
embrace open-source development compared to other fields. This hesitancy stems
from concerns related to composability, reproducibility, software quality
maintenance, and complexities within the software infrastructure, making it
challenging for them to extend to new applications and algorithms. The Flash-X
team has dedicated substantial efforts over the past three years to tackle
these challenges and enhance the robustness of the ecosystem for scientific
software development and use. This article serves as an overview of this
intricate system, offering a starting point for future Flash-X developers to
familiarize themselves with the tools and practices established over time.

Many physics and engineering problems of interest involve interfaces between
different types of fluids, such as gases and liquids, coexisting with complex
geometrical configurations and moving or deforming solid boundaries.
Effectively capturing these diverse interactions within an application demands
a rigorous software development approach, an aspect that has been lacking in
the field of scientific computing. FLASH, a community code for fluid dynamics,
astrophysics, and magnetohydrodynamics simulations [15], has played a crucial
role in this field for the past 24 years. It laid the groundwork for a
versatile computational software framework that seamlessly integrates Adaptive
Mesh Refinement (AMR) with various Partial Differential Equation (PDE)-based
physics problems. Over time, FLASH has evolved to encompass a broader range of
applications. It was expanded to model single-phase fluid-structure
interaction [37, 39] and multiphase pool boiling simulations [10, 11]. Recent
developments have led to the creation of Flash-X [17], involving a substantial
redesign to enhance performance and ensure compatibility with modern hardware
architectures [16, 5]. This showcases Flash-X’s adaptability and commitment to
remaining pertinent in the ever-evolving landscape of scientific computing.

The discussion presented in this work centers on the software design of a
subset of Flash-X applications dedicated to modeling engineering problems
involving interactions between gas, liquid, and moving/stationary solids [8].
Examples encompass a range of scenarios, including convective air-cooling of
electronic devices [43], boiling-based cooling of machine components [22],
air-water dynamics during the breaking of surface waves [29], solidification-
melt dynamics in metal casting [20], liquid-gas chemical reactions in carbon
capture [21], cold plasma-based treatment of cancer cells [27], and more. In
these applications, the interplay between interface dynamics and surrounding
fluid motion is intricate. Factors such as density and viscosity ratios,
temperature jumps across the interface, surface-tension effects, and boundary
conditions significantly influence the dynamics. These complexities are
captured through interface-resolving DNS, relying on the implicit
representation of the interface, whose motion is tracked using a level-set
function. We intentionally avoid delving into the mathematical and numerical
formulation, providing references when necessary. Instead, our focus is on
outlining details of the object-oriented design of physics and infrastructure
units that leverages directory-based inheritance within the source code to
allow for composability and extensibility.

The paper is structured as follows: Section 2 provides and overview of the
incompresible multiphase fluid dynamics implementation in Flash-X along with
example applications. Section 3 focuses on details of the software design to
serve as a guideline for Flash-X users and developers. Section 4 focuses on
tools and practices that we been developed around Flash-X for software
maintenance and reproduciblity and finally we provide conclusion and the
direction of future work.

##  2 Multiphase Fluid Dynamics System in Flash-X

Flash-X employs a staggered-grid, finite-difference, fractional-step
projection method to solve incompressible fluid dynamics equations. These
equations, derived from the Reynolds transport theorem for momentum, mass, and
energy, are applicable to single-phase, multiphase, and phase-change heat
transfer problems. The computations take place on a block-structured Cartesian
grid, where each block $\Omega$ is treated as a data structure within the
software design context. A block is composed of cells, further divided into
points across three spatial dimensions. Partial differential equations are
solved over time and space within these cells. The schematic in Figure 2
illustrates a multiphase computational scenario, where a solid sphere
$\lambda$ falls under gravity $g$, disturbing the free surface between air and
water from time $t_{1}$ to $t_{3}$. $\lambda$ consists of triangular nodes
treated as separate data structures. $\Omega$ and $\lambda$ represent distinct
spatial reference frames, Eulerian and Lagrangian, respectively. Blocks
$\Omega$ can exist at different resolutions, as demonstrated at time $t_{3}$
in Figure 2, where the resolution of $\Omega_{2}$ is higher than that of
$\Omega_{1}$, implying smaller cell sizes in the former. Different resolutions
within an Adaptive Mesh Refinement (AMR) grid enhance the accuracy of
numerical solutions, particularly in regions of interest like the air-water
interface. This interface is represented using a signed level-set distance
function $\phi$, which is positive in air, negative in liquid, and zero at the
implicit location of the interface.

Figure 2: Schematic of an example problem in Flash-X where a solid sphere,
$\lambda$, disturbs the air-water free surface from $t_{1}$ to $t_{3}$ as it
falls under the influence of gravity, $g$. The sphere is represented using a
collection of Lagrangian triangles, and the air-water interface is defined
using a signed level-set distance function, $\phi$, which is greater than zero
in air ($\phi^{+}$) and less than zero in water ($\phi^{-}$). An example of
Adaptive Mesh Refinement (AMR) is shown at $t_{3}$. AMR is applied at
$\phi=0$, which corresponds to the implicit location of the interface.
$\Omega_{1}$ and $\Omega_{2}$ represent two different resolutions for a block,
$\Omega$, which contains a collection of cells with a staggered arragement of
Cartesian points on faces ($\times$) and cell-centers ($\bullet$). This
collection is knows as a Stencil, $S$ where partial differentiation equations
are solved.

.1 Flash-X/source. .2 Driver. .2 Grid. .2 IO. .2 Particles. .2
RuntimeParameters. .2 Simulation. .2 numericalTools. .3 Stencils. .2 physics.
.3 IncompNS. .3 HeatAD. .3 Multiphase. .3 ImBound. .3 SolidMechanics.

Figure 3: Top-level overview of Flash-X source code for multiphase fluid
dynamics applications.

An essential design pattern to highlight is the stencil, denoted as $S$. It is
characterized by a staggered arrangement of face ($\times$) and cell
($\bullet$) center points. Vector quantities are positioned at face centers,
while scalars are situated at cell centers. Consider the following example of
a partial differentiation equation representing the transport of a physical
quantity, $Q$ (which can be either a vector or a scalar), through space and
time,

$\begin{split}\mathchoice{\leavevmode\hbox to42.35pt{\vbox
to15.83pt{\pgfpicture\makeatletter\hbox{\hskip 21.17293pt\lower-3.47221pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-21.17293pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{magenta!17}$\displaystyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\textstyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\scriptstyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\scriptscriptstyle\mathstrut\frac{\partial
Q}{\partial t}$}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to42.35pt{\vbox to15.83pt{\pgfpicture\makeatletter\hbox{\hskip
21.17293pt\lower-3.47221pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-21.17293pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{magenta!17}$\displaystyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\textstyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\scriptstyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\scriptscriptstyle\mathstrut\frac{\partial
Q}{\partial t}$}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to31.64pt{\vbox to11.08pt{\pgfpicture\makeatletter\hbox{\hskip
15.82101pt\lower-2.43054pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-15.82101pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{magenta!17}$\displaystyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\textstyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\scriptstyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\scriptscriptstyle\mathstrut\frac{\partial
Q}{\partial t}$}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to24.51pt{\vbox to7.92pt{\pgfpicture\makeatletter\hbox{\hskip
12.25307pt\lower-1.7361pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-12.25307pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{magenta!17}$\displaystyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\textstyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\scriptstyle\mathstrut\frac{\partial
Q}{\partial
t}$}}{\hbox{\pagecolor{magenta!17}$\scriptscriptstyle\mathstrut\frac{\partial
Q}{\partial t}$}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}-\mathchoice{\leavevmode\hbox
to95.89pt{\vbox to9.09pt{\pgfpicture\makeatletter\hbox{\hskip
47.9474pt\lower-1.94443pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-47.9474pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{teal!17}$\displaystyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\textstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\scriptstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\scriptscriptstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to95.89pt{\vbox to9.09pt{\pgfpicture\makeatletter\hbox{\hskip
47.9474pt\lower-1.94443pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-47.9474pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{teal!17}$\displaystyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\textstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\scriptstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\scriptscriptstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to77.64pt{\vbox to8.51pt{\pgfpicture\makeatletter\hbox{\hskip
38.81938pt\lower-1.36108pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-38.81938pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{teal!17}$\displaystyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\textstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\scriptstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\scriptscriptstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to65.47pt{\vbox to8.12pt{\pgfpicture\makeatletter\hbox{\hskip
32.73404pt\lower-0.9722pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-32.73404pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{teal!17}$\displaystyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\textstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\scriptstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}{\hbox{\pagecolor{teal!17}$\scriptscriptstyle\mathstrut\vec{u}\cdot\nabla{Q}$}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}=\mathchoice{\leavevmode\hbox
to97.7pt{\vbox to10.58pt{\pgfpicture\makeatletter\hbox{\hskip
48.84882pt\lower-1.94443pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-48.84882pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{red!17}$\displaystyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\textstyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\scriptstyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\scriptscriptstyle\mathstrut
K\nabla^{2}Q$}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to97.7pt{\vbox to10.58pt{\pgfpicture\makeatletter\hbox{\hskip
48.84882pt\lower-1.94443pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-48.84882pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{red!17}$\displaystyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\textstyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\scriptstyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\scriptscriptstyle\mathstrut
K\nabla^{2}Q$}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to70.39pt{\vbox to7.41pt{\pgfpicture\makeatletter\hbox{\hskip
35.1941pt\lower-1.3611pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-35.1941pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{red!17}$\displaystyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\textstyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\scriptstyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\scriptscriptstyle\mathstrut
K\nabla^{2}Q$}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to52.18pt{\vbox to5.29pt{\pgfpicture\makeatletter\hbox{\hskip
26.09099pt\lower-0.9722pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-26.09099pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{red!17}$\displaystyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\textstyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\scriptstyle\mathstrut
K\nabla^{2}Q$}}{\hbox{\pagecolor{red!17}$\scriptscriptstyle\mathstrut
K\nabla^{2}Q$}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}+\mathchoice{\leavevmode\hbox
to19.11pt{\vbox to8.89pt{\pgfpicture\makeatletter\hbox{\hskip
9.5556pt\lower-1.94444pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-9.5556pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{blue!17}$\displaystyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\textstyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\scriptstyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\scriptscriptstyle\mathstrut f$}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to19.11pt{\vbox to8.89pt{\pgfpicture\makeatletter\hbox{\hskip
9.5556pt\lower-1.94444pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-9.5556pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{blue!17}$\displaystyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\textstyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\scriptstyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\scriptscriptstyle\mathstrut f$}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to13.38pt{\vbox to6.22pt{\pgfpicture\makeatletter\hbox{\hskip
6.6889pt\lower-1.3611pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-6.6889pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{blue!17}$\displaystyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\textstyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\scriptstyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\scriptscriptstyle\mathstrut f$}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to9.56pt{\vbox to4.44pt{\pgfpicture\makeatletter\hbox{\hskip
4.77777pt\lower-0.97221pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.77777pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\definecolor{currentcolor}{rgb}{0,0,0}\mathchoice{\hbox{\pagecolor{blue!17}$\displaystyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\textstyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\scriptstyle\mathstrut
f$}}{\hbox{\pagecolor{blue!17}$\scriptscriptstyle\mathstrut f$}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\end{split}$ (1)
Temporal EvolutionAdvectionDiffusionSource Terms

Equation 1 is derived from the Reynolds transport theorem and identifies
various continuum forces acting on $Q$. Temporal evolution denotes changes
over time. Advection forces arise from the transport induced by the velocity
$u$, while the diffusion force is attributed to variations in the
concentration of $Q$ over space and is a multiple of the diffusion coefficient
$K$. The term $f$ signifies the impact of additional source terms, such as
gravity, pressure gradients, and other body forces. The operators for each
force can be generalized over the stencil, $S$, and become separate design
patterns. Effective software design can be achieved by exploiting these design
patterns and the data structures $\lambda$ and $\Omega$ discussed above.

Figure 4: Venn diagram of multiphysics categories that describe interactions
involved in practical engineering problems

The Flash-X source code organization provides a solid foundation for
incorporating software design elements for $\lambda$, $\Omega$, and $S$. The
codebase is derived from FLASH which was extensively developed by Vanella [37,
39], Delaney [6], Wang [40] for multiphase and fluid-structure interaction
problems. While the design of the source code will be discussed at length in
the following sections, a top-level overview is provided in Figure 3.
Directories under Flash-X/source are termed as units, each containing code
pertinent to its functionality. For instance, the Grid and Particles units
contain code relevant to the management of $\Omega$ and $\lambda$,
respectively. The Stencils unit under numericalTools manages various stencils
related to advection, diffusion, and temporal integration in Equation 1. The
Driver unit manages the parallel execution environment and the
integration/evolution of various physics components, while the IO oversees
read and write functionality for data. Definition and management of execution
time parameters is handled by RuntimeParameters unit.

The physics units contains specific algorithms for solving various
mathematical models,

  * 1.

IncompNS: Routines for solving incompressible Navier-Stokes equations, both
with constant and variable coefficients, to compute solutions for velocity and
pressure.

  * 2.

HeatAD: Routines for solving heat diffusion and advection equations,
accommodating both constant and variable coefficients, to compute solutions
for temperature.

  * 3.

Multiphase: Routines for handling multiphase fluid properties and boundary
conditions on physical variables, with and without evaporation. These routines
are designed for solving the level-set function $\phi$ and setting source
terms for velocity and temperature.

  * 4.

ImBound: Modeling effects of the solid, $\lambda$, on continuum fluid.

  * 5.

SolidMechanics: Modeling effects of fluid and other body forces on $\lambda$.

Figure 5: An example of a target problem. (a) Flow boiling over a heat-
exchanger (b) Schematic of the problem over a rough heater surface (c) Zoomed-
in view of microlayer (d) Textured heater surface.

These physics units are combined under the special unit Simulation to generate
different application configurations. Figure 4 provides an overview of
application configurations built using a combination of different solvers
discussed above. A broad classification of these applications can be described
under fluid-solid interactions, fluid-fluid interactions, and scalar
transport, typically involving heat transfer and/or species
chemistry/evolution. Practical engineering simulations often utilize two or
more of these units to account for interactions occurring at similar or
different length and time scales. Flash-X brings all these capabilities
together to create a powerful software framework that can be utilized for
different types of fluid dynamics simulations. The details of numerical
implementation that corresponding to the discussion in this article are
documented in [9, 8, 10, 11].

Consider the problem of boiling in a heat exchanger which lies at the center
of the Venn diagram in Fig 4. Boiling is the most efficient way of cooling
machine components on earth, and is extensively used in internal combustion
engines, industrial power plants, and nuclear reactors. Figure 5a shows a
section of a liquid cooled heat-exchanger of length $\sim 1$ m, which is
representative of the scale of industrial components. Due to high temperature
on the heated surface the liquid undergoes phase change, enabling formation of
vapor bubbles and leading to boiling (Figure 5b). The process of evaporation
occurs at the solid-liquid-gas triple contact line, near the heater surface,
and within the thermal boundary layer of thickness, $\delta_{thermal}\sim
10^{-4}$ m, and microlayer of thickness, $\delta_{micro}\sim 10^{-6}$ m. The
former consists of superheated liquid which by definition has temperature
greater than the saturation temperature ($T_{liq}>T_{sat}$). The temperature
of the superheated liquid, $T_{liq}$, is dependent on heat transfer through
the heater wall, whereas the saturation temperature, $T_{sat}$, is based on
fluid-vapor properties. An enlarged view is shown in Figure 5c which
highlights the microlayer which is comprised of liquid that gets trapped below
the bubble base during rapid growth. The topology of the heater surface with
the length scale of surface textures, $l~{}\sim 10^{-5}$m, also affects
evaporation by exerting wall adhesion force on the bubbles and through
conjugate heat transfer with the fluids (see Figure 5d).

Figure 6: (a) Demonstration of metal casting
https://www.youtube.com/@LAMachines (b) Schematic of the problem.

The disparity of scales that exist in the example problem of boiling has
presented challenges in scientific understanding of the process. Experimental
investigations undertaken in the past have been helpful in providing an
understanding of marcoscale phenomena like scaling of average wall heat flux
and bubble dynamics based on gravity and heater temperature [22]. It has also
enabled calculation of quantitative data for growth and departure during
single bubble nucleate boiling [24, 7], and paved development of heuristic and
data-driven models for nucleation site distribution and microlayer
contribution [33, 35, 36]. However, fundamental knowledge relating to events
associated with bubble dynamics (sliding, mergers, departure, etc.), and their
relationship to fluid flow and heat transfer is still lacking, which can be
acquired by running simulations.

This is the case with many problems described in Fig 4. The problem of
impinging liquid jet during metal casting is another example where complexity
is introduced through scale and physical dynamics of the process. Fig 6
provides a schematic of the problem which shows a liquid metal jet falling on
a pool of liquid metal due to gravity. The jet is laminar in air and becomes
turbulent inside the pool as described by the vorticity contours in Fig 6b.
Due to this sudden change in dynamics, disturbances are created at the air-
liquid free surface which lead to entrainment of air bubbles. Furthermore,
chemical reactions that occur at liquid-air interface (at free surface and
near bubbles) lead to formation of metal-oxide impurity, which makes the metal
brittle after solidification. The amount of metal-oxide formed during this
process depends on nozzle design, shape of the liquid metal jet in air, and
impact velocity at the free surface, whose relationship with one another is
difficult to quantify experimentally [20].

For the class of problems discussed above and others shown in Fig 4, numerical
simulations become imperative for scientific discovery. They can provide high-
fidelity spatio-temporal data, which can be used for data analysis and the
discovery of new physics. The open-source development of Flash-X for modeling
these problems serves as a significant incentive for the broader research
community.

##  3 Software Design and Composability

In this section, we provide a more detailed description of the directory-based
inheritance that allows for composability shown in Figure 4. In the simplest
terms, directory-based inheritance implies that a directory tree starting from
Flash-X/source to a target directory within a unit becomes a collection of
node objects which inherit source code files and functionality from their
parent node. To illustrate, refer to Figure 7, which provides details for the
infrastructure units–Grid, Driver, and IO. When an application requests a
specific directory, such as Grid/GridMain/AMR/Amrex/Incomp, all the files
along this directory tree are unpacked into Flash-X/object directory. The
files in the child node replace or supersede the files from the parent node.

Each node contains a Config file written in a Domain-Specific Language (DSL),
parsed by a special setuptool to create Flash-X/object (an instance of the
Flash-X/source). Links to source files under Flash-X/object can be compiled to
build an executable for one of many simulations in Figure 4. Details of the
DSL for Config files and the usage of setuptool have been extensively
documented in the user’s guide (https://flash-x.org/pages/documentation).

.1 Flash-X/source. .2 Grid. .3 GridMain/AMR. .4 Amrex. .5 Incomp. .4
Paramesh4. .5 Incomp. .3 GridSolvers. .4 Grid_solvePoisson.F90.

.1 Flash-X/source. .2 Driver. .3 DriverMain. .4 Driver_initParallel.F90. .4
Driver_initAll.F90. .4 Driver_finalizeAll.F90. .4 Incomp. .5
Driver_evolveAll.F90.

.1 Flash-X/source. .2 IO. .3 IOMain/hdf5. .4 serial. .5 AM. .5 PM. .4
parallel. .5 AM. .5 PM.

Figure 7: Organization of infrastructure units for Grid, Driver, and I/O. The
highlighted boxes correspond to the units/source files required for simulation
shown in Figure 5.

Grid/GridMain/AMR unit provides interface to two mutually exclusive AMR
packages–AMReX [44] and custom implementation of Paramesh [26], each
containing special directories Incomp for customizations related to
incompressible Navier-Stokes formulation. The Grid/GridSolvers unit provides a
parallel directory structure to Grid/GridMain containing implementation for
multigrid Poisson solvers. AMReX-based simulations use its native Poisson
solver, while Paramesh-based simulations leverage linear solvers provided by
HYPRE [19]. The Driver/DriverMain unit, responsible for integrating different
units and managing physics evolution, follows a similar structure and set of
rules to Grid/GridMain. The file Driver_evolveAll.F90 under
Driver/DriverMain/Incomp controls the evolution of multiphase fluid dynamics
solution. Further details about this file will be discussed later, following
the details of the physics and Stencils unit.

Finally, the IO/IOMain/hdf5 unit provides an interface to the Hierarchical
Data Format (HDF5) library for writing and reading data to and from the disk.
Interfaces are available for both serial and parallel versions of the HDF5
library, tuned to the available AMR implementations–AM for AMReX and PM for
Paramesh. Other infrastructure units like Particles and RuntimeParameters
organize source code in a similar fashion. We strongly encourage reading the
Flash-X user guide for the details (https://flash-x.org/pages/documentation).

.1 Flash-X/source. .2 physics. .3 IncompNS. .4 IncompNSMain. .5 constDens. .5
varDens. .4 IncompNSExtras. .4 localAPI.

.1 Flash-X/source. .2 physics. .3 HeatAD. .4 HeatADMain. .5 constDiffusion. .5
varDiffusion. .4 HeatADAdvection. .4 localAPI.

.1 Flash-X/source. .2 physics. .3 Multiphase. .4 MultiphaseMain. .4
MultiphaseEvap. .4 localAPI.

Figure 8: Organization of physics units for incompressible fluid dynamics,
heat transfer, and multiphase equations. The highlighted boxes correspond to
the units/source files required for simulation shown in Figure 5.

Figure 8 provides details of three physics units which model incompressible
fluid dynamics, heat transfer, and multiphase interactions. The
physics/IncompNS/IncompNSMain unit provides functionality for fractional-step
project method for solving Navier-Stokes equations using either constant
coefficient formulation (constDens) or variable coefficient formulation
(varDens). The equations follow the style of Equation 1 and leverage the
design patterns by using functionality from StencilsDiffusion,
StencilsAdvection, and StencilsTemporal from numericalTool/Stencils unit (see
Figure 9 for details). The auxiliary unit physics/IncompNS/IncompNSExtras
provides additional functionality to manage data interpolation from face-
centers to cell-centers. This is useful for post processing and visualization.

The parallel units physics/HeatAD/HeatADMain and
physics/HeatAD/HeatADAdvection offer implementations for heat diffusion and
advection, respectively. The parallel design of this unit permits independent
utilization for heat transfer problems that do not involve fluid flow. When
using heat transfer with fluid flow, the advection unit will always be
invoked. Similar to physics/IncompNS/IncompNSMain, these units leverage
stencil design patterns based on their individual requirements. The sub-
directories constDiffusion and varDiffusion correspond to constant and
variable values for diffusion coefficient diffusion $K$ in Equation 1.

.1 Flash-X/source. .2 physics. .3 sourceTerms. .4 Inlet. .5 InletMain. .4
Outlet. .5 OutletMain. .4 Heater. .5 HeaterMain.

.1 Flash-X/source. .2 numericalTools. .3 Stencils. .4 StencilsMain. .4
StencilsDiffusion. .4 StencilsAdvection. .4 StencilsLevelSet. .4
StencilsTemporal.

Figure 9: Organization of units for source terms and stenciled operations. The
highlighted boxes correspond to the units/source files required for simulation
shown in Figure 5.

The physics/Multiphase/MultiphaseMain unit provides functionality for treating
the level-set function $\phi$ that models the air-water (gas-liquid) interface
in Figure 2. Currently, this unit has a hard dependency on the varDens
configuration under physics/IncompNS/IncompNSMain, which is explicitly
specified in its Config file. In addition to utilizing functionality from
StencilsAdvection and StencilsTemporal, this unit also exploits design
patterns for the level-function by using StencilsLevelSet to handle level-set
specific operations for setting variable fluid properties and other relevant
numerical procedures. Level-set operations are general and can be extended to
other numerical algorithms that do not involve multiphase flows. Therefore,
separating out this functionality provides a composable design for future
work.

Evaporation is modelled using physics/Multiphase/MultiphaseEvap unit which
provides additional functionality to the physics/Multiphase/MultiphaseMain
implementation. It is important to note that for any unit implementation, the
main implementation will always be invoked before the auxiliary
implementations. This unit has a hard dependency on varDiffusion configuration
under physics/HeatAD/HeatADMain.

Figure 8 also shows the localAPI for the physics units. localAPI exists under
many other units and provides functionality that is private to the unit it
belongs to. Public interface for a unit are named as, Unit_interfaceName, and
use a "Unit_" prefix. The interfaces under localAPI use a different prefix
(usually lower case) to denote that they should never be used in a unit
outside their scope.

.1 Flash-X/source. .2 physics. .3 ImBound. .4 ImBoundMain. .4 localAPI.

.1 Flash-X/source. .2 physics. .3 SolidMechanics. .4 SolidMechanicsMain. .4
localAPI.

Figure 10: Organization of immersed boundary and solid mechanics units for
solid body, $\lambda$.

The solid body $\lambda$ is primarily managed by
physics/SolidMechanics/SolidMechanicsMain. This module utilizes the Particles
unit to handle the Lagrangian data structure. The SolidMechanics unit manages
the evolution of the body, influenced either by prescribed kinematics or
fluid-induced forces.

The effect of body on the continuum flow is modelled using
physics/ImBound/ImBoundMain which map the Lagrangian representation to a
level-set function $\lambda_{\phi}$. This unit also leverages the level-set
design patterns in StencilsLevelSet. The mapping uses an an Approximate
Nearest Library (ANN) algorithm for efficient adaptation as the body moves
within the domain. $\lambda_{\phi}$ effectively converts the Lagrangian
reference frame to an Eulerian one, enabling the computation of $f$ in
Equation 1 to model the effects of the solid body on velocity, temperature,
and pressure.

Figure 10 and 11 provide details of the unit organization and the schematic of
the Lagrangian-to-Eulerian mapping. The numerical details of interpolating and
extrapolating fluid-body forces, implemented using a Moving Least Squares
(MLS) scheme, are discussed in [8, 37, 39].

Figure 11: Representation of the solid body $\lambda$ under two different
units: SolidMechanics unit uses Lagrangian reference frame and ImBound maps it
to a level set function $\lambda_{\phi}$.

.1 Flash-X/source. .2 Simulation. .3 main.F90. .3 SimulationMain. .5
incompFlow. .6 Grid_bcApplyToRegion.F90. .6 ImpingingJet. .6 ChannelFlow. .6
FlowBoiling. .7 Grid_bcApplyToRegionSpecialized.F90. .7
Simulation_initBlock.F90. .7 Simulation_init.F90. .7 Simulation_finalize.F90.
.7 flash.par. .7 flash_hdf5_htr_0001. .7 tests. .8 tests.yaml. .8 test_2d.par.
.8 test_2d_hdf5_htr_0001.

Figure 12: Organization of the simulation unit which contains different
applications and use cases. The highlighted boxes correspond to the
units/source files required for simulation shown in Figure 5.

As mentioned earlier, the Simulation unit in Figure 3 is a special unit where
physics and infrastructure units are assembled to generate different
applications. Figure 12 provides details for this unit. We would like to note
that the actual Simulation under Flash-X provides a wide range of applications
assembled using physics units that are not discussed in this article. To
maintain focus on multiphase fluid dynamics applications, we intentionally
left out references to other physics and simulation units, which have been
extensively documented elsewhere [5].

The main.F90 file under Simulation is the main program file for a Flash-X
application. The contents of this file are shown in Figure 13, which show
calls to four top-level Driver interface procedures including
Driver_evolveAll, located under Driver/DriverMain/Incomp for fluid dynamics
simulations.

⬇

program Flashx

use Driver_interface, ONLY : Driver_initParallel, Driver_initAll, &

Driver_evolveAll, Driver_finalizeAll

implicit none

call Driver_initParallel() ! located under Driver/DriverMain, intializes
parallel environment

call Driver_initAll() ! located under Driver/DriverMain, calls Unit_init for
all units

call Driver_evolveAll() ! located under Driver/DriverMain/Incomp

call Driver_finalizeAll() ! located under Driver/DriverMain, call
Unit_finalize for all units

end program Flashx

Figure 13: Contents of Simulation/main.F90

Different multiphase fluid dynamics applications are encoded further along the
directory under incompFlow implementation of Simulation/SimulationMain, which
shows three different applications, FlowBoiling (Figure 5), ImpingingJet
(Figure 6), and a simple single-phase ChannelFlow simulation. The actual
Flash-X source contains many other applications. To build the Flash-X/object
directory for a given simulation, we can invoke the following setuptool call
from Flash-X root directory,

⬇

./setup incompFlow/FlowBoiling -auto -maxblocks=100 -3d -nxb=16 -nyb=16
-nzb=16 \ +amrex +incomp +parallelIO -site=hello

which builds a three dimensional flow boiling simulation shown in Figure 5
using files and directories highlighted in Figures 7 \- 12. The parameter file
flash.par contains information for runtime parameters used to configure
boundary conditions, domain size, AMR refinement levels, etc. The
flash_hdf5_htr_0001 file contains HDF5 encoded information related to the
heater configuration which is handled by physics/sourceTerms/Heater.

The corresponding snippet of Driver_evolveAll.F90 is shown in Figure 14 which
shows of different infrastructure and grid units come together to calculate
predicted velocities for the fractional-step project method. The procedures
IncompNS_advection and IncompNS_predictor are called each time step between
dr_nBegin and dr_nend (the prefix "dr_" indicates that variables are owned by
the Driver/DriverMain unit). These procedures invoke design patterns in
StencilsAdvection and StencilsTemporal, respectively. IncompNS_diffusion uses
a specialized design pattern for the varDens configuration, which is provided
through its localAPI procedure ins_diffusion_vardens. Note that all the
physics procedures operate on the data structure tileDesc, which represents a
single or a collection of blocks ($\Omega$). tileDesc is allocated during loop
over a specialized iterator which is managed through the Grid unit to allow
for performance portability across different computing platforms. The details
on the implementation of this performance portability layer are beyond the
scope of the current article, and we aim to address them in a follow-up work.

An important design feature in Figure 14 is the call to Grid_fillGuardCells,
which applies physical boundary conditions and performs halo exchange between
AMR blocks. In FLASH, these calls were buried inside the physics units, making
performance portability and optimization cumbersome. Within Flash-X, these
calls are exposed to the top-level Driver interface, leading to a cleaner
software design.

⬇

subroutine Driver_evolveAll()

! use statements referencing prodecures from different units that are needed…

implicit none

! variable declarations and book-keeping…

do dr_nstep = dr_nBegin, dr_nend ! Evolution loop over steps

! preceeding physics related calls…

! …

call Grid_fillGuardCells() ! Perform halo-exchange and apply boundary
conditions

\parcall Grid_getTileIterator(itor, nodetype=LEAF) ! Get the tile iterator for
LEAF blocks

do while (itorcall itorcall IncompNS_advection(tileDesc) ! uses procedures
from StencilsAdvection

call IncompNS_diffusion(tileDesc) ! uses procedures from IncompNS/localAPI

call IncompNS_predictor(tileDesc, dr_dt) ! uses procedures from
StencilsTemporal

call Multiphase_velForcing(tileDesc, dr_dt) ! dr_dt is time-step

call itorend do

call Grid_releaseTileIterator(itor) ! Release the iterator

! …

! succeding physics related calls…

end do

! book-keeping and clean up…

end subroutine Driver_evolveAll

Figure 14: Snippet of Driver/DriverMain/Incomp/Driver_evolveAll.F90 to solve
the predictor step for the fractional-step projectio method [9].

##  4 Testing Framework and Scientific Reproduciblity

To ensure the accuracy and sustainability of the codebase, the Flash-X team
has dedicated a significant amount of time to designing tools and procedures
for testing, maintenance, and reproducibility. Two command line tools, in
particular, have been developed:

  * 1.

flashxtest: A testing framework enabling uniform testing practices across
different computing platforms, research groups, and individual developers
https://github.com/Flash-X/Flash-X-Test.

  * 2.

jobrunner: A utility to build and manage laboratory notebooks for systematic
curation of execution environments, simulation data, and post-processing
scripts and procedures. This tool facilitates reproducibility and
collaboration efforts https://github.com/Lab-Notebooks/Jobrunner.

Figure 15: Overview of testing framework: (a) Comparison tests that run a
simulation from time $t_{0}$ to $t_{1}$ and compare the results against an
approved benchmark, (b) Composite tests involve restarting simulations from
$t_{1}$ to $t_{2}$ to ensure that restarts are transparent, and (c) Testing
workflow for on different sites – a cluster hosted at Argonne National
Laboratory (ANL) and a public docker container. Gatekeepers manage testing on
the ANL cluster and maintain control over the quality of the source code in
the main branch of the GitHub repository, while developers leverage the public
docker container to perform unrestricted testing of their code on a feature
branch [13].

Figure 15 provides an overview of the testing framework. Flash-X test
specifications are essentially shorter version of applications located under
Simulation/SimulationMain unit. See Figure 12 for example. The file
tests/tests.yaml consists test specifications for FlowBoiling application
under Simulation/SimulationMain/incompFlow. Figure 16 provides an example of
the contents within this file, which contains information related to setup
options, parfile, and auxiliary file transfers required to run the
Comparison/incompFlow/FlowBoiling/2d/AMReX test.

⬇

Comparison/incompFlow/FlowBoiling/2d/AMReX:

setupOptions: -auto -maxblocks=100 +amrex +parallelIO +incomp -2d -nxb=16
-nyb=16

parfiles: test_2d.par

transfers:
source/Simulation/SimulationMain/incompFlow/FlowBoiling/tests/test_2d_hdf5_htr_0001

Figure 16: Example contents of
Simulation/SimulationMain/incompFlow/FlowBoiling/tests/tests.yaml.

Flash-X supports unit tests and regression tests. Regression tests are
comparison or composite tests as shown in Fig 15a and b. Comparison tests are
based on the availability of an inspected and approved benchmark to compare
the output generated by the application instance run as a test of correctness.
Composite tests include two comparison tests, one in which the application
starts from initial conditions and runs to a time $t_{1}$ and another in which
the application starts from the checkpoint at time $t_{1}$ and runs to a time
$t_{2}$. Therefore, each composite test requires two benchmarks. These two
types of regression tests capture the real world need for simulations to
reproduce correct data over time, and also perform transparent restarts when
running production simulation on a HPC cluster where allocation time may not
match the required simulation time. Testing against approved benchmarks is
performed using Serial Flash Output Comparison Utility (SFOCU), available
under Flash-X/tools.

Flash-X has brought together developers from different scientific domains,
necessitating the need to employ uniform testing practices among different
domains and research groups. This is facilitated using test suites, which
represent collections of tests covered on a site (a HPC platform or a cloud
architecture). Following is an example entry for a suite file which follows
the style of Flash-X setup command,

⬇

incompFlow/FlowBoiling -t ”Comparison/incompFlow/FlowBoiling/2d/AMReX” -np 4
-cbase <yyyy-mm-dd>

This command utilizes the test specifications provided in Figure 16. It
executes a test with four MPI processes and compares its results against the
benchmark established on the date <yyyy-mm-dd>. These tests are managed and
deployed using flashxtest which is extensively documented in [13].

Figure 15c provides an overview of the complete testing workflow. Individual
developers, seeking an efficient process for setting up automated tests when
implementing new features, employ a public Docker container to conduct
unrestricted testing on their feature branches. In contrast, Flash-X
gatekeepers, responsible for ensuring the quality and accuracy of scientific
results, follow a more restrictive workflow with well-defined testing
protocols. When individual developers create a pull request to the main
branch, gatekeepers initiate testing on the Argonne National Laboratory
cluster, utilizing a private and protected test suite. Test failures
necessitate manual intervention, communication, and resolution between
gatekeepers and developers before any new updates are merged into the main
branch.

Scientific experiments carried out with Flash-X are meticulously cataloged and
documented in individual repositories accessible via https://github.com/Lab-
Notebooks. This approach enables collaborators and new developers to initiate
their work with validated and well-documented studies, facilitating a smoother
transition into the intricacies of the Flash-X codebase. Figure 17 showcases
various laboratory notebooks crafted by a Flash-X developer, which serve as
valuable resources for conducting individual experiments or for collaborative
research endeavors.

Figure 17: A common scenario in computational sciences where a developer has
to conduct individual experiments and collaborate with other researchers who
may not be experts in using their software. Developers can create lab
notebooks for different experiments related to performance and physics related
studies and use them to document their investigation and collaborate with
others. Copyright images from our IEEE publication [12].

Laboratory notebooks capture the execution environment, software
configurations, and other scientific decisions and leverage a directory-based
inheritance similar to Flash-X to conduct and archive studies using jobrunner.
A detailed documentation of this process can be found in [12].

The jobrunner utility also ensures the organization of runtime parameters. It
achieves this by utilizing TOML configuration files as an interface to
generate application-specific flash.par files. An illustrative snippet is
provided in Figure 18, showcasing the use of a nested implementation of TOML
keys to declare runtime parameters under the respective units managing them.

⬇

[Grid]

xmin = 0.0 # Domain lower bound

xmax = 1.0 # Domain upper bound

xl_boundary_type = ”inflow_ins” # Boundary condition at lower bound

xr_boundary_type = ”outflow_ins” # Boundary condition at upper bound

nblockx = 10 # Number of blocks

\par[IncompNS]

ins_invReynolds = 0.0001 # Inverse Reynolds number

ins_gravY = -0.01 # Inverse square-root of Froude number

\par[Outlet]

out_buffer = 1.0 # Outlet parameters

\par[Multiphase]

mph_rhoGas = 0.001 # Ratio of gas to liquid density

mph_muGas = 0.02 # Ratio of gas to liquid viscosity

mph_invWeber = 0.004 # Inverse Weber number

\par[Driver]

nend = 1000 # Total evolution steps

tmax = 10 # Maximum simulation time

\par[IO]

basenm = ”INS_Simulation_” # Basename for HDF5 IO files

plot_var_1 = ”pres” # Plotfile variable

Figure 18: Snippet of a TOML configuration file used by jobrunner to generate
a parfile https://github.com/Lab-Notebooks/Liquid-Jet-Crossflow.

Assigning runtime parameters under unit-specific keys enhances implicit
documentation, thereby improving scientific reproducibility. Additionally,
this practice serves as a foundational step in understanding the design of the
Flash-X architecture. When a user or developer modifies a parameter, they can
readily identify the specific unit they are interfacing with. This knowledge
proves invaluable for resolving bugs, incorporating new functionality, or
gaining deeper insights into the codebase.

##  5 Conclusion

In this article, we present a comprehensive overview of the infrastructure and
physics capabilities within Flash-X. Our exploration includes the
identification of various data structures and design patterns inherent in the
mathematical framework of a computational system. We provide detailed insights
into how Flash-X strategically leverages these structures and patterns to
construct a robust software architecture specifically tailored for multiphase
fluid dynamics simulations.

We proceed to demonstrate how this resulting architecture achieves
composability within Flash-X, showcasing its adaptability in configuring
diverse numerical simulations. These simulations encompass solid-fluid
interactions, fluid-fluid dynamics featuring heat transfer, and phase
transition phenomena.

We extend our discussion to encompass broader system design considerations,
emphasizing the critical facets of software maintenance and reproducibility.
We provide insight into established tools and practices that have evolved over
the years in the development of Flash-X. Our intention is to not only showcase
the current state of the software but also to contribute valuable knowledge to
the community regarding sustainable design practices in scientific computing.

This article is intended to serve as a guideline for Flash-X developers and
the broader research community on the sustainable and modular design of
scientific computing software. Additionally, we touch on aspects of
performance portability, which we plan to elaborate on in a follow-up work.

Acknowledgements: The author expresses gratitude to Dr. Anshu Dubey (Argonne
National Laboratory), Dr. Klaus Weide (University of Chicago), and Jared
O’Neal (Argonne National Laboratory) for their invaluable guidance and support
in developing the Flash-X software design for multiphase fluid dynamics
simulations. They have also been co-authors on other Flash-X related articles
and proceedings. Additionally, sincere appreciation is extended to Dr. Elias
Balaras (George Washington University), Dr. Marcos Vanella (National Institute
of Standards and Technology), Keegan Delaney (Dsc. George Washington
University), and Dr. Shizao Wang for their significant contributions to FLASH,
which laid the foundation for the developments discussed in this article.

The material is based upon work supported by Laboratory Directed Research and
Development (LDRD) funding from Argonne National Laboratory, provided by the
Director, Office of Science, of the U.S. Department of Energy under contract
DE-AC02-06CH11357 and the Exascale Computing Project (17-SC-20-SC), a
collaborative effort of the US Department of Energy Office of Science and the
National Nuclear Security Administration.

We also acknowledge that The City of Chicago is located on land that is and
has long been a center for Native peoples. The area is the traditional
homelands of the Anishinaabe, or the Council of the Three Fires: the Ojibwe,
Odawa, and Potawatomi Nations. Many other Nations consider this area their
traditional homeland, including the Myaamia, Ho-Chunk, Menominee, Sac and Fox,
Peoria, Kaskaskia, Wea, Kickapoo, and Mascouten.

The submitted manuscript has been created by UChicago Argonne, LLC, operator
of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of
Energy Office of Science laboratory, is operated under Contract No. DE-
AC02-06CH11357. The U.S. Government retains for itself, and others acting on
its behalf, a paid-up nonexclusive, irrevocable worldwide license in said
article to reproduce, prepare derivative works, distribute copies to the
public, and perform publicly and display publicly, by or on behalf of the
Government. The Department of Energy will provide public access to these
results of federally sponsored research in accordance with the DOE Public
Access Plan. http://energy.gov/downloads/doe-public-access-plan.

During the preparation of this work the author(s) used ChatGPT in order to
clean up language in the manuscript. After using this tool/service, the
author(s) reviewed and edited the content as needed and take(s) full
responsibility for the content of the publication.

## References

  * [1] 17.0, N. V. Argonne national laboratory, illinois.  Available: https://nek5000.mcs.anl.gov..
  * [2] 7, O. V. The openfoam foundation.  Available: https://openfoam.org..
  * [3] Bolotnov, I., Behafarid, F., Shaver, D., Guo, T., Wang, S., Wei, H., Antal, S., Jansen, K., Samulyak, R., and Podowski, M. Interaction of computational tools for multiscale multiphysics simulation of generation-iv reactors.  Proceedings of ICAPP (2010).
  * [4] Chong, K. L., Yang, Y., Huang, S.-D., Zhong, J.-Q., Stevens, R. J. A. M., Verzicco, R., Lohse, D., and Xia, K.-Q. Confined rayleigh-bénard, rotating rayleigh-bénard, and double diffusive convection: A unifying view on turbulent transport enhancement through coherent structure manipulation.  Phys. Rev. Lett. 119 (Aug 2017), 064501.
  * [5] Couch, S. M., Carlson, J., Pajkos, M., O’Shea, B. W., Dubey, A., and Klosterman, T. Towards performance portability in the spark astrophysical magnetohydrodynamics solver in the flash-x simulation framework.  Parallel Computing 108 (2021), 102830.
  * [6] Daley, C., Vanella, M., Dubey, A., Weide, K., and Balaras, E. Optimization of multigrid based elliptic solver for large scale simulations in the FLASH code.  2346–2361.
  * [7] Dhir, V., Warrier, G., Aktinol, E., Chao, D., Sheredy, W., Eggers, J., and Booth, W. Observations on ISS of Bubble Dynamics During Boiling.
  * [8] Dhruv, A. A Multiphase Solver for High-Fidelity Phase-Change Simulations over Complex Geometries.  PhD thesis, George Washington University, 2021.
  * [9] Dhruv, A. A vortex damping outflow forcing for multiphase flows with sharp interfacial jumps.
  * [10] Dhruv, A., Balaras, E., Riaz, A., and Kim, J. A formulation for high-fidelity simulations of pool boiling in low gravity.  International Journal of Multiphase Flow 120 (2019), 103099.
  * [11] Dhruv, A., Balaras, E., Riaz, A., and Kim, J. An investigation of the gravity effects on pool boiling heat transfer via high-fidelity simulations.  International Journal of Heat and Mass Transfer 180 (2021), 121826.
  * [12] Dhruv, A., and Dubey, A. Managing software provenance to enhance reproducibility in computational research.  Computing in Science & Engineering 25, 3 (May 2023), 60–65.
  * [13] Dhruv, A., Jain, R., O’Neal, J., Weide, K., and Dubey, A. Framework and methodology for verification of a complex scientific simulation software, flash-x.
  * [14] Dubey, A., Almgren, A., Bell, J., Berzins, M., Brandt, S., Bryan, G., Colella, P., Graves, D., Lijewski, M., Löffler, F., O’Shea, B., Schnetter, E., Straalen, B. V., and Weide, K. A survey of high level frameworks in block-structured adaptive mesh refinement packages.  Journal of Parallel and Distributed Computing 74, 12 (2014), 3217–3227.
  * [15] Dubey, A., Daley, C., ZuHone, J., Ricker, P. M., Weide, K., and Graziani, C. Imposing a Lagrangian particle framework on an Eulerian hydrodynamics infrastructure in FLASH.  ApJ Supplement 201 (aug 2012), 27.
  * [16] Dubey, A., Lee, Y., Klosterman, T., and Vatai, E. A tool and a methodology to use macros for abstracting variations in code for different computational demands.  Future Generation Computer Systems (2023).
  * [17] Dubey, A., Weide, K., O’Neal, J., Dhruv, A., Couch, S., Harris, J. A., Klosterman, T., Jain, R., Rudi, J., Messer, B., Pajkos, M., Carlson, J., Chu, R., Wahib, M., Chawdhary, S., Ricker, P. M., Lee, D., Antypas, K., Riley, K. M., Daley, C., Ganapathy, M., Timmes, F. X., Townsley, D. M., Vanella, M., Bachan, J., Rich, P. M., Kumar, S., Endeve, E., Hix, W. R., Mezzacappa, A., and Papatheodore, T. Flash-x: A multiphysics simulation software instrument.  SoftwareX 19 (2022), 101168.
  * [18] F. D. Witherden, A. M. Farrington, P. E. V. Pyfr: An open source framework for solving advection-diffusion type problems on streaming architectures using the flux reconstruction approach.  Computer Physics Communications 185 (2014), 3028–3040.
  * [19] Falgout, R. D., and Yang, U. M. hypre: A library of high performance preconditioners.  632–641.
  * [20] Griffin, J., and Bates, C. Ladle Treating, Pouring, and Gating for the Production of Clean Steel Castings.  Research report (Steel Founders’ Society of America). Technical Steering Committee, Steel Founders’ Society of America, 1991.
  * [21] Holems, G., and Keith, D. An air–liquid contactor for large-scale capture of co2 from air.  Philosophical transcations of the royal society (2012).
  * [22] Kim, J., Benton, J. F., and Wisniewski, D. Pool boiling heat transfer on small heaters: effect of gravity and subcooling.  International Journal of Heat and Mass Transfer 45, 19 (2002), 3919 – 3932.
  * [23] Kim, J., Bodony, D. J., and Freund, J. B. Adjoint-based control of loud events in a turbulent jet.  Journal of Fluid Mechanics 741 (2014), 28–59.
  * [24] Kim, J., Oh, B. D., and Kim, M. H. Experimental study of pool temperature effects on nucleate pool boiling.  International Journal of Multiphase Flow 32, 2 (2006), 208 – 231.
  * [25] Lee, M., and Moser, R. D. Direct numerical simulation of turbulent channel flow up to ${Re}\approx 5200$.  Journal of Fluid Mechanics 774 (2015), 395–415.
  * [26] MacNeice, P., Olson, K. M., Mobarry, C., de Fainchtein, R., and Packer, C. Paramesh: A parallel adaptive mesh refinement community toolkit.  Computer Physics Communications 126, 3 (2000), 330–354.
  * [27] Martinez, L. G., Dhruv, A., Lin, L., Balaras, E., and Keidar, M. Interaction between a helium atmospheric plasma jet and targets and dynamics of the interface.  Plasma Sources Science and Technology (2019).
  * [28] Nagrath, S., Jansen, K. E., and Lahey, R. T. Computation of incompressible bubble dynamics with a stabilized finite element level set method.  Computer Methods in Applied Mechanics and Engineering 194, 42 (2005), 4565 – 4587.
  * [29] Nasiri, F., and Balaras, E. The effects of froude number on a turbulent boundary layer with a free-surface.  Flow, Turbulence and Combustion (2019).
  * [30] O’Neal, J., Weide, K., and Dubey, A. Experience reort: Refactoring the mesh interface in flash, a multiphysics software.
  * [31] Ousterhout, J. A Philosophy of Software Design.  Yaknyam Press, 2018.
  * [32] O’Dea, M. L., and Guessous, L. Simulation of Wind Turbine Flow Using the Actuator Line Method in NEK5000.  Volume 7B: Fluids Engineering Systems and Technologies, ASME International Mechanical Engineering Congress and Exposition (2015).  V07BT09A052.
  * [33] Raj, R., Kim, J., and McQuillen, J. Pool Boiling Heat Transfer on the International Space Station: Experimental Results and Model Verification.  Journal of Heat Transfer 134, 10 (2012), 101504.
  * [34] Rodriguez, J. M., Sahni, O., Lahey, R. T., and Jansen, K. E. A parallel adaptive mesh method for the numerical simulation of multiphase flows.  Computers and Fluids 87 (2013), 115 – 131.  USNCCM Moving Boundaries.
  * [35] Urbano, A., Tanguy, S., Huber, G., and Colin, C. Direct numerical simulation of nucleate boiling in micro-layer regime.  International Journal of Heat and Mass Transfer 123 (2018), 1128 – 1137.
  * [36] Utaka, Y., Kashiwabara, Y., and Ozaki, M. Microlayer structure in nucleate boiling of water and ethanol at atmospheric pressure.  International Journal of Heat and Mass Transfer 57, 1 (2013), 222 – 230.
  * [37] Vanella, M., and Balaras, E. Short Note A moving-least-squares reconstruction for embedded-boundary formulations.  Journal of Computational Physics 228, 18 (2009), 6617–6628.
  * [38] Vanella, M., McGrattan, K., McDermott, R., Forney, G., Mell, W., Gissi, E., and Fiorucci, P. A multi-fidelity framework for wildland fire behavior simulations over complex terrain.  Atmosphere 12, 2 (2021).
  * [39] Vanella, M., Rabenold, P., and Balaras, E. A direct-forcing embedded-boundary method with adaptive mesh refinement for fluid–structure interaction problems.  Journal of Computational Physics 229, 18 (2010), 6427–6449.
  * [40] Wang, S., Vanella, M., and Balaras, E. A hydrodynamic stress model for simulating turbulence/particle interactions with immersed boundary methods.  Journal of Computational Physics 382 (2019), 240–263.
  * [41] Wu, X., Moin, P., Wallace, J. M., Skarda, J., Lozano-Durán, A., and Hickey, J.-P. Transitional–turbulent spots and turbulent–turbulent spots in boundary layers.  Proceedings of the National Academy of Sciences 114, 27 (2017), E5292–E5299.
  * [42] Yildiz, M. A., Yuan, H., Merzari, E., and Hassan, Y. Numerical simulation of isothermal flow across slant five-tube bundle with spectral element method code nek5000.  Nuclear Technology 0, 0 (2019), 1–11.
  * [43] Young, T. J., and Vafai, K. Convective cooling of a heated obstacle in a channel.  International Journal of Heat and Mass Transfer 41, 20 (1998), 3131 – 3148.
  * [44] Zhang, W., Almgren, A., Beckner, V., Bell, J., Blaschke, J., Chan, C., Day, M., Friesen, B., Gott, K., Graves, D., Katz, M., Myers, A., Nguyen, T., Nonaka, A., Rosso, M., Williams, S., and Zingale, M. AMReX: a framework for block-structured adaptive mesh refinement.  Journal of Open Source Software 4, 37 (May 2019), 1370.