# Flash-X, a multiphysics simulation software instrument

A. Dubey<EMAIL_ADDRESS>K. Weide<EMAIL_ADDRESS>J. O’Neal
<EMAIL_ADDRESS>A. Dhruv<EMAIL_ADDRESS>S. Couch<EMAIL_ADDRESS>J.A. Harris
<EMAIL_ADDRESS>T. Klosterman<EMAIL_ADDRESS>R. Jain<EMAIL_ADDRESS>J.
Rudi<EMAIL_ADDRESS>O.E.B. Messer<EMAIL_ADDRESS>M. Pajkos
<EMAIL_ADDRESS>J. Carlson<EMAIL_ADDRESS>R. Chu
<EMAIL_ADDRESS>M. Wahib<EMAIL_ADDRESS>S. Chawdhary
<EMAIL_ADDRESS>P.M. Ricker<EMAIL_ADDRESS>D. Lee
<EMAIL_ADDRESS>K. Antypas<EMAIL_ADDRESS>K.M. Riley<EMAIL_ADDRESS>C.
Daley<EMAIL_ADDRESS>M. Ganapathy<EMAIL_ADDRESS>F.X. Timmes
<EMAIL_ADDRESS>D.M. Townsley<EMAIL_ADDRESS>M. Vanella
<EMAIL_ADDRESS>J. Bachan<EMAIL_ADDRESS>P.Rich
<EMAIL_ADDRESS>S. Kumar<EMAIL_ADDRESS>E. Endeve<EMAIL_ADDRESS>W. R. Hix<EMAIL_ADDRESS>A. Mezzacappa<EMAIL_ADDRESS>T. Papatheodore
<EMAIL_ADDRESS>Argonne National Laboratory, Lemont, IL, 60439, USA  Oak
Ridge National Laboratory, Oak Ridge, TN, 37831, USA  Michigan State
University  University of Illinois, Urbana Champaign  University of
California, Santa Cruz  George Washington University  Lawrence Berkeley
National Laboratory  University of Chicago, IL, 60637, USA  Google Inc
Arizona State University  National Institute of Standards and Technology
University of Tennessee, Knoxville, TN, 37996, USA  University of Alabama,
Tuscaloosa, AL, 35487, USA  RIKEN, Kobe, Japan

###### Abstract

Flash-X is a highly composable multiphysics software system that can be used
to simulate physical phenomena in several scientific domains. It derives some
of its solvers from FLASH, which was first released in 2000. Flash-X has a new
framework that relies on abstractions and asynchronous communications for
performance portability across a range of increasingly heterogeneous hardware
platforms. Flash-X is meant primarily for solving Eulerian formulations of
applications with compressible and/or incompressible reactive flows. It also
has a built-in, versatile Lagrangian framework that can be used in many
different ways, including implementing tracers, particle-in-cell simulations,
and immersed boundary methods.

###### keywords:

Multiphysics , Simulation software , high-performance computing , performance
portability

††journal: SoftwareX

## Metadata

Nr. |  Code metadata description |
---|---|---
C1 |  Current code version |  1.0
C2 |  Permanent link to code/repository used for this code version |  https://github.com/Flash-X/Flash-X
C3 |  Code capsule |  https://github.com/Flash-X/ Workflows/tree/main/incompFlow/ FlowBoiling
C4 |  Legal Code License |  Apache 2.0
C5 |  Code versioning system used |  git
C6 |  Software code languages, tools, and services used |  Fortran, C, C++, Python3, MPI, OpenMP, OpenACC, HDF5
C7 |  Compilation requirements, operating environments and dependencies |  Unix, Linux, OSX based compilers for languages and libraries mentioned above
C8 |  Developer documentation/manual |  https://flash-x.org/ pages/documentation/
C9 |  Support email for questions |<EMAIL_ADDRESS>
Table 1: Code metadata. Please note that the code repository is private only
because our funding agency requires us to keep a list of people who obtain the
code directly from our repo. Anyone can furnish their github id and be added
to the list of collaborators.

##  1 Motivation and Significance

Flash-X [1] is a new incarnation of FLASH [2, 3], a multiphysics software
system that has been used by multiple science communities. Flash-X is meant
for use beyond existing FLASH science communities. It is designed to be easily
adaptable for use by any computational scientists who rely upon differential
equations as their primary mathematical model with finite-volume or finite-
difference discretization. FLASH was designed only for a homogeneous,
distributed-memory parallel model with bulk-synchronism, which has rendered it
unsuitable for use on many newer system architectures that are heavily reliant
on disparate memory spaces (e.g., accelerators). This difficulty is further
exacerbated by increasing heterogeneity in hardware as well as solvers within
the code. Flash-X has a fundamentally redesigned architecture that uses
abstractions and asynchronous operations for performance portability across a
variety of platforms, both with and without accelerators. Our design is
forward-looking in that it makes minimal assumptions about which
parallelization or memory models are likely to be prevalent in future
platforms. The design relies upon self-describing code components of varying
granularity and a toolchain that can interpret the metadata of the code
components to synthesize application instances. The synthesis is done partly
through assembly, partly through code translation, and partly through code
generation. Some code assembly features have been imported from FLASH, but
have been significantly enhanced to discretize components at a finer scope
than subroutines or functions. Tools for code translation and runtime
management are new and will enable orchestration of computation and data
movement between distinct compute devices on a node.

In addition to the new architecture, Flash-X has newer and higher- fidelity
physics solvers. Most notable among these are Spark [4] for
magnetohydrodynamics, XNet [5, 6] for nuclear burning, thornado [7, 8] for
neutrino radiation transport, and WeakLib [9, 10, 11] for tabulated
microphysics. Additionally, Flash-X can support multiphase flow through a
level-set method, which did not exist in FLASH releases [12]. Flash-X has been
exercised on small clusters at Argonne National Laboratory and on leadership-
class machines at Oak Ridge National Laboratory and Argonne National
Laboratory. Flash-X will showcase the key performance parameters of ExaStar
[13], a project under the Exascale Computing Project[14, 15] (ECP), through a
core-collapse supernova (CCSN) simulation on exascale machines to be deployed
by the US Department of Energy. To run effectively at scale, Flash-X will rely
upon the toolchain described above. Some components of the toolchain are
embedded in Flash-X, while others are encapsulated into independent libraries
that can be used by other codes. Note that compilation and execution of the
code do not require using these external libraries; they are used only to
orchestrate data movement and computation for better performance.

Along with a new architecture, Flash-X also adopts a community-based, open
development model. The stewardship of the code is guided by a Council
representing all the major science communities of FLASH/Flash-X. More details
of our community development model are available at https://flash-x.org.

##  2 Software Description

The Flash-X code is a component-based software system for simulation of
multiphysics applications that can be formulated largely as a collection of
partial and ordinary differential equations (PDEs and ODEs), as well as
algebraic equations. The equations are discretized and solved on a domain that
can have uniform resolution (UG) or adaptive mesh refinement (AMR). In
Flash-X, one can select between PARAMESH [16], an octree-based library written
in Fortran, or AMReX [17, 18], a highly-flexible, patch-based, C++ AMR
library. Both AMR frameworks can interface to math libraries such as hypre
[19] and PETSc [20], making those solvers available to Flash-X. Physics units
are designed to be oblivious of domain decomposition. Bulk of their code is
written for block-by-block update, interspersed with invocation of fine-coarse
boundary resolution related API functions of the Grid unit as needed.

Hyperbolic equations are solved using explicit methods commonly used for
compressible flows with strong shocks, described in Section 2.2. For elliptic
equations, one can either use an included multipole solver [21], AMReX’s
multigrid solver, or an interface to one of the math libraries. For parabolic
equations, one must rely upon library interfaces.

The maintained code components are written in a combination of high-level
languages such as Fortran, C, and C++, with an embedded domain-specific
configuration language (DSCL) that also supports Flash-X custom macros. The
DSCL permits multiple alternative definitions of macros with a built-in
arbitration mechanism to select the appropriate definition for an instance of
code assembly. The accompanying configuration toolchain can translate and
assemble different combinations of the components to configure a diverse set
of applications. Flash-X has been designed from the outset to be performant
with increasing heterogeneity of both the platforms and the solvers within the
code.

The code uses the Message-Passing Interface (MPI) library for communication
between nodes, though more than one MPI rank can also be placed on a node.
HDF5 is the default mode for IO. Support for OpenMP, both for threading and
for offloading to accelerators, is built into several, though not all, of the
solvers.

###  2.1 Software Architecture

Flash-X has composable components with accompanying metadata that can express,
for example, inter-component dependency and exclusivity, necessary state
variables, etc. The metadata is encapsulated within the code components by
accompanying config files and is parsed and interpreted by the configuration
tool, Setup. Setup parses config files recursively, aggregates requisite
components, and assembles a complete application. It also assembles the
compilation/make system and runtime parameters for each component included in
the application. The Setup tool also implements code inheritance through a
combination of keywords in the config files and the Unix directory structure
instead of using programming language supported inheritance mechanisms. When
Setup parses the source tree, it treats each subdirectory as inheriting all of
the files in its parent’s directory. While source files at a given level of
the directory hierarchy override files with the same name at higher levels,
config files accumulate all definitions encountered. The schematic for
inheritance is shown in Figure 1.

Figure 1: Schematic for the implementation of inheritance in Flash-X. Handling
of inheritance and variants assumes three lists: one for files, one for
macros, and one for runtime parameters. The flowchart in the right box gives
details of how keys and runtime parameters are updated as the source tree is
traversed.

In Flash-X parlance, the highest-level code component for a specific type of
functionality is called a unit. Units can have subunits. A unit includes an
API accessible to the whole code through which it interacts with other units
and the driver. While each subunit can have its own sub-components with no
restriction on how fine-grained they can become, the general rule of thumb is
to keep them as coarse-grained as feasible for ease of maintenance. A unit can
have multiple alternative implementations, one of which is required to be a
null implementation. If a unit is not needed in a simulation, the null
implementation is included. This feature facilitates maintaining very few
implementations of the main driver while permitting many combinations of
capabilities to be included in an application. Any code component can have
multiple alternative implementations, though unlike the unit-level API, lower-
level components do not require null implementations.

A different mechanism is used when a code component needs to become smaller
than a function or a subroutine. Here, we rely on macros to implement
alternative definitions of an operation, including the null case. The
inheritance mechanism shown in Figure 1 arbitrates on which definition to
select. The macros may also have arguments, be inlined, and be recursive. This
mechanism serves two purposes. The first is for developer convenience. Certain
code patterns repeat often in the code – for example, invocation of iterators,
bounds for loop-nests, and bounds for arrays. We have provided macros for such
repeated patterns, and developers can use these at their discretion. Macros
make the code compact, reduce cut-and-paste errors, and help to clarify the
control flow and semantics of the code. The second, more powerful motivation
is that with alternative definitions, we can generate many variants of a code
component from the same source. This functionality is particularly useful when
different control flow is more suitable for different compute devices. We can
keep arithmetic expressions invariant while using macros for the control flow,
or vice-versa, thus not only eliminating code duplication but also keeping the
maintained code more compact. The schematic for generating variants from a
single source where specializations are obtained through alternative macro
definitions is shown Figure in 2.

Figure 2: Schematic for generating variants from single source using
inheritance and macros.

###  2.2 Software Functionalities

The Flash-X distribution includes solvers for compressible and incompressible
fluids, several methods for handling equations of state (EOS), source terms
for nuclear burning, several methods for computing effects of gravity, level-
set methods for multiphase flow, and several others. The primary formulation
for PDEs in Flash-X is Eulerian, although a versatile Lagrangian framework is
also included that can be configured to do computations such as tracers,
particles-in-cell, immersed boundaries, etc. The vast majority of applications
using Flash-X include some form of hydrodynamics or magnetohydrodynamics in
their configuration. However, it is possible to configure applications that
completely bypass those solvers.

Magnetohydrodynamics and Hydrodynamics: a compressible
magnetohydrodynamics/hydrodynamics solver with second- or third-order strong
stability preserving (SSP) Runge-Kutta (RK) time integration (Spark) [4],
another compressible hydrodynamics solver with a predictor-corrector
formulation [22, 23], and an incompressible hydrodynamics solver with fluid-
structure interaction [24] are included in the distribution. All of the
solvers can be used in 1-, 2-, or 3- dimensional configurations.

Equations of State: the code supports several EOS versions suitable for a
range of regimes in astrophysical flows. The simplest one is a perfect-gas EOS
with a multispecies variant. Another implementation with two variants uses a
fast Helmholtz free-energy table interpolation to handle degenerate
relativistic electrons and positrons and also includes radiation pressure and
ions (via the perfect gas approximation) [25].

Nuclear Burning: three nuclear reaction networks of varying numbers of species
are included in the distribution. Approx-13 and approx-19 [26] are inherited
from FLASH. XNet is a standalone code for evolving astrophysical nuclear
burning and is generalizable to arbitrarily large networks as needed for
improved physical fidelity of some applications.

Gravity: the gravitational potential can be treated very simply as constant,
or through a Poisson solve using a multipole or multigrid method depending
upon the symmetry of the density field.

Particles: this component of the code forms the basis for the Lagrangian
framework [27]. Particles maintain their own spatial coordinates and are
independently integrated in time. They interact with the Eulerian mesh either
to obtain physical quantities needed for their advancement or to deposit
quantities such as mass, charge, or energy to the mesh, depending on usage.

Incompressible Fluid Dynamics: this component of the code solves
incompressible Navier-Stokes equations for single and multiphase flow
simulations with options for heat transfer and phase transitions [12]. The
Navier-Stokes solver is implemented using a fractional-step temporal
integration scheme that uses Poisson solver for pressure. Multiphase
interfaces are tracked with a level-set function and use ghost-fluid methods
to account for forces due to surface tension and mass transfer [28]. The
effect of solid bodies on the fluid is modeled using an immersed boundary
method that uses Lagrangian particles[29].

Importable Modules: Flash-X uses GitHub’s submodules to import some
capabilities that are independently developed and hosted in their own
repositories. These include WeakLib for tabulated, nuclear EOS and neutrino-
matter interaction rates, and thornado for spectral neutrino radiation
transport.

##  3 Illustrative Examples

We describe two example simulations using Flash-X from two different science
communities. The first is a CCSN simulation that uses compressible
hydrodynamics, nuclear EOS, neutrino radiation transport, and self-gravity
solvers. The second is a subcooled flow boiling simulation that uses
multiphase incompressible Navier-Stokes and heat advection diffusion solver.

We perform a CCSN simulation in spherical symmetry, initiated with a low-mass
pre-collapse progenitor star previously modeled throughout all stages of
stellar evolution [30]. Electron-type neutrinos and anti-neutrinos are evolved
using thornado’s two-moment neutrino transport solver and WeakLib’s tabulated
nuclear EOS [31] and neutrino-matter interaction rates [32]. Compressible
hydrodynamics are evolved with Spark, and Newtonian self-gravity is computed
using the multipole Poisson solver. For a more detailed description of the
physics included, see [33]. Figure 3 shows the evolution of the ratio of
electrons to baryons (electron fraction) versus radius during a critical epoch
in the simulation that spans the formation of the primary shock-wave during
core “bounce” — the phenomena of infalling matter colliding with, and
_bouncing_ off of, the newly-formed neutron-star.

Figure 3: The electron fraction is plotted versus stellar radius for various
times (relative to the bounce-time, $t_{b}$) in a CCSN simulation.

Figure 4 provides details for the subcooled flow-boiling simulation which was
designed to replicate experiments performed at different gravity levels by
Lebon et al. [34]. These computations used the multiphase incompressible
Navier–Stokes solver along with the phase transition capability, and were
preformed at a resolution almost twice the previous state-of-the-art [35, 28].
Liquid coolant flows over a heater surface with a mean velocity $U_{0}$,
leading to phase-change and formation of vapor bubbles. These vapor bubbles
grow, merge, and finally depart the heater surface due to buoyancy which
introduces turbulence in the domain. The heat transfer associated with this
turbulence is an important parameter in designing cooling systems for
automotive and industrial components, but is difficult to quantify through
experimental observations/measurements. With Flash-X we are able to address
this challenge through targeted high-fidelity simulations to quantify the
contribution of turbulent heat flux.

Figure 4: Example of a flow boiling simulation with Flash-X. Liquid coolant
flows over a heater surface with a mean velocity $U_{0}$, leading to phase-
change and formation of vapor bubbles. The bubble dynamics introduce
turbulence which enhances heat transfer between the heater surface and
coolant, shown by temperature ($T$) contours

##  4 Impact

FLASH has been an influential code for computing astrophysical flows almost
since its inception. FLASH’s scientific impact is clearly demonstrated by the
citation history of the original paper describing the code, as shown in Figure
5. Analysis in [36, 37] further quantifies the scientific significance and
impact of the code on science. FLASH has not only been used extensively for
science, it has also been among the pioneers in giving due importance to
software quality and adopting rigorous auditing and productivity practices
[38].

Figure 5:  Citation history of the original FLASH paper from Google Scholar
and Astrophysics Data System (ADS).

In recent years, FLASH’s use has been diminishing in several communities
because of its inability to use accelerators effectively. Flash-X is designed
to fill this gap and become a reliable multiphysics simulation code for the
communities that earlier relied on FLASH. At least two major communities of
FLASH users, stellar astrophysics[4, 33] and fluid-structure interactions[12],
are already transitioning to Flash-X, with some users now exclusively using
Flash-X. These use cases have also reported on performance gains with the use
of GPUs. Not all of FLASH’s physics capabilities are available in Flash-X yet.
However, since Flash-X is open source, it is expected that interested users
will assist in transitioning their capabilities of interest to the Flash-X
architecture and help grow the Flash-X community. Additionally, the new tool-
chain for orchestration of data and work movement is still in the early stages
of being exercised. Preliminary performance studies of the runtime tool have
been very encouraging [39]. It is expected that full performance gains will
have been realized by the next major release.

##  5 Conclusions

Sustained funding under the ECP has permitted modernization of a highly
capable community code for current and future platforms. With Flash-X, the
FLASH science communities can embrace heterogeneity and use available hardware
effectively. With the move to an open, community-based development model,
users are assured of continuity and support for the code without depending on
a single funding source. FLASH has had a long history of scientific discovery,
and Flash-X aims to follow in that tradition. With more modern solvers and
flexible architecture, Flash-X can continue to be a very useful resource for
science domains that rely on modeling of partial differential equations.

##  6 Conflict of Interest

We wish to confirm that there are no known conflicts of interest associated
with this publication and there has been no significant financial support for
this work that could have influenced its outcome.

## Acknowledgements

The authors acknowledge all contributors to the Flash-X code, including
contributors to the FLASH code from whose work Flash-X has inherited.

This work was supported by the U.S. Department of Energy Office of Science
Office of Advanced Scientific Computing Research under contract number DE-
AC02-06CH1137.

This research was supported by the Exascale Computing Project (17-SC-20-SC), a
collaborative effort of two U.S. Department of Energy organizations (Office of
Science and the National Nuclear Security Administration) that are responsible
for the planning and preparation of a capable exascale ecosystem, including
software, applications, hardware, advanced system engineering, and early
testbed platforms, in support of the nation’s exascale computing imperative.

This research used resources of the Oak Ridge Leadership Computing Facility,
which is a DOE Office of Science User Facility supported under Contract DE-
AC05-00OR22725.

## References

  * [1] Flash-X.
URL https://github.com/Flash-X/Flash-X

  * [2] A. Dubey, K. Antypas, M. K. Ganapathy, L. B. Reid, K. Riley, D. Sheeler, A. Siegel, K. Weide, Extensible component-based architecture for FLASH, a massively parallel, multiphysics simulation code, Parallel Computing 35 (10-11) (2009) 512–522.  doi:10.1016/j.parco.2009.08.001.
  * [3] B. Fryxell, K. Olson, P. Ricker, F. Timmes, M. Zingale, D. Lamb, P. MacNeice, R. Rosner, J. W. Truran, H. Tufo, FLASH: An adaptive mesh hydrodynamics code for modeling astrophysical thermonuclear flashes, The Astrophysical Journal Supplement Series 131 (2000) 273.  doi:10.1086/317361.
  * [4] S. M. Couch, J. Carlson, M. Pajkos, B. W. O’Shea, A. Dubey, T. Klosterman, Towards performance portability in the spark astrophysical magnetohydrodynamics solver in the flash-x simulation framework, Parallel Computing 108 (2021) 102830.
  * [5] XNet, https://github.com/starkiller-astro/xnet (Apr. 2022).
  * [6] W. R. Hix, F. K. Thielemann, Computational methods for nucleosynthesis and nuclear energy generation., Journal of Computational and Applied Mathematics 109 (1999) 321–351.
  * [7] R. Chu, E. Endeve, C. D. Hauck, A. Mezzacappa, Realizability-preserving DG-IMEX method for the two-moment model of fermion transport, Journal of Computational Physics 389 (2019) 62–93.  doi:10.1016/j.jcp.2019.03.037.
  * [8] M. P. Laiu, E. Endeve, R. Chu, J. A. Harris, O. E. B. Messer, A DG-IMEX Method for Two-moment Neutrino Transport: Nonlinear Solvers for Neutrino-Matter Coupling, The Astrophysical Journal Supplement Series 253 (2) (2021) 52.  doi:10.3847/1538-4365/abe2a8.
  * [9] WeakLib, https://github.com/starkiller-astro/weaklib (Apr. 2022).
  * [10] D. Pochik, B. L. Barker, E. Endeve, J. Buffaloe, S. J. Dunham, N. Roberts, A. Mezzacappa, thornado-hydro: A discontinuous galerkin method for supernova hydrodynamics with nuclear equations of state, The Astrophysical Journal Supplement Series 253 (1) (2021) 21.  doi:10.3847/1538-4365/abd700.
  * [11] R. E. Landfield, Sensitivity of neutrino-driven core-collapse supernova models to the microphysical equation of state, Ph.D. thesis, University of Tennessee, https://trace.tennessee.edu/utk{_}graddiss/5294 (2018).
  * [12] A. Dhruv, E. Balaras, A. Riaz, J. Kim, An investigation of the gravity effects on pool boiling heat transfer via high-fidelity simulations, International Journal of Heat and Mass Transfer 180 (2021) 121826.  doi:10.1016/j.ijheatmasstransfer.2021.121826.
  * [13] Exastar, multi-physics stellar astrophysics at exascale.
URL https://sites.google.com/lbl.gov/exastar

  * [14] The Exascale Computing Project (2020).
URL https://www.exascaleproject.org/

  * [15] F. Alexander, A. Almgren, J. Bell, A. Bhattacharjee, J. Chen, P. Colella, D. Daniel, J. DeSlippe, L. Diachin, E. Draeger, A. Dubey, T. Dunning, T. Evans, I. Foster, M. Francois, T. Germann, M. Gordon, S. Habib, M. Halappanavar, S. Hamilton, W. Hart, Z. (Henry) Huang, A. Hungerford, D. Kasen, P. R. C. Kent, T. Kolev, D. B. Kothe, A. Kronfeld, Y. Luo, P. Mackenzie, D. McCallen, B. Messer, S. Mniszewski, C. Oehmen, A. Perazzo, D. Perez, D. Richards, W. J. Rider, R. Rieben, K. Roche, A. Siegel, M. Sprague, C. Steefel, R. Stevens, M. Syamlal, M. Taylor, J. Turner, J.-L. Vay, A. F. Voter, T. L. Windus, K. Yelick, Exascale applications: skin in the game, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 378 (2166) (2020) 20190056.  doi:10.1098/rsta.2019.0056.
  * [16] P. MacNeice, K. Olson, C. Mobarry, R. de Fainchtein, C. Packer, PARAMESH: A parallel adaptive mesh refinement community toolkit, Computer Physics Communications 126 (3) (2000) 330–354.
  * [17] Amrex, https://amrex-codes.github.io/ (2020).
  * [18] W. Zhang, A. Almgren, V. Beckner, J. Bell, J. Blaschke, C. Chan, M. Day, B. Friesen, K. Gott, D. Graves, et al., AMReX: A framework for block-structured adaptive mesh refinement, JOSS 4 (37) (2019) 1370–1370.
  * [19] R. Falgout, U. Yang, hypre: A library of high performance preconditioners, Computational Science-ICCS 2002 (2002) 632–641.
  * [20] S. Balay, et al., Petsc web page (2001).
  * [21] S. M. Couch, C. Graziani, N. Flocke, An Improved Multipole Approximation for Self-gravity and Its Importance for Core-collapse Supernova Simulations, The Astrophysical Journal 778 (2) (2013) 181.
  * [22] D. Lee, A. E. Deane, An unsplit staggered mesh scheme for multidimensional magnetohydrodynamics, Journal of Computational Physics 228 (4) (2009) 952–975.
  * [23] D. Lee, A solution accurate, efficient and stable unsplit staggered mesh scheme for three dimensional magnetohydrodynamics, Journal of Computational Physics 243 (2013) 269–292.
  * [24] M. Vanella, P. Rabenold, E. Balaras, A direct-forcing embedded-boundary method with adaptive mesh refinement for fluid–structure interaction problems, Journal of Computational Physics 229 (18) (2010) 6427–6449.
  * [25] F. X. Timmes, F. D. Swesty, The Accuracy, Consistency, and Speed of an Electron-Positron Equation of State Based on Table Interpolation of the Helmholtz Free Energy, The Astrophysical Journal Supplement Series 126 (2) (2000) 501–516.  doi:10.1086/313304.
  * [26] F. Timmes, Integration of nuclear reaction networks, The Astrophysical Journal Supplement Series 124 (1999) 241–263.
  * [27] A. Dubey, C. Daley, J. ZuHone, P. M. Ricker, K. Weide, C. Graziani, Imposing a Lagrangian particle framework on an Eulerian hydrodynamics infrastructure in FLASH, The Astrophysical Journal Supplement Series 201 (2012) 27.  doi:10.1088/0067-0049/201/2/27.
  * [28] A. Dhruv, E. Balaras, A. Riaz, J. Kim, A formulation for high-fidelity simulations of pool boiling in low gravity, International Journal of Multiphase Flow 120 (2019) 103099.  doi:10.1016/j.ijmultiphaseflow.2019.103099.
  * [29] M. Vanella, E. Balaras, Short note: A moving-least-squares reconstruction for embedded-boundary formulations, J. Comput. Phys. 228 (18) (2009) 6617–6628.  doi:10.1016/j.jcp.2009.06.003.
  * [30] T. Sukhbold, T. Ertl, S. Woosley, J. M. Brown, H.-T. Janka, Core-collapse supernovae from 9 to 120 solar masses based on neutrino-powered explosions, The Astrophysical Journal 821 (1) (2016) 38.
  * [31] A. W. Steiner, J. M. Lattimer, E. F. Brown, The equation of state from observed masses and radii of neutron stars, The Astrophysical Journal 722 (1) (2010) 33\.
  * [32] S. W. Bruenn, Stellar core collapse - numerical model and infall epoch, The Astrophysical Journal Supplement Series 58 (1985) 771–841.  doi:10.1086/191056.
  * [33] J. A. Harris, R. Chu, S. M. Couch, A. Dubey, E. Endeve, A. Georgiadou, R. Jain, D. Kasen, M. P. Laiu, O. B. Messer, et al., Exascale models of stellar explosions: Quintessential multi-physics simulation, The International Journal of High Performance Computing Applications (2021) 10943420211027937.
  * [34] M. T. Lebon, C. F. Hammer, J. Kim, Gravity effects on subcooled flow boiling heat transfer, International Journal of Heat and Mass Transfer 128 (2019) 700–714.  doi:https://doi.org/10.1016/j.ijheatmasstransfer.2018.09.011.
  * [35] Y. Sato, B. Niceno, Pool boiling simulation using an interface tracking method: From nucleate boiling to film boiling regime through critical heat flux, International Journal of Heat and Mass Transfer 125 (2018) 876 – 890.  doi:10.1016/j.ijheatmasstransfer.2018.04.131.
  * [36] A. Dubey, P. Tzeferacos, D. Q. Lamb, The dividends of investing in computational software design: A case study, The International Journal of High Performance Computing Applications 33 (2) (2019) 322–331.
  * [37] A. Grannan, K. Sood, B. Norris, A. Dubey, Understanding the landscape of scientific software used on high-performance computing platforms, The International Journal of High Performance Computing Applications 34 (4) (2020) 465–477.
  * [38] A. Dubey, K. Antypas, A. C. Calder, C. Daley, B. Fryxell, J. B. Gallagher, D. Q. Lamb, D. Lee, K. Olson, L. B. Reid, et al., Evolution of FLASH, a multi-physics scientific simulation code for high-performance computing, The International journal of high performance computing applications 28 (2) (2014) 225–237.
  * [39] J. O’Neal, M. Wahib, A. Dubey, K. Weide, T. Klosterman, J. Rudi, Domain-specific runtime to orchestrate computation on heterogeneous platforms, in: R. Chaves, D. B. Heras, A. Ilic, D. Unat, R. M. Badia, A. Bracciali, P. Diehl, A. Dubey, O. Sangyoon, S. L. Scott, L. Ricci (Eds.), Euro-Par 2021: Parallel Processing Workshops, Springer International Publishing, Cham, 2022, pp. 154–165.