# Potential long-term habitable conditions on planets with primordial H-He
atmospheres

Marit Mol Lous 1,2 Ravit Helled2 & Christoph Mordasini1

###### Abstract

Cold super-Earths which retain their primordial, H-He dominated atmosphere
could have surfaces that are warm enough to host liquid water. This would be
due to the collision induced absorption (CIA) of infra-red light by hydrogen,
which increases with pressure. However, the long-term potential for
habitability of such planet has not been explored yet. Here we investigate the
duration of this potential exotic habitability by simulating planets of
different core masses, envelope masses and semi-major axes. We find that
terrestrial and super-Earth planets with masses of $\sim$1 - 10$M_{\oplus}$
can maintain temperate surface conditions up to 5 - 8 Gyr at radial distances
larger than $\sim$2 AU. The required envelope masses are $\sim
10^{-4}\,M_{\oplus}$ (which is 2 orders of magnitude more massive than
Earth’s), but can be an order of magnitude smaller (when close-in) or larger
(when far out). This result suggests that the concept of planetary
habitability should be revisited and made more inclusive with respect to the
classical definition.

Physikalisches Institut, Universität Bern, Gesellschaftstrasse 6, 3012 Bern,
Switzerland

Institute for Computational Science, University of Zürich, Winterthurerstrasse
190, CH-8057 Zürich, Switzerland

The emergence of life as we know it is generally understood to require three
building blocks: an energy source, access to nutrition and the presence of
liquid water [1, 2]. Apart from these requirements, it remains unknown to what
extent conditions on other planets need to resemble Earth in order to be
habitable. Since the possibility of in-situ solar system exploration,
habitable conditions are considered on planets and moons that have liquid
water oceans underneath a layer of ice [3]. These habitats would have to be
inhabited by organisms that are adapted to high pressures and use a metabolism
independent of photosynthesis. Such organisms have already been found on
Earth: while today the majority of Earth’s biomass is concentrated on its
surface (due to complex photosynthetic organisms such as land plants[4]), the
subsurface biomass likely outweighed the surface biosphere for most of
geologic history [5]. Life has adapted to many other, relatively extreme,
environments such as the depths of the ocean at kbar pressures [6], although
it is unknown how many of these organisms live independently from life on the
surface. In the search for life on extraterrestrial planets it needs to be
considered that life might manifest and thrive under conditions that would be
considered extreme on Earth.
Observations of exoplanets have shown that the solar system might not be a
standard planetary system. This was initially implied after the first
detections of Hot Jupiters [7] (although these turned out later to be rather
rare but seemingly prevalent due to a detection bias) and further confirmed by
the fact that super-Earths, absent in the solar system, are very common [8, 9,
10]. To date, super-Earths have only been observed at short orbits, but planet
formation models indicate they are also prevalent at large radial distances
[11] where they could retain a non-negligible H-He envelope, which is the
original atmosphere accreted from the protoplanetary disk [12, 13]. At several
AU away from the host star, stellar radiation is insufficient to thermally
evaporate a significant amount of the atmospheric H-He [14].
The primordial atmosphere, dominated by hydrogen and helium, would have
insufficient greenhouse gases that are significant on Earth, such as
$\text{CO}_{2}$ or methane. However, if the atmosphere is massive enough,
$\text{H}_{2}$ will act as a greenhouse gas. At sufficient pressures the
$\text{H}_{2}$ molecules undergo enough collisions to create a dipole moment,
causing them to absorb the infrared radiation coming from the planet; this is
known as collision induced absorption (CIA) [15, 16]. It could raise the
surface temperature enough to allow a liquid water ocean [17, 18, 19].
Past analytical work has shown that in the case of a massive atmosphere an
intrinsic heat source is sufficient to warm the planetary surface, possibly
even enabling unbound planets to be temperate [17]. Alternatively, planets
with a smaller envelope can have stellar irradiation as the sole energy
source, assuming that the accreted envelope is neither too massive/small for
the given equilibrium temperature [18]. Recently Madhusudhan et al.[20]
investigated the mass-radius relationship of ’Hycean worlds’, planets with a
liquid water layer above a rocky core and below H-He primordial atmosphere.
Similar to previous studies [18, 21], they found that liquid water can persist
at a wide range of equilibrium temperatures. Therefore, the range of orbital
distances where liquid surface water can exist (the so-called habitable zone
[22, 23, 24]) is wider than in the case of Earth-like planets.
These past studies concentrated on the static properties necessary for liquid
water. However, liquid water conditions on these cold planets could be a
transient phase [19]. It is important to consider atmospheric escape since
this can give limits to the separations at which H-He envelopes allow the
existence and stability of liquid water. Here, we study the temporal evolution
in particular to address the duration of a primordial H-He habitat.

We investigate how long surface liquid water could be permitted using an
evolution model and define the duration of adequate surface pressure-
temperature conditions as $\tau_{\text{lqw}}$. First we assume that the core
mass, envelope mass and semi-major axis remain constant in time. We run a
large grid of evolution models and vary these three parameters. During the
planetary evolution the equilibrium temperature changes due to the evolution
of the host-star, which is modelled as sun-like [25]. The planet’s intrinsic
luminosity declines as the envelope and interior cool down and radiogenic
components decay. For a given core mass we run 1144 models, with the semi-
major axis varied from 1 AU to 100 AU and the envelope mass from $10^{-6}$ to
$10^{-1.8}$ $M_{\oplus}$.
Figure 1 shows these grids for models with core masses of 1.5, 3 and 8
$M_{\oplus}$. Every grid-point represents a planet with a given core mass,
envelope mass, and semi-major axis. The evolution models are run for 8 Gyr
(see Method for discussion). Colored, circular grid-points represent
$\tau_{\text{lqw}}$ of at least 10 Myr. Planets with the longest duration,
$\tau_{\text{lqw}}\,>$ 5 Gyr, are indicated in yellow. Within the range of
considered core masses, the values of $\tau_{\text{lqw}}$ have the same trends
with respect to envelope masses and semi-major axes. At orbits within $\sim$10
AU long-term $\tau_{\text{lqw}}$, indicated by the yellow data-points, are
distributed over envelope masses of $10^{-4}$ to $10^{-6}$. For envelopes
smaller than $\sim 10^{-5}$ the surface temperatures are temperate, similar to
Earth’s. The stellar radiation is expected to be the dominant source of energy
and there is a scaling relation between the received flux (through the semi-
major axis) and the envelope mass for long-term $\tau_{\text{lqw}}$. Beyond
$\sim$10 AU liquid water conditions follow the cooling of the interior.
Planets with small envelope masses have liquid water conditions relatively
early while planets with more massive envelope reach liquid water conditions
later in the evolution.
In addition to $\tau_{\text{lqw}}$ we show a case where an additional
constraint is imposed: the surface temperature must remain below 400 K. This
is based on the temperature limit for life on Earth [26]. The bottom right
panel of Figure 1 shows the results for this case for a planet with a 3
$M_{\oplus}$ core.

###  0.1 Including atmospheric loss

The results presented so far do not include the effect of atmospheric loss.
This is important for small planets close to their host stars since they are
much more vulnerable to stellar radiation [18, 19, 14]. We apply the
evaporation model discussed in the Methods to study which planets in the
previous result would be stable against thermal atmosphere loss.
Figure 2 in the methods shows how the envelope mass changes with time with
hydrodynamic escape (top) and Jeans loss (bottom). The Jeans escape model only
has a significant effect on the mass of envelopes around small core masses of
1.5 $M_{\oplus}$. For the Jeans model we show the highest considered exosphere
temperature of 2,000 K . This is relatively high as exosphere temperatures for
H-He compositions are estimated at $\sim$1000 K [27]. Since a higher exosphere
temperature leads to a higher mass loss rate, we consider this as an upper
limit to Jeans escape. Lower values of 300 K and 1000 K were also considered,
but we find almost no effect on the envelope’s mass evolution. Even for the
cases in which Jeans escape removes some of the planetary envelope, we still
find a similar $\tau_{\text{lqw}}$, suggesting that the results presented in
Figure 1 are robust if escape occurs in the Jeans regime. The XUV-driven
hydrodynamic model, however, can significantly affect the evolution of small
close-in planets over a wide range of parameters, as shown in Figure 2.
The results when the hydrodynamic escape model is included are shown in Figure
3. In this case we find that there are no long-term liquid water conditions
possible on planets with a primordial atmosphere within 2 AU. Madhusudhan et
al. 2021[20] found that for planets around sun-like stars, liquid water
conditions are allowed at a distance of $\sim$1 AU. We find that the pressures
required for liquid water conditions between 1 and 2 AU are too low to be
resistant against atmospheric escape, assuming that the planet does not
migrate at a late evolutionary stage.

###  0.2 Unbound Planetary Mass Objects

The cases labelled as ’Unbound’ shown in Figure 1 and 3 correspond to planets
that are detached from their host star. Planets can be ejected after formation
by gravitational interaction [28]. Such unbound planets might be very common
[29]. Without the influence of a host-star, these planets are not affected by
post main-sequence evolution, which might terminate the duration of habitable
conditions for bound planets. Instead, the habitable conditions would end when
an internal heat source can no longer provide enough energy. Note that unbound
planets are no longer protected by the stellar heliosphere and therefore
galactic cosmic-rays and gamma ray could affect the planet’s structure and its
potential habitability [30].
We simulate such cold planets on a longer timescale and investigate whether
$\tau_{\text{lqw}}$ persists beyond the life-time of a sun-like star. The
results are shown in Figure 4. The simulations are performed up to 100 Gyr,
when most of the radioactive nuclei have decayed and the intrinsic luminosity
is nearly zero. The different results are caused by the longer integration
time than for the unbound cases in Figures 1 and 3.
We find that many unbound planets are too hot shortly after their formation to
host liquid water but do have the right conditions at later times and for long
$\tau_{\text{lqw}}$. In this case the core mass is an important factor for
$\tau_{\text{lqw}}$, since we assume that the radiogenic heat source scales
with core mass. The envelope’s mass influences the planetary cooling
efficiency and therefore has a smaller, but still significant, effect on
$\tau_{\text{lqw}}$. Planets with cores more massive than 5 $M_{\oplus}$ can
have liquid water conditions lasting over 50 Gyr for envelopes with masses of
$\sim$0.01 $M_{\oplus}$. The longest duration for liquid-water is found to be
84 Gyr for a planet with 10 $M_{\oplus}$ core and 0.01 $M_{\oplus}$ envelope.
The current set-up of our intrinsic luminosity model limits
$\tau_{\text{lqw}}$ of unbound planets by the half-life of the radio-active
components in the core, assuming a chondritic abundance. A potential planet
with radioactive components with a longer half-life could therefore have an
even longer $\tau_{\text{lqw}}$. Many of the inferred $\tau_{\text{lqw}}$
values for unbound planets are much longer than the age of the universe. It
should be noted, however, that it takes $\sim$10 Gyrs of cooling before these
conditions are reached. Consequently it could be that these types of planets
are currently too hot for liquid water but will be in a far future.

Our results indicate that habitable conditions underneath H-He dominated
atmospheres might be long-lasting and prevalent. Planets that retain their
primordial atmosphere should have a relatively simple formation and evolution
path compared to a secondary atmosphere. However, this is under a lot of
assumptions and simplifications on the formation and evolution.
It is commonly assumed that habitable planets require a negative climate
feedback to remain persistently habitable [31, 22, 23]. However, this work
suggests that the right set of initial conditions for H2-rich super-Earths can
alone lead to persistently temperate surface conditions over Gyr timescales.
There are no accurate predictions on the occurrence of super-Earth sized
planets with these initial conditions, but it is likely enough that these
alternatively habitable planets constitute a significant fraction of the
habitable worlds in the galaxy.
We made the assumption that the combination of core masses and envelope masses
of these initial conditions can form. Theory [32] and simulations [33, 11]
indicate more massive envelopes are likely to form beyond the snow-line.
Studies of observed close-in super-Earths suggest that these planets consist
of H-He envelopes of several percent in mass [34, 35]. Rogers & Owen [13]
showed that the inferred post-formation H-He mass fraction peaks at $\sim$4%,
but extends well into the range needed for long $\tau_{\text{lqw}}$
(M${}_{\text{env}}\,\leq\,10^{-2.5}\,M_{\oplus}$ ). The same work showed that
formation models predict too high envelope mass fractions, motivating the
improvement of planet formation models. Less-massive envelopes could be the
result of additional mechanisms, such as core-powered mass-loss [36, 37] or
collisions [38, 39]. We suggest that future studies should investigate whether
the planetary evolution of unbound planets differs from that of bound planets,
and if so, how it may affect their potential habitability. A detailed study on
the formation likelihood of planets with liquid water conditions underneath a
H-He envelopes that are observable at current time is beyond the scope of this
research and we hope to address it in future studies.
Future works should investigate in detail the formation likelihood of planets
with the right initial conditions. The possibility of their formation around
M-dwarfs should also be considered since these stars have a prolonged pre-
main-sequence with higher UV fluxes. Planets around such stars are expected to
lose more of their primordial envelope and water [40, 41]. Therefore, future
work should consider the initial volatile abundances of such planets in order
to predict the water mass fraction at different evolutionary stages [42, 43].
In addition, while a sufficient amount of water left after atmospheric escape
is required, the amount of water cannot be too high in order to prevent the
formation of ice layers between the silicate core and the liquid water. The
exchange of nutrients would then be inhibited [44, 45] or at least limited in
the possible scenario that the ice layer is fully convective [46].
Another simplification is that the planets accrete a solar-composition
envelope and remain this composition. However, processes during the evolution
can increase the metallicity. This can lead to a different total greenhouse
effect, with contributions of $\text{CO}_{2}$ as well as collision induced
absorption by $\text{H}_{2}-\text{N}_{2}$ and $\text{H}_{2}-\text{CO}_{2}$
[47, 40, 48, 49]. We have explored the sensitivity of our result to different
compositions in the Supplementary Materials.
Life on the type of planet described in this work would live under
considerably different conditions than most life on Earth. The surface
pressures in our results are on the order of 100-1000 bar, the pressure range
of oceanic floors and trenches. There is no theoretical pressure limit on life
and some of the extremest examples on Earth’s biosphere thrive at $\sim$500
bar [50]. The habitats also receive a negligible amount of direct sunlight,
and therefore photosynthesis would not be an optional mechanism to provide to
a metabolisms. Chemoautotrophic life on Earth [51] would be a more likely
analogue to possible life. Earth is inhabited by lifeforms that might be
adapted to life underneath a H-He envelope [52]. Since chemotrophic life arose
before photoautotrophic lifeforms, it can exist independently as it presumably
did on Earth [53]. However it is not trivial if the emergence of life could
happen on such planets in a similar way as on Earth, given that the planets we
simulated spend a longer time of their evolution significantly too hot [54].
Furthermore, the advent of photosynthesis on Earth introduced a much more
productive metabolism not dependent on preexisting chemical energy gradients.
Consequently, planets dominated by photosynthetic life likely produce more
readily observed signatures of their inhabitation. The lack of sunlight means
there would also be no temporal variation from a day-night cycle or season.
This would lead to stable conditions (which could influence the kind of
lifeforms that are best adapted[55, 2].) Another property that can affect
planetary habitability is the presence of a magnetic field [56]. It is yet to
be determined whether the planets we consider in this study are expected to
generate a dynamo and we hope to address this in future research.
More recent interpretations of planetary habitability do not only correspond
to harboring life, but also for the ability to detect it from Earth [57].
Currently it is still challenging to observe small planets at large radial
distances, but great advancement is expected from future telescope missions.
The relatively large scale-height of H-He atmospheres could make it possible
for instruments like JWST and Ariel or the Extremely Large Telescope to detect
and characterize biomarkers in such atmospheres [52]. While life in a
$H_{2}$-dominated atmosphere environment could produce biomarkers[58], one
challenge is that habitats that lack photosynthesis life do not produce a
chemical disequilibrium, but rather destroy it by its metabolism [59]. Future
studies should predict if certain chemical disequilibria are biomarkers or
anti-biomarkers for these specific habitats. Because of the importance of the
system’s temporal evolution, age determination by PLATO [60] is also critical
for the assessment of planetary habitability. The Roman space telescope using
gravitational microlensing could detect colder exoplanets and even unbound
planets [61, 62] to constrain their population. We therefore expect that our
understanding of this exoplanetary population and its potential habitability
would significantly improve in the near future.

We simulate the gaseous envelope using a spherically symmetric model that
solves the structure equations of a planet’s interior assuming hydrostatic
equilibrium [63, 64, 65]:

$\frac{\mathrm{d}m}{\mathrm{~{}d}r}=4\pi r^{2}\rho$ (1)
$\frac{\mathrm{d}P}{\mathrm{~{}d}r}=-\frac{Gm}{r^{2}}\rho$ (2)
$\frac{\mathrm{d}\tau}{\mathrm{d}r}=\kappa_{\mathrm{\text{R}}}\rho$ (3)
$\frac{\mathrm{d}l(r)}{\mathrm{~{}d}r}=0$ (4)

This relates the local cumulative mass $m$ to the radius $r$ within the
planet. $P$ and $\rho$ are the local pressure and density, respectively, and
$G$ is the gravitational constant. $l$ is the local intrinsic luminosity: the
energy flux going through a shell of radius $r$. Contributions to the
luminosity due to contraction and cooling of both the envelope and the iron-
silicate core (see Figure 5), are included as well as radiogenic heating (see
Linder et al. 2019[65]). The implication of Equation 4 is that the luminosity
does not vary through the gaseous envelope. This simplification is valid when
the envelope has a small contribution to the total luminosity compared to the
core (see Intrinsic Luminosity).
The temperature gradient at high optical depth $\tau$ is represented either by
the adiabatic or radiative gradient, using the Schwarzschild criterion [66,
67]. In case of a convective region we use the adiabatic gradient:

$\frac{d\,T}{d\,P}=\frac{T}{P}\left(\frac{\text{ln}\,T}{\text{ln}\,P}\right)_{\text{adi}}$
(5)

At low optical depths, a double-gray atmosphere model [68] is adapted. This
atmosphere model uses the assumption that the spectral energy distribution of
received stellar radiation is mostly optical light, while the outgoing
radiation of the planet is mostly in the infrared. The temperature as a
function of optical depth is given by:

$T(\tau)^{4}=\frac{3T_{\mathrm{\text{int}}}^{4}}{4}\left\\{\frac{2}{3}+\tau\right\\}+\frac{3T_{\mathrm{\text{eq}}}^{4}}{4}\left\\{\frac{2}{3}+\frac{2}{3\gamma}\left[1+\left(\frac{\gamma\tau}{2}-1\right)e^{-\gamma\tau}\right]+\frac{2\gamma}{3}\left(1-\frac{\tau^{2}}{2}\right)E_{2}(\gamma\tau)\right\\}$
(6)

The equilibrium temperature $T_{\text{eq}}$ is defined as
$T_{\text{eq}}=T_{\star}\sqrt{\frac{R_{\star}}{2a}}\left(1-A_{\text{B}}\right)^{\frac{1}{4}}$,
where $T_{\star}$ and $R_{\star}$ are the effective temperature and radius of
the star, respectively, $a$ is the distance between planet and star and
$A_{\text{B}}$ is the Bond albedo fixed to Jupiter’s value of 0.343. The
intrinsic heat, $T_{\text{int}}$, is defined as
$T_{\text{int}}=\left(\frac{L_{\text{int}}}{\sigma_{B}4\,\pi r^{2}}\right)$.
Here, $L_{\text{int}}$ is the total intrinsic luminosity and $\sigma_{B}$ is
the Stefan-Boltzmann constant. $E_{2}(\gamma\tau)$ is a form of the
exponential integral $E_{n}(z)\equiv\int_{1}^{\infty}t^{-n}e^{-zt}dt$, where
$n=2$. The greenhouse effect is parametrised as the ratio between the optical
opacity and the infrared opacity as
$\gamma=\kappa_{\text{vis}}/\kappa_{\text{IR}}$. The level of greenhouse
effect is thus contained in the parameter $\gamma$. We use a parametrisation
of $\gamma$ (Table 2 in Jin et al. 2014 [64]). Since most of our planets have
equilibrium temperatures that are lower than the low limit in their table
($T_{\text{eq, min}}=260$ K), we extrapolate the table for lower $\gamma$
values. It should be noted that the term in Equation 6 containing the
intrinsic heat becomes the dominant term in the extrapolated regime of low
equilibrium temperatures, and therefore different gamma values have a small
effect on the resulting temperature. We therefore find that the exact value of
$\gamma$ in the extrapolated regime has a negligible effect on the inferred
state of water underneath the atmosphere. The atmosphere then reduces to a
classical Eddington grey atmosphere.
At high optical depths, the temperature in convectively stable regions is
calculated with diffusive radiative energy transport as:

$\frac{\mathrm{d}T}{\mathrm{~{}d}r}=-\frac{3\kappa_{R}\rho
l}{64\pi\sigma_{\text{B}}T^{3}r^{2}}.$ (7)

The transition between the low optical depth regime (Equation 6) and high
optical depth regime (Equation 7) is determined by two boundaries, between
which we interpolate the temperature. These two boundaries are given by:
$\tau=10/\sqrt{3}\gamma$ and $\tau=100/\sqrt{3}\gamma$. The transitions occurs
at 10 bar and 100 bar, which means that planets which have an envelope of
$\lessapprox\,10^{-5}M_{\oplus}$ are completely in the low optical depth
regime.
The model uses Rossland-mean opacities for a solar or a scaled-solar
composition gas. One source of opacities are the molecular, grain-free tables
[69]. These are tables in temperature-density space and depend on the
metallicity [M/H], covering a range of [M/H] = 0 to 1.7 (close to 50 times
solar). They include CIA that is expected to dominate at low temperatures. The
lower limit on temperature for this opacity table is 75 K. Our simulations
sometimes contain planets with lower temperatures in certain regions of the
atmosphere (down to 50 K). In this case we extrapolate the opacities as
$\propto T^{2}$ as suggested by the temperature dependency close to 75 K. To
assess the consequences of the extrapolation we also repeated our simulations
with a fixed opacity in the extrapolated regime. This made no difference to
our final result, since the planets that we find to allow liquid water do not
have a significant part of their atmosphere in the extrapolated regime.
We use a non-ideal equation of state [70] for H-He [15]. For
$\text{H}_{2}\text{O}$ we use AQUA [71].

###  0.3 Solid Core

Our internal structure model [72] solves Equation 1 and 2 for the solid core.
We assume for all cores a silicate to iron ratio of 2:1, similar to Earth, and
assume there are no ices. It seems likely that planets forming beyond the
snowline accrete a significant amount of water ice. On the other hand, several
mechanisms dehydrate the planetary building blocks [73]. Including ices in our
model has several possible consequences: First, it changes the mass-radius
relationship. Since the radii of water-rich planets are more sensitive to
temperature we expect that the core radii will vary more during the evolution.
Second, at a fixed total core mass, the presence of water would reduce the
silicate mantle mass and thus the amount of estimated radiogenic heating (see
Importance of Intrinsic Heat). Lastly it would allow more energy to be stored
in the core which is released over longer timescales, due to the fact that ice
has a higher specific heat capacity than silicates. We plan to account for
different planetary compositions and investigate their effect on the planetary
evolution in future research.
The core model uses a modified polytropic EOS [74] that relates the density
$\rho$ and the pressure $P$ as:

$\rho(P)=\rho_{0}+cP^{n}$ (8)

$\rho_{0}$, $c$ and $n$ are parameters specific to the material [74]. We use
MgSiO3 for silicates [72]. The external pressure that the envelope imposes on
the core is included, although this does not have a significant effect on our
calculations, as the most massive envelopes considered are only 0.01
$M_{\oplus}$.

###  0.4 Intrinsic luminosity

The intrinsic luminosity of the planet and its temporal evolution are included
in the simulation. We estimate the initial value of this intrinsic luminosity
and its evolution in the following way.
First, we estimate the initial intrinsic luminosity that the planet has
shortly after formation. We use an analytical fit [75], which uses the results
of numerous planet formation simulations. The intrinsic luminosity of planets
in the formation simulation is calculated by solving the 1D structure
equations, including the heating from cooling and contractions as well as
accretion of gas and planetesimals. The analytical approximation yields the
intrinsic luminosity as it depends on core mass, envelope mass and age of the
planet. We fix the starting age of the planet at the same value as the start
of our simulation, namely 20 Myr. The initial luminosity is then found by
interpolating in core and envelope mass. Our simulations also show how much of
the luminosity is generated in the core or the envelope (an example of which
is show in Figure 4). This is also described in detail in Linder et al.[65].
It is found that luminosity contribution of the H-He envelope is much smaller
than the contribution of the solid core, in agreement with earlier work [34,
65]. This justifies the assumption of a uniform luminosity in the envelope
made by Equation 4. If a significant part of the total luminosity comes from
envelope cooling and contraction, this assumption would not be valid.
The second contribution used to calculate the total intrinsic luminosity is
based on radiogenic heating [72]. We model the heat due to the decay of
radioactive nucleids by assuming an initial abundance of radioactive material
(${}^{40}\text{K}$, ${}^{232}\text{Th}$, and ${}^{238}\text{U}$) of a
chondritic composition. Then we assume that the total abundance of the
radiogenic heating scales with the mass of the silicate mantle. The radiogenic
luminosity becomes:

$L_{\text{radio}}(t)=L_{\text{radio},\oplus}(t)\cdot\frac{M_{\text{mantle}}}{M_{\text{mantle},\oplus}}$
(9)

Once the initial luminosity is found, its temporal evolution is given by
energy conservation, i.e. by the condition that the total energy difference
between two points in time is equivalent to the luminosity radiated between
them [63]. The radioactive component evolves according to the decay time of
the radioactive nucleids.
Figure 5 shows an example of the evolution of the intrinsic luminosity of a 3
$M_{\oplus}$ core and a 0.001 $M_{\oplus}$ envelope for a planet that receives
a negligible amount of stellar radiation. After 300 Myr the radiogenic
luminosity becomes the dominant source of intrinsic heat. Since the abundance
of radioactive nucleids in other planets is unknown [76], we also consider
cases where these are a factor of ten larger/smaller, which are presented in
the Supplementary Materials under Importance of Internal Heat.
Figure 6 shows how the atmosphere structure evolves in 3 planets from 50 Myr
to 5 Gyr. All planets have a core mass of 3 $M_{\oplus}$ and an envelope of
$10^{-3}$ M⊕. No atmosphere escape model is implemented, the evolution is due
to the host star and the intrinsic luminosity. The planets at 1 and 10 AU
remain too hot for liquid water conditions, while the planet at 100 AU reached
liquid water conditions after 1 Gyr.

###  0.5 Atmosphere loss

We model atmospheric loss through two distinct thermal escape models: one
based on Jeans escape and one based on hydrodynamic escape.
Jeans escape is applicable for planets that remain in hydro-static equilibrium
everywhere [77, 78, 79, 27]. The fraction of escaping particles depends on the
escape velocity, particle number density and the temperature
($T_{\text{exo}}$) at the exobase. For the escape velocity we use the total
radius and total mass of the planet and therefore assume the escape velocity
at the exobase is the same as at the outer radius. We also assume a 100%
hydrogen composition for the number density and neglect that helium particles
are heavier and thus have lower velocities at the same temperature when
assuming hydro-static equilibrium. This results in an overestimate of the
escape rate, but using an upper limit of Jeans escape serves our goal of
investigating whether Jeans escape has a significant effect on liquid water
duration.
The exosphere temperature is complex to estimate, since it is determined by
the absorption of X-ray and extreme-UV, and by the composition at the
exosphere [80]. H-He dominated atmosphere are expected to have a relatively
warm exosphere of $\sim$1000 K [27], while planets dominated by e.g.
$\text{CO}_{\text{2}}$, $\text{H}_{2}\text{O}$ or $\text{N}_{2}$ should have a
colder $T_{\text{exo}}$ of $\sim$300 K [81]. We remain agnostic about the
specific exosphere temperature. Instead we put the exosphere temperature to a
fixed value during the simulation ($T_{\text{exo},\,\text{min}}$) and use a
range (300 - 2000 K ). If the temperature of the outer radius of the
atmosphere (where $\tau=2/3$) is warmer than the assumed exosphere
temperature, we allow the exosphere temperature to be increased to this value
at any moment in the evolution. However, we find this never to be the case.

The second escape model is based on hydrodynamic escape [64, 82]. Hydrodynamic
escape is mostly driven by X-ray radiation at early times and later by
extreme-UV radiation [83]. We assume X-ray dominated evaporation until a
threshold of EUV flux is reached [83, 64]. The evolution of the stellar X-ray
and EUV luminosity are from solar models [84].
During X-ray-dominated evaporation the loss is energy limited [85]. We
estimate the mass-loss by [86, 64]:

$\dot{M}_{\text{X-ray}}=\epsilon\frac{\pi
F_{\text{X-ray}}R_{2/3}^{3}}{G\,M_{\text{pl}}\,K_{\text{tide}}}$ (10)

$F_{\text{X-ray}}$ is the received X-ray flux, $R_{2/3}$ the radius at optical
depth of 2/3, $M_{\text{pl}}$ the planet’s total mass and $K_{\text{tide}}$ is
an extra factor to account for a higher mass-loss when planets have their
Roche-lobe boundary close to the surface [87]. Since we simulate relatively
small planets at large distances, $K_{\text{tide}}$ has a negligible effect.
In contrast to earlier work [64, 82] we now parameterise the efficiency factor
$\epsilon$ by the escape velocity [9].
The extreme-UV-dominated loss can also be energy limited. In this case the
mass-loss is also calculated with Equation 10, but using the extreme UV flux
and the radius where the optical depth in UV is 1[88]. When the extreme UV
radiation is high, a significant amount of the heat can be lost to radiation.
In this case the radiation/recombination limited (rr-limited) description [88]
is used. Our model computes both the energy-limited and rr-limited mass-loss
and applies the lowest value.

###  0.6 Liquid water conditions

During the simulations we determine whether there are liquid water conditions
by checking the pressure and temperature at the bottom of our simulated
atmosphere. We compare these to the phase-diagram of water to determine if a
$\text{H}_{2}\text{O}$ water layer could be in the liquid phase [89]. We
define the duration of liquid water conditions as the time that water is
permitted in the liquid phase without interruptions, referred to as
$\tau_{\text{lqw}}$. In some of our results we will apply, with mentioning, an
extra constraint to the liquid water conditions: that the surface temperature
remains below 400 K . This is based on the observation on Earth that
terrestrial live thrives best at temperatures of around 300 K and an upper
limit for the chemistry of life is estimated at $\sim$400 K [26].

###  0.7 Comparison to Pierrehumbert & Gaidos (2011) [18] (P&G)

To test our model in time independent calculations, we first compare our
models to their results in Figure 7. Similarly to this work we calculate the
atmosphere mass that is necessary for a surface temperature of 280 K. In
addition to our own model COMPLETO we use another atmosphere model: PETITcode
[90]. PETITcode is also a 1-dimensional radiative-convective equilibrium code,
but with a wavelength dependent treatment of radiative transport. It uses
specific chemical abundances and assumes a chemical equilibrium. At low
temperatures, however, non-equilibrium effects can play an important role in
the atmosphere. It assumes a uniform gravity throughout the atmosphere.
The simulations of P&G used a constant gravity of 1700 cm/$\text{s}^{2}$,
without an interior heat source. To match this we set the core mass at 3
$M_{\oplus}$, which resulted in a radius of 1.32 $R_{\oplus}$. The intrinsic
heat value was set to a negligible temperature (1 K internal temperature for
PETITcode, $10^{-6}\,L_{\text{jup}}$ for COMPLETO). Furthermore P&G[18] used a
pure $H_{2}$ composition. In PETIT we set the metallicity to 0.01 times solar,
so that there is predominantly hydrogen and a non-negligible fraction of
helium, though we do not expect this helium to influence the opacities. In
COMPLETO we use solar composition, since [69] only provide tabulated opacities
for [M / H] =0 to 1.7. The age of the host star is set at 5 Gyr.
The results of the PETITcode models agree very well with those of P&G[18]. The
results are also shown with the inclusion of an intrinsic heat source of 35 K
, which is the average intrinsic temperature we find for a 3 $M_{\oplus}$
planet when applying our intrinsic luminosity model. These results show that,
as expected, at far semi-major axis this intrinsic heat source can reduce the
amount of atmosphere that is needed to warm the surface. Quite good agreement
is found inside of about 7 AU. Outside, higher pressures are found, likely a
consequence of extra absorption of incoming visual light in the solar
composition atmosphere relative to more H-He.

###  0.8 Illustrative cases of evolving models

Figure 8 shows the evolution of surface pressures and temperatures for
different models, and whether the pressure and temperature allow liquid water
(solid lines) or not (dashed lines). We consider typical low-mass planets
($M_{\text{core}}$ 1.5 to 8 $M_{\oplus}$) with envelopes of $10^{-5}$ to
$10^{-3}$ $M_{\oplus}$. The duration of continuous liquid water conditions,
$\tau_{\text{lqw}}$, is therefore the integration of the (longest) solid line.
Figure 8 (a) shows the effect of changing the semi-major axis. Planets
relatively close to their star at 2 AU are too hot for the first 500 Myr when
the combination of high irradiation (from a small semi-major axis) and a high
intrinsic luminosity (from a young age) lead to high temperatures. These
planets develop liquid water conditions at a later stage, while the same
planet further away (at 6 AU or 10 AU) can host an ocean earlier. The effect
of changing the atmospheric mass is shown in Figure 8 (b), where the core mass
and semi-major axis are fixed. This directly influences the pressure at the
bottom of the atmosphere. An envelope of $10^{-3}\,M_{\oplus}$ leads to
temperatures that are too high. A smaller envelope of $10^{-4}\,M_{\oplus}$
has liquid water conditions after 70 Myr, at high pressures and temperatures.
An even smaller atmosphere of $10^{-5}M_{\oplus}$ has liquid water conditions
at the beginning of the evolution, but then becomes too cold. In Figure 8(c)
we show the effect of the core mass. Changing the core mass has two different
effects on our model. First of all it will result in different gravities in
the atmosphere model with consequences on the radiative-convective profile.
Secondly, the intrinsic luminosity depends on the core mass.

###  0.9 Dependence on Model Parameters

In this section we test the sensitivity of our results to the assumptions we
made. We show in Figure 9 how different model parameters influence duration of
liquid water conditions in comparison to the nominal case presented in Figure
1.
The effect of including a 0.5 ice mass fraction in the core is shown in Figure
59 (b). The nominal case assumes no ice in the core. The biggest consequence
of assuming the core is half made up of ice is that it leads to an estimated
mantle that is half as massive and therefore the amount of estimated
radiogenics that is a factor two lower.

###  0.10 Intrinsic Luminosity

The second parameter we investigate is the intrinsic heat. The value and
evolution of the intrinsic heat is an important, if not the main contributor
to habitable surface temperatures for the planets considered in this work. We
change our intrinsic heat model in two separate ways: by changing the initial
luminosity due to cooling and contraction at the start of our simulation, and
by changing the magnitude of the radiogenic luminosity.
A change of the initial luminosity (excluding the radiogenic component) has
little influence on the long-term surface conditions. Planets with a more
(less) initial luminosity will start with more (less) entropy. This results in
more (less) efficient cooling and contractions. In our simulations, it takes
around 100 Myr for planets with different initial luminosities to converge to
the same luminosity. Since we find $\tau_{\text{lqw}}$ on the timescale of
billions of years, it remains unaffected even when we change the initial
luminosity by a factor of 4.
The radiogenic luminosity component is independent to the rest of the
intrinsic luminosity sources. As we show in Figure 5, this component becomes
the dominant term on long timescales and therefore we expect it to be critical
on the determination of $\tau_{\text{lqw}}$. We perform our simulations, but
multiply (divide) the radiogenic heat source by a factor 10. This would
correspond with a planet that has 10 times more or less radioactive material
as a fraction of the mantle core.
In Figure 9 (c) and (d) we show how $\tau_{\text{lqw}}$ is influenced by
changes in $L_{\text{radio}}$ of a factor 10. Up to $\sim$10 AU,
$\tau_{\text{lqw}}$ seems mostly unaffected. The exception are planets with an
envelope of $\sim$ $10^{-5}$ at $\sim$7 AU that are too hot for liquid water
in the nominal case, but cool enough in the low $L_{\text{radio}}$ case. At
distances on the order of $\sim$10-100 AU, $\tau_{\text{lqw}}$ depends on
$L_{\text{int}}$. We only took into consideration the effect of a different
magnitude of $L_{\text{radio}}$, which would correspond to a different total
amount of radioactive particles. We did not take into account that a different
composition of radioactive material could additionally lead to a different
half-life.

###  0.11 Variations in Composition

It is likely that planets which retain their primordial atmosphere still have
varying degrees of metalicity compared to a solar metallicities. Also the
relative abundances of elements will vary. Given the uncertainty in ranges of
possible exoplanet compositions, we investigate how they potentially affect
our results by changing two general parameters: the greenhouse parametrization
$\gamma$ in Equation 6 and the opacities.
The ratio between visible and thermal opacities
$\gamma=\frac{\kappa_{\text{vis}}}{\kappa_{\text{IR}}}$, incorporates the
greenhouse effect in the atmosphere model. Ideally, the value of $\gamma$
takes into account condensation of any molecule that absorbs in the infra-red,
with contributions of all possibly relevant species such as $\text{CH}_{4}$
and $\text{CO}_{2}$.
In the nominal case, $\gamma$ is extrapolated from Table 2 in Jin et al. 2014
[64] from the effective temperature. Therefore the value of $\gamma$ changes
in time. As discussed in Methods, the atmosphere model has a small dependency
on $\gamma$ when the intrinsic temperature is significantly larger than the
equilibrium temperature. This is the case for young planets and/or those that
are at large semi major-axis.
Figure 9 (e) and (f) show the effect of fixing the value of $\gamma$ to 0.01
or 0.001. It is in line with the double-grey atmosphere model that the value
of $\gamma$ is an important parameter for determining $\tau_{\text{lqw}}$ at
shorter distances ($>$ 3 AU), but does not effect the planets at larger
distances.
Finally we investigate the effect of enhanced infra-red opacities in Figure 9
(g). We increase the calculated opacities by a factor 10. Enhanced infra-red
opacities could be the result of relatively more greenhouse gases present in
the envelope, such as CH4, NH3 or H2O. Our results show that enhanced infra-
red opacities lead to larger surface pressures and therefore liquid water
conditions would be found on planets with smaller envelopes. In reality
different greenhouse gasses would absorb at different wavelengths. To account
for the contributions of specific enhanced greenhouse gasses would require a
more complicated model.

###  0.12 Comparison to Planet Population Synthesis

Given the uncertainty in planet formation it is not possible to give an
accurate prediction of the occurrence rate of planets that satisfy the right
initial conditions leading to liquid water conditions. Nevertheless we use the
New Generation Planet Population Synthesis (NG76)[91, 11] to compare the
predicted initial conditions to those that favour long-term liquid water
conditions. NG76 consists of 1000 systems, all formed around a 1 M⋆ and
integrated to 100 million years. After the integration time, 34635 embryos are
formed.
Most planets accrete envelopes that are too big for liquid water conditions,
making a potential surface temperature too hot. Nevertheless planets with the
required initial conditions do occur. Out of the 34635 embryo’s formed, 6851
have envelope masses of $10^{-6}<M_{\text{env}}/M_{\oplus}<0.01$ and 5160 have
$10^{-6}<M_{\text{env}}/M_{\oplus}<0.001$. Out of these, most have a smaller
core mass below 1 M⊕, with only 187 planets having a core mass between 1 and
10 M⊕. There are several reasons why the envelope mass can be overestimated in
the NGPPS. First of all it applies 1-dimensional, hydrostatic models. More
realistic models find reduced gas accretion because of gas exchange [92, 93].
The assumed grain opacities can also be an underestimation, leading to too
efficient gas accretion. Other assumptions on the formation and evolution
could also lead to smaller primordial envelopes than estimated in NGPPS, for
example core-powered mass-loss [36, 37] or collisions [38, 39].
The fact that the NGPPS predict (too) high envelope masses is in line with the
finding that planet formation theory overestimates envelope masses in
comparison to observations [13]. It is therefore desirable to identify the
missing physical/chemical processes that lead to the higher envelope masses.
We also suggest that more sophisticated planet formation and structure models
are required to estimate the occurrence rate of planets with liquid water at
current time.

Data from the simulation is public and available at
https://github.com/mollous/Data_Liquid_Water_Conditions.

The used code is available upon reasonable request with the corresponding
author.

This work has been carried out within the framework of the National Centre of
Competence in Research PlanetS supported by the Swiss National Science
Foundation. The authors acknowledge the financial support of the SNSF.

M. M. L. generated the data, created figures and wrote the manuscript. C.M.
Provided the initial code which was modified by M. M. L. All authors analyzed
the data and discussed the implications of the model assumptions, as well as
the results.

The authors declare that they have no competing financial interests.

Correspondence and requests for materials should be addressed to M.M.L.
(email: marit@ics.uzh.ch).

## Figures

Figure 1: Duration of liquid water conditions for planets at a wide range of
semi-major axis (1 AU to 100 AU) and envelope masses ($10^{-1.8}$ to $10^{-6}$
$M_{\oplus}$). Planets receive insolation based on the luminosity evolution of
a sun-like star. Panels a-c correspond to core masses of 1.5, 3 and 8
$M_{\oplus}$ respectively. The duration of the total evolution is 8 Gyr. The
color of a grid-point indicates how long there were continuous surface
pressures and temperatures allowing liquid water, $\tau_{\text{lqw}}$. These
range from 10 Myr (purple) to over 5 Gyr (yellow). Grey crosses correspond to
cases with no liquid water conditions lasting longer than 10 Myr. Atmospheric
loss is not considered in these simulations. Panel d shows results for planets
with a core mass of 3 $M_{\oplus}$, but including the constraint that the
surface temperature must remain between 270 and 400 K. Every panel contains an
’unbound’ case where the distance is set to 106 AU and solar insolation has
become negligible. Figure 2: The evolution of the mass of the simulated
envelope for two different atmosphere loss models: Hydrodynamic escape (top)
and Jeans escape (bottom). The default values of the model are a core mass of
3 $M_{\oplus}$ and envelope mass of $10^{-4}\,M_{\oplus}$ at a distance of 6
AU. The top panel shows how the envelope loss is affected by different semi-
major axis, the middle panel shows the effect of different envelope masses and
the bottom panel shows the effect of different core masses. Dashed lines show
when a layer of water between the envelope and core would not be in the liquid
phase, solid lines show when it would. Hydrodynamical escape can significantly
reduce or completely evaporate the envelope of some planets, which influences
the duration of liquid water conditions. The Jeans escape model, even when a
high exosphere temperature is chosen of 2000 K , shows only a somewhat
significant loss for a small core mass of 1.5 $M_{\oplus}$. Figure 3: Same as
Figure 1, but when including the effect of hydrodynamic atmospheric photo-
evaporation. The y-axis shows the envelope mass after 8 Gyr. Escape inhibits
liquid water conditions by removing the atmosphere for close-in planets with
low initial envelope masses. Lower core masses are more affected. Figure 4:
Duration of liquid water conditions ($\tau_{\text{lqw}}$) on unbound planets.
The planets are simulated with different core masses and envelope masses. The
longest duration simulated was 84 billion years for a 10 $M_{\oplus}$ core and
a 0.01 $M_{\oplus}$ envelope. If planets with a primordial atmosphere can host
liquid water the duration can be much longer on unbound planets, since the
internal heat source can evolve slower than the host-star. Contour lines
indicate the start of liquid water conditions for planets with
$\tau_{\text{lqw}}>$ 100 Myr. Figure 5: Evolution of the intrinsic luminosity
(given in Jupiters current luminosity[94] of 3.35$\cdot\,10^{25}$ erg s-1) for
a planet with a core of 3 $M_{\oplus}$ and a 0.001 $M_{\oplus}$ envelope. In
the beginning of the evolution the cooling and contraction of the core are
dominant components of the total luminosity. After $\sim$300 Myr the
radiogenic luminosity dominates. Figure 6: Evolution of the atmosphere
structure of 3 planets between 50 Myr and 5 Gyr, without atmosphere loss. All
planets have a core mass of 3 $M_{\oplus}$ and an envelope of
$10^{-3}\,M_{\oplus}$ and are at 1, 10 and 100 AU separation from their sun-
like host-star. The bar in the bottom of the figure shows the state of water
for the temperature range of the figure and at a fixed pressure of about 1
kbar, the resulting surface pressure. Figure 7: Comparison between the
PETITcode models and the results from Pierrehumbert & Gaidos [18]. The figure
shows the mass of a primordial, hydrogen-dominated atmosphere that is
necessary for a surface temperature of 280 K, as a function of the semi-major
axis around a sun-like star. The P&G models had a surface gravity of 1700
cm/$\text{s}^{2}$, which was constant through the coordinates of the model. To
match this value at the surface, the models simulate a planet with a core mass
of 3 $M_{\oplus}$. Figure 8: Evolution of the temperature at the envelope-core
boundary for planets with different semimajor-axes (a), envelope masses (b)
and core masses (c). The default values used are a distance of 6 AU, an
envelope of $10^{-4}\,M_{\oplus}$ and a 3 $M_{\oplus}$ core. Dashed lines show
when a layer of water between the envelope and core would not be in the liquid
phase, solid lines show when it would. The simulations ran from 20 Myr to 10
Gyr. Our stellar model has no post Main-Sequence luminosity evolution and
therefore we only consider our results until our model leaves the Main-
Sequence at 8 Gyr. Atmospheric escape was not taken into account. Figure 9:
Comparison of $\tau_{\text{lqw}}$ for the nominal case and cases where a
parameter has been changed. The semi-major axis and envelope masses are varied
over a range of values on the x-axis and y-axis respectively. All cases have a
total core mass of 3 M⊕. (a) The nominal case that is presented in Figure 1.
(b) The core has a 50% ice mass fraction, rather than 0% in the nominal case.
In (c) and (d) the amount of assumed radiogenic abundances is scaled by a
factor 10 lower (higher), leading to a 10 times weaker (stronger) radiogenic
luminosity. In (e) and (f) the ratio of visible to thermal opacities is fixed
to values of 0.01 and 0.001. For close-in planets where the irradiation is the
main source of energy these are relatively high (low, respectively) compared
to the nominal case. Finally in (g) the opacities are increased by a factor
10.

## References

## References

  * [1] Lammer, H. _et al._ What makes a planet habitable?  _Astron. & Astroph. Review._ 17, 181–249 (2009).
  * [2] Irwin, L. N. & Schulze-Makuch, D.  The Astrobiology of Alien Worlds: Known and Unknown Forms of Life.  _Universe_ 6, 130 (2020).
  * [3] Hussmann, H., Sohl, F. & Spohn, T.  Subsurface oceans and deep interiors of medium-sized outer planet satellites and large trans-neptunian objects.  _Icarus_ 185, 258–273 (2006).
  * [4] Bar-On, Y. M., Phillips, R. & Milo, R.  The biomass distribution on Earth.  _Proc Natl Acad Sci U S A_ 115, 6506–6511 (2018).
  * [5] McMahon, S. & Parnell, J.  The deep history of earth’s biomass.  _Journal of the Geological Society_ 175, 716 – 720 (2018).
  * [6] Edwards, K. J., Becker, K. & Colwell, F.  The Deep, Dark Energy Biosphere: Intraterrestrial Life on Earth.  _Annual Review of Earth and Planetary Sciences_ 40, 551–568 (2012).
  * [7] Mayor, M. & Queloz, D.  A Jupiter-mass companion to a solar-type star.  _Nature_ 378, 355–359 (1995).
  * [8] Fulton, B. J. _et al._ The California-Kepler Survey. III. A Gap in the Radius Distribution of Small Planets.  _AJ_ 154, 109 (2017).  1703.10375.
  * [9] Wu, Y.  Mass and Mass Scalings of Super-Earths.  _ApJ_ 874, 91 (2019).  1806.04693.
  * [10] Otegi, J. F., Bouchy, F. & Helled, R.  Revisited mass-radius relations for exoplanets below 120 M⊕.  _A &A_ 634, A43 (2020).  1911.04745.
  * [11] Emsenhuber, A. _et al._ The New Generation Planetary Population Synthesis (NGPPS). II. Planetary population of solar-like stars and overview of statistical results.  _A &A_ 656, A70 (2021).  2007.05562.
  * [12] Owen, J. E., Shaikhislamov, I. F., Lammer, H., Fossati, L. & Khodachenko, M. L.  Hydrogen Dominated Atmospheres on Terrestrial Mass Planets: Evidence, Origin and Evolution.  _Space Sci. Rev._ 216, 129 (2020).  2010.15091.
  * [13] Rogers, J. G. & Owen, J. E.  Unveiling the planet population at birth.  _MNRAS_ 503, 1526–1542 (2021).  2007.11006.
  * [14] Lammer, H. _et al._ Origin and loss of nebula-captured hydrogen envelopes from ‘sub’- to ‘super-Earths’ in the habitable zone of Sun-like stars.  _MNRAS_ 439, 3225–3238 (2014).  1401.2765.
  * [15] Borysow, A.  Collision-induced absorption coefficients of H2 pairs at temperatures from 60 K to 1000 K.  _A &A_ 390, 779–782 (2002).
  * [16] Frommhold, L.  _Collision-induced Absorption in Gases_ (Cambridge University Press, 2006).
  * [17] Stevenson, D. J.  Life-sustaining planets in interstellar space?  _Nature_ 400, 32 (1999).
  * [18] Pierrehumbert, R. & Gaidos, E.  Hydrogen Greenhouse Planets Beyond the Habitable Zone.  _ApJl_ 734, L13 (2011).  1105.0021.
  * [19] Wordsworth, R.  Transient conditions for biogenesis on low-mass exoplanets with escaping hydrogen atmospheres.  _Icarus_ 219, 267–273 (2012).  1106.1411.
  * [20] Madhusudhan, N., Piette, A. A. A. & Constantinou, S.  Habitability and Biosignatures of Hycean Worlds.  _ApJ_ 918, 1 (2021).  2108.10888.
  * [21] Seager, S.  Exoplanet Habitability.  _Science_ 340, 577–581 (2013).
  * [22] Kasting, J. F., Whitmire, D. P. & Reynolds, R. T.  Habitable Zones around Main Sequence Stars.  _Icarus_ 101, 108–128 (1993).
  * [23] Kopparapu, R. K. _et al._ Habitable Zones around Main-sequence Stars: New Estimates.  _ApJ_ 765, 131 (2013).  1301.6674.
  * [24] Kopparapu, R. K. _et al._ Habitable Zones around Main-sequence Stars: Dependence on Planetary Mass.  _ApJl_ 787, L29 (2014).  1404.5292.
  * [25] Baraffe, I., Homeier, D., Allard, F. & Chabrier, G.  New evolutionary models for pre-main sequence and main sequence low-mass stars down to the hydrogen-burning limit.  _A &A_ 577, A42 (2015).  1503.04107.
  * [26] Takai, K. _et al._ Cell proliferation at 122°C and isotopically heavy CH4 production by a hyperthermophilic methanogen under high-pressure cultivation.  _Proceedings of the National Academy of Science_ 105, 10949–10954 (2008).
  * [27] Tian, F.  Atmospheric escape from solar system terrestrial planets and exoplanets.  _Annual Review of Earth and Planetary Sciences_ 43, 459–476 (2015).  URL https://doi.org/10.1146/annurev-earth-060313-054834.  https://doi.org/10.1146/annurev-earth-060313-054834.
  * [28] Lissauer, J. J.  Timescales for planetary accretion and the structure of the protoplanetary disk.  _Icarus_ 69, 249–265 (1987).
  * [29] Strigari, L. E., Barnabè, M., Marshall, P. J. & Blandford, R. D.  Nomads of the Galaxy.  _MNRAS_ 423, 1856–1865 (2012).  1201.2687.
  * [30] Ávila, P. J. _et al._ Presence of water on exomoons orbiting free-floating planets: a case study.  _International Journal of Astrobiology_ 20, 300–311 (2021).  2106.08349.
  * [31] Walker, J. C. G., Hays, P. B. & Kasting, J. F.  A negative feedback mechanism for the long-term stabilization of the earth’s surface temperature.  _J. Geophys. Res._ 86, 9776–9782 (1981).
  * [32] Rafikov, R. R.  Atmospheres of Protoplanetary Cores: Critical Mass for Nucleated Instability.  _ApJ_ 648, 666–682 (2006).  astro-ph/0405507.
  * [33] Mordasini, C.  _Planetary Population Synthesis_ , 143 (Springer Nature, 2018).
  * [34] Lopez, E. D. & Fortney, J. J.  Understanding the Mass-Radius Relation for Sub-neptunes: Radius as a Proxy for Composition.  _ApJ_ 792, 1 (2014).  1311.0329.
  * [35] Rogers, L. A.  Most 1.6 Earth-radius Planets are Not Rocky.  _ApJ_ 801, 41 (2015).  1407.4457.
  * [36] Ginzburg, S., Schlichting, H. E. & Sari, R.  Core-powered mass-loss and the radius distribution of small exoplanets.  _MNRAS_ 476, 759–765 (2018).  1708.01621.
  * [37] Misener, W. & Schlichting, H. E.  To cool is to keep: residual H/He atmospheres of super-Earths and sub-Neptunes.  _MNRAS_ 503, 5658–5674 (2021).  2103.09212.
  * [38] Liu, S.-F., Hori, Y., Lin, D. N. C. & Asphaug, E.  Giant Impact: An Efficient Mechanism for the Devolatilization of Super-Earths.  _ApJ_ 812, 164 (2015).  1509.05772.
  * [39] Biersteker, J. B. & Schlichting, H. E.  Atmospheric mass-loss due to giant impacts: the importance of the thermal component for hydrogen-helium envelopes.  _MNRAS_ 485, 4454–4463 (2019).  1809.06810.
  * [40] Ramirez, R. M. & Kaltenegger, L.  The Habitable Zones of Pre-main-sequence Stars.  _ApJl_ 797, L25 (2014).  1412.1764.
  * [41] Luger, R. _et al._ Habitable Evaporated Cores: Transforming Mini-Neptunes into Super-Earths in the Habitable Zones of M Dwarfs.  _Astrobiology_ 15, 57–88 (2015).  1501.06572.
  * [42] Raymond, S. N., Scalo, J. & Meadows, V. S.  A Decreased Probability of Habitable Planet Formation around Low-Mass Stars.  _ApJ_ 669, 606–614 (2007).  0707.1711.
  * [43] Ogihara, M. & Ida, S.  N-Body Simulations of Planetary Accretion Around M Dwarf Stars.  _ApJ_ 699, 824–838 (2009).  0904.4543.
  * [44] Alibert, Y.  On the radius of habitable planets.  _A &A_ 561, A41 (2014).  1311.3039.
  * [45] Noack, L. _et al._ Water-rich planets: How habitable is a water layer deeper than on Earth?  _Icarus_ 277, 215–236 (2016).
  * [46] Cowan, N. B. & Abbot, D. S.  Water Cycling between Ocean and Mantle: Super-Earths Need Not Be Waterworlds.  _ApJ_ 781, 27 (2014).  1401.0720.
  * [47] Wordsworth, R. & Pierrehumbert, R.  Hydrogen-Nitrogen Greenhouse Warming in Earth’s Early Atmosphere.  _Science_ 339, 64 (2013).
  * [48] Batalha, N., Domagal-Goldman, S. D., Ramirez, R. & Kasting, J. F.  Testing the early Mars H2-CO2 greenhouse hypothesis with a 1-D photochemical model.  _Icarus_ 258, 337–349 (2015).  1507.02569.
  * [49] Ramirez, R. M. & Kaltenegger, L.  A Volcanic Hydrogen Habitable Zone.  _ApJl_ 837, L4 (2017).  1702.08618.
  * [50] Dalmasso, C. _et al._ Thermococcus piezophilus sp. nov., a novel hyperthermophilic and piezophilic archaeon with a broad pressure range for growth, isolated from a deepest hydrothermal vent at the mid-cayman rise.  _Systematic and Applied Microbiology_ 39, 440–444 (2016).  URL https://www.sciencedirect.com/science/article/pii/S0723202016300753.
  * [51] Edwards, B., Mugnai, L., Tinetti, G., Pascale, E. & Sarkar, S.  An Updated Study of Potential Targets for Ariel.  _AJ_ 157, 242 (2019).  1905.04959.
  * [52] Seager, S., Huang, J., Petkowski, J. J. & Pajusalu, M.  Laboratory studies on the viability of life in H2-dominated exoplanet atmospheres.  _Nature Astronomy_ 4, 802–806 (2020).  2005.01668.
  * [53] Wächtershäuser, G.  The case for the chemoautotrophic origin of life in an iron-sulfur world.  _Origins of Life and Evolution of the Biosphere_ 20, 173–176 (1990).
  * [54] Islas, S., Velasco, A. M., Becerra, A., Delaye, L. & Lazcano, A.  Hyperthermophily and the origin and earliest evolution of life.  _International Microbiology_ 6, 87 – 94 (2003).  URL https://doi.org/10.1007/s10123-003-0113-4.
  * [55] Jørgensen, B. & Boetius, A.  Jorgensen bb, boetius a.. feast and famine-microbial life in the deep-sea bed. nat rev microbiol 5: 770-781.  _Nature reviews. Microbiology_ 5, 770–81 (2007).
  * [56] Meadows, V. S. & Barnes, R. K.  _Factors Affecting Exoplanet Habitability_ , 2771–2794 (Springer International Publishing, Cham, 2018).  URL https://doi.org/10.1007/978-3-319-55333-7_57.
  * [57] Forget, F., Turbet, M., Selsis, F. & Leconte, J.  Definition and Characterization of the Habitable Zone.  In _Habitable Worlds 2017: A System Science Workshop_ , vol. 2042 of _LPI Contributions_ , 4057 (2017).
  * [58] Seager, S., Bains, W. & Hu, R.  Biosignature Gases in H2-dominated Atmospheres on Rocky Exoplanets.  _ApJ_ 777, 95 (2013).  1309.6016.
  * [59] Wogan, N. F. & Catling, D. C.  When is chemical disequilibrium in Earth-like planetary atmospheres a biosignature versus an anti-biosignature? Disequilibria from dead to living worlds.  _The Astrophysical Journal_ 892 (2020).
  * [60] Rauer, H. _et al._ The PLATO 2.0 mission.  _Experimental Astronomy_ 38, 249–330 (2014).  1310.0696.
  * [61] Penny, M. T. _et al._ Predictions of the WFIRST Microlensing Survey. I. Bound Planet Detection Rates.  _ApJs_ 241, 3 (2019).  1808.02490.
  * [62] Johnson, S. A. _et al._ Predictions of the Nancy Grace Roman Space Telescope Galactic Exoplanet Survey. II. Free-floating Planet Detection Rates.  _AJ_ 160, 123 (2020).  2006.10760.
  * [63] Mordasini, C., Alibert, Y., Klahr, H. & Henning, T.  Characterization of exoplanets from their formation. I. Models of combined planet formation and evolution.  _A &A_ 547, A111 (2012).  1206.6103.
  * [64] Jin, S. _et al._ Planetary Population Synthesis Coupled with Atmospheric Escape: A Statistical View of Evaporation.  _ApJ_ 795, 65 (2014).  1409.2879.
  * [65] Linder, E. F. _et al._ Evolutionary models of cold and low-mass planets: cooling curves, magnitudes, and detectability.  _A &A_ 623, A85 (2019).  1812.02027.
  * [66] Schwarzschild, M.  _Structure and evolution of the stars._ (Princeton University Press, 1958).
  * [67] Hansen, C. J., Kawaler, S. D. & Trimble, V.  _Stellar interiors : physical principles, structure, and evolution_ (Springer-Verlag, 2004).
  * [68] Guillot, T.  On the radiative equilibrium of irradiated planetary atmospheres.  _A &A_ 520, A27 (2010).  1006.4702.
  * [69] Freedman, R. S. _et al._ Gaseous Mean Opacities for Giant Planet and Ultracool Dwarf Atmospheres over a Range of Metallicities and Temperatures.  _ApJs_ 214, 25 (2014).  1409.0026.
  * [70] Saumon, D., Chabrier, G. & van Horn, H. M.  An Equation of State for Low-Mass Stars and Giant Planets.  _ApJs_ 99, 713 (1995).
  * [71] Haldemann, J., Alibert, Y., Mordasini, C. & Benz, W.  AQUA: a collection of H2O equations of state for planetary models.  _A &A_ 643, A105 (2020).  2009.10098.
  * [72] Mordasini, C. _et al._ Characterization of exoplanets from their formation. II. The planetary mass-radius relationship.  _A &A_ 547, A112 (2012).  1206.3303.
  * [73] Lichtenberg, T. _et al._ A water budget dichotomy of rocky protoplanets from 26Al-heating.  _Nature Astronomy_ 3, 307–313 (2019).  1902.04026.
  * [74] Seager, S., Kuchner, M., Hier-Majumder, C. A. & Militzer, B.  Mass-Radius Relationships for Solid Exoplanets.  _ApJ_ 669, 1279–1297 (2007).  0707.2895.
  * [75] Mordasini, C.  Planetary evolution with atmospheric photoevaporation. I. Analytical derivation and numerical study of the evaporation valley and transition from super-Earths to sub-Neptunes.  _A &A_ 638, A52 (2020).  2002.02455.
  * [76] Wang, H. S., Morel, T., Quanz, S. P. & Mojzsis, S. J.  Europium as a lodestar: diagnosis of radiogenic heat production in terrestrial exoplanets. Spectroscopic determination of Eu abundances in $\alpha$ Centauri AB.  _A &A_ 644, A19 (2020).
  * [77] Öpik, E. J. & Singer, S. F.  Distribution of Density in a Planetary Exosphere. II.  _Physics of Fluids_ 4, 221–233 (1961).
  * [78] Chamberlain, J. W.  Planetary coronae and atmospheric evaporation.  _Planet. Space Sci._ 11, 901–960 (1963).
  * [79] Shu, F. & Kranakis, E.  _The Physical Universe: An Introduction to Astronomy_.  Series of books in astronomy (University Science Books, 1982).  URL https://books.google.ch/books?id=NfhrH6FS7TYC.
  * [80] Lammer, H. _et al._ Atmospheric Loss of Exoplanets Resulting from Stellar X-Ray and Extreme-Ultraviolet Heating.  _ApJl_ 598, L121–L124 (2003).
  * [81] Pierrehumbert, R. T.  _Principles of Planetary Climate_ (Cambridge University Press, 2010).
  * [82] Jin, S. & Mordasini, C.  Compositional Imprints in Density-Distance-Time: A Rocky Composition for Close-in Low-mass Exoplanets from the Location of the Valley of Evaporation.  _ApJ_ 853, 163 (2018).  1706.00251.
  * [83] Owen, J. E. & Jackson, A. P.  Planetary evaporation by UV & X-ray radiation: basic hydrodynamics.  _MNRAS_ 425, 2931–2947 (2012).  1206.2367.
  * [84] Ribas, I., Guinan, E. F., Güdel, M. & Audard, M.  Evolution of the Solar Activity over Time and Effects on Planetary Atmospheres. I. High-Energy Irradiances (1-1700 Å).  _ApJ_ 622, 680–694 (2005).  astro-ph/0412253.
  * [85] Watson, A. J., Donahue, T. M. & Walker, J. C. G.  The dynamics of a rapidly escaping atmosphere: Applications to the evolution of Earth and Venus.  _Icarus_ 48, 150–166 (1981).
  * [86] Jackson, A. P., Davis, T. A. & Wheatley, P. J.  The coronal X-ray-age relation and its implications for the evaporation of exoplanets.  _MNRAS_ 422, 2024–2043 (2012).  1111.0031.
  * [87] Erkaev, N. V. _et al._ Roche lobe effects on the atmospheric loss from “Hot Jupiters”.  _A &A_ 472, 329–334 (2007).  astro-ph/0612729.
  * [88] Murray-Clay, R. A., Chiang, E. I. & Murray, N.  Atmospheric Escape From Hot Jupiters.  _ApJ_ 693, 23–42 (2009).  0811.0006.
  * [89] Wagner, W. & Pruß, A.  The iapws formulation 1995 for the thermodynamic properties of ordinary water substance for general and scientific use.  _Journal of Physical and Chemical Reference Data_ 31, 387–535 (2002).  URL https://doi.org/10.1063/1.1461829.  https://doi.org/10.1063/1.1461829.
  * [90] Mollière, P., van Boekel, R., Dullemond, C., Henning, T. & Mordasini, C.  Model Atmospheres of Irradiated Exoplanets: The Influence of Stellar Parameters, Metallicity, and the C/O Ratio.  _ApJ_ 813, 47 (2015).  1509.07523.
  * [91] Emsenhuber, A. _et al._ The New Generation Planetary Population Synthesis (NGPPS). I. Bern global model of planet formation and evolution, model tests, and emerging planetary systems.  _A &A_ 656, A69 (2021).  2007.05561.
  * [92] Cimerman, N. P., Kuiper, R. & Ormel, C. W.  Hydrodynamics of embedded planets’ first atmospheres \- III. The role of radiation transport for super-Earth planets.  _MNRAS_ 471, 4662–4676 (2017).  1707.08079.
  * [93] Moldenhauer, T. W., Kuiper, R., Kley, W. & Ormel, C. W.  Steady state by recycling prevents premature collapse of protoplanetary atmospheres.  _A &A_ 646, L11 (2021).  2102.00939.
  * [94] Guillot, T. & Gautier, D.  Giant Planets.  In Schubert, G. (ed.) _Treatise on Geophysics_ , 529–557 (2015).


# ReProHRL: Towards Multi-Goal Navigation in the Real World using
Hierarchical Agents

Tejaswini Manjunath, Mozhgan Navardi, Prakhar Dixit, Bharat Prakash, Tinoosh
Mohsenin

###### Abstract

Robots have been successfully used to perform tasks with high precision. In
real-world environments with sparse rewards and multiple goals, learning is
still a major challenge and Reinforcement Learning (RL) algorithms fail to
learn good policies. Training in simulation environments and then fine-tuning
in the real world is a common approach. However, adapting to the real-world
setting is a challenge. In this paper, we present a method named Ready for
Production Hierarchical RL (ReProHRL) that divides tasks with hierarchical
multi-goal navigation guided by reinforcement learning. We also use object
detectors as a pre-processing step to learn multi-goal navigation and transfer
it to the real world. Empirical results show that the proposed ReProHRL method
outperforms the state-of-the-art baseline in simulation and real-world
environments in terms of both training time and performance. Although both
methods achieve a 100% success rate in a simple environment for single goal-
based navigation, in a more complex environment and multi-goal setting, the
proposed method outperforms the baseline by 18% and 5%, respectively. For the
real-world implementation and proof of concept demonstration, we deploy the
proposed method on a nano-drone named Crazyflie with a front camera to perform
multi-goal navigation experiments.

## Introduction

Reinforcement Learning (RL) is a popular method used for a variety of
applications like robotics, autonomous navigation, games, energy management,
and so on. Autonomous drone navigation is one such widely used application
that enables many capabilities like search and rescue (zuluaga2018deep), gas
leakage detection (duisterhof2021sniffy), etc. Despite achieving great success
in such applications, training the agent is a compute-intensive and time-
consuming process because of the large amount of trial and error required to
learn new policies. Recent works in Sim2Real (truong2021bi; zhang2021sim2real;
zhang2020sim2real) have shown promising results in learning navigation and
manipulation policies where the Reinforcement learning based policy is trained
in simulators and then transferred to real-world environments. However, such
Sim2Real adaptation is still not trivial due to the lack of high-fidelity
replication of real-world scenes and the imprecise simulation models.

With progress in computer vision research, Unmanned Aerial Vehicles (UAVs)
with computer vision capabilities have gained a lot of interest
(duisterhof2021tiny; navardi2022optimization). UAVs equipped with high-
resolution cameras can be used in real-world environments for a number of
applications like transportation systems, search and rescue, agricultural and
land surveys, safety, etc. Recent proposals such as You Only Look Once (YOLO)
(benjumea2021yolo) have achieved efficient and fast object detection by
obtaining cost-free region proposals sharing full-image convolutional features
with the detection network. One of the advantages of using YOLO-based object
detection is the availability of pre-trained models trained on millions of
real-world images. Only minimal fine-tuning is necessary to fit the model to
different use cases.

Training in simulation and deploying in the real world, usually called
Sim2Real (doersch2019sim2real), is a popular way to deploy such agents in the
real-world and production environments. A lot of existing work in image-based
robotic navigation tends to use raw images to learn policies, whether they are
in the simulation or the real world. But, the visual gap between the
simulation and the real world makes Sim2Real deployment challenging.
Therefore, it is important to address this challenge to have a Ready for
Production (RePro) agent. Another challenge is that real-world tasks are
complex and have longer horizons. But high-level tasks can be broken down into
smaller sub-tasks by leveraging the natural hierarchy that exists in most
cases. Hierarchical Reinforcement Learning (HRL) agents are used to decompose
complex tasks into multiple sub-tasks (prakash2021interactive; li2020hrl4in;
staroverov2020real; prakash2021semantic; prakashhierarchical). The agent
consists of a high-level policy that provides a sub-goal and a low-level
policy that takes the sub-goal and performs primitive actions in the
environment.

In this paper, we propose a framework named ReProHRL that trains Ready for
Production Hierarchical Reinforcement Learning agents in simulation which can
be easily deployed in the real world. The ReProHRL agent has a fixed high-
level module that provides sub-goals and a low-level policy that performs
those sub-goals. To address the challenges of learning with high-dimensional
observations like images, we use an object detection model, YOLO
(benjumea2021yolo), to pre-process the raw images. We extract object
representation in the form of the Bounding box (Bbox) coordinates to improve
generalization and for robust sim-to-real transfer. Few works
(taghibakhshi2021local; nguyen2019reinforcement) have explored this idea, but
none have successfully deployed it in a real-world environment for UAVs in the
multi-goal setting. Moreover, we show that our approach can be generalized to
multiple goals. We one-hot encode the goal vector to represent the goals
(skills) and their relations.

One of the crucial requirements for goal-based navigation is obstacle
avoidance. While other methods exist that use an image, depth map, or bounding
box, we use a lidar sensor(available in the drone) as a laser ranger.

The main contributions of this paper can be summarized as follows:

  * •

A generalized framework to train hierarchical agents in simulation and deploy
in the real world for a variety of multi-goal tasks.

  * •

Using YOLOv5 as an object detector to pre-process images and greatly reduce
the state space to RL.

  * •

A low-level policy that is trained to reach goals using the pre-processed
inputs and distance sensors. Additionally, a controller that switches sub-
goals to enable sequential goal completion in virtual and physical
environments.

  * •

A multi-goal simulation environment using Airsim and unreal engine which
enables RL research for deployment on UAVs.

  * •

Deployment and evaluation of the proposed technique on a drone named Crazyflie
for multi-goal navigation in the real-world environment.

To evaluate the proposed approach, we measured the training success rate of
the HRL agent by applying the ReProHRL approach in simple and complex
environments with multiple objects. Moreover, we compare the proposed approach
with state-of-the-art works (navardi2022toward; shiri2022efficient). The
experimental results show that the ReProHRL approach reaches a 100% success
rate in different environments. We also deploy the proposed approach in a
real-world environment on a drone named Crazyflie (giernacki2017crazyflie).

The rest of the paper is organized as follows: First, we discuss related work,
followed by our proposed approach in Section Proposed Methodology. We show the
experimental setup in detail in Section Experimental Setup. Further in Section
Experimental Results, we discuss our findings based on experiments before we
summarize the proposed work in Section Conclusion.

Figure 1: Illustration of Reinforcement Learning process showing how the agent
interacts with the environment to maximize the reward. Figure 2: Illustration
of the architecture for a hierarchical agent used in this work. The High-level
controller which acts as a state machine generates a sub-goal when the agent
completes an episode. The Low-level Policy outputs an action at every step and
the agent performs the action in the environment to reach the goals.

## Related Work

There is considerable work shown in high-fidelity simulators for visual
navigation in large-scale indoor settings where control policies are trained
for end-to-end navigation. Some methods try to solve the problem of multi-goal
detection and navigation for complex tasks in the real world. But there are
not many works that use pre-processing on images to solve the problem of
Sim2Real for production in the real world. We discuss the related work in the
following subsections.

### Vision Based Navigation

Existing research in Vision-Based Navigation tends to pass sensory input into
a meaningful state before sending it to a Reinforcement Learning (RL) agent,
with the expectation that the robot can implicitly learn the navigation
problem through recursive trials. (Zhu2017TargetdrivenVN) proposed to solve
this problem by finding the similarity between different objects using Siamese
Network (Koch2015SiameseNN). The work of (inproceedings) incorporated a meta-
learning approach where the agent learns a self-supervised interaction loss
during inference to avoid collisions. However, none of these methods use any
prior information or semantic context. Moreover, the learned features are
expected to vary with the test setting, thereby limiting the scope of this
method in unseen environments.

There are a few works on autonomous drone navigation in the real world. In
reducing the sim2real gap, the work by (ALLAMAA2022385; 9196730;
wang2022tacto) shows promising results but requires high-fidelity simulators
and high simulation time. But policies trained in the simulated environment do
not generalize well in unknown environments. Therefore, purely vision-based
navigation seems to lack in production settings.

### Hierarchical Agent

Hierarchical goal-based tasks can be modeled as a Markov decision process
(MDP), consisting of a state-space $S$, an action space $A$, a reward function
$r$: $S$ × $A$ → $R$, an initial state distribution $s0$, and a transition
probability $p(st+1|st,at)$ that defines the probability of transitioning from
state $st$ and action $at$ at time $t$ to next state $st+1$. A high-level
illustration of the MDP process is shown in Figure 1.

(kaelbling1993learning) first proposed an algorithm addressing dynamically
changing goals in a multi-goal setting. For each goal object, the algorithm
learned a specific value function using Q-learning. More recently,
(pmlr-v37-schaul15) proposed Universal Value Function Approximators (UVFA), a
unique policy able to address an infinite number of goals by concatenating the
current state and goal state to feed to the network. This information could be
the textual embedding about the target (shiri2022efficient), an image
(fang2022target), or the position of the goal in the environment
(pmlr-v37-schaul15).

Multi-goal approaches prove better than simply training a policy per goal
because knowledge can be transferred between different goals using off-policy
learning. A naive solution to the multi-goal problem would be to concatenate
the state and goal data as a unique input to the network. But this has proven
to not be the best solution as the agent only sees a small subset of mappings
between states and goals (pmlr-v37-schaul15). But with reduced state space,
such mappings become easier. With our approach, the state space grows only by
a vector of five values for every object and is easily transferable to other
environments. Thus, allowing us to use the naive approach of universal Value
Function Approximators (UVFA) (pmlr-v37-schaul15). The overview of the same is
elaborated in Figures 2 and 3.

### Object Detection

Object Detection is used for classifying and localizing objects in a bounding
box. A classifier is prepared by training on a set of images in all different
methods for object detection. Object detectors are used generally to detect
one object out of all the other objects. The models are then evaluated on
unseen datasets, and the accuracy is measured accordingly.

There are two types of object detectors - namely, region-based, and region-
free methods. Region proposal-based methods include R-CNN, Faster R-CNN
(ren2015faster), etc., and classification-based methods include YOLO, single
shot detector (SSD) (liu2016ssd), etc.

In (navardi2022toward), our previous work, we used the DQN agent for single-
agent, single-goal navigation. Object detection was performed on grayscale
images using a classification-based method - YOLO. In this work, a success
rate of 92% was achieved for navigation towards a single object in simulation.
Similar results were observed in a real-world environment. This is shown in
Figure 6.

Our current work outperforms the previous work owing to the colored camera.
Most importantly, this work differs from the previous work, as here, the agent
learns to navigate towards multiple goals hierarchically, thus showing its
generalization towards multi-goal learning. While object detectors are mainly
used to detect objects, they can also be used for collision avoidance when
equipped with the right sensors (8919366). But we use four-point lidar
embeddings and equivalent distance sensors in the real world to get distances
from the obstacles in all four directions (left, right, front, back).

Figure 3: Details of Low-level Policy for goal-dependent navigation. The
Bounding box (Bbox) coordinates from pre-processed image and goal vector
(received from the High-level controller shown in Figure 2) are concatenated
with the lidar embedding (used for collision avoidance) as an input to the
low-level policy.  Figure 4: Three different examples for state-space showing
values of Boundary box (Bbox) coordinates, goal encoding, and lidar embedding.
The Final vector, state space, is the concatenated vector of the three
vectors.

## Proposed Methodology

As illustrated in Figures 2 and 3, we design a framework to train hierarchical
agents for multi-goal navigation. The proposed architecture consists of a
high-level controller that provides sub-goals and a low-level policy that uses
pre-processed inputs and lidar readings to perform these sub-goals. The
proposed architecture is explained in the following subsections.

### High-level Controller

The High-level controller in our method is used purely to give sub-goal
instructions to the Low-level policy. Although it can be trained to learn
different sequences (shu2018hierarchical), we’ve limited the scope of the
High-level controller to provide sub-goals from a fixed sequence and the
current observation.

The High-level controller relies on the current observation to provide sub-
goals to the Low-level planner. The images captured by the agent in the
environment are passed through an object detector(YOLOv5) to receive the
bounding box coordinates as shown in Figure 2.

In the simulation, we switch sub-goals on the successful completion of a sub-
task. A sub-task is considered successful when the agent reaches the sub-goal
within a distance of 0.1 meters. In a real-world environment, there is no way
of knowing the success distance without sensors. One way to address this issue
is by setting a minimum number of steps = k by when we know the agent can
reach the goal. This “k" can be found by a trial-and-error method in the real
world.

Figure 5: Simple and complex environments for both simulation and real-world
environments. The simple environment comprises the multi-goal objects and the
UAV. The complex environment has additional obstacles that need to be avoided
by the agent while navigating the environment.

A more realistic way of deciding this factor is by defining a precondition
based on the size of the image. Since we use bounding box coordinates, we can
use the maximum size of the Bounding box(Bbox) as the precondition. When the
Bounding box of the goal object reaches a certain size, the High-level-
controller can switch the sub-goal.

The high-level controller uses the latter method to switch sub-goals. Figure 4
describes some examples of Bounding box coordinates. Each Bounding box output
consists of 4 predictions: $x1,y1,x2,y2$. The $(x1,y1)$ corresponds to the x
and y coordinates of the top left corner of the rectangle. The $(x2,y2)$
corresponds to the x and y coordinates of the bottom right corner of the
rectangle. Therefore, $width=(x2-x1)$ and $height=(y2-y1)$. In our
experiments, the agent is closest to the objects when either the height or
width of the bounding box coordinate is at least 70% of the whole frame’s
height and weight respectively. In both simulation and real-world
environments, the agent operates on constraints of a maximum number of
$steps=50$ to reach the sub-goal. As illustrated in Figures 2 and 3, we design
a DQN-based deep learning model for the goal-dependent navigation task. The
model takes the goal as part of the input and enables the agent to learn a
series of different goals jointly. The proposed approach is also used to go to
a sequence of goals.

### Low-level Planner

The architecture for the Low-level planner is shown in Figure 3 which
comprises two modules - an object detection module and a collision avoidance
module. The details of the object detection module are described as follows:

First, the pre-processing step consists of a YOLOv5 model. The input image
from the environment is processed by the YOLO module to produce outputs in the
form of the Bounding box (Bbox) coordinates of the objects. YOLO splits the
input image into an $m$x$m$ grid, and each grid generates b bounding boxes and
confidence scores for those boxes.

Second, outputs from the YOLO model in the form of BBox coordinates (1x12) are
concatenated with the sub-goal vector(1x3) received from the High-level
controller to form a (1x15) vector for three objects.

In the collision avoidance module, the lidar sensor in the simulation
environment is used to train the DQN agent to avoid obstacles. This is
important because the object detection module only provides details about the
goal objects. This process is represented in Figure 3 in Collision Avoidance
Module. The lidar embeddings are then concatenated with the vector from the
object detection module to form the state space (1x19).

The final state space is the concatenated vector of the Bounding box
coordinates, goal vector, and lidar embeddings, which are inputs to the DQN
policy. We use Q-learning as it is more sample-efficient than policy gradient
and Monte Carlo value-based methods. The action space is right, left, and
straight.

$R_{t}=\begin{cases}R_{goal}\quad d_{t}\leq d_{p}\\\ R_{coll}\quad d_{o}\leq
d_{q}\\\ R_{fail}\quad step=max\\_step\\\ 0\end{cases}$ (1)

We tuned the reward function to learn to navigate to all three objects. The
reward function is shown above where $R\textsubscript{goal}$ =
$2~{}-~{}0.2~{}\times~{}(current\\_step/max\\_step)$ is the reward on reaching
the target position, $R\textsubscript{coll}$ = $-0.5$ is a negative reward
when the agent collides with obstacles in the environment.
$R\textsubscript{fail}$ = $-1.5$ is the penalty when the agent cannot reach
the goal object in $max\\_step=300$ and $0$ in other situations.
$d\textsubscript{t}$ is the distance between the goal and the drone at time
$t$, and $d\textsubscript{o}$ is the distance from the obstacle.
$d\textsubscript{p}$ and $d\textsubscript{q}$ are predefined distances used to
check if the target is reached or a collision has occurred.

Figure 6: Success rate result of agent training in an environment with one
object using grayscale (navardi2022toward) and RGB camera (ReProHRL). The
result illustrates the agent with an RGB camera achieves 100% success rate in
about 20K steps.

## Experimental Setup

An overview of our simulation environment, training methods, and real-world
environment will be discussed in this Section. The performance of the RL agent
with the proposed configuration is tested in both simulation and real-world
environments with different complexities. The air-learning environment in
simple and complex settings is shown in Figure 5.

### Simulation Environment

In this section, we elaborate on our simulation environment. Our experiments
are based on the simulation environment shown in Figure 5. Air-Learning
(krishnan2021air), which is based on Airsim is specifically designed for
aerial robotics and autonomous UAVs in 3D environments. It generates high-
fidelity photo-realistic environments with domain randomization and flight
physics model for the UAVs to fly in. One of the other advantages of Air-
Learning environment is that it comes with an energy module that can be used
to measure the energy expended by UAVs. This is especially useful when we
transfer the models to resource-constrained UAVs since they come with a
limited amount of battery. We designed an environment with a 25x15m room and
multiple goals in it. In addition to the environment with only goal objects,
we also evaluate the performance of the proposed methodology in a complex
environment shown in Figure 5 (2).

Figure 7: Success rate achieved by the proposed approach (ReProHRL) in the
Air-Learning environments with respect to the number of episode steps. For the
baseline, we used the language-guided approach presented in
(shiri2022efficient). We reported results for simple environments with (a) one
object, (b) two objects, and (c) three objects. For three objects shown in
(c), the DQN agent trained using curriculum learning shows training for one
object for 20K steps, two objects for 70K steps, and finally for three objects
until 150K steps. Figure 8: Success rate graph for agent trained to navigate
to a single-goal in a complex environment with multiple obstacles that is
shown in Figure 5 (2).

### Training Details

We first trained YOLOv5 on the training data sampled from both Air-learning
and real-world environments. We sampled a total of 4k images. Although the
pre-trained models can be used for real-world object detection, to improve the
accuracy and performance of our model, we manually annotated the images and
trained them on the YOLOv5 model. These images are trained with YOLOv5s to fit
the low-powered drone. The final map score for YOLOv5s is 0.96 which is over
0.05 more than when trained with Gray-scale images.

### Non-Curriculum Learning

We train the DQN agent to navigate to one or two objects in both simple and
complex environments shown in Figure 5 (1) and (2) respectively. To determine
the performance of our algorithm compared to the baseline, we train each agent
until we reach a satisfactory success rate using non-curriculum learning. We
randomize the position of the goal every episode to be anywhere in the arena.
The checkpoints are saved every 10000 steps and we use the last saved
checkpoint for evaluation and further experiments.

### Curriculum Learning

To improve the performance of the DQN agent in a multi-goal setting, we use a
curriculum learning methodology where the task is progressively made difficult
- the agent first learns to navigate to a single object and then to two
objects and finally three. As seen in Figure 7 (c), the success rate curve
goes up quickly before dropping down since the goal in the initial steps was
to navigate to a single object.

### Real-world Deployment

The Crazyflie platform features easily replaceable parts, so we use Crazyflie
to bridge the ‘Sim2Real’ gap and to test directly in our real-world
experiments. Crazyflie weighs 27 grams and has dimensions 92 mm×92 mm×29 mm.
Stackable chips like AIdeck, a Multi-ranger deck, a location module, etc., can
be used for additional processing, location, and other functionalities. The
processing is done on an ARM CortexM4 operating at 160 MHz on an STM32F405
microcontroller, with 192 KB of SRAM and 1024 KB of flash memory. We use the
same Yolo model which was trained on different environments for our real-world
evaluation.

Deployment of models on UAVs is a challenging task especially on resource-
constrained UAVs since they need to accomplish the given task with limited
energy. Additionally, onboard memory is scarce (shiri2022efficient), and RL
models are computationally intensive. Therefore, we need to design the models
carefully. As the size of the drone decreases, its battery capacity also
decreases, which aggravates the problems. Therefore, we carefully monitor the
energy both in simulation and real-world environments. We give a higher
penalty to the agent if it fails to complete the task to encourage faster task
completion. On average, the agent completes the task under 25 steps in a
15mx25m room operated under actions of lower time duration.

In the real world, to avoid collisions, we used a multi-ranger deck, an
extension deck for Crazyflie. This gives the distance from the obstacle to the
drone in all four directions, similar to our lidar readings. In the real
world, we performed experiments in a 5mx5m room similar to our simulation
environment as shown in Figure 5 (3) and (4), simple and complex environments,
respectively. The actions and action duration of Crazyflie is tuned for better
efficiency in the real world.

## Experimental Results

This section discusses the evaluation of the proposed method on the Air-
learning environment, an open-source setting within the Unreal game engine
that enables navigation in near-photo-realistic indoor scenes and real-world
settings. The performance is evaluated quantitatively in terms of success rate
- the number of times the agent reaches the goal over the total number of
episodes. For each episode, we randomly initialized the starting location of
the agent in the same simulation environment with a different number of goals,
i.e., one, two, and three in a simple and complex environment.

Methods | Number of Goals | Environment
---|---|---
Simple Env | Complex Env
Success Rate (%) |  Training time (steps) |  Success Rate (%) |  Training Time (steps)
Baseline | Single goal | 100% | 7K | 80% | 90K
Two goals | 95% | 25K | - | -
Three goals | - | - | - | -
ReProHRL | Single goal | 100% | 15K | 98% | 100K
Two goals | 100% | 22K | - | -
Three goals | 95% | 150K | - | -
Table 1: Summary of Success rate and Training time comparison between our
approach and the baseline (shiri2022efficient) all trained for under 150K
steps in Air-learning Environment shows that our approach outperforms the
baseline while taking fewer steps in multi-goal settings.

### Simulation Results

To see how our method compares to state-of-the-art baseline
(shiri2022efficient), an ablation study is made between the two approaches for
a different number of goals, i.e., one, two goals, and three goals in both
simple and complex environments. The baseline is a visual navigation-based
model trained in simple and complex environments. The baseline
(shiri2022efficient), one of our previous works encodes the goal information
that is in the form of English language instructions into a vector. Therefore,
state space comprises of a goal-state encoding similar to our approach.

The learning curves in Figure 7 show that our approach obtains better
performance than state-of-the-art in most cases, both in terms of convergence
speed and success rate. For single object navigation, both models performed
exceptionally due to the simpler task.In navigating towards two goal objects
that are selected at random, our approach reached a convergence to near 100%
success rate at 25k steps, and the baseline failed to reach 100% even after 1
million steps. This shows that our approach performs better in multi-goal
settings even with a pre-processing step. Finally, for the three-object
navigation setting, our method reached a 95% success rate in 150k steps. In
the complex setting shown in Figure 5 (2), with our approach, the success rate
reaches 98% at 100k steps but the stat-of-the-art baseline reaches only 80% as
shown in Figure 8.

The performance could be attributed to our reduced state space that can
capture the navigation-relevant association between state and goal and
encourage cooperation when training an agent to navigate towards multiple
goals.

For single and two-goal navigation settings, we trained the DQN agent using a
non-curriculum approach. But, to improve the training time for navigation in a
three-goal setting, we used a curriculum learning approach where we trained
for one object until the DQN agent learned to navigate to a single object in
20K steps. Next, we trained the agent for two goals until 70K steps. It
learned quickly for single-object navigation, but the success rate dropped low
when the complexity increased before the agent learned to navigate to all
three objects at 1.5 million steps. The success rate graph in Figure 7 (c)
reflects the same.

All models were trained to reach an acceptable success rate. But any training
above 1.5 million steps is time-consuming in high-fidelity simulators. The
Success Rate for all methods is summarized in Table 1.

### Performance in Real-world

In (navardi2022toward), a single object-based navigation model trained on
grayscale images in the simulation was deployed in the real-world environment.
It also shows that the drone reaches the goal as long as it is within a visual
distance from the object. With ReProHRL, the performance of the YOLO model is
better on account of RGB images used in training. Additionally, the model
generalizes well to unseen observations, which is vital for the production of
RL algorithms.

Though we observe an improvement in single-goal-based navigation, both in
terms of performance and time, the onboard energy proves insufficient for
multi-goal navigation as the drone takes around 30 seconds to 1 minute in a
5mx5m room to reach an object. Although the battery lasts up to 7 minutes on
Crazyflie, with the addition of 2 decks and additional processing, the energy
lasts at most for 3 minutes. In high-performance UAVs like DJI-Mavic Pro
(elkhrachy2021accuracy) that have 15 times more onboard energy than Crazyflie,
this would not be concerning.

## Conclusion

In this work, we proposed a Ready for Production Hierarchical Reinforcement
Learning (ReProHRL) agent scheme that enables the agent to learn the skill of
navigating to multiple goals hierarchically which can be deployed in the real
world. We introduced two components in our architecture: a High-level
controller and a Low-level planner to enhance sequential goal completion in
visual navigation. We validated the proposed approach by experimenting with
navigation tasks for multi-goal navigation and comparing it with two state-of-
art works. Experimental results in the Air-learning environment demonstrated
the superiority of the proposed architecture, ReProHRL, over existing methods
in goal-dependent navigation tasks. With the proposed approach, we achieved
near 100% success rate in the simple environment, multi-goal setting as well
as the complex environment, single-goal setting. The latter has 18%
improvement over the baseline (shiri2022efficient). These settings resulted in
a fast response time and adaptation to unseen environments. Moreover, we have
evaluated the proposed approach in physical environments by deploying it on a
drone named Crazyflie in a real room with multi objects.

## Acknowledgement

This project was sponsored by the U.S. Army Research Laboratory under
Cooperative Agreement Number W911NF2120076.

aaai22, eehpc

## References

zuluaga2018deep

duisterhof2021tiny

navardi2022optimization

truong2021bi

zhang2021sim2real

duisterhof2021sniffy

zhang2020sim2real

nguyen2019reinforcement

benjumea2021yolo

doersch2019sim2real

taghibakhshi2021local

prakash2021interactive

staroverov2020real

li2020hrl4in

pmlr-v37-schaul15

wang2022tacto

kaelbling1993learning

navardi2022toward

shiri2022efficient

giernacki2017crazyflie

Zhu2017TargetdrivenVN.

Koch2015SiameseNN.

inproceedings.

ALLAMAA2022385

9196730

fang2022target

ren2015faster

liu2016ssd

8919366

shu2018hierarchical

krishnan2021air

elkhrachy2021accuracy

prakash2021semantic

prakashhierarchical