Goswami Goswami 1]Indian Institute of Astrophysics, Koramangala, Bangalore
560034, India 2]Pondicherry University, R.V. Nagar, Kalapet, 605014
Puducherry, India<EMAIL_ADDRESS>
# Formation scenarios of Ba stars: new evidence from the masses of the
companion AGBs

Partha Pratim  Aruna  [  [

(19th April 2023)

###### Abstract

The abundances of the slow neutron-capture (s-) process elements observed in
barium stars can be explained by considering mass transfer in a binary system
from an asymptotic giant branch (AGB) star onto a smaller mass and less
evolved binary companion star. Based on the abundances of neutron-capture
elements the barium stars are broadly divided into two categories, “strong”
and “mild” barium stars. However, an understanding of the role of the
characteristic properties of the companion AGBs on the formation of the
different types of barium stars is still lacking. This work focuses on
investigating the role of the mass of the companion AGB stars in this context.
In a recent high-resolution spectroscopic study, we studied in detail the
chemical composition of a few barium stars that we have identified. In order
to understand the nature and the mass of the companion stars, we first
estimated the mass of these objects and then used a parametric model-based
analysis to derive the masses of the primary companion stars. The calculation
is extended to 205 barium stars taken from literature. An analysis of the mass
distribution revealed that both the strong and the mild barium stars occupy
the same range of masses. However, their companion AGB stars’ mass
distributions peak at two different values 2.5 M⊙ and 3.7 M⊙, for the strong
and the mild barium stars respectively. This provides clear evidence that the
formation of mild and strong Ba stars is greatly influenced by the initial
masses of the companion AGB stars. It remains, however, to be seen the
possible impact of other factors such as orbital period and metallicity on the
formation scenarios of barium stars.

###### keywords:

Barium stars, AGB, mass-distribution

##  1 Introduction

Bidelman and Keenan (1951) first recognised abnormally strong Ba II (4554 Å)
and Sr II (4077 Å) features in some giant stars and categorised these objects
as “Barium Stars”. In addition to these features barium stars are also
characterised by enhanced abundances of slow neutron-capture process elements
(Bidelman and Keenan, 1951; Warner, 1965). Several authors (Bond, 1984; Luck
and Bond, 1991) have suggested that the CH strong subgiants are likely
progenitors of barium stars as they exhibit similar carbon and s-process
elements. With the s-process signatures of AGB stars similar to CH stars, the
barium stars are also believed to be the metal-rich Population I analogues of
CH stars (Allen and Barbuy, 2006). From a long-term radial velocity
observations of a large number of barium stars McClure et al. (1980) have
confirmed the binary nature of barium stars. Jorissen and van Eck (2000) have
suggested that the abundances of heavy elements observed in barium stars are
the result of a mass transfer process and can be explained with the help of a
binary picture including low-mass AGB star. Based on the abundances of
neutron-capture elements barium stars are divided into two groups, the strong
barium stars and the mild barium stars. However, the characteristic properties
of the companion stars that are likely to play a vital role in making a barium
star “strong” or “mild” are not clearly understood. In this work, we have
addressed this problem by considering the mass distribution of the companion
stars of a large set of barium stars. In a recent work, from a detailed
analysis of seven stars (BD+75348, BD+093019, HD238020, HE0319–0215,
HE0507–1653, HE0930–0018, and HE1023–1504) based on high-resolution
spectroscopic analysis we have identified two strong barium stars (BD+75348,
BD+093019) and one mild barium star (HD238020) in the sample. These objects
are included in the present study. In Section 2, we present a brief discussion
on the radial velocities, stellar atmospheric parameters and elemental
abundances of the stars. Section 3 briefly discusses the determination of
masses of the programme stars as well as the masses of their primary companion
stars. A discussion on the mass distribution and clues derived from the
distribution patterns are also presented in this section. The conclusions are
drawn in section 4.

##  2 Radial Velocities, Stellar Atmospheric Parameters and Elemental
Abundances

For our analysis we have used High-resolution ($\lambda/\delta\lambda$ $\sim$
60 000) spectra acquired using Hanle Echelle SPectrograph (HESP) for the
objects BD+093019, BD+75348, and HD238020. High-resolution
($\lambda/\delta\lambda$ = 50 000) spectra of the other stars were taken from
the SUBARU archive. IRAF was used for data reduction and continuum fitting. We
have used broadband colours, optical, and infrared (IR) to determine the
photometric temperatures of the programme stars. The colour-temperature
calibrations used in this study are based on the IR flux method (Alonso et
al., 1996, 1999). The Teff(J–K) is independent of metallicity and hence we
have used this temperature as an initial guess to select the model atmospheres
by an iterative method. The radial velocities of the objects are calculated by
measuring the shifts of the elemental absorption lines from the laboratory
wavelengths, using Doppler’s formula. The noticeable discrepancies between the
radial velocities obtained by us and the values reported by Gaia Collaboration
et al. (2018) suggest the possibility of the programme stars being binary
systems with unseen companions. Stellar atmospheric parameters such as Teff,
log $g$, microturbulent velocity ($\xi$), and [Fe/H] have been derived using
the MOOG code (Sneden, 1973) and the method described in Goswami et al.
(2021); Goswami and Goswami (2022). The model atmospheres have been selected
from the Kurucz grid of model atmospheres. Using MOOG code, we determined the
abundances of the light and neutron-capture elements of the programme stars.
Spectrum synthesis calculations were used for deriving the abundances of C, N,
O and the elements showing hyperfine splitting. The equivalent width method
was used for the rest of the elements. Kurucz database and linemake provided
the atomic line information.

The details of the methodology, linelists and the results of this study can be
found in Goswami and Goswami (2023).

##  3 Results and Discussion

###  3.1 Estimation of masses of Ba stars and mass distribution

Our analysis shows that BD+75 348 and BD+09 3019 are strong Ba ([hs/Fe] $>$
0.60) stars and HD 238020 a mild Ba star (0.17 $<$ [hs/Fe] $<$ 0.60). In
addition to these objects, we have compiled the abundance data and other
relevant data of a large sample of Ba stars from the literature to derive the
mass distribution of these stars. We have estimated the masses of the Ba stars
by determining their position on the HR diagram and used Equation 1 to
calculate their luminosities (L).

$log(L/L_{\odot})=0.4(M_{bol\odot}-V-5-5log(\pi)+A_{v}-BC)$ (1)

We have used various sources to obtain the necessary data required for the
analysis. The V magnitudes of the objects are obtained from SIMBAD, while the
parallaxes ($\pi$) are sourced from Gaia. Interstellar extinction (Av) values
are calculated using the equation of Chen et al. (1998), and the bolometric
corrections (BC) are determined using empirical calibrations by Alonso et al.
(1999). To estimate the masses of the Ba stars, we used updated BASTI-IAC
evolutionary tracks, corresponding to the metallicities of the Ba stars. In
total, 205 Ba stars are analysed, including 52 mild and 153 strong Ba stars.
We found that mild and strong Ba stars have similar mass ranges (0.8 to 4.5
M⊙), with tail ends going to 4.5 $\pm$ 0.25 M⊙ and 4.0 $\pm$ 0.25 M⊙,
respectively (Figure 1). Additionally, the distribution of Ba stars does not
follow a single Gaussian distribution but is scattered throughout the mass
range. On average, the mass of Ba stars is estimated to be 1.9 M⊙ (Figure 1).
Contrary to Escorza et al. (2017) that reported a shortage of Ba stars in the
1.0 – 2.0 M⊙ mass range, our study shows that this range is significantly
populated by Ba stars, as shown in Figure 1.

Figure 1: Mass distributions of Ba stars. Panel (a) displays the mass
distributions of mild and strong Ba stars separately, while Panel (b) shows
the mass distribution of all mild and strong Ba stars together. Taken from
Goswami and Goswami (2023) with permission.

###  3.2 Estimation of masses of AGB companions and mass distribution

We have used AGB yields of the FRUITY models (Cristallo et al., 2015) to
perform a parametric-model-based analysis to estimate the initial masses of
the companions of the programme stars and the sample of Ba stars compiled from
the literature. The FRUITY models are used to calculate the s-process yields
for various masses that correspond to the metallicities of the stars. Next,
the observed abundances of the neutron-capture elements are compared to the
model function to identify the best fit. The required mass of the companion
AGB of the programme star is determined as the corresponding mass of the AGB
model for which the minimum value of $\chi^{2}$ is achieved. The detailed
methodology followed for the parametric-model-based study is same as that
explained at length in Goswami and Goswami (2023). We have determined the
masses of the primary companions for 158 Ba stars, including 52 mild Ba stars
and 106 strong Ba stars.

The mass distribution of the primary companions for mild and strong Ba stars
is found to have two peaks with a larger mass range for mild Ba stars. The
distribution of primary companion masses of strong Ba stars peaks at 2.5 M⊙,
while that of mild Ba stars peaks at 3.7 M⊙ (Figure 2). The distribution of
primary companion masses of Ba stars, when considered as a whole, peaks at 2.9
M⊙ (Figure 2).

Figure 2: Mass distributions of the primary stars (progenitor AGBs) of Ba
stars. Panel (a) displays the mass distributions of the progenitors of mild
and strong Ba stars separately, while Panel (b) presents the mass distribution
of the progenitors of both mild and strong Ba stars together. Taken from
Goswami and Goswami (2023) with permission.

###  3.3 Derived clues on the formation Scenarios of the two classes of Ba
Stars

Theoretical analysis by Han et al. (1995) indicates that most mild Ba stars
are formed through wind accretion and wind exposure, whereas strong Ba stars
are formed through wind accretion, wind exposure, and stable Roche lobe
overflow (RLOF). According to de Castro et al. (2016), mildly enhanced
interstellar matter with s-process elements is the source of mild Ba stars.
However, subsequent studies by Jorissen et al. (2019) have revealed that both
mild and strong Ba stars are formed through AGB mass transfer in binary
systems, rendering the previous scenario invalid. Yang et al. (2016) proposed
two possible formation scenarios for mild Ba stars, but observations by
Jorissen et al. (1998, 2019) have shown that mild Ba stars are not restricted
to long-period systems and that the orbital periods of mild and strong Ba
stars overlap.

The differences in the peaks of primary masses for mild and strong Ba stars
indicate that the companions of mild Ba stars are more massive. This implies
that the initial mass of the companion is the main factor controlling the
abundance peculiarities of mild Ba stars, with mass transfer from 4.0–6.0 M⊙
AGB companions capable of explaining their formation (Goswami and Goswami,
2023). White dwarf masses of approximately 1.0 M⊙ around Ba stars also
indicate a massive (5 M⊙) companion AGB (Jorissen et al., 2019). However, in
the case of primary masses of mild and strong Ba stars in the range of 1.3–4.0
M⊙, the system’s metallicity and dilution are essential in their formation
(Goswami and Goswami, 2023). Mild Ba stars have a higher metallicity
distribution than strong Ba stars, and the s-process efficiency decreases at
higher metallicities, resulting in lower AGB yields.

##  4 Conclusion

We have investigated the mass distribution of Ba stars and found that it
cannot be represented by a single Gaussian. The average mass of the
distribution is found to be 1.9 M⊙ with strong and mild Ba stars extending up
to 4.0 M⊙ and 4.5 M⊙, respectively. We confirmed that mild and strong Ba stars
occupy a similar mass range. A previous claim (i.e., Escorza et al. (2017))
that there is a lack of Ba stars in the mass range of 1.0–2.0 M⊙ is not
supported by our study. Our investigation shows that the progenitor mass
distributions of mild and strong Ba stars peak at different values. We,
therefore, suggest that the initial mass of the companion AGB is the primary
factor in controlling the heavy elements’ enrichment in barium stars, and thus
plays a vital role in the making of strong and mild barium stars. However, the
impact of orbital period and metallicity in the formation of mild Ba stars
also need to be considered and investigated in detail, as there is significant
overlap in the orbital periods of mild and strong Ba stars. Such a study has
been undertaken and the results will be discussed in a separate paper.

###### Acknowledgements.

This work made use of the SIMBAD astronomical database, operated at CDS,
Strasbourg, France, the NASA ADS, USA and data from the European Space Agency
(ESA) mission Gaia, processed by the Gaia Data Processing and Analysis
Consortium (DPAC, https://www.cosmos.esa.int/web/gaia/dpac/consortium). AG
acknowledges the support received from the Belgo-Indian Network for Astronomy
& Astrophysics project BINA - 2 (DST/INT/Belg/P-02 (India) and BL/11/IN07
(Belgium)).

PPG and AG designed the research; PPG compiled the data and performed the
research; PPG and AG interpreted the results and wrote the paper.

“The authors declare no conflict of interest.”

## References

  * Allen and Barbuy (2006) Allen, D. M. and Barbuy, B. (2006) Analysis of 26 barium stars. I. Abundances.  A&A, 454(3), 895–915.  https://doi.org/10.1051/0004-6361:20064912.
  * Alonso et al. (1996) Alonso, A., Arribas, S. and Martinez-Roger, C. (1996) The empirical scale of temperatures of the low main sequence (F0V-K5V).  A&A, 313, 873–890.
  * Alonso et al. (1999) Alonso, A., Arribas, S. and Martínez-Roger, C. (1999) The effective temperature scale of giant stars (f0-k5) - ii. empirical calibration of $t_{eff}$ versus colours and [fe/h].  A&AS, 140(3), 261–277.  https://doi.org/10.1051/aas:1999521.
  * Bidelman and Keenan (1951) Bidelman, W. P. and Keenan, P. C. (1951) The Ba II Stars.  ApJ, 114, 473.  https://doi.org/10.1086/145488.
  * Bond (1984) Bond, H. E. (1984) A search for white-dwarf companions of subgiant CH stars.  In NASA Conference Publication, edited by Mead, J. M., Chapman, R. D. and Kondo, Y., vol. 2349 of _NASA Conference Publication_ , pp. 289–292.
  * Chen et al. (1998) Chen, B., Vergely, J. L., Valette, B. and Carraro, G. (1998) Comparison of two different extinction laws with HIPPARCOS observations.  A&A, 336, 137–149.
  * Cristallo et al. (2015) Cristallo, S., Straniero, O., Piersanti, L. and Gobrecht, D. (2015) Evolution, Nucleosynthesis, and Yields of AGB Stars at Different Metallicities. III. Intermediate-mass Models, Revised Low-mass Models, and the ph-FRUITY Interface.  ApJS, 219(2), 40.  https://doi.org/10.1088/0067-0049/219/2/40.
  * de Castro et al. (2016) de Castro, D. B., Pereira, C. B., Roig, F., Jilinski, E., Drake, N. A., Chavero, C. and Sales Silva, J. V. (2016) Chemical abundances and kinematics of barium stars.  MNRAS, 459(4), 4299–4324.  https://doi.org/10.1093/mnras/stw815.
  * Escorza et al. (2017) Escorza, A., Boffin, H. M. J., Jorissen, A., Van Eck, S., Siess, L., Van Winckel, H., Karinkuzhi, D., Shetye, S. and Pourbaix, D. (2017) Hertzsprung-Russell diagram and mass distribution of barium stars.  A&A, 608, A100.  https://doi.org/10.1051/0004-6361/201731832.
  * Gaia Collaboration et al. (2018) Gaia Collaboration, Katz, D., Antoja, T., Romero-Gómez, M., Drimmel, R., Reylé, C., Seabroke, G. M., Soubiran, C., Babusiaux, C., Di Matteo, P. and et al. (2018) Gaia Data Release 2. Mapping the Milky Way disc kinematics.  A&A, 616, A11.  https://doi.org/10.1051/0004-6361/201832865.
  * Goswami and Goswami (2022) Goswami, P. P. and Goswami, A. (2022) The peculiar abundances of HE 1005-1439. A carbon-enhanced extremely metal-poor star contaminated with products of both s- and i-process nucleosynthesis.  A&A, 657, A50.  https://doi.org/10.1051/0004-6361/202141775.
  * Goswami and Goswami (2023) Goswami, P. P. and Goswami, A. (2023) Spectroscopic Study of Ba and CEMP-s Stars: Mass Distribution of AGB Progenitors.  AJ, 165(4), 154.  https://doi.org/10.3847/1538-3881/aca971.
  * Goswami et al. (2021) Goswami, P. P., Rathour, R. S. and Goswami, A. (2021) Spectroscopic study of CEMP-(s & r/s) stars. Revisiting classification criteria and formation scenarios, highlighting i-process nucleosynthesis.  A&A, 649, A49.  https://doi.org/10.1051/0004-6361/202038258.
  * Han et al. (1995) Han, Z., Eggleton, P. P., Podsiadlowski, P. and Tout, C. A. (1995) The formation of barium and CH stars and related objects.  MNRAS, 277(4), 1443–1462.  https://doi.org/10.1093/mnras/277.4.1443.
  * Jorissen et al. (2019) Jorissen, A., Boffin, H. M. J., Karinkuzhi, D., Van Eck, S., Escorza, A., Shetye, S. and Van Winckel, H. (2019) Barium and related stars, and their white-dwarf companions. I. Giant stars.  A&A, 626, A127.  https://doi.org/10.1051/0004-6361/201834630.
  * Jorissen and van Eck (2000) Jorissen, A. and van Eck, S. (2000) Barium Stars and Tc-Poor S Stars: Binary Masqueraders within the Carbon-Star Family.  In The Carbon Star Phenomenon, edited by Wing, R. F., vol. 177, p. 259\.
  * Jorissen et al. (1998) Jorissen, A., Van Eck, S., Mayor, M. and Udry, S. (1998) Insights into the formation of barium and Tc-poor S stars from an extended sample of orbital elements.  A&A, 332, 877–903.
  * Luck and Bond (1991) Luck, R. E. and Bond, H. E. (1991) Subgiant CH Stars. II. Chemical Compositions and the Evolutionary Connection with Barium Stars.  ApJS, 77, 515.  https://doi.org/10.1086/191615.
  * McClure et al. (1980) McClure, R. D., Fletcher, J. M. and Nemec, J. M. (1980) The binary nature of the barium stars.  ApJL, 238, L35–L38.  https://doi.org/10.1086/183252.
  * Sneden (1973) Sneden, C. A. (1973) Carbon and Nitrogen Abundances in Metal-Poor Stars.  Ph.D. thesis, The University of Texas at Austin.
  * Warner (1965) Warner, B. (1965) The barium stars.  MNRAS, 129, 263.  https://doi.org/10.1093/mnras/129.3.263.
  * Yang et al. (2016) Yang, G.-C., Liang, Y.-C., Spite, M., Chen, Y.-Q., Zhao, G., Zhang, B., Liu, G.-Q., Liu, Y.-J., Liu, N., Deng, L.-C., Spite, F., Hill, V. and Zhang, C.-X. (2016) Chemical abundance analysis of 19 barium stars.  RAA, 16(1), 19.  https://doi.org/10.1088/1674-4527/16/1/019.


# Juvenile state hypothesis: What we can learn from lottery ticket hypothesis
researches?

Di Zhang

###### Abstract

> The proposition of lottery ticket hypothesis revealed the relationship
> between network structure and initialization parameters and the learning
> potential of neural networks. The original lottery ticket hypothesis
> performs pruning and weight resetting after training convergence, exposing
> it to the problem of forgotten learning knowledge and potential high cost of
> training. Therefore, we propose a strategy that combines the idea of neural
> network structure search with a pruning algorithm to alleviate this problem.
> This algorithm searches and extends the network structure on existing
> winning ticket sub-network to producing new winning ticket recursively. This
> allows the training and pruning process to continue without compromising
> performance. A new winning ticket sub-network with deeper network structure,
> better generalization ability and better test performance can be obtained in
> this recursive manner. This method can solve: the difficulty of training or
> performance degradation of the sub-networks after pruning, the forgetting of
> the weights of the original lottery ticket hypothesis and the difficulty of
> generating winning ticket sub-network when the final network structure is
> not given. We validate this strategy on the MNIST and CIFAR-10 datasets. And
> after relating it to similar biological phenomena and relevant lottery
> ticket hypothesis studies in recent years, we will further propose a new
> hypothesis to discuss which factors that can keep a network juvenile, i.e.,
> those possible factors that influence the learning potential or
> generalization performance of a neural network during training.

## Introduction

After the proposition of the lottery ticket hypothesis (Frankle and Carbin
2019), more effective methods for neural network pruning appeared, and at the
same time, new methods for neural network architect searching(NAS) were
emerging. NAS algorithm searching for the network’s architectural parameters
to construct the mapping function from the network’s input to the network’s
output (Ren et al. 2021). Network pruning reduces the total number of the
network’s parameters by removing those parameters of lesser importance from
the network weight parameters (LeCun, Denker, and Solla 1990). The main
difference between NAS and network printing is that the former is based on the
network’s architecture, while the latter is based on the network’s parameters.
To some extent,they applied similar procedures to neural networks since
network pruning can cut unnecessary links between neurons by eliminating the
network’s unimportant parameters and finding a better architecture for the
network.

Therefore, we can consider the pruning algorithm as a network structure
optimization operator. The pruning algorithm optimizes the inter-neuron
connection structure at a finer granularity when the neural architecture
search algorithm for the network layer width and the path of forwarding
propagation flow.

However, in practice, pruning algorithms are usually used to prune neural
networks trained close to convergence to optimize their inference speed and
memory efficiency (Liang et al. 2021), and many researchers point out that
continuing training on the neural network after pruning could cause
performance degradation on the validation set, and that training from scratch
is hard using pruned structures (Li et al. 2016; Han et al. 2015). To address
the latter issue, Frankle et al. proposed the lottery ticket hypothesis(LTH)
that the initialization is a key point for one neural network and its sub-
networks to get better performance on the validation set during a similar
training process. By pruning the neural networks close to the training fit,
the unmasked network weights are re-trained after re-adopting the parameters
from the original initialization, and the process is repeated until obtaining
a sub-network with the required amount of parameters and accuracy.

Even so, in large neural networks, the overhead of the pruning-reset-
retraining cycle is unaffordable, and Frankle et al. then tried using weights
after trained for iterations to reset the pruned network (Frankle et al. 2019;
Renda, Frankle, and Carbin 2020) instead of the totally original initial
weights. But the recovery of the original parameters and weights means
forgetting of most knowledge learned in training, even some researcher assumed
that pruned architecture keeps the memory of the network’s inductive bias
(Frankle and Carbin 2019; Cohen and Shashua 2016).

But after trying to add new trainable layers and parameters to pruned near-
convergent neural networks,also known as _winning tickets_ , we found that the
tested network could continue training until convergence nearly at a higher
test accuracy.

Therefore, we proposed a new iterative pruning algorithm that is based on a
generalized inference of LTH, This pruning algorithm combines a simple network
architecture search Process and pruning-retraining cycle into a pruning-
growing-retraining approach to shape a deep and thin neural network to meet
the requirement of parameter’s amount and test accuracy.

Then we further generalize the lottery ticket hypothesis into recursive
lottery ticket hypothesis, and by this hypothesis and other inferences of
lottery ticket hypotheses, we will further explore the juvenile state of
neural networks beyond the luckiness of initialization to try to explain what
determines the learning potential and convergence speed of neural networks.

## Approach

### Lottery Ticket Hypothesis

First, we would have a review of the naive Lottery ticket hypothesis (LTH).

#### Notion

We follow the description of the lottery ticket hypothesis by Frankle et al.
(Frankle and Carbin 2019). but with an additional symbol $\mathbf{\theta}$ to
represent the architecture of the network and shaping the vector space of
weight parameters. Given a feed-forward neural network $f$ with architecture
$\mathbf{\theta}$ , weights $\mathbf{\omega}$ and taking $x$ as input.

Furthermore,the weight parameter $\mathbf{\omega}$ defined a set of parameters
$\mathbf{\omega}_{i}$ for each layer $i$. The architecture $\mathbf{\theta}$
defines the number of neurons, width $w$ in each layer $i$ and the number of
layers $L$, as well as the path of forwarding propagation flow especially
inter-neuron connections through mask matrix $m_{i}$ in each layer $i$.

#### Formal Definitions

At start, the network weight parameters $\mathbf{\omega}$ are randomly
initialized with $\mathbf{\omega}^{0}\sim\mathcal{D}_{\mathbf{\omega}}$to
adapt the structure $\mathbf{\theta}^{0}$ of the network. Then,the network
$f(x;\mathbf{\theta}^{0},\mathbf{\omega}^{0})$ trained on a dataset
$\mathbf{D}$ and optimized by a gradient descent algorithm optimizer
$\mathcal{L}$ close to convergence.

Then the network $f(x;\mathbf{\theta}^{0},\mathbf{\omega}^{1})$is pruned by
removing the parameters of lesser importance from the network weight
parameters $\mathbf{\omega}$,and produce mask matrices $m_{i}$ for each layer
$i$ to define inter-neuron connections between layers.

The cycle of training and pruning could execute iteratively until the final
network $f(x;\mathbf{\theta}^{*},\mathbf{\omega}^{*})$ achieve the expected
test accuracy $Acc^{*}$ and compression ratio $r_{comp}^{*}$of the number of
parameters or meet the maximum iterations $j$, and produce a mask matrix $m$.

The network $f(x;\mathbf{\theta}^{*},\mathbf{\omega}^{*})$is said to be a
_lottery ticket_ network if the following conditions are satisfied:

1.$\frac{\left\|\mathbf{\omega}^{*}\right\|_{0}}{\left\|\mathbf{\omega}^{0}\right\|_{0}}\leq
r_{comp}^{*}$

2\. Existing a $(\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$ with the
same architecture of $\mathbf{\theta}^{*}$ and the same number of parameters
of $\mathbf{\omega}^{*}$ and a policy $p$, follow this policy $p$,we can
training network$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$ close
to convergence,and achieve a new test accuracy $Acc^{\prime}$ close enough to
$Acc^{*}$.

So,We take the identifying method which frankle et al. used in their work
(Frankle and Carbin 2019) as the weight reset policy,its work process can be
described by the following pseudo-code:

1Given $f(x;\mathbf{\theta}^{0},\mathbf{\omega}^{0})$

Result: _winning ticket_
$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$

2 initialize $\mathbf{\omega}^{0}\sim\mathcal{D}_{\mathbf{\omega}}$to adapt
the structure $\mathbf{\theta}^{0}$

3 record $(\mathbf{\theta}^{0},\mathbf{\omega}^{0})$

4 training and pruning $f(x;\mathbf{\theta}^{0},\mathbf{\omega}^{0})$ for $j$
iterations,arriving at $f(x;\mathbf{\theta}^{*},\mathbf{\omega}^{*})$

5 $\omega^{\prime}\leftarrow m\odot\omega^{0}$, reset $\mathbf{\omega}^{*}$ to
$\mathbf{\omega}^{0}$ according to the mask matrix $m$,and denote as
$\mathbf{\omega}^{\prime}$

6 produce _winning ticket_
$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$

7 (Optional) Verifying the _winning ticket_
$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$

8 training $f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$ for $j$
iterations and get a new test accuracy $Acc^{\prime}$

9
If$\frac{\left\|\mathbf{\omega}^{\prime}\right\|}{\left\|\mathbf{\omega}^{0}\right\|}\leq
r_{comp}^{*}$ And $Acc^{\prime}$ is closing enough to $Acc^{*}$

10 Then$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$ is a _winning
ticket_

End

Algorithm 1 Weight reset policy

With this policy, we can solve the accuracy degradation problem when we
continue to train the pruned neural network.

That is, by resetting the network weights to the original initialization
values, we can retrain the network with a smaller number of parameters and
keep the final test accuracy with a little degradation or even better compared
to the network with a larger number of parameters.

### Recursive Lottery Ticket Hypothesis

Then,we propose the recursive lottery ticket hypothesis (RLTH), which is a
generalization of the former naive lottery ticket hypothesis allows us to
obtain a new _winning ticket_ with more trainable parameters from an existing
_winning ticket_ by a recursive way like pruning-growing-retraining cycle.

Up to this point, we have used the pruning algorithm $\mathcal{L}_{prun}$ only
for optimizing the number of parameters of a trained neural network with a
fixed structure $\theta$.

In the following part, we will go a step further and use the pruning algorithm
$\mathcal{L}_{prun}$ as a fine-grained structure optimization operator to get
a sub-network with optimal structure from original network. And try to extend
the existed _winning ticket_ with new neural network layers and trainable
parameters, and obtain a new _winning ticket_ recursively by a similar
training-pruning process.

Under the weight reset strategy, the network weights must be reset if we want
continue training on the pruned structure without compromising the test
accuracy, but it implies discarding and forgetting the learned knowledge while
preserving only a little learned inductive bias preserved by the structural
information.

Assuming that we have already obtained a _winning ticket_ sub-network by the
original lottery ticket hypothesis, we can make following assumptions to
induce the recursive lottery ticket hypothesis, which is a generalized
inference of original lottery ticket hypothesis.

Inference 1: When neural network
$f(x;\mathbf{\theta}^{*},\mathbf{\omega}^{*})$ is a _winning ticket_ obtained
following a policy $p$ from neural network
$f(x;\mathbf{\theta}^{0},\mathbf{\omega}^{0})$, and $m$ is a mask matrix, a
_winning ticket_ structure parameterized by $\theta^{*}$ is a sub-network of
structure parameterized by $\theta^{0}$,and on their corresponding graphical
representations $G^{*},G^{0}$, Also meet that $G^{*}$ is a _spanning subgraph_
of $G^{0}$.

Then we can construct a new father graph of that _winning ticket_ by add new
nodes and links to existing graph through specific network layers with new
trainable parameters.

Inference 2: Given a _winning ticket_
network$f(x;\mathbf{\theta}^{*},\mathbf{\omega}^{*})$ ,there existing a
specific network which structure parameterized by $\theta^{\prime}$ with
weights $\omega^{\prime}$.

by adding that,we can construct a new neural network
$f(x;\mathbf{\theta}^{1},\mathbf{\omega}^{1})$, where

$\left\\{\begin{aligned} \theta^{1}&=\theta^{*}\cup\theta^{\prime}\\\
\omega^{1}&=\omega^{*}\cup\omega^{\prime}\end{aligned}\right.$

and according to the original LTH, neural network
$f(x;\mathbf{\theta}^{*},\mathbf{\omega}^{*})$ can also be a _winning ticket_
sub-network of $f(x;\mathbf{\theta}^{1},\mathbf{\omega}^{1})$,only if there
exisit a policy $p$ and a pruning algorithm $\mathcal{L}$ could train and
prune $f(x;\mathbf{\theta}^{1},\mathbf{\omega}^{1})$ into
$f(x;\mathbf{\theta}^{*},\mathbf{\omega}^{*})$.

Further more, we can also learn a simple fact from original LTH, that one
neural network could have more than one _winning ticket_ sub-networks. Since
our goal is to get a new _winning ticket_ sub-network,if our training policy
and pruning algorithm meet the requirements fortunately, we can relax the
premise of ”specific extending layer” of Inference 2. Then the old _winning
ticket_ sub-network can be extended with a new randomly initialized layer, and
thus obtain a new _winning ticket_ through above training-pruning cycle.

So in summary, we can draw a generalized conclusion from the original lottery
ticket hypothesis.

inference 3 Given a _winning ticket_ neural networks
$f(x;\mathbf{\theta}^{*},\mathbf{\omega}^{*})$ and a randomly initialized
extending layer which struture parameterized by $\theta^{\prime}$ with weights
$\omega^{\prime}$, there exisit a policy $p$ and a pruning algorithm
$\mathcal{L}$ could train and prune its extended network
$f(x;\mathbf{\theta}^{1},\mathbf{\omega}^{1})$ and produce a new _winning
ticket_ sub-network.

That is the Recurisive lottery ticket hypothesis(RLTH). With this conclusion
we can start training with a simple small neural network and expand it after
getting a _winning ticket_ sub-network. And repeat for several times,
eventually getting a deeper _winning ticket_ recursively without forgetting of
learned knowledge during the training process.

Formally, the basic training policy for the recursive lottery ticket
hypothesis can be outlined in the following form.

1Given $f(x;\mathbf{\theta}^{0},\mathbf{\omega}^{0})$

Result: _winning ticket_
$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$

2 initialize $\mathbf{\omega}^{0}\sim\mathcal{D}_{\mathbf{\omega}}$to adapt
the structure $\mathbf{\theta}^{0}$

3 while  _sub-network can’t meet requirements_ do

4 training and pruning $f(x;\mathbf{\theta}^{0},\mathbf{\omega}^{0})$ for $j$
iterations,arriving at $f(x;\mathbf{\theta}^{*},\mathbf{\omega}^{*})$

5 initialize a new layer
$\mathbf{\omega}^{1}\sim\mathcal{D}_{\mathbf{\omega}}$to adapt the structure
$\mathbf{\theta}^{1}$

6
$\theta^{1}\leftarrow\theta^{*}\cup\theta^{\prime},\omega^{1}\leftarrow\omega^{*}\cup\omega^{\prime}$

7 construct a new sub-network
$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$

8

9 end while

10produce _winning ticket_
$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$

11 (Optional) Verifying the _winning ticket_
$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$

12 training $f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$ for $j$
iterations and get a new test accuracy $Acc^{\prime}$

13
If$\frac{\left\|\mathbf{\omega}^{\prime}\right\|}{\left\|\mathbf{\omega}^{0}\right\|}\leq
r_{comp}^{*}$ And $Acc^{\prime}$ is closing enough to $Acc^{*}$

14 Then$f(x;\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime})$ is a _winning
ticket_

End

Algorithm 2 Basic structure growing policy

This approach allows the neural network to grow like an oyster, growing to a
deeper structure and keeping the structurally optimal with a proper parameters
amount at the same time.

## Experiment

### RLTH on Dense Networks

First, we test the weight reset policy of the original LTH and the basic
structure growing policy of RLTH on the MNIST dataset (Lecun et al. 1998).The
experiment was executed on Google Colab platform.

Then we iteratively training neural networks for 5 times to the same parameter
amount by former two policies without any hyperparameter tunings, this process
is a circle of training, obtain _winning ticket_ and training again.

The neural network to be trained with weight reset policy iteratively is a
random-initialized dense network. The network have a input layer project the
input image to a vector of 128 elements. And those 5 hidden layers have been
set to a linear layer with a shape of (128,128) and a ReLU activation layer.
Finally, the output layer produce the category probability of the output image
with a Softmax activation layer.

And The neural network to be trained with structure growing policy have the
same input and output layer and have no hidden layers at begin. When
iteratively training, hidden layers would be insert to the network via so-
called structure growing operation. To simplify the structure growing
operation, we insert a linear layer with a shape of (128,128) and a ReLU
activation layer per iteration. And to avoid the interruption of the training
process, the inserted layer was initialized with a identity matrix for weight
and a zero matrix for bias.

Notion that, in more complicated scenarios, the structure growing operation
can be implement more properly by a sophisticated methods like other NAS
algorithms.

For the convenience of reproduction, The other settings of the experiment are
as follows. All random seeds set to 42. Optimizer of networks was Adam (Kingma
and Ba 2017) with a initial learning rate of 0.001. And the loss function is
using categorical cross entropy. The prunning algorithm is Level pruner with a
sparsity ratio of 95% from NNI toolbox (Microsoft 2021).

The loss and accuracy changes during training are shown in the Figure 1 and
Figure 2, different color referring to different training iterations.

(a) training

(b) validating

Figure 1: Accuracy on MNIST Dataset

(a) training

(b) validating

Figure 2: Loss on MNIST Dataset

### RLTH on ResNet and Convolution Networks

This part of experiment was executed on CIFAR-10 Dataset (Krizhevsky, Hinton
et al. 2009). To fitting the multi-channel image data of this dataset. The
input and output layer was different form above. And training process was
iteratively executed for 7 times.

The neural network to be trained with weight reset policy iteratively is a
random-initialized residual convolution neural network. The network have a
input layer with input channel of 3 and output channel of 8, and kernel size
is 7. Thus those 7 hidden layers have been set to basicblocks of ResNet (He et
al. 2016) with input channel of 8 and output channel of 8. Finally, the output
layer produce the category probability of the output image with a Softmax
activation layer.

And The neural network to be trained with structure growing policy have the
same input and output layer and have no hidden layers at begin. When
iteratively training, hidden layers would be insert to the network via so-
called structure growing operation, too. We insert a basicblocks of ResNet
with input channel of 8 and output channel of 8 per iteration, the convolution
kernels of inserted layer was initialized with a identity matrix for weight.

The other settings of the experiment are the same with above except the
sparsity ratio chaged to 80% and the pruning algorithms only compress hidden
layers of convolution Basicblocks.

The loss and accuracy changes during training are shown in the Figure 3 and
Figure 4, different color referring to different training iterations.

(a) training

(b) validating

Figure 3: Accuracy on CIFAR-10 Dataset

(a) training

(b) validating

Figure 4: Loss on CIFAR-10 Dataset

From the above two parts of the experiments, we can see that although the
weight reset policy avoids the training difficulties of the pruned neural
network, its forgetting of knowledge is irrecoverable, and the forgetting
slows down the training process and the final test performance when the
_winning ticket_ sub-networks are repeatedly acquired and trained. In
contrast, the structural growth strategy avoids the training difficulties of
the pruned neural network by adding new trainable parameters after the
_winning ticket_ sub-networks are acquired, and ensuring the transfer of the
learned knowledge at the same time. This policy is useful for accelerating
convergence and reducing overhead in the early training of large neural
networks.

## Discussion

### Juvenile state of neural networks

Based on the above experiments, we can see that the weight reset policy and
the structure growing policy are similar in that both strategies play a
similar role in the training of neural networks and the acquisition of
_winning ticket_. To clarify this role, we need to go back to the early
starting point of frankle’s work, i.e., to overcome the training difficulties
and test performance degradation faced by the re-training of pruned neural
networks. And the weight reset policy address this difficulty by re-empowering
the pruned neural networks to obtain the speed of convergence and test
performance comparable to the original neural networks just after initialized.

Therefore, we can make a qualitative hypothesis that when the initialization
method and network structure are appropriate, the original initialized neural
network has a property that has been weakened during the pruning or improper
training, and it is this property that affects the so-called learning
potential, also known as the ability of the network to converge at a better
test accuracy with an appropriate training policy.

As for this property, when applied to the learning process of neural networks,
it has a high degree of similarity to the learning process of natural animals,
so we can introduce a bio-behavioral phenomena, juvenile state, to define this
property, where the learning skills and proficiency of young animals progress
significantly faster than those of adults (Delacour 1994; Boesch et al. 2019).
The main manifestation of this in humans is in language learning, where young
children who have not acquired any language skills learn their native language
significantly faster than adults who learn the same language as a foreign
language (Snow 1972; Leung, Tunkel, and Yurovsky 2021). For canines, there is
a golden period of training, with puppies around half a year old acquiring
tricks and skills much faster than adult dogs after sexual maturity (of
Veterinary Medicine Vienna 2016; Wallis et al. 2016).

For neural networks, the juvenile state is the state where the model has good
learning potential and fast convergence speed. When the neural network is in
the juvenile state or the juvenile state has good properties, the neural
network can converge at a good test performance level by a fast speed with
appropriate learning strategies and optimizers.

In contrast, the juvenile state or property of juvenile of a neural network is
weakened during a inappropriate training process or model pruning, which is
most intuitively manifested by the difficulty of convergence and the
degradation of test performance when the neural network continues training
after pruning.

### winning tickets and juvenile state

Returning to the lottery ticket hypothesis, we can consider that the _winning
ticket_ sub-networks are the sub-networks with better learning potential among
the different sub-networks obtained from the same original neural network.
Thus the weight reset policy and structure growing policy, to some extent,
exerted a rejuvenation of the sub-networks by modifying the sub-networks
structure or weights to restore the learning potential of the sub-networks
that were damaged during the pruning process.

Therefore, based on the above hypothesis and analysis, we can consider that
the process of obtaining a _winning ticket_ sub-networks is to obtain a sub-
networks with smaller structure and smaller parameters amount from a neural
network who was already converged. And at the same time, the juvenile
properties of this sub-network would also close to the original just
initialized neural network. A good _winning ticket_ sub-networks has a good
juvenile properties to ensure it can converge as fast as the original just
initialized neural network with a smaller parameters amount and a comparable
test performance after convergence.

### Factors affecting Juvenile state

In the above analysis, we have covered some behaviors that are harmful or
beneficial to the neural network juvenile state and properties. In the
following, we would discuss those factors that may affect the neural network’s
juvenile state.

First, the weights and initialization of the neural network. The most
important contribution of the work of frankle et al. (Frankle and Carbin 2019)
is that the convergence speed and testing performance of the network are
highly correlated with the initialization process of the network, and the
weight resetting policy restores the property of juvenile and learning
potential of the network precisely by resetting the network to the initial
weights after pruning.

Second, the structure of the neural network. The impact of the structure of
the neural network on the neural network juvenile properties is reflected in
the common sense. When a model structure is not suitable for training data,
the neural network will face the situation of training difficult or degraded
test performance even from the beginning of training.

Thus, If the number of model parameters is too large, it would make the model
difficult to train and slow to converge. While if the number of parameters is
too small, it would make the neural network easy to fall into the local optima
and cause a overfitting of the model.

In contrast, experiments on the structural growth strategy show that
appropriate overparameterization of structure is helpful for improving the
property of juvenile of the model.

Third, the training strategy. In our usual sense, It contains the loss
function, the optimization algorithm and its hyperparameter settings, such as
learning rate and momentum, as well as other settings such as batch size and
data set sampling methods appeared in deep learning practices.

Specifically, just for the gradient descent method, from a viewpoint of the
_loss function surface landscape_ (Li et al. 2017), the projection points (or
the center of gravity of multiple projection points) of the same (or the same
batch) dataset sample points on the loss surface, during the iterative
optimization process, are also in constant motion, and ideally most of the
projection points can reach a local elevation minimum along the fastest
descending line, i.e., the model converges on a local optimal solution. And
models can leave the local optima by the momentum methods to reach a better
approximate solution.

The proper loss function and proper training policy provided a good _loss
function surface landscape_ , which enables faster convergence, avoidance of
local optima traps and better test performance of the neural network in the
above process, which corresponds to the juvenile state assumption.

Finally, better test scores tend to reflect better generalization performance
for different _winning ticket_ sub-networks within the same dataset. on the
other hand, the work of Ibrahim et al. (Alabdulmohsin et al. 2021; Morcos et
al. 2019) shown the existence of _winning ticket_ sub-networks with the
ability to generalize across datasets and training policies. So, we can assume
that the learning potential of a neural network and juvenile state properties
on datasets also affects the neural network’s generalization ability.

## Conclusion

In this paper,we introduced the recursive lottery ticket hypothesis by
generalizing the lottery ticket hypothesis for neural networks. By recursively
acquiring the _winning ticket_ sub-networks and continuing the training after
add new trainable parameters, we can avoid the training difficulty of pruned
network and the knowledge forgetting of the lottery ticket hypothesis does.

From the perspective of connectionism,the recursive lottery ticket hypothesis
uses the pruning algorithm as a fine-grained optimizer of the link structure
between neuron layers.The structure growth operation allows the introduction
of more complex neural architecture search algorithms to above procedure,thus
a bridge between structural search and structural optimization have been
constructed. Finally, we further propose the juvenile state hypothesis by
linking previous lottery ticket hypothesis studies with similar phenomena in
biological behavior, and finally we made a qualitative analysis of those
factors may affecting the juvenile state of neural networks.

## Future works

Due to hardware and time constraints, the datasets and pruning algorithms
involved in the experiments and the architectural search operations in this
paper are relatively simple and naive. In the near future, we will verify that
hypothesis on more complex problems and introduce more complex algorithms for
the structural operations of neural networks.

In addition, another terminology closely related to juvenile state in biology
is Neoteny (Shea 1989; Price 1999), also known as juvenile state’s
continuation, which is a state in which an organism can maintain its physical
and mental young and strong learning ability for a long time even last for
life, similar to continuous learning in machine learning that can combat with
catastrophic forgettings (Delange et al. 2021), and it is one of the main
directions of our future work.

## References

  * Alabdulmohsin et al. (2021) Alabdulmohsin, I.; Markeeva, L.; Keysers, D.; and Tolstikhin, I. 2021.  A Generalized Lottery Ticket Hypothesis.  _arXiv:2107.06825 [cs]_.  ArXiv: 2107.06825.
  * Boesch et al. (2019) Boesch, C.; Bombjaková, D.; Meier, A.; and Mundry, R. 2019.  Learning curves and teaching when acquiring nut-cracking in humans and chimpanzees.  _Scientific Reports_ , 9(1): 1515.  Bandiera_abtest: a Cc_license_type: cc_by Cg_type: Nature Research Journals Number: 1 Primary_atype: Research Publisher: Nature Publishing Group Subject_term: Behavioural ecology;Biological anthropology Subject_term_id: behavioural-ecology;biological-anthropology.
  * Cohen and Shashua (2016) Cohen, N.; and Shashua, A. 2016.  Inductive bias of deep convolutional networks through pooling geometry.  Edition: arXiv preprint arXiv: 1605.06743.
  * Delacour (1994) Delacour, J. 1994.  _The Memory System of the Brain_.  World Scientific.  ISBN 978-981-02-1021-2.  Google-Books-ID: CCtqDQAAQBAJ.
  * Delange et al. (2021) Delange, M.; Aljundi, R.; Masana, M.; Parisot, S.; Jia, X.; Leonardis, A.; Slabaugh, G.; and Tuytelaars, T. 2021.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 1–1.
  * Frankle and Carbin (2019) Frankle, J.; and Carbin, M. 2019.  The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.  _arXiv:1803.03635 [cs]_.  ArXiv: 1803.03635.
  * Frankle et al. (2019) Frankle, J.; Dziugaite, G. K.; Roy, D. M.; and Carbin, M. 2019.  Stabilizing the lottery ticket hypothesis.  _arXiv preprint arXiv:1903.01611_.
  * Han et al. (2015) Han, S.; Pool, J.; Tran, J.; and Dally, W. J. 2015.  Learning both Weights and Connections for Efficient Neural Networks.  _CoRR_ , abs/1506.02626.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Kingma and Ba (2017) Kingma, D. P.; and Ba, J. 2017.  Adam: A Method for Stochastic Optimization.  arXiv:1412.6980.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Lecun et al. (1998) Lecun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11): 2278–2324.
  * LeCun, Denker, and Solla (1990) LeCun, Y.; Denker, J. S.; and Solla, S. A. 1990.  Optimal brain damage.  In _Advances in neural information processing systems_ , 598–605.
  * Leung, Tunkel, and Yurovsky (2021) Leung, A.; Tunkel, A.; and Yurovsky, D. 2021.  Parents Fine-Tune Their Speech to Children’s Vocabulary Knowledge.  _Psychological Science_ , 32(7): 975–984.  Publisher: SAGE Publications Inc.
  * Li et al. (2016) Li, H.; Kadav, A.; Durdanovic, I.; Samet, H.; and Graf, H. P. 2016.  Pruning Filters for Efficient ConvNets.  _CoRR_ , abs/1608.08710.
  * Li et al. (2017) Li, H.; Xu, Z.; Taylor, G.; and Goldstein, T. 2017.  Visualizing the Loss Landscape of Neural Nets.  _CoRR_ , abs/1712.09913.
  * Liang et al. (2021) Liang, T.; Glossner, J.; Wang, L.; Shi, S.; and Zhang, X. 2021.  Pruning and quantization for deep neural network acceleration: A survey.  _Neurocomputing_ , 461: 370–403.
  * Microsoft (2021) Microsoft. 2021.  microsoft/nni.  Original-date: 2018-06-01T05:51:44Z.
  * Morcos et al. (2019) Morcos, A. S.; Yu, H.; Paganini, M.; and Tian, Y. 2019.  One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers.  _arXiv preprint arXiv:1906.02773_.
  * of Veterinary Medicine Vienna (2016) of Veterinary Medicine Vienna, U. 2016.  You can teach an old dog new tricks, but younger dogs learn faster.
  * Price (1999) Price, E. O. 1999.  Behavioral development in animals undergoing domestication.  _Applied Animal Behaviour Science_ , 65(3): 245–271.
  * Ren et al. (2021) Ren, P.; Xiao, Y.; Chang, X.; Huang, P.-Y.; Li, Z.; Chen, X.; and Wang, X. 2021\.  A comprehensive survey of neural architecture search: Challenges and solutions.  _ACM Computing Surveys (CSUR)_ , 54(4): 1–34.
  * Renda, Frankle, and Carbin (2020) Renda, A.; Frankle, J.; and Carbin, M. 2020.  Comparing rewinding and fine-tuning in neural network pruning.  _arXiv preprint arXiv:2003.02389_.
  * Shea (1989) Shea, B. T. 1989.  Heterochrony in human evolution: The case for neoteny reconsidered.  _American Journal of Physical Anthropology_ , 32(S10): 69–101.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ajpa.1330320505.
  * Snow (1972) Snow, C. E. 1972.  Mothers’ Speech to Children Learning Language.  _Child Development_ , 43(2): 549–565.  Publisher: [Wiley, Society for Research in Child Development].
  * Wallis et al. (2016) Wallis, L. J.; Virányi, Z.; Müller, C. A.; Serisier, S.; Huber, L.; and Range, F. 2016.  Aging effects on discrimination learning, logical reasoning and memory in pet dogs.  _AGE_ , 38(1): 6.