Department of Energy Conversion and Storage, Technical University of Denmark,
Fysikvej, Building 310, 2800 Kgs. Lyngby These two authors contributed equally
These two authors contributed equally Danish National Metrology Institute,
Kogle Alle 5, 2970 Hørsholm, Denmark Department of Energy Conversion and
Storage, Technical University of Denmark, Fysikvej, Building 310, 2800 Kgs.
Lyngby

# Photon assisted tunneling of high order multiple Andreev reflections in
epitaxial nanowire Josephson junctions

Damon James Carrad  Center for Quantum Devices, Niels Bohr Institute,
University of Copenhagen, 2100 Copenhagen, Denmark<EMAIL_ADDRESS>Lukas
Stampfer  Center for Quantum Devices, Niels Bohr Institute, University of
Copenhagen, 2100 Copenhagen, Denmark  Dāgs Ols̆teins  Center for Quantum
Devices, Niels Bohr Institute, University of Copenhagen, 2100 Copenhagen,
Denmark  Christian Emmanuel Noes Petersen  Center for Quantum Devices, Niels
Bohr Institute, University of Copenhagen, 2100 Copenhagen, Denmark  Sabbir A.
Khan  Center for Quantum Devices, Niels Bohr Institute, University of
Copenhagen, 2100 Copenhagen, Denmark  Peter Krogstrup  Center for Quantum
Devices, Niels Bohr Institute, University of Copenhagen, 2100 Copenhagen,
Denmark  Thomas Sand Jespersen  Center for Quantum Devices, Niels Bohr
Institute, University of Copenhagen, 2100 Copenhagen, Denmark<EMAIL_ADDRESS>
###### Abstract

Semiconductor/superconductor hybrids exhibit a range of phenomena that can be
exploited for the study of novel physics and the development of new
technologies. Understanding the origin the energy spectrum of such hybrids is
therefore a crucial goal. Here, we study Josephson junctions defined by shadow
epitaxy on InAsSb/Al nanowires. The devices exhibit gate-tunable supercurrents
at low temperatures and multiple Andreev reflections (MARs) at finite voltage
bias. Under microwave irradiation, photon assisted tunneling (PAT) of MARs
produces characteristic oscillating sidebands at quantized energies, which
depend on MAR order, $n$, in agreement with a recently suggested modification
of the classical Tien-Gordon equation. The scaling of the quantized energy
spacings with microwave frequency provides independent confirmation of the
effective charge $ne$ transferred by the $n^{\mathrm{th}}$ order tunnel
process. The measurements suggest PAT as a powerful method for assigning the
origin of low energy spectral features in hybrid Josephson devices.

Keywords: semiconductor/superconductor hybrids, nanowires, multiple Andreev
reflections, photon assisted tunneling, Tien-Gordon

Traditional Josephson junctions (JJ)1 – where two superconductors are coupled
through a thin tunnel barrier – constitute the most important device element
in superconducting electronics and in state-of-the-art superconducting quantum
information processors2. Recently, there has been an increasing interest in
merging the quantum properties of superconductors with the electrostatic
control of semiconductor electronics. For example, semiconductor-based JJs
allow electrostatic tunability of the Josephson coupling,3 enabling gate-
tunable superconducting qubits,4 and qubit implementations based on the
Andreev bound states of few-channel JJs5 or topologically protected low-energy
states in one-dimensional proximity-coupled semiconductor/superconductor
hybrid devices6, 7. In all cases, the electronic states in
semiconductor/superconductor hybrid materials are of fundamental importance.
The spectrum can be probed using normal/superconductor spectroscopy8, 9, 10,
microwave spectroscopy11 or by measuring finite-bias transport in
semiconductor JJs12. In the latter case, however, low energy spectral features
can both be related to states of the leads or to higher order transport
processes13, 14, 15, 16. Sharp spectral features are expected to appear at
voltages $\pm 2\Delta/ne$ due to the sequential opening and closing of the
allowed orders $n=1,2,..$ of multiple Andreev reflection (MAR) processes. The
MAR energies are independent of the junction transparency and analysis of the
sub-gap spectrum allows extraction of the superconducting gap17, 12 and the
determination of channel transmission of the junction18, 19. In nanoscale
semiconductor JJs, unambiguous assignment of MAR features in transport becomes
problematic due to potential modifications of the sub-gap spectrum including
from charging phenomena20, 21, 22 and intrinsic properties of the
semiconductor such as spin-orbit coupling23, 24. This is increasingly
important for processes of higher-orders – i.e. lower energy – where strong
peaks unrelated to MAR can appear in the tunnel spectrum25, heating effects
may modify the gap and thus the MAR positions, and MAR orders have been
observed to be missing from the spectrum for reasons not yet understood26.
Looking beyond the hybrid device studied in this manuscript, understanding of
sub-gap spectra is crucial for interpreting results from the powerful tool of
superconducting scanning tunneling microscopy,27 which allows special mapping
of sub-gap states and is applicable to a wide variety of materials.

A characteristic feature which could distinguish MAR processes and their
orders is the transferred charge: The $n^{\mathrm{th}}$-order process,
dominating at $V_{sd}=2\Delta/ne$, transfers a charge $q_{n}=ne$ across the
junction28. While the sub-gap spectrum has been extensively investigated13,
14, 15, 16, only a few studies have directly addressed $q_{n}$. In Refs. 29,
30, 31 $q_{n}$ was investigated through the bias scaling of the noise spectrum
of JJ which was found to be consistent with the prediction. Recently, an
alternative method for measuring $q_{n}$ has been suggested based on the
evolution of the sub-gap spectrum under microwave irradiation32, 33, 27. Due
to the Tien-Gordon effect34, 35 – or photon assisted tunneling (PAT) –
modulating sidebands are expected in the non-linear sub-gap spectrum around
the $n^{\mathrm{th}}$ MAR structure. While the conventional Tien-Gordon model
predicts a characteristic modulation scale of $\delta V=\hbar f/e$ where $f$
is the frequency of the microwave field, Ref. 33, 32 suggested that in higher
order processes the total transferred charge should be used. This was
consistent with measurement of quasi-particle tunneling $(1e)$ and incoherent
Cooper pair tunneling ($2e$), and is consistent with early studies of the sub-
gap spectrum at fixed microwave frequency and power11. In this letter we
present measurements at varying microwave power and frequency of PAT of the
sub-gap spectrum in epitaxial semiconductor/superconductor InAsSb/Al nanowire
JJs realized by shadow lithography36, 37. The measurements show clear
signatures of the charge $q_{n}=ne$ of the $n^{\mathrm{th}}$ order MAR up to
$n=3$.

InAsSb nanowires (NWs) were grown by molecular beam epitaxy following the
procedure of Ref. 36. Parallel trenches were etched in a $(100)$ InAs
substrate and $90-120\,\mathrm{nm}$ diameter NWs were grown from Au catalyst
particles positioned on opposing $(111)$B inclined side facets. As shown in
Fig. 1a, the growth position of the NWs were laterally offset such that
neighboring NWs defined $\sim 100\,\mathrm{nm}$ wide shadow gaps38, 39 in the
$15\,\mathrm{nm}$ thick, epitaxially matched Al film (arrow) 40, 36, 37, 41.
NWs were transferred to a doped silicon substrate capped with
$100\,\mathrm{nm}$ SiO2, and contacts to the Al half-shell were defined using
established lithographic methods40, 42, 37, 36. Figure 1b shows a SEM
micrograph of a typical finished device and schematic measurement circuit. The
conducting Si substrate acted as back-gate, which was used to alter the
carrier density in the exposed InAsSb. Each contact electrode was split into
two individual bond pads to facilitate pseudo-four terminal measurements of
the device resistance, eliminating series resistances from e.g. low pass
filters in the cryostat cabling. The remaining series resistance – likely
dominated by the contacts – $\sim 3\,\mathrm{k}\Omega$, was identified as a
constant off-set in the supercurrent regime and was subtracted in the
following. Measurements were performed in a dilution refrigerator with a base
temperature of $15\,\mathrm{mK}$ and microwave radiation was coupled to the
device by a coaxial line with $\sim 12\,\mathrm{mm}$ of the inner conductor
left exposed and situated $\sim 5\,\mathrm{mm}$ above the sample.

Figure 1:  (a) Scanning electron micrograph (SEM) of as-grown InAsSb/Al
nanowires. Al deposition was performed from roughly bottom-right to top-left
across the image, and forward-positioned nanowires shadow those behind
resulting in Al gaps (arrow). Scale bar in inset is $1\,\mu\mathrm{m}$. (b)
False colour SEM of a device featuring InAsSb nanowire (grey), shadow-
patterned Al layer (blue) and Ti/Au ohmic contacts (gold). (c,d) Gate
dependence at $T=15\,\mathrm{mK}$ of the differential resistance
$\mathrm{d}V/\mathrm{d}I$ vs. measured current $I$, and voltage $V$,
respectively. A zero resistance supercurrent is present between switching and
retrapping currents $I_{\mathrm{sw}}$, $I_{\mathrm{r}}$. (e) Line trace in (d)
at fixed $V_{\mathrm{G}}=-25\,\mathrm{V}$. Dashed lines indicate features at
voltages $\pm 2\Delta/e$ (orange), $\pm\Delta/e$ (purple), and $\pm
2\Delta/3e$ (blue) with $\delta=225\,\mu\mathrm{eV}$ attributed to $n=1,2,3$
MAR. The corresponding transport processes are schematically illustrated in
(f).

Figure 1c shows the differential resistance, $dV/dI$, as a function of current
through the device, $I$, and back-gate voltage $V_{\mathrm{G}}$. A zero
resistance supercurrent is observed for low $|I|$. The current was swept from
positive to negative (arrow) and the switching(re-trapping) current where the
device transitions to(from) a resistive state from(to) the superconducting
state is indicated. The differences in $I_{\mathrm{sw}}$ and $I_{\mathrm{r}}$
are commonly observed in nanowire JJs3 and ascribed to a consequence of Joule
heating in the dissipative regime and/or underdamped dynamics of the JJ43, 44.
Both $I_{\mathrm{sw}}$ and $I_{\mathrm{r}}$ decrease upon lowering
$V_{\mathrm{G}}$ as expected from a JJ with constant
$I_{\mathrm{c}}R_{\mathrm{N}}=\pi\Delta/2e$ and an $n$-type semiconductor
which depletes with lowering $V_{\mathrm{G}}$. Here $I_{\mathrm{c}}$ is the
critical current and $R_{\mathrm{N}}$ is the normal-state device resistance.
The aperiodic oscillations in $I_{\mathrm{sw}}(V_{\mathrm{G}})$ appearing in
addition to the overall gate-dependence is attributed to universal conductance
fluctuations in $R_{\mathrm{N}}(V_{\mathrm{G}})$ 3. At
$V_{\mathrm{G}}\sim-25\,\mathrm{V}$ the normal state resistance is
$R_{\mathrm{N}}\sim 4.5\mathrm{k}\Omega$ and
$I_{\mathrm{sw}}R_{\mathrm{N}}=200~{}\mathrm{\mu V}$, relatively close to the
theoretical value $\pi\Delta_{\mathrm{Al}}/(2e)=350~{}\mathrm{\mu V}$, as
previously reported for similar hybrids36. The measured excess current
(Supporting Fig. S1) $I_{\mathrm{ex}}\sim 50\,\mathrm{nA}$ corresponds to a
junction transmission of $\sim 0.7$45.

For $I$ above/below the zero-resistance region in Fig. 1c, $dV/dI$ exhibits a
rich structure which follows the trend of $I_{\mathrm{sw}}$ and
$I_{\mathrm{r}}$. To gain insight into the origin of these features, Fig. 1d
shows the same data as a function of the measured voltage $V$. Upon sweeping
$I$ down from zero, the voltage remains $V=0\,\mathrm{V}$ until it abruptly
switches at $I_{\mathrm{sw}}$ to a finite value. Therefore no data is obtained
with $V$ below this value, and this region is greyed-out in Fig. 1d.
Conversely, $V$ gradually decreases to zero around $I_{\mathrm{r}}$ when
sweeping $I$ down from positive values due to the underdamped junction
dynamics,43, 44 enabling data extraction down to $V=0$ for $I>0$. As expected
for MAR, the structure in Fig. 1d appears at fixed voltages despite the
variation in conductance with $V_{\mathrm{G}}$. Figure 1e shows a line trace
of $dV/dI$ vs. $V$ for fixed $V_{\mathrm{G}}=-25\,\mathrm{V}$. The three most
pronounced dips are highlighted by dashed lines and follow $\pm 2\Delta/e$
(orange), $\pm\Delta/e$ (purple), and $\pm 2\Delta/3e$ (blue) with
$\Delta=225\,\mu\mathrm{eV}$ consistent with prior results from 15 nm thick Al
46, 42, 36. This series is expected for MAR and the relevant processes
transfer charges of $1e,2e$ and $3e$ and are illustrated in Fig. 1f, where the
slopes of the arrows represent voltage biases. Direct quasi particle tunneling
$(1e)$ occurs for $|V|\geq 2\Delta/e$. In the general case of $n\geq 2$, the
$n^{\mathrm{th}}$ order process involves $n-1$ Andreev reflections, a charge
transfer of $ne$, and is allowed for $|V|\geq 2\Delta/(ne)$. Suppression of
$n\geq 3$ processes occur for $e|V|\geq 2\Delta/(n-2)$ due to the low
probability of Andreev reflection above the gap47. Overall, the dc transport
characteristics in Fig. 1 are comparable to previous NW-based JJs3, 25, 48,
26.

Figure 2:  (a) Differential resistance at $V_{\mathrm{G}}=-25\mathrm{V}$ as a
function of bias, $V$, and the rms amplitude of $6\,\mathrm{GHz}$ microwave
radiation. The bottom axis represents energy $E$ in units of $2\Delta$, with
$\Delta=225~{}\mu$eV and the energy intervals for the first three MAR
processes are indicated by horizontal lines. (b) The second derivative,
$d^{2}V/dI^{2}$, in a region of (a) around the higher order MAR showing the
Tien-Gorden modulation in the $n=2$ and $n=3$ MAR processes. The changing
period of modulation is indicated by arrows. (c) A simulation of the data in
(b), based on Eq. 1 modified to account for the charge of the higher order
transport processes. Processes of different order are assumed to dominate in
the indicated regions.

We now consider the features arising from irradiating the device with
microwaves. Figure 2a shows $dV/dI$ vs. $V$ and the rms amplitude,
$V_{\mathrm{app}}^{\mathrm{rms}}$, of $6\,\mathrm{GHz}$ microwave radiation,
as applied to the antenna (Fig. 1b). The gate voltage was fixed at
$V_{\mathrm{G}}=-25\mathrm{V}$ as in Fig. 1e – similar features were observed
at other $V_{\mathrm{G}}$, see Supporting Section II – and $I$ was swept from
negative to positive. A series of ‘V’-shaped patterns are observed, fanning
out from $V_{\mathrm{app}}^{\mathrm{rms}}=0$. The structure emerging from
$V=0\,\mathrm{V}$ – i.e., when the JJ carries a supercurrent – is due to
Shapiro steps49, 1 caused by phase-locking between the microwave field and the
superconducting phase difference across the JJ leading to at step-wise
increase of $V$ in units of $hf/(2e)$. In Supporting Fig. S3, we replot this
regime as a function $I$ as is common for studies of Shapiro steps, showing
the conventional Shapiro pattern3, 50, 48, 51. The measurements are well-
described by the extended RCSJ model and will not be considered further here.

The remaining V-shaped structures in Fig. 2a, which evolve from the MAR
features at finite $V$, have a different origin and are the main focus of this
work. Since the junction does not carry a supercurrent, phase locking and the
resultant Shapiro steps do not occur. Instead, we attribute the structures to
photon assisted tunneling. In general, for any two terminal device absorbing
radiation at frequency $f$, charges can absorb/emit photons and thereby gain
energy $\pm lhf$ and tunnel across the junction – here $h$ is the Planck
constant and $l=0,\pm 1,\pm 2,\ldots$. Thus, at finite irradiation amplitude,
the current at the dc voltage $V$ is related to the dc current in the absence
of irradiation, $I^{0}$, at $V\pm hfl/e$. For smooth $I^{0}(V)$, this has no
consequence, however, non-linearities in $I^{0}(V)$ will repeat at finite
radiation amplitude as sidebands with an intensity modulated by the radiation
amplitude. In the simplest case where the radiation field on the sample is
sinusoidal, $V_{\mathrm{abs}}=V_{\mathrm{abs}}^{\mathrm{pp}}\cdot\sin(2\pi
ft)$, the current at finite radiation amplitude is given by the Tien-Gordon
equation 34:

$I_{\mathrm{dc}}(V)=\sum_{l}J^{2}_{l}\left(\frac{eV_{\mathrm{abs}}^{\mathrm{pp}}}{hf}\right)\cdot
I^{0}\left(V+\frac{hfl}{e}\right)$ (1)

where $J_{l}(x)$ is the $l^{\mathrm{th}}$ Bessel function. The amplitude of
each sideband oscillates with increasing $V_{\mathrm{abs}}^{\mathrm{pp}}$
according to $J_{l}^{2}$ (Supporting Fig. S4). This explains the overall shape
of the structures in Fig. 2a which evolve from the non-linearities at
$V=2\Delta/ne$ associated with MAR. The highly asymmetric MAR features in Fig.
1e makes the inflection points of $dV/dI$ – i.e. peaks in $d^{2}V/dI^{2}$ –
well suited for identifying the onset of MAR as shown in Fig. 2b. The
modulation period of the features associated with the $n=1,2,3$ order MARs can
be clearly resolved and as highlighted by the black arrows, the modulation
period is not the same for each ‘V’-fan but rather decreases with decreasing
bias. This is inconsistent with the conventional Tien-Gordon model described
above which predicts the same characteristic pattern evolving from all non-
linearities in $I^{0}$. Instead, as discussed in the introduction, the pattern
in Fig. 2a,b follows the work of Refs 32, 33 suggesting that Eq. 1 describes
the microwave response for higher-order tunnel processes if the electron
charge, $e$, is replaced by the total transferred charge $q=ne$: Refs 32, 33
experimentally confirmed this for quasiparticle ($e$) and incoherent Cooper
pair tunneling at zero bias ($2e$). For our case, the $n^{\mathrm{th}}$ order
MAR transfers $ne$ electrons (Fig. 1f), and the corresponding modulation
period should be $\Delta V_{n}=hf/ne$. For $f=6\,\mathrm{GHz}$ this amounts to
$24.8\,\mu\mathrm{V}$, $12.4\,\mu\mathrm{V}$, and $8.3\,\mu\mathrm{V}$ for
$n=1,2,3$, respectively, in good agreement with the measured splittings
$23.8\,\mu\mathrm{V}$,$12.2\,\mu\mathrm{V}$, and $9.3\,\mu\mathrm{V}$ from
Fig. 2a,b (see below). Using this assumption we show in Fig. 2c the result of
a simulation of $d^{2}I/dV^{2}$ taking the $I-V$ curve in absence of radiation
as input and evolving according to Eq. 1 under the simplifying assumption that
the $n^{\mathrm{th}}$ order MAR dominate in the region around $2\Delta/ne$ as
shown by the dashed lines Fig. 2c (see Supporting Section IV for details).
Comparing to Fig. 2b, the simulation accurately captures the main features of
the measurement.

Figure 3:  (a) $dV/dI$ vs $V$ and normalised absorbed microwave voltage
$eV^{\mathrm{pp}}_{\mathrm{abs}}/(hf)$ for $f=6$,8 and 10 GHz. The 6 GHz data
is a subset of Fig. 2a. The sideband spacing within each MAR order increases
with increasing $f$. (b) Average sideband spacing $\left<\delta E\right>$ as a
function of $f$ for $n=1,2,3$ (blue, orange and green data points). The blue,
orange and green dashed lines show the theoretical $\delta E$ vs $f$
relationships for 1, 2 and 3 electron processes.

To further quantify the splitting, the measurements of Fig. 2a were repeated
for different frequencies $f=5-12$ GHz. Three representative examples are
shown in Fig. 3a for $f=6,8,10$ GHz. The coupling between the antenna and
device depends strongly and non-monotonically on $f$, and therefore the
amplitude of absorbed radiation, $V_{\mathrm{abs}}^{\mathrm{pp}}$ in Fig. 3a
was for visualization purposes normalized to the first zero of $J_{0}$ which
occurs at $eV_{\mathrm{abs}}^{pp}/(hf)=2.4$ (see Supporting information for
non-normalized data at all eight frequencies). Figure 3b shows the average of
the sideband spacing $\left<\delta E\right>=e\left<\delta V\right>$, as a
function of $f$ (data points) and the theoretical relationship $\delta E=hf/n$
(dashed lines) for $n=1,2$ and 3 (blue, orange and green). The average was
obtained across all the measured $\delta E$ within each order, and the error
bars represent the standard deviation. The good agreement between theory and
experiment quantitatively confirms that the $n^{\mathrm{th}}$ order MAR
process involves the transmission of $n$ electrons across the junction.

In conclusion, we have studied the finite-bias sub-gap structure in epitaxial
InAsSb/Al JJs under microwave radiation. We showed that the microwave response
provides a sensitive probe of the transmitted charge $q=ne$ up to order $n=3$
– i.e. a MAR process including two Andreev reflections. The modulation period
was found to decrease with increasing MAR order, and the dependence of the
sideband spacings on the radiation amplitude agrees quantitatively with the
expectation that each order transmits $n$ electrons across the junction. A
simulation based on a modified version of the Tien-Gordon equation was found
to accurately reproduce the measurements. We expect the effect to be
observable in non-epitaxially matched hybrids, albeit likely with less sharp
features than observed here,42 and possibly modified weight of the MAR due to
changes in the interface transmission.16 We believe that the results will be
valuable tool for understanding the low-energy sub-gap spectrum in hybrid
nanoscale devices. This becomes increasingly relevant with the rapid
developments in the quality of hybrid quantum materials, device fabrication
and filtering and measurement techniques, which allows for increasing
coherence lengths and increased spectral resolution and thus even higher order
MAR at progressively lower energies. Finally, the identification of sub-gap
states is a crucial task for the powerful technique of spatially resolved
superconducting scanning tunneling microscopy.

Acknowledgement This research was supported by the Danish National Research
Foundation, Microsoft Quantum Materials Lab, and by research grants from
Villum Fonden (00013157), The Danish Council for Independent Research
(7014-00132), and European Research Council (866158).

## Competing interests

The authors declare no competing interests.

## Supporting Information

The supporting information contains additional data relating to the excess
current, Shapiro steps, PAT of MAR at a different gate voltage, the underlying
measurements for Fig. 3, as well as extra information regarding the simulation
in Fig. 2c. This material is available free of charge via the internet at
http://pubs.acs.org.

## References

  * Josephson 1962 Josephson, B. Possible new effects in superconductive tunnelling. _Phys. Lett._ 1962, _1_ , 251–253
  * Kjaergaard et al. 2020 Kjaergaard, M.; Schwartz, M. E.; Braumüller, J.; Krantz, P.; Wang, J. I.-J.; Gustavsson, S.; Oliver, W. D. Superconducting Qubits: Current State of Play. _Annu. Rev. Condens. Matter Phys._ 2020, _11_ , 369–395
  * Doh et al. 2005 Doh, Y.; van Dam, J.; Roest, A.; Bakkers, E.; Kouwenhoven, L.; De Franceschi, S. Tunable supercurrent through semiconductor nanowires. _Science_ 2005, _309_ , 272–275
  * Larsen et al. 2015 Larsen, T. W.; Petersson, K. D.; Kuemmeth, F.; Jespersen, T. S.; Krogstrup, P.; Nygård, J.; Marcus, C. M. Semiconductor-Nanowire-Based Superconducting Qubit. _Phys. Rev. Lett._ 2015, _115_ , 127001
  * Hays et al. 2018 Hays, M.; de Lange, G.; Serniak, K.; van Woerkom, D. J.; Bouman, D.; Krogstrup, P.; Nygård, J.; Geresdi, A.; Devoret, M. H. Direct Microwave Measurement of Andreev-Bound-State Dynamics in a Semiconductor-Nanowire Josephson Junction. _Phys. Rev. Lett._ 2018, _121_ , 047001
  * Oreg et al. 2010 Oreg, Y.; Refael, G.; von Oppen, F. Helical Liquids and Majorana Bound States in Quantum Wires. _Phys. Rev. Lett._ 2010, _105_ , 177002
  * Lutchyn et al. 2010 Lutchyn, R. M.; Sau, J. D.; Das Sarma, S. Majorana Fermions and a Topological Phase Transition in Semiconductor-Superconductor Heterostructures. _Phys. Rev. Lett._ 2010, _105_ , 077001
  * Giaever 1960 Giaever, I. Energy Gap in Superconductors Measured by Electron Tunneling. _Phys. Rev. Lett._ 1960, _5_ , 147–148
  * Deacon et al. 2010 Deacon, R. S.; Tanaka, Y.; Oiwa, A.; Sakano, R.; Yoshida, K.; Shibata, K.; Hirakawa, K.; Tarucha, S. Tunneling Spectroscopy of Andreev Energy Levels in a Quantum Dot Coupled to a Superconductor. _Phys. Rev. Lett._ 2010, _104_ , 076805
  * Pillet et al. 2010 Pillet, J.-D.; Quay, C. H. L.; Morfin, P.; Bena, C.; Yeyati, A. L.; Joyez, P. Andreev Bound States in Supercurrent-Carrying Carbon Nanotubes Revealed. _Nature Phys_ 2010, _6_ , 965–969
  * Chauvin et al. 2006 Chauvin, M.; vom Stein, P.; Pothier, H.; Joyez, P.; Huber, M. E.; Esteve, D.; Urbina, C. Superconducting Atomic Contacts under Microwave Irradiation. _Phys. Rev. Lett._ 2006, _97_ , 067006
  * Heedt et al. 2021 Heedt, S. et al. Shadow-Wall Lithography of Ballistic Superconductor–Semiconductor Quantum Devices. _Nat. Commun._ 2021, _12_ , 4914
  * Octavio et al. 1983 Octavio, M.; Tinkham, M.; Blonder, G.; Klapwijk, T. Subharmonic energy-gap structure in superconducting constrictions. _Phys. Rev. B_ 1983, _27_ , 6739–6746
  * Blonder et al. 1982 Blonder, G.; Tinkham, M.; Klapwijk, T. Transition from metallic to tunneling regimes in superconducting micro-constrictions - excess current, charge imbalance, and super-current conversion. _Phys. Rev. B_ 1982, _25_ , 4515–4532
  * Bratus et al. 1995 Bratus, E.; Shumeiko, V.; Wendin, G. Theory of subharmonic gap structure in superconducting mesoscopic tunnel contacts. _Phys. Rev. Lett._ 1995, _74_ , 2110–2113
  * Averin and Bardas 1995 Averin, D.; Bardas, A. ac Josephson Effect in a Single Quantum Channel. _Phys. Rev. Lett._ 1995, _75_ , 1831–1834
  * Kjaergaard et al. 2017 Kjaergaard, M.; Suominen, H. J.; Nowak, M. P.; Akhmerov, A. R.; Shabani, J.; Palmstrøm, C. J.; Nichele, F.; Marcus, C. M. Transparent Semiconductor-Superconductor Interface and Induced Gap in an Epitaxial Heterostructure Josephson Junction. _Phys. Rev. Applied_ 2017, _7_ , 034029
  * Scheer et al. 1998 Scheer, E.; Agrait, N.; Cuevas, J.; Yeyati, A.; Ludoph, B.; Martin-Rodero, A.; Bollinger, G.; van Ruitenbeek, J.; Urbina, C. The signature of chemical valence in the electrical conduction through a single-atom contact. _Nature_ 1998, _394_ , 154–157
  * Goffman et al. 2017 Goffman, M. F.; Urbina, C.; Pothier, H.; Nygård, J.; Marcus, C. M.; Krogstrup, P. Conduction Channels of an InAs-Al Nanowire Josephson Weak Link. _New J. Phys._ 2017, _19_ , 092002
  * Buitelaar et al. 2003 Buitelaar, M.; Belzig, W.; Nussbaumer, T.; Babic, B.; Bruder, C.; Schönenberger, C. Multiple Andreev reflections in a carbon nanotube quantum dot. _Phys. Rev. Lett._ 2003, _91_ , 057005
  * Eichler et al. 2007 Eichler, A.; Weiss, M.; Oberholzer, S.; Schönenberger, C.; Levy Yeyati, A.; Cuevas, J. C.; Martin-Rodero, A. Even-odd effect in Andreev transport through a carbon nanotube quantum dot. _Phys. Rev. Lett._ 2007, _99_ , 126602
  * Sand-Jespersen et al. 2007 Sand-Jespersen, T.; Paaske, J.; Andersen, B. M.; Grove-Rasmussen, K.; Jorgensen, H. I.; Aagesen, M.; Sorensen, C. B.; Lindelof, P. E.; Flensberg, K.; Nygård, J. Kondo-enhanced Andreev tunneling in InAs nanowire quantum dots. _Phys. Rev. Lett._ 2007, _99_ , 126603
  * Dolcini and Dell’Anna 2008 Dolcini, F.; Dell’Anna, L. Multiple Andreev Reflections in a Quantum Dot Coupled to Superconducting Leads: Effect of Spin-Orbit Coupling. _Phys. Rev. B_ 2008, _78_ , 024518
  * Hajer et al. 2019 Hajer, J.; Kessel, M.; Brüne, C.; Stehno, M. P.; Buhmann, H.; Molenkamp, L. W. Proximity-Induced Superconductivity in CdTe–HgTe Core–Shell Nanowires. _Nano Lett._ 2019, _19_ , 4078–4082
  * Abay et al. 2014 Abay, S.; Persson, D.; Nilsson, H.; Wu, F.; Xu, H. Q.; Fogelström, M.; Shumeiko, V.; Delsing, P. Charge transport in InAs nanowire Josephson junctions. _Phys. Rev. B_ 2014, _89_ , 214508
  * Xiang et al. 2006 Xiang, J.; Vidan, A.; Tinkham, M.; Westervelt, R. M.; Lieber, C. M. Ge/Si nanowire mesoscopic Josephson junctions. _Nat. Nanotechnol._ 2006, _1_ , 208–213
  * Peters et al. 2020 Peters, O.; Bogdanoff, N.; Acero González, S.; Melischek, L.; Simon, J. R.; Reecht, G.; Winkelmann, C. B.; von Oppen, F.; Franke, K. J. Resonant Andreev Reflections Probed by Photon-Assisted Tunnelling at the Atomic Scale. _Nat. Phys._ 2020, _16_ , 1222–1226
  * Cuevas et al. 1999 Cuevas, J.; Martin-Rodero, A.; Yeyati, A. Shot noise and coherent multiple charge transfer in superconducting quantum point contacts. _Phys. Rev. Lett._ 1999, _82_ , 4086–4089
  * Cron et al. 2001 Cron, R.; Goffman, M.; Esteve, D.; Urbina, C. Multiple-charge-quanta shot noise in superconducting atomic contacts. _Phys. Rev. Lett._ 2001, _86_ , 4104–4107
  * Kozhevnikov et al. 2000 Kozhevnikov, A.; Schoelkopf, R.; Prober, D. Observation of photon-assisted noise in a diffusive normal metal-superconductor junction. _Phys. Rev. Lett._ 2000, _84_ , 3398–3401
  * Ronen et al. 2016 Ronen, Y.; Cohen, Y.; Kang, J.-H.; Haim, A.; Rieder, M.-T.; Heiblum, M.; Mahalu, D.; Shtrikman, H. Charge of a quasiparticle in a superconductor. _P. Natl. Acad. Sci._ 2016, _113_ , 1743–1748
  * Kot et al. 2020 Kot, P.; Drost, R.; Uhl, M.; Ankerhold, J.; Cuevas, J. C.; Ast, C. R. Microwave-Assisted Tunneling and Interference Effects in Superconducting Junctions under Fast Driving Signals. _Phys. Rev. B_ 2020, _101_ , 134507
  * Roychowdhury et al. 2015 Roychowdhury, A.; Dreyer, M.; Anderson, J. R.; Lobb, C. J.; Wellstood, F. C. Microwave Photon-Assisted Incoherent Cooper-Pair Tunneling in a Josephson STM. _Phys. Rev. Applied_ 2015, _4_ , 034011
  * Tien and Gordon 1963 Tien, P.; Gordon, J. Multiphoton process observed in interaction of microwave fields with tunneling between superconductor films. _Phys. Rev._ 1963, _129_ , 647
  * Platero and Aguado 2004 Platero, G.; Aguado, R. Photon-assisted transport in semiconductor nanostructures. _Phys. Rep._ 2004, _395_ , 1–157
  * Khan et al. 2020 Khan, S. A. et al. Highly Transparent Gatable Superconducting Shadow Junctions. _ACS Nano_ 2020, _14_ , 14605–14615
  * 37 Carrad, D. J.; Bjergfelt, M.; Kanne, T.; Aagesen, M.; Krizek, F.; Fiordaliso, E. M.; Johnson, E.; Nygård, J.; Jespersen, T. S. Shadow Epitaxy for In Situ Growth of Generic Semiconductor/Superconductor Hybrids. _Adv. Mater._ _32_ , 1908411
  * Gazibegovic et al. 2017 Gazibegovic, S. et al. Epitaxy of advanced nanowire quantum devices. _Nature_ 2017, _548_ , 434
  * Krizek et al. 2017 Krizek, F.; Kanne, T.; Razmadze, D.; Johnson, E.; Nygård, J.; Marcus, C. M.; Krogstrup, P. Growth of InAs Wurtzite Nanocrosses from Hexagonal and Cubic Basis. _Nano Lett._ 2017, _17_ , 6090–6096
  * Krogstrup et al. 2015 Krogstrup, P.; Ziino, N. L. B.; Chang, W.; Albrecht, S. M.; Madsen, M. H.; Johnson, E.; Nygård, J.; Marcus, C. M.; Jespersen, T. S. Epitaxy of semiconductor-superconductor nanowires. _Nat. Mater._ 2015, _14_ , 400–406
  * Sestoft et al. 2018 Sestoft, J. E.; Kanne, T.; Gejl, A. N.; von Soosten, M.; Yodh, J. S.; Sherman, D.; Tarasinski, B.; Wimmer, M.; Johnson, E.; Deng, M.; Nygård, J.; Jespersen, T. S.; Marcus, C. M.; Krogstrup, P. Engineering Hybrid Epitaxial InAsSb/Al Nanowires for Stronger Topological Protection. _Phys. Rev. Materials_ 2018, _2_ , 044202
  * Chang et al. 2015 Chang, W.; Albrecht, S. M.; Jespersen, T. S.; Kuemmeth, F.; Krogstrup, P.; Nygård, J.; Marcus, C. M. Hard gap in epitaxial semiconductor–superconductor nanowires. _Nat. Nanotechnol._ 2015, _10_ , 232–236
  * McCumber 1968 McCumber, D. E. Effect of ac Impedance on dc Voltage‐Current Characteristics of Superconductor Weak‐Link Junctions. _Journal of Applied Physics_ 1968, _39_ , 3113–3118
  * Tinkham et al. 2003 Tinkham, M.; Free, J. U.; Lau, C. N.; Markovic, N. Hysteretic I-V curves of superconducting nanowires. _Phys. Rev. B_ 2003, _68_ , 134515
  * Flensberg et al. 1988 Flensberg, K.; Hansen, J. B.; Octavio, M. Subharmonic energy-gap structure in superconducting weak links. _Phys. Rev. B_ 1988, _38_ , 8707–8711
  * Deng et al. 2016 Deng, M. T.; Vaitiekėnas, S.; Hansen, E. B.; Danon, J.; Leijnse, M.; Flensberg, K.; Nygård, J.; Krogstrup, P.; Marcus, C. M. Majorana bound state in a coupled quantum-dot hybrid-nanowire system. _Science_ 2016, _354_ , 1557–1562
  * Klapwijk et al. 1982 Klapwijk, T.; Blonder, G.; Tinkham, M. Explanation of subharmonic energy gap structure in superconducting contacts. _Physica B+C_ 1982, _109-110_ , 1657–1664, 16th International Conference on Low Temperature Physics, Part 3
  * Ridderbos et al. 2019 Ridderbos, J.; Brauns, M.; Li, A.; Bakkers, E. P. A. M.; Brinkman, A.; van der Wiel, W. G.; Zwanenburg, F. A. Multiple Andreev reflections and Shapiro steps in a Ge-Si nanowire Josephson junction. _Phys. Rev. Materials_ 2019, _3_ , 084803
  * Shapiro 1963 Shapiro, S. Josephson Currents in Superconducting Tunneling: The Effect of Microwaves and Other Observations. _Phys. Rev. Lett._ 1963, _11_ , 80–82
  * Larson et al. 2020 Larson, T. F. Q.; Zhao, L.; Arnault, E. G.; Wei, M.-T.; Seredinski, A.; Li, H.; Watanabe, K.; Taniguchi, T.; Amet, F.; Finkelstein, G. Zero Crossing Steps and Anomalous Shapiro Maps in Graphene Josephson Junctions. _Nano Letters_ 2020, _20_ , 6998–7003
  * Perla et al. 2021 Perla, P.; Fonseka, H. A.; Zellekens, P.; Deacon, R.; Han, Y.; Kölzer, J.; Mörstedt, T.; Bennemann, B.; Espiari, A.; Ishibashi, K.; Grützmacher, D.; Sanchez, A. M.; Lepsa, M. I.; Schäpers, T. Fully _in Situ_ Nb/InAs-nanowire Josephson Junctions by Selective-Area Growth and Shadow Evaporation. _Nanoscale Adv._ 2021, _3_ , 1413–1421


* (59) W. Florkowski, E. Maksymiuk, R. Ryblewski, M. Strickland, Exact solution of the (0+1)-dimensional Boltzmann equation for a massive gas, Phys. Rev. C 89 (5) (2014) 054908.  arXiv:1402.7348, doi:10.1103/PhysRevC.89.054908.
  * (60) W. Florkowski, O. Madetko, Kinetic description of mixtures of anisotropic fluids, Acta Phys. Polon. B 45 (5) (2014) 1103–1118.  arXiv:1402.2401, doi:10.5506/APhysPolB.45.1103.
  * (61) C. Chattopadhyay, S. Jaiswal, L. Du, U. Heinz, S. Pal, Non-conformal attractor in boost-invariant plasmas, Phys. Lett. B 824 (2022) 136820.  arXiv:2107.05500, doi:10.1016/j.physletb.2021.136820.
  * (62) W. Israel, J. M. Stewart, Transient relativistic thermodynamics and kinetic theory, Annals Phys. 118 (1979) 341–372.  doi:10.1016/0003-4916(79)90130-1.
  * (63) A. Behtash, C. N. Cruz-Camacho, M. Martinez, Far-from-equilibrium attractors and nonlinear dynamical systems approach to the Gubser flow, Phys. Rev. D97 (2018) 044041.  arXiv:1711.01745, doi:10.1103/PhysRevD.97.044041.
  * (64) S. Kamata, J. Jankowski, M. Martinez, Novel features of attractors and transseries in non-conformal Bjorken flowsarXiv:2206.00653.
  * (65) A. Jaiswal, B. Friman, K. Redlich, Relativistic second-order dissipative hydrodynamics at finite chemical potential, Phys. Lett. B 751 (2015) 548–552.  arXiv:1507.02849, doi:10.1016/j.physletb.2015.11.018.
  * (66) A. Muronga, Causal theories of dissipative relativistic fluid dynamics for nuclear collisions, Phys. Rev. C 69 (2004) 034903.  arXiv:nucl-th/0309055, doi:10.1103/PhysRevC.69.034903.
  * (67) C. Chattopadhyay, A. Jaiswal, S. Pal, R. Ryblewski, Relativistic third-order viscous corrections to the entropy four-current from kinetic theory, Phys. Rev. C91 (2) (2015) 024917.  arXiv:1411.2363, doi:10.1103/PhysRevC.91.024917.


# On the Effectiveness of Neural Ensembles for Image Classification with Small
Datasets

Lorenzo Brigato
Sapienza University of Rome
<EMAIL_ADDRESS>Luca Iocchi
Sapienza University of Rome
<EMAIL_ADDRESS>
###### Abstract

Deep neural networks represent the gold standard for image classification.
However, they usually need large amounts of data to reach superior
performance. In this work, we focus on image classification problems with a
few labeled examples per class and improve data efficiency by using an
ensemble of relatively small networks.
For the first time, our work broadly studies the existing concept of neural
ensembling in domains with small data, through extensive validation using
popular datasets and architectures. We compare ensembles of networks to their
deeper or wider single competitors given a total fixed computational budget.
We show that ensembling relatively shallow networks is a simple yet effective
technique that is generally better than current state-of-the-art approaches
for learning from small datasets. Finally, we present our interpretation
according to which neural ensembles are more sample efficient because they
learn simpler functions.

##  1 Introduction

The advent of deep learning (DL) has revolutionized the computer vision field
[43]. However, the cost to reach high recognition performances involves the
collection and labeling of large quantities of images. This requirement can
not always be fulfilled since it may happen that collecting images is
extremely expensive or not possible at all. Different approaches have been
proposed by the research community to mitigate the necessity of training data,
tackling the problem from different perspectives. Among them of particular
interest are definitely transfer and few-shot learning [4], [36] [37]. Still,
these approaches rely on a large set of image-annotation pairs on which re-
usable representations can be learnt.

In this work, we propose the use of neural ensembles composed of smaller
networks to tackle the problem of learning from a small sample and show the
superiority of such methodology. Similarly to what has been done in recent
works, we benchmark the approaches by varying the number of data points in the
training sample while keeping it low with respect to the current standards of
computer vision datasets [1], [3]. Due to its great difficulty, this problem
is still unsolved and hardly experimented despite its primary importance.

It has been shown that large convolutional neural networks (CNNs) can handle
overfitting and generalize well even if they are severely over-parametrized
[19], [25]. A recent study has also empirically shown that such behavior might
also be valid in the case of tiny datasets, making large nets a viable choice
even when the training sample is limited [6]. On the other hand, a well-known
technique to reduce model variance is to average predictions from a set of
weak learners (e.g. random forests [7]). An ensemble of low-bias decorrelated
learners, combined with randomized inputs and prediction averaging, generally
mitigates overfitting. Despite the high popularity of neural ensembles, our
study is the first one that systematically trains them from scratch on
datasets with few samples per class and gives empirical evidence of their
advantage over state-of-the-art approaches. Our strict experimental
methodology compares ensembles with state-of-the-art methods and single
networks keeping a fair comparison in terms of model resources.

Therefore, we study ensembles of CNNs in small-data tasks by a) fixing a
computational budget and b) comparing them to corresponding deeper or wider
single variants. According to our empirical study, ensembles are preferable
over wider networks that are in turn better than deeper ones. Moreover,
obtained results make ensembles of smaller networks a strong baseline and an
advantageous basic building block for future works that will tackle the
problem of learning from small datasets.

In summary, the contributions of our work are the following: _i_) we
systematically study neural ensembles with small datasets and show that they
generally outperform state-of-the-art methods; _ii_) we make a structured
study comparing ensembles of smaller-scale networks and their computationally
equivalent single competitors with increased depth or width; _iii_) we explain
the better performance of ensembles by showing their bias towards learning
less complex functions.

##  2 Related Work

#### Learning from limited data

is still largely unsolved. As previously said, few works have tried to tackle
the problem of training DL architectures with a small number of samples due to
its difficulty. A series of works focused on the classification of vector data
and mainly used the UCI Machine Learning Repository as a benchmark [12], [28],
[1].

However, in this work, we are interested in image classifiers. Transformations
that synthesize new images are a straightforward approach to improve
generalization (e.g. data augmentation [33]). Whit limited data, it was shown
that even standard data augmentation is fundamental to raise the performance
of neural networks [8]. More particular domains like face recognition instead,
should rely on had-hoc approaches [16]. Training generative models (e.g. GANs)
represents an attractive approach to increase the dataset size and
consequently, performance [22]. However, training a generative model might be
computationally intensive or present severe challenges in the small sample
domain (e.g. overfitting, mode collapse) [18]. [2] proposed to generate images
directly optimizing a fixed latent space to obviate such difficulties. [32]
suggested training CNNs with a greedy layer-wise method, analogous to that
used in unsupervised deep networks. [5] presented diverse kernel-based
regularization techniques to regularize deep networks. [3] proposed the use of
the cosine loss to prevent overfitting. On the other hand, [1] performed
experiments with convolutional neural tangent kernel (CNTK). Finally, [42]
proposed a rotation-invariant and scale-invariant architecture to improve
generalization and data efficiency.

#### Ensembles of neural networks

are widely successful for improving both the accuracy and predictive
uncertainty of very different predictive tasks, ranging from supervised
learning [14] to few-shot learning [11].
Recent studies empirically show that deep ensembles can outperform single
models with equivalent computational budgets on image classification tasks
[10], [9], [13], [40]. We have in common with these works the outcome, i.e.
the superiority of neural ensembles over single networks. However, our study
spans different directions of the problem space. Indeed, [13] suggested the
possibility of obtaining better performance with ensembles but without
performing an extensive empirical evaluation with state-of-the-art
architectures. [10] mainly focused their attention on the possible strategies
to combine the predictions of the ensemble members. Differently from [9], we
compare neural ensembles with single networks not only of the same depth but
also of the same width and greater depth. Furthermore, given few training
examples as in our study, we empirically show that claims regarding the
advantage of depth over width do not hold [40]. Indeed, we present evidence
that deeper networks tend to struggle in the case of small datasets. Finally,
in contrast with previous work, we provide a more solid interpretation of the
better performance of ensembles. We base our reasoning on a sensitivity metric
that was proposed to measure neural function complexity [27].
A drawback of neural ensembles regards the computational cost that generally
grows linearly with the ensemble dimension. Some works mitigated such problem
in the field of continual and supervised learning [41], [39]. We share this
concern by fixing a budget, making the computational cost of training
ensembles equivalent or lower to the one of a single network [40].

##  3 Problem Definition and Methodology

###  3.1 Image recognition with small data

We are dealing with a problem in which we assume to have a dataset of samples
and corresponding labels
$\mathcal{D}=\\{(\mathbf{x}_{1},\mathbf{y}_{1})\dots,(\mathbf{x}_{s},\mathbf{y}_{s})\\}$
where each label $\mathbf{y}$ is a one-hot vector that is encoding $K$
different classes.
Furthermore, we are assuming that $\mathcal{D}$ is balanced and that the
number of samples per class is equal to $N$ and relatively small. The notion
of ”small” is intrinsically related to the difficulty of the task. Indeed, it
may be possible that good recognition performance can be reached with few
samples per class. Yet, we quantify this lack of data by assuming to have at
most tens/hundreds of samples per class. Concerning the dimensions of current
computer vision benchmarks, this can be generally described as a reduction of
one/two orders of magnitudes.
In general, $\mathbf{x}$ might be a vector of any size. In our work, we are
tackling a classification problem where each input sample is an image, hence,
$\mathbf{x}\in\mathbb{R}^{H\times W\times D}$. The objective is to learn the
input-output mapping $\mathbf{y}=f_{\theta}(\mathbf{x})$ with $f(\cdot)$
representing a function approximator and $\mathbf{\theta}$ its parameters.

###  3.2 Neural ensembles

The operational pipeline of neural ensembles consists of training multiple
networks individually and then averaging their predictions obtaining, in the
end, better results than single estimates. More precisely, since we are
implementing a weighted-average ensemble, we define a set
$\mathcal{M}=\\{g_{\theta_{m}}(\cdot):m=1,\dots M\\}$ of $M$ functions. In our
particular implementation, the ensemble is homogeneous, therefore all
functions $g$ are equal. At testing time, given an input sample $\mathbf{x}$,
the ensemble prediction is going to be the mean of the predictions coming from
each member in $\mathcal{M}$ scaled by a function $\phi$ depending on the type
of the loss used. More precisely,
$\mathbf{y}=f_{\theta}(\mathbf{x})=\frac{1}{M}\sum_{m=1}^{M}\phi\left(g_{\theta_{m}}(\mathbf{x})\right)$.
The weights $\theta_{m}$ are randomly initialized and ensemble members update
their parameters receiving the same input batch. Throughout the optimization
process, the members are independent, therefore each network will minimize its
own loss $\mathcal{L}(\mathbf{y},\phi(g_{\theta_{m}}(\mathbf{x})))$.

###  3.3 Networks’ design space

In our analysis, we will evaluate neural ensembles and compare them with deep
or wide single networks. Given an ensemble $\mathcal{M}$ with $M$ base members
$g_{\theta_{m}}$, each network has depth $d$ and width $w$. The set of
networks account for a total computational budget $B$. To perform a fair
ranking, the competing single networks have the same $B$. More in detail, deep
or wide single competitors have, respectively, the same $w$ but increased
depth $d^{\prime}>d$ or the same $d$ but increased width $w^{\prime}>w$. We
express the budget as the number of floating-point operations (FLOPs) to make
a prediction. This is a standard metric that measures algorithmic/model
complexity.111In our implementation, we used Tensorflow APIs to compute the
FLOPs of networks. Note that two networks of the same family and layout not
only have a similar number of FLOPs but also of trainable parameters.

##  4 Experimental Evaluation

###  4.1 Evaluation metrics

In our work, we will use two metrics to evaluate the performance of the
trained networks.
The first one is the classic accuracy over the testing set. Indeed, we will
use benchmarks that are balanced in terms of samples per class, making
accuracy an appropriate metric to measure generalization. To ensure
consistency of the results, we perform five independent runs and then compute
mean and standard deviations.
The second metric measures the sensitivity of the network to input
perturbations and is computed as the Frobenius norm of the input-output
Jacobian ($||J(\mathbf{x})||_{F}$) given
$J(\mathbf{x})=\partial{f_{\theta}}(\mathbf{x})/\partial\mathbf{x}^{T}$. We
will use this metric to support our interpretation of the effectiveness of
neural ensembles with limited data. [35] theoretically showed the connection
between input-change robustness and generalization, and [27] provided
empirical evidence of this result. On a qualitative level, this metric tracks
the first-order term of the Taylor expansion of the function around a point.
Therefore, low sensitivity values (i.e. low norm of input-output Jacobian)
suggest smoother functions less influenced by input perturbations.

###  4.2 Datasets

We perform our study on five popular benchmarks for image classification:
CIFAR-10, CIFAR-100 [21], SVHN [24], Stanford Dogs [20] and CUB-200 [38]. More
details regarding the datasets are available in the Appendix.

In this work, we are interested in benchmarking the capabilities of neural
ensembles with limited data. To this end, we sub-sample the training sets of
the chosen datasets to match our specifications. More in detail, for all the
datasets, except CUB-200, the number of samples per class is varied in the set
$\\{10,50,100,250\\}$. For Stanford Dogs, we stop at 100 samples having
reached the original size of the training dataset. On the other hand, since
CUB-200 has at most 30 samples per class, we train with $\\{5,10,20,30\\}$
samples per class. A very similar procedure was already proposed in [3], and
[2].
Note that the test datasets are not changed and remain fixed throughout all
the evaluations.

###  4.3 Experimented networks

Table 1: Results of neural ensembles and state-of-the-art approaches evaluated
on sub-sampled versions of popular datasets. We report in blue our ensembles
and in black the other methods. We highlight the highest value in black bold
font. Values correspond to $mean\pm std$. (a) Comparison with the best of
Table 4 from [5] ($\dagger$).

Dataset | Aug. | Model |  N = 100 | Dataset | Aug. | Model |  N = 500
---|---|---|---|---|---|---|---
CIFAR-10 | - |  VGG-11 + grad-$l_{2}$ \+ SN proj† | 46.88 |  CIFAR-10 | - |  VGG-11 + PGD-$l_{2}$ \+ SN proj† | 64.50
10 VGG-5 | 52.16 $\pm$ 0.38 | 10 VGG-5 | 68.59 $\pm$ 0.21
- |  ResNet-18 + $||\nabla f||^{2}$† | 44.97 | - |  ResNet-18 + $||f||^{2}_{\delta}$ SN proj† | 59.03
20 ResNet-8 | 52.66 $\pm$ 0.45 | 20 ResNet-8 | 69.49 $\pm$ 0.36
+ |  VGG-11 + grad-$l_{2}$ \+ SN proj† | 55.32 | + |  VGG-11 + grad-$l_{2}$† | 75.38
10 VGG-5 | 57.73 $\pm$ 0.52 | 10 VGG-5 | 76.26 $\pm$ 0.09
+ |  ResNet-18 + grad-$l_{2}$† | 49.30 | + |  ResNet-18 grad-$l_{2}$ \+ SN proj† | 77.73
20 ResNet-8 | 63.64 $\pm$0.61 | 20 ResNet-8 | 81.78 $\pm$ 0.19

(b) Comparison with the best of Table 2 from [1] ($\sharp$)

Dataset | Aug. | Model |  N = 1 |  N = 2 |  N = 4 |  N = 8 |  N = 16 |  N = 32 |  N = 64 |  N = 128
---|---|---|---|---|---|---|---|---|---|---
CIFAR-10 | - |  ResNet-34♯ |  14.59 $\pm$ 1.99 |  17.5 $\pm$ 2.47 |  19.52 $\pm$ 1.39 |  23.32 $\pm$ 1.61 |  28.3 $\pm$ 1.38 |  33.15 $\pm$ 1.2 |  41.66 $\pm$ 1.09 |  49.14 $\pm$ 1.31
CNTK♯ |  15.33 $\pm$ 2.43 |  18.79 $\pm$ 2.13 |  21.34 $\pm$ 1.91 |  25.48 $\pm$ 1.91 |  30.48 $\pm$ 1.17 |  36.57 $\pm$ 0.88 |  42.63 $\pm$ 0.68 |  48.86 $\pm$ 0.68
20 ResNet-8 | 16.73 $\pm$ 1.02 | 20.66 $\pm$ 0.53 | 24.47 $\pm$ 0.9 | 29.54 $\pm$ 0.91 | 34.55 $\pm$ 0.92 | 40.48 $\pm$ 1.28 | 47.5 $\pm$ 0.24 | 55.04 $\pm$ 0.46

(c) Comparison with results from Figure 6 and Table 1 of [2] ($\ast$), Figure
10 of [42] ($\diamond$) and Figure 2 of [3] ($\ddagger$). The values of the
plots were provided by the authors or extracted with an on-line tool.

Dataset | Aug. | Model |  N = 10 |  N = 50 |  N = 100 |  N = 250 |
---|---|---|---|---|---|---|---
CIFAR-10 | \+ & Cutout |  1 WRN-28-10∗ |  27.41 $\pm$ 0.57 |  46.39 $\pm$ 0.39 |  57.97 $\pm$ 0.31 |  73.86 $\pm$ 0.68 |
GLICO |  1 WRN-28-10∗ |  32.12 $\pm$ 0.38 |  54.32 $\pm$ 1.35 |  67.19 $\pm$ 0.36 | 78.23 |
+++ | 20 WRN-10-5 | 34.01 $\pm$ 0.27 | 59.24 $\pm$ 0.23 | 67.96 $\pm$ 0.48 |  78.15 $\pm$ 0.26 |
CIFAR-100 | GLICO |  1 WRN-28-10∗ | 28.55 $\pm$ 0.4 | 52.95 $\pm$ 0.2 | 64.27 $\pm$0.04 | – |
+++ | 20 WRN-10-5 |  24.47 $\pm$ 0.48 |  51.09 $\pm$ 0.45 |  61.82 $\pm$ 0.31 | – |
+ |  Data-Eff. CNN⋄ | 14 | 36 | 52.67 | – |
\+ & Cutout |  1 ResNet-110-32 (cosine)‡ | 18.47 | 42.86 | 55.89 | 68.06 |
1 ResNet-110-32 (cosine + xe)‡ | 16.99 | 45.44 | 57.14 | 68.66 |
+++ | 20 ResNet-8-32 (xe) | 23.74 $\pm$ 0.69 | 51.14 $\pm$ 0.35 | 61.41 $\pm$ 0.36 | 70.68 $\pm$ 0.12 |
|  |  |  N = 5 |  N = 10 |  N = 20 |  N = 30 |
CUB-200 | \+ & Cutout |  1 ResNet-50 (cosine)‡ | 10.65 | 23.20 | 53.14 | 67.6 |
1 ResNet-50 (cosine + xe)‡ | 11.25 | 30.82 | 55.18 | 67.95 |
+ | 5 ResNet-10 (cosine) |  15.77 $\pm$ 0.05 |  31.9 $\pm$ 0.08 |  55.97 $\pm$ 0.19 |  66.69 $\pm$ 0.05 |
5 ResNet-10 (cosine + xe) | 16.3 $\pm$ 0.09 | 32.94 $\pm$ 0.01 | 56.03 $\pm$ 0.51 |  66.87 $\pm$ 0.01 |

We considered four different families of convolutional architectures: VGG
[34], ResNet [15], Wide ResNet (WRN) [44] and DenseNet [17]. We tested the
first three on the CIFAR datasets, while DensNet on SVHN and Stanford Dogs. We
also tested the Imagenet version of ResNet on CUB-200.

For ResNet and VGG on CIFAR, the maximum computational budget was set to be
roughly close to the FLOPs of a single ResNet-110 and VGG-9. On CUB-200, due
to the larger image size, the budget was equivalent to the one of a ResNet-50
with 64 base filters. For WRN we considered a single WRN-28-10. For DenseNet
on SVHN, we have chosen the complexity of DenseNet-BC-52 while, for Stanford
Dogs, DenseNet-BC-121.
On the CIFAR datasets with ResNets, we also set two smaller computational
budgets matching the FLOPs of a single ResNet-26 and ResNet-50 with 16 base
filters.
As base members for the ensembles, we chose to have ResNet-8 with 16 base
filters, WRN-10 with width factor equal to 5, WRN-10 with width factor equal
to 5, ResNet-10 with 64 base filters, VGG-5 with 32 base filters, DenseNet-
BC-16 with growth rate ($k$) equal to 12 and DenseNet-BC-62 with $k$ = 56.
For more details regarding the computational complexity and structure of the
networks refer to the Appendix.

###  4.4 Training set-up

To train our networks we used standard data pre-processing and optimization
techniques. We set three different augmentation levels based on conventional
transformations to ensure that results were not biased in this respect. We
added to the standard random crop and flipping policy widely used (+), random
color distortion (++), and finally random erasing (+++).

More details regarding training schedules and augmentation strategies are
available in the Appendix.

##  5 Results

###  5.1 Comparing with the state of the art

Table 2: Comparison between deep/wide CNNs and ensembles of less complex
networks. $M$ indicates the number of networks in the ensemble and $N$ the
number of training samples per class. These results are obtained using medium
data augmentation (++). Means and standard deviations are obtained from five
independent runs.

Dataset |  Budget (MFLOPs) | Model | M |  N = 10 |  N = 50 |  N = 100 |  N = 250
---|---|---|---|---|---|---|---
CIFAR-10 |  $\sim$ 9.5 | ResNet-110-16 | 1 |  28.65 $\pm$ 1.06 |  43.68 $\pm$ 1.29 |  51.16 $\pm$ 0.96 |  63.0 $\pm$ 1.7
ResNet-8-72 | 1 |  32.12 $\pm$ 0.28 |  54.58 $\pm$ 0.36 |  62.82 $\pm$ 0.62 |  74.76 $\pm$ 0.1
ResNet-8-16 | 20 | 34.9 $\pm$ 1.13 | 58.11 $\pm$ 0.42 | 66.95 $\pm$ 0.58 | 76.89 $\pm$ 0.11
$\sim$ 4.5 | VGG-9-32 | 1 |  28.5 $\pm$ 1.6 |  42.25 $\pm$ 1.29 |  47.51 $\pm$ 1.68 |  58.07 $\pm$ 0.44
VGG-5-76 | 1 |  32.99 $\pm$ 1.57 |  48.06 $\pm$ 0.12 |  55.41 $\pm$ 0.73 |  65.45 $\pm$ 0.24
VGG-5-32 | 5 | 34.15 $\pm$ 1.54 | 51.5 $\pm$ 0.41 | 60.28 $\pm$ 0.63 | 70.73 $\pm$ 0.33
CIFAR-100 |  $\sim$ 9.5 | ResNet-110-16 | 1 |  13.13 $\pm$ 0.54 |  29.53 $\pm$ 0.26 |  38.23 $\pm$ 1.61 |  53.17 $\pm$ 0.73
ResNet-8-72 | 1 |  19.65 $\pm$ 1.09 |  45.16 $\pm$ 0.02 |  54.85 $\pm$ 0.19 | 65.47 $\pm$ 0.44
ResNet-8-16 | 20 | 22.62 $\pm$ 0.82 | 48.76 $\pm$ 0.35 | 57.75 $\pm$ 0.11 | 65.41 $\pm$ 0.15
$\sim$ 4.5 | VGG-9-32 | 1 |  10.36 $\pm$ 0.53 |  22.95 $\pm$ 0.79 |  30.97 $\pm$ 0.64 |  42.92 $\pm$ 0.51
VGG-5-76 | 1 |  14.35 $\pm$ 0.3 |  28.25 $\pm$ 0.09 |  36.84 $\pm$ 0.27 |  45.49 $\pm$ 0.31
VGG-5-32 | 5 | 17.47 $\pm$ 0.38 | 36.39 $\pm$ 0.13 | 46.05 $\pm$ 0.22 | 55.21 $\pm$ 0.31
SVHN |  $\sim$ 0.5 | DenseNet-BC-52, k=12 | 1 | 16.72 $\pm$ 1.75 |  78.42 $\pm$ 1.19 |  86.52 $\pm$ 0.24 |  89.6 $\pm$ 0.7
DenseNet-BC-16, k=30 | 1 |  16.44 $\pm$ 3.8 |  76.41 $\pm$ 1.65 |  85.41 $\pm$ 0.52 |  89.28 $\pm$ 0.06
DenseNet-BC-16, k=12 | 6 |  14.01 $\pm$ 2.5 | 82.02 $\pm$ 1.67 | 87.73 $\pm$ 0.44 | 91.61 $\pm$ 0.32
St. Dogs |  $\sim$ 14.3 | DenseNet-BC-121, k=32 | 1 |  6.93 $\pm$ 0.86 |  28.32 $\pm$ 1.33 |  47.7 $\pm$ 1.17 | –
DenseNet-BC-62, k=56 | 1 |  7.33 $\pm$ 0.35 |  29.25 $\pm$ 0.76 |  47.82 $\pm$ 0.83 | –
DenseNet-BC-62, k=32 | 3 | 8.42 $\pm$ 0.02 | 35.12 $\pm$ 0.68 | 53.39 $\pm$ 0.45 | –

First, we show the effectiveness of deep ensembles comparing them with state-
of-the-art techniques that were proposed for learning from a small sample.
More precisely, diverse kernel-based regularizations proposed in [5], the CNTK
tested in [1], classifiers augmented with the generative model GLICO in [2],
the cosine loss of [3] and Data-Efficient CNNs [42]. The first two works have
chosen to test their models on sub-sampled versions of CIFAR-10 while the
second two used CIFAR and CUB-200 datasets. The results are shown in Table
1(c).

[5] evaluate their models with and without standard augmentation composed of
random cropping and mirroring. One hundred and five hundred samples per class
comprised their training sets. For sake of completeness, we also evaluate our
ensembles in the latter case, despite the training set already contains a
consistent number of samples. They trained a VGG-11 and ResNet-18 with various
regularization techniques. To keep a fair comparison, we train an ensemble of
10 VGG-5 and 20 ResNet-8 and compare them with the corresponding regularized
architectures tested in [5]. Note, however, that with the default widths, both
VGG-11 and ResNet-18 have more training parameters than our ensembles.
Moreover, in the original work, authors tuned the hyper-parameters of their
methods on a held-out validation set.222This is a simplifying assumption that
inevitably increases training set dimension. We note from Table 1(c) that our
ensembles always outperform the respective models from a minimum of $\sim 2\%$
to a maximum of $\sim 14\%$ average accuracy.

[1] used a very restrictive training protocol varying the number of samples
per class from 1 to 128 and not using any kind of data augmentation. Here, we
compare our ensemble of 20 ResNet-8 with their baseline ResNet-34 and proposed
CNTK. Our ensemble has fewer total trainable parameters than a single
ResNet-34. Also, in this case, gains in terms of average accuracy are
noticeable and in the range of approximately $1\%-7\%$.

Figure 1: Results considering, respectively, the variation of depth/width in
single networks, and the number of networks in the ensemble for ResNets on
CIFAR datasets trained with standard augmentation. Left: allocation of
resources in network’s depth. Middle: allocation of resources in network’s
width. Right: allocation of resources in multiple networks.

[2] tested WRN-28-10 on CIFAR datasets and ResNet-50 on CUB-200 augmented by
GLICO. We do not compare our approach with GLICO on the latter dataset since,
in that case, they used a pre-trained ResNet-50 classifier. Note also that
GLICO uses a pre-trained-VGG based perceptual loss to synthesize images.
Therefore, differently from our method, GLICO requires, in any case, a pre-
trained model. We see from Table 1(c) that, on CIFAR-10, ensembles are
comparable to or better than the single network augmented with GLICO (gains of
$\sim 2\%-5\%$ average accuracy). This fact is surprising considering the
previously cited advantages that GLICO has over our standard ensembles. On the
other hand, on the larger CIFAR-100, the single WRN plus GLICO outmatches 20
WRNs by roughly $2\%-5\%$ average accuracy.

[3] trained a ResNet-110 with 32 base filters on CIFAR-100, ResNet-50 on
CUB-200 with multiple losses, and one-hot or semantic labels using standard
augmentation plus Cutout. In our case, we did not train ensembles with any
kind of a priori knowledge, therefore, we considered their results obtained
with the standard one-hot labels. A combination of the cosine and cross-
entropy loss scored the best values in their experiments. We compared an
ensemble of 20 ResNet-8-32 trained with the cross-entropy loss to their single
ResNet-110-32 on CIFAR-100 and show that we get improvements of roughly
$2\%-6\%$ average accuracy. On CUB-200 we train an ensemble of 5 ResNet-10 to
match their ResNet-50. As in [3] we obtained poor results with the cross-
entropy on this fine-grained dataset. Therefore, we perform the comparison
using the cosine and smoothed cosine losses. In this case, as well, we get
comparable results using the full dataset and gains of up to $\sim 5\%$
average accuracy on smaller versions.

Finally, we report the results of Data-Efficient CNNs obtained on CIFAR-100 in
[42]. Our ensemble of 20 ResNet-8-32 has comparable complexity to this
competitor in terms of parameters but better data efficiency. Indeed, gains in
terms of average accuracy reach $15\%$.

In fair comparisons, neural ensembles outperform the competing techniques. In
the more complex challenge with a generative model, ensembles can still be a
valid choice. Indeed, on the one hand, they require less computational needs.
On the other hand, they could be still strengthened by the generative model.
These results clearly show that standard ensembles with equivalent model
complexity is a simple yet effective way to reduce model variance with few
training samples.

###  5.2 Comparing deep or wide single nets with ensembles

In this section, we report the results of the ensembles and analyze their
relation with their deeper or wider single competitors. From Table 2, we note
that ensembles are the best option in terms of testing accuracy for almost all
cases.

We notice that deeper networks struggle with small datasets. On CIFAR
benchmarks, ResNet-8 and VGG-5 ensembles obtain very large gains over deeper
architectures ranging from a minimum of $\sim 3\%$ to a maximum of $\sim 15\%$
average accuracy. To be sure that these results are not just due to a poor
regularization of the deepest model, we also made tests with more aggressive
data augmentation which are available in the Appendix. However, results
remained consistent with the ones with medium augmentation. Considering
DenseNets, we appreciate a similar trend, with more moderate but still
significant gains, from $\sim 1.5\%$ to $\sim 6\%$ average accuracy.

On the other hand, wider networks seem to handle the lack of training data
better than deeper models. Still, model averaging and independent training has
a clear advantage over wider single networks. ResNet, VGG and DenseNet
ensembles generalize better than wider nets in terms of average accuracy from
a minimum of $\sim 1\%$ to a maximum of $\sim 5\%$. Additional experiments and
discussions regarding depth, width, and ensembles will follow in the next
sections.

###  5.3 Comparing design spaces

Figure 2: tSNE visualization of networks sensitivity to testing samples fixing
the feature space of an oracle network. The first two rows and the third one
respectively correspond to CIFAR-10/SVHN with 100/50 samples per class.

We believe it is worth discussing in more detail the experiments with ResNets
on the CIFAR datasets to better understand what is the best design choice
among depth or width in single networks or the number of members in ensembles.
To provide a complete picture, we set, as mentioned in Section 4.3, additional
computational budgets in terms of FLOPs that are lower than the FLOPs of a
single ResNet-110. The results of this analysis are visible in Figure 1 but
also present in table format in the Appendix. In the plots, the four budgets
are represented by the different colors assuming green as the lowest budget
(single ResNet-8-16) and black as the highest budget (e.g. single
ResNet-110-16). Therefore, an increment of testing accuracy from green to
black indicates a favorable design direction.

A first result is that increasing the computational budget in the depth
direction is generally not beneficial (left plots in Figure 1). Indeed, we
note that the deepest model is always the worst, except for the case of
CIFAR-100 with more samples. This is reasonable since it was empirically shown
that deep networks coupled with larger datasets can learn very complex
functions that translate in good representational capacity [23].
Investing on network’s width is slightly better (middle plots in Figure 1). On
CIFAR-100, wider networks outperform the baseline model. This result is
expected. Indeed, on a more complex problem, more capacity gives an advantage
to wide networks. On the other hand, on CIFAR-10, wide models perform
comparably well to ResNet-8-16.

Finally, we note that ensembles have a clear advantage over the single
ResNet-8-16 on both CIFAR-10 and CIFAR-100, with larger gains obtained on the
latter (right plots in Figure 1). This makes ensembles the most preferable
approach having the possibility of increasing the computational budget. We
also note that the largest gap is generally between the single network and 5
ResNet-8 proving that the performance gain is already considerable with
relatively small ensembles.

###  5.4 Interpreting data efficiency of neural ensembles

In this final section, we provide our interpretation of why ensembles perform
better than competitors with small datasets.
Given our design choices, deep or wide single nets and ensembles of networks
potentially have the same complexity expressed in terms of free parameters.
However, it was shown that this general conformity does not automatically
ensure equivalently performing models [31]. Also in our case, different
designs of the same architecture led to different performances. Therefore, we
are led to believe that the functions implemented by ensembles are superior in
terms of generalization and should differ from the ones learned by deeper or
wider single nets. A straightforward way to characterize the learned input-
output mapping is to measure its complexity. We anticipated in Section 4.1
that the complexity of the learned function on the data manifold can be
qualitatively measured by the Frobenius norm of the input-output Jacobian. Low
values of $||J(\mathbf{x})||_{F}$ indicate smooth output variations given
perturbations of $\mathbf{x}$, i.e. low local sensitivity of the learnt
function. A more global estimate can be obtained computing the mean
sensitivity on the test sample [27]. To better understand this phenomenon and
visually show it, we perform the following analysis.

First, we train oracle networks on full datasets to fix a perceptually robust
representation space. In this manner, perceptually similar inputs are
generally mapped close to each other in the oracle feature space. Secondly, we
feed the test samples to the oracle and extract features right before the
classification layer. Then, we perform a 2D tSNE to decrease their dimensions.
Now, each test sample corresponds to a 2D vector $(p_{1},p_{2})$. In the left
plots of Figure 2, we assign a different color to each of them depending on
the class to which they belong. In other words, each point is a 3D tuple
$(p_{1},p_{2},c)$ with $c$ being the class color. As a third step, we consider
networks trained on smaller datasets and compute $||J(\mathbf{x})||_{F}$ for
each testing sample obtaining the local sensitivity at $\mathbf{x}$. To
visualize it on the available version of the input space, i.e. the test set,
we assign $||J(\mathbf{x})||_{F}$ to previously fixed positions
$(p_{1},p_{2})$ obtaining a scatter plot of a 3D tuple
$(p_{1},p_{2},||J(\mathbf{x})||_{F})$ being $||J(\mathbf{x})||_{F}$ the color
(right plots of Figure 2). Note also that running tSNE for each network,
obtaining independent embeddings (i.e. different $(p_{1},p_{2})$), would not
have been possible for ensembles since their members do not share a common
representation space.

In Figure 2 we show the just introduced visualizations along with the average
sensitivity over testing sets of ResNets/VGGs trained with 100 samples per
class on CIFAR-10, and DenseNets trained with 50 per class on SVHN. We also
report additional results concerning network sensitivity in the Appendix.
Ensembles have the lowest sensitivity even in regions where there is the
highest perceptual similarity among samples of different classes. We believe
that this quality is a direct consequence of model averaging and independent
training. Qualitatively speaking, as we change the input, the predictions of
the overall ensemble model are smoothed thanks to the intrinsic property of an
ensemble to have members with decorrelated errors. Indeed, even if a subset of
networks undergoes large input-output oscillations, there are still the
remaining ones that can balance out their influence.

We notice that deeper ResNet-110 and VGG-9 have higher sensitivity than their
wider competitors. While single DenseNets are closer in terms of average
sensitivity ($0.95$ and $0.99$). Previous works studied the expressive power
of ReLU networks and argued that the complexity of the computed function grows
exponentially with the depth and polynomially with the width [29], [23], [30].
Our empirical results confirm the bias of deeper nets to learn more complex
functions. While this might be an advantage on larger and more complex
datasets, it does not seem so with scarce data. Reasonably, at test time
networks experience a larger input variance with respect to scenarios with
large datasets. These input perturbations grow exponentially in the depth of
the network [30] causing poorer performance.

On the other hand, our shallower and wider tested networks seem to learn less
sensible functions. [26] analyzed the influence of depth and width on CNNs and
predicted that network width smoothes the optimization landscape of the loss.
We align with this claim since also after increasing the width on very small
training sets (e.g. CIFAR-10 with 50 samples per class), networks perform
comparably well or better than thinner ones (Figure 1).

To sum up, taking previous arguments and results into account, it is
reasonable to believe that deeper networks are more prone to overfit on small
datasets due to their inherent property of learning more complex functions
that scale exponentially with depth. Wide networks seem to be more suited to
learn less complex functions and to improve the optimization landscape.
Finally, ensembles add the advantage of model averaging and independent
training to the low sensitivity of a single shallow network. Note that
ensemble members need to have enough representational capacity to fit the
available training sample. In the case of limited data, this can also be
reached with relatively small computational budgets as we have seen in our
experiments. Yet, a smaller ensemble of larger-size nets or a single wider
model might outperform a larger ensemble of smaller-size nets as the task
complexity increases [40].

##  6 Conclusions

In this paper, we have investigated the capability of deep ensembles on image
classification problems with limited data. Through comparing ensembles given a
fixed computational budget, our results show that ensembles are generally
better than current state-of-the-art approaches and their deeper or wider
single competitors.

Furthermore, we analyzed the function space of tested models using the popular
sensitivity metric based on the input-output Jacobian. Functions implemented
by ensembles reported lower sensitivity to input perturbations. Therefore, we
conjecture that neural ensembles are more suitable for small datasets because
they learn simpler functions.

The encouraging results presented in this work suggest the potential of using
ensembles for small datasets and promise further developments of this line of
research in the field of small data.

## References

  * [1] Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu.  Harnessing the power of infinitely wide deep nets on small-data tasks.  In International Conference on Learning Representations, 2020.
  * [2] Idan Azuri and Daphna Weinshall.  Generative latent implicit conditional optimization when learning from small sample.  2020\.
  * [3] Bjorn Barz and Joachim Denzler.  Deep learning on small datasets without pre-training using cosine loss.  In The IEEE Winter Conference on Applications of Computer Vision, 2020.
  * [4] Yoshua Bengio.  Deep learning of representations for unsupervised and transfer learning.  In Proceedings of ICML workshop on unsupervised and transfer learning, 2012.
  * [5] Alberto Bietti, Grégoire Mialon, Dexiong Chen, and Julien Mairal.  A kernel perspective for regularizing deep neural networks.  In International Conference on Machine Learning, 2019.
  * [6] Jörg Bornschein, Francesco Visin, and Simon Osindero.  Small data, big decisions: model selection in the small-data regime.  International Conference on Machine Learning, 2020.
  * [7] Leo Breiman.  Random forests.  Machine learning, 2001.
  * [8] Lorenzo Brigato and Luca Iocchi.  A close look at deep learning with small data.  2020\.
  * [9] Nadezhda Chirkova, Ekaterina Lobacheva, and Dmitry Vetrov.  Deep ensembles on a fixed memory budget: One wide network or several thinner ones?  arXiv preprint arXiv:2005.07292, 2020.
  * [10] Anuvabh Dutt, Denis Pellerin, and Georges Quénot.  Coupled ensembles of neural networks.  In International Conference on Content-Based Multimedia Indexing, 2018.
  * [11] Nikita Dvornik, Julien Mairal, and Cordelia Schmid.  Diversity with cooperation: Ensemble methods for few-shot classification.  In International Conference on Computer Vision, 2019.
  * [12] Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim.  Do we need hundreds of classifiers to solve real world classification problems?  The Journal of Machine Learning Research, 2014.
  * [13] Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d’Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart.  Scaling description of generalization with number of parameters in deep learning.  Journal of Statistical Mechanics: Theory and Experiment, 2020.
  * [14] Lars Kai Hansen and Peter Salamon.  Neural network ensembles.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 1990\.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  * [16] Guosheng Hu, Xiaojiang Peng, Yongxin Yang, Timothy M Hospedales, and Jakob Verbeek.  Frankenstein: Learning deep face representations using small data.  IEEE Transactions on Image Processing, 2017.
  * [17] G Huang, Z Liu, L Maaten, and KQ Weinberger.  Densely connected convolutional networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [18] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  Advances in Neural Information Processing Systems, 2020.
  * [19] Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio.  Generalization in deep learning.  CoRR, 2017.
  * [20] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.  Novel dataset for fine-grained image categorization: Stanford dogs.  In CVPR Workshop on Fine-Grained Visual Categorization, 2011.
  * [21] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [22] Lanlan Liu, Michael Muelly, Jia Deng, Tomas Pfister, and Li-Jia Li.  Generative modeling for small-data object detection.  In Proceedings of the IEEE International Conference on Computer Vision, 2019.
  * [23] Guido F. Montúfar, Razvan Pascanu, KyungHyun Cho, and Yoshua Bengio.  On the number of linear regions of deep neural networks.  In Advances in Neural Information Processing Systems, 2014.
  * [24] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [25] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro.  Towards understanding the role of over-parametrization in generalization of neural networks.  CoRR, 2018.
  * [26] Quynh Nguyen and Matthias Hein.  Optimization landscape and expressivity of deep cnns.  In International Conference on Machine Learning, 2018.
  * [27] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein.  Sensitivity and generalization in neural networks: an empirical study.  In International Conference on Learning Representations, 2018.
  * [28] Matthew Olson, Abraham Wyner, and Richard Berk.  Modern neural networks generalize on small data sets.  In Advances in Neural Information Processing Systems, 2018.
  * [29] Razvan Pascanu, Guido Montufar, and Yoshua Bengio.  On the number of response regions of deep feed forward networks with piece-wise linear activations.  arXiv preprint arXiv:1312.6098, 2013.
  * [30] Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein.  On the expressive power of deep neural networks.  In International Conference on Machine Learning, 2017.
  * [31] Carl Edward Rasmussen and Zoubin Ghahramani.  Occam’s razor.  Advances in Neural Information Processing Systems, 2001.
  * [32] Diego Rueda-Plata, Raúl Ramos-Pollán, and Fabio A. González.  Supervised greedy layer-wise training for deep convolutional networks with small datasets.  In Computational Collective Intelligence, 2015.
  * [33] Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of Big Data, 2019.
  * [34] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2014.
  * [35] Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues.  Robust large margin deep neural networks.  IEEE Transactions on Signal Processing, 2017.
  * [36] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu.  A survey on deep transfer learning.  In International Conference on Artificial Neural Networks, 2018\.
  * [37] Joaquin Vanschoren.  Meta-learning: a survey.  CoRR, 2018.
  * [38] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [39] Abdul Wasay, Brian Hentschel, Yuze Liao, Sanyuan Chen, and Stratos Idreos.  Mothernets: Rapid deep ensemble learning.  In Machine Learning and Systems, 2020.
  * [40] Abdul Wasay and Stratos Idreos.  More or less: When and how to build convolutional neural network ensembles.  In International Conference on Learning Representations, 2021.
  * [41] Yeming Wen, Dustin Tran, and Jimmy Ba.  Batchensemble: an alternative approach to efficient ensemble and lifelong learning.  In International Conference on Learning Representations, 2020.
  * [42] Wenju Xu, Guanghui Wang, Alan Sullivan, and Ziming Zhang.  Towards learning affine-invariant representations via data-efficient cnns.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 904–913, 2020.
  * [43] Y. Bengio Y. LeCun and G. E. Hinton.  Deep learning.  Nature, 2015.
  * [44] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In Proceedings of the British Machine Vision Conference, 2016.

  *[DL]: deep learning
  *[CNNs]: convolutional neural networks
  *[CNTK]: convolutional neural tangent kernel