# Few-mode-fiber technology fine-tunes losses of quantum communication systems

A. Alarcón  Institutionen för Systemteknik, Linköpings Universitet, 581 83
Linköping, Sweden  J. Argillander  Institutionen för Systemteknik, Linköpings
Universitet, 581 83 Linköping, Sweden  G. Lima  Departamento de Física,
Universidad de Concepción, 160-C Concepción, Chile  Millennium Institute for
Research in Optics, Universidad de Concepción, 160-C Concepción, Chile  G. B.
Xavier<EMAIL_ADDRESS>Institutionen för Systemteknik, Linköpings
Universitet, 581 83 Linköping, Sweden

###### Abstract

A natural choice for quantum communication is to use the relative phase
between two paths of a single-photon for information encoding. This method was
nevertheless quickly identified as impractical over long distances and thus a
modification based on single-photon time-bins has then become widely adopted.
It however, introduces a fundamental loss, which increases with the dimension
and that limits its application over long distances. Here, we are able to
solve this long-standing hurdle by employing a few-mode fiber space-division
multiplexing platform working with orbital angular momentum modes. In our
scheme, we maintain the practicability provided by the time-bin scheme, while
the quantum states are transmitted through a few-mode fiber in a configuration
that does not introduce post-selection losses. We experimentally demonstrate
our proposal by successfully transmitting phase-encoded single-photon states
for quantum cryptography over 500 m of few-mode fiber, showing the feasibility
of our scheme.

##  I Introduction

Quantum Communication (QC) is one of the main pillars of the applied field of
Quantum Technologies Acin_2018 , which deals with information processing tasks
that rely on individual and entangled quantum systems. QC includes many
applications such as quantum cryptography Xu_2020 ; Pirandola_2020 , quantum
bit commitment Lunghi_2013  and quantum secret sharing Pinnell_2020 . Although
traditionally many experiments have been performed while resorting to
polarization-encoded quantum states Xavier_2009 ; Grunenfelder_2018 , time-
bin-based phase-coding quantum cryptography has always been regarded as the
optimal choice for optical fiber communication links due to strong robustness
to environmental disturbances Bennett_1992 ; Gisin_2002 ; Townsend_1993 ;
Gobby_2004 ; Wang_2008 ; Yuan_2008 ; Wang_2012 ; Rubenok_2013 ; Yang_2013 ;
Islam_2017a ; Boaron_2018 . The phase-coding quantum cryptography protocol was
initially discussed considering the relative phase between two different
single-photon spatial modes defining a long Mach-Zehnder interferometer
Bennett_1992 ; Gisin_2002 . The main limitation of this original scheme was
that the interferometer length needed to be as long as the physical separation
between the communicating parties (Alice and Bob), thus subjected to strong
environmental phase disturbances. More recently, the technology to stabilise
long Mach-Zehnder interferometers supporting the propagation of single photons
with short coherence length has become available Xavier_2012 ; Cuevas_2013 ;
Carvacho_2015 , but it nevertheless adds to the experimental complexity.

The time-bin based configuration was proposed to solve this issue Bennett_1992
; Gisin_2002 . It trades the long single Mach-Zehnder interferometer of the
previous scheme with two unbalanced and short Mach-Zehnder interferometers
(UMZIs), one located at Alice and the other within Bob’s station. Both
interferometers are connected with a single optical fiber comprising the
communication channel linking Alice and Bob. The key point is that the two
long parallel paths of the original phase-encoding scheme are traded for two
temporally-separate time-bins that travel through the same path. This
guarantees intrinsic phase stability, since any phase disturbance needs to
occur on a time scale shorter than the separation between the two time-bins.
Only the short UMZIs need to be stabilized, a task considerably easier than
actively compensating phase drifts on a long interferometer. The trade-off is
an intrinsic loss of 50% at the detection stage due a post-selection of non-
interfering time-bins. Even more dramatic, this intrinsic loss grows with the
dimension $d$ of the encoded system as $(d-1)/d$, having a larger impact on
high-dimensional time-bin QC systems Islam_2017b . In spite of this, time-bin
has been extremely popular since it is much more practical to stabilise two
short interferometers instead of a very long one Gisin_2002 .

In this work, we adopt modern space-division multiplexing (SDM) fiber optics
technology Richardson_2013 ; Xavier_2020  to solve the issue of the
irreversible loss at the detection stage of the time-bin scheme, thus
providing a pathway towards efficient fiber-based phase-coding quantum
communication. Specifically, in our scheme we use a commercial SDM fiber that
supports a few linearly polarised (LP) spatial modes, the so-called few-mode
fiber (FMF). Similar to the time-bin configuration, our scheme is based on a
single (few-mode) fiber interconnecting two local interferometers for the
state generation and detection. These states are encoded as a superposition of
the LP optical modes supported by the FMF. The key component that allows this
implementation is the mode-selective photonic lantern, which takes $N$ input
single-mode fibers, and maps each one-to-one to $N$ linearly polarised modes
of the FMF fiber Birks_2015 . As there is no difference in time between the
different interferometers’ paths, no temporal post-selection is needed, and
thus there is no irreversible loss in the detection stage. Last, we highlight
that our scheme allows the generation and measurement of light modes carrying
orbital angular momentum (OAM) in an all-fiber platform, thus dispensing
complex mode multiplexers and sorters based on bulk optics Berkhout_2010 ;
Lavery_2012 ; Huang_2015 ; Wen_2020 ; Cao_2020 ; Liu_2020 . The all-in-fiber
generation of OAM light modes is not only useful for classical and quantum
communication. The compactness that can be achieved for our OAM
source/detection modules can certainly find applications in biophysics Padg11
; Grie03 ; Fran08 , metrology Damb13nc , and astronomy Tamb11  for instance.

##  II Experiment

In the original phase-encoded scheme Alice has a single-photon source (SPS)
producing time-localised single-photons, whose output is split into two paths
through a 50:50 bidirectional fiber coupler (FC). A phase modulator $\phi_{A}$
is placed in one of the arms, thus producing quantum states given by
$|\psi\rangle=(1/\sqrt{2})\left(|0\rangle+e^{i\phi_{A}}|1\rangle\right)$,
where $|0\rangle$ and $|1\rangle$ correspond to the upper and lower path modes
respectively (Fig. 1a). Both paths are connected to Bob, who also has another
phase modulator ($\phi_{B}$) allowing him to choose between the two required
measurement bases. The measurement procedure is concluded with both paths
superposed on another 50:50 FC, followed by a single-photon detector placed in
each outcome mode. In this case, the single-photon detection probability at D1
(D2) is proportional to $\mathrm{cos}^{2}(\phi_{A}-\phi_{B})$
[$\mathrm{sin}^{2}(\phi_{A}-\phi_{B})$], which is a requirement for quantum
key distribution for instance Gisin_2002 .

Figure 1: Phase-coding quantum cryptography. a) Original scheme based on a
long Mach-Zehnder interferometer that must be phase-stabilised. Phase
modulators are used for preparing and measuring the required states. The inset
shows that photons transmitted through paths ${\lvert 0\rangle}$ and ${\lvert
1\rangle}$ will arrive in the same detection time window. b) Typical time-bin
based scheme, relying on a single fiber interconnecting two asymmetrical short
interferometers (easier to stabilise). Temporal post-selection is required.
Photons will arrive in three different time-bins (see inset). In order to
extract a secret key the ones arriving too early or too late must be
discarded, thus, drastically reducing the communication rate by $50\%$. c) Our
setup resorting to few-mode fiber technology. It borrows concepts of the
robust time-bin configuration: two short interferometers interconnected
through a single (few-mode) fiber. Photons travelling through paths ${\lvert
0\rangle}$ and ${\lvert 1\rangle}$ are mapped to orthogonal optical modes
supported by the fiber using a photonic lantern, and demultiplexed in an
inverse fashion by another lantern. Since the local interferometers are now
symmetrical, these photons will again arrive in the same time bin and no
temporal post-selection is needed.

In the time-bin modification (Fig. 1b) Alice first splits an attenuated
optical pulse over an early (e) and a late (l) time-bin with an unbalanced
Mach-Zehnder interferometer (UMZI), where the imbalance must be greater than
the pulse length. Phase modulator $\phi_{A}$ is placed inside the
interferometer encoding the state
$|\psi\rangle=(1/\sqrt{2})\left(|e\rangle+e^{i\phi_{A}}|l\rangle\right)$,
which propagates towards Bob through an optical fiber. Bob possesses an
identical UMZI with phase modulator $\phi_{B}$ placed in the long arm, which
is also used to choose his measurement projection. At the outputs of Bob’s
UMZI, a temporal post-selection procedure is used to only detect photons that
took the $|e\rangle|l\rangle$ or $|l\rangle|e\rangle$ path combinations. These
instances are indistinguishable in principle, and thus display a sinusoidal
interference pattern that depends on $\phi_{A}-\phi_{B}$, which is the same as
in the original scheme (Fig. 1a). The other two possibilities that the photons
may take ($|e\rangle|e\rangle$ and $|l\rangle|l\rangle$) are distinguishable,
do not interfere and are thus discarded with temporal post-selection (Inset
Fig. 1b). This represents a loss by a factor two at the detector stage, which
is the trade-off when employing only one fiber for connecting the
communicating parties.

In our experimental demonstration (Fig. 1c), the single-photon source (SPS)
consists of a continuous wave distributed feedback telecom semiconductor laser
(Exfo OS-DFB) operating at 1546 nm. The signal is split into two paths
($|0\rangle$ and $|1\rangle$) using a 50:50 FC at Alice’s station. A variable
optical attenuator (ATT) placed before the fiber coupler is used to change the
power intensity to the single-photon level. A lithium niobate pigtailed
telecom phase modulator (Thorlabs LN65S-FC) is placed on one path
($\phi_{A}$). The phase modulator is driven by an electrical signal from a
function generator. The two paths are then connected to a commercial mode-
selective photonic lantern (Phoenix Photonics 3PLS-GI-15). It allows for
multiplexing information into the spatial modes supported by a FMF through the
implementation of the following mode mapping:
$|1\rangle\to|\textrm{LP}_{11a}\rangle$ ;
$|2\rangle\to|\textrm{LP}_{11b}\rangle$. polarization controllers (PC), not
shown for simplicity, are placed at each path $|0\rangle$ and $|1\rangle$ to
ensure the polarization state of each input mode is the same. We also employ a
variable attenuator in tandem with a 90:10 fiber coupler before each input to
the photonic lantern in order to ensure each single-mode input has the same
optical intensity, thus generating the intended state superpositions.

The FMF link following the lantern consists of either a direct back-to-back
connection with only a 10 m long FMF manual polarization controller (MPC), or
an added spool of FMF with a length of 500 m, with a measured total loss of
1.2 dB, including two home-made fiber connectors. The MPC is employed to
optimize mode demultiplexing at Bob’s lantern. The FMFs used in this
experiment are commercially available graded-index telecom fiber (OFS 80730),
with a loss coefficient of less than 0.22 dB/km as specified by the
manufacturer. The detection stage consists of another photonic lantern that is
now used as a demultiplexer, where the inverse mapping is performed. The
lantern outputs are the single-mode path-states ${\lvert 0\rangle}$ and
${\lvert 1\rangle}$, which are then recombined on another 50:50 fiber coupler.
The measurement basis implemented is defined by a second phase modulator
($\phi_{B}$). Standard MPCs in each arm are also employed to align the photon
polarization state such that in the final interferometer there is no path-
information available QErasure ; DecQErasure , which would compromise the
visibility of the observed interference. Following the final beamsplitter, we
place InGaAs single-photon counting modules (IdQ id210) operating with a gate
width of 2.5 ns, internal trigger rate of 1 MHz, overall detection efficiency
of 10%, and dark count probability per gate of $2.4\times 10^{-6}$. An in-
fiber polariser is placed before each detector. Overall, we obtain -14.6 and
-16.2 dB of extinction rate when measuring the outputs $|1\rangle$ and
$|2\rangle$ respectively for the opposing inputs ($|2\rangle$ and
$|1\rangle$). The insertion loss given by our commercial lanterns is 6.5 dB.

##  III Results

We focus on the generation of the states that are used for phase-encoding BB84
QKD Gisin_2002 . These are based on sets of orthogonal states divided between
two mutually unbiased basis (MUBs) Mafu_2013 . In our case, these states are
based on coherent superpositions of LP11 modes:

$|\mathrm{LP}_{\pm}\rangle=\frac{1}{\sqrt{2}}(|\mathrm{LP}_{11a}\rangle\pm|\mathrm{LP}_{11b}\rangle).$
(1)
$|\mathrm{OAM}_{\pm}\rangle=\frac{1}{\sqrt{2}}(|\mathrm{LP}_{11a}\rangle\pm
i|\mathrm{LP}_{11b}\rangle).$ (2)

The states $|\mathrm{OAM}_{\pm}\rangle$ are associated to the well known
Laguerre-Gaussian beams carrying orbital angular momentum (OAM) Li_2019 .
Figure 2a shows the theoretical transverse intensity profiles of the
$|\mathrm{LP}_{11a}\rangle$ and $|\mathrm{LP}_{11b}\rangle$ modes as well as
experimental results in the back-to-back case and after 500 m of propagation,
obtained with a linear polariser and an infrared CCD camera placed after the
FMF link. We also measured the intensity profiles associated to the states
$|\mathrm{LP}_{\pm}\rangle$ and $|\mathrm{OAM}_{\pm}\rangle$, which are shown
respectively in Figs. 2b and 2c. Here, each profile is prepared by setting the
appropriate phase $\phi_{A}=\\{0,\pi,\pi/2,3\pi/2\\}$ through a slow driving-
signal (0.2 Hz) applied to the phase modulator, thus sequentially preparing
them. The driving voltage of the phase modulator was previously calibrated. To
obtain the intensity profiles of the $|\mathrm{LP}_{11a}\rangle$ and
$|\mathrm{LP}_{11b}\rangle$ modes, each input arm was individually blocked.
For all these measurements, we worked with the source at the classical regime
bypassing the variable optical attenuator. These initial results show that the
photonic lantern can be used to prepare the coherent quantum superpositions of
the spatial states $|\mathrm{LP}_{11a}\rangle$ and
$|\mathrm{LP}_{11b}\rangle$, and also show that these superpositions suffer
little degradation following propagation after an extra 500 m, indicating
longer distances may be feasible with the current setup.

Figure 2: Spatial intensity profiles of the output of the FMF as measured by
an InGaAs CCD camera. a) Theoretical spatial profiles of the LP11a and LP11b
modes and experimental results in the back-to-back case and after an extra 500
m of FMF. b) and c) Theoretical and experimental spatial profiles (back-to-
back and 500 m) associated to the $|\mathrm{LP}_{\pm}\rangle$ and
$|\mathrm{OAM}_{\pm}\rangle$ states respectively.

In order to demonstrate the feasibility of quantum communication protocols
Xu_2020 ; Pirandola_2020  in our setup, we removed the CCD camera, reconnected
the variable optical attenuator, and connected the output of the FMF to Bob’s
station as shown in Fig. 1c. The required states $|\mathrm{LP}_{\pm}\rangle$
and $|\mathrm{OAM}_{\pm}\rangle$ are again prepared by driving $\phi_{A}$
sequentially. The variable attenuator is set to create a weak coherent state
with an average mean photon number of $\mu=0.4$ per gate width at Alice’s
output. Therefore, contribution of multi-photon events is negligible in our
experiment. We acquire the detection counts with a short integration time (14
ms) to visualise the interference pattern without active phase stabilisation
of the first local interferometer. The single-photon interference patterns
obtained after 500 m of propagation over the FMF are shown in Fig. 3. Figure
3a is the case corresponding to the detection at the first mutually unbiased
base defined by $\phi_{B}=0$. In Fig. 3b, we have the single-photon
interference curves obtained with the detection apparatus operating at the
second MUB defined by $\phi_{B}=\pi/2$.

Figure 3: Single-photon interference curves after 500 m propagation over the
few-mode fiber. The relative phase $(\phi_{A})$ is modulated with a slow-
driving triangular signal. a) Interference curves recorded while the detection
stage is set for measuring at the first mutually unbiased base defined by
$\phi_{B}=0$. b) Interference curves recorded while the detection stage is set
for measuring at the second mutually unbiased base defined by
$\phi_{B}=\pi/2$. The standard deviation is calculated assuming Poissonian
statistics. The deviation that can be seen from the experimental data to the
theoretical fit is due to the residual phase drift present acting on Alice and
Bob’s local interferometers that are not actively phase-stabilized.

From the data points displayed in Fig. 3 we calculate the probabilities of the
projection onto state $i$ given that $j$ was sent, where
$i,j=\\{{\lvert\mathrm{LP}_{+}\rangle},{\lvert\mathrm{LP}_{-}\rangle},{\lvert\mathrm{OAM}_{+}\rangle},{\lvert\mathrm{OAM}_{-}\rangle}\\}$.
The results for the transmitted vs. the projected states are plotted in Fig. 4
for both the back-to-back and 500 m cases. The average probability for the
main diagonal (corresponding to the cases where the same state is being
transmitted and projected onto) is $0.955\pm 0.022$ and $0.951\pm 0.024$ for 0
and 500 m respectively. This points to a lower bound in the error rate for
quantum key distribution using this setup and employing the BB84 protocol of
less than 5% at 500 m, showing the feasibility of this scheme. Out of that
number, 0.016% is due to dark counts of the detector. Also, as expected in
BB84 protocol, the probabilities when performing measurements in the non-
matching bases is around 50%. Furthermore, by considering a simple attenuation
model in the single-photon interference curves, we calculate that an extra
3.85 dB loss (corresponding to 17.5 kms of FMF) is needed to bring the QKD
error rate to 11%, a higher bound where usually no secret key can be
generated.

Figure 4: Probabilities associated to the states of the BB84 quantum
cryptography protocol. The vertical axis show the states being prepared, while
the horizontal one shows to which state the projection is made onto. a) The
probabilities in the back-to-back case, while b) shows the same probabilities
following 500 m of propagation over the FMF. Please see the text for details.

##  IV Conclusion

We have proposed and experimentally demonstrated a phase-encoded quantum
communication system based on few-mode fibers, which removes the irreversible
detection loss that is present on all time-bin quantum communication systems.
While it would be possible to employ the polarization degree-of-freedom to
implement this idea, i.e. sending orthogonal polarization states in the same
fiber, it would not allow an expansion to higher dimensions. Our scheme on the
other hand can be further upgraded to higher dimensions by using few-mode
fibers Sillard_2014 , lanterns Velazquez_2018  and beamsplitters Carine_2020
supporting more modes, while still not presenting extra intrinsic detection
losses, thus solving this fundamental issue regarding high-dimensional time-
bin encoding (more details in the Appendix). Furthermore, our proposal becomes
even more effective in a $d$-dimensional space, due to the growth of the loss
with $(d-1)/d$. Recently multi-core fibers (MCFs) have been successfully used
to transport spatially encoded quantum states Canas_2017 , but they still
suffer from a slow phase drift due to the fact each mode takes a separate core
in the fiber DaLio_2019 . FMFs on the other hand show no such drift, since the
modes propagate in the same core (See the Appendix), showing they may be ideal
for this application. Other recently demonstrated lanterns have had losses as
low as 0.7 dB for a 6-mode lantern Velazquez_2015 , with recent simulations
pointing out that lanterns of much lower losses (0.1 dB) could be reached
Li_2019 , further demonstrating the attractiveness of our scheme as a path in
the future use of time-bin encoding for quantum communication. For the
specific case of qubit-based time-bin quantum communication systems, if one
employs improved lanterns with 0.7 dB losses, then there is an increase in
approximately 70% in the overall detection efficiency compared to a standard
time-bin system, with the gain increasing if higher-dimensional systems are
employed. Finally, although we used BB84 as an illustrative example of the
possibilities of our setup for quantum information, other applications can
benefit directly from the removed detection losses such as high-dimensional
quantum cryptography Islam_2017b  and quantum random access codes
Tavakoli_2015 .

Another major achievement, that further shows that SDM technology can provide
more significant benefits to quantum information Xavier_2020 , is the use of
photonic lanterns to generate and decode OAM spatial states completely in-
fiber, which also has important applications in optical communications
Bozinovic_2013 . Finally, recent developments in integrated photonic circuits
Wang_2020 , could completely replace Alice and Bob’s optical setups with
integrated chips, greatly increasing compactness and robustness, and thus
dismissing the need for active phase stabilisation. We therefore envisage
these results will have significant and an imminent impact in areas such as
long-distance quantum communication, high-dimensional quantum information and
as a tool to further increase the capacity of classical communication
networks.

##  V Appendix

###  V.1 Expansion to higher dimensions

Our scheme is directly scalable by employing SDM components that support a
higher number of modes: lanterns Velazquez_2018 , few-mode fibers Sillard_2014
and multi-port beamsplitters Carine_2020 . Furthermore, our scheme becomes
even more beneficial at higher dimensions, due to the greatly increasing loss
with $d$ for the time-bin configuration. The proposal is shown in Fig. 5,
where we split the output of the single-photon source into a total of $d$
paths, using a multi-port beamsplitter (MBS). Each path has a different phase
modulator $\phi_{n}$, with $n$ ranging from 0 to $d-1$, allowing the
preparation of the high-dimensional path state
$|\Psi\rangle=1/\sqrt{d}\sum_{n=0}^{d-1}e^{i\phi_{n}}|n\rangle$, where
$|n\rangle$ represents the $n$th path and $1/\sqrt{d}$ is a normalization
factor. Then each path is connected to a $d$-mode photonic lantern in order to
excite the corresponding LP mode in a $d$-mode FMF. Finally, another $d$-mode
lantern is used to demultiplex the LP modes into the corresponding $d$ paths,
which contain the phase modulators $\phi_{n}^{\prime}$ to choose the state
projection. Finally, an $d\times d$ MBS is employed to superpose the different
paths, and the $d$ outputs are connected to single-photon detectors. Please
note that no temporal post-selection is needed, as in the bi-dimensional case.

Figure 5: Qudit-based quantum communication scheme using phase-encoding with
few-mode fibers.

###  V.2 Phase stability of phase-encoded states over a FMF

As discussed in the main text, phase encoding is not usually employed in the
original Mach-Zehnder configuration since the channel is not held stable long
enough for a key exchange to take place, due to fast phase instabilities.
Although MCFs show a considerable improvement in this aspect when compared to
independent single-mode fibers, they still have a residual drift which
requires active phase drift compensation Canas_2017 ; DaLio_2019 . FMFs on the
other hand, have the advantage that the individual spatial modes travel
through the same core, which is more intrinsically stable than independent
cores in the same cladding.

Figure 6: Fourier spectra of 50 min-long measurements of the interferometer
output, when the input state is continuously modulated with a 100 Hz
sinusoidal wave, for both the back-to-back (a) and after an extra 500 m of FMF
(b).

In order to demonstrate the stability, we adjusted the attenuator at the
output of the laser source to work with classical optical power levels. We
then continuously modulated the input state through the phase modulator
$\phi_{A}$ with a 100 Hz sinusoidal wave during 50 mins, both in the back-to-
back case (10 m), and with the 500 m long spool connected. The output from the
final beamsplitter in Fig. 1c was measured with a p-i-n photodiode. The
driving voltage is meant to impose a clear modulation signal creating well-
defined interference fringes. The output from the photodiode is recorded with
an oscilloscope. Figures 6a and 6b shows the Fourier transforms of the
recorded 50 minutes time signal for the back-to-back and 500 m cases
respectively. The inset shows zooms in around the centre peak at 0 Hz. The
environmental phase drift acting on an interferometric setup in the lab is
typically characterised by low frequency components. In our case, there is
substantial phase drift acting over Alice and Bob’s components, which consist
of parallel paths (single-mode fibers). Comparing both spectra (back-to-back
and 500 m) it is clear there is no significant difference in the low frequency
region, or even across a broader frequency range. This result points out to
the fact that adding an extra 500 m of FMF does not increase the environmental
phase drift, which leads to the possibilities that FMFs can be used as a
platform for phase-encoded states over longer distances.

To further corroborate the benefits of using a FMF in this regard, we perform
a brief comparison based on two multi-core fiber interferometers previously
used in other experiments. In Canas_2017 , it was demonstrated that tens of
minutes were needed for the environmental phase drift to cause a complete
inversion in the detected state in a 300 m long multi-core fiber
interferometer. In DaLio_2019  it can be seen that an equivalent measurement
on a 2-km long multi-core fiber interferometer is on the order of tens of
seconds. While it is difficult to make a precise measurement on the
difference, due to the fact the two experiments were done in different labs,
it still shows a clear difference when increasing the distance from 300 m to 2
kms in multi-core fiber interferometers. In our case, as we can observe no
appreciable difference when moving from back-to-back to 500 m, this further
shows the benefits of using FMFs in this regard.

## Acknowledgments

We acknowledge Ceniit Linköping University, the Swedish Research Council (VR
2017-04470), QuantERA grant SECRET (VR grant no. 2019-00392) and the Knut and
Alice Wallenberg Foundation through the Wallenberg Center for Quantum
Technology (WACQT) for financial support. G.L. was supported by Fondo Nacional
de Desarrollo Científico y Tecnológico (FONDECYT) (1200859) and ANID -
Millennium Science Initiative Program - ICN17_012).

## References

  * (1) A. Acín, I. Bloch, H. Buhrman, T. Calarco, C. Eichler, J. Eisert, D. Esteve, N. Gisin, S. J. Glaser, F. Jelezko, et al, The quantum technologies roadmap: a European community view, New J. Phys. 20, 080201 (2018).
  * (2) F. Xu, X. Ma, Q. Zhang, H-K. Lo, and J-W. Pan, Secure quantum key distribution with realistic devices, Rev. Mod. Phys. 92, 025002 (2020).
  * (3) S. Pirandola, U. L. Andersen, L. Banchi, M. Berta, D. Bunandar, R. Colbeck, D. Englund, T. Gehring, C. Lupo, C. Ottaviani, et al, Advances in Quantum Cryptography, Adv. Opt. Photon. 12,1012-1236 (2020).
  * (4) T. Lunghi, J. Kaniewski, F. Bussières, R. Houlmann, M. Tomamichel, A. Kent, N. Gisin, S. Wehner, and H. Zbinden, Experimental Bit Commitment Based on Quantum Communication and Special Relativity, Phys. Rev. Lett. 111, 180504 (2013).
  * (5) J. Pinnell, I. Nape, M. de Oliveira, N. TabeBordbar and A. Forbes, Experimental Demonstration of 11-Dimensional 10-Party Quantum Secret Sharing, Laser Photonics Rev. 14, 2000012 (2020).
  * (6) G. B. Xavier, N. Walenta, G. Vilela de Faria, G. P. Temporão, N. Gisin, H. Zbinden and J. P. von der Weid, Experimental polarisation encoded quantum key distribution over optical fibres with real-time continuous birefringence compensation. New J. Phys. 11, 045015 (2009).
  * (7) F. Grünenfelder, A. Boaron, D. Rusca, A. Martin and H. Zbinden, Simple and high-speed polarisation-based QKD. Appl. Phys. Lett. 112, 051108 (2018).
  * (8) C. H. Bennett, Quantum cryptography using any two nonorthogonal states. Phys. Rev. Lett. 68, 3121 (1992).
  * (9) N. Gisin, G. Ribordy, W. Tittel, and H. Zbinden, Quantum cryptography. Rev. Mod. Phys. 74, 145–195 (2002).
  * (10) P. Townsend, J. G. Rarity and P. R. Tapster, Single photon interference in a 10 km long optical fibre interferometer, Electron. Lett. 29, 634 (1993).
  * (11) C. Gobby, Z. L. Yuan and A. J. Shields, Quantum key distribution over 122 km of standard telecom fibre, Appl. Phys. Lett. 84, 3762 (2004).
  * (12) Q. Wang, W. Chen, G. Xavier, M. Swillo, T. Zhang, S. Sauge, M. Tengner, Z.-F. Han, G.-C. Guo and A. Karlsson, Experimental decoy-state quantum key distribution with a sub-poissionian heralded single-photon source, Phys. Rev. Lett. 100, 090501 (2008).
  * (13) Z. L. Yuan, A. R. Dixon, J. F. Dynes, A. W. Sharpe and A. J. Shields, Gigahertz quantum key distribution with InGaAs avalanche photodiodes, Appl. Phys. Lett. 92, 201104 (2008).
  * (14) S. Wang, W. Chen, J.-F. Guo, Z.-Q. Yin, H.-W. Li, Z. Zhou, G.-C. Guo, and Z.-F. Han, 2 GHz clock quantum key distribution over 260 km of standard telecom fibre, Opt. Lett., 37, 1008 (2012).
  * (15) A. Rubenok, J.A. Slater, P. Chan, I. Lucio-Martinez, and W. Tittel, Real-World Two-Photon Interference and Proof-of-Principle Quantum Key Distribution Immune to Detector Attacks, Phys. Rev. Lett. 111, 130501 (2013).
  * (16) Y. Liu, T.-Y. Chen, L.-J. Wang, H. Liang, G.-L. Shentu, J. Wang, K. Cui, H.-L. Yin, N.-L. Liu, L. Li, et al, Experimental Measurement-Device-Independent Quantum Key Distribution, Phys. Rev. Lett. 111, 130502 (2013).
  * (17) N. T. Islam, C. C. W. Lim, C. Cahall, J. Kim and D. J. Gauthier, Provably secure and high-rate quantum key distribution with time-bin qudits, Sci. Adv. 3, e1701491 (2017).
  * (18) A. Boaron, B. Korzh, R. Houlmann, G. Boso, D. Rusca, S. Gray, M.-J. Li, D. Nolan, A. Martin, and H. Zbinden, Simple 2.5 GHz time-bin quantum key distribution, Appl. Phys. Lett. 112, 171108 (2018).
  * (19) G. B. Xavier, G. P. Temporão and J. P. von der Weid, Employing long fibre-optical Mach-Zehnder interferometers for quantum cryptography with orthogonal states, Electron. Lett. 48, 775 (2012).
  * (20) Á. Cuevas, G. Carvacho, G. Saavedra, J. Cariñe, W. A. T. Nogueira, M. Figueroa, A. Cabello, P. Mataloni, G. Lima and G. B. Xavier, Long-distance distribution of genuine energy-time entanglement, Nat. Comm. 4, 2871 (2013).
  * (21) G. Carvacho, J. Cariñe, G. Saavedra, Á. Cuevas, J. Fuenzalida, F. Toledo, M. Figueroa, A. Cabello, J.-Å. Larsson, P. Mataloni, et al, Postselection-Loophole-Free Bell Test Over an Installed Optical fibre Network, Phys. Rev. Lett. 115, 030503 (2015).
  * (22) N. T. Islam, C. Cahall, A. Aragoneses, A. Lezama, J. Kim, and D. J. Gauthier, Robust and Stable Delay Interferometers with Application to $d$-Dimensional Time-Frequency Quantum Key Distribution. Phys. Rev. Appl. 7, 044010 (2017).
  * (23) G. B. Xavier and G. Lima, Quantum information processing with space-division multiplexing optical fibres, Commun. Phys. 3, 9 (2020).
  * (24) D. J. Richardson, J. M. Fini and L .E. Nelson, Space-division multiplexing in optical fibres, Nat. Photon. 7, 354 (2013).
  * (25) T. A. Birks, I. Gris-Sánchez, S. Yerolatsis, S. G. Leon-Saval and R. R. Thomson, The photonic lantern. Adv. Opt. Phot. 7, 107 (2015).
  * (26) G. C. G. Berkhout, M. P. J. Lavery, J. Courtial, M. W. Beijersbergen, and M. J. Padgett, Efficient sorting of orbital angular momentum states of light. Phys. Rev. Lett. 105, 153601 (2010).
  * (27) M. P. J. Lavery, D. J. Robertson, G. C. G. Berkhout, G. D. Love, M. J. Padgett, and J. Courtial, Refractive elements for the measurement of the orbital angular momentum of a single photon. Opt. Express 20, 2110 (2012).
  * (28) H. Huang, G. Milione, M. P. J. Lavery, G. Xie, Y. Ren, Y. Cao, N. Ahmed, T. A. Nguyen, D. A. Nolan, M.-J. Li, et al, Mode division multiplexing using an orbital angular momentum mode sorter and MIMO-DSP over a graded-index few-mode optical fibre, Sci. Rep. 5, 2015.
  * (29) Y. Wen. I. Chremmos, Y. Chen, G. Zhu, J. Zhang, J. Zhu, Y. Zhang, J. Liu, and S. Yu, Compact and high-performance vortex mode sorter for multi-dimensional multiplexed fibre communication systems, Optica 7, 254 (2020).
  * (30) Huan Cao, She-Cheng Gao, Chao Zhang, Jian Wang, De-Yong He, Bi-Heng Liu, Zheng-Wei Zhou, Yu-Jie Chen, Zhao-Hui Li, Si-Yuan Yu et al, Distribution of high-dimensional orbital angular momentum entanglement over a 1 km few-mode fiber Optica 7, 232 (2020).
  * (31) J. Liu, I. Nape, Q. Wang, A. Vallés, J. Wang and A. Forbes, Multidimensional entanglement transport through single-mode fiber, Sci. Adv. 6, eaay0837 (2020).
  * (32) M. J. Padgett and R. Bowman, Tweezers with a twist. Nat. Photon. 5, 343 (2011).
  * (33) D. G. Grier, A revolution in optical manipulation. Nature 424, 810 (2003).
  * (34) S. Franke-Arnold, L. Allen and M. J. Padgett, Advances in optical angular momentum. Laser & Photon. Rev. 2, 299 (2008).
  * (35) V. D’Ambrosio, N. Spagnolo, L. Del Re, S. Slussarenko, Y. Li, L. C. Kwek, L. Marrucci, S. P. Walborn, L. Aolita and F. Sciarrino, Photonic polarization gears for ultra-sensitive angular measurements, Nat. Comm. 4, 2432 (2013).
  * (36) F. Tamburini, B. Thidé, G. Molina-Terriza and G. Anzolin, Twisting of light around rotating black holes. Nat. Phys. 7, 195 (2011).
  * (37) S. P. Walborn, M. O. Terra Cunha, S. Pádua, and C. H. Monken, Double-slit quantum eraser, Phys. Rev. A 65, 033818 (2002).
  * (38) F. A. T. Ruiz, G. Lima, A. Delgado, S. Pádua, and C. Saavedra, Decoherence in a double-slit quantum eraser, Phys. Rev. A 81, 042104(2010).
  * (39) M. Mafu, A. Dudley, S. Goyal, D. Giovannini, M. McLaren, M. J. Padgett, T. Konrad, F. Petruccione, N. Lütkenhaus, and A. Forbes, Higher-dimensional orbital-angular-momentum-based quantum key distribution with mutually unbiased bases. Phys. Rev. A 88, 032305 (2013).
  * (40) Y. Li, Y. Li, L. Feng, C. Yang, W. Li, J. Qiu, X. Hong, Y. Zuo, H. Guo, W. Tong, et al, Mode-Selective Photonic Lanterns for Orbital Angular Momentum Mode Division Multiplexing. Appl. Sci. 9, 2233 (2019).
  * (41) P. Sillard, M. Bigot-Astruc and D. Molin, Few-Mode fibres for Mode-Division-Multiplexed Systems. IEEE J. Lightwave Tech. 32, 2824 (2014).
  * (42) A. M. Velázquez-Benítez, J. E. Antonio-López, J. C. Alvarado-Zacarías, N. K. Fontaine, R. Ryf, H. Chen, J. Hernández-Cordero, P. Sillard, C. Okonkwo, Chigo Okonkwo, et al, Scaling photonic lanterns for space-division multiplexing. Sci. Rep. 8, 8897 (2018).
  * (43) J. Cariñe, G. Cañas, P. Skrzypczyk, I. Šupić, N. Guerrero, T. Garcia, L. Pereira, M. A. S. Prosser, G. B. Xavier, A. Delgado, et al, Multi-core fibre integrated multi-port beam splitters for quantum information processing. Optica, 7, 542 (2020).
  * (44) G. Canãs, N. Vera, J. Cariñe, P. González, J. Cardenas, P. W. R. Connolly, A. Przysiezna, E. S. Gómez, M. Figueroa, G. Vallone, at al. High-dimensional decoy-state quantum key distribution over multicore telecommunication fibres. Phys. Rev. A 96, 022317 (2017).
  * (45) B. Da Lio, D. Bacco, D. Cozzolino, N. Biagi, T. N. Arge, E. Larsen, K. Rottwitt, Y. Ding, A. Zavatta and L. K. Oxenløwe, Stable transmission of high-dimensional quantum states over a 2-km multicore fibre. IEEE J. Sel. Topics in Quant. Electron. 26, 6400108 (2019).
  * (46) A. M. Velázquez-Benítez, J. C. Alvarado-Zacarías, G. Lopez-Galmiche, J. E. Antonio-López, A. Schülzgen, D. V. Ras, P. Sillard, C. Okonkwo, and R. Amezcua-Correa, Six Spatial Modes Photonic Lanterns, Optical fibre Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2015), paper W3B.3 (2015).
  * (47) N. Bozinovic, Y. Yue, Y. Ren, M. Tur, P. Kristensen, H. Huang, A. E. Willner and S. Ramachandran, Terabit-Scale Orbital Angular Momentum Mode Division Multiplexing in fibres, Science 340, 1545 (2013).
  * (48) J. Wang, F. Sciarrino, A. Laing and M. G. Thompson, Integrated photonic quantum technologies. Nat. Photonics 14, 273 (2020).
  * (49) A. Tavakoli, A. Hameedi, B. Marques and M. Bourennane, Quantum Random Access Codes Using Single $d$-Level Systems, Phys. Rev. Lett. 114, 170502 (2015).


# RHNAS: Realizable Hardware and Neural Architecture Search

Yash Akhauri*
Intel Labs, India
&Adithya Niranjan*
Intel Labs, India
&J. Pablo Muñoz
Intel Labs, USA
&Suvadeep Banerjee
Intel Labs, USA
&Abhijit Davare
Intel Labs, USA
&Pasquale Cocchini
Intel Labs, USA
&Anton A. Sorokin
Intel Labs, USA
&Ravi Iyer
Intel Labs, USA
&Nilesh Jain
Intel Labs, USA

###### Abstract

The rapidly evolving field of Artificial Intelligence necessitates automated
approaches to co-design neural network architecture and neural accelerators to
maximize system efficiency and address productivity challenges. To enable
joint optimization of this vast space, there has been growing interest in
differentiable NN-HW co-design. Fully differentiable co-design has reduced the
resource requirements for discovering optimized NN-HW configurations, but fail
to adapt to general hardware accelerator search spaces. This is due to the
existence of non-synthesizable (invalid) designs in the search space of many
hardware accelerators. To enable efficient and realizable co-design of
configurable hardware accelerators with arbitrary neural network search
spaces, we introduce RHNAS. RHNAS is a method that combines reinforcement
learning for hardware optimization with differentiable neural architecture
search. RHNAS discovers realizable NN-HW designs with $\mathbf{1.84\times}$
lower latency and $\mathbf{1.86\times}$ lower energy-delay product (EDP) on
ImageNet and $\mathbf{2.81\times}$ lower latency and $\mathbf{3.30\times}$
lower EDP on CIFAR-10 over the default hardware accelerator design.

##  1 Introduction

The success of deep neural networks (DNNs) in fields such as computer vision,
recommender systems, and virtual reality is driving an exponential increase in
deployment scenarios for deep learning (DL) systems. This success has also
been driven by larger, compute-intensive models. These large models have
higher latency and energy costs - one of the primary hurdles in developing
effective artificial intelligence (AI) systems. To meet the increasing
engineering and deployment requirements needed to optimize AI systems, there
has been a shift to utilizing automated neural architecture search techniques
[1, 2, 3], as well as automated hardware optimization [4, 5]. While HW-Aware
ML on fixed hardware does improve efficiency, this is often sub-optimal for
configurable accelerators. Such accelerators present significant space for
improvement through optimization on components such as number of processing
elements, buffer sizes, and data-types, among others. Fixing the NN
architecture and optimizing the HW design also leaves performance on the table
- different accelerators have different modes of operations and optimizations.
Therefore, an optimal layer choice for a given accelerator may not be the best
choice for another. To model these inter-dependencies, there has been growing
interest in evolutionary algorithm-based [6], reinforcement learning-based
methods [7, 8, 9], as well as fully differentiable methods [10, 11] to co-
design both hardware accelerators and neural network architectures.

Fully differentiable approaches often introduce HW parametrization to
represent HW design choices. While some accelerators are fully parametrizable,
several of them are _sparsely valid_. Sparse validity can have many reasons,
be it Instruction Set Architecture (ISA) limitations, arbitrary constraints in
synthesizable designs or NN-HW interactions. In the context of developing HW
accelerators and compiler stacks, such issues with validity can arise quite
often during early stages of development. Reducing this sparsity can take
considerable verification and development efforts, delaying product and
innovation life-cycles. We therefore feel that the assumption of fully
parametrizable/valid spaces design spaces to be fairly restrictive and hope to
address some of these issues with our work. Optimizing on sparsely valid
spaces is a strictly harder problem than having non-differentiable hardware
costs, as no hardware cost exists for non-realizable designs. We discuss the
difficulty of co-design in sparsely valid spaces in more detail in Section 3.
For the remainder of this paper, we use _non-realizable_ and _invalid_
hardware designs to refer to choices that are not synthesizable.

In this paper, we look into finding a _realizable_ solution in two aspects.
First, finding optimal yet synthesizable designs in the hardware accelerator
space. Second, proposing a way of co-discovering NN-HW designs with reasonable
run-time and compute requirements by leveraging differentiable neural
architecture search, something that is becoming increasingly important
considering the carbon impact of AI [12]. For example, BestOB [7] utilizes a
RL controller for co-design and requires 1000 GPU hours to co-design NN-HW on
the CIFAR-100 dataset.

We propose a method that integrates a reinforcement learning-based hardware
optimizer with differentiable neural architecture search. This method allows
us to efficiently explore a prohibitively large NN-HW joint space with non-
differentiable hardware costs and sparsely valid hardware designs.  With these
improvements, we co-design NN-HW on the ImageNet dataset in 186.7 GPU hours
and 4 GPU hours on the CIFAR-10 dataset. In this paper, we discuss the
following contributions:

  1. 1.

We introduce RHNAS, an NN-HW co-design method that integrates RL-based
hardware optimizers with differentiable NAS. Our method successfully overcomes
the challenges associated with sparse validity - a failure point for existing
differentiable co-design works. It does so with minimal overhead to a standard
DNAS solution.

  2. 2.

We benchmark our RL-based hardware optimizer and use Bayesian hyperparameter
optimization to identify the best hyper-parameters for a fair study of a range
of standard RL algorithms (PPO, A2C, DQN) [13, 14, 15].

  3. 3.

We empirically verify that using fully differentiable approaches to co-design
does not aid in discovering valid hardware accelerator designs. We benchmark
our RL-based hardware optimizer against a multi-layer perceptron (MLP) based
hardware generator [11]. Further, we study the convergence of differentiable
algorithms which learn optimal hardware parametrizations [10] on our HW
accelerator search space.

##  2 Related Work

Hardware-aware NAS - Early efforts towards the discovery of NN architectures
primarily focused on optimizing accuracy using reinforcement learning (RL) and
evolutionary algorithms (EA) [16, 17, 18]. These algorithms generally require
considerable computational resources and need to evaluate and train many
candidate architectures to completion. Differentiable NAS strategies were
introduced to reduce the cost of discovering efficient neural networks [19,
2]. Most of the early focus was on maximizing accuracy, which leads to very
large neural networks with high latency and energy requirements. To address
these demanding requirements, many works [3, 20, 21, 22, 23] model the
latency, floating-point operations, or other performance-related metrics as
regularizers to guide NAS.

Hardware design optimization - DNN accelerators [24, 25, 26, 27, 28, 29] have
been developed with the prospect of potential gains over general-purpose
hardware. Eyeriss is an energy-efficient CNN accelerator [30] that delivered
10 times the energy efficiency over mobile GPUs. Such improvements lead to
further research in novel microarchitectures and data flow styles. To get the
best performance on a range of workloads, many accelerators are being designed
for configurability based on the selected workload. This results in many
possible hardware resource assignments dictated by the NN topology to be
deployed. Several approaches have been proposed to identify optimal resource
assignments, e.g., ConfuciuX [5], GAMMA [4], MARVEL [31] utilize RL and GA
(Genetic Algorithms) to decide the hardware resource assignment for a given
workload.

Hardware-Neural Network Co-design - While NAS and HW design optimization give
good results when applied standalone, the significant inter-dependencies
between NN and HW components suggest an opportunity for improvement by jointly
modeling NN topologies and HW designs. DANCE [11] introduces an MLP based HW
generator along with a differentiable performance estimator to guide the joint
differentiable search. DNA [10] converts the hardware and neural network
design space into a categorical probability distribution and uses an MLP HW
performance predictor for the co-design of hardware and neural network. We
find that these fully differentiable solutions cannot adapt to spaces that has
many invalid hardware design choices. Some works propose an RL controller for
co-design [7, 32]. However, these works suffer from high search cost problems
[11].

##  3 Configurable Hardware Accelerators and Sparsely Valid Design Spaces

We use the Versatile Tensor Accelerator (VTA) as our configurable hardware
accelerator. VTA is a programmable deep learning accelerator [33] with
parametrizable architecture paired with Apache TVM [34], a deep learning
compiler stack. This integrated end-to-end deep learning stack provides a
flexible platform to implement our NN-HW co-design framework. The VTA HW is
designed in Constructing Hardware in a Scala Embedded Language (CHISEL) [35]
that facilitates full parametrization where underlying memory elements can be
programmatically customized for the HW backend (ASIC/FPGA), but the hardware
designer also retains the ability to reuse existing hardware components. The
CHISEL flow generates HW modules expressed in register-transfer language
(RTL), which can be used for downstream ASIC/FPGA synthesis. The TVM framework
utilizes a HW simulation guided schedule auto-tuning platform known as autoTVM
[36] for computing optimal data access patterns in adaptation to changes in
the workload and HW configurations. Details on how we calculate latency and
energy-delay product (EDP) are provided in the appendix.

For this work, we have modified the TVM compiler to utilize an analytical
bandwidth minimization technique to compute the optimal tiling scheme of NN
operator schedules rapidly. In addition, we have extended the TVM-VTA stack to
support depth-wise convolutions to allow simulation of our NN search space.
Table 1 shows the HW architecture parameters we have considered and their
possible values (in $log_{2}$ scale). The product of $block\\_in$ and
$block\\_out$ decides the size of the GEMM core of the accelerator, the input
(Inp), weight (Wgt), and accumulator (Acc) buffers are on-chip SRAMs, $\mu$Op
width defines the size of the RISC micro-ops, and the $\mu$Op buffer defines
the size of the micro-op cache.

Table 1: VTA parameter ranges ($log_{2}$). HW Feature | Range
---|---
block_in | [3, 6]
block_out | [3, 6]
$\mu$Op Width (bits) | [5, 6]
$\mu$Op Buffer (kB) | [5, 6]
Inp Buffer (kB) | [13, 20]
Wgt Buffer (kB) | [13, 20]
Acc Buffer (kB) | [13, 20]

While independent consideration of each architectural parameter in Table 1
leads to a large theoretical design space (32768 possible configurations),
many parameter combinations are invalid. This is a common problem on hardware
accelerators. On VTA, it is due to ISA restrictions. For example, large
compute cores and local memory buffers can improve accelerator performance,
but the number of bits required to address such cores/buffers may exceed the
instruction width, thus limiting the scope of joint architectural parameter
exploration. Another issue involves hardware resource limitations. For
example, FPGA targets may lack the SRAM required, or ASIC synthesis may not
reach the specified frequency. There can also exist arbitrary points in the
parameter space which are not synthesizable due to routing congestion of
larger designs. While such designs are within the parameter space, it is not
possible to realize them. As configurable accelerators may have invalid
parameter choices, we implement a validity checker function that analytically
detects whether a hardware design is valid.

##  4 The RHNAS Flow

###  4.1 Proposed Co-Exploration

Figure 1: Overall RHNAS flow diagram

We introduce an RL HW optimizer whose training is disentangled from
differentiable NAS to address the dual problem of sparsely valid hardware
design space and high NN architecture search cost associated with RL and EA-
based NAS algorithms. Figure 1 shows an overview of our RHNAS flow. Our
differentiable NAS algorithm samples neural network architectures from a
super-net. This sampling is done from a super-net using a categorical
architectural distribution. We use a pre-trained RL HW optimizer to generate
an optimized accelerator configuration for the sub-sampled NN architecture.
For every NN architecture sampled during the differentiable neural
architecture search process, the RL HW optimizer receives the subsampled NN
architecture along with a default HW accelerator configuration template and
outputs an optimized HW accelerator configuration. A performance predictor is
then used to get the performance of the optimized NN-HW design. This
performance is used as a regularizer in the differentiable NAS minimization
objective.

Table 2: RHNAS’s Optimization
$\min\limits_{\alpha,\theta}\,\,L_{train}(Z_{\alpha,\theta})+\lambda
L_{hw}(\sigma(\alpha),H^{opt})$
---
$s.t.\quad L_{hw}=\text{PerfPredictor}(\sigma(\alpha),H^{opt})$
$s.t.\quad H^{opt}=\text{HWOpt}(\sigma(\alpha),H_{0})$

RHNAS’s optimization is formulated as shown in Table 2. $\sigma$ is the
Softmax function. $\alpha$ is the continuous variable parametrization of the
NN architecture. $\sigma(\alpha)$ is the categorical probability distribution
of the NN supernet. $Z_{\alpha,\theta}$ is the one-hot representation of the
subnetwork sampled using the $\sigma(\alpha)$ and $\theta$ weights. HWOpt is
the RL HW optimizer. $H_{0}$ is the template HW accelerator configuration that
the HWOpt is initially provided with. If HWOpt generates an invalid
accelerator configuration, it simply outputs the template HW accelerator
configuration $H_{0}$. $L_{hw}$ is the HW cost loss determined by the NN-HW
Performance Predictor (e.g., Clock Cycles, Energy-Delay-Product) and is used
to guide the neural architecture search.

As indicated in Table 2, we use the NN architecture categorical distribution
$\sigma(\alpha)$ instead of the sub-sampled one-hot representation
$Z_{\alpha,\theta}$ for the HW loss. In our tests, using the one-hot
architectural choice $Z_{\alpha,\theta}$ in Table 2 leads to noisy training,
whereas $\sigma(\alpha)$ leads to smoother training as it is a continuous
probability distribution.

###  4.2 HW Performance Predictors

For any given accelerator configuration and neural network architecture, the
simulator takes significant time to output a performance metric, as seen in
Table 4. For example, we need 2 minutes to get a single cycle-accurate RTL
simulation of an end-to-end ResNet50v2model. The large query time makes
computing HW metrics on the fly impractical. Metrics such as FLOPs are not
capable of capturing the intricacies of the NN-HW design space, we find that
FLOPS are weakly correlated with cycle count. To quantify this - the Pearson’s
correlation coefficient between cycle count and FLOPs for 55420 randomly
sampled NN-HW design choices was 0.15 (weakly positive correlation).

We use an MLP to avoid large query times and capture the relations between NN-
HW designs and performance. The MLP takes as input the joint NN-HW embedding
and predicts the Latency/EDP. As seen in Table 2, the NN embedding the MLP
takes is a probability distribution ($\sigma(\alpha)$) instead of a one-hot
encoding. We include a study in the appendix to validate that the performance
predictor is able to accurately interpolate between different discrete NN
architectural choices.

###  4.3 Reinforcement Learning for HW Optimization

We use an RL agent to optimize over our sparsely valid hardware design space.
The agent receives a representation of the neural network ($N$) and an initial
HW accelerator template ($H_{0}$) of our accelerator. The agent modifies the
accelerator configuration to improve the NN-HW design latency/EDP. In order to
make the agent minimize the latency/EDP, we provide the negative of the
predicted latency/EDP as the reward. In a sparsely valid design space,
optimizing only latency/EDP can lead to invalid accelerator designs. To
address this challenge, we penalize the agent with $C_{inv}$ for providing
invalid accelerator design parameters. We study two intuitive settings based
on this formulation:

Composite setting: In this formulation, we use a concatenated representation
of the NN and HW $(N,H_{t})$ as the environment state. The agent takes a
discrete action that specifies the whole HW configuration ($H_{t}$) every time
step and runs for a fixed number of timesteps $t_{max}$ every episode. As the
composite agent provides a new hardware configuration in each time step, it
may lose incentive to optimize a suboptimal accelerator configuration due to
penalty for invalid states. To encourage continuous improvement, we introduce
a penalty $B$ imposed when the accelerator configuration the composite agent
provides is worse than its previous accelerator configuration
($P(N,H_{t})>P(N,H_{t-1}$).
Sequential setting: In this approach, the RL agent optimizes only one HW
accelerator parameter in a time step, making $t_{max}$ passes in an episode.
Here, we modify the state to include a one-hot vector $S_{t}$ to provide the
agent context about which parameter to optimize. At each time step, the agent
takes a discrete action choice ($H^{p}_{t}$ for the $p^{th}$ step). After the
agent has generated an action for every HW parameter (we have 7 parameters)
$t_{max}$ times, we get the final HW configuration. In this setting, we found
that providing a reward after generating a full template to be more stable
than rewards at every time-step.
We discuss implementation details of the agent and the state representation
further in Section 5.2 and the appendix.

Table 3: RL formulation. $O_{t}$, $A_{t}$, $R_{t}$ are state, action and reward respectively Composite setting | Sequential setting
---|---
$O_{t}=(N,H_{t})$; |  $O_{t}=(N,H_{t},S_{t})$;
$A_{t}=(H_{t})$ | $A_{t}=(H^{p}_{t})\quad s.t.\quad H^{p}_{t}\in[0,7]$
$R_{t}=\begin{cases}&-P(N,H_{t})-C_{inv}(H_{t})\text{, if }t=t_{max}\\\ &-B\text{, if }P(N,H_{t})>P(N,H_{t-1})\\\ &0\text{, otherwise}\\\ \end{cases}$ | $R_{t}=\begin{cases}&-P(N,H_{t})-C_{inv}(H_{t})\text{, if }S_{7}=1\\\ &0\text{, otherwise}\end{cases}$

##  5 Evaluation of RHNAS Components

###  5.1 Performance Predictor Network

Table 4: Testing the predictor network performance by measuring the Kendall Tau rank correlation coefficient. ($\tau$) Predictor Network |  $\tau$ metric
---|---
EDP | 0.982
Cycle Count | 0.982
GEMM Cycle Count | 0.984
$\mu$Op load (bytes) | 0.975
Weight load (bytes) | 0.985
Input load (bytes) | 0.983
Accumulator load (bytes) | 0.984

Our performance predictor network is a fully connected neural network with 3
layers with 512 neurons in each hidden layer and intermediate ReLU activation
functions. Table 4 shows the clock cycles, EDP and other simulation metrics.
We used the VTA simulator to randomly sample 53600 NN-HW pairs to generate our
dataset. We train our networks with the L1 loss between predicted and actual
performance. Our design space changes the buffer and GEMM core sizes by powers
of two. There can be a large difference in the clock cycles for different NN-
accelerator designs. We found that the _Robust Scaler_ normalization from
sklearn [37] works well on our data. Since we use the performance predictor
output to guide our search, it is important that the ranking of different NN-
HW architecture pairs are preserved. To verify this, we use the Kendall rank
correlation coefficient ($\tau$) [38] \- a $\tau$ of 1 indicates that the rank
ordering is perfectly preserved. Since we find that $\tau>0.97$, we conclude
that our predictors can be used in the RHNAS.

###  5.2 Hardware Optimizers

For the RL HW optimizer, we experimented with the Proximal Policy Optimization
(PPO) [13], Advantage Actor-Critic (A2C) [14], and Deep Q-Network (DQN) [15]
algorithms. We implemented the composite and sequential action spaces and
environments with the OpenAI gym environment and use implementations of PPO,
A2C and DQN algorithms from the stable-baselines3 library [39].

Table 5: Optimality of HW generated by RL HW generators. Cycle Count | Composite | Sequential
---|---|---
PPO |  99.82% | 99.79%
A2C | 99.82% | 56.51%
DQN | - | 98.10%
EDP | Composite | Sequential
PPO |  98.61% | 97.56%
A2C | 97.53% | 44.05%
DQN | - | 95.21%

For the RL environment, we treat the validity penalty ($C_{inv}$), maximum
timesteps ($t_{max}$), and the intermediate penalty ($B$), as environment
hyper-parameters. We found that tuning these parameters carefully was key to
finding agents that generated both optimal and synthesizable hardware
configurations .For example, for penalty values that were too low ($C_{inv}$ <
1.5), the RL agent would generate very large accelerator configurations, which
do not fit on our target hardware. On the other hand, larger penalties meant
the agent would quickly stop at a valid sub-optimal HW configuration. Using
HPO (details in appendix), we were able to find agents that give near-optimal
synthesisable configurations. From Table 5, we observe that PPO [13] with the
composite setting gives close to optimal configuration ($\sim 99\%$ optimal)
for both cycle count and energy delay product (EDP) while generating no
invalid configurations

####  5.2.1 Ablation Study on Hardware Optimizers

A recent work, DANCE [11] introduced an MLP based hardware generator. The
data-set is generated by sampling random NNs from the search space and using
the output of a hardware generation toolkit as the target HW. This toolkit
uses exact algorithms such as exhaustive search or branch-and-bound algorithms
to output optimized HW configurations for an input NN. We reference this
method as Exhaustive-HWGEN in Table 6. In our tests, Exhaustive-HWGEN is a 3
layer MLP with 512 neurons and is trained by taking a categorical cross
entropy loss with respect to the optimal HW design for a given NN
architecture. The optimal HW design is found by exhaustive search. While this
method slightly exceeds the optimality of hardware generated by our
reinforcement learning agents, it has a significantly larger training time for
a general accelerator space because of the exhaustive search. Exhaustive
search is not practically feasible as accelerator designs get increasingly
complex. If we look at most accelerators, the search space is of the order of
$10^{30}$ to $10^{70}$ [10, 5] choices, generating the data-set for
Exhaustive-HWGEN would be completely infeasible.

Table 6: Optimality of HW generated by MLP HW generators. Cycle Count | Optimality | % invalid
---|---|---
Exhaustive-HWGEN | 99.9% | 0%
Perf-HWGEN | N/A | 100%
EDP | Optimality | % invalid
Exhaustive-HWGEN | 99.4% | 0.8%
Perf-HWGEN | N/A | 100%

As an alternative to exhaustive search, we test _Perf-HWGEN_ , which is a
hardware generator akin to Exhaustive-HWGEN but with a training procedure that
does not require exhaustive enumeration. We utilize the performance predictor
MLP from Section 5.1 in our loss formulation for the Perf-HWGEN hardware
generator. Since we are not using exhaustive enumeration as a search strategy,
we also have to penalize invalid configurations given by the Perf-HWGEN. For
this purpose, we use _ValidNet MLP_. This MLP indicates whether an input HW
accelerator configuration is valid. Details on ValidNet MLP architecture and
training is provided in the appendix.

The training loss for Perf-HWGEN is formulated as,

$\displaystyle\begin{split}&\min\limits_{\beta}\,\,L_{hw}(Z,\text{Perf-
HWGEN}_{\beta}(Z))+\lambda L_{valid}(\text{Perf-HWGEN}_{\beta}(Z))\end{split}$
(1)

As indicated in Equation 1, the training objective of Perf-HWGEN is to jointly
minimize the hardware cost ($L_{hw}$) and validity loss ($L_{valid}$). $Z$ is
a randomly sampled NN architecture and $\beta$ are the parameters of Perf-
HWGEN. If $\lambda=0$, there is no penalty for invalid configurations. In such
cases, the Perf-HWGEN generates invalid configurations by merely maximizing
the performance.

Our Perf-HWGEN uses the $L_{valid}$ loss to guide the search for valid
configurations. We sweep $\lambda\in[10^{-4},10^{2}]$ but do not converge to a
Perf-HWGEN that generates valid configuration. We study the behavior of
ValidNet in guiding search for valid hardware configurations in Section 6.4.
It is also important to note that the performance predictor network has only
been trained on valid configurations, thus the optimality numbers for Perf-
HWGEN are not reliable, and labeled as _Not Applicable (N/A)_ in Table 6 for
invalid configurations.

Table 7:  RHNAS Ablation studies. EDP has been normalized, Lat is Latency.  Target | Acc | Lat | EDP
---|---|---|---
ASIC | (%) | $ms$ |
DSNAS | 74.4 | 22.13 | 28
HW Aware NAS | 74.1 | 23.67 | 31
Sequential Optimization | 74.2 | 13.54 | 17
RHNAS Lat | 74.5 | 12.03 | 16
RHNAS EDP | 74.5 | 12.73 | 15
RHNAS Lat+FLOPs | 73.4 | 11.77 | 12
DSHWNAS | 74.2 | Invalid Design

Table 8:  Evaluating RHNAS with other co-design works. Target | Latency | Area | Acc
---|---|---|---
ASIC | $ms$ | ($mm^{2}$) | (%)
DANCE | 8.13 | 2.73 | 68.70
DNA 4bit | 1.25 | 5.5 | 71.7
DNA 16bit | 2.85 | 2.12 | 72.2
NHAS | 1.58 | 5.87 | 70.74
EDD | 11.15 | - | 74.7
RHNAS | 12.03 | 1.21 | 74.5

##  6 RHNAS Evaluation

Table 9: Comparing against other efficient neural networks. Target | Latency | Acc
---|---|---
ASIC | $ms$ | (%)
MNASNet-1 | 17.31 | 75.20
MobileNetV2 | 12.16 | 71.66
RHNAS | 12.03 | 74.5

For our neural architecture search strategy, we employ the super-net and
choice of blocks as used in the released implementation of DSNAS [2]. Our NN
architecture sampling, hyper-parameter settings and parameter update strategy
is the same as DSNAS [2]. DSNAS introduces a task-specific direct neural
architecture search without parameter retraining. DSNAS requires 420 GPU hours
to discover networks with accuracy comparable to two-stage (search and train)
methods. As we train on an NVIDIA V-100 GPU, we complete the search in 187 GPU
hours.

We have conducted several experiments on RHNAS using the ImageNet (ILSVRC2012)
[40] and CIFAR-10 [41] data-sets. We include in our evaluation three ASIC
baselines: DANCE [11], DNA [10] and NHAS [6]. Due to differences in
accelerator designs and the neural network backbone, it is not feasible to
fairly compare the latency and EDP without further standardization. The HW
metrics for different works are largely different based on the capabilities
and optimization of the accelerators. We draw the attention of the reader to
the fact that DNA cannot discover valid designs on our hardware search space,
and is studied in detail by our DSHWNAS test in Table 8 and Section 6.4. The
MLP-based hardware generator in DANCE has been studied in Section 5.2.1. DANCE
can work on our hardware design space by exhaustive enumeration, which is not
a scalable approach as the complexity of hardware accelerators increase. To
address the lack of standardization across accelerators, we deploy MNASNet-1
[42] and MobileNetV2 [42] on the same VTA accelerator as discovered by RHNAS
Latency in Table 9. We also demonstrate the effectiveness of RHNAS, by
implementing HW-Aware NAS and NN-HW sequential optimization methods on our
accelerator search space as shown in Table 8. All our networks have been
quantized to 8 bits.

###  6.1 RHNAS on CIFAR-10

To demonstrate the effectiveness of RHNAS in discovering solutions with low
latency and high accuracy, we use a smaller variant of the DSNAS network with
9 layers instead of 20 layers and try to find good HW-NN pairs on the smaller
network. There are $4^{9}=262144$ neural networks in our CIFAR-10 search
space. We randomly sample 45 neural networks and train them using the DSNAS
[2] SPOSretrain240 strategy detailed in the appendix along with more
experimental details. We measure the latency of these neural networks on the
default VTA accelerator configuration and are represented by "Random NN
Sampling w/ Default HW". We then measure the latency of these neural networks
on the optimized HW accelerator design generated by the RL HW Optimizer
referenced as "Random NN Sampling w/ Optimized HW". Sequential Optimization
refers to a method where we first optimize the neural network architecture for
accuracy, and use the RL HW Optimizer to generate the hardware accelerator
design for the neural network. We also include HW-Aware NAS, which attempts to
find an optimal neural network architecture on the default HW accelerator
design. As seen from Figure 2, RHNAS designs outperform all other methods
provided in the figure.

Figure 2: RHNAS discovers efficient NN-HW choices that give high accuracy on
CIFAR-10.

###  6.2 Hardware-Aware NAS

In Hardware-Aware NAS, we try to minimize cycle count and EDP without a HW
optimizer in the loop. As described by Equation 2, we utilize the default HW
configuration and optimize the NN parametrization on the fixed HW. RHNAS finds
solutions with $1.97\times$ lower latency and $2.07\times$ lower EDP with a
$0.4\%$ accuracy gain on ImageNet and $2.81\times$ lower latency and
$3.30\times$ lower EDP on CIFAR-10.

$\displaystyle\begin{split}&\min\limits_{\alpha,\theta}\,\,L_{train}(Z_{\alpha,\theta})+\lambda
L_{hw}(\sigma(\alpha),H_{0})\end{split}$ (2)

###  6.3 Sequential Optimization

One of the primary points of comparison for a joint search algorithm is to
benchmark with sequential optimization algorithms. Since our RL HW optimizer
agent is disentangled from the NN architecture search, we can take the
resultant network from the NAS algorithm with no simulation/FLOPs loss and
generate HW optimized for that topology. Further, we also simulate the
original DSNAS algorithm on our HW. As seen from Table 8, RHNAS Latency is
able to find a solution with $1.13\times$ lower latency for $0.1\%$ higher
accuracy on the ImageNet dataset and $1.5\times$ lower latency on CIFAR-10
dataset.

###  6.4 Fully Differentiable Joint Search

Several works [10, 11] introduce a fully differentiable approach to NN-HW co-
design. While these works produce excellent results in accelerator design
spaces with sparse optimas, we attempt to reproduce a generalized fully
differentiable search algorithm as shown in Equation 3 and demonstrate its
inability to navigate our sparsely valid accelerator design space. Our joint
differentiable architecture search is formulated below. We use the same
$L_{valid}$ as described in Equation 1.

$\displaystyle\begin{split}&\min\limits_{\alpha,\gamma,\theta}\,\,L_{train}(Z_{\alpha,\theta})+\lambda
L_{hw}(\sigma(\alpha),\gamma)+\beta L_{valid}(\gamma)\end{split}$ (3)

As seen in Table 8, this DSHWNAS formulation converges with an accuracy of
74.2%,

Table 10: Gradient Interpolation Study
$\Delta_{j}=\frac{\partial}{\partial\widetilde{\gamma_{j}}}\text{L1Loss}(\text{ValidNet}(\widetilde{\gamma}),1)$
---
$\widetilde{\gamma}=\begin{cases}&(1-\varphi)\text{v}+(\varphi)\text{r}\;\;\;\,\,\;\;\;\forall\varphi\in[0,1)\\\
&(2-\varphi)\text{r}+(\varphi-1)\text{i}\;\;\forall\varphi\in[1,2)\\\
\end{cases}$

but discovers an invalid HW design. We ran this test with $\beta=10^{x}$ for
$x\in[-7,7]$ but were not able to converge to a valid HW choice. We suspect
that this is due to the irregular gradients given our sparsely valid design
space. We analyze the gradient of our ValidNet model to understand this
behaviour, and observe that the gradients in proximity of invalid design
choices are smaller in magnitude than the gradients around valid design
choices. This may mean the gradients are ineffective in navigating the design
space with $L_{valid}$ given in Equation 3.

Figure 3: Gradient magnitudes when interpolating from valid to invalid.

To study this, we measure the gradient to the HW parametrization (input to the
ValidNet) with respect to a target output of 1 (where the HW parametrization
is valid.). We smoothly interpolate the input HW parametrization from a valid
design ($v$) to a random design ($r$) and finally to an invalid design ($i$).
We record the gradient magnitude to the HW parametrization. Explicitly, we
convert the information in Table (10) ($\delta_{j}$) to a heat map in Figure
3. We smoothly interpolate for $\varphi\in[0,2)$ and use the resulting state
to fetch the gradients through ValidNet.

As evident in Figure 3, the gradients are zero almost everywhere other than
the valid design point around $\varphi\in[0,0.25]$. The observed behavior is
consistent for many iterations of random sampling. As seen in our Perf-HWGEN
tests previously, $\lambda\neq 0$ does not aid in the discovery of valid
designs. This may be explained by the gradient study shown in Figure 3.

##  7 Conclusion

In this paper, we demonstrate the need for an efficient solution to address
issues with differentiable HW optimization (DSHWNAS) and MLP based HW
generation (Exhaustive/Perf-HWGEN) on sparsely valid spaces. Our method
proposes to enables co-design on sparsely valid search spaces by disentangling
the training of a non-differentiable RL HW optimizer from NAS. The trained RL
HW optimizer is used along with a performance predictor to guide the
differentiable NAS algorithm. Our formulation is compatible wth other
differentiable NAS strategies and is task agnostic (applicable to general
tasks in classification, segmentation, detection, among others.). We can also
adapt our RL HW optimizer to more complex HW search spaces, as explored in
ConfuciuX [43].

Sparsely valid spaces are common in HW accelerator design. Arbitrary
constraints on architectural parameters can arise due to resource limitations
on accelerators that support different PE (Processing Element) connectivity
and compiler mapping strategies. Reducing the sparsity of valid design choices
incurs a cost in hardware design/verification effort, physical design effort,
fabrication cost, and energy efficiency. More importantly, it increases
development time and delays deployment. We hope that the presented technique
can help push accelerated deep learning into deployment faster. In future
work, we hope to introduce multi-task optimization as well as develop more
sample-efficient NAS methods to enable co-exploration of more complex HW
search spaces.

##  8 Broader Impact

Our work introduces a co-design method to discover efficient and realizable
neural network and hardware design pairs on a broader class of hardware
accelerators. Our proposed technique integrates a single-stage NAS process
with an RL agent for hardware optimization. This method allows for co-design
with marginally more resource requirements. For instance, training a sub-
sampled architecture from our NN search space would require 160 GPU hours
using the SPOSRetrain240 strategy described in the appendix. Our algorithm,
RHNAS, discovers efficient NN-HW designs in only 187 GPU hours. Considering
the large energy, compute, and carbon costs of previous non-differentiable NN-
HW co-design works, we hope that our work will help drive more efficient NN-HW
architecture search research. Our algorithm is task agnostic and can be
adapted to heterogeneous workload optimization by introducing multi-workload
simulation losses as a regularizer. The proposed method is particularly useful
for discovering optimized configurations for ASICs before deployment. We
believe that our work will encourage early co-design of accelerators in the
development stages, allowing developers to make wiser decisions about the
capabilities, data-flow styles, and hardware resource assignments their
accelerators should have. Our modest compute requirements should allow for the
deployment of accelerators in various stages of development as well as
frequent prototyping of the design space.

## References

  * [1] Liu, H., K. Simonyan, Y. Yang.  Darts: Differentiable architecture search, 2019.
  * [2] Hu, S., S. Xie, H. Zheng, et al.  Dsnas: Direct neural architecture search without parameter retraining.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12081–12089, 2020.
  * [3] Cai, H., L. Zhu, S. Han.  Proxylessnas: Direct neural architecture search on target task and hardware, 2019.
  * [4] Kao, S., T. Krishna.  Gamma: Automating the hw mapping of dnn models on accelerators via genetic algorithm.  _2020 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_ , pages 1–9, 2020.
  * [5] Kao, S., G. Jeong, T. Krishna.  Confuciux: Autonomous hardware resource assignment for dnn accelerators using reinforcement learning.  _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_ , pages 622–636, 2020.
  * [6] Lin, Y., D. Hafdi.  Neural-hardware architecture search.  2019\.
  * [7] Abdelfattah, M., L. Dudziak, T. Chau, et al.  Best of both worlds: Automl codesign of a cnn and its hardware accelerator.  _2020 57th ACM/IEEE Design Automation Conference (DAC)_ , pages 1–6, 2020.
  * [8] Jiang, W., X. Zhang, E. H. M. Sha, et al.  Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search, 2019.
  * [9] Yang, L., Z. Yan, M. Li, et al.  Co-exploration of neural architectures and heterogeneous asic accelerator designs targeting multiple tasks, 2020.
  * [10] Zhang, Y., Y. Fu, W. Jiang, et al.  Dna: Differentiable network-accelerator co-search.  _ArXiv_ , abs/2010.14778, 2020.
  * [11] Choi, K., D.-K. Hong, H. Yoon, et al.  Dance: Differentiable accelerator/network co-exploration.  _ArXiv_ , abs/2009.06237, 2020.
  * [12] Dhar, P.  The carbon impact of artificial intelligence.  _Nature Machine Intelligence_ , 2(8):423–425, 2020.
  * [13] Schulman, J., F. Wolski, P. Dhariwal, et al.  Proximal policy optimization algorithms, 2017.
  * [14] Mnih, V., A. P. Badia, M. Mirza, et al.  Asynchronous methods for deep reinforcement learning, 2016.
  * [15] Mnih, V., K. Kavukcuoglu, D. Silver, et al.  Playing atari with deep reinforcement learning, 2013.
  * [16] Howard, A., M. Sandler, G. Chu, et al.  Searching for mobilenetv3, 2019.
  * [17] Tan, M., B. Chen, R. Pang, et al.  Mnasnet: Platform-aware neural architecture search for mobile, 2019.
  * [18] Tan, M., Q. V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
  * [19] Liu, H., K. Simonyan, Y. Yang.  Darts: Differentiable architecture search.  _ArXiv_ , abs/1806.09055, 2019.
  * [20] Wu, B., X. Dai, P. Zhang, et al.  Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10726–10734, 2019.
  * [21] Xie, S., H. Zheng, C. Liu, et al.  Snas: Stochastic neural architecture search, 2020.
  * [22] Xu, Y., L. Xie, X. Zhang, et al.  Latency-aware differentiable neural architecture search.  _ArXiv_ , abs/2001.06392, 2020.
  * [23] Zhang, L., Y. Yang, Y. Jiang, et al.  Fast hardware-aware neural architecture search.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 2959–2967, 2020.
  * [24] Han, S., X. Liu, H. Mao, et al.  Eie: Efficient inference engine on compressed deep neural network, 2016\.
  * [25] Umuroglu, Y., N. J. Fraser, G. Gambardella, et al.  Finn.  _Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_ , 2017.
  * [26] Umuroglu, Y., Y. Akhauri, N. J. Fraser, et al.  Logicnets: Co-designed neural networks and circuits for extreme-throughput applications, 2020.
  * [27] Chen, Y.-H., T.-J. Yang, J. Emer, et al.  Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices, 2019.
  * [28] Gao, M., J. Pu, X. Yang, et al.  Tetris: Scalable and efficient neural network acceleration with 3d memory.  _SIGARCH Comput. Archit. News_ , 45(1):751–764, 2017.
  * [29] Chen, T., Z. Du, N. Sun, et al.  Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning.  _SIGARCH Comput. Archit. News_ , 42(1):269–284, 2014.
  * [30] Chen, Y.-H., T.-J. Yang, J. S. Emer, et al.  Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices.  _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , 9:292–308, 2019.
  * [31] Chatarasi, P., H. Kwon, N. Raina, et al.  Marvel: A data-centric compiler for dnn operators on spatial accelerators, 2020.
  * [32] Lu, Q., W. Jiang, X. Xu, et al.  On neural architecture search for resource-constrained hardware platforms.  _ArXiv_ , abs/1911.00105, 2019.
  * [33] Moreau, T., T. Chen, L. Vega, et al.  A hardware-software blueprint for flexible deep learning specialization, 2019.
  * [34] Chen, T., T. Moreau, Z. Jiang, et al.  Tvm: An automated end-to-end optimizing compiler for deep learning, 2018\.
  * [35] Bachrach, J., H. Vo, B. Richards, et al.  Chisel: constructing hardware in a scala embedded language.  In _DAC Design Automation Conference 2012_ , pages 1212–1221. IEEE, 2012.
  * [36] Chen, T., L. Zheng, E. Yan, et al.  Learning to optimize tensor programs.  _Advances in Neural Information Processing Systems_ , 31:3389–3400, 2018.
  * [37] Pedregosa, F., G. Varoquaux, A. Gramfort, et al.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011.
  * [38] Kendall, M. G.  A new measure of rank correlation.  _Biometrika_ , 30(1/2):81–93, 1938.
  * [39] Raffin, A., A. Hill, M. Ernestus, et al.  Stable baselines3.  https://github.com/DLR-RM/stable-baselines3, 2019.
  * [40] Deng, J., W. Dong, R. Socher, et al.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR09_. 2009.
  * [41] Krizhevsky, A.  Learning multiple layers of features from tiny images.  2009\.
  * [42] Paszke, A., S. Gross, S. Chintala, et al.  torchvision.models https://pytorch.org/vision/stable/models.html.
  * [43] Kao, S.-C., G. Jeong, T. Krishna.  Confuciux: Autonomous hardware resource assignment for dnn accelerators using reinforcement learning, 2020.
  * [44] Bergstra, J., D. Yamins, D. D. Cox.  Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures.  In _Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28_ , ICML’13, page I–115–I–123. JMLR.org, 2013.
  * [45] Bergstra, J., R. Bardenet, Y. Bengio, et al.  Algorithms for hyper-parameter optimization.  In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, K. Q. Weinberger, eds., _Advances in Neural Information Processing Systems_ , vol. 24, pages 2546–2554. Curran Associates, Inc., 2011.
  * [46] Akiba, T., S. Sano, T. Yanase, et al.  Optuna: A next-generation hyperparameter optimization framework, 2019\.
  * [47] Bohr, M.  22ffl technology.  https://simplecore.intel.com/newsroom/wp-content/uploads/sites/11/2017/04/2017_TMD_MarkBohr_22FFL.pdf.
  * [48] Pedram, A., S. Richardson, M. Horowitz, et al.  Dark memory and accelerator-rich system optimization in the dark silicon era.  _IEEE Design and Test_ , 34(2):39–50, 2017.
  * [49] Kingma, D. P., J. Ba.  Adam: A method for stochastic optimization, 2017.

##  Appendix A Appendix

###  A.1 RL Agent Details

Agent details For the policy and value networks for PPO and A2C, and Q-network
for the DQN, we use MLPs with two hidden layers with sizes [64, 64].We fixed
the number of training episodes to 300000 and tuned the agents learning rate,
discount factor ($\gamma$) and number of steps per update across a wide range
of settings.111We run all our tests in this paper on an NVIDIA DGX-2™ server
with 4 NVIDIA V-100 32GB GPUs.

State representation details:

To represent the HW embeddings, we use a flattened one-hot vector of size 36
for our HW accelerator design as there are 7 HW features we control, with two
features having 2 choices, two with 4 and three with 8 (Listed in Table 1 in
the paper).

As for the neural network state, we use a concatenated representation of the
one-hot choices for each layer. For ImageNet, we use a backbone with 20 blocks
having 4 choices each - giving us a vector of length 80. For CIFAR-10 we use a
smaller network with 9 blocks - giving us a vector of length 36.

We utilize the same representations for the performance predictor - which we
use in our reward formulation. We found experimentally that normalizing the
rewards lead to agents learning faster - we used the VecNormalize wrapper from
[39]

Hyperparameter optimization for Reinforcement Learning

In order to choose the right hyperparameters, we required significant manual
tuning and domain knowledge. To automate our flow and achieve further
improvement, we set up hyper-parameter tuning using Optuna [44, 45, 46]. Here,
we focused on jointly optimizing the environment parameters ( $P$, $B$,
$t_{max}$) and a few algorithm specific parameters - (learning rate, discount
factor $\gamma$, number of steps per update) for all three algorithms and the
number of environment steps per update for PPO and A2C. We utilized a Tree
Parzen estimator based sampler from Optuna [44, 45, 46] and optimized each
algorithm using 40 trials. We optimize agents based on how close they came to
the optimal solutions on fixed test set of neural network architecture
samples.

Hyper-parameter optimization significantly reduced wall time ($\approx$3x
less) required to tune algorithms and enabled discovery of trained agents that
achieved within 0.2 $\%$ of the optimal cycle count and 2 $\%$ of the optimal
EDP values on average.

We utilized the hyper-parameters discovered for the ImageNet RL task, and used
the same settings for training our CIFAR-10 RL optimizer. We found that we
were able to achieve optimality rate of 99.2$\%$ (latency) on NNs sampled for
the CIFAR-10 task.

###  A.2 Optimality

In order to benchmark our RL algorithms, we compare the performance of
hardware designs found by our RL solution to the optimal hardware design found
through grid search using a metric Optimality $(O)$.

We measure Optimality by computing the ratio of the HW performance
(Latency/EDP) of the best performing hardware on a neural network and the
performance of the design suggested by the RL HW optimizer on a test set of NN
architectures $(T)$. Formally, with $H_{v}$ denoting the set of all
synthesisable HW designs

$O=\frac{100}{\left\|T\right\|}\sum_{Z_{\alpha}\in\text{T}}\frac{\operatorname*{min}_{h\in
H_{v}}\text{PerfPredictor}(Z_{\alpha},h)}{\text{PerfPredictor}(\sigma(\alpha),\text{HWOpt}(Z_{\alpha},H_{0}))}$

Note that we only use grid search on the HW space for evaluation and for HPO
(on a small subset of the NN space) Thus, our approach achieves near optimal
performance without incurring the significant cost of grid search over the
whole NN space.

###  A.3 Latency and EDP calculations on VTA

We target the Intel 22FFL process [47] with a 1 GHz frequency for our ASIC
designs. Cycle accurate RTL simulation is used to calculate latency. For
energy calculation, we assume that off-chip memory transfers take 320 pJ/Byte
[48]. For ASIC power analysis, we ran a large VTA design synthesis for 2
million cycles at 1 GHz on our ASIC. This required 0.52W on the VTA core and
5W of power for DRAM memory transactions. Given that over 90% of the overall
power is used for DRAM access, we can leverage the data transfer count from
RTL simulation (for a given neural network workload running on a particular
VTA instance) and approximate energy as the product of bytes transferred and
320 pJ.

###  A.4 Predictor Network Training

We use L1 Loss for training the networks for 80 epochs with the Adam optimizer
[49] with a learning rate of 0.001 and a step decay by a factor of 0.1 at
epochs 40 and 60.

###  A.5 Kendall Tau Metric

The Kendall Tau metric ranges from -1 to 1. If $\tau=1$, the ranking is fully
preserved, if $\tau=-1$, the ranking is reversed. We measure the Kendall Tau
metric by comparing the true ranking of the cycle count (as an example metric)
from the test data-set for a given set of NN - HW accelerator pairs, and the
ranking produced by the cycle count predictor network for the same set of NN -
HW accelerator pairs.

###  A.6 ImageNet search space

Supernet architecture

Table 11: Supernet architecture. CB - choice block. GAP - global average pooling. FC - fully connected layer. Each line describes a sequence of 1 or more identical layers, repeated Repeat times. All layers in the same sequence have the same number of output channels. The first layer of each sequence has a stride Stride and all others use stride 1. Input | Block | Channels | Repeat | Stride
---|---|---|---|---
$224^{2}\times 3$ |  $3\times 3$ Conv | 16 | 1 | 2
$112^{2}\times 16$ | CB | 64 | 4 | 2
$56^{2}\times 64$ | CB | 160 | 4 | 2
$28^{2}\times 160$ | CB | 320 | 8 | 2
$14^{2}\times 320$ | CB | 640 | 4 | 2
$7^{2}\times 640$ |  $1\times 1$ Conv | 1024 | 1 | 1
$7^{2}\times 1024$ | GAP | - | 1 | -
1024 | FC | 1000 | 1 | -

###  A.7 CIFAR10 search space

Supernet architecture

Table 12: Supernet architecture. CB - choice block. GAP - global average pooling. FC - fully connected layer. Each line describes a sequence of 1 or more identical layers, repeated Repeat times. All layers in the same sequence have the same number of output channels. The first layer of each sequence has a stride Stride and all others use stride 1. Input | Block | Channels | Repeat | Stride
---|---|---|---|---
$32^{2}\times 3$ |  $3\times 3$ Conv | 64 | 1 | 1
$32^{2}\times 64$ | CB | 256 | 4 | 2
$16^{2}\times 256$ | CB | 640 | 4 | 2
$8^{2}\times 640$ | CB | 1280 | 1 | 2
$4^{2}\times 1280$ | GAP | - | 1 | -
1280 | FC | 10 | 1 | -

###  A.8 Structures of choice blocks for ImageNet and CIFAR-10

222We follow the setting including choice blocks used in the released
implementation of DSNAS[2]

(a)  Choice blocks with stride=1. Choice blocks in search space. From left to
right: Choice_3, Choice_5, Choice_7, Choice_x. (b)  Choice blocks with
stride=2. Choice blocks in search space. From left to right: Choice_3,
Choice_5, Choice_7, Choice_x.

Figure 4: Choice blocks in our search space (Figure from [2])

###  A.9 CIFAR-10 experimental details

We first do 80 epochs of NN architecture search. The NN architecture is then
trained from scratch using the SPOSretrain240 strategy. We found that this
gave better accuracy for CIFAR-10. Thus, the total number of epochs required
for CIFAR-10 search and train is 320.
SPOSretrain240
Epochs: 240
Batch size: 1024
Input data has been augmented with random crops, random horizontal flops.
Torch Transforms Normalize: transforms.Normalize((0.4914, 0.4822, 0.4465),
(0.2023, 0.1994, 0.2010))
Categorical Cross Entropy
SGD: Momentum = 0.9, weight decay=0.00004
lr_mode : cosine
base_lr: 0.34
targetlr: 0.0
power: 1
warmup_mode: linear
warmup_lr: 0.0001
warmup_epochs: 5

###  A.10 ValidNet MLP

This MLP has two hidden layers with ReLU activation function and a SoftMax
output to indicate whether an input HW accelerator design is valid. The data-
set is generated by randomly sampling design choices and their output from our
analytical validity checker. Since there are very sparsely distributed valid
HW design choices, we train the Validity MLP network with a cross entropy loss
and a weighted loss vector $[0.05,0.95]$ for valid and invalid designs
respectively. The network is trained for 150 epochs and samples 10240 design
points. This validity checker MLP delivered an accuracy of $>99\%$.

###  A.11 Performance Predictor Interpolation test.

We utilize the $\sigma(\alpha)$ NN categorical distribution in RHNAS. Thus we
need to ensure that our performance predictor is able to take in probability
distribution of neural network architectural choices and estimate the expected
performance.

To do this, we conduct a test where we compute the ratio between the
performance predictor output for a continuous NN categorical distribution to
the expected value for this output (Equation 4). We obtain this expected value
by summing over the predicted metric from discrete architectures sampled from
the categorical distribution obtained from $\sigma(\alpha)$ multiplied by
their likelihood of being sampled. We found that the ratio is $0.9910$ with a
variance of $0.0012$ after sampling over 400 NN distributions. This indicates
that the performance predictor is able to accurately interpolate performance
between NN architectural choices.

$\displaystyle\begin{split}&\frac{\text{PerfPredictor}(\sigma(\alpha),H_{0})}{\displaystyle\mathop{\mathbb{E}}_{Z_{\alpha}\sim\sigma(\alpha)}(\text{PerfPredictor}(Z_{\alpha},H_{0}))}\vspace{-0.1cm}\end{split}$
(4)