# Cold Seeded Epitaxy and Flexomagnetism in Smooth GdAuGe Membranes Exfoliated
from graphene/Ge(111)

Zachary LaDuca  Tamalika Samanta  Nicholas Hagopian  Taehwan Jung  Katherine
Su  Materials Science and Engineering, University of Wisconsin-Madison,
Madison, WI 53706, United States of America  Konrad Genser  Karin M. Rabe
Physics and Astronomy, Rutgers University, NJ 08854, United States of America
Paul M. Voyles  Michael S. Arnold  Jason K. Kawasaki<EMAIL_ADDRESS>Materials Science and Engineering, University of Wisconsin-Madison, Madison,
WI 53706, United States of America

###### Abstract

Remote and van der Waals epitaxy are promising approaches for synthesizing
single crystalline membranes for flexible electronics and discovery of new
properties via extreme strain; however, a fundamental challenge is that most
materials do not wet the graphene surface. We develop a cold seed approach for
synthesizing smooth intermetallic films on graphene that can be exfoliated to
form few nanometer thick single crystalline membranes. Our seeded GdAuGe films
have narrow x-ray rocking curve widths of 9-24 arc seconds, which is two
orders of magnitude lower than their counterparts grown by typical high
temperature methods, and have atomically sharp interfaces observed by
transmission electron microscopy. Upon exfoliation and rippling, strain
gradients in GdAuGe membranes induce an antiferromagnetic to
ferri/ferromagnetic transition. Our smooth, ultrathin membranes provide a
clean platform for discovering new flexomagnetic effects in quantum materials.

keywords: graphene, epitaxy, membrane, Heusler, strain

Remote 1, 2 and van der Waals epitaxy 3, 4, 5, 6 on graphene-covered surfaces
are promising strategies for synthesizing single crystalline membranes. Here,
the weak van der Waals interactions between the film and graphene enable etch-
free membrane exfoliation. These membranes enable applications for flexible
electronics 1 and discovery of new properties induced by extreme strain 7, 8,
9, 10. For example, flexomagnetism is magnetic ordering induced by strain
gradients 11. While there has been some success in producing strain gradients
in epitaxial thin films via partial relaxation 12, 13, greater opportunities
for control exist for bending of ultrathin single crystalline membranes 14.
This includes experimental demonstration of an antiferromagnetic to
ferromagnetic transition in rippled GdPtSb membranes 9, predicted cycloidal
spin textures induced by bending in CrI3 15, and predicted bending-induced
skyrmions via control of Dzyaloshinskii-Moriya Interaction (DMI) in CrBr2 and
MnSe2 16, since strain gradients and bending break inversion symmetry.

There are two significant challenges for epitaxy on graphene. First, since
graphene has very low surface energy, most materials do not wet the graphene
surface and instead grow with a disconnected island morphology. This includes
GaN islands on graphene/SiC 17, oxide islands on graphene/SrTiO3 18, III-V
islands 19, 20, 21 or nanowires 22 on graphene, and metal islands on graphene
23, 24. The wetting challenge limits the ability to access the largest strains
that are only possible in the ultrathin limit: whereas $8\%$ strain has been
reported for few unit cell thick (La,Ca)MnO3 (LCMO) membranes, LCMO with
thickness greater than 20 nm could only undergo $2\%$ strain before cracking
7. Second, the sticking coefficients on graphene can be highly temperature and
element dependent, which poses challenges for controlling the stoichiometry of
alloys and compounds that are not adsorption-controlled 24.

Here we demonstrate a cold seeded epitaxy approach to solve the dewetting and
sticking coefficient challenges. Using molecular beam epitaxy (MBE) of GdAuGe
on graphene/Ge and graphene/SiC, we demonstrate few nanometer thick films on
graphene with a smooth step and terrace morphology and a 9-24 arc sec x-ray
rocking curve width that is two orders of magnitude sharper than their
counterparts grown at high temperature. The cold seed suppresses surface
diffusion during nucleation, while subsequent annealing and continued growth
at elevated temperatures recrystallizes the seed. The cold seed also provides
a template with a well defined stoichiometry, since sticking coefficients on
graphene at room temperature are much closer to unity. The GdAuGe films can be
exfoliated to produce continuous membranes with minimal tears that are
supported on a polymer handle. We demonstrate that strain gradients in rippled
GdAuGe membranes induce an antiferromagnetic to ferromagnetic transition,
similar to that observed for GdPtSb membranes 9. We anticipate that this
strategy is generally applicable to epitaxy of other thin film systems on
graphene.

Figure 1 illustrates the key steps in our seeded growth approach. We start
from graphene that is grown by chemical vapor deposition on Ge (111),
following Ref 25. After annealing the graphene in ultrahigh vacuum ($<2\times
10^{-10}$ Torr) at $600\degree$ C for one hour, the reflection high energy
electron diffraction (RHEED) pattern along the Ge $[1\bar{1}0]$ azimuth
exhibits a sharp ring of spots (Fig. 1a), indicating a nearly atomically
smooth and clean surface. Previous low energy electron diffraction studies
show that epitaxial graphene on Ge (111) has two domain orientations that are
rotated 30 degrees from one another: a primary domain with graphene
$[1\bar{1}00]\parallel$ Ge $[1\bar{1}0]$ and a secondary domain with graphene
$[1\bar{2}10]\parallel$ Ge $[1\bar{1}0]$ 25.

To suppress the dewetting and island formation that are typical for high
temperature growth of metals on graphene surfaces 24, we initiate growth of a
5 nm thick GdAuGe seed at room temperature ($30\degree$C). Here, equiatomic
fluxes of Gd, Au, and Ge were supplied by thermal effusion cells at rates of
$1.2\times 10^{13}$ atoms/cm2s, respectively, as measured by quartz crystal
microbalance and calibrated by Rutherford backscattering spectrometry (RBS) on
test samples. The corresponding RHEED pattern indicates an epitaxial seed with
a semi-rough surface: the RHEED spots are connected by extended streaks with
well-defined lattice spacings (Fig. 1b). Subsequent annealing of the seed
layer at $480\degree$C recrystallizes it and smooths the surface, as evidenced
by the sharp and streaky RHEED pattern in Fig. 1(c).

The smooth surface of the annealed seed layer serves as a template for
continued GdAuGe growth at elevated temperatures. Fig. 1(d) shows the RHEED
pattern of a GdAuGe film regrown at $480\degree$ C to a total thickness of 16
nm GdAuGe. The bright, sharp, and extended streaks indicate a flat epitaxial
film. The final GdAuGe surface exhibits two sets of streaks: a primary set
with with spacing $\Delta Q$ (arrows), and a secondary set with spacing
$\sqrt{3}\cdot\Delta Q$ (asterisks), indicative of two GdAuGe domains that are
rotated 30 degrees from one another. An azimuthal $\phi$ scan by x-ray
diffraction confirms two GdAuGe orientations rotated around the $c$ axis by 30
degrees (Fig. 1(e)).

Figure 1: Cold seeded epitaxy of GdAuGe on graphene. (a) RHEED pattern of
graphene on Ge (111), after annealing at $600\degree$C to remove surface
adsorbates. All RHEED patterns are recorded along a Ge $\langle 110\rangle$
zone axis at a beam energy of 15 kV. (b) 5 nm thick seed of GdAuGe grown at
$30\degree$C. (c) Seed after recrystallizing by annealing at $480\degree$C.
(d) Continued growth of GdAuGe to a total thickness of 18 nm. Arrows mark the
primary GdAuGe and graphene domain reflections. Stars mark the 30 degree
rotated orientation. (e) X-ray diffraction azimuthal $\phi$ scan showing the
in-plane epitaxial alignment. (f) Crystal structures of GdAuGe, graphene, and
Ge showing the primary epitaxial alignment. For graphene, the primitive unit
cell is marked by dotted lines and a $(\sqrt{3}\times\sqrt{3})R30\degree$
supercell is marked by solid lines.

An interesting question is whether GdAuGe films on graphene/Ge are epitaxial
to the graphene or to the Ge substrate, e.g., via “remote” substrate
interactions that permeate through graphene 1. Here we find a direct
correspondence between two graphene orientations rotated by 30 degrees, and
two GdAuGe orientations rotated by 30 degrees. This implies that the GdAuGe is
epitaxial to graphene, since our control experiment of GdAuGe epitaxy directly
on Ge (111) produces a single orientation of GdAuGe. The lattice mismatch of
$2.3\%$ between GdAuGe and a $(\sqrt{3}\times\sqrt{3})R30\degree$ supercell of
graphene is smaller than the $9.7\%$ mismatch between GdAuGe and Ge (111)
(Fig. 1(f)), which also suggests a preferred epitaxial relationship to
graphene. As an additional test, we grew GdAuGe on graphene/Ge(110), where the
Ge (110) surface has rectangular symmetry compared to the hexagonal symmetry
of GdAuGe. We find that GdAuGe again grows (0001)-oriented, with the same in-
plane orientation with respect to graphene: GdAuGe
$[2\bar{1}\bar{1}0]\parallel$ graphene $[1\bar{1}00]$ (Supplemental Figure 1).

Figure 2: Comparison of low temperature seeded growth versus direct high
temperature growth of GdAuGe on graphene/Ge (111). (a) X-ray diffraction scan
(Cu $K\alpha$) of a film grown using the low temperature seed approach (blue)
versus a film grown directly at high temperature (black). Insert shows the
$0004$ reflection. (b,c) Rocking curves of $0004$ reflection for the the
seeded and unseeded samples. (d,e) Atomic force microscopy showing a step and
terrace morphology for the seeded sample and an island morphology for the no
seed sample. Inserts show the RHEED patterns at the end of GdAuGe growth.

Figure 2 compares the structure and morphology of a 16 nm GdAuGe film grown
with a cold seeded growth approach on graphene/Ge(111) to that of an 18 nm
film grown directly on graphene/Ge(111) at $540\degree$C. For the seeded
sample, the $2\theta-\omega$ x-ray diffraction scan displays bright GdAuGe
$000l$ reflections, including the higher order $0006$ and $0008$ Bragg
reflections. The sharp Kiessig fringes around the GdAuGe $0004$ reflection
(insert) indicate a smooth surface and interface morphology (Fig. 2(a), blue
trace). These x-ray fringes are consistent with the narrow x-ray rocking curve
width of 24 arc seconds that indicates a low mosaicity (Fig. 2(b)). This
rocking curve width is significantly narrower than the 245 arc sec reported
for GaAs on graphene/SiO2 19, 300 arc sec for GaN on graphene/GaN 26, 115 arc
sec for SrTiO3 on graphene/SrTiO3 27, 180 arc sec for VO2 on graphene 28, 770
arc sec for ZnO on graphene 29, and 1.2 degrees (4320 arc sec) for BaTiO3 on
graphene/Ge (110) 30. We observe a flat atomic step and terrace morphology by
atomic force microscopy (AFM, Fig. 2(d)). In the AFM scan we also observe an
arc of circular openings with depth of a few nanometers into the sample. We
attribute these depressions to defects induced by a graphene grain boundary or
wrinkle.

In contrast, the sample without seed layer displays weaker $000l$ reflections,
indicative of poorer crystallinity, no Kiessig fringes, and XRD reflections
from secondary phases (diamond symbols). We attribute the secondary phases to
the fact that the sticking coefficients of metals on graphene deviate
significantly from unity at elevated temperatures, as observed previously for
other metals on graphene 24, and confirmed for GdAuGe on graphene by RBS
(Supplemental Fig. 2). AFM of the non seeded sample displays a disconnected
island morphology (Fig. 2e), consistent with large rocking curve width of 1
degree that indicates large mosaicity (Fig. 2c).

Cross-sectional high angular annular dark field (HAADF) STEM imaging of the
seeded sample reveals a well ordered GdAuGe /graphene/Ge interface (Figure 3).
The GdAuGe /graphene/Ge interface appears atomically sharp, terminated with an
Au/Ge plane above the graphene interface. Due to Z-contrast image intensity,
heavier elements such as Gd and Au are bright and easily visible whereas
lighter carbon in the graphene layer exhibits a relatively weak signal. Just
below the graphene, the top layer of the Ge substrate displays brighter
contrast than the bulk Ge substrate, indicating possible Gd intercalation. Gd
intercalation under graphene has been reported previously, although at
slightly higher temperatures of $800\degree$C 31, compared to the $30\degree$
C deposition and $480\degree$C anneal and regrowth we employ here. Finally, we
observe that the film is a continuous single crystal between the annealed seed
layer and continued growth layer, with no obvious defects induced by the
temperature cycling. Our combined diffraction (XRD and RHEED), surface
topography (AFM), and electron microscopy support the enhanced structural
quality and smoothness for films grown by our low temperature seed approach
compared to direct growth at elevated temperature.

Figure 3: High-angle annular dark-field scanning transmission electron
microscopy (HAADF-STEM) of GdAuGe/graphene/Ge(111). (a) Wide field of view
showing a highly ordered film with no boundary between seed and continued
growth. (b) Zoom in showing a sharp GdAuGe/graphene/Ge interface.

Our low temperature seed layer approach appears generalizable to other
graphene-covered substrates. As shown in Supplementary Fig. 3, seeded GdAuGe
growth on graphene/SiC (0001) produces films with narrower rocking curve width
($\Delta\omega=9$ arc sec), smoother AFM morphology, and sharper Kiessig
fringes compared to films grown directly at high temperature. Additionally,
the seeded samples show the expected metallic transport, compared to the
hopping transport observed for high temperature growth due to the disconnected
island morphology (Supplementary Fig. 4).

Figure 4: Strain gradient induced ferri/ferromagnetism in a rippled GdAuGe
membrane. (a) SQUID magnetometry at 2K with in plane field. Insert: AFM
topography of the rippled membrane. (b) DFT calculation of the difference in
total energy for ferromagnetic versus A-type antiferromagnetically ordered
GdAuGe with moments oriented along the crystallographic $c$ axis.

Our smooth GdAuGe films grown using the seed approach provide an excellent
starting point for membrane exfoliation and manipulation of magnetic
properties via extreme strain. To produce membranes with large strain
gradients, we adhere a polyurethane handle under tensile strain to the GdAuGe
film and exfoliate by cooling the sample in liquid nitrogen. Upon exfoliation,
the polyurethane contracts to induce ripples in the GdAuGe membrane. For
GdAuGe films grown with a seed layer, this exfoliation produces continuous
membranes with well ordered ripples over length scales of several hundred of
microns (Fig. 4(a) insert and Supplemental Fig. 5). In contrast, exfoliation
of samples grown directly at high temperature on graphene/Ge show a rough and
disconnected morphology with no ripples, suggesting that the polymer
contraction does not produce a well controlled strain pattern in the unseeded
samples.

Fig. 4 compares SQUID magnetometry of a rippled GdAuGe membrane grown using
the seed layer, compared with a relaxed GdAuGe film. At the measurement
temperature of 2 K, the film shows a weak linear dependence of magnetization
on applied field, consistent with the antiferromagnetism observed in bulk
crystals 32, 33, 34. In contrast, the rippled sample displays a clear
hysteresis loop indicative of ferro or ferrimagnetic ordering. The behavior is
similar to GdPtSb membranes for which we also observe bending-induced
ferri/ferromagnetism 9, but with a factor of 5 smaller magnetic saturation
than GdPtSb at similar bend radius.

An important distinction is whether the magnetic ordering is induced by
homogeneous strain (piezomagnetism and magnetotostriction) or by strain
gradients (flexomagnetism). For a smoothly curved membrane with no plastic
deformation, the strain along the curved path $s$ is approximately
$\varepsilon_{s}\approx(R+t)/R-1$, where $t$ is the height or depth along the
thickness axis and $R$ is the radius of curvature 9. Based on the measured
thickness of 20 nm, period of 12 $\mu$m, and peak to peak height of 2.6
$\mu$m, we estimate a maximum strain of $\varepsilon_{s}\approx\pm 0.36\%$ and
maximum strain gradient of $\partial\varepsilon_{s}/\partial z\approx\pm
36\%/\mu m$ at the peaks and valleys. Compared with our density functional
theory calculations of antiferromagnetic vs ferromagnetic ordering (Fig 4(b)
and Supplemental Figure 6), we find that the peak strain in our rippled
membrane is an order of magnitude smaller than the calculated homogeneous
strain to induce ferromagnetic ordering. This suggests that coupling to strain
gradients is more likely responsible for the observed ferro/ferrimagnetic
ordering than coupling to homogeneous strain, similar to GdPtSb 9. We note,
however, that the antiferromagnetic ordering of GdAuGe has not yet been solved
32, and in our calculation we assume $A$-type ordering with moments pointed
along $c$. The particular choice of AFM structure will change the expected
strain induced crossover between AFM and FM ordering. Further experiments and
theory are required to solve the magnetic structure of GdAuGe and understand
its response to strain and strain gradients.

In conclusion, we demonstrated a cold seeded epitaxy approach for growing few
nanometer thick, smooth GdAuGe films on graphene-covered Ge and SiC
substrates. Seeded films display sharper Bragg reflections, lower mosaicity,
and a smoother atomic step and terrace morphology than their counterparts
grown directly at high temperature. These films are epitaxial to graphene
rather than the underlying Ge or SiC substrate. The interfaces with graphene
are atomically sharp. Using a rippled membrane exfoliate from graphene, we
demonstrated an antiferromagnetic to ferri/ferromagnetic transition induced by
strain gradients. This flexomagnetic response is similar to GdPtSb. Our cold
seed strategy provides a path for access to extreme strains in membranes in
the ultrathin limit.

## Supporting Information

Contains the following:

Methods for STEM and XRD.

Fig. 1. Epitaxial orientation of GdAuGe on graphene/Ge (110).

Fig. 2. Composition of GdAuGe films on graphene/Ge by RBS.

Fig. 3. XRD of GdAuGe films on graphene/SiC.

Fig. 4. Electrical transport of GdAuGe films on graphene/SiC.

Fig. 5. Optical microscopy of exfoliated GdAuGe membranes.

Fig. 6. DFT calculated energy difference between FM and AFM configurations for
strain along $a$ and $b$ axes.

## Data Availability

All raw data that appears in this manuscript are posted on the XX repository,
doi number YY.

## Acknowledgments

We thank Chengye Dong and Joshua Robinson at The Pennsylvania State University
Two-Dimensional Crystal Consortium – Materials Innovation Platform (2DCC-MIP)
for graphene on SiC samples. 2DCC-MIP is supported by NSF cooperative
agreement DMR-2039351. We thank Greg Haugstad at the University of Minnesota
Characterization Facility for RBS measurements.

This project was primarily supported by the the Department of Energy Office of
Science (DE-SC00203958, Heusler synthesis by Z.L., T.S., T.J., and J.K.K.).
Magnetic measurements (Z.L., T.S., and J.K.K.) and electron microscopy (N.H.
and P.M.V.) were supported by NSF through the University of Wisconsin
Materials Research Science and Engineering Center (DMR-2309000). We
acknowledge user facilities also supported by the Wisconsin MRSEC. Transport
and preliminary synthesis (Z.L. and J.K.K.) were supported by the Air Force
Office of Scientific Research (FA9550-21-0127). Graphene on germanium
synthesis and characterization (K.S. and M.S.A.) are supported by the U.S.
Department of Energy, Office of Science, Basic Energy Sciences, (DE-SC0016007)
and the NSF Graduate Research Fellowship Program (DGE-1747503). DFT
calculations (K.T.G. and K.M.R.) were supported by the Office of Naval
Research (ONR N00014-21-1-2107) and Wisconsin MRSEC.

## References

  * Kim et al. 2017 Kim, Y.; Cruz, S. S.; Lee, K.; Alawode, B. O.; Choi, C.; Song, Y.; Johnson, J. M.; Heidelberger, C.; Kong, W.; Choi, S.; others Remote epitaxy through graphene enables two-dimensional material-based layer transfer. _Nature_ 2017, _544_ , 340–343
  * Kong et al. 2018 Kong, W.; Li, H.; Qiao, K.; Kim, Y.; Lee, K.; Nie, Y.; Lee, D.; Osadchy, T.; Molnar, R. J.; Gaskill, D. K.; others Polarity governs atomic interaction through two-dimensional materials. _Nature materials_ 2018, _17_ , 999–1004
  * Koma 1992 Koma, A. Van der Waals epitaxy—a new epitaxial growth method for a highly lattice-mismatched system. _Thin Solid Films_ 1992, _216_ , 72–76
  * Shi et al. 2012 Shi, Y.; Zhou, W.; Lu, A.-Y.; Fang, W.; Lee, Y.-H.; Hsu, A. L.; Kim, S. M.; Kim, K. K.; Yang, H. Y.; Li, L.-J.; others van der Waals epitaxy of MoS2 layers using graphene as growth templates. _Nano letters_ 2012, _12_ , 2784–2791
  * Kim et al. 2014 Kim, J.; Bayram, C.; Park, H.; Cheng, C.-W.; Dimitrakopoulos, C.; Ott, J. A.; Reuter, K. B.; Bedell, S. W.; Sadana, D. K. Principle of direct van der Waals epitaxy of single-crystalline films on epitaxial graphene. _Nature communications_ 2014, _5_ , 4836
  * Du et al. 2022 Du, D.; Jung, T.; Manzo, S.; LaDuca, Z.; Zheng, X.; Su, K.; Saraswat, V.; McChesney, J.; Arnold, M. S.; Kawasaki, J. K. Controlling the balance between remote, pinhole, and van der Waals epitaxy of Heusler films on graphene/sapphire. _Nano Letters_ 2022, _22_ , 8647–8653
  * Hong et al. 2020 Hong, S. S.; Gu, M.; Verma, M.; Harbola, V.; Wang, B. Y.; Lu, D.; Vailionis, A.; Hikita, Y.; Pentcheva, R.; Rondinelli, J. M.; others Extreme tensile strain states in La0. 7Ca0. 3MnO3 membranes. _Science_ 2020, _368_ , 71–76
  * Harbola et al. 2021 Harbola, V.; Crossley, S.; Hong, S. S.; Lu, D.; Birkholzer, Y. A.; Hikita, Y.; Hwang, H. Y. Strain gradient elasticity in SrTiO3 membranes: bending versus stretching. _Nano letters_ 2021, _21_ , 2470–2475
  * Du et al. 2021 Du, D.; Manzo, S.; Zhang, C.; Saraswat, V.; Genser, K. T.; Rabe, K. M.; Voyles, P. M.; Arnold, M. S.; Kawasaki, J. K. Epitaxy, exfoliation, and strain-induced magnetism in rippled Heusler membranes. _Nature communications_ 2021, _12_ , 1–7
  * Dong et al. 2019 Dong, G.; Li, S.; Yao, M.; Zhou, Z.; Zhang, Y.-Q.; Han, X.; Luo, Z.; Yao, J.; Peng, B.; Hu, Z.; others Super-elastic ferroelectric single-crystal membrane with continuous electric dipole rotation. _Science_ 2019, _366_ , 475–479
  * Lukashev and Sabirianov 2010 Lukashev, P.; Sabirianov, R. F. Flexomagnetic effect in frustrated triangular magnetic structures. _Physical Review B_ 2010, _82_ , 094417
  * Zhang et al. 2021 Zhang, Y.; Liu, J.; Dong, Y.; Wu, S.; Zhang, J.; Wang, J.; Lu, J.; Rückriegel, A.; Wang, H.; Duine, R.; others Strain-driven Dzyaloshinskii-Moriya interaction for room-temperature magnetic skyrmions. _Physical Review Letters_ 2021, _127_ , 117204
  * Makushko et al. 2022 Makushko, P.; Kosub, T.; Pylypovskyi, O. V.; Hedrich, N.; Li, J.; Pashkin, A.; Avdoshenko, S.; Hübner, R.; Ganss, F.; Wolf, D.; others Flexomagnetism and vertically graded Néel temperature of antiferromagnetic Cr2O3 thin films. _Nature Communications_ 2022, _13_ , 6745
  * Du et al. 2023 Du, D.; Hu, J.; Kawasaki, J. K. Strain and strain gradient engineering in membranes of quantum materials. _Applied Physics Letters_ 2023, _122_
  * Edström et al. 2022 Edström, A.; Amoroso, D.; Picozzi, S.; Barone, P.; Stengel, M. Curved magnetism in CrI 3. _Physical review letters_ 2022, _128_ , 177202
  * Ga et al. 2022 Ga, Y.; Cui, Q.; Liang, J.; Yu, D.; Zhu, Y.; Wang, L.; Yang, H. Dzyaloshinskii-Moriya interaction and magnetic skyrmions induced by curvature. _Physical Review B_ 2022, _106_ , 054426
  * Qiao et al. 2021 Qiao, K.; Liu, Y.; Kim, C.; Molnar, R. J.; Osadchy, T.; Li, W.; Sun, X.; Li, H.; Myers-Ward, R. L.; Lee, D.; others Graphene buffer layer on SiC as a release layer for high-quality freestanding semiconductor membranes. _Nano letters_ 2021, _21_ , 4013–4020
  * Chang et al. 2023 Chang, C. S.; Kim, K. S.; Park, B.-I.; Choi, J.; Kim, H.; Jeong, J.; Barone, M.; Parker, N.; Lee, S.; Zhang, X.; others Remote epitaxial interaction through graphene. _Science Advances_ 2023, _9_ , eadj5379
  * Alaskar et al. 2014 Alaskar, Y.; Arafin, S.; Wickramaratne, D.; Zurbuchen, M. A.; He, L.; McKay, J.; Lin, Q.; Goorsky, M. S.; Lake, R. K.; Wang, K. L. Towards van der Waals epitaxial growth of GaAs on Si using a graphene buffer layer. _Advanced Functional Materials_ 2014, _24_ , 6629–6638
  * Zulqurnain et al. 2022 Zulqurnain, M.; Burton, O. J.; Al-Hada, M.; Goff, L. E.; Hofmann, S.; Hirst, L. C. Defect seeded remote epitaxy of GaAs films on graphene. _Nanotechnology_ 2022, _33_ , 485603
  * Manzo et al. 2022 Manzo, S.; Strohbeen, P. J.; Lim, Z. H.; Saraswat, V.; Du, D.; Xu, S.; Pokharel, N.; Mawst, L. J.; Arnold, M. S.; Kawasaki, J. K. Pinhole-seeded lateral epitaxy and exfoliation of GaSb films on graphene-terminated surfaces. _Nature communications_ 2022, _13_ , 1–9
  * Hong et al. 2012 Hong, Y. J.; Lee, W. H.; Wu, Y.; Ruoff, R. S.; Fukui, T. van der Waals epitaxy of InAs nanowires vertically aligned on single-layer graphene. _Nano letters_ 2012, _12_ , 1431–1436
  * Liu et al. 2013 Liu, X.; Wang, C.-Z.; Hupalo, M.; Lin, H.-Q.; Ho, K.-M.; Tringides, M. C. Metals on graphene: interactions, growth morphology, and thermal stability. _Crystals_ 2013, _3_ , 79–111
  * LaDuca et al. 2023 LaDuca, Z.; Su, K.; Manzo, S.; Arnold, M. S.; Kawasaki, J. K. Control of ternary alloy composition during remote epitaxy on graphene. _Physical Review Materials_ 2023, _7_ , 083401
  * Kiraly et al. 2015 Kiraly, B.; Jacobberger, R. M.; Mannix, A. J.; Campbell, G. P.; Bedzyk, M. J.; Arnold, M. S.; Hersam, M. C.; Guisinger, N. P. Electronic and mechanical properties of graphene–germanium interfaces grown by chemical vapor deposition. _Nano letters_ 2015, _15_ , 7414–7420
  * Badokas et al. 2021 Badokas, K.; Kadys, A.; Mickevičius, J.; Ignatjev, I.; Skapas, M.; Stanionytė, S.; Radiunas, E.; Juška, G.; Malinauskas, T. Remote epitaxy of GaN via graphene on GaN/sapphire templates. _Journal of Physics D: Applied Physics_ 2021, _54_ , 205103
  * Yoon et al. 2022 Yoon, H.; Truttmann, T. K.; Liu, F.; Matthews, B. E.; Choo, S.; Su, Q.; Saraswat, V.; Manzo, S.; Arnold, M. S.; Bowden, M. E.; others Free-Standing Epitaxial SrTiO $\\_3$ Nanomembranes via Remote Epitaxy using Hybrid Molecular Beam Epitaxy. _arXiv preprint arXiv:2206.09094_ 2022,
  * Guo et al. 2019 Guo, Y.; Sun, X.; Jiang, J.; Wang, B.; Chen, X.; Yin, X.; Qi, W.; Gao, L.; Zhang, L.; Lu, Z.; others A reconfigurable remotely epitaxial VO2 electrical heterostructure. _Nano letters_ 2019, _20_ , 33–42
  * Wang et al. 2022 Wang, N.; Wang, P.; Wang, F.; He, H.; Huang, J.; Pan, X.; Zhu, G.; Wang, J.; Ye, Z. Improved epitaxy of ZnO films by regulating the layers of graphene. _Applied Surface Science_ 2022, _585_ , 152709
  * Dai et al. 2022 Dai, L.; Zhao, J.; Li, J.; Chen, B.; Zhai, S.; Xue, Z.; Di, Z.; Feng, B.; Sun, Y.; Luo, Y.; others Highly heterogeneous epitaxy of flexoelectric BaTiO3-$\delta$ membrane on Ge. _Nature Communications_ 2022, _13_ , 1–10
  * Link et al. 2019 Link, S.; Forti, S.; Stöhr, A.; Küster, K.; Rösner, M.; Hirschmeier, D.; Chen, C.; Avila, J.; Asensio, M.; Zakharov, A.; others Introducing strong correlation effects into graphene by gadolinium intercalation. _Physical Review B_ 2019, _100_ , 121407
  * Ram et al. 2023 Ram, D.; Singh, J.; Hooda, M.; Singh, K.; Kanchana, V.; Kaczorowski, D.; Hossain, Z. Multiple magnetic transitions, metamagnetism, and large magnetoresistance in GdAuGe single crystals. _Physical Review B_ 2023, _108_ , 235107
  * Kurumaji et al. 2024 Kurumaji, T.; Gen, M.; Kitou, S.; Arima, T.-h. Metamagnetism and anomalous magnetotransport properties in rare-earth-based polar semimetals $R$ AuGe ($R=$ Dy, Ho, and Gd). _arXiv preprint arXiv:2405.00628_ 2024,
  * Gibson et al. 1996 Gibson, B.; Schnelle, W.; Pöttgen, R.; Bartkowski, K.; Kremer, R. Susceptibility, specific heat, and transport properties of CeAuGe and GdAuGe. _Czechoslovak Journal of Physics_ 1996, _46_ , 2573–2574


# Taming Resource Heterogeneity In Distributed ML Training With Dynamic
Batching

Sahil Tyagi Prateek Sharma  _Indiana University Bloomington_
_{ styagi<EMAIL_ADDRESS>
###### Abstract

Current techniques and systems for distributed model training mostly assume
that clusters are comprised of homogeneous servers with a constant resource
availability. However, cluster heterogeneity is pervasive in computing
infrastructure, and is a fundamental characteristic of low-cost transient
resources (such as EC2 spot instances). In this paper, we develop a dynamic
batching technique for distributed data-parallel training that adjusts the
mini-batch sizes on each worker based on its resource availability and
throughput. Our mini-batch controller seeks to equalize iteration times on all
workers, and facilitates training on clusters comprised of servers with
different amounts of CPU and GPU resources. This variable mini-batch technique
uses proportional control and ideas from PID controllers to find stable mini-
batch sizes. Our empirical evaluation shows that dynamic batching can reduce
model training times by more than $4\times$ on heterogeneous clusters.

##  I Introduction

Distributed training of machine learning models by using large clusters of
servers is a popular technique to decrease the model training time. Techniques
and system-architectures for distributed ML training such as Stochastic
Gradient Descent (SGD) and parameter servers are widely used to train in data
centers and cloud platforms to provide reasonable parallel speedup.

However, current techniques and systems for distributed model training mostly
assume that the workers (i.e., the servers) will all have the same performance
and resource configuration, i.e., will be homogeneous. However, virtual
clusters in data centers and especially clouds _do not_ always exhibit this
resource homogeneity. The performance of different workers can be affected due
to performance interference with co-located applications; workers may be
throttled by the cloud or data center provider; or the cluster may have
servers with vastly different resource configurations.

This _resource heterogeneity_ is a key characteristic of cloud-based
applications, and distributed ML model training must be able to tolerate and
perform well even in heterogeneous environments. However, heterogeneity
presents many fundamental challenges to distributed training: synchronous
model updates result in stragglers causing poor parallel efficiency, and
asynchronous updates result in gradient and model staleness causing poor
statistical efficiency [1].

In this paper, we address the challenges of distributed ML training in
heterogeneous environments. Our goal is to make model training “omnivorous”,
and be able to run efficiently on dynamic and heterogeneous cluster
configurations in shared data center and cloud environments. Our key insight
is that having variable, instead of uniform mini-batch sizes on different
workers, is a simple yet powerful technique for alleviating many of the
performance degradation problems in heterogeneous environments.

Our dynamic batch sizing mechanism adjusts the mini-batch size on each worker
based on the worker’s throughput, by using a proportional-controller [2] that
minimizes the differences in the workers’ iteration times. This dynamic batch
sizing technique permits training on clusters made up of servers with vastly
different resource configurations; and on clusters with dynamic resource
availability due to resource elasticity, over-commitment, or preemption. The
technique enables us to train models efficiently on clusters comprising of
servers with different CPUs and GPUs, which is a key differentiator from prior
work in heterogeneous distributed training [3, 4] that instead focuses on
random worker slowdowns. The prior work has shown that even small random
slowdowns can result in the training times increase by an order of magnitude.
This is only exacerbated with the systemic heterogeneity that we aim to
alleviate.

The dynamic batching mechanism is able to reduce stragglers in Bulk
Synchronous Parallel (BSP) training, and is designed as zero-configuration,
black-box approach that can effectively work with different training, model,
and resource configurations. Our approach allows distributed training on
clusters with dynamic resource availability that are ubiquitous in cloud
environments. By mitigating the performance degradation due to heterogeneity,
our contributions enable low-cost training on heterogeneous collections of
transient cloud servers such as EC2 spot instances [5] and Google Preemptible
VMs [6], that are up to $10\times$ cheaper than conventional cloud servers. We
implement our dynamic batching mechanism and policies in TensorFlow, and make
the following contributions:

  1. 1.

We develop a dynamic batching mechanism for data-parallel training that
continuously balances the load between workers by assigning them different
sized mini-batches based on their throughput. Our proportional-control based
technique reduces stragglers in BSP, and allows the mixing of CPU and GPU
servers. It is able to ameliorate both static as well as dynamic
heterogeneity.

  2. 2.

We implement all our mechanisms and policies in TensorFlow using the estimator
API, which allows most models to directly run in heterogeneous environments
without any modifications.

  3. 3.

We conduct a large scale study of training performance in various static and
dynamic heterogeneity environments using popular ML workloads. Our techniques
can reduce the training times by as much as $4\times$ compared to existing
uniform-batching.

##  II Background & Motivation

###  II-A Heterogeneity in Data Centers and Clouds

Resource heterogeneity is pervasive in modern data centers and clouds. In
cloud environments, applications are often deployed on clusters composed of
servers (i.e, VMs) of different resource capacities and sizes. This _static
heterogeneity_ is _necessary_ for effectively using low-cost transient servers
such as Amazon EC2 spot instances [5], Google Preemptible VMs [6], etc.

Since distributed model training is highly computationally intensive, using
low-cost transient VMs or low-priority data center resources is a key
technique for reducing training costs [7, 8]. Transient VMs can be
unilaterally preempted by the cloud provider, which are akin to fail-stop
failures. Distributed applications that can tolerate a failure of a (small)
subset of their servers failing can benefit greatly from running on VMs of
different sizes. Past work on transient computing [9, 10] has found that
transient VMs of different sizes are usually _uncorrelated_ in their
preemptions, and this diversification significantly reduces the risk of all
the VMs preempted at the same time. Thus distributed training needs to be
“omnivorous”, capable of using different types of low-cost low-priority
servers and cannot assume homogeneous clusters with constant resource
availability.

###  II-B Distributed Training

Training of machine learning models entails learning the model parameters
(a.k.a weights) of a given model (such as a deep neural network) over an input
training dataset. This process is typically done through an iterative-
convergent process that gradually minimizes some loss function of the model
over the dataset, by using an optimization technique such as Stochastic
Gradient Descent (SGD) [11].

Since ML training is highly compute intensive, parallelizing it using
computational accelerators such as GPUs and TPUs via distributed training is
vital [12, 13]. In distributed training, multiple _workers_ participate to
iteratively refine the model. A common parallelization approach is _data-
parallelism_ , where training is launched on multiple workers, and each worker
learns and updates the model parameters by processing a small batch of the
training data [14]. Each iteration comprises of computing model updates to the
previous model parameters, and sharing the updates with other workers to form
a new global model. Training a popular image recognition model such as ResNet
[15] requires tens of thousands of iterations until the model’s error
converges to a low-enough target.

Conventionally, workers send their updates to a smaller number of parameter
servers that apply the updates and compute an “averaged” model that is
broadcasted to workers before the next iteration [16]. Concretely, the
learning process involves iteratively computing the model parameters over $K$
workers, each processing a mini-batch of $b$ items at iteration $t$ and
computing the gradient $\nabla f(\mathbf{x}_{k,t})$. The gradients from all
the workers are then collected and averaged by the parameter servers, and the
update rule for the model parameters $\mathbf{x}$ is given by :

$\mathbf{x}_{t+1}=\mathbf{x}_{t}-\eta\dfrac{1}{K}\dfrac{1}{b}\sum_{k=1}^{k=K}{\nabla
f(\mathbf{x}_{k,t})},$ (1)

where $\eta$ is the learning rate parameter which is one of the
“hyperparameters” of the model that is found through empirical search
techniques.

###  II-C Training Challenges in Heterogeneous Environments

Figure 1: Increase in the total training time compared to a homogeneous
cluster for three popular ML workloads. Both the homogenous and heterogenous
clusters have the same total amount of computing resources.

If the computing capacities of the workers is not uniform and constant, then
data-parallel training suffers from severe performance degradation. The
performance and model quality (i.e., model accuracy) of distributed training
is _highly_ dependent on the communication and synchronization of the gradient
updates to compute new model parameters. In bulk synchronous parallel (BSP)
SGD, new model parameters are computed after gradients from _all_ workers have
been received. Even in homogeneous conditions, stragglers are an important
concern in synchronous data-parallel training. In heterogeneous environments,
straggler workers with lower computational resources will take much longer to
process their mini-batches. Thus, BSP suffers from poor parallel efficiency in
heterogeneous environments because of stragglers that significantly increase
the total training time.

##  III Dynamic Mini-Batching

In this section, we describe our dynamic batching mechanism and policies for
distributed training. Our focus is on data-parallel training on heterogeneous
clusters of data center or cloud servers.

Figure 2: With variable batching, we can decrease the batch size on the slower
worker, and increase the batch size on the larger worker, so that no worker
“waits” for another.

###  III-A Key Idea: Variable Mini-Batch Sizes

Conventional data-parallel training uses mini-batch SGD for distributing and
parallelizing the model training process. This approach entails each worker
processing a mini-batch of training samples independently and computing the
gradients. The gradients are computed over the mini-batch of size $b$ by all
the workers. Due to resource heterogeneity, the mini-batch processing times
(i.e., iteration times) across workers can be different. This results in
stragglers in the case of BSP and staleness in the case of ASP—both of which
cause a significant increase in the model training time to a desired accuracy
level.

The main insight is that the mini-batch sizes need not be uniform across
workers—instead, the mini-batch sizes should be proportional to the server
resource availability. This _variable_ batching allows workers to process
different amount of data. The goal is to reduce the differences in the
workers’ iteration times to reduce stragglers and staleness. This is
illustrated in Figure 2, which shows the use of variable batching during the
training process to adjust the worker batch sizes to minimize stragglers.

Such variable batching is compatible with distributed SGD—we assign a mini-
batch size of $b_{k}$ to worker $k$. Because workers are processing different
amounts of training data, their contribution in the training process is no
longer uniform. In conventional SGD, the gradients from all workers are
averaged as per Equation 1. However, with variable batching, the gradients
computed by workers with larger batch sizes need to be “weighted” more than
the gradients computed using smaller batches. Thus we _scale_ the gradients
computed by each worker based on its mini-batch size, and the final gradients
are computed using a weighted average.

We use linear gradient scaling: gradients on worker $k$ are multiplied by
$\lambda_{k}$ such that $\lambda_{k}\propto b_{k}$. To maintain equivalence
with conventional uniform batching, we also require that
$\sum_{k}{\lambda_{k}}=1$. This yields
$\lambda_{k}=\frac{b_{k}}{\sum_{i=1}^{i=K}b_{i}}$. The new weights for the
next iteration are then computed by doing a weighted average of the gradients:

$\displaystyle g_{k,t}$ $\displaystyle=\lambda_{k}\nabla
f(\mathbf{x}_{b_{k},t})$ (2) $\displaystyle\mathbf{x}_{t+1}$
$\displaystyle=\mathbf{x}_{t}-\frac{1}{K}\eta\sum_{k=1}^{k=K}g_{k,t},$ (3)

where $\nabla f(\mathbf{x}_{b_{k},t})$ is the gradient computed using mini-
batch $b_{k}$ by worker $k$. The weighted averaging is done by the parameter
server, and preserves the convergence properties of SGD [17].

Ideally, we want perfect load balancing, and assign mini-batch-sizes to
workers such that all workers finish their iterations at the same time. Due to
servers of different sizes and dynamic resource availability due to
interference or over-commitment, the processing power on workers also varies.
In the next two subsections, we describe two approaches for assigning mini-
batches to workers—a simple static open-loop allocation technique, and a
closed-loop dynamic allocation that can respond to cluster resource dynamics.

###  III-B Static Mini-batch Allocation Policy

Instead of uniform mini-batches for all workers, our _static assignment_
policy computes mini-batch sizes proportional to the worker’s computing power.
Because model training is highly computation-bound, the throughput of workers
is proportional to the available CPU and GPU resources. Thus given a
heterogeneous cluster of $K$ workers, we want $b_{k}\propto X_{k}$, where
$X_{k}$ is the throughput (i.e., training samples processed per second) of
server $k$.

We seek to maintain the initial average mini-batch size, $b_{0}$ that is
provided for a given ML model. We then allocate the mini-batches to different
workers such that the batch sizes are proportional the worker throughput, and
the global batch size is maintained: $b_{k}=\frac{b_{0}X_{k}}{\sum_{i}X_{i}}$
. This ensures that $\sum_{i}b_{i}=Kb_{0}$, where $b_{0}$ corresponds to the
conventional uniform mini-batch size. Importantly, this approach keeps the
total global batch size constant, and invariant to variable batching.

Static mini-batch allocation seeks to “equalize” the iteration times on
different workers, as illustrated in Figure 3, which shows the distribution of
iteration times for ResNet-50 (BSP)111We train ResNet with BSP for all the
examples and figures in this section., with three workers in a heterogeneous
cluster with $(3,5,12)$ CPU cores respectively. With uniform batching in
Figure 3a, the iteration times for the workers are different, due to the
differences in their processing powers. In contrast, with the variable mini-
batching approach, the iteration times for all the workers have similar
frequency distributions, as seen in Figure 3b.

By reducing the gap between iteration times among workers, the variable
batching technique can reduce stragglers in the case of BSP and thus the total
training time in heterogeneous environments. Unlike for BSP, our approach does
not _directly_ address the root cause of slowdowns for ASP training. With ASP,
the slowdown is a result of the statistical inefficiency arising due to
multiple factors, including gradient update staleness. However the relation
between staleness and training time is not as simple to model as the effect of
stragglers on BSP [18, 19], and is not necessarily linear. Nevertheless,
reducing the iteration gap allows us to ameliorate the staleness and improve
the total training time even for ASP, albeit not as effectively as BSP.

(a) Uniform batching.  (b) Variable batching.

Figure 3: Frequency distributions of iteration times across workers in a
heterogeneous cluster. Worker 3 is 3x larger than worker 1, which is 2x larger
than worker 2. Variable batching ensures that the iteration times across
workers are similar.

Estimating throughput. We can estimate the worker throughput required for the
variable batch allocation based on the server’s resource configuration. When
workers are using only CPU resources, a simple way is to assign batch sizes
proportional to the number of CPU cores. In case a distributed training job is
running on both CPU and GPU servers, we assign batch sizes proportional to the
half-precision FLOPs (Floating Point Operations per Second) for each server.
This a one-shot method that is black box and requires no adjustment, and is
“open-loop”.

However,throughput may not be exactly proportional to the server FLOPs. This
error can cause imperfect load-balancing, and can result in sub-optimal mini-
batching. Compared to the ideal batch sizes that equalize all iteration times,
some workers may get smaller batches and wait for workers with larger batches.
We can address this problem by _dynamically_ assigning mini-batch sizes, which
we describe in the next subsection.

###  III-C Proportional-Control Based Dynamic Policy

To mitigate stragglers and staleness, it is crucial for workers to finish
processing their mini-batches simultaneously. In the previous static
technique, the mini-batches were allocated based on the estimated relative
throughput of different workers.

This open-loop estimation, based on the hardware FLOPs, is not accurate in
predicting the training throughput, in two major situations. First, the
training throughput depends on intra-worker parallel scaling characteristics
governed by Amdahl’s law. Thus the observed throughput on large workers (with
more CPUs) may be lower than what is indicated by their core counts. Second,
many scenarios yield _dynamic_ resource availability, which the static mini-
batching approach is ill-suited for.

Our _dynamic_ mini-batching technique is designed to handle throughput
estimation errors, as well as handling dynamic resource availability due to
server overcommittment or intermittent performance interference that lead to
variable effective throughput on the affected workers. The key idea is to
continuously adjust the mini-batch sizes on the workers. The goal is to
equalize the iteration times among all the workers. Let worker $k$ finish
computing gradients for its mini-batch in time $t_{k}$. Ideally, we want
$t_{i}=t_{j}$ for all workers $i,j$.

The dynamic mini-batch adjustment uses a simple proportional-control approach
to compute the mini-batch size of all workers. Since the goal is to equalize
the batch processing times, the “error” is $\tau_{k}=t_{k}-\bar{t}$, where
$\bar{t}$ is the average iteration time across all the workers. To minimize
this error, the mini-batch size is updated by $\Delta(b)$ by the following
proportional control rule:

$\Delta(b_{k})=-X_{k}\tau_{k},$ (4)

where, $X_{k}$ is the throughput of worker $k$, which can be empirically
determined as $X_{k}=b_{k}/t_{k}$. The new batch size for iteration $i+1$ is
computed as follows:

$b_{k}^{i+1}=b_{k}^{i}+\Delta(b_{k}^{i}),$ (5)

Thus slower workers ($t>\bar{t}$) will have a positive error $\tau$, and their
batch sizes will be decreased. Workers whose batch processing times are faster
than average, are capable of handling a higher load, and will get a larger
batch size after the dynamic adjustment. Simplifying the above two equations,
we can compute the new batch size $b_{k}^{(1)}$ from the initial batch size
$b_{k}^{(0)}$ as : $b_{k}^{(1)}=b_{k}^{(0)}\bar{t}/t_{k}$.

This policy essentially combines model-based and conventional black-box PID
controllers. Instead of using and tuning an arbitrary proportionality constant
like in most PID controllers, we use the (estimated) throughput.

Initial mini-batch sizes. The dynamic mini-batching approach works with any
initial batch size. By default, the initial batch sizes are allocated based on
the throughput-based open-loop variable batching approach described in the
previous subsection. In that case, any error in the throughput approximation
(based on the CPU/GPU FLOPs) is corrected by the control mechanism.

While a good starting point is desirable, it is not necessary. The dynamic
batching approach permits _any_ initial batch size allocation, with the caveat
that the farther the initial batch size is from the ideal (i.e., throughput
proportional), the larger number of batch adjustment steps are required to
reach the equilibrium steady-state batch sizes. For example, Figure 4a shows
the progress of the batch adjustment on three heterogeneous workers when all
the workers are assigned the same initial batch size (which is sub-optimal).
We can see that the mini-batch sizes on the different workers converge to
their stable throughput-proportional values after only two batch adjustments.
Thus, the dynamic batching technique is useful in situations where apriori
throughput estimates are not be known.

####  III-C1 Control stability

The dynamic batch size adjustment can be done at the end of every iteration.
However, it is neither prudent nor necessary to do so. Changing the batch size
on workers is _not_ a zero-cost operation, because it involves terminating and
restarting the training. Furthermore, due to the stochastic nature of
training, iteration times on workers will never converge to the exact average,
and there will always be some error which the proportional control mechanism
will try to chase. This is illustrated in Figure 4b, which shows the mini-
batch sizes “oscillating” due to the dynamic batching adjustments.

(a) Batch sizes quickly converge within two adjustments.  (b) Oscillations in
mini-batch sizes without dead-banding.

Figure 4: Dynamic batch size adjustments.

To prevent these oscillations and reduce the overhead of batch adjustments, we
use three main techniques: dead-banding, exponential smoothing iteration
times, and lower-upper bounds on batch sizes. We describe these approaches
below:

Dead-banding. After every iteration, we compute the new batch sizes using the
proportional-control technique as described so far. We use a dead-band for our
controller: batch sizes are updated only if the change is substantial. We
compute the difference between $b^{i+1}-b^{i}$ and do not update if this is
smaller than threshold, $\Delta_{min}(b)$. If the change in the batch sizes on
all workers is less than $\Delta_{min}(b)$, then no batch readjustment is
made. The threshold can be chosen based on how sensitive we want the
adjustment to be, and it also depends on the performance overhead of
readjusting the batch sizes. For instance, current ML frameworks such as
TensorFlow do not support graceful dynamic adjustment of batch sizes and
require terminating and restarting the entire training process, in which case
a larger threshold is preferable. Based on the TensorFlow overheads, we use a
dead-band threshold of 0.05: meaning that the new batch sizes on all workers
must be atleast change by 5%.

Exponential Smoothing. With dead-banding, we only need to make batch
adjustments at the start of the training process and whenever the underlying
resource availability of the workers changes due to resource over-commitment
or preemption. To improve the controller stability and avoid spurious
readjustments, we compute the error (deviation of iteration time from the
cluster average) on multiple iterations. Specifically, the error is computed
using an EWMA (Exponentially Weighted Moving Average) across all the
iterations since the previous batch readjustment. This provides us with the
“Integrator” component in the controller, and particularly useful to prevent
outliers.

With the dead-banding, we don’t update batches on every iteration, and the
moving average is computed in the interval with no batch size updates. Assume
that last batch update happened on iteration $j$, and the current iteration is
$i$. We then compute the average of worker $k$: $\mu(k,i,j)$ =
EWMA($t_{k}^{i},t_{k}^{i-1},...t_{k}^{j}$). The smoothed iteration times
($\mu$) are used in Equation 4 to compute the error and the batch size update.

Batch size bounds. Finally, we enforce lower and upper bounds on mini-batch
sizes on all workers. These bounds prevent extreme batch sizes in cases of
extreme heterogeneity, and ensure that the total throughput does not drop
because of variable batching. Extremely small batches cannot use all the
hardware parallelism and yield low throughput. Similarly, large batches may
exhaust memory resources and also result in lower throughput. This is
illustrated in Figure 5, which shows the throughput increasing with the batch
size, until a sharp decline due to memory exhaustion in the GPU, and a gradual
decline for CPU workers.

(a) Throughput on GPU  (b) Throughput on CPU

Figure 5: Training throughput (img/sec) increases with batch size, then
declines because of hitting resource (memory) limits on the workers,
especially on GPUs where the memory limit is strict.

We thus allow users to specify estimates of lower and upper bounds
($b_{\text{min}},b_{\text{max}}$) of the batch sizes for all the workers. As
the training progresses and we readjust batch sizes, we get more data points
for the throughput curve. If we observe a drop in worker throughput after
increasing its batch size from $b_{0}$ to $b$, then we update its
$b_{\text{max}}=b_{0}$. This ensures that future batch readjustments will not
result in a drop in throughput.

Putting it all together. We can integrate all the control stability techniques
into the proportional controller. Assume that the latest iteration is $i$, and
the last batch-update was made in iteration $j$. The pseudo-code for our
dynamic batching can be expressed as:

  1. 1.

Compute exponential moving average iteration times $\mu(k,i,j)$ for all
workers $k$.

  2. 2.

Use $\mu(k,i,j)$ in Eqn 4, 5 to compute $\Delta(b_{k})$ and $b_{k}^{i+1}$.

  3. 3.

Enforce batch size bounds: $b_{k,min}\leq b_{k}\leq b_{k,max}$

  4. 4.

Apply deadbanding check. If $\max_{k}{\Delta(b_{k})/b_{k}}>\Delta_{min}(b)$,
update all batch sizes. Otherwise do nothing.

##  IV Experimental Evaluation

We conduct all our evaluation using our modified TensorFlow implementation
that monitors differences in iteration times and dynamically adjusts per-
worker batch sizes. We use the following standard well-known training
workloads:

  * •

ResNet-50: TensorFlow’s ResNet benchmark [20], trained on the standard
CIFAR-10 dataset. We use a momentum optimizer with a learning rate schedule of
[0.1, 0.01, 0.001, 0.0002].

  * •

MNIST CNN [21]: with Adam [22] and learning rate of 0.0001.

  * •

Linear Regression: To show our system effectively sustains heavy as well as
comparatively lighter workloads, we perform Linear Regression (LR) on
Harvard’s bar crawl dataset [23].

(a) ResNet-50  (b) MNIST CNN  (c) Linear Regression

Figure 6: With BSP synchronization, variable batching can reduce the total
training time to accuracy by up to $4\times$.

Experimental environment and setup. We use the parameter server distribution
strategy for all model training. We appropriately scale the number of
parameter servers to ensure that they are not the bottleneck. All TensorFlow
processes (master, parameter servers, and workers) are deployed inside Docker
containers for ease of management and fine-grained resource accounting and
control. We conduct all our empirical evaluation on a local cluster as well as
on Google Cloud Platform. The local cluster’s CPU servers have 48-core Intel
Xeon Platinum 2.10GHz CPUs and 256 GB of RAM. The GPU is Nvidia Tesla
P100-PCIe-16GB.

###  IV-A CPU Training

In this subsection, we focus on static heterogeneity when the cluster is
composed of VMs/containers of different sizes. We are primarily interested in
determining the impact of heterogeneity, and not parallel scaling. Therefore,
we evaluate on clusters with different heterogeneity levels but the same total
resource capacity. For instance, we compare a cluster configuration with two
workers with (4, 16) CPUs, vs. two workers with (8, 12) CPUs. For CPU-only
clusters, we define the heterogeneity level as:
$\text{H-level}=\max\text{number of cores}/\min\text{number of cores}$.

Local cluster. We first present the training performance across different
heterogeneity levels on our local cluster with three CPU workers. The total
number of CPU cores across the three workers is 39, and so a H-level of 2
would yield a (9, 12, 18) CPU cores configuration. The total training time to
reach a desired level of model accuracy across the three different workloads
is shown in Figure 6. Compared to vanilla TensorFlow’s uniform batching, our
variable batching approach can significantly reduce the training time. In
general, the variable batching does better compared to the uniform batching at
higher heterogeneity levels, because it is able to mitigate the stragglers.
For computationally intensive ResNet, our variable batching improves training
times by $2\times$ at H-level of 2, and $2.4\times$ at the highest H-level of
10. The high heterogeneity levels result in very small workers (e.g., H-level
10 is a (2,17,20) configuration). The small workers end up being stragglers
even with variable batching’s load balancing, because we are not able to use
any parallelism inside these small workers, yet still face the same
communication and model synchronization overhead.

The MNIST CNN also sees a performance improvement of $2\times$—$4\times$.
Finally, the Linear Regression workload is the least computationally
expensive, and sees the least benefit ($\sim 15\%$) from the load-balancing
that variable batching provides, because it is communication and
synchronization bound.

Importantly, our variable batching can ameliorate the heterogeneity-induced
slowdown, and can “flatten the curve”. At a high H-level of 6, ResNet training
time only increases by $2\times$ compared to the homogenous setup (Figure 6).
Similarly, MNIST time increases by by $4\times$, and Linear Regression by only
$5\%$.

Result: _Variable batching can mitigate stragglers in BSP and can reduce
training time by $4\times$ for high heterogeneity levels. Our technique is
particularly effective in scenarios that are computation and not communication
bound._

###  IV-B GPU Training

For GPU training, we first consider an extreme heterogeneity case where the
cluster comprises of both CPU and GPU workers. Specifically, we use a single
GPU worker (Tesla P100) and CPU worker (48-core Intel Xeon). We compare the
performance of uniform, variable, and dynamic batching in Figure 7a.

Recall that variable batch allocation is an open-loop approach that assigns
batch sizes based on the hardware FLOPs performance and not actual throughput.
Compared to uniform batching, we are able to reduce the training time by more
than $4\times$ for the computationally intensive ResNet workload. For MNIST,
the cluster is underutilized, since workload is not computationally bound, and
we see a more modest 20% improvement in training time with our approach.

(a) GPU-CPU mix.  (b) ResNet with multiple GPU types.

Figure 7: GPU training.

The performance of the Xeon Platinum CPUs used in our local cluster
experiments is far closer to GPU performance most cloud CPUs. For instance,
the ratios of the FLOPs and the batch size between the GPU and CPU was
$0.813:0.187$, and thus the GPU worker is “only” $4.3\times$ faster.

Interestingly, the dynamic batching improves performance by about 3% compared
to static variable batching for MNIST CNN, and has a negligible effect for
ResNet. This intriguing result is because of the tradeoff of dynamic batching.
For a computationally intensive workload like ResNet, hardware FLOPs
approximates throughput, so there was not enough opportunity for the dynamic
readjustments. The kill-restart approach poses a small performance overhead
too. These two factors “cancel out” and in most cases, static variable
batching is “good enough”.

Result: _Variable batching allows efficient use of mixed GPU-CPU clusters, and
can reduce the training time by up to $4\times$._

We also examine training performance on a cloud cluster with two different
types of GPUs. Specifically, we run two VMs with Tesla T4 and two VMs with
Tesla P4 GPUs. The training time of ResNet (BSP) was 90 minutes with uniform
batching, and only 20 minutes with variable batching—-a $4.5\times$
improvement.

##  V Related Work

Heterogeneous Training. The closest work is [3], which develops
synchronization techniques (DynSGD and ConSGD), for mitigating the effects of
staleness and stragglers by explicitly accounting for staleness using a
vector-clock technique. However much like other work in this area [4], the
cluster heterogeneity they consider is only a result of stochastic performance
variations (random worker slowdowns). Instead, we focus on _systemic_ and
severe heterogeneity due to vastly different resource sizes of workers. Our
fundamental idea of variable mini-batch sizes is agnostic to the
synchronization technique and can also be integrated with ConsSGD to provide
support for alleviating the random slowdowns due to performance interference.

Heterogeneity in training is being recognized as an important missing feature
and many approaches are being developed. [24] uses a gradient coding scheme to
tolerate stragglers due to static heterogeneity in a BSP setup. Our variable
batching technique is applicable in existing parameter server based
architectures and does not require gradient coding. Heterogeneity for
decentralized training is explored in Hop [4], which uses a bounded staleness
approach and bound the iteration-gap. The technique is shown to be effective
in case of random worker slowdowns. Its effectiveness at high static
heterogeneity levels is less clear, since the large iteration gaps may pose
fundamental synchronization challenges in the decentralized setting.

Resource allocation for training is also an active area of work, and is
challenging due to our incomplete first-principles understanding of SGD
scaling, and profiling-driven empirical models are typically used. [25] shows
how to do cluster resource allocation and scheduling for ML training jobs by
developing and using an empirical performance model to determine number of
workers and parameter servers to use. Similarly, Cynthia [26] uses an
analytical performance model for cost efficient cloud resource provisioning.
In contrast, our approach can directly start training without the need for
apriori modeling. Our design goal was to design a generally usable mechanism
that is plug-in compatible with different resource allocation approaches,
training algorithms, and treats ML models as “black boxes”. Integrated systems
and training algorithm co-design, like in Orpheus [27] that improves
consistency via periodic centralized synchronization, is an alternative
approach.

Model synchronization impacts training performance, especially in cloud
environments with higher stochasticity in server performance and network
latencies. This has motivated many synchronization techniques such as stale
synchronous parallel [28] and others [29, 30, 31, 32, 33]. The performance
tradeoffs of synchronization techniques in dynamic cloud environments is
studied in [34]. Although asynchronous approaches [1] seem promising in
heterogeneous environments, gradient staleness is still a pernicious problem
[35, 19, 18, 36].

Batch size in distributed training is one of the most crucial hyper-parameters
that affects the training performance as well as the model convergence.
Understanding these tradeoffs is a key problem in machine learning [37, 38,
18, 39]. Due to the duality between learning rates and global batch sizes
[40], adjusting the _global_ batch size is a known technique to regulate the
errors in SGD training [41]. Adabatch [42] and [43] describe a “batch size
schedule” analogous to a learning rate schedule. This is distinct from our
dynamic mini-batch adjustment, and the dynamic global batch schedules can
easily be incorporated into our approach. Finally, the theoretical soundness
of variable mini-batch sizes can be found in [17]. They also propose a new
synchronization technique where gradient updates are “pulled” from workers
periodically, irrespective of their mini-batch processing, resulting in
different sized worker updates.

Acknowledgments. This work was partially supported by the Google Cloud
research credits program.

## References

  * [1] B. Recht, C. Re, S. Wright, and F. Niu, “Hogwild: A lock-free approach to parallelizing stochastic gradient descent,” in _NIPS 2011_ , pp. 693–701.
  * [2] “PID controllers,” https://en.wikipedia.org/wiki/PID_controller.
  * [3] J. Jiang, B. Cui, C. Zhang, and L. Yu, “Heterogeneity-aware Distributed Parameter Servers,” in _SIGMOD ’17_ , pp. 463–478.
  * [4] Q. Luo, J. Lin, Y. Zhuo, and X. Qian, “Hop: Heterogeneity-aware Decentralized Training,” in _24th ASPLOS ’19_ , pp. 893–907.
  * [5] “Amazon EC2 Spot Instances,” https://aws.amazon.com/ec2/spot/ .
  * [6] “Google Preemptible VMs,” https://cloud.google.com/preemptible-vms/, September 2015, https://cloud.google.com/preemptible-vms/.
  * [7] S. Li, R. J. Walls, L. Xu, and T. Guo, “Speeding up Deep Learning with Transient Servers,” _16th IEEE ICAC ’19_ , arXiv: 1903.00045.
  * [8] R. W. Shijian Li and T. Guo, “Characterizing and modeling distributed training with transient cloud gpu servers,” _40th IEEE ICDCS’20_.
  * [9] P. Sharma, D. Irwin, and P. Shenoy, “Portfolio-driven resource management for transient cloud servers,” in _ACM POMACS ’17_ , vol. 1, no. 4.
  * [10] P. Sharma, T. Guo, X. He, D. Irwin, and P. Shenoy, “Flint: Batch-interactive data-intensive processing on transient servers,” in _11th EuroSys ’16_ , 2016.
  * [11] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in _COMPSTAT’2010_ , pp. 177–186.
  * [12] T. Ben-Nun and T. Hoefler, “Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis,” _arXiv:1802.09941 [cs]_.
  * [13] R. Mayer and H.-A. Jacobsen, “Scalable Deep learning on Distributed Infrastructures: Challenges, Techniques, and Tools,” _ACM Comput. Surv._ , vol. 53, Feb. 2020. [Online]. Available: https://doi.org/10.1145/3363554
  * [14] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang _et al._ , “Large scale distributed deep networks,” in _NIPS ’12_ , pp. 1223–1231.
  * [15] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _31st AAAI ’17_.
  * [16] M. Li, “Scaling Distributed Machine Learning with the Parameter Server,” in _ACM BigDataScience ’14_ , pp. 1–1.
  * [17] N. S. Ferdinand, H. Al-Lawati, S. C. Draper, and M. S. Nokleby, “Anytime Minibatch: Exploiting Stragglers in Online Distributed Optimization,” in _ICLR ’19_.
  * [18] Z. Zhou, P. Mertikopoulos, N. Bambos, P. Glynn, Y. Ye, L.-J. Li, and L. Fei-Fei, “Distributed asynchronous optimization with unbounded delays: How slow can you go?” in _Proceedings of the 35th ICML_ , vol. 80, 2018.
  * [19] S. Dutta, G. Joshi, S. Ghosh, P. Dube, and P. Nagpurkar, “Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD,” _AISTATS 2018, arXiv:1803.01113_.
  * [20] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE CVPR_ , pp. 770–778.
  * [21] “Tensorflow official Github repository MNIST.” [Online]. Available: https://github.com/tensorflow/models/tree/master/official/r1/mnist
  * [22] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd ICLR ’15_ , Y. Bengio and Y. LeCun, Eds.
  * [23] J. A. Killian, K. M. Passino, A. Nandi, D. R. Madden, and J. D. Clapp, “Learning to detect heavy drinking episodes using smartphone accelerometer data,” in _KHD@IJCAI_ , 2019.
  * [24] H. Wang, S. Guo, B. Tang, R. Li, and C. Li, “Heterogeneity-aware Gradient Coding for Straggler Tolerance,” _IEEE 39th ICDCS_ , 2019\.
  * [25] Y. Peng, Y. Bao, Y. Chen, C. Wu, and C. Guo, “Optimus: an efficient dynamic resource scheduler for deep learning clusters,” in _13th EuroSys ’18_ , pp. 1–14.
  * [26] H. Zheng, F. Xu, L. Chen, Z. Zhou, and F. Liu, “Cynthia: Cost-Efficient Cloud Resource Provisioning for Predictable Distributed Deep Neural Network Training,” in _48th ICPP 2019_ , pp. 1–11.
  * [27] P. Xie, J. K. Kim, Q. Ho, Y. Yu, and E. Xing, “Orpheus: Efficient Distributed Machine Learning via System and Algorithm Co-design,” in _ACM SoCC ’18_ , pp. 1–13.
  * [28] Q. Ho, J. Cipar, H. Cui, J. K. Kim, S. Lee, P. B. Gibbons, G. A. Gibson, G. R. Ganger, and E. P. Xing, “More effective distributed ml via a Stale Synchronous Parallel (SSP) Parameter Server,” _26th NIPS - Volume 1 ’13_ , pp. 1223–1231.
  * [29] H. Cui, J. Cipar, Q. Ho, J. K. Kim, S. Lee, A. Kumar, J. Wei, W. Dai, G. R. Ganger, P. B. Gibbons _et al._ , “Exploiting bounded staleness to speed up big data analytics,” in _USENIX ATC ’14_ , pp. 37–48.
  * [30] A. Harlap, H. Cui, W. Dai, J. Wei, G. R. Ganger, P. B. Gibbons, G. A. Gibson, and E. P. Xing, “Addressing the straggler problem for iterative convergent parallel ML,” in _7th ACM SoCC ’16_ , pp. 98–111.
  * [31] G. Zhao, L. Gao, and D. Irwin, “Sync-on-the-fly: A Parallel Framework for Gradient Descent Algorithms on Transient Resources,” in _IEEE ICBD (Big Data) ’18_ , pp. 392–397.
  * [32] C. Zhang, H. Tian, W. Wang, and F. Yan, “Stay Fresh: Speculative Synchronization for Fast Distributed Machine Learning,” in _IEEE 38th ICDCS ’18_ , pp. 99–109.
  * [33] J. Wei, W. Dai, A. Qiao, Q. Ho, H. Cui, G. R. Ganger, P. B. Gibbons, G. A. Gibson, and E. P. Xing, “Managed communication and consistency for fast data-parallel iterative analytics,” in _6th ACM SoCC ’15_ , pp. 381–394.
  * [34] P. Ambati, D. Irwin, P. Shenoy, L. Gao, A. Ali-Eldin, and J. Albrecht, “Understanding synchronization costs for distributed ml on transient cloud resources,” _IEEE IC2E ’19_ , pp. 145–155.
  * [35] J. Chen, R. Monga, S. Bengio, and R. Jozefowicz, “Revisting Distributed Synchronous SGD,” _arXiv:1604.00981_ , 2016.
  * [36] F. Haddadpour, M. M. Kamani, M. Mahdavi, and V. Cadambe, “Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization,” _32nd NIPS ’19_.
  * [37] C. J. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl, “Measuring the Effects of Data Parallelism on Neural Network Training,” _Journal of ML Research 2019, arXiv:1811.03600_.
  * [38] P. Yin, P. Luo, and T. Nakamura, “Small Batch or Large Batch?: Gaussian Walk with Rebound Can Teach,” in _23rd ACM SIGKDD 2017_ , pp. 1275–1284.
  * [39] S. Gupta, W. Zhang, and F. Wang, “Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study,” in _IEEE 16th ICDM_ , 2016, pp. 171–180.
  * [40] S. L. Smith and Q. V. Le, “A Bayesian Perspective on Generalization and Stochastic Gradient Descent,” in _ICLR_ , 2018.
  * [41] H. Yu and R. Jin, “On the computation and communication complexity of parallel SGD with dynamic batch sizes for stochastic non-convex optimization,” in _Proceedings of the 36th ICML_ , vol. 97, 2019.
  * [42] A. Devarakonda, M. Naumov, and M. Garland, “Adabatch: Adaptive Batch sizes for training Deep Neural Networks,” _ICLR workshop, arXiv:1712.02029_ , 2018.
  * [43] S. L. Smith, P.-J. Kindermans, C. Ying, and Q. V. Le, “Don’t Decay the Learning Rate, Increase the Batch Size,” _ICLR ’18_ , p. 11.