# Strong large scale magnetic fields in rotating convection-driven dynamos:
the important role of magnetic diffusion

Ming Yan and Michael A. Calkins  Department of Physics, University of
Colorado, Boulder, CO 80309, USA

###### Abstract

Natural dynamos such as planets and stars generate global scale magnetic field
despite the inferred presence of small scale turbulence. Such systems are
known as large scale dynamos and are typically driven by convection and
influenced by rotation. Previous numerical studies of rotating dynamos
generally find that the large scale magnetic field becomes weaker as the flow
becomes more turbulent. The underlying physical processes necessary for
sustaining so-called large scale dynamos is therefore still debated. Here we
use a suite of numerical simulations to show that strong large scale magnetic
fields can be generated in rotating convective turbulence provided that two
conditions are satisfied: (1) the flow remains rotationally constrained; and
(2) magnetic diffusion is important on the small convective length scale.
These findings are in agreement with previous asymptotic predictions and
suggest that natural dynamos might satisfy these two conditions.

Planets and stars generate magnetic field through the dynamo process that
converts the kinetic energy of fluid motion into magnetic energy [e.g. 1, 2].
These fields are often dominated by their global (e.g. dipolar) components,
whereas the underlying fluid motion that powers the dynamo is turbulent and
characterized by length scales much smaller than the global scale. The
Coriolis force likely plays a fundamental role in the generation of such large
scale structure by leading to domain-scale correlations in the flow field that
generate coherent large scale magnetic field [3, 4, 5]. This correlation can
be quantified by the helicity of the fluid motion [6], defined as the dot
product between the velocity field and the curl of the velocity field.
However, simulations tend to find that the relative contribution of the large
scale component of the dynamo generated magnetic field decreases as the fluid
becomes more strongly forced and turbulent [7, 8, 9, 10]. This loss of a
predominantly large scale magnetic field is associated with a decrease in the
relative (as measured to a maximum value) helicity. The fundamental question
pertaining to natural systems is then how a large scale dynamo is maintained
in the presence of strongly forced, small scale turbulence. Asymptotic theory
relevant to rapidly rotating flows suggests that the deficit in relative
helicity can be overcome by enhancing the influence of magnetic diffusion on
the small convective length scale [11]. In the present work we utilize direct
numerical simulations (DNS) of rapidly rotating convective turbulence to
confirm these asymptotic predictions, thus helping to shed light on a major
problem in dynamo theory.

Buoyancy is thought to provide the main source of power for natural dynamos.
Rayleigh-Bénard convection, consisting of a fluid layer of depth $H$ heated
from below and cooled from above, is a canonical system for studying buoyancy-
driven turbulence and dynamos. The buoyancy force is controlled by the non-
dimensional Rayleigh number,

$Ra=\frac{g\alpha\Delta TH^{3}}{\nu\kappa},$ (1)

where $g$ is the (constant) gravitational acceleration, $\alpha$ is the
thermal expansion coefficient, $\Delta T$ is the temperature difference
between the top and bottom boundaries, $\nu$ is the kinematic viscosity and
$\kappa$ is the thermal diffusivity. The relative importance of viscous and
inertial forces to the Coriolis force is quantified by the Ekman number and
the Rossby number defined by, respectively,

$E=\frac{\nu}{2\Omega H^{2}},\quad Ro=\frac{U}{2\Omega H},$ (2)

where $\Omega$ is the system rotation rate and $U$ is the characteristic
dimensional flow speed. The Reynolds number, quantifying the relative
importance of inertia and viscous forces, is then related via $Re=Ro/E$.

The electrical properties of the fluid are specified by the magnetic Prandtl
number, $Pm=\nu/\eta$, where $\eta$ is the magnetic diffusivity. Dynamos
require sufficiently large flow speeds to overcome, or at least balance, the
resistive effects of ohmic diffusion. Therefore, the magnetic Reynolds number,
$Rm=RePm$, must exceed a threshold value so that the self-induced magnetic
field does not diffuse away. Planetary and stellar dynamos are in a regime of
rapidly rotating magnetized convective turbulence in which $E\ll Ro\ll 1$ and
$Re\gg Rm\gg 1$, which is a parameter space that is challenging to study
numerically given the broad range of scales characterizing the dynamics.

Asymptotic theory can provide insight into the extreme parameter space that
characterizes most natural dynamos. A brief summary of this theory is provided
here to aid in the interpretation of the simulation results. For details the
reader is referred to Refs. [11, 12]. In the limit $E\rightarrow 0$ and
$Ro\rightarrow 0$, the dynamics of convection depend on the reduced Rayleigh
number [e.g. 13]

$\widetilde{Ra}\equiv RaE^{4/3}.$ (3)

At leading asymptotic order the flow is geostrophically balanced on
$O\left(E^{1/3}\right)$ horizontal convective length scales. Thus, when the
governing equations are non-dimensionalized on this small convective length
scale, horizontal derivatives are $O\left(1\right)$ and vertical derivatives
are $O\left(E^{1/3}\right)$. Variables are decomposed into horizontal averages
(mean) and fluctuating quantities such that the magnetic field and velocity
field become $\mathbf{B}=\overline{\bf B}(z,t)+\mathbf{B}^{\prime}(x,y,z,t)$
and $\mathbf{u}=\overline{\bf u}(z,t)+\mathbf{u}^{\prime}(x,y,z,t)$,
respectively. In the plane layer geometry studied here the mean velocity field
is negligibly small and so $\mathbf{u}\approx\mathbf{u}^{\prime}$. In the
asymptotic regime, the small scale magnetic Reynolds number, defined as

$\widetilde{Rm}=RmE^{1/3},$ (4)

plays the key role in determining the relative sizes of the mean and
fluctuating magnetic field. In particular, accessing what we refer to as the
energetically robust large scale dynamo regime, in which the energy contained
in the large scale (mean) component of the magnetic field is asymptotically
larger than the energy contained in the fluctuating magnetic field, is
achieved by balancing the emf with large scale magnetic diffusion,
${\bf\widehat{z}}\times\partial_{z}\overline{\left(\mathbf{u}^{\prime}\times\mathbf{B}^{\prime}\right)}\sim\widetilde{Rm}^{-1}\partial_{z}^{2}\overline{\bf
B}$, and mean stretching with small scale magnetic diffusion, $\overline{\bf
B}\cdot\nabla\mathbf{u}\sim\widetilde{Rm}^{-1}\nabla^{2}\mathbf{B}^{\prime}$;
these two balances give, respectively,

$|\mathbf{B}^{\prime}|/|\overline{\bf B}|\sim
E^{1/3}/\widetilde{Rm},\qquad|\mathbf{B}^{\prime}|/|\overline{\bf
B}|\sim\widetilde{Rm},$ (5)

where we have used the fact that $|\mathbf{u}^{\prime}|=O(1)$ to ensure
geostrophic balance at leading order [11]. For the two relations given above
to be consistent we then require

$\widetilde{Rm}=O\left(E^{1/6}\right).$ (6)

This relationship states that the energetically robust large scale dynamo
regime requires that both $E$ and $\widetilde{Rm}$ are small, i.e. equation
(6) is the distinguished limit that allows for the large scale magnetic field
to be energetically larger than the small scale magnetic field.

Connecting the asymptotic scaling of equation (6) with $Pm$ can be made upon
noting that $\widetilde{Rm}=Pm\widetilde{Re}$, where
$\widetilde{Re}=ReE^{1/3}$ is the small scale Reynolds number. In a given
simulation, $\widetilde{Re}$ is controlled by $\widetilde{Ra}$. Thus, reaching
small values of $\widetilde{Rm}$ requires that $Pm$ is reduced; since
$\widetilde{Re}=O(1)$ this implies we need $Pm=O\left(E^{1/6}\right)$.
Importantly, the energetically robust large scale dynamo regime is not limited
to small values of $\widetilde{Ra}$ (i.e. limited to near the onset of
convection), and is therefore not limited to convective states in which the
helicity is maximal. We note that the simulations are consistent with a
$E^{1/3}$ scaling for the convective length scale, though a dependence on
$\widetilde{Ra}$ is also observed. The inertial theory for rotating convection
suggests that the length scale should behave like $Ro^{1/2}$ [e.g. 14, 15].
Though we do not test this inertial scaling with the present simulations, such
a scaling would still imply that $\widetilde{Rm}$ must be small to observe
strong large scale magnetic field.

In the present study we simulate dynamo action of a Boussinesq fluid driven by
rotating convection as $E$, $Ra$, and $Pm$ are varied. Reduced Rayleigh
numbers up to $\widetilde{Ra}\approx 80$ are reached for each value of $E$;
the so-called geostrophic turbulence regime occurs for $\widetilde{Ra}\gtrsim
40$ [13]. For Ekman numbers within the range $10^{-6}\leq E\leq 10^{-4}$ we
use $Pm=1$, and for $E\leq 10^{-6}$ we reduce the magnetic Prandtl number down
to $Pm=0.05$ at the smallest Ekman number considered, $E=10^{-8}$. The largest
Reynolds number achieved in the survey was $Re=1.53\times 10^{4}$ for
$E=10^{-8}$ and $\widetilde{Ra}\approx 80$. The maximum Rossby was $Ro=0.06$
for $E=10^{-4}$ and $\widetilde{Ra}\approx 80$. The simulations use a de-
aliased pseudospectral method in which all flow variables are expanded as
Chebyshev polynomials in the vertical dimension and Fourier series in the
horizontal dimensions. Resolutions up to $432$ Fourier modes and $864$
Chebyshev polynomials were used for the most demanding calculations. A third
order accurate implicit-explicit time-stepping scheme is used [e.g. 16, 17] in
which timestep sizes (in non-dimensional rotation time) down to $\Delta
t=5\times 10^{-4}$ were used. Each simulation was run until a statistically
stationary state was achieved and statistics were computed only once this
state was reached. We found that this state was reached in approximately
hundreds to thousands of rotation times. Approximately 5 million CPU hours
were required to generate the dataset used in the present study. In all cases
shown the horizontal dimension of the simulation domain is scaled such that 10
critical wavelengths are present. The boundary conditions are stress-free,
isothermal and the magnetic field is required to be purely vertical at the top
and bottom boundaries. The thermal Prandtl number is fixed at
$Pr=\nu/\kappa=1$.

(a)

(b)

Figure 1:  Helicity (rms) for all simulations: (a) $\overline{\mathcal{H}}$
versus $Ra$; (b) asymptotically rescaled rms helicity
$E\overline{\mathcal{H}}$ versus reduced Rayleigh number,
$\widetilde{Ra}=RaE^{4/3}$. For the symbols the colors denote different values
of $Pm$ and the shapes denote different values of $E$.

The mean helicity is denoted by
$\overline{\mathcal{H}}(z)\equiv\overline{\mathbf{u}^{\prime}\cdot\boldsymbol{\zeta}^{\prime}}$,
where $\boldsymbol{\zeta}^{\prime}=\nabla\times\mathbf{u}^{\prime}$ is the
vorticity. Helicity can stretch and twist magnetic field lines collectively
over the horizontal plane, and therefore induce large scale magnetic field. It
is well known that when strongly influenced by rotation, convection is helical
[e.g. 18]. Fig. 1(a) shows the rms value of the helicity for all simulations.
All combinations of $E$ and $Pm$ follow similar behavior with increasing $Ra$
and larger values of helicity are observed for decreasing Ekman numbers,
whereas $Pm$ tends to have only a weak influence on $\overline{\mathcal{H}}$.
In our DNS we non-dimensionalize the equations using the depth $H$ and speed
$\nu/H$; adapting this scaling to the asymptotic theory gives
$|\mathbf{u}^{\prime}|=O(E^{-1/3})$ and
$|\boldsymbol{\zeta}^{\prime}|=O(E^{-2/3})$ so that
$|\overline{\mathcal{H}}|=O(E^{-1})$ as $E\rightarrow 0$. Fig. 1(b) shows the
asymptotically rescaled rms helicity versus $\widetilde{Ra}$; the collapse of
the data shows that these simulations are in a quasi-geostrophic dynamical
state [19].

(a)

(b)

(c)

(d)

(e)

Figure 2:  (a) Relative (rms) helicity for all simulations versus
$\widetilde{Ra}$; the arrow shows the parameter space location for the
visualizations shown in (b)-(e). (b,d) Visualizations of (b) vertical
vorticity and (d) $x$-component of the magnetic field for $E=10^{-7}$,
$Pm=0.3$ and $\widetilde{Ra}\approx 44$. (c,e) Visualizations of (c) vertical
vorticity and (e) $x$-component of the magnetic field for $E=10^{-8}$,
$Pm=0.05$ and $\widetilde{Ra}\approx 44$. The symbols have the same meaning as
in Fig. 1.

It is further helpful to define the relative helicity
$\overline{\mathcal{H}}_{r}=\overline{\mathcal{H}}/(u\,\zeta)$, where $u$ and
$\zeta$ are volumetric rms values, such that maximally helical flows are
characterized by $|\overline{\mathcal{H}}_{r}|=O(1)$. In agreement with
previous studies, Fig. 2(a) shows that $|\overline{\mathcal{H}}_{r}|$ becomes
small as $\widetilde{Ra}$ increases for all of the simulations, yet remains
finite [e.g. 20, 21, 22]. These results suggest that natural dynamos, for
which $\widetilde{Ra}\gg 1$, are also characterized by small values of the
relative helicity. However, reducing the magnetic Prandtl number shows that
despite the small relative helicity, large scale dynamo action remains
possible, as predicted by theory. Figs. 2(b)-(e) compare flow visualizations
for (b,d) $E=10^{-7}$ and $Pm=0.3$ with (c,e) $E=10^{-8}$ and $Pm=0.05$. As
indicated by the black arrow in Fig. 2(a), both cases are characterized by
similar values of $\overline{\mathcal{H}}_{r}$. The renderings of the vertical
component of the vorticity shown in (b) and (c) provide visual evidence that
both simulations are characterized by dynamically similar turbulent flows.
Whereas the large scale Reynolds numbers are $Re\approx 4.5\times 10^{3}$ and
$Re\approx 1.1\times 10^{4}$ for $E=10^{-7}$ and $E=10^{-8}$, respectively,
the corresponding small scale Reynolds numbers for both cases are
$\widetilde{Re}\approx 21$ and $\widetilde{Re}\approx 24$. The corresponding
$x$-component of the magnetic field vector is shown for the two cases in Figs.
2(d) and (e) where we find a more obvious coherent large scale component for
the $E=10^{-8}$ ($Pm=0.05$) case.

(a)

(b)

Figure 3:  Fraction of the mean magnetic energy to the total magnetic energy,
$\overline{M}/M$, in all simulations: (a) $\overline{M}/M$ versus
$\widetilde{Ra}$; (b) $\overline{M}/M$ versus $\widetilde{Rm}$. The symbols
have the same meaning as in Fig. 1.

The relative size of the large scale magnetic field can be quantified by
computing $\overline{M}/M$, where $\overline{M}$ is the magnetic energy of the
mean magnetic field and $M$ is the total magnetic energy. Fig. 3(a) shows this
mean energy fraction versus $\widetilde{Ra}$. We find that smaller values of
$Pm$ typically yield larger values of $\overline{M}/M$ for a given value of
$\widetilde{Ra}$. For a fixed value of $Pm$ the mean energy fraction decreases
with increasing $\widetilde{Ra}$. Conversely, for a fixed value of
$\widetilde{Ra}$, decreasing $Pm$ typically yields larger values of
$\overline{M}/M$. Fig. 3(b) shows that a collapse of the data occurs when
$\overline{M}/M$ is plotted versus $\widetilde{Rm}$. In particular, we find
energetically robust large scale dynamos, as characterized by
$\overline{M}/M\rightarrow 1$, only when $\widetilde{Rm}\lesssim O(1)$.

We note that the data shown in Fig. 3(b) also demonstrates that dynamo action
is achieved for smaller values of $\widetilde{Rm}$ as both $E$ and $Pm$ are
reduced. This reduction in the value of $\widetilde{Rm}$ needed for dynamo
action as the parameter values are made more extreme is because these dynamos
are intrinsically multiscale and anisotropic; convective motions take place
over the depth $H$ of the system, yet the small $O(HE^{1/3})$ horizontal
length scale of the convection is crucial to these dynamics. Although
$\widetilde{Rm}$ is becoming smaller as both $E$ and $Pm$ are reduced, the
large scale magnetic Reynolds number scales as $Rm=O(E^{-1/6})$, and therefore
becomes large even as $\widetilde{Rm}=O(E^{1/6})$ becomes small as
$E\rightarrow 0$.

Figure 4:  The fraction of the mean magnetic energy to the total magnetic
energy $\overline{M}/M$ versus the rms relative helicity
$\overline{\mathcal{H}}_{r}$. The symbols have the same meaning as in Fig. 1.

The relationship between the mean energy fraction and relative helicity is
shown in Fig. 4. We find that robust large scale dynamo action is achieved at
smaller values of the relative helicity as both $E$ and $Pm$ are reduced.
These findings show that small relative helicity by itself does not imply that
large scale dynamo action is not achievable, and that a deficit of helicity
can be offset by enhancing the influence of magnetic diffusion on the small
convective length scale. A possible explanation for these observations is that
the stretching-diffusion balance present in the small scale induction equation
prevents small scale magnetic field from cascading to smaller length scales.
We note that simulations of non-rotating helical turbulence have found a
similar effect in which large scale dynamos can be maintained with decreasing
helicity so long as the forcing wavenumber is increased [23].

In comparison to previous studies of rotating dynamos in the plane parallel
geometry, the present investigation extends the parameter space to smaller
Ekman numbers and smaller magnetic Prandtl numbers. Refs. [24, 25] find that
so-called large scale vortices (LSVs), generated by the inverse kinetic energy
cascade mechanism, can play an important role in sustaining large scale
magnetic field. However, none of the cases reported in Refs. [24, 25] show
evidence of what we refer to as the energetically robust large scale dynamo
regime in which $\overline{M}/M\approx 1$. This finding is consistent with the
trends observed in our simulations. Moreover, no general conclusion on the
importance of LSVs in our simulations could be reached; whereas some cases
show relatively strong LSVs, in many of the cases reported they are not
energetically dominant, similar to the findings of Ref. [26]. Ref. [9] found
transitions in dynamo behavior for $Pm\geq 1$ when $\widetilde{Rm}\gtrsim 13$,
though we do not observe a similar behavior that may be due to the smaller
Ekman and Rossby numbers employed here. Ref. [27] finds large scale dynamos
for sufficiently small Ekman numbers and $Pm\geq 1$, though energetically
robust large scale dynamos are not observed.

The combination of the plane parallel geometry and the Boussinesq
approximation has enabled a systematic exploration of parameter space in which
$E\ll 1$ and $Pm<1$. Sphericity and compressibility are both important
physical effects in natural systems such as stars and planets. However, due to
their global nature, spherical simulations require enormous resolution
requirements as the Ekman number reduced [e.g. 28]. Small Mach number
compressible convection simulations are regularly used [27, 29, 26], but the
increased computational cost relative to Boussinesq simulations nevertheless
prevents such studies from accessing values of $E$ and $Pm$ comparable to
those studied here. We note that neither compressibility [30] nor sphericity
[28] influence the leading order balances on the small convective length
scales, suggesting that the distinguished limit identified in Ref. [10] and
studied here may still play the key role in controlling the relative size of
the large scale magnetic field.

The authors gratefully acknowledge funding from the National Science
Foundation (NSF) through grants EAR-1620649 (MY and MAC), EAR-1945270 (MAC)
and SPG-1743852 (MAC). This work used the Extreme Science and Engineering
Discovery Environment (XSEDE) Stampede2 supercomputer at the Texas Advanced
Computing Center (TACC) through allocation PHY180013. Additional computing
resources were provided on the RMACC Summit supercomputer, which is supported
by the NSF (awards ACI-1532235 and ACI-1532236), the University of Colorado
Boulder, and Colorado State University. Flow visualization was performed with
VAPOR [31].

## References

  * Jones [2011] C. A. Jones, Planetary magnetic fields and fluid dynamos, Annu. Rev. Fluid Mech. 43, 583 (2011).
  * Tobias [2021] S. M. Tobias, The turbulent dynamo, J. Fluid Mech. 912, P1 (2021).
  * Parker [1955] E. N. Parker, Hydromagnetic dynamo models, Astrophys. J. 122, 293 (1955).
  * Steenbeck  _et al._ [1966] M. Steenbeck, F. Krause, and K.-H. Rädler, A calculation of the mean electromotive force in an electrically conducting fluid in turbulent motion, under the influence of Coriolis forces, Z. Naturforsch. 21a, 369 (1966).
  * Moffatt [1972] H. K. Moffatt, An approach to a dynamic theory of dynamo action in a rotating conducting fluid, J. Fluid Mech. 53, 385 (1972).
  * Moffatt [1970] H. K. Moffatt, Dynamo action associated with random inertial waves in a rotating conducting fluid, J. Fluid Mech. 44, 705 (1970).
  * Christensen and Aubert [2006] U. Christensen and J. Aubert, Scaling properties of convection-driven dynamos in rotating shells and applications to planetary magnetic fields, Geophys. J. Int. 166, 97 (2006).
  * Cattaneo and Hughes [2006] F. Cattaneo and D. W. Hughes, Dynamo action in a rotating convective layer, J. Fluid Mech. 553, 401 (2006).
  * Tilgner [2012] A. Tilgner, Transitions in rapidly rotating convection driven dynamos, Phys. Rev. Lett. 109, 248501 (2012).
  * Favier and Bushby [2013] B. Favier and P. J. Bushby, On the problem of large-scale magnetic field generation in rotating compressible convection, J. Fluid Mech. 723, 529 (2013).
  * Calkins  _et al._ [2015a] M. A. Calkins, K. Julien, S. M. Tobias, and J. M. Aurnou, A multiscale dynamo model driven by quasi-geostrophic convection, J. Fluid Mech. 780, 143 (2015a).
  * Calkins  _et al._ [2017] M. A. Calkins, K. Julien, and S. M. Tobias, Inertia-less convectively-driven dynamo models in the limit of low Rossby number, Phys. Earth Planet. Int. 266, 54 (2017).
  * Julien  _et al._ [2012] K. Julien, A. M. Rubio, I. Grooms, and E. Knobloch, Statistical and physical balances in low Rossby number Rayleigh-Bénard convection, Geophys. Astrophys. Fluid Dyn. 106, 392 (2012).
  * Guervilly  _et al._ [2019] C. Guervilly, P. Cardin, and N. Schaeffer, Turbulent convective length scale in planetary cores, Nature 570, 368 (2019).
  * Aurnou  _et al._ [2020] J. M. Aurnou, S. Horn, and K. Julien, Connections between nonrotating, slowly rotating, and rapidly rotating turbulent convection transport scalings, Phys. Rev. Res. 2, 043115 (2020).
  * Marti  _et al._ [2016] P. Marti, M. A. Calkins, and K. Julien, A computationally efficent spectral method for modeling core dynamics, Geochem. Geophys. Geosys. 17, 3031 (2016).
  * Yan  _et al._ [2019] M. Yan, M. A. Calkins, S. Maffei, K. Julien, S. M. Tobias, and P. Marti, Heat transfer and flow regimes in quasi-static magnetoconvection with a vertical magnetic field, J. Fluid Mech. 877, 1186 (2019).
  * Vogt  _et al._ [2021] T. Vogt, S. Horn, and J. M. Aurnou, Oscillatory thermal–inertial flows in liquid metal rotating convection, J. Fluid Mech. 911 (2021).
  * Calkins [2018] M. A. Calkins, Quasi-geostrophic dynamo theory, Phys. Earth Planet. Int. 276, 182 (2018).
  * Schmitz and Tilgner [2010] S. Schmitz and A. Tilgner, Transitions in turbulent rotating Rayleigh-Bénard convection, Geophys. Astrophys. Fluid Dyn. 104, 481 (2010).
  * Soderlund  _et al._ [2012] K. M. Soderlund, E. M. King, and J. M. Aurnou, The influence of magnetic fields in planetary dynamo models, Earth Planet. Sci. Lett. 333-334, 9 (2012).
  * Calkins  _et al._ [2016] M. A. Calkins, L. Long, D. Nieves, K. Julien, and S. M. Tobias, Convection-driven kinematic dynamos at low Rossby and magnetic Prandtl numbers, Phys. Rev. Fluids. 1, 083701 (2016).
  * Graham  _et al._ [2012] J. P. Graham, E. G. Blackman, P. D. Mininni, and A. Pouquet, Not much helicity is needed to drive large-scale dynamos, Phys. Rev. E 85, 066406 (2012).
  * Guervilly  _et al._ [2015] C. Guervilly, D. W. Hughes, and C. A. Jones, Generation of magnetic fields by large-scale vortices in rotating convection, Phys. Rev. E 91, 041001 (2015).
  * Guervilly  _et al._ [2017] C. Guervilly, D. W. Hughes, and C. A. Jones, Large-scale-vortex dynamos in planar rotating convection, J. Fluid Mech. 815, 333 (2017).
  * Bushby  _et al._ [2018] P. J. Bushby, P. J. Käpylä, Y. Masada, A. Brandenburg, B. Favier, C. Guervilly, and M. J. Käpylä, Large-scale dynamos in rapidly rotating plane layer convection, Astron. Astrophys. 612, A97 (2018).
  * Käpylä  _et al._ [2009] P. J. Käpylä, M. J. Korpi, and A. Brandenburg, Large-scale dynamos in rigidly rotating turbulent convection, Astrophys. J. 697, 1153 (2009).
  * Schaeffer  _et al._ [2017] N. Schaeffer, D. Jault, H.-C. Nataf, and A. Fournier, Turbulent geodynamo simulations: a leap towards earth’s core, Geophys. J. Int. 211, 1 (2017).
  * Favier and Proctor [2013] B. Favier and M. R. E. Proctor, Kinematic dynamo action in square and hexagonal patterns, Phys. Rev. E 88, 053011 (2013).
  * Calkins  _et al._ [2015b] M. A. Calkins, K. Julien, and P. Marti, The breakdown of the anelastic approximation in rotating compressible convection: implications for astrophysical systems, Proc. Roy. Soc. A 471 (2015b).
  * Li  _et al._ [2019] S. Li, S. Jaroszynski, S. Pearse, L. Orf, and J. Clyne, VAPOR: A visualization package tailored to analyze simulation data in earth system science, Atmosphere 10, 488 (2019).


Membership Inference Attacks and Defenses in Neural Network Pruning
Xiaoyong Yuan, Lan Zhang

Michigan Technological University

Neural network pruning has been an essential technique to reduce the computation and memory requirements for using deep neural networks for resource-constrained devices. Most existing research focuses primarily on balancing the sparsity and accuracy of a pruned neural network by strategically removing insignificant parameters and retraining the pruned model. Such efforts on reusing training samples pose serious privacy risks due to increased memorization, which, however, has not been investigated yet.

In this paper, we conduct the first analysis of privacy risks in neural network pruning. Specifically, we investigate the impacts of neural network pruning on training data privacy, , membership inference attacks. We first explore the impact of neural network pruning on prediction divergence, where the pruning process disproportionately affects the pruned model’s behavior for members and non-members. Meanwhile, the influence of divergence even varies among different classes in a fine-grained manner. Enlighten by such divergence, we proposed a self-attention membership inference attack against the pruned neural networks. Extensive experiments are conducted to rigorously evaluate the privacy impacts of different pruning approaches, sparsity levels, and adversary knowledge. The proposed attack shows the higher attack performance on the pruned models when compared with eight existing membership inference attacks.
In addition, we propose a new defense mechanism to protect the pruning process by mitigating the prediction divergence based on KL-divergence distance, whose effectiveness has been experimentally demonstrated to effectively mitigate the privacy risks while maintaining the sparsity and accuracy of the pruned models.

§ INTRODUCTION

Much of the progress in artificial intelligence over the past decade has been the result of deep neural networks (DNNs). The powerful DNNs with a large number of parameters consume considerable storage and memory bandwidth, which makes it challenging to deploy the state-of-the-art neural networks on resource-constrained devices.
To address this issue, neural network pruning as one of the most popular compression technologies has attracted great attention <cit.>.
By removing insignificant parameters from a DNN, recent research has shown that neural network pruning can substantially reduce the size of a DNN and speedup the inference process without largely compromising prediction accuracy <cit.>.

In general, neural network pruning includes three main stages: 1) train an original DNN; 2) remove the insignificant parameters; 3) fine-tune the remaining parameters with the training dataset. Most existing research on neural network pruning has focused on improving the trade-off between accuracy and sparsity by strategically design the last two stages <cit.>. However, such efforts on reusing training samples pose serious privacy risks of the pruned neural networks due to the potentially increased memorization of training samples.

Original model confidence

Original model sensitivity

Pruned model confidence

Pruned model sensitivity

Histograms of the prediction confidences and the prediction sensitivity of the ground-truth label. We remove 70% of the parameters in the original DenseNet121 model using l1 unstructured pruning on the CIFAR10 dataset. The figures show the frequency of prediction confidence (a) and (c) and prediction sensitivity (b) and (d) belonging to the ground-truth class on the training and test data. The vertical lines indicate the average values of training data, , members (black), and test data, , non-members (red), respectively. In both prediction confidence and sensitivity measurements, neural network pruning makes the distances between the two vertical lines in the pruned model larger than that in the original model, which indicates a larger confidence gap and sensitivity gap between members and non-members due to pruning.

The privacy risks of DNNs have already been pointed out, where a DNN is prone to memorizing sensitive information of the training dataset <cit.>. Taking the membership inference attack (MIA) as an example, an adversary can infer whether a given data sample was used to train a DNN, seriously threatening the individual privacy. For instance, an adversary can infer an individual was a confirmed case, if it is known that the individual's record was used to train an infectious disease model. The MIA was first proposed against black-box models in <cit.>, where the adversary only has access to the data sample and predictions of the target model. Later on, more attention has been attracted against various DNN models, such as generative models <cit.>, graph models <cit.>, machine translation <cit.>, text generation <cit.>, genomic analysis <cit.>, and transfer learning <cit.>. Although extensive analysis has been conducted, none of the existing efforts have been put into analyzing MIAs against pruned neural networks.

In view of this, the paper focuses on one fundamental question: comparing with original deep neural networks, are the pruned networks more vulnerable to membership inference attacks? Specifically, most MIAs infer a sample's membership based on the different behaviors of a target model between members (, training samples) and non-members (, test samples), such as the different prediction confidences <cit.>. Since most neural network pruning approaches rely on reusing training dataset to fine-tune the parameters after pruning the insignificant parameters, the additional training at the pruned neural network inevitably increases its memorization of the training samples. Moreover, the pruned neural network enforces a small number of parameters to achieve similar prediction capabilities, which also increases the memorization of training data and makes the pruned model more sensitive to the training data. Hence, such increased memorization can intuitively lead to a larger divergence of the prediction confidences and sensitivities between members and non-members.
Figure <ref> illustrates the prediction confidence and the prediction sensitivity[The definitions of the prediction confidence and the prediction sensitivity are detailed respectively in Section <ref>.] of members and non-members in the original DNN and the pruned network, respectively.

The larger divergence of the confidences and the sensitivities in the pruned model at (c) and (d) confirms our intuition: neural network pruning can aggravate the privacy issues of the original deep neural network. Therefore, in the following paper, we conduct a comprehensive analysis to reveal the impacts of neural network pruning on training data privacy, , MIAs. Specifically, we first explore the impact of neural network pruning on prediction divergence: the pruning process disproportionately affects the pruned model's behavior for members and non-members.

Enlighten by this insight, a new MIA is proposed against the pruned neural networks. In addition, with the proposed new attack, we propose a new defense mechanism to protect the fine-tuning process by mitigating the prediction divergence based on KL-divergence distance. Extensive experiments are conducted to rigorously evaluate our proposals. To the best of our knowledge, this is the first study to investigate the privacy risks of neural network pruning. Our main contributions are summarized below:

* We investigate the privacy risk of neural network pruning and propose a new MIA: self-attention membership inference attack (SAMIA). By exploring the impacts of neural network pruning on prediction divergence, the proposed attack results in high attack accuracy of revealing the membership status from the pruned models. In particular, SAMIA has advantages in identifying the pruned models' prediction divergence by using finer-grained prediction metrics. We recommend SAMIA as a competitive baseline attack model for future privacy risk study of neural network pruning.

* To rigorously evaluate the privacy impacts of different pruning approaches, sparsity levels and adversary knowledge, we conduct extensive experiments on seven commonly used datasets, four neural network architectures, four pruning approaches, five sparsity levels, and 255 pruned models in total. Experimental results demonstrate the effectiveness of the proposed attacks against pruned neural networks, which further indicates that neural network pruning can aggravate the privacy issues of the original DNN. The adversary can successfully reveal the membership status, even without the knowledge of the pruning approach used in the target model. Furthermore, we evaluate the privacy impacts of different pruning approaches and various sparsity levels.

To defend the pruned models against MIAs, we propose a new defense mechanism: pair-based posterior balancing (PPB). PPB protects the fine-tuning process of neural network pruning by narrowing down the divergences of posterior predictions and reducing the prediction sensitivities based on their KL-divergence distances.
Experimental results demonstrate the effectiveness of the PPB mechanism, which significantly mitigates the privacy risks while maintaining the sparsity and accuracy of the pruned model.
Besides, compared with the state-of-the-art defenses, PPB achieves a better trade-off between prediction performance and privacy in most cases.

§ BACKGROUND AND RELATED WORK

§.§ Neural Network Pruning

The state-of-the-art neural networks are usually deep and resource hungry, requiring large amounts of computation and memory, which becomes a particular challenge on resource-constrained end devices. As one of the most popular network compression approaches, neural network pruning has attracted great attention in recent years <cit.>. In general, most network pruning studies follow the pruning workflow: "train-prune-finetuning." For example, Han  <cit.> proposed to remove the individual parameters with the lowest magnitude. Randomly removing individual parameters reduces the model size, but may not be efficient to facilitate hardware optimization and accelerate the neural network computation. Therefore, many methods were proposed to remove parameters in an organized way by removing a group of parameters (, structured pruning). For example, Li  <cit.> removed the entire filters with the lowest magnitude in the neural network, which leads to significant speedup compared with the unstructured pruning.
Liu  <cit.> removed the entire channels according to the corresponding scaling factors in the followed batch normalization layers. In this paper, we investigate the privacy risks of both unstructured and structured pruning approaches.

More recently, new pruning approaches have been proposed, which prune parameters by searching the optimal neural architecture <cit.> or fine-tune the pruned model by rewinding the parameters to the previous states <cit.>. The privacy risks discussed in this paper might exist in these new pruning approaches.
We will investigate their privacy risks in our future work.

On the other hand, recent efforts have been put into neural network pruning from other important perspectives. Paganini <cit.> investigated the unfairness and systematic biases in the pruned models. Hooker  <cit.> demonstrated the biased performance on different groups and classes after pruning. Given the potential of pervasively implementing neural network pruning, this work targets on another critical and urgent aspect regarding neural network pruning, ,  training data privacy.

§.§ Membership Inference Attacks (MIAs)

Membership inference attacks have raised serious privacy threats by determining if a record was in the training dataset of a neural network model via querying that model. Given a target neural network model $f: \mathbb{R}^n \rightarrow \mathbb{R}$, the process of MIA can be formally defined as:

\begin{equation}
\mathcal{A}: \bm{x}, f \rightarrow \{0, 1\},\vspace{-0.5em}
\end{equation}

where $\mathcal{A}$ denotes the attack model, which is a binary classifier. If the data sample $\bm{x}$ is used to train the target model $f$ , the attack model $\mathcal{A}$ outputs $1$ (, member), and $0$ otherwise (, non-member).

Due to the practical consideration, most MIAs focused on the black-box setting, where an adversary only has access to the target model's outputs. By leveraging the target model's prediction confidences, Shokri , <cit.> proposed a black-box MIA. They constructed several shadow models to mimic the behavior of a target model. The well-established shadow models will then be used to generate data to train a neural network-based binary classifier to determine the membership of a record against the target model, , whether a record belongs to the target model's training dataset or not. Salem , <cit.> further boosted this attack successfully by only using a single shadow model. To further improve the attack accuracy, Nasr , <cit.> included more features, such as the class labels of data samples, to train the binary classifier. In addition to the aforementioned neural network-based binary classifier, Leino , <cit.>, Yeom , <cit.>, and Song , <cit.> proposed the metric-based binary classifier, where the membership of a record is directly determined by a predefined threshold based on the metrics, such as the prediction confidences, entropy, or modified entropy of the record.

Song and Mittal showed that by setting a class-dependent threshold, the metric-based classifier could achieve comparable or even better accurate inference performance compared with the neural network-based classifier <cit.>.
Despite the extensive research on MIAs, none of them is designed towards pruned models. Therefore, we propose SAMIA to investigate the privacy risks of pruned models.

§.§ Defenses against MIAs

Recent efforts have been made to defend against MIAs. As one of the most popular privacy-preserving techniques, differential privacy (DP) provides provable defense against MIAs by adding noise to the gradient or parameter during model training <cit.>. However, DP usually requires a large magnitude of noises to achieve a meaningful privacy guarantee, which seriously degrades the performance of the protected models <cit.>.
On the other hand, regularization <cit.>, dropout, and model stacking <cit.> have been used in model training to reduce the privacy risks caused by overfitting. Although these approaches reduced the vulnerability by bridging the generalization gap between member and non-member data samples, in many cases, the privacy risks after applying these approaches are still high.
Recent adversarial learning techniques <cit.>,
has been introduced in defending against MIAs by adding noises to the prediction confidences for misleading the adversary <cit.>.

In a recent analysis of the defense mechanisms, Song and Mittal showed that the early stopping mechanism achieved comparable performance with most defenses <cit.>.
In this paper, we provide a comprehensive analysis of defenses in neural network pruning, including our proposed PPB defense along with the existing defense mechanisms.

§ SYSTEM OVERVIEW

A typical workflow of neural network pruning.

§.§ Neural Network Pruning Workflow

This paper is focused on a general neural network pruning process, whose workflow includes three key stages: original network training, coarse pruning, and fine-tuning, as illustrated in Figure <ref>. Specifically,

* Original network training: A large size original neural network model $f(\bm{x}; \bm{W})$ (sometime over-parameterized) is first trained at this stage, where $\bm{x}$ is the training data and $\bm{W}$ is the model parameters;

* Pruning: Upon the original network, the pruning is conducted by removing insignificant parameters or groups of parameters according to a specific criterion. The pruned network can be given by $f(\bm{x}; \bm{M}\odot \bm{W})$, where $\bm{M}\in {0, 1}^{|\bm{W}|}$ denotes the binary mask that can set a parameter to be 0, and $\odot$ denotes the element-wise multiplication;

* Fine-tuning: To recover the performance loss due to pruning, a pruned network can be fine-tuned by reusing the training data. After $N$-epoch fine-tuning, a pruned network can be given by $f(\bm{x}; \bm{M}\odot \bm{W}^N)$.

For the sake of simplicity, we use $f$ to denote the original model $f(\bm{x}; \bm{W})$ and $f_p$ to denote the pruned model $f(\bm{x}; M\odot \bm{W}^N)$ in the following paper.

§.§ Adversarial Knowledge

The goal of MIAs is to find the membership of a data sample, , whether the sample is used to train a target model or not.  In this paper, we assume the adversary of the MIAs against a pruned neural network has the following knowledge.

* Access to query the pruned network. The pruned model is made available to the public, , queryable. Due to practical considerations, the original model is assumed not published and inaccessible.

* Access to the prediction confidences. We consider the practical black-box MIAs <cit.>. The adversary can only acquire the output, , the prediction confidences, of the pruned network. Any internal information about the pruned model and the original model, such as the network architecture and activation functions, are inaccessible to the adversary.

* Access to the pruning approach and the sparsity level. We consider two different types of adversaries with or without knowledge of the pruning approach and the sparsity level.

* Access to the defense approach.
The arms race between attacks and defenses is one main challenge in machine learning privacy. If the defense mechanisms are designed without considering the adversary's knowledge, their performance might be substantially degraded when adaptive attacks are used against those defensive mechanisms <cit.>. Hence, we consider both non-adaptive and adaptive attacks to evaluate defense mechanisms: 1) non-adaptive attacks, , the adversary has no access to the defense mechanisms; 2) adaptive attacks, where the adversary has full knowledge of the defense mechanisms and performs the MIAs by taking the defensive mechanisms into account.

§ MIA AGAINST NEURAL NETWORK PRUNING

Framework of membership inference attacks (MIA) against neural network pruning.

Given the workflow of neural network pruning presented in Section <ref>, this section focuses on investigating the privacy risks introduced by the pruning process. A general framework of MIAs against the pruned model is illustrated in Figure <ref>. Specifically, to extract the membership information from the pruned model, the adversary first derives the predictions of the given input sample by querying the target pruned model. The adversary then feeds the predictions into the trained attack model and provides the binary classification of the membership status.
The attack model is derived following the shadow-training technique, which was originally proposed by Shokri  <cit.> and is widely used in MIAs <cit.>: a shadow model is trained and pruned to imitate the behavior of the target pruned model. The adversary trains an attack model based on the pruned shadow model's predictions over shadow training and test data.

Original Model Confidence

Original Model Sensitivity

Pruned Model Confidence

Pruned Model Sensitivity

Divergence of the pruned model's prediction confidences and prediction sensitivities over different classes, respectively. We prune DenseNet121 models with 70% sparsity on the CIFAR10 datasets. The blue bar indicates the average prediction confidence/sensitivity of members in different classes. The green bar indicates the average prediction confidence/sensitivity of non-members. The divergence of confidence (a) and sensitivity (b) between members and non-members is increased after pruning as shown in (c) and (d), respectively. Such divergence also differs among classes.

§.§ Divergence of Prediction Behaviors

To investigate the prediction behaviors of pruned neural networks, we first introduce two metrics: prediction confidence and prediction sensitivity. Specifically, given an input sample $\bm{x}$ and an pruned model $f_p$, the prediction confidence is defined as $\mathrm{PC} = f_p(\bm{x})$.
In addition, to further measure the prediction behavior changes in terms of slight input change, we introduce prediction sensitivity, which is defined as

\begin{equation}\label{eq:sensitivity}
\mathrm{PS} = \frac{1}{n} \sum_{i=1}^n  \frac{|f_p(\bm{x} + \epsilon\bm{\delta_i}) - f_p(\bm{x})|}{\epsilon},
\end{equation}

where $\bm{\delta_i} \sim \mathcal{N}(0, 1)$ is a random Gaussian noise vector added to the input data $\bm{x}$, and  $\epsilon$ controls the magnitude of input changes.
Similar idea has been used in the gradient estimation for black-box adversarial attacks <cit.>. It has been shown that a small number of noise vectors can achieve good estimation of prediction changes, so that we set a small query budget in the evaluation ($n=10$) <cit.>.
Accordingly, we use the confidence and sensitivity to measure the divergence between members and non-members. We define the confidence gap as

\begin{equation}
\frac{1}{|\mathcal{D}_{train}|}\sum_{(\bm{x_i}, y_i)\in \mathcal{D}_{train}}f_p^{y_i}({\bm{x_i}}) - \frac{1}{|\mathcal{D}_{test}|}\sum_{(\bm{x_i}, y_i)\in \mathcal{D}_{test}}f_p^{y_i}(\bm{x_i}),
\end{equation}

where $f_p^{y_i}$ denotes the prediction confidence of ground-truth class $y_i$.
Confidence gap calculates the difference of average confidence between members and non-members in the ground-truth class.
Similarly, we define the sensitivity gap as

\begin{equation}
\frac{1}{|\mathcal{D}_{train}|}\sum_{(\bm{x_i}, y_i)\in \mathcal{D}_{train}}\mathrm{PS}^{y_i}(\bm{x_i}) - \frac{1}{|\mathcal{D}_{test}|}\sum_{(\bm{x_i}, y_i)\in \mathcal{D}_{test}}\mathrm{PS}^{y_i}(\bm{x_i}),
\end{equation}

where $\mathrm{PS}^{y_i}$ denotes the prediction sensitivity (Eq. <ref>) of ground-truth class $y_i$. The sensitivity gap calculates the difference of average sensitivity between members and non-members in the ground-truth class.

As illustrated in Figure <ref>, the divergence of prediction confidences and prediction sensitivities is increased due to neural network pruning, which introduces the new attack vectors for MIAs and thus makes the pruned models more vulnerable. Moreover, the divergences of prediction confidences and sensitivities from the pruned model vary widely among the different classes of training and test data.
Figure <ref>  shows that the divergences of the pruned models' prediction behavior (confidence and sensitivity) over members and non-members are significantly different among classes. Similar observations of prediction confidences on different classes after pruning have been made in other fields such as model fairness and transparency <cit.>.

§.§ SAMIA: Self-Attention MIA

Upon the above observations, we propose one hypothesis: the divergences among classes, , confidence gap and sensitivity gap, can provide fine-grained “evidence” for MIAs, leading to serious privacy leakage. In addition, most existing MIA research only considers the confidence gap and a single threshold of the ground-truth class, which may underestimate the privacy risks of MIAs in neural network pruning. Hence, we propose SAMIA, a self-attention MIA, to fully utilize the increased divergence information along with the class information to conduct a finer-grained analysis. Specifically, self-attention is a neural network module to capture global dependencies among inputs and allows the inputs to interact with each other. Despite the recent success of self-attention mechanism in many areas, such as natural language processing <cit.> and computer vision <cit.>, it has not been well exploited in the research of privacy attacks yet.

In SAMIA, we leverage the self-attention mechanism to automatically extract the finer-grained “thresholds” from different classes by capturing the dependency between predicted information (confidence and sensitivity) and class information and allowing them to interact with each other. Specifically, SAMIA takes the pruned model's prediction confidence and sensitivity and ground-truth labels as inputs.  Given a specific class, the self-attention mechanism finds out the specific confidence information and sensitivity information that the attack “threshold” should pay more “attention” to.

Attack model architecture in SAMIA.

Figure <ref> illustrates the network architecture of the attack model used in SAMIA, enlighten by the idea of Transformer <cit.>, , one of the most widely used self-attention architectures. We first convert the ground-truth label into a one-hot vector and then feed both pruned model's prediction confidence, sensitivity and the one-hot vector into the attack model as the input features.
The input features are encoded into a vector using a Fully Connected (FC) layer, which are then fed into the multi-head self-attention modules. In each module, we encode the features as query $Q$, key $K$, and value $V$ vectors using a linear function following the self-attention strategy. The attention module calculates the attention scores of the subgroups in a scaled dot-product way:
$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}(QK^T)V,$

where $\mathrm{softmax}()$ denotes the softmax function to make the attention scores sum up to $1$.
The output of the attention module is the weighted sum of the value vector, where the weight assigned to each value is derived by the attention scores $\mathrm{softmax}(QK^T)$. In addition, we calculate four attention scores (, multi-head attention) to capture the different attention strategies. Followed by the attention module, we add the result to the input features and apply the layer normalization <cit.> to stabilize the attack model training. The result will be fed into another FC layer with layer normalization. We consider these operations as a block and repeat the block for three times, followed by two fully connected layers.
A non-linear activation function, ReLU is applied to the output of the first few FC layers. A softmax function is applied to the last FC layer to provide the binary prediction on the membership status.

Compared with existing MIAs that learn a single threshold of prediction confidence to determine the membership, the proposed SAMIA captures the information of confidences and sensitivities and intuitively better learns the diverse thresholds to multiple classes. Our evaluation results demonstrate that SAMIA leads to higher attack accuracy compared with the state-of-the-art attacks.

§ ATTACK EVALUATION

This section conducts comprehensive experiments[Due to the space limit, we only present the major results in main paper. More details can be found in Appendix.] to thoroughly investigate the privacy risks of the proposed MIAs against neural network pruning. In the following, we first introduce the experimental setup, and then evaluate the privacy risks of the pruned models by comparing them with those of original models. Next, we investigate the impact of the confidence gap, sensitivity gap, and generalization gap, respectively. Finally, we evaluate the privacy risks without the knowledge of pruning approaches and sparsity levels.

§.§ Evaluation Setup

In the evaluation, we consider the most widely used datasets, neural network architectures, and optimization approaches following recent research of MIAs <cit.>.

CIFAR10, ResNet18

CIFAR10, DenseNet121

CIFAR100, DenseNet121

Location, FC

Prediction accuracy (test accuracy) of the pruned models using different pruning approaches and sparsity levels. Each point indicates the prediction accuracy achieved by the pruned model with a certain pruning approach and sparsity level. The black line indicates the prediction accuracy of the original models.

§.§.§ Datasets

We consider seven popular datasets in the experiments: CIFAR10, CIFAR100, CHMNIST, SVHN, Texas, Location, and Purchase.

* CIFAR10 and CIFAR100 <cit.>. These are two benchmark datasets for image classification. CIFAR10 dataset contains 60,000 $32\times 32$ color images in 10 classes, with 6,000 images per class. CIFAR100 dataset contains 60,000 color images in 100 classes, with 600 images per class.

* CHMNIST <cit.>. This dataset consists of 5,000 histological images of human colorectal cancer containing 10 classes of tissues. We resize all images to $32\times 32$, the same dimension as CIFAR10 and CIFAR100.

* SVHN <cit.>. This dataset consists of 99,289 $32\times 32$ color images from house numbers in the Google Street View dataset, containing 10 classes from 0 to 9.

* Location <cit.>. This dataset contains location “check-in” records of mobile users in the Foursquare social network, restricted to the Bangkok area. The dataset is used to predict users' geosocial type based on the geographical history record features: whether the user visited a certain region or location type. We use the preprocessed purchase dataset provided by Shokri  <cit.>, which contains 5,010 data samples, 446 binary features, and 30 classes.

* Texas <cit.>. This dataset is presented in the Hospital Discharge Data Public Use Data File provided by the Texas Department of State Health Services. The dataset is used to predict the types of patient's main procedure based on a wide range of features, such as external causes of injury, diagnosis of the patient, procedures the patient underwent, and other generic information. We use the preprocessed purchase dataset provided by <cit.>, which contains 67,330 data samples, 6,169 binary features, and 100 classes.

* Purchase <cit.>. This dataset is presented in Acquire Valued Shoppers Challenge to predict which shoppers will become repeat buyers based on the purchase history. We use the preprocessed purchase dataset provided by Shokri  <cit.>, which contains 197,324 data samples, 600 binary features, and 100 classes.

Each above dataset is first randomly and equally split into two parts: one for target model, one for shadow model. In each part, we split the data into three datasets: training (45%), validation (10%), and test (45%). We use the validation dataset to determine if the model needs to stop training or fine-tuning for early stopping.
Therefore, the membership inference via random guessing results in 50% attack accuracy.
Due to the space limit, we only show the results of the CIFAR10 and Purchase datasets. The rest results are presented in the Appendix.

§.§.§ Neural Network Architectures

For the four image datasets, , CIFAR10, CIFAR100, CHMNIST, and SVHN, we consider three representative neural network architectures: ResNet18, VGG16, and DenseNet121[All neural networks are trained using <https://github.com/huyvnphan/PyTorch_CIFAR10>]. For the other three datasets, , Texas, Purchase, and Location, we implement fully connected (FC) neural networks with two layers, and the numbers of neurons for each layer are 256 and 128, respectively. All the FC layers except the last one are followed by ReLU activation functions.  In addition, Adam optimizer <cit.> is implemented with a learning rate of 0.001 and the batch size of 128 to train all the original models and fine-tune all the models after pruning.

§.§.§ Neural Network Pruning Approaches

Four representative neural network pruning approaches are considered, including L1 unstructured pruning,  L1 structured pruning, L2 structured pruning, and Network slimming.

* L1 unstructured pruning <cit.> (L1 unstructured), which removes the weights with the lowest absolute values individually. This pruning approach can produce a sparse neural network with a small size, but may not improve efficiency given the existing hardware and software optimization.

* L1 structured pruning <cit.> (L1 structured), which removes the entire filters with the lowest absolute values from the convolution layers. By removing the entire filters, this method leads to significant speedup compared with the unstructured pruning since optimization for dense matrix can be applied for efficient computation.

* L2 structured pruning (L2 structured), which removes the entire filters with the lowest L2 norm values from the convolution layers, similar to L1 structured pruning.

* Network slimming <cit.> (Slimming), which associates scaling factors used in the batch normalization layer with each channel and removes the entire channels with the lowest scaling factors. This method automatically identifies the insignificant channels and finds the target architectures.

We apply the L1 unstructured pruning to all models.
Since structured pruning approaches, , L1 structured and L2 structured pruning and Slimming, can only be applied to pruning convolution layers, we evaluate the structured pruning approaches on the ResNet18, VGG16, and DenseNet121 models trained on CIFAR10, CIFAR100, SVHN datasets. In addition, five sparsity levels $\gamma=\{0.5, 0.6, 0.7, 0.8, 0.9\}$ are investigated for all pruning approaches, which denote the portions of the removed parameters[Since structured pruning only removes the parameters in the convolution layers, the sparsity levels for structured pruning only count the removed parameters in the convolution layers instead of the entire neural network.]. We follow typical pruning procedures: train the original model, prune the model using the above approaches, and finally fine-tune the pruned model.

Figure <ref> shows the prediction accuracy of the original model and the pruned models with different pruning approaches and sparsity levels. We observe that the pruned models achieve close performance compared to the original model if the sparsity level is not high. The accuracy of pruned models is reduced with the increase of the pruning sparsity. Sometimes, pruned models can achieve higher accuracy than the original models, which has been shown in recent studies of neural network pruning <cit.>. Unstructured pruning usually performs better than structured pruning in the evaluation, since structured pruning forces the removed parameter in a restricted way, which limits the performance of the pruned model but increases the speed of model inference.

§.§.§ State-of-the-art MIAs

To thoroughly evaluate the proposed SAMIA, we investigate eight state-of-the-art MIAs along with SAMIA.
[We implement Conf, Xent, Mentr, and Top1-Conf attacks based on <https://github.com/inspire-group/membership-inference-evaluation> and BlindMI attack based on <https://github.com/hyhmia/BlindMI/blob/master/BlindMI_Diff_W.py>. ].

* Ground-truth class confidence-based threshold attack (Conf). Yeom used the prediction confidence of ground-truth class to identify membership status <cit.>. The adversary learns a threshold to determine the membership of a data sample based on the confidence of ground-truth class. Given an input sample $\bm{x}$, its class $y$, and the pruned model $f_p$, the attack function is defined as $I_{\mathrm{conf}}(f_p, (\bm{x},y)) = \mathbbm{1}\{f_p^{(y)}(\bm{x})\geq\zeta_y\}$, where $f_p^{(y)}$ is the prediction confidence of class $y$ and $\zeta_y$ is the threshold of class $y$ derived from the shadow pruned model.

* Cross-Entropy-based threshold attack (Xent). The entropy loss can be used to derive the threshold from the shadow pruned model <cit.>. The attack function is defined as $I_{\mathrm{xent}}(f_p, (\bm{x},y)) = \mathbbm{1}\{\mathrm{xent}(f_p^{(y)}(\bm{x}))\geq\zeta_y\}$, where $\mathrm{xent}$ denotes the cross entropy loss.

* Modified-entropy-based threshold attack (Mentr). Song and Mittal proposed modified entropy by including the information about the ground-truth class, which achieved better performance than using prediction confidence <cit.>. The attack function is defined as $I_{\mathrm{mentr}}(f_p, (\bm{x},y)) = \mathbbm{1}\{\mathrm{mentr}(f_p^{(y)}(\bm{x}))\geq\zeta_y\}$, where $\mathrm{mentr}(f_p(\bm{x}), y) = - (1 - f_p^{(y)}(\bm{x}))\log (f_p^{(y)}(\bm{x})) - \sum_{t\neq y}f_p^{(t)}(\bm{x})\log(1-f_p^{(t)}(\bm{x}))$.

* Top1 Confidence-based threshold attack (Top1-Conf). Salem proposed to derive the threshold from the highest prediction confidence <cit.>. The attack function is defined as $I_{\mathrm{top1}}(f_p, (\bm{x})) = \mathbbm{1}\{\mathrm{top1}(f_p(\bm{x}))\geq\zeta_y\}$, where $\mathrm{top1}$ calculates the highest value from the prediction confidence.

* Confidence-based Neural Network attack (NN). Shokri proposed to use prediction confidence as features to train a neural network from the shadow model <cit.>, which is used to distinguish member and non-member data.

* Top-3 Confidence-based Neural Network attack (Top3-NN). Salem proposed to use the top-3 prediction confidences as features <cit.> to train a neural network classifier.

* Confidence-based Neural Network attack with ground-truth class (NNCls). Nasr combined one-hot encoded class labels with the prediction confidence as features to train a neural network classifier <cit.>.

* Blind Membership Inference Attack (BlindMI). Hui proposed to determine the membership of a data sample by moving it to a non-member set and check if the moving operation increases the distance between member and non-member sets <cit.>. BlindMI considers the data sample as a non-member if the distance is increased. We use the default BlindMI attack provided in <cit.>.

In the main paper, we present the results of five attacks, that achieve the highest attack accuracies in most experiments, , Conf, Mentr, NNCls, BlindMI, and SAMIA. The results of the rest of the attacks are reported in the Appendix.

Besides, it should be mentioned that to provide a practical analysis of privacy risks, we adopt early stopping and l2 regularization as a baseline defense mechanism and apply it to all the following experiments of membership inference attacks. Other defense mechanisms will be discussed in Section <ref>.

§.§.§ SAMIA Settings

Following the experimental setting in <cit.>, we first train five shadow models and their pruned models. The predictions of the shadow models on shadow training and shadow test datasets are used to train an attack model. In the attack model, we use four attention heads, 64 neural units, and GeLU activation function <cit.> in each self-attention module, with a 20% dropout rate.
We use SGD optimizer <cit.> to train the attack models for 100 epochs with batch size 128. The learning rate of the SGD optimizer is set as 0.01 and reduced to 0.001 and 0.0001 at the 1/2 and 3/4 of the training process (, the 50th and 75th epoch).

Due to the large number of settings evaluated in the attacks and defenses and the high computational cost in each setting, we conduct all the experiments only once. Thus experimental variation may be observed due to the randomness in neural network pruning and membership inference attacks (, parameter initialization, dataset shuffling).

CIFAR10, ResNet18

CIFAR10, DenseNet121

CIFAR100, DenseNet121

Location, FC

Privacy Risks of Neural Network Pruning (w.r.t. prediction accuracy). Most pruning approaches results in a higher attack accuracy when considering a similar prediction accuracy, compared with the original models. We present the attack accuracy of SAMIA for pruned models and the attack accuracy of Conf attack for the original models.

CIFAR10, ResNet18

CIFAR10, DenseNet121

CIFAR100, DenseNet121

Location, FC

Privacy Risks of Neural Network Pruning (w.r.t. model sparsity). We present the attack accuracy of SAMIA for pruned models and the attack accuracy of Conf attack for the original models.

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (CIFAR10, ResNet18). We present the relationship between the gap and the attack accuracy of SAMIA.

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (CIFAR10, ResNet18). We present the attack accuracy of state-of-the-art membership inference attacks and compared them with the proposed SAMIA. We present the results of pruned CIFAR10 ResNet18 models using four pruning approaches. The black line presents the attack accuracy of original models using Conf attack, , Conf (Org.).

§.§ Privacy Risk Discussions

In this section, we evaluate the privacy risks of the pruned models and compare them with the original models and then investigate several key factors on privacy risks of neural network pruning. Additionally, we investigate the privacy risks of different pruning approaches and  discuss the effectiveness of the proposed SAMIA and the impact of unknown sparsity levels and pruning approaches.

§.§.§ Privacy Risks of Neural Network Pruning

Since different pruning approaches and sparsity levels may achieve distinct prediction accuracy, to make a fair comparison, we evaluate the privacy risks of pruning by taking the prediction accuracy into consideration.
Figure <ref> show the relationship between prediction accuracy and (SAMIA) attack accuracy when we apply different pruning approaches and sparsity levels on the CIFAR10, CIFAR100, and Location datasets.
We observe that when the pruned model achieves a comparable prediction accuracy with the original model, most pruning approaches result in an increased attack accuracy (, privacy risk).

The attack accuracy may be decreased with the loss of prediction accuracy, as the pruned model becomes less effective for both prediction and attack.
However, we still observe that in most cases, when the pruned model performs worse than the original model, the pruned model's attack accuracy remains higher than the original one's.
Therefore, the pruned models become more vulnerable to membership inference attacks than the original models.

When a low sparsity level is used, we always observe the increased privacy risk of the pruned model (Figure <ref>). Since with a low sparsity level, the pruned model is more likely to achieve a comparable or even higher prediction accuracy compared with the original model, which increases the accuracy of prediction confidence used in MIAs and further increases the privacy risk.

CIFAR10, DenseNet121

CIFAR10, DenseNet121

CIFAR100, DenseNet121

CIFAR100, DenseNet121

Location, FC

Location, FC

Attack performance comparison of MIAs on different datasets (L1 Unstructured). We present the attack accuracy of state-of-the-art membership inference attacks and compared them with the proposed SAMIA.  We present the attack accuracy of three models (CIFAR10 DenseNet121, CIFAR100 DenseNet121, Location FC) pruned by L1 unstructured pruning. The black line presents the attack accuracy of original models using Conf attack, , Conf (Org.).

§.§.§ Impact of Confidence, Sensitivity, and Generalization Gap.

As aforementioned, we hypothesize that neural network pruning leads to the increased confidence gap and sensitivity gap (in ground-truth class) of pruned models, thus increasing their membership inference risks.
Meanwhile, overtraining is considered as one of the key causes of membership leakage in previous research <cit.>, leading to our evaluation on generalization gap, , the difference between training accuracy and testing accuracy.
From Figure <ref>, we observe that neural network pruning increases the gaps between members and non-members, , confidence gap, sensitivity gap, and generalization gap, in most settings. Further, with the increase of gaps, we observe the increase of attack accuracy, which indicates  the strong correlation between the gaps, , confidence gap, sensitivity gap, and generalization gap, and the increased privacy risk.

The strong correlation for confidence gap and sensitivity gap validates our intuition that these gaps can be leveraged by the adversary to infer the membership status, introducing a new attack surface in neural network pruning.
By investigating the attack results, we find that the confidence gap plays the most important role in the privacy risk. L1 unstructured and slimming pruning usually lead to an increased confidence gap, which will be leveraged by the adversary and result in a higher attack accuracy. Additionally, sensitivity gap can also leak the membership information. For example, the confidence gap of L1 unstructured pruning on a CIFAR10 ResNet18 model (Figure <ref>) is close to the original model, but due to the increased sensitivity gap (Figure <ref>), the pruning still results in an increased attack accuracy.

§.§.§ Privacy Risks of Pruning Approaches.

Following the same settings above, we investigate the privacy risks of different pruning approaches by comparing the attack accuracy under the similar prediction accuracy.
As shown in Figure <ref> and <ref>, given the similar prediction accuracy of the pruned models, L1 unstructured and slimming pruning result in the highest attack accuracy.
Besides, L1 structured pruning achieves the lowest attack accuracy among all pruning approaches, but still in some cases, the attack accuracy is higher than the original model, even with the similar or lower prediction accuracy.
The structured constraint used in L1 structured pruning regularizes the model in the fine-tuning and thereby reduces the privacy risk.

§.§.§ Effectiveness of SAMIA

To investigate the effectiveness of the proposed SAMIA, we compare SAMIA with the state-of-the-art MIAs in terms of attack accuracy. As shown in Figure <ref>, we observe that our proposed SAMIA achieves the highest attack accuracy in most cases compared with baseline attacks, which is mainly due to the fact that SAMIA best leverages both confidence gap and sensitivity gap (in ground-truth class) introduced in pruning. Besides, Top1-conf and Mentr attacks are also effective, as both attacks take advantage of the confidence gap, the most important factor for model privacy.
We also observe that when the pruning introduces a high generalization gap , all attacks can achieve a high attack accuracy (, CIFAR100 DenseNet121 in Figure <ref>, <ref> and Appendix Figure <ref>)), which has been discussed in the previous MIA research.

§.§.§ Unknown Sparsity Level and Pruning Approach

In the evaluation above, we assume the adversary has the knowledge of sparsity levels and pruning approaches used in network pruning.
In this section, we explore the privacy risks of a more realistic scenario, , when the adversary has no prior knowledge of the sparsity levels and the pruning approaches.

Unknown sparsity level. We assume the adversary only knows the pruning approach but not the sparsity level that is the major factor of model efficiency. We evaluate the attack accuracy of SAMIA when the adversary prunes target models and shadow models using different sparsity levels. We also consider the case when the target model is not pruned, , sparsity level $= 0$. As shown in Figure <ref> and <ref>, the attack accuracy is not affected too much due to the different sparsity levels between target models and shadow models. In some cases, using a different sparsity level in pruning shadow models can even increase the attack accuracy.
The attack accuracy mainly depends on the performance of the shadow model, and thus the adversary can attack victim models with higher attack accuracy by selecting a good pruned shadow model.
For instance, the adversary can use each shadow model to attack other shadow models with different sparsity levels and select the one with the highest attack accuracy.

L1 Unstructured

L1 Structured

L2 Structured

Attack accuracy with unknown sparsity levels (CIFAR10, ResNet18).

CIFAR10, DenseNet121

CIFAR100, DenseNet121

Location, FC

Attack accuracy with unknown sparsity levels (L1 Unstructured).

Unknown sparsity level and pruning approach.
Since we assume the adversary has no prior knowledge of the sparsity level and pruning approach, the adversary may randomly pick a sparsity level and a pruning approach to prune a shadow model for attacks. To evaluate the attack accuracy, we conduct 20 experiments for the aforementioned four image datasets and the corresponding neural networks. In each experiment, we randomly select the sparsity levels and pruning approaches for target models and shadow models, respectively. For example, the target model uses L1 Structured pruning with 0.5 sparsity level while the shadow model uses Slimming pruning with 0.8 sparsity level. The sparsity levels are selected from the set of $\{0.5, 0.6, 0.7, 0.8, 0.9\}$ and the pruning approaches are selected from the four pruning approaches.
To measure the privacy risks, we define the attack accuracy loss as $(acc_{known} - acc_{unknown}) /acc_{known}$, where $acc_{known}$ denotes the attack accuracy when the adversary knows all the pruning information and $acc_{unknown}$ denotes the attack accuracy without knowing any pruning information. Table <ref> shows the average attack accuracy loss over 20 experiments for each dataset and model.
We observe that without knowing the sparsity levels and pruning approaches, the attack is still effective in most cases except the CIFAR10 VGG16 and CIFAR100 ResNet18 models.
The poor attack performance in these two models is due to the ineffectiveness of shadow models using specific sparsity levels and pruning approaches for attacks.
For example, we observe a significant drop of attack accuracy (from 90% to 50%) when applying L1 structured and slimming pruning with sparsity levels 0.7 to 0.9, on the CIFAR10 VGG16 model (Figure <ref> in Appendix). The large gap makes the shadow models pruned using these settings ineffective in attacking the unknown victim model.

Attack accuracy loss with unknown sparsity levels and pruning approaches.

Dataset                                          Model           Attack Acc Loss
3*CIFAR10     ResNet18        4.77%

DenseNet121     1.63%

VGG16           26.83%
3*CIFAR100                        ResNet18        12.41%

DenseNet121     6.90%

VGG16           2.43%
3*SVHN                            ResNet18        0.60%

DenseNet121     0.12%

VGG16           0.05%
3*CHMNIST                         ResNet18        0.78%

DenseNet121     0.52%

VGG16           -0.58%

§ DEFENSES AGAINST MIAS

Given the privacy risks of pruned neural networks, this section focuses on defenses against the proposed SAMIA.
We first present the design principle of defenses for pruned neural networks, then describe the proposed defensive design, and lastly compare the performance of the proposed defense with the state-of-the-art defenses.

In addition, to rigorously evaluate the defense performance, we consider the defenses against both the non-adaptive attacks and adaptive attacks, where the adversary of adaptive attacks is put into the last step of the arms race between privacy attacks and defenses (, the adversary knows all the details of defense mechanism and performs adaptive attacks against the defended models) <cit.>. Extensive experiments are conducted to evaluate our defensive proposals.

§.§ Design Principles of Defenses

Two major design principles are considered for the defenses of pruned neural networks. On the one hand, effective defenses should be able to reduce the behavior discrepancy introduced by pruning. The above attack evaluation has demonstrated that the privacy risks introduced by MIAs in the pruned models are due to the increased divergence of prediction confidences and sensitivities. Hence, it is essential to reduce such divergence between members and non-members of the pruned neural networks for defense. On the other hand, the defenses need to take into consideration the resource constraints imposed by low-end devices. Neural network pruning aims to reduce the computational cost during inference.
Such cost cannot be increased by the defenses. Therefore, the defenses should be designed to mitigate the privacy risks of pruned models before deploying them on devices, thus without introducing additional defense costs in the inference phase.

Confidence gap

Sensitivity gap

Divergence of the pruned model's prediction confidences and sensitivities using PPB defense (CIFAR10, DenseNet121).

Confidence gap

Sensitivity gap

Divergence of the pruned model's prediction confidences and sensitivities over different classes with PPB defense (CIFAR10, DenseNet121).

§.§ Proposed Defense: PPB

Following the two design principles, we propose a countermeasure approach named by pair-based posterior balancing (PPB).
The main idea of PPB defense is to mitigate the new prediction behaviors on prediction confidence and sensitivity by aligning the posterior predictions of different input samples. In this way, PPB can reduce the divergence of prediction confidence between members and non-members as well as the degree of sensitivities. Specifically, given any pair of two input samples, we try to make the distributions of their ranked posterior predictions as close as possible.
The difference between ranked posteriors' distributions is measured by the Kullback–Leibler divergence (KL divergence) <cit.>. Give two posterior predictions $P$ and $Q$, the KL divergence is defined as:

\begin{equation}
\mathcal{L}_{\mathrm{KL}}(P, Q) = \sum_x P(x)\log\frac{P(x)}{Q(x)}.
\end{equation}

KL-divergence is considered as a regularization term in neural network pruning.
The loss function includes both the prediction loss and KL divergence loss, which can be given by:

\begin{align}
\label{eq:defense_obj}
\mathcal{L}(f_p(\bm{x}), \bm{y}) =& \sum_i \mathcal{L}_{\mathrm{predict}}(f_p(\bm{x}_i), y_i) \nonumber\\
&+ \lambda \sum_{j, k (j\neq k)}\mathcal{L}_{\mathrm{KL}}(R(f_p(\bm{x}_j)), R(f_p(\bm{x}_k))),
\end{align}

where $\mathcal{L}_{\mathrm{KL}}$ and $\mathcal{L}_{\mathrm{predict}}$ denote the KL-divergence loss and the prediction loss (, cross-entropy loss for the classification tasks), respectively. $R(\cdot)$ sorts the posteriors provided by the pruned model $f_p$ in decreasing order and $\lambda$ is a hyper-parameter to balance the two losses. It is computationally costly to calculate the KL loss for all possible pairs of data samples in the training dataset. To address this issue, we sample training pairs in each mini-batch during fine-tuning by randomly selecting two data samples as a pair without replacement. Hence, in each mini-batch with batch size $B$, KL loss consists of $B/2$ pairs of training samples.

L1 Unstructured

L1 Structured

L2 Structured

Performance of defenses for different pruning approaches (CIFAR10, ResNet18, Sparsity 0.6).

CIFAR10, DenseNet121

CIFAR10, VGG16

CIFAR100, DenseNet121

Location, FC

Performance of defenses for different datasets (L1 Unstructured, Sparsity 0.6).

In addition, the PPB defense is only applied in fine-tuning of neural network pruning by using KL-divergence as a regularization term. Thus, the defense does not include the additional computational costs in the inference phase.

After applying PPB defense, we observe the divergence between the member and non-member data is significantly reduced by comparing Figure <ref> with Figure <ref>.

Such decreased divergence can be observed in different classes by comparing Figure <ref> with Figure <ref>.
Both changes on the distributions of the pruned model's posterior predictions indicate that the PPB defense makes the attack model fail to learn the binary classification thresholds from the prediction confidence and sensitivity.
Moreover, the PPB defense is designed to change the distribution of predictions instead of their orders. In other words, the PPB defense will not change the predicted classes of the pruned models during fine-tuning, which largely preserves the prediction accuracy of pruned models.

As shown in Figure <ref>, such decreased divergence can also be preserved in different classes. After applying PPB defense, the divergence of the pruned model is close to that of the original model (comparing Figure <ref> with Figure <ref>), which indicates the effectiveness of the PPB defense.

§.§ Defense Evaluation

This section evaluates the effectiveness of PPB by comparing the performance of PPB with that of state-of-the-art defenses[The defenses are evaluated by the empirical experiments. We will investigate the strict privacy guarantee in future work.

§.§.§ State-of-the-art Defenses

We investigate three state-of-the-art defenses against MIA attacks in neural network pruning.

Early Stopping and L2 Regularization (Basic).
Early stopping and l2 regularization have been used to successfully defend membership inference attacks with competitive performance <cit.>. As discussed in Section <ref>, an adversary infers the membership of a sample based on the divergence of the prediction confidences between members and non-members. Such divergence becomes more severe as the number of training epochs increases, due to the increased memorization. Hence, the early stopping mechanism with fewer training epochs and l2 regularization for penalizing the over-training can tradeoff a slight reduction in model accuracy with lower privacy risk.
In the evaluation, we stop the training and fine-tuning when the validation loss is not decreased for five epochs using early stopping mechanism. In l2 regularization, we set the regularization factor as $0.0005$. Note that we use early stopping and l2 regularization in all the other defenses to improve the defense performance.

Differential Privacy (DP).
Differential privacy is a strategy to bound the individual information exposure when running an algorithm $f$ and has been widely investigated for preventing privacy leakage against membership inference attacks <cit.>.
We implement differentially private SGD (DPSGD) <cit.>, one of the most widely-used defense techniques, to train neural networks with DP guarantees.
Following DPSGD, we first clip the gradient, then add noise to the gradient, and use the generated noisy gradient to update the model's parameters. The noise is sampled from a Gaussian distribution $\mathcal{N}(0,\sigma)$. To achieve ($\epsilon, \delta$)-DP, the standard deviation of the Gaussian distribution, i.e., $\delta$, should be in the order of $\Omega(q\sqrt{T\log (1/\delta)/\epsilon}$, where $q$ denotes the sampling ratio and $T$ denotes the total number of iterations.
Accordingly, the privacy guarantee of the DP defense can be derived from $\delta$, which plays an important role in balancing utility and privacy.
Therefore, in the defense evaluation, we evaluate the effectiveness of DP defense and explore the impact of different privacy budgets (, different values of $\delta$).

Adversarial Regularization (ADV).
Nasr proposed to consider the membership inference adversary in the training process <cit.>. The defender first trains a surrogate attack model to distinguish between members and non-members and then trains the target model to minimize the prediction loss while maximizing the classification loss of the surrogate attack model. A parameter $\alpha$ is used to balance the prediction performance and privacy risk.
ADV is applied in the fine-tuning process of pruning to protect the privacy of pruned models.

CIFAR10, ResNet18

CIFAR10, DenseNet121

CIFAR100, DenseNet121

Location, FC

Performance of defenses against adaptive attacks for different pruning approaches (CIFAR10, ResNet18, Sparsity 0.6).

CIFAR10, DenseNet121

CIFAR10, VGG16

CIFAR100, DenseNet121

Location, FC

Performance of defenses against adaptive attacks for different datasets (L1 Unstructured, Sparsity 0.6).

§.§.§ Experimental Results of Defenses

We use the same settings of attack evaluations in Section <ref> and conduct the following experiments with defenses in the process of pruning.
Since there is always a trade-off between privacy and prediction accuracy when implementing defenses, we explore different settings of hyper-parameters in the defensive mechanisms to thoroughly evaluate the defense performance. Specifically, we set hyper-parameter $\lambda \in \{1, 2,4,8, 16\}$ in PPB, $\sigma \in \{0.01, 0.1, 1, 10, 100\}$) in the DP noise vectors, $\alpha \in \{0.5, 1, 2, 4, 8, 16\}$ in ADV, respectively.

Figure <ref> and <ref> illustrate the prediction accuracy and attack accuracy with different defense mechanisms. For better illustration, we remove the results if the model with a specific hyper-parameter cannot achieve 75% of the basic defense's prediction accuracy, , poor prediction performance, or result in a higher attack accuracy, , ineffective defense.

We observe that PPB is especially effective in protecting all pruning approaches from attacks, which can reduce the attack accuracy to around 50% (random guessing accuracy), while not degrading the prediction accuracy too much.
Hence, PPB defense provides a privacy-preserving approach with minimal degradation of prediction accuracy.
In addition, ADV is also effective in the L1 unstructured and Slimming pruning, but fails to achieve a good balance between prediction performance and privacy in the L1 structured and L2 structured pruning. Similar to the fact shown in recent work <cit.>, we observe DP can hardly balance this utility-privacy tradeoff.

§.§.§ Defenses against Adaptive Attacks

To rigorously evaluate the defense performance, we consider adaptive attacks, where the adversary knows all the details of defenses along with the pruning information.
In adaptive attacks, the adversary trains a shadow pruned model following the same defense mechanism (, Basic, DP, ADV, and proposed PPB) and pruning process. The adversary then performs the SAMIA attack based on the shadow pruned model.

As shown in Figure <ref> and <ref>, we observe that PPB reduces the accuracy of adaptive attacks compared to the attacks on the pruned model without defenses and provides the best protection in L1 structured and L2 structured pruning. Besides, for the L1 unstructured and Slimming pruning, PPB and ADV are the best two defenses.
PPB is designed towards pruned models by reducing the confidence and sensitivity gap. Therefore, in general, PPB provides good protection in all pruning approaches.

In addition, ADV is designed to mitigate the confidence gap, which is largely increased in L1 unstructured and slimming pruning (as discussed in Section <ref>). Hence, ADV is also effective in protecting pruned models using L1 unstructured and slimming pruning.

§ CONCLUSION

This paper conducted the first analysis of privacy risks in neural network pruning. We first explored the impacts of neural network pruning on prediction divergence, based on which, a new membership inference attack, , self-attention membership inference attack (SAMIA), is proposed against the pruned neural network models. Through comprehensive and rigorous evaluation, we demonstrated the substantially increased privacy risks of the pruned models. We found that the privacy risks of the pruned models are tightly related to the confidence gap, sensitivity gap, and generalization gap due to pruning. Besides, even without knowing the pruning approach, the membership inference attacks can still achieve high attack accuracy against the pruned model.
Especially, the proposed SAMIA showed superiority in identifying the pruned models' prediction divergence by using finer-grained prediction metrics, which is recommended as a competitive baseline attack model for future privacy risk study of neural network pruning.

In addition, to defend the attacks, we proposed a pair-based posterior balancing named as PPB by reducing the prediction divergence of fine-tuning process during neural network pruning. We experimentally demonstrated that PPB could reduce the attack accuracy to around 50% (random guessing accuracy) without considering adaptive attacks and achieve the best protection compared with the three existing defenses. Besides, PPB showed competitive performance even when defending adaptive attacks.

The proposed SAMIA attack will be further explored under more challenging MIA settings, such as the label-only MIA without available confidences, where the existing label-only MIA attacks using data augmentation <cit.> and black-box adversary <cit.> can be potentially integrated for more powerful attack capability.
We hope our work convinces the community about the importance of exploring innovative neural network pruning approaches by taking privacy-preserving into consideration.

§ ACKNOWLEDGEMENT

We would like to thank our shepherd, Yinzhi Cao, and the anonymous reviewers for their constructive suggestions. This work was supported in part by National Science Foundation (CCF-2106754).

§ AVAILABILITY

Our code is publicly available at <https://github.com/Machine-Learning-Security-Lab/mia_prune> for the purpose of reproducible research.

In the Appendix, we first present the attack evaluation for the remaining experimental settings (Section A). Then we compare the defense performance of the proposed PPB with other state-of-the-art defenses for the remaining experimental settings (Section B).

§ EXTENDED EXPERIMENTS FOR ATTACK EVALUATION

In this section, we provide the extended experiments for attack evaluation and report the prediction accuracy and the attack performance in the remaining experimental settings, ,  datasets (CIFAR10, CIFAR100, CHMNIST, SVHN, Location, Purchase, and Texas), model architectures (VGG16, ResNet18, DenseNet121).

§.§ Prediction Accuracy of Pruned Models

This section presents the prediction accuracy of pruned models using different pruning approaches and sparsity levels on the remaining experiment settings, including the model architectures (ResNet18, DenseNet121, VGG16, and FC) and the datasets (CIFAR10, CIFAR100, CHMNIST, SVHN, Location, Purchase, and Texas).

CIFAR10, VGG16

CIFAR100, ResNet18

CIFAR100, VGG16

CHMNIST, ResNet18

CHMNIST, DenseNet121

CHMNIST, VGG16

SVHN, ResNet18

SVHN, DenseNet121

SVHN, VGG16

Purchase, FC

Texas, FC

Prediction accuracy (test accuracy) of the pruned models using different pruning approaches and sparsity levels. Each point indicates the prediction accuracy achieved by the pruned model with a specific pruning approach and sparsity level. The black line indicates the prediction accuracy of the original models.

§.§ Privacy Risks of Pruned Models

This section presents the attack accuracy of the remaining experiment settings.

Similar to the observation in the main paper, we find that in most cases, the pruned models become more vulnerable to the membership inference attacks than the original models.

CIFAR10, VGG16

CIFAR100, ResNet18

CIFAR100, VGG16

CHMNIST, ResNet18

CHMNIST, DenseNet121

CHMNIST, VGG16

SVHN, ResNet18

SVHN, DenseNet121

SVHN, VGG16

Texas, FC

Purchase, FC

Privacy Risks of Neural Network Pruning (w.r.t. prediction accuracy). Most pruning approaches results in a higher attack accuracy when considering a similar prediction accuracy, compared with the original models. We present the attack accuracy of SAMIA for pruned models and the attack accuracy of Conf attack for the original models.

§.§ Effectiveness of SAMIA

We investigate the effectiveness of SAMIA.
Although most investigated MIAs are effective in attacking pruned models, SAMIA results in higher attack accuracy than other attacks in most cases.
SAMIA does not perform among the best attacks mainly in two cases: 1) the attack accuracy is close to 50% (, random guessing); 2) the sparsity level is low (, $0.5$). When the sparsity level is low, the divergence of confidence and sensitivity is reduced. In this case, the features of confidence and sensitivity cannot be fully utilized for SAMIA, which makes the SAMIA's attack performance close to the existing attacks.

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (CIFAR10, DenseNet121).

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (CIFAR10, VGG16).

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (CIFAR100, ResNet18).

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (CIFAR100, VGG16).

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (CHMNIST, ResNet18).

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (CHMNIST, DenseNet121).

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (CHMNIST, VGG16).

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (SVHN, ResNet18).

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (SVHN, DenseNet121).

L1 unstructured

L1 structured

L2 structured

L1 unstructured

L1 structured

L2 structured

Attack performance comparison of MIAs (SVHN, VGG16).

Purchase, FC

Purchase, FC

Texas, FC

Texas, FC

Attack performance comparison of MIAs (L1 Unstructured).

§.§ Impact of Confidence Gap, Sensitivity Gap, and Generalization Gap

This section reports the impact of confidence gap, sensitivity gap, and generalization gap in the remaining experimental settings. We observe the strong correlation between gaps (, confidence gap, sensitivity gap, and generalization gap) and attack accuracy.

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (CIFAR10, DenseNet121).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (CIFAR10, VGG16).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (CIFAR100, ResNet18).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (CIFAR100, DenseNet121).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (CIFAR100, VGG16).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (CHMNIST, ResNet18).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (CHMNIST, DenseNet121).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (CHMNIST, VGG16).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (SVHN, ResNet18).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (SVHN, DenseNet121).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (SVHN, VGG16).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (Location, FC).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (Purchase, FC).

Confidence gap

Sensitivity gap

Generalization gap

Impact of confidence gap, sensitivity gap, and generalization gap (Texas, FC).

§.§ Impact of Dataset Split Strategies

This section reports the attack performance when the target model is trained on an unevenly split dataset.
In the main paper, we split datasets into training, test, and validation in the ratio 45%, 45%, 10%, so that we get the same number of training and test data and the random guessing attack accuracy is 50%. Here we evaluate the attack performance using different and uneven ratios. Given the uneven dataset, we use Area Under the Curve (AUC) to report the attack performance. As shown in Table <ref>, the uneven split may cause a slight decrease of attack AUC compared with an even split, but the privacy risk is still high (over 70% attack AUC).

Dataset Split Ratio                  2*Attack AUC

45/45/10                      75.5

50/40/10                       72.7

60/30/10                       71.6

70/20/10                    72.7

Impact of different dataset split strategies (CIFAR10, ResNet18, L1 Unstructured, Sparsity 0.7).

§.§ Attack Accuracy With Unknown Sparsity Levels

This section reports the attack accuracy with unknown sparsity levels in the CIFAR10, CIFAR100, Purchase, and Texas datasets. Without knowing the sparsity level (different sparsity levels used in the target model and the shadow model), the adversary can still achieve similar (even higher in some cases) attack accuracy.

L1 Unstructured

L1 Structured

L2 Structured

Attack accuracy with unknown sparsity levels (CIFAR10, DenseNet121).

L1 Unstructured

L1 Structured

L2 Structured

Attack accuracy with unknown sparsity levels (CIFAR10, VGG16).

L1 Unstructured

L1 Structured

L2 Structured

Attack accuracy with unknown sparsity levels (CIFAR100, DenseNet121).

Purchase, FC

Texas, FC

Attack accuracy with unknown sparsity levels.