# Combining Deep Reinforcement Learning And Local Control
For The Acrobot Swing-up And Balance Task

Sean Gillen, Marco Molnar, and Katie Byl *This work was funded in part by NSF
NRI award 1526424.Sean Gillen and Katie Byl are with the Electrical and
Computer Engineering Department at the University of California, Santa Barbara
CA 93106<EMAIL_ADDRESS><EMAIL_ADDRESS>Marco Molnar is with TU Berlin
<EMAIL_ADDRESS>code for this paper can be found here:
https://github.com/sgillen/ssac

###### Abstract

In this work we present a novel extension of soft actor critic, a state of the
art deep reinforcement algorithm. Our method allows us to combine traditional
controllers with learned neural network policies. This combination allows us
to leverage both our own domain knowledge and some of the advantages of model
free reinforcement learning. We demonstrate our algorithm by combining a hand
designed linear quadratic regulator with a learned controller for the acrobot
problem. We show that our technique outperforms other state of the art
reinforcement learning algorithms in this setting.

##  I INTRODUCTION

Advances in machine learning have allowed researchers to leverage the massive
amount of compute available today in order to better control robotic systems.
The result is that modern model-free reinforcement learning has been used to
solve very difficult problems. Recent examples include controlling a 47 DOF
humanoid to navigate a variety of obstacles [1], dexterously manipulating
objects with a 24 DOF robotic hand [2], and allowing a physical quadruped
robot to run [3], and recover from falls [4].

Despite this, these algorithms can struggle on certain low dimensional
problems from the nonlinear control literature. Namely the acrobot [5] and the
cart pole pendulum. These are both under-actuated mechanical systems that have
unstable fixed points in their unforced dynamics (see section II-B).
Typically, the goal is to bring the system to this fixed point and keep it
there. In this paper we focus on the acrobot as we found less examples of
model free reinforcement learning performing well on this task.

Figure 1: System diagram for the new technique proposed in this paper. Rounded
boxes represent learned neural networks, squared boxes represent static, hand
crafted functions. The local controller is a hand designed LQR, the swing-up
controller is obtained via reinforcement learning, and the gating function is
trained as a neural network classifier

It is not uncommon to see some variation of these systems tackled in various
reinforcement benchmarks, but we have found these problems have usually been
artificially modified to make them easier. For example the very popular OpenAI
Gym benchmarks [6] includes an acrobot task. But the objective is only to get
the system in the rough area of the unstable fixed point, and the dynamics are
integrated with a fixed time-step of .2 seconds, which makes the problem much
easier and unrepresentative of a physical system. We have found that almost
universally, modern model free reinforcement learning algorithms fail to solve
a more realistic version of the task. Notably, the Deep Mind control suite [7]
includes the full acrobot problem, and all but one algorithm that they tested
(the exception being [8]) learned nothing, the average return after training
was the same as before training.

Despite this, there are many traditional model based solutions [5], [9], that
can solve this problem well. In this work we do not seek to improve upon the
model based solutions to this problem, but to extend to the class of problems
that model free reinforcement learning methods can be used to solve. We
believe the methods used here to solve the acrobot can be extended to other
problems, such as making robust walking policies.

One of the primary reasons why this problem is difficult for RL is that the
region of state space that can be brought to the unstable fixed point is very
small, even with generous torque limits. An untrained RL agent explores by
taking random actions in the environment. Reaching the region of attraction is
rare, we found that for our system, random actions will reach the basin of
attraction for a well designed LQR in about 1% of trials. However an RL agent
doesn’t have access to a well designed LQR at the start of training, in
addition to reaching the region where stabilization is possible, the agent
must also stabilize the acrobot for the agent to receive a strong reward
signal. This results in successful trials in this environment being extremely
rare, and therefore training is in-feasibly slow and sample inefficient.

Our solution to add a predesigned balancing controller into the system, this
is comparatively easy to design, and can be done with a linear controller. Our
contribution is a novel way to combine this balancing controller with an
algorithm that is learning the swing-up behavior. We simultaneously learn the
swing-up controller, and a function that switches between the two controllers.

###  I-A Related Work

Work done by Randolov et. al. [10] is closely related to our own. In that work
they construct a local controller, an LQR, and combine it with a learned
controller to swing-up and balance an inverted double pendulum (similar to the
acrobot we study but with actuators at both joints). The primary differences
between our work and theirs is that they hard code the transition between
their two controllers. In contrast we learn our transition function online and
in parallel with our swing-up controller.

Work done by Yoshimoto et. al. [11], like ours, learns the transition function
between controllers in order to swing-up and balance an acrobot. However,
unlike our work they limit the controllers they switch between to pre-computed
linear functions. In contrast our work simultaneously learns a nonlinear
swing-up controller and the transition between a learned and pre-computed
balance controller.

Wiklednt et. al [12] too swing-up and balance an acrobot using a combined
neural network and LQR. However they only learn to swing-up from a single
initial condition, whereas our method learns to solve the task from any
initial position.

Doya [13] also learns many controllers using reinforcement learning, and
adaptively switches between them. However unlike our work, the switching
function is not learned using reinforcement learning, but is instead selected
according to which of the controllers currently makes best prediction of the
state at the current point in state space. We believe our model free updates
will avoid the model bias that can be associated with such approaches.
Furthermore our work allows for combining learned controllers with hand
designed controllers, such as the LQR.

##  II Background

###  II-A Nomenclature

We formulate our problem as a Markov decision process, $M=(S,A,R)$. At each
time step $t$, Our agent receives the current state $s_{t}\in S$ and chooses
an action $a_{t}\in A$. It then receives a reward according to the reward
function $r_{t}=R(s_{t},a_{t},s_{t+1})$. The goal is to find a policy
$\pi:S\rightarrow p(A=a|s)$ that satisfies:

$\pi^{*}=\operatorname*{arg\,max}_{\pi}\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},s_{t+1})\right]$
(1)

###  II-B Acrobot

The acrobot is described in Figure 2. It is a double inverted pendulum with a
motor only at the elbow. We use the parameters from Spong [5]:

TABLE I: Mass and inertial parameters used in simulation

Parameter | Value | Units
---|---|---
$m_{1},m_{2}$ | 1 | Kg
$l_{1},l_{2}$ | 1 | m
$l_{c1},l_{c2}$ | .5 | m
$I_{1}$ | .2 | Kg*m2
$I_{2}$ | 1.0 | Kg*m2

Figure 2: Diagram for the acrobot system

The state of this system is
$s_{t}=[\theta_{1},\theta_{2},\dot{\theta}_{1},\dot{\theta}_{2}]$. The action
$a_{t}=\tau$, is the torque at the elbow joint. The goal we wish to achieve is
to take this system from any random initial state, to the upright state
$gs=[\pi/2,0,0,0]$, which we will refer to as the goal state. To achieve this
goal, we seek the maximum of the following reward function:

$r_{t}=l_{1}\sin(\theta_{1})+l_{2}\sin(\theta_{1}+\theta_{2})$ (2)

This was motivated by the popular Acrobot-v1 environment [14], We found
empirically that for our algorithm this reward signal led to the same
solutions as the more typical $\lVert s_{t}-gs\rVert$. However we found that
some of the other algorithms we compared to perform better with the sinusoidal
reward function.

We implement the system in python (all source code is provided, see footnote
on page one), the dynamics are implemented using Euler integration with a
time-step of .01 seconds, and the control is updated every .2 seconds. We
experimented with smaller timesteps and higher order integrators, generally we
found these made the balancing task easier, but made the wall clock time for
the learning much slower.

###  II-C Soft Actor Critic

Soft actor critic (SAC) is an off policy deep reinforcement learning algorithm
shown to do well on control tasks with continuous actions spaces [15]. To aid
in exploration, rather than directly optimize the discounted sum of future
rewards, SAC attempt to find a policy that optimizes a surrogate objective:

$J^{\mathrm{soft}}=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\bigg{(}R_{t}+\alpha
H(\pi(\cdot|s_{t}))\bigg{)}\right]$ (3)

Where $H$ is the entropy of the policy.

SAC introduces several neural networks for the training. We define a soft
value function $V_{\phi}(s_{t})$, a neural network defined by weights $\phi$,
which approximate $J^{soft}$ given the current state. Next we define two soft
Q functions $Q_{\rho_{1}}(s_{t},a_{t})$ and $Q_{\rho_{2}}(s_{t},a_{t})$ which
approximate $J^{soft}$ given both the current state and the current action.
Using two Q networks is a trick that aids the training by avoiding
overestimating the Q function. We must also define a target soft value
function $V_{\overline{\phi}}(s_{t})$, which follows the value function via
polyak averaging:

$V_{\overline{\phi}^{+}}(s_{t})=c_{py}V_{\overline{\phi}}(s_{t})+(1-c_{py})V_{\phi}$
(4)

With $c_{py}$ a fixed hyper parameter. We also define $\Pi_{\theta}$, a neural
network that outputs $\mu_{\theta}(s_{t})$ and $\log(\sigma_{\theta}(s_{t}))$
which define the probability distribution of our policy $\pi_{\theta}$. The
action is given by:

$a_{t}=\tanh(\mu_{\theta}(s_{t})+\sigma_{\theta}(s_{t})\epsilon_{t})$ (5)

where $\epsilon_{t}$ is drawn from $N(0,1)$.

SAC also make use of a replay buffer $D$ which stores the tuple
$(s_{t},a_{t},r_{t})$ after policy rollouts. When it is time to update we
sample randomly from this buffer, and use those samples to compute our losses
and update our weights.

With this we can define the losses for each of these networks (originated from
[15])

The loss for our two Q functions is:

$L^{Q}=\mathbb{E}_{s_{t},a_{t}\sim
D}\left[\frac{1}{2}\left(Q_{\rho}(s_{t},a_{t})-\hat{Q}(s_{t},a_{t})\right)^{2}\right]$
(6)

where

$\hat{Q}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}}\left[V_{\overline{\phi}}(s_{t+1})\right]$
(7)

Our policy seeks to minimize:

$L^{\pi}=\mathbb{E}_{s_{t}\sim D,\epsilon_{t}\sim
N(0,1)}\left[\log\pi_{\theta}(f_{\theta}(\epsilon_{t},s_{t})|s_{t})-Q_{\rho_{1}}(s_{t},f_{\theta}(\epsilon_{t},s_{t})\right]$
(8)

And our value function:

$L^{\text{V}}=\mathop{\mathbb{E}}_{s_{t}\sim
D}\left[\frac{1}{2}\left(V_{\phi}(s_{t})-\hat{V}_{\phi}(s_{t})\right)^{2}\right]$
(9)

Where

$\hat{V}_{\phi}=\mathbb{E}_{a_{t}\sim\pi_{\theta}}\left[Q^{min}(s_{t},a_{t})-log\pi_{\theta}(a_{t}|s_{t})\right]$
(10)

And $Q^{min}=\min(Q_{\rho_{1}}(s_{t},a_{t}),Q_{\rho_{2}}(s_{t},a_{t}))$

SAC starts by doing policy roll outs, recording the state, action, reward, and
the active controller at each time step. It stores these experiences in the
replay buffer. After enough trials have been run, we run our update step. We
sample from the replay buffer, and use these sampled states to compute the
losses above. We then run one step of Adam [16] to update our network weights.
We repeat this update $n_{u}$ times with different samples. Finally we copy
our weights to our target network and repeat until convergence (or some other
stopping metric).

##  III Switched Soft Actor Critic

Our primary contribution is to extend SAC in two key ways, we call the
modified algorithm switched soft actor critic (SSAC). The first modification
is a change to the structure of the learned controller in order to inject our
domain knowledge into the learning. Our controller consists of three distinct
components. The gate function, the balancing controller, and the swing-up
controller. The gate, $G_{\gamma}:S\rightarrow[0,1]$, is a neural network
parameterized by weights $\gamma$ which takes the observations at each time
step and outputs a number $g_{t}$ representing which controller it thinks
should be active. $g_{t}\approx 1$ implies high confidence that the balancing
controller should be active, and $g_{t}\approx 0$ implies the swing-up
controller is active. This output is fed through a standard switching hysteris
function, to avoid rapidly switching on the class boundary, parameters given
in the appendix. The swing-up controller can be seen as the policy network
from vanilla SAC, the action then is determined by equation (5). The
parameters for these networks are given in the appendix. The balancing
controller is a linear quadratic regulator $C:S\rightarrow A$ about the
acrobot’s unstable equilibrium. We use the LQR designed by Spong [5]:

Using

$Q=\begin{pmatrix}1000&-500&0&0\\\ -500&1000&0&0\\\ 0&0&1000&-500\\\
0&0&-500&1000\end{pmatrix},R=\begin{pmatrix}.5\end{pmatrix}$

The resulting control law is:

$u=-Ks$

with

$K=[-1649.8,-460.2,-716.1,-278.2]$

These three functions together form our policy, $\pi_{\theta}$. Algorithm 1
demonstrates how the action is computed at each timestep.

We learn the basin of attraction for the regulator by framing it as a
classification problem, our neural network takes as input the current state,
and outputs a class prediction between 0-1. A one implying that the LQR is
able to stabilize the system, and a zero implying that it cannot. We then
define a threshold function $T(s)$, as a criteria for what we consider a
successful trial:

${T(s)=\lVert s_{t}-gs\rVert<\epsilon_{thr}}\quad\forall
t\in\\{N_{e}-b,...,N_{e}\\}$ (11)

Here $s$ is understood to be an entire trajectory of states, $N_{e}$ is the
length of each episode, $e_{thr}$ and $b$ hyper parameters with values given
in the appendix. We are following the convention of a programming language
here, (11) returns one when the inequality holds, and zero otherwise. To
gather data, we sample a random initial condition, do a policy roll out using
the LQR, and record the value of 11 as the class label.

To train the gating network we minimize the binary cross entropy loss:

$L^{\text{G}}=\mathop{\mathbb{E}}_{\gamma}-\left[c_{w}y_{i}\log(G_{\gamma}(s_{i}))+(1-y_{i})\log(1-G_{\gamma}(s_{i}))\right]$
(12)

Where $y_{i}$ is the class label for the ith sample, $c_{w}$ is a class weight
for positive examples. we set $c_{w}=\frac{n_{t}}{n_{p}}w$ where $n_{t}$ is
the total number of samples, $n_{p}$ is the number of positive examples, and
$w$ is a manually chosen weighting parameter to encourage learning a
conservative basin of attraction. We found that the learned basin was very
sensitive to this parameter, a value of .01 empirically works well. Note that
unlike the other losses above, the data here is not computed over a sample but
is instead computed over the entire replay buffer. We found the gate was prone
to ”forgetting” the basin of attraction early in the training otherwise. This
also allows us to update the gate infrequently compared to the other networks,
and so the total impact on wall clock time is modest.

The second extension is a modification of the replay buffer $D$. We do this by
constructing $D$ from two separate buffers, $D_{n}$ and $D_{r}$. Only roll
outs that ended in a successful balance (as defined by equation (11)) are
stored in $D_{r}$. The other buffer stores all trials, the same as the
unmodified replay buffer. Whenever we draw experience from $D$, with
probability $p_{d}$ we sample from $D_{n}$, and with probability $(1-p_{d})$
we sample from $D_{r}$. We found this to speed up learning dramatically, as
even with the LQR and a decent gating function in place, the swing-up
controller finds the basin of attraction only in a tiny minority of trials.

Algorithm 1 Do-Rollout($G_{\gamma},\Pi_{\theta}$, K)

1:$s=r=a=g=r=\\{\\}$

2:Reset environment, collect $s_{0}$

3:for $t\in\\{0,...,T\\}$ do

4: $g_{t}=hyst(G_{\gamma}(s_{t}))$

5: if $(g_{t})==1$ then

6: $a_{t}=-Ks_{t}$

7: else

8: Sample $\epsilon_{t}$ from $N(0,1)$

9: $a_{t}=\beta\tanh(\mu_{\theta}(s_{t})+\sigma_{\theta}(s_{t})*\epsilon_{t})$

10: Take one step using $a_{t}$, collect $\\{s_{t+1},r_{t}\\}$

11: $s=s\bigcup s_{t}$, $r=r\bigcup r_{t}$

12: $a=a\bigcup a_{t}$, $g=g\bigcup g_{t}$

13:return $s,a,r,g$

Algorithm 2 Switched Soft Actor Critic

1:Initialize network weights $\theta,\phi,\gamma,\rho_{1},\rho_{2}$ randomly

2:set $\overline{\phi}=\phi$

3:for $n\in\\{0,...,N_{e}\\}$ do

4: $s,r,a,g=\text{Do-Rollout}(G_{\gamma},\Pi_{\theta},K)$

5: if $T(s)$ then

6: Store $s,r,a$ in $D_{n}$

7: Store $s,r,a$ in $D_{r}$

8: Store $s,g,T(s)$ in $D_{g}$

9: if Time to update policy then

10: sample $s^{r},a^{r},r^{r}$ from $D$

11: $\hat{Q}\approx R+\gamma V_{\overline{\phi}}(S)$

12: $Q^{min}=\min(Q_{\rho_{1}}(s^{r},a^{r}),Q_{\rho_{2}}(s^{r},a^{r}))$

13: $\hat{V}\approx Q^{min}-\alpha H(\pi_{\theta}(A|S))$

14: Run one step of Adam on $L^{Q}(s^{r},q^{r},r^{r})$

15: Run one step of Adam on $L^{\pi}(s^{r})$

16: Run one step of Adam on $L^{V}(s^{r})$

17: $\overline{\phi}=q\overline{\phi}+(1-q)\phi$

18: if Time to update gate then

19: Run one step of Adam on $L^{G}$ using all samples in $D_{g}$

##  IV Results

###  IV-A Training

To train SSAC we first start by training the gate exclusively, using the
supervised learning procedure outlined in section III This allows us to form
an estimate of the basin of attraction before we try to learn to reach it. We
trained the gate for 1e6 timesteps, and then trained both in parallel using
algorithm 2 for another 1e6 timesteps. The policy, value, and Q functions are
updated every 10 episodes, and the gate every 1000. The disparity is because,
as mentioned earlier, the gate is updated using the entire replay buffer,
while all the other losses are updated with one sample batch from the buffer.
Hyperparameters were selected by picking the best performing values from a
manual search, which are reported in the appendix.

In addition to training on our own version of SAC and Switched SAC we also
examined the performance of several algorithms written by OpenAI and cleaned
up by the community [17]. We examine PPO and TRPO, two popular trust region
methods. A2C was included to compare to a non trust region, modern policy
gradient algorithm. We also include TD3, which has been shown in the
literature to do well on the acrobot and cartpole problems [18]

Stable baselines includes hyperparameters that were algorithmically tuned for
each environment. For algorithms where parameters for Acrobot-v1 were
available we chose those, some algorithms were missing tuned Acrobot-v1
examples, and for those we used parameters for Pendulum-v0, simply because it
is another continuous, low dimensional task. Note we don’t expect the hyper-
parameters to impact the learned policy’s score in this case, only how fast
learning occurs. Reported rewards are averaged over 4 random seeds. Every
algorithm makes 2e6 interactions with the environment. Also note that this
project was largely inspired by spending a large amount of time manually
tuning these parameters to work on this task (with no success better than what
we see here). Figure 3 shows the reward curve for our algorithm and the
algorithms from stable baselines. Table II shows the mean and standard
deviation for the final rewards obtained by all algorithms.

Figure 3: Reward curve for SSAC and the other algorithms we compare to. the
solid line is the smoothed average of episode reward, averaged over four
random seeds. The shaded area indicates the best and worst rewards at each
epoch across the four seeds. SSAC is shown starting later to account for the
time training the gating function alone.

Algorithm (implementation) | Mean Reward $\pm$ Standard Deviation
---|---
SSAC (Ours) | 92.12 $\pm$ 2.35
SAC | 73.01 $\pm$ 11.41
PPO | 0.43 $\pm$ 8.89
TD3 | 78.67 $\pm$ 61.85
TRPO | 17.63 $\pm$ 3.39
A2C | 2.57 $\pm$ 3.63

TABLE II: Rewards after training for across learning algorithms. This table
shows results after 2 million environment interactions

As we can see, for this environment, with the number of steps we have
allotted, our approach outperforms the algorithms we compared to, with TD3
making it the closest to our performance. This is a necessarily flawed
comparison. These algorithms are meant to be general purpose, so it is unfair
to compare them to something designed for a particular problem. But that is
part of the point we are making, that adding just a small amount of domain
knowledge can improve performance dramatically.

###  IV-B Analyzing performance

To qualitatively evaluate the performance of our learned agent we examine the
behavior during individual episodes. SSAC gives us a deterministic controller
(we can set $\epsilon_{t}$ from 5 to zero). We chose the initial condition
$s_{0}=(-\pi/2,0,0,0)$ and record a rollout. The actions are displayed in
figure 4, and the positions in 5.

Figure 4: Torque exerted during the sampled episode Figure 5: Observations
during the sampled episode

We have also found that despite achieving relatively high rewards, the other
algorithms we compare to often fail to meet the balance criteria (11). We
often see solutions where the first link is constantly rotating, with the
second link constantly vertical. To demonstrate this, as well as to
demonstrate our algorithms robustness, we run roll outs with the trained
agents across a grid of initial conditions, recording if the trajectory
satisfies (11) or not. We compare our method with TD3, which was the best
performing model free method we could find on this task. Figure 6 show the
results, when these initial conditions were run for SSAC, it satisfied (11)
for every initial condition.

Figure 6: Balance map for TD3, X and Y indicate the initial position for the
trial, a black dot indicates that the trial started from that point satisfies
equation (11), and red indicates the converse. when these initial conditions
were run for SSAC, it satisfied (11) for every initial condition

##  V CONCLUSIONS

We have presented a novel control design methodology that allows engineers to
leverage their domain knowledge, while also reaping many of the benefits from
recent advances in deep reinforcement learning. In our case study we
constructed a policy to swing-up and balance an acrobot while only needing to
manually design a linear controller for the balancing task. We believe this
method of control will be straightforward to apply to the double or triple
cartpole problems, which to our knowledge no model free algorithm is reported
as solving. We also think that this general methodology can be extended to
more complex problems, such as legged locomotion. In that case the linear
controller here could be a nominal walking controller obtained via trajectory
optimization, and the learned controller could be a recovery controller to
return to the basin of attraction of this nominal controller.

## References

  * [1] N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A. Eslami, M. Riedmiller, and D. Silver, “Emergence of Locomotion Behaviours in Rich Environments,” _arXiv:1707.02286 [cs]_ , July 2017, arXiv: 1707.02286. [Online]. Available: http://arxiv.org/abs/1707.02286
  * [2] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning Dexterous In-Hand Manipulation,” _arXiv:1808.00177 [cs, stat]_ , Aug. 2018f, arXiv: 1808.00177. [Online]. Available: http://arxiv.org/abs/1808.00177
  * [3] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” _Science Robotics_ , vol. 4, no. 26, p. eaau5872, Jan. 2019. [Online]. Available: http://robotics.sciencemag.org/lookup/doi/10.1126/scirobotics.aau5872
  * [4] J. Lee, J. Hwangbo, and M. Hutter, “Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning,” _arXiv:1901.07517 [cs]_ , Jan. 2019, arXiv: 1901.07517. [Online]. Available: http://arxiv.org/abs/1901.07517
  * [5] M. W. Spong, “Swing up control of the acrobot using partial feedback linearization *,” _IFAC Proceedings Volumes_ , vol. 27, no. 14, pp. 833–838, Sept. 1994. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S1474667017474040
  * [6] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  * [7] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller, “DeepMind control suite,” https://arxiv.org/abs/1801.00690, DeepMind, Tech. Rep., Jan. 2018. [Online]. Available: https://arxiv.org/abs/1801.00690
  * [8] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. TB, A. Muldal, N. Heess, and T. Lillicrap, “Distributed Distributional Deterministic Policy Gradients,” _arXiv:1804.08617 [cs, stat]_ , Apr. 2018, arXiv: 1804.08617. [Online]. Available: http://arxiv.org/abs/1804.08617
  * [9] M. W. Spong, “Energy Based Control of a Class of Underactuated Mechanical Systems,” _IFAC Proceedings Volumes_ , vol. 29, no. 1, pp. 2828–2832, June 1996. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S1474667017581057
  * [10] J. RandlÃžv, A. G. Barto, and M. T. Rosenstein, “Combining Reinforcement Learning with a Local Control Algorithm,” in _Proceedings of the Seventeenth International Conference on Machine Learning_ , ser. ICML ’00. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2000, pp. 775–782. [Online]. Available: http://dl.acm.org/citation.cfm?id=645529.657804
  * [11] J. Yoshimoto, M. Nishimura, Y. Tokita, and S. Ishii, “Acrobot control by learning the switching of multiple controllers,” _Artificial Life and Robotics_ , vol. 9, no. 2, pp. 67–71, May 2005. [Online]. Available: http://link.springer.com/10.1007/s10015-004-0340-6
  * [12] L. Wiklendt, S. Chalup, and R. Middleton, “A small spiking neural network with LQR control applied to the acrobot,” _Neural Computing and Applications_ , vol. 18, no. 4, pp. 369–375, May 2009. [Online]. Available: http://link.springer.com/10.1007/s00521-008-0187-1
  * [13] K. Doya, K. Samejima, K.-i. Katagiri, and M. Kawato, “Multiple Model-Based Reinforcement Learning,” _Neural Computation_ , vol. 14, no. 6, pp. 1347–1369, June 2002. [Online]. Available: http://www.mitpressjournals.org/doi/10.1162/089976602753712972
  * [14] P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y. Wu, and P. Zhokhov, “Openai baselines,” https://github.com/openai/baselines, 2017.
  * [15] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” _arXiv:1801.01290 [cs, stat]_ , Aug. 2018, arXiv: 1801.01290. [Online]. Available: http://arxiv.org/abs/1801.01290
  * [16] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” _arXiv:1412.6980 [cs]_ , Dec. 2014, arXiv: 1412.6980. [Online]. Available: http://arxiv.org/abs/1412.6980
  * [17] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, “Stable baselines,” https://github.com/hill-a/stable-baselines, 2018.
  * [18] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv:1509.02971 [cs, stat]_ , Sept. 2015, arXiv: 1509.02971. [Online]. Available: http://arxiv.org/abs/1509.02971

## APPENDIX

### Hyperparameters

Hyperparameter | Value
---|---
Episode length ($N_{e}$) | 50
Exploration steps | 5e4
Initial policy/value learning rate | 1e-3
Steps per update | 500
Replay batch size | 4096
Policy/value minibatch size | 128
Initial gate learning rate | 1e-5
Win criteria lookback (b) | 10
Win criteria threshold ($\epsilon_{thr}$) | .1
Discount ($\gamma$) | .95
Policy/value updates per epoch | 4
Gate update frequency | 5e4
Needle lookup probability $p_{n}$ | .5
Entropy coefficient ($\alpha$) | .05
Polyak constant ($c_{py}$) | .995
Hysteresis on threshold | .9
Hysteresis off threshold | .5

### Network Architecture

The policy, value, and Q networks are all made of four fully connected layers,
with 32 hidden nodes and Relu activations. The gate network is composed of two
hidden layers with 32 nodes each, also with Relu activations, the last output
is fed through a sigmoid to keep the result between 0-1.


On phase transition signal in inelastic collision

J.Manjavidze

JINR, Dubna, Russia111 On leave in absence: Andronikashvili Institute of
Physics, Tbilisi, Georgia

###### Abstract

The paper is devoted to retrieval of the first order phase transition signal
in the inelastic collisions. The primary intent is to show that the
experimentally observable signal exist iff the multiplicity is sufficiently
large. We discuss corresponding phenomenology from the point of view of
experiment.

I

The question of possibility to observe first order phase transition in the
hadron and heavy ion collisions is discussed widely at present time [1]. The
aim of paper is discuss the phenomenology of that problem.

The first order phase transition in statistics appears in the result of
creation of local critical fluctuations (e.g. bubbles of vapor if the
(liquid$\to$gas) transition is considered), in contrast to the second order
phase transition where the whole system undergo the transition. The dimension
of such fluctuations increase, if they are under-critical, and the whole
system in result undergo the transition.

It is hard to imagine that exactly such picture appears in the hadron or ion
inelastic collisions. The point is that in the event-by-event measurement the
dimension of bubbles, if they exist, may be both smaller or larger critical
one. Therefore, in the best case, we observe the mixture of two-phase medium
and the question how one may increase on experiment the weight of events with
under-critical bubbles is the first problem. It is evident that the density,
usually used in statistics, can not be introduced as the ”order parameter”
since only the result of particles $production$ process is observed.

In the paper [2] the ”chemical potential”, $\mu(n,s)$, was offered as the
”order parameter”. It is the work which is necessary for creation of one
particle and it was shown that if the role of under-critical babbles dominate
then $\mu(n,s)$ must decrease with number of produced particles $n$. In
another words, it was offered to define the boiling, i.e. creation of under-
critical bubbles, through $intensity$ of the process of particles production
(evaporation).

This is the main idea why the very high multiplicity (VHM) processes were
considered. It is not important what additional criterium is used searching
first order phase transition. In any case we must consider the VHM domain to
have intensive production of particles. In addition, the kinetic degrees of
freedom must be suppressed in the VHM region.

Therefore, the special attention will be given to the VHM processes. Then
exist approximation [2]:

$\mu(n,s)\simeq-\frac{T(n,s)}{n}\ln\sigma_{n}(s)$ (1)

in this multiplicity region. Here $T$ is the mean energy, including mass, of
produced particles, i.e. $T$ is associated with temperature, and $\sigma_{n}$
is the normalized to unite multiple production cross section which can be
considered in the VHM region as the ”partition function” of the $equilibrium$
system. The equilibrium condition will be defined later, see inequality (17),
definition (18) and [3] where the detailed explanation was given. See also the
footnote 5.

Continuing the analogy with thermodynamics one can say that
$(-T\ln\sigma_{n})/n$ is the Gibbs free energy per one particle. Then $\mu$
can be interpreted as the ”chemical potential” measured with help of
$observed$ free particles222Notice that one may consider $n$ as the
multiplicity in the experimentally observable range of phase space..

The definition (1) is quiet general. It can be used both for hadron-hadron and
ion-ion collisions, both for low and high energies, both for ”boiling” media
of coloured partons and colorless hadrons. Definition (1) is model free and
operates only with ”external” directly measurable parameters. The single
indispensable condition: we work in the VHM region.

We will discuss in the paper the chance of experimental measurement of
$\mu(n,s)$ defined by (1), what kind uncertainties hides it from
$experimental$ point of view noting the the cross sections in VHM domain are
small. The correction to (1) are not essential in the VHM region but
nevertheless the field-theoretical definition of chemical potential in using
Wigner functions formalism [4] will be published, see also [3].

II.

It is necessary to remind main steps toward (1) to understand hidden
phenomenological uncertainties. The starting point [2] was the generating
function

$\rho(z,s)=\sum_{n=0}^{\infty}z^{n}\sigma_{n}(s),~{}\rho(1,s)=1,~{}\sigma_{n}=0~{}{\rm
at}~{}n>n_{max}=\sqrt{s}/m,$ (2)

where $m$ is the hadron mass. One may use inverse Mellin transformation:

$\sigma_{n}(s)=\frac{1}{2\pi i}\oint\frac{dz}{z^{n+1}}\rho(z,s)$ (3)

to find $\sigma_{n}$ if $\rho(z,s)$ is known. One may calculate integral (3)
by saddle point method. The equation (of state):

$n=z\frac{\partial}{\partial z}\ln\rho(z,s)$ (4)

defines mostly essential value $z=z(n,s)$. Therefore, only

$z<z_{max}=z(n_{max},s)$ (5)

have the physical meaning.

One may write $\rho(z,s)$ in the form:

$\rho(z,s)=\exp\left\\{\sum_{l=0}^{\infty}z^{l}b_{l}(s)\right\\},$ (6)

where the ”Mayer group coefficient” $b_{l}$ can be expressed through
correlators $c_{k}(s)$:

$b_{l}(s)=\sum_{k=l}^{\infty}\frac{(-1)^{(k-l)}}{l!(k-l)!}c_{k}(s).$

Let us assume now that we have Poisson distribution, i.e. if in the sum:

$\ln\rho_{(}z,s)=\sum_{k}\frac{(z-1)^{k}}{k!}c_{k}(s)$ (7)

one may leave first term, then it is easy to see that

$z(n,s)=n/c_{1}(s),~{}c_{1}(s)\equiv\bar{n}(s),$ (8)

are essential and in the VHM region:

$\ln\sigma_{n}(s)=-n\ln\frac{n}{c_{1}(s)}(1+O(1/\ln n))=-n\ln z(n,s)(1+O(1/\ln
n)).$ (9)

Therefore, in considered case with $c_{k}=0,~{}k>1,$ exist following
asymptotic estimation for $n>>1$:

$\ln\sigma_{n}\simeq-n\ln z(n,s),$ (10)

i.e. $\sigma_{n}$ is defined in VHM region mainly by the solution of Eq.(4)
and the correction can not change this conclusion. It will be shown that that
kind estimation is hold for arbitrary asymptotics of $\sigma_{n}$. The
definition (1) based on this observation.

If we understand $\sigma_{n}$ as the ”partition function” in the VHM region
then $z$ is the $activity$ usually introduced in statistical physics.
Correspondingly the chemical potential $\mu$ is defined trough $z$:

$\mu=T\ln z.$ (11)

Combining this definition with estimation (10) we define $\sigma_{n}$ through
$\mu$. But, $if$ this estimation does not depend from asymptotics of
$\sigma_{n}$ over $n$, i.e. if it has general meaning, then it can be used for
definition of $\mu(n,s)$ through $\sigma_{n}(s)$ and $T(n,s)$ at $n>>1$. Just
this formal idea is realized in (1): it can be shown in Sec.III that (1) is
correct at the asymptotical value of $n$.

III.

Now we will make the important step. To put in a good order our intuition it
is useful to consider $\rho(z,s)$ as the $nontrivial$ function of $z$. In
statistical physics the thermodynamical limit is considered for this purpose.
In our case the finiteness of energy $\sqrt{s}$ and of the hadron mass $m$ put
obstacles on this way since the system of produced particles necessarily
belongs to the energy-momentum surface333It must be noted that the canonical
thermodynamic system belongs to the energy-momentum shell because of the
energy exchange, i.e. interaction, with thermostat. The width of the shell is
defined by the temperature. But in particle physics there is no thermostat and
the physical system completely belongs to the energy momentum surface.. But we
can continue $theoretically$ $\sigma_{n}$ to the range $n>n_{max}$ and
consider $\rho(z,s)$ as the nontrivial function of $z$. This step hides the
assumption that nothing new appear at $n>n_{max}$, i.e. the VHM interval
$\bar{n}<<n<n_{max}$ is sufficiently wide to represent main physical
processes.

Let us consider the analog generating function which has the first $n<n_{max}$
coefficient of expansion over $z$ equal to $\sigma_{n}$ and higher
coefficients for $n\geq n_{max}$ are deduced from continuation of theoretical
value of $\sigma_{n}$ to $n\geq n_{max}$. Then the inverse Mellin
transformation (3) gives a good estimation of $\sigma_{n}$ through this
generating function if the fluctuations near $z(n,s)$ are Gaussian or, it is
the same, if

$\left.\frac{|2n-z^{3}\partial^{3}\ln\rho(z,s)/\partial
z^{3}|}{|n+z^{2}\partial^{2}\ln\rho(z,s)/\partial
z^{2}|^{3/2}}\right|_{z=z(n,s)}<<1.$ (12)

Notice that if the estimation (10) is generally rightful then one can easily
find that l.h.s. of (12) is $\sim 1/n^{1/2}$. Therefore, one may consider
$\rho(z,s)$ as the nontrivial function of $z$ considering $z(n,s)<z_{max}$ if
$\bar{n}<<n<n_{max}$.

Then it is easily deduce that the asymptotics of $\sigma_{n}(s)$ is defined by
the leftmost singularity, $z_{c}$, of function $\rho(z,s)$ since, as it
follows from Eq.(4), the singularity ”attracts” the solution $z(n,s)$ in the
VHM region. In result we may classify asymptotics of $\sigma_{n}$ in the VHM
region if (12) is hold.

Our problem is reduced to the definition of possible location of leftmost
singularity of $\rho(z,s)$ over $z>0$444The singularities in complex $z$ plane
will not be considered since they lead only to oscillations in multiplicity
distribution.. It must be stressed that the character of singularity is not
important for definition of $\mu(n,s)$ in the VHM region at least with
$O(1/\ln n)$ accuracy. One may consider only three possibility at
$n\to\infty$: (I) $z(n,s)\to z_{c}=1$; (II) $z(n,s)\to
z_{c},~{}~{}1<z_{c}<\infty$; (III) $z(n,s)\to z_{c}=\infty$. The structure of
complex $z$ plane is much more complicate but for our purpose the above
described picture is sufficient.

Correspondingly one may consider only three type of asymptotics in the VHM
region: (I) $\sigma_{n}>O(e^{-n})$; (II) $\sigma_{n}=O(e^{-n})$. Such
asymptotics is typical for hard processes with large transverse momenta, like
for jets [3]; (III) $\sigma_{n}<O(e^{-n})$. That asymptotic behavior is
typical for multiperipheral-like kinematics, where the longitudinal momenta of
produced particles are noticeably higher than the transverse ones [3].

Therefore the case (I) is the best candidate for phase transition since in
this case the cross sections are comparatively large in the VHM region, i.e.
particles ”intensively” produced in that case. Notice that if (I) is not
realized in nature then the (II) kind processes would dominate in the VHM
region.

Let us consider now the estimation (1). It follows from (3) that, up to the
preexponential factor,

$\ln\sigma(n,s)\approx-n\ln z(n,s)+\ln\rho(z(n,s),s).$ (13)

We want to show that, in a vide range of $n$ from VHM region,

$n\ln z(n,s)\sim\ln\rho(z(n,s),s).$ (14)

Let as consider now the mostly characteristic examples.

(I) Singularity at $z=1$. The physical meaning of singularity at $z=1$ may be
illustrated by the droplet model [6]. The Mayer’s group coefficient, see (6),
for cluster from $l$ particle is $b_{l}(\beta)\sim\exp\\{-\beta\tau
l^{(d-1)/d}\\},$ where $\tau l^{(d-1)/d}$, $l>>1$, is the surface tension
energy, $d$ is the dimension. Therefore, if $d>1$ the series over $l$ in (6)
diverges at $z=1$.

This case was considered in [2] in details. In the used lattice gas
approximation $\ln{z}(n)\sim n^{-5}$ and $\ln\sigma_{n}\approx-n^{-4}=-n\ln
z(n)(1+O(1/n)).$ Notice that the simplest droplet model predicts unphysical
asymptotics: $\sigma_{n}\to const$ in the VHM region.

(II) Singularity at $1<z_{c}<\infty$. Let us consider one jet contribution:
$\ln\rho(z,s)=-\gamma\ln(1-\bar{n}_{j}(s)(z-1))$. In this case
$z(n,s)=z_{c}(1-\gamma/n),~{}n>>\gamma,$ and $\ln\sigma_{n}=-n\ln
z(n,s)(1+O(\ln n/n)).$

(III) Singularity at $z=\infty$. For $k$ Pomeron exchange:
$\ln\rho(z,s)=c_{k}(s)(z-1)^{k}$. In this case $z(n)=(n/kc_{k})^{1/k}>>1$ and
$\ln\sigma_{n}\approx-n\ln z(n)(1+O(1/\ln n)).$

One can conclude:

(i) The definition (1) in the VHM region is rightful since the correction
falls down with $n$. On this stage we can give only the qualitative estimation
of corrections. Nevertheless (1) gives the correct $n$ dependence in the VHM
region.

(ii) Activity $z(n,s)$ tends to $z_{c}$ from the right in the case (I) and
from the left if we have the case (II) or (III).

(iii) The accuracy of estimation of the chemical potential (1) increase from
(III) to (I).

IV.

The temperature $T$ is the next problem. The temperature is introduced usually
using Kubo-Martin-Schwinger (KMS) periodic boundary conditions. But this way
assumes from the very beginning that the system (a) is equilibrium and (b) is
surrounded by thermostat through which the temperature is determined. The
first condition (a) we take as the simplification which gives the equilibrium
state.

The second one (b) is the problem since there is no thermostat in particle
physics. For this reason we introduce the temperature as the Lagrange
multiplier $\beta=1/T$ of energy conservation law [3]. In such approach the
condition that the system is in equilibrium with thermostat replaced by the
condition that the fluctuations in vicinity of $\beta$ are Gaussian.

The interesting for us $\rho(z,{s})$ we define through inverse Laplace
transform of $\rho(z,\beta)$:

$\rho(z,s)=\int\frac{d\beta}{2\pi i\sqrt{s}}e^{\beta\sqrt{s}}\rho(z,\beta).$
(15)

It is known that if the interaction radii is finite, i.e. the hadron mass is
finite, then the equation (of state):

$\sqrt{s}=-\frac{\partial}{\partial\beta}\rho(z,\beta)$ (16)

have real positive solution $\beta(n,s)$ at $z=z(n,s)$. We will assume that
the fluctuations near $\beta(n,{s})$ are Gaussian. This means that the
inequality [3]:

$\left.\frac{|\partial^{3}\ln\rho(z,\beta)/\partial\beta^{3}|}{|\partial^{2}\ln\rho(z,\beta)/\partial\beta^{2}|^{3/2}}\right|_{z=z(n,s),\beta=\beta(n,s)}<<1$
(17)

is satisfied. Therefore, we prepare the formalism to find ”thermodynamic”
description of the processes of particle production assuming that this
$S$-matrix condition of equilibrium (17) is hold555Introduction of
$\beta(n,s)$ allows to describe the system of large number of degrees of
freedom in terms of single parameter $\beta(n,s)$, i.e. it is nothing but the
useful trick. It is no way for this reason to identify entirely $1/\beta(n,s)$
with thermodynamic temperature where it has the self-contained physical sense.
Nevertheless path-integral representation of $\rho(\beta,z)$ defined from
$S$-matrix coincides with Feynman-Kac representation of grand partition
function [3] if (17) is hold. It must be noted also that the energy spectrum
of produced particle in this case have Boltzmann form, $e^{-\beta\eta}$..

I want to underline that our thermal equilibrium condition (17) have absolute
meaning: if it is not satisfied then $\beta(n,s)$ loses every sense since the
expansion in vicinity of $\beta(n,s)$ leads to the asymptotic series. In this
case only the dynamical description of $S$-matrix can be used.

It is not hard to see [3] that

$\frac{\partial^{l}}{\partial\beta^{l}}\ln
R(z,\beta)|_{z=z(n,s),\beta=\beta(n,s)}=<\prod_{i=1}^{l}(\eta_{i}-<\eta>)>_{n,s}$
(18)

is the $l$-point energy correlator, where $<...>_{n,s}$ means averaging over
all events with given multiplicity and energy. Therefore (17) means
”relaxation of $l$-point correlations”, $l>2$, measured in units of the
dispersion of energy fluctuations, $l=2$. One can note here the difference of
our definition of thermal equilibrium from thermodynamical one [5].

V.

We may conclude that:

(i) The definition of chemical potential (1) was discussed. This important
observable can be measured on the experiment directly. Chemical potential,
$\mu(n,s)$, must decrease in the VHM region if the first order phase
transition occur, case (I), and it rise in opposite case, see (II) and (III),
see Sec.III.

(ii) We are forced to assume that the energy and the multiplicity are
sufficiently large, i.e. the experimental value $z^{exp}(n,s)$ is sufficiently
close to $z_{c}=1$. In opposite case the leading leftmost singularity over $z$
would not be ”seen” on experiment and the production processes constitutes
from the complicated mixture of subprocesses.

(iii) The cross section $\sigma_{n}$ falls down rapidly with $n$ and for this
reason the VHM events are hardly observable. One may avoid this problem
considering the finite energy heavy ion collisions as the most candidates of
processes described by methods of thermodynamics and $z_{c}$ is easier
”reachable” in this case.

(iv) One can define $z(n,s)$ also directly from Eq.(4):

$n=z\frac{\partial}{\partial z}\ln\sum_{n}z^{n}\sigma_{n}^{exp}(s),$ (19)

using experimental values $\sigma_{n}^{exp}(s)$. But comparing (19) with
definition (10),

$n\ln z\simeq-\ln\sigma_{n}^{exp}(s),$ (20)

it seems that last one gives more definite value of $z^{exp}(n,s)$ than the
”integral” equation (19) especially since the statistical errors are large in
the VHM region and the theoretical correction to Eq.(20) are small, $\sim
1/n$.

Summarizing the results we conclude: if the energy is sufficiently large, i.e.
if $z_{max}$ is sufficiently close to $z_{c}=1$, if the multiplicity is
sufficiently large, so that (17) is satisfied and $z(n,s)$ can be sufficiently
close to $z_{c}$, then one may have confident answer on the question:
observable or not the first order phase transition in hadron/ion collisions.
The heavy ion collisions are favorable to observe the phase transition.

Acknowledgements.

I would like to thank participants of 7-th International Workshop on the ”Very
High Multiplicity Physics” (JINR, Dubna) for stimulating discussions. I am
grateful to V.Priezzhev, A.Sissakian, A.Sorin and V.Kekelidze for valuable
attention.

## References

  * [1] BNL Report, Hunting the Quark Gluon Plasma, BNL-73847-2005; C.Alt et al., The NA-49 Collaboration, nucl-ex/0710.0118; M.Creutz, Phys. Rev., D15 (1977) 1128; M.Gazdzicki and M.I.Gorenstein, Acta Physica Polonica, B 30, 2705, (1999)
  * [2] J.Manjavidze and A.Sissakian, Proc. VHM Physics Workshops, (World Scient., 2008)
  * [3] J.Manjavidze and A.Sissakian, Phys. Rep., 346 (2001) 1, hep-ph/0105245
  * [4] J.Manjavidze, Phys.Part.Nucl. 30 (1999) 49, hep-ph/9802318
  * [5] N.N.Bogolyubov, Studies in Statistical Mechanics, (North-Holland Publ. Co., Amsterdam, 1962)
  * [6] T.D.Lee and C.N.Yang, Phys.Rev., 87 (1952) 404, 410; J.S.Langer, Ann.Phys., 41 (1967) 108


# Region of Interest Loss for Anonymizing Learned Image Compression

Christoph Liebender1,2 Ranulfo Bezerra1 Kazunori Ohno1 Satoshi Tadokoro1
1Graduate School of Information Sciences, Tohoku University, Japan.2Christoph
Liebender is with the robotics group of Julius Maximilian University Würzburg,
Germany. This paper originated during his research stay at Tohoku University.
<EMAIL_ADDRESS>work was partially supported by the
robotics group of the Julius Maximilian University Würzburg, the Innovation
and Technology Commission of the HKSAR Government under the InnoHK initiative,
and this research was performed by the commissioned research fund provided by
F-REI (JPFR23010101).

###### Abstract

The use of AI in public spaces continually raises concerns about privacy and
the protection of sensitive data. An example is the deployment of detection
and recognition methods on humans, where images are provided by surveillance
cameras. This results in the acquisition of great amounts of sensitive data,
since the capture and transmission of images taken by such cameras happens
unaltered, for them to be received by a server on the network. However, many
applications do not explicitly require the identity of a given person in a
scene; An anonymized representation containing information of the person’s
position while preserving the context of them in the scene suffices. We show
how using a customized loss function on region of interests (ROI) can achieve
sufficient anonymization such that human faces become unrecognizable while
persons are kept detectable, by training an end-to-end optimized autoencoder
for learned image compression that utilizes the flexibility of the learned
analysis and reconstruction transforms for the task of mutating parts of the
compression result. This approach enables compression and anonymization in one
step on the capture device, instead of transmitting sensitive, nonanonymized
data over the network. Additionally, we evaluate how this anonymization
impacts the average precision of pre-trained foundation models on detecting
faces (MTCNN) and humans (YOLOv8) in comparison to non-ANN based methods,
while considering compression rate and latency.

##  I Introduction

Figure 1: Depiction of example usage application: collision avoidance system
with humans in the action space of mobile robots. Cameras use our method to
anonymize and compress images of the scene, for a server to first decode the
data and then to further process it by detecting the position of individuals
(yellow bounding boxes) while their biometric information is obfuscated (boxes
on faces) by the compression / decompression process. Robots rely on the
detected positions plan their path in the shared space.

Utilizing cameras to obtain visual information of a given scene is a
fundamental part of remote sensing and at the core of many applications.
Digital twin generation of a scene is one of these applications. In this task,
image data of many cameras can be used to digitally replicate a 3D scene with
its spatial and semantic information. An emerging use case for digital twin
generation are industry applications, where a manufacturing site or warehouse
is monitored by such cameras. With a digital twin of such a site, mobile
robots may be added to the industry process much easier, since task and route
planning is aided by an accurate digital twin. Shown in Fig. 1, this
generation of semantic information becomes even more important as soon as
humans join the action space of those mobile robots, as robots then need to
adapt to the actions of humans, which requires the detection of both positions
and actions of them in the scene.

To generate an accurate digital twin, multiple viewpoints of the scene are
necessary to reduce occlusions. This yields a requirement for many cameras,
that then transmit their image data to a processing server. As it turns out,
many cameras sharing a network to send their high-resolution image data
quickly reach the bandwidth limit of that network, yielding a requirement of
some form of image or video compression. This pre-processing before
transmission requires computational resources present on the camera, turning
it into an edge-computing device.

Since these edge-devices now share a network, we must consider the security
implications that come with this approach. The transmitted image data contains
potentially sensitive information, especially in public environments. Even a
network that is considered trusted bears a chance that an attacker may recover
the data that is transmitted over it. Implementations of such an approach must
ensure that sensitive data is kept private.

When image data is collected in public spaces, privacy is not the only
consideration to make, as the continuous recording and tracking of individuals
records biometric information that is not necessary for a task like location
and action detection. This yields an additional requirement of sufficient
anonymization of the data. In fact, this matter has reached as high a level as
the European Parliament: the so-called _Artificial Intelligence Act_ as
proposed in 2021 puts “real-time remote biometric identification systems in
publicly accessible spaces” in the category of “prohibited AI practices”, the
highest prohibition category [1].

However, edge-side anonymization requires edge devices to have the
computational resources to both employ anonymization and compression as
_seperate_ tasks. Current ANN-based anonymization techniques have not been
developed with the aspect of platform constraints in mind; That is: edge
devices like those that we use for this research cannot run state of the art
anonymization models since their weights do not fit into device memory. As an
example, similar approaches like that of Yang _et al_. [2] and More _et al_.
[3] use graphics cards with 11GB of memory for their experiments, while we
manage to reduce this requirement down to 4GB of a Jetson Nano. Furthermore,
edge devices may not come with hardware implementations of the employed
codecs, adding to the latency of ANN anonymization. Ideally, the computational
load of edge-side anonymization is shifted towards the server-side, while
there is no need in trusting a vulnerable communication channel.

((a))

((b))

((c))

((d))

Figure 2: Results of our anonymizing image compression using a region-of-
interest loss based on bounding boxes. MTCNN (face) and YOLOv8 (human)
detections are annotated in cyan and yellow, respectively. Compression rates
are given in bits per pixel. (2(a)): original. (2(b)): our method, no face
detections, persons in foreground stay detected. (2(c)): AV1 (libaom) at
similar bitrate, all four faces detected. (2(d)): JPEG at similar bitrate, one
face detection, and no human detections.

To address this issue, we propose a learned image compression method that
anonymizes human faces in the image. Our learned method employs a
convolutional autoencoder that uses a generalized ROI loss function during
training. By using a symmetric autoencoder, the computational complexity of
anonymization is effectively split between the edge- and server-side. Since
the analysis transformation into a latent space compresses the input data, we
obtain an image codec that depends on both model architecture and the
individual weights. This adds two unknown variables to an attacker without
hardware access. Our approach utilizes the flexibility provided by the
parameter sets of the analysis and synthesis transforms to inhibit the
reconstruction of human faces while reconstructing other parts. This way, the
extraction of information using other algorithms or neural networks trained on
raw image input remains feasible. Since lossy image compression will
inevitably cause a loss of precision of detection methods, we evaluate this
deviation by comparing the results of the YOLO [4, 5] object detection
approach on the validation dataset of CrowdHuman [6], by first compressing the
image and then feeding it through the detection network. The bounding boxes of
persons returned are then compared to the ground truth using the the average
precision (AP) metric of the Pascal VOC Challenge [7, 8]. Using the same
method, we show how our approach leaves faces undetectable, by using bounding-
box outputs of the MTCNN face detection network [9]. To set these results into
context, we compute the AP of JPEG [10] and AV1 [11] at similar compression
ratios, while also comparing compression latency and rate, as exemplified in
Fig. 2.

The main contributions of this paper are:

  1. 1.

the combination of a novel ROI-loss function that only operates on bounding
boxes and learned compression to achieve a anonymizing compression result,

  2. 2.

the evaluation of the change of precision of YOLO and MTCNN caused by
anonymization / image compression, and

  3. 3.

the comparison of compression latency between our approach and JPEG/AV1, on
two different systems.

##  II Related Work

Classical image compression is a well-studied field of computer science.
Algorithms like JPEG [10] and AV1 [11] rely on hand-crafted algorithms to
reduce the entropy of the encoded representation. Recent research shows that
learned image compression outperforms non-learned methods in distortion
metrics like PSNR and MS-SSIM. [12]

###  II-A Rate-distortion optimized compression

One of the first competitive learned approaches was introduced by Ballé _et
al_. in 2017 [13]. It uses layered convolutions for its analysis and synthesis
transforms, utilizing the generalized divisive normalization (GDN)
nonlinearity [14] between layers and an uniform quantizer at the bottleneck to
achieve coding-efficiency. In their work, they define a _rate-distortion loss_
, which addresses both the reconstruction error of the compression
(distortion) and the size of the encoded image (rate), by combining them as a
weighted sum. Since then, improvements to this method have been made partly by
improving the entropy model [15, 16, 17], at the cost of increased
computational requirements. One recent approach by Liu _et al_. [18] combines
layered convolutions with transformers to reach state-of-the-art rate-
distortion values. The idea is that our method is to be used in environments
where computational costs for compression cannot rise indefinitely, which
renders complex ANN-based compression methods unsuitable.

###  II-B Task-optimized compression

Classical compression algorithms try to preserve the image quality perceived
by humans, not regarding a possible downstream usage of machine vision
applications. Chaiman _et al_. [19] present end-to-end optimized image
compression, where the detection confidence of a downstream algorithm is added
to the rate-distortion loss of the learned compression. Xiao _et al_. [20]
present an identity preserving loss that employs this principle for the task
of face recognition, to achieve low compression rates with the model focuses
on only reconstructing faces. Work by Ma _et al_. [21] optimizes perceptual
quality by allocating most bits for a given ROI and generating fake texture
elsewhere. Our generalized ROI-loss can also be used to allocate more bits for
arbitrary regions in an image. However, we do not use a ROI network that
extracts this foreground for each single compression during preprocessing; We
simply use pre-annotated bounding boxes during training to achieve this
behaviour.

###  II-C ANN based anonymization and censorship

Leaving the field of compression shows that there has already been progress
made considering the task of hiding sensitive data in an image. One use-case
is nudity censorship using adversarial training, as shown by Simões _et al_.
[22] and More _et al_. [3]. The task of face anonymization has thus far been
achieved by face generation and replacement or inpainting [23, 24, 25, 2].
Most notably however, previous research on face anonymization does not
integrate this task into compression.

Up to now, literature shows that the main interest lies in optimizing for
distortion metrics, as well as some in generative-based anonymization and less
in task-optimization. The main standing point of our paper is the combination
of both rate-distortion optimized compression and anonymization, as this has
not yet been explored by previous research on learned image compression.

##  III Methodology

To achieve compression, one needs to reduce the entropy of a given input. For
this, a nonlinear analysis transform is used first, to reduce the
dimensionality of the data by only extracting significant parts. (That is:
parts deemed significant by the application.) An example is the discrete
cosine transform of JPEG [10]. Then, the data needs to be quantized and coded
into a compact representation, for example, by arithmetic coding. In order to
reconstruct the original image from this compact representation, this process
is inverted. The general approach of learned image compression is replacing
the nonlinear transform by a convolutional neural network, and introducing
entropy coding at the bottleneck. This also applies in our case. Shown in Fig.
3, our input image $x$ is analyzed by the encoder to produce a latent
representation $y$ that is then quantized and efficiently coded into a
representation to be sent over the network channel. At the receiving end, an
entropy decoder reconstructs the latent representation $\hat{y}$ which is then
used by the convolutional decoder to synthesize an image $\hat{x}$ that
ideally comes close to the original.

Now, we introduce the details of our architecture and introduce other
parameters present in Fig. 3, which are key to the definition of the used
loss-function.

Figure 3: Autoencoder architecture with loss components. Convolutions have
kernel size 5 with stride 2, and map the RGB image onto $N$ channels. To
assess how AP and bpp vary with different architecture depths and widths, we
test different numbers of hidden layers $M\in\left\\{1,2\right\\}$ and feature
maps $N\in\left\\{128,256\right\\}$. The loss consists of a sum of the
compression rate provided by the entropy bottleneck, background-loss and
losses for each head-boxes and visual-boxes of persons. (Images of both Jetson
Nano and DGX Station A100 were taken from the Nvidia website.)

###  III-A Architecture

We use the fully factorized prior architecture introduced by Ballé _et al_. in
2018 [15]. In their work, they use a generalized divisive normalization non-
linearity that introduces multiple learnable parameters. Johnston _et al_.
[26] argue that by using this non-linearity, unnecessary computational
overhead is introduced by computing square roots, while performance-
improvements of the model are insignificant. They continue to simplify this
activation by removing parameters that cause this computational overhead,
resulting in a simplified GDN. We use this simplified version to improve
latency on the embedded device and denote it by GDN1. As shown in Fig. 3, we
vary the depth $M$ and width $N$ of the networks to compare the effects on AP
and compression rate. Each convolutional layer in the analysis (synthesis)
transform reduces (increases) both width and height dimensions of the input
tensor by two, caused by a step of 2 for each convolution kernel. The size of
these kernels are fixed to $5\times 5$.

###  III-B Training

During training, we utilize the bounding boxes provided by the CrowdHuman [6]
dataset to achieve the anonymization behaviour. This dataset includes pre-
annotated bounding boxes for both the head and visible region of persons.
Images are rescaled to $512\times 512$ and bounding boxes are adjusted
accordingly. We start defining our loss-function with the rate-distortion
components:

$\displaystyle\mathcal{R}$ $\displaystyle=\underbrace{\mathbb{E}_{x\sim
p_{x}}\left[-\log_{2}p_{\hat{y}}(\hat{y})\right]}_{\text{Shannon cross entropy
(Rate)}}$ (1) $\displaystyle\mathcal{L}_{\text{bg}}$
$\displaystyle=\underbrace{\mathbb{E}_{x\sim p_{x}}\lVert
x-\hat{x}\rVert^{2}}_{\text{MSE (Distortion)}},$ (2)

similar to Ballé _et al_. [15], where $p_{x}$ is the distribution of images
$x$ (during training: a batch), and $p_{\hat{y}}(\hat{y})$ the entropy-model.

####  III-B1 ROI-Loss: generalized region-of-interest loss

Now, we introduce a generalized ROI loss that operates on bounding boxes:

$\displaystyle\mathcal{L}_{k}(B)=\mathbb{E}_{x\sim p_{x},b\in
B_{x}}\left[k-1^{k}\cdot\lVert x_{b}-\hat{x}_{b}\rVert^{2}\right],$ (3)

as the expectancy over the set of bounding boxes $B$ for images of $p_{x}$.
$B_{x}\subseteq B$ are bounding boxes present in an image $x$, while $x_{b}$
is an image cropped w.r.t. $b\in B_{x}$. For $k=0$, this yields the average
loss over given ROI boxes in an image for each image in a batch $p_{x}$. Given
that inputs and outputs are normalized such that
$x,\hat{x}\in\left[0,1\right]$ holds, this loss can be inverted for each box
by setting $k=1$, such that close reconstructions compute to a higher loss. We
utilize both variants for boxes provided in the training dataset, non-inverted
for persons (vbox; visible bounding box) and inverted for heads (hbox; head
bounding box):

$\displaystyle\mathcal{L}_{\text{vbox}}$
$\displaystyle=\mathcal{L}_{0}(\text{vbox}),$ (4)
$\displaystyle\mathcal{L}_{\text{hbox}}$
$\displaystyle=\mathcal{L}_{1}(\text{hbox}).$ (5)

The total loss for one batch then computes as a weighted sum over all
components:

$\displaystyle\mathcal{L}=\lambda_{r}\mathcal{R}+\lambda_{\text{bg}}\mathcal{L}_{\text{bg}}+\lambda_{\text{hbox}}\mathcal{L}_{\text{hbox}}+\lambda_{\text{vbox}}\mathcal{L}_{\text{vbox}}$
(6)

introducing four new hyperparameters $\lambda_{i}$ that can be used to
prioritize each task. We use the same hyperparameters for all depth and width
configurations, but stop the training of the model with one hidden layer early
after 293 epochs. The two-hidden-layer model was trained up to an epoch of
874, with the narrow variant up to 750. The hyperparameters are:
$\lambda_{r}=0.04,\>\lambda_{\text{bg}}=1,\>\lambda_{\text{hbox}}=0.6,\>\lambda_{\text{vbox}}=1$.
Although there remains room for optimization on both hyperparameters and depth
/ width of the architecture, we do not conduct a hyperparameter study in this
work.

##  IV Evalutaion

We choose CrowdHuman [6] as the dataset for training and validation. To
evaluate our method, we use both pretrained models YOLOv8 [5] and MTCNN [27]
with frozen weights. For YOLOv8, we choose the largest detection model type
YOLOv8x for best precision, which is pretrained on the COCO [28] dataset. To
assess the precision loss by compression, we first apply compression for each
image of the CrowdHuman validation dataset (N = 4370), and then run both YOLO
and MTCNN detection. The detected bounding boxes output by YOLO of class
person are compared to the visible bounding boxes of CrowdHuman; In the case
of MTCNN, it is the output face bounding boxes against ground truth head
bounding boxes. This will inevitably cause low precision values for MTCNN,
since it is not trained on detecting heads; However since we are only
interested in the differences of precision, this is still viable because a
certain set of heads in the dataset will show faces. The same applies for YOLO
trained on the COCO dataset, which includes more classes that can be detected.
Here, the drop in precision is lower than for MTCNN. We believe however that
since these foundation models can be considered widespread, it makes sense to
evaluate their accuracy instead of the accuracy of some specialized models.

We choose the average precision metric of the Pascal VOC challenge [7], the
implementation being provided by Padilla _et al_ [8]. This gives us both
average precision, as well as the number of true- and false-positives. In
order to compare our results to a baseline, we choose JPEG and AV1 at
different quality presets; Implementations for both codecs are provided by the
python imaging library PIL [29] with AV1 support [30].

For a latency assessment, we record the runtime of compression and
decompression for every image in the validation dataset, by repeating the
process ten times per image and pooling the runtimes over all images, to then
compute minimum, arithmetic mean and sample standard deviation. It is
important to note that the compression processes for AV1 run on the CPU, since
the support for AV1 encoding is just now starting to be implemented in
hardware. In our application, the runtime of decoding will happen on a
powerful server, while encoding happens on a device with constraints on
computing power. The support for hardware-accelerated AV1 encoding on low-
power devices will probably only happen in the far future, which is why it is
reasonable to look at CPU encoding latency for this method. In contrast, we
provide both GPU and CPU measurements for our method where possible. Our
actual test hardware consists of two devices:

  * •

Nvidia DGX Station A100

    * –

4 x 80GB Nvidia A100 GPU

    * –

64-core AMD EPYC 7742 CPU

  * •

Nvidia Jetson Nano

    * –

4GB Nvidia Maxwell GPU

    * –

Quad-core ARM Cortex-A57. (We skip benchmarking our method on this CPU.)

When using GPU computations, we do not rescale to half-precision floating
point, nor decrease the accuracy of matrix calculations. In case of the DGX
station, we use only one GPU for latency measurements.

##  V Results

###  V-A Precision

Fig. 4 shows visualized results of average precision over compression rate,
where Fig. 4(a) are the results for YOLOv8x and Fig. 4(b) for MTCNN. We
include JPEG and AV1 at three quality presets, and our method at three
configurations: $(N,M)\in\left\\{(128,2),(256,1),(256,2)\right\\}$. Note that
there is only one value for each configuration, since our method does not
facilitate setting a quality preset at runtime; This precision-distortion
trade-off is to be adjusted during training with the hyperparameters
$\lambda_{i}$ shown in Eq. 6. Our method with the configuration $(256,2)$
achieves similar precision of 19% AP on persons to AV1 at 10% quality (20% AP,
0.14 bpp) and JPEG at 10% quality (20% AP, 0.39 bpp), at a bitrate of 0.24
bits per pixel. Removing one hidden layer $(256,1)$ decreases the precision to
16% AP and increases the bitrate to 0.26 bits per pixel. Reducing the width
$(128,2)$ requires a higher bitrate to achieve similar precision values to
$(256,2)$. The precision on heads / faces of two hidden layers (0.1% AP) is
seven times lower in comparison to 0.7% AP of AV1 at 1% quality, while one
hidden layer reduces this precision to 0.08% AP at a similar bitrate to JPEG
at 5% quality with a precision of 0.9% AP. Reducing the width of the network
to 128 and training up to 750 epochs yields 0.07% AP on faces.

Considering true and false positive detections, Tab. I shows that using
$(128,2)$ as a model configuration decreases the number of true positives to
150 in comparison to the lowest achieved rate of JPEG at 1% quality of 424
true positives.

((a))

((b))

Figure 4: Average precision over compression rate for JPEG, AV1 and our method with three configurations, at an IOU-threshold of 0.5. For JPEG and AV1, we choose three quality presets: 1, 5 and 10 percent. Error-bars indicate one sample standard deviation of compression rate over the validation dataset. Subfigure 4(a) shows the average precision of YOLOv8 on persons, 4(b) the average precision of MTCNN on faces; Both after compression. TABLE I: Number of true and false positive face detections. Ours (best in total) in bold. Method | Preset | TP | FP
---|---|---|---
$(256,2)$ |  |  234 |  1159
$(256,1)$ |  |  191 |  1488
$(128,2)$ |  |  150 |  1105
AV1 |  1% |  1408 |  1949
|  5% |  2253 |  2966
|  10% |  3138 |  4345
JPEG |  1% |  424 |  613
|  5% |  1995 |  3180
|  10% |  4031 |  6399

###  V-B Latency

Fig. 5 compares each method on each of our devices in encoding latency. For
further reference, numeric values are provided in Tab. II and Tab. III. On the
Nvidia DGX station, our encoding method achieves lower latency than AV1 at 1%
quality at 52 ms, with the configuration $(256,2)$ while running on the GPU at
21 ms. Decreasing the depth of the model does not reduce the runtime of the
model, while reducing width does significantly; Increasing the runtime three
times and reducing it by a factor of two, respectively. On the Jetson Nano,
runtimes of both $(256,2),(256,1)$ configurations do not outperform AV1. The
reduced-width of $(128,2)$ decreases the latency to 239 ms, resulting in 110
ms less than minimum AV1.

((a))

((b))

Figure 5: Encoding latency comparison of JPEG and AV1 at 1, 5 and 10 to the left, against our method with three configurations to the right. Error-bars indicate one sample standard deviation in latency over the validation dataset. Logarithmic runtime scale. TABLE II: Jetson Nano: encoding latency. Values of Fig. 5(a). Lowest latency in italics. Best of ours in bold. Method | Preset | $\overline{t}\pm s_{t}$ | $\min t$ | Unit
---|---|---|---|---
$(256,2)$ |  GPU |  $0.634\pm 0.069$ |  $0.610$ |  [s]
$(256,1)$ |  |  $0.789\pm 0.010$ |  $0.755$ |
$(128,2)$ |  |  $\mathbf{0.239\pm 0.049}$ |  $0.221$ |
AV1 |  1% |  $0.35\pm 0.10$ |  $0.18$ |
|  5% |  $0.42\pm 0.11$ |  $0.20$ |
|  10% |  $0.49\pm 0.13$ |  $0.22$ |
JPEG |  1% |  $\mathit{4.03\pm 0.15}$ |  $3.72$ |  [ms]
|  5% |  $4.10\pm 0.15$ |  $3.76$ |
|  10% |  $4.24\pm 0.16$ |  $3.79$ |
TABLE III: DGX Station: encoding latency. Values of Fig. 5(b). Lowest latency in italics. Best of ours in bold. Method | Preset | $\overline{t}\pm s_{t}$ | $\min t$ | Unit
---|---|---|---|---
$(256,2)$ |  GPU |  $21.43\pm 0.46$ |  $20.81$ |  [ms]
|  CPU |  $134.1\pm 4.5$ |  $111.3$ |
$(256,1)$ |  GPU |  $72.9\pm 1.0$ |  $70.7$ |
|  CPU |  $181.6\pm 6.6$ |  $147.5$ |
$(128,2)$ |  GPU |  $\mathbf{11.50\pm 0.71}$ |  $10.41$ |
|  CPU |  $66.4\pm 4.9$ |  $40.9$ |
AV1 |  1% |  $52\pm 20$ |  $20$ |
|  5% |  $64\pm 21$ |  $25$ |
|  10% |  $76\pm 24$ |  $29$ |
JPEG |  1% |  $\mathit{633\pm 73}$ |  $564$ |  [µs]
|  5% |  $637\pm 28$ |  $570$ |
|  10% |  $671\pm 29$ |  $577$ |

##  VI Discussion

The results show that anonymization behavior can be learned using less-complex
architectures. Reducing the number of available parameters decreases
downstream people detection precision at similar bit-rates, while providing
latency benefits in case of networks with reduced width but unchanged depth.
Tab. I shows that while our method successfully reduces the number of true
positive face detections down to 150 over the whole test dataset, the number
of true positive face detections is not zero. The reason for this are large
faces, of which the training dataset includes only few. In comparison with
JPEG, we achieve a better precision-rate trade-off across all of our
configurations. Comparing to AV1 shows that ANN-based compression is able to
outperform it on latency.

While standardized codecs excel in many areas, they fall short in providing
application flexibility. One notable example of this limitation is the
inability to anonymize faces during the encoding and decoding process. We
especially argue that the combination of both compression and anonymization
comes with benefits that other approaches cannot provide; Considering the
handling of sensitive data, using our method alleviates any distrust present
in the party tasked with decompression, since the original image data is
already altered during encoding. Our method also renders man-in-the-middle
attacks meaningless, since access to the encoded data requires knowledge of
both decoder architecture and weights to reconstruct the image data. Lastly,
by combining anonymization and compression, we show that it is unnecessary to
chain standardized codecs and high-complexity ANN-based anonymization
techniques introduced by previous research. This especially benefits the use
of our method in edge-devices, since a part of the process happens on another
device tasked with decoding.

Low-bitrate compression on lower-power devices remains a rarely approached
topic. In this field, learned image compression is a competitive alternative,
especially when dedicated hardware encoding implementations are unavailable.
On lower-power devices that have access to a dedicated GPU or TPU, the
advantage of using ANN-based compression is even greater, as these
accelerators further decrease latency. Furthermore, the implementation of more
generalized matrix extensions like those implemented by Intel [31] and
presented by ARM [32] into lower-power CPUs will make the usage of methods
like ours more feasible, even when a dedicated accelerator is unavailable.

Additionally, the application of our loss-function operating on bounding boxes
is not limited to face anonymization; It can easily be generalized by using it
with any dataset that comes with bounding boxes. This opens up the possibility
to hide or accentuate any kind of region of interest in images, like censoring
nudity or highlighting weapons.

##  VII Conclusion

In this paper, we show how using a region-of-interest loss operating on pre-
annotated bounding boxes can be used to utilize the flexibility of ANN based
compression methods to highlight or hide specific parts in an image. As an
application, we introduce the task of face anonymization. Our network learns
to in-paint faces in a given image, making them undetectable in the decoded
result, while keeping persons detectable. We show that this detectability of
humans is kept possible while faces become unrecognizable by comparing the
detection results of the foundation models YOLOv8 (people) and MTCNN (faces)
against the pre-annotated ground truth of the CrowdHuman dataset. Comparing
these results to widespread JPEG and state-of-the-art AV1-based image
compression as a baseline, we show that our method achieves much higher
precision on persons than JPEG at similar bitrates, while keeping face
detections at an absolute minimum. From a latency perspective, our method
outperforms AV1 on encoding while utilizing the flexibility of the ANN network
for the additional step of anonymization. Additionally, we discuss the
implications of the usage of our solution in edge computing environments.

## References

  * [1] European Parliamentary Research Service, “EU Artificial intelligence act,” June 2023. [Online]. Available: https://www.europarl.europa.eu/RegData/etudes/BRIE/2021/698792/EPRS˙BRI(2021)698792˙EN.pdf
  * [2] J. Yang, S. Qiao, Z. Wang, and Z. Zuo, “Adversarial Secret-Identity Generation Model for Face Anonymization in the Internet of Vehicles,” _IEEE Systems Journal_ , pp. 1–10, 2023.
  * [3] M. D. More, D. M. Souza, J. Wehrmann, and R. C. Barros, “Seamless Nudity Censorship: an Image-to-Image Translation Approach based on Adversarial Training,” in _2018 International Joint Conference on Neural Networks (IJCNN)_ , 2018, pp. 1–8.
  * [4] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [5] G. Jocher, A. Chaurasia, and J. Qiu, “YOLO by Ultralytics,” Jan. 2023. [Online]. Available: https://github.com/ultralytics/ultralytics
  * [6] S. Shao, Z. Zhao, B. Li, T. Xiao, G. Yu, X. Zhang, and J. Sun, “CrowdHuman: A Benchmark for Detecting Human in a Crowd,” _arXiv preprint arXiv:1805.00123_ , 2018.
  * [7] M. Everingham, L. V. Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (VOC) challenge,” _International Journal of Computer Vision_ , vol. 88, no. 2, pp. 303–338, sep 2009.
  * [8] R. Padilla, W. L. Passos, T. L. B. Dias, S. L. Netto, and E. A. B. da Silva, “A comparative analysis of object detection metrics with a companion open-source toolkit,” _Electronics_ , vol. 10, no. 3, p. 279, jan 2021.
  * [9] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks,” _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, Oct. 2016, arXiv:1604.02878 [cs]. [Online]. Available: http://arxiv.org/abs/1604.02878
  * [10] G. Wallace, “The JPEG still picture compression standard,” _IEEE Transactions on Consumer Electronics_ , vol. 38, no. 1, pp. xviii–xxxiv, Feb. 1992.
  * [11] J. Han, B. Li, D. Mukherjee, C.-H. Chiang, A. Grange, C. Chen, H. Su, S. Parker, S. Deng, U. Joshi, Y. Chen, Y. Wang, P. Wilkins, Y. Xu, and J. Bankoski, “A technical overview of av1,” _Proceedings of the IEEE_ , vol. 109, no. 9, pp. 1435–1462, 2021.
  * [12] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja, “CompressAI: a PyTorch library and evaluation platform for end-to-end compression research,” 2020.
  * [13] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” in _International Conference on Learning Representations_ , 2017. [Online]. Available: https://openreview.net/forum?id=rJxdQ3jeg
  * [14] ——, “Density modeling of images using a generalized normalization transformation,” _Int’l Conf on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016_ , Nov. 2015.
  * [15] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=rkcQFMZRb
  * [16] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Deep convolutional autoencoder-based lossy image compression,” in _2018 Picture Coding Symposium (PCS)_ , 2018, pp. 253–257.
  * [17] ——, “Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [18] J. Liu, H. Sun, and J. Katto, “Learned Image Compression With Mixed Transformer-CNN Architectures,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2023, pp. 14 388–14 397.
  * [19] L. D. Chamain, F. Racapé, J. Bégaint, A. Pushparaja, and S. Feltman, “End-to-End optimized image compression for machines, a study,” in _2021 Data Compression Conference (DCC)_ , Mar. 2021, pp. 163–172, iSSN: 2375-0359.
  * [20] J. Xiao, L. Aggarwal, P. Banerjee, M. Aggarwal, and G. Medioni, “Identity Preserving Loss for Learned Image Compression,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2022, pp. 517–526.
  * [21] Y. Ma, Y. Zhai, C. Yang, J. Yang, R. Wang, J. Zhou, K. Li, Y. Chen, and R. Wang, “Variable Rate ROI Image Compression Optimized for Visual Quality,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2021, pp. 1936–1940.
  * [22] G. S. Simões, J. Wehrmann, and R. C. Barros, “Attention-based Adversarial Training for Seamless Nudity Censorship,” in _2019 International Joint Conference on Neural Networks (IJCNN)_ , 2019, pp. 1–8.
  * [23] M. Maximov, I. Elezi, and L. Leal-Taixe, “CIAGAN: Conditional Identity Anonymization Generative Adversarial Networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [24] Q. Sun, L. Ma, S. J. Oh, L. Van Gool, B. Schiele, and M. Fritz, “Natural and Effective Obfuscation by Head Inpainting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [25] Z. Ren, Y. J. Lee, and M. S. Ryoo, “Learning to Anonymize Faces for Privacy Preserving Action Detection,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [26] N. Johnston, E. Eban, A. Gordon, and J. Ballé, “Computationally Efficient Neural Image Compression,” 2019.
  * [27] T. Esler, “Face Recognition Using Pytorch,” July 2023, original-date: 2019-05-25T01:29:24Z. [Online]. Available: https://github.com/timesler/facenet-pytorch
  * [28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [29] PIL Contributors, “Pillow Imaging Library (Fork),” July 2023, original-date: 2012-07-24T21:38:39Z. [Online]. Available: https://github.com/python-pillow/Pillow
  * [30] A. Piskun, “pillow-heif,” July 2023, original-date: 2021-09-10T16:24:22Z. [Online]. Available: https://github.com/bigcat88/pillow˙heif
  * [31] Intel, “Intel® Advanced Matrix Extensions Overview.” [Online]. Available: https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/advanced-matrix-extensions/overview.html
  * [32] M. Weidmann, “Introducing the Scalable Matrix Extension for the Armv9-A Architecture.” [Online]. Available: https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/scalable-matrix-extension-armv9-a-architecture