# Edit One for All: Interactive Batch Image Editing

Thao Nguyen Utkarsh Ojha Yuheng Li Haotian Liu Yong Jae Lee
University of Wisconsin–Madison
https://thaoshibe.github.io/edit-one-for-all

###### Abstract

In recent years, image editing has advanced remarkably. With increased human
control, it is now possible to edit an image in a plethora of ways; from
specifying in text what we want to change, to straight up dragging the
contents of the image in an interactive point-based manner. However, most of
the focus has remained on editing single images at a time. Whether and how we
can simultaneously edit large batches of images has remained understudied.
With the goal of minimizing human supervision in the editing process, this
paper presents a novel method for interactive batch image editing using
StyleGAN as the medium. Given an edit specified by users in an example image
(e.g., make the face frontal), our method can automatically transfer that edit
to other test images, so that regardless of their initial state (pose), they
all arrive at the same final state (e.g., all facing front). Extensive
experiments demonstrate that edits performed using our method have similar
visual quality to existing single-image-editing methods, while having more
visual consistency and saving significant time and human effort.

##  1 Introduction

Image editing has undergone a transformation in recent years with the aid of
modern generative models. The process has been greatly democratized, where
many of the sophisticated edits, which previously might have required hours
and niche expertise, can now be completed with relative ease in a matter of
minutes. For example, different types of learning based algorithms can be used
for image correction/adjustment [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] and
for manipulating the semantic contents of real images [13, 14, 15, 16, 17, 18,
19, 20, 21, 22]. Moreover, there are many different ways to edit an image. For
instance, a user can specify in text what changes they want - e.g., “make the
hair darker” [14, 23, 15, 24], or they can drag the contents of the image in
an interactive manner to shrink, enlarge, or move a part of an object [17, 19,
25, 16, 12].

Figure 1: Interactive Batch Image Editing. Given a single user edited image,
the goal is to automatically transfer that edit to new unseen images, so that
all edited images end up with the same final state as the user edited image.

Figure 2: Single Image Editing vs. Batch Image Editing. (a) Prior work (e.g.,
[17, 19, 25]) focuses on single image editing. (b) We focus on batch image
editing, where the user’s edit on a single image is automatically transferred
to new images, so that they all arrive at the same final state regardless of
their initial starting state. In this way, we can achieve time speed up and
reduce human effort in editing.

A common theme across many of these works is that the edits are designed to
work for _a single image_ at a time. For example, given an image of a cat with
open eyes, DragGAN [17] allows the user to drag the contents of _that
particular image_ so that both the eyes can be closed in the resulting cat
image. But what if we wish to apply the same edit to _many_ different types of
cats so that we can close all of their eyes? While one could perform dragging
on each cat separately, the whole process would be quite cumbersome (requiring
lots of human annotation) and time intensive (requiring optimization for each
image).

In this paper, we introduce the new problem of _Interactive Batch Image
Editing_. Given a user-specified edit $I\rightarrow I^{\prime}$, the goal is
to automatically transfer that edit to new unseen images, so that all edited
images end up with the same _final state_ as $I^{\prime}$ (e.g., all cats with
eyes closed) regardless of their original starting states (e.g., varying
degrees of original eye openness); see Figure 1. Hence, a solution to this
problem requires two components: (i) modeling the user edit in the example
pair $(I,I^{\prime})$ so that it can be transferred to new images; and (ii)
controlling the degree of the edit so that all edited images have the same
final state.

To model the user’s edit, we build upon existing literature on Generative
Adversarial Networks (GANs) [26, 27, 28, 23, 29], and in particular StyleGAN
[30, 31], which shows that their learned latent space emits globally linear
editing directions where the edited attribute (e.g., eye closeness) varies
linearly in magnitude along such a direction for all semantically related
instances (e.g., all closed eyes cat). To try to discover the global direction
corresponding to a user edit, we perform optimization in the latent space of
StyleGAN2 [31] so that (i) the edited image with the discovered direction is
visually similar to the original edited image $I^{\prime}$, and (ii) the
linear correlation between the distance along the direction and the strength
of the visual attribute is highest (e.g., going twice the amount in that
direction closes a cat’s eyes by twice the amount).

To control the degree of the edit so that all edited images end up with the
same final state, we derive a closed-form solution. Taking motivation from
[29], we model the attribute strength (e.g., how much the cat’s eyes are open)
as the distance along the normal vector from a hyperplane. By setting the
example edited image as lying on that hyperplane (i.e., its distance is 0),
the objective is to move all other images along that same direction so that
their distances to the hyperplane also become 0. In this way, regardless of
the start state of any image (e.g., varying degree of eye openness), their
edited end states all become the same (e.g., closed eyes). We show that we can
analytically compute the exact amount of traversal along the direction for any
image with a closed-form solution. Editing images in this way helps improve
the results visually, as the final state of any new image matches that of the
edited example given by the user.

Our method works with various edits given by an editing framework, e.g., it
can be a change given by a dragging operation [17], or it can be a change
given by a text-based edit [23, 14] (e.g., “make eyes bigger”), as long as it
is a direction compatible with a StyleGAN2 [31] model. We present qualitative
and quantitative results in transferring edits for a variety of objects
(cats/dogs/faces/humans/lions/arts/etc.), parts (mouths/ears/legs/etc.), and
corresponding attributes (big eyes/short faces/pale skins/etc.). Importantly,
we show that the final state of the edited images is comparable to the
scenario in which a user performs the edit for each image separately. But
since our method does this automatically (see Fig. 2), we show that it takes
much less time (e.g., only 0.05s per image, compared to DragGAN [17] about 2s
per image), and does not require laborious human annotation (e.g., only need 1
annotated image, compared to 4 required for DragGAN [17]). Finally, we show
practical applications of batch image editing; e.g., changing wheel size in
every photo of a car collection by editing just one.

In sum, our contributions are: (1) We introduce the novel problem of
interactive batch image editing, wherein a user-driven edit is automatically
transferred to other similar test images. (2) We study what _having the same
final state_ means in a geometrical sense, and propose a principled approach
to achieving that in StyleGAN2 [31]’s latent space. (3) We show that our
method works on a wide variety of domains e.g., cats, dogs, humans etc.,
taking significantly less time than editing each image individually while
having similar visual quality and more visual consistency.

Figure 3: Different editing strategies. (a) Setting. (b) Naive Approach: The
editing direction effective for an example may not generalize well to test
images. (c) Optimizing Editing Direction: We optimize for a globally
consistent direction that is effective for both example and test images. (d)
Adjusting Editing Strength: Ensuring consistent final states requires
adjusting the editing strength for each test image.

##  2 Related Work

#### Latent space of GANs.

The capabilities of generative adversarial networks (GANs) have transformed
drastically, from the first GAN [32] designed for simple image datasets like
MNIST [33] to big and powerful models like BigGAN [34], Progressive GAN [35]
and StyleGANs [30, 31] designed for much more complex datasets [36, 37].
Parallel to the efforts to enhance their capabilities, there has also been
work done to better understand and make use of their learned latent space.
Specifically, there is a line of work which tries to find interpretable
_directions_ in that space, so that moving in such a direction majorly changes
only one discernible attribute in the image. Some methods [38, 39] try to find
such directions or particular activations using (self) supervised methods.
Others [26, 28, 27] try getting rid of the need for supervision: discovering
the directions representing the most important factors of variation, whatever
they may be. Going beyond the $\mathcal{W}$ space, the authors in [40] explore
the $\mathcal{W+}$ space, and show that it has even better disentanglement
useful for spatial image editing. All of these hidden capabilities have made
StyleGANs a useful tool for editing purposes. However, prior work mainly
focuses on the effectiveness of the editing direction (e.g., whether the
discovered direction can change yaw pose). The question of whether these
directions can be applied to diverse images to yield consistent results (e.g.,
all faces facing frontal) and, if so, how to achieving such consistency,
remains unanswered.

#### Image editing with generative models.

Researchers have used StyleGAN for _real_ image editing by designing an
encoder to invert a real image into StyleGAN’s latent space [41, 42, 43, 44].
StyleCLIP [23] presents a way to perform image editing through a text-based
interface by making use of the CLIP encoder [45]. Very recently, StyleGAN has
also been used to perform point-based editing [25, 17] so as to move any start
point to reach a target location in the image; thereby elongating, rotating,
shifting the objects. Most work on GANs has been for single object category
datasets, but with the rise of text-to-image Diffusion Models which can
generate complex images [13, 46, 47], editing such images is now possible.
Complex scenes can be edited either using text [48, 49, 14, 24, 15, 21], or
using the same idea of point-based manipulation [16, 19]. There have been some
works which discuss the possibility to transfer the edit of one image to
another. EditGAN [50] uses segmentation mask manipulations to edit an image,
but to successfully transfer them to an unseen image, it needs to do post hoc
manipulation of the editing scale. RewriteGAN [51] involves editing one
generated sample; e.g., adding a patch of a tree onto a church tower.
Following this, the rules of the GAN are manipulated so that all churches have
_some_ tree on the top. Within diffusion models, visual prompting/ image
analogies tries something similar, where users can define a triplet {before,
after, test} to learn the edit and transfer it to a test image [52, 53, 54,
55, 56]. Our task is similar, in that we wish to transfer user-edits from the
training example to new images, but it differs in one crucial aspect: along
with transferring the edit, we wish to automatically learn its strength so
that the edit produces the same final state for a new image.

##  3 Approach

Our focus is on the setting in which an image editing process edits an
attribute of an image so that the attribute’s _value_ reaches a desired state;
e.g., rotating the pose of a face looking sideways so that it faces front
(attribute = face’s pose, value = front).

With this view, we explain our framework for _batch image editing_ , which can
be broken down into two stages: (i) A user edits an image $I_{0}$ (e.g., using
DragGAN [17]) to obtain an edited image $I^{\prime}_{0}$. We describe in Sec.
3.1 how we capture this user edit $I_{0}\rightarrow I^{\prime}_{0}$, so that
the same edit can be applied to new images. (ii) Next, we describe in Sec.
3.2, how for any new image, we apply the modeled edit by automatically
adjusting its _strength_ so that the attribute’s value in this new image
matches that of $I^{\prime}_{0}$ (e.g., any face, regardless of its initial
pose, now faces front after the edit).

###  3.1 Modeling the User Edit

We start with an image $I_{0}$. This image could be a real image or a
generated one. Either way, we get its latent representation in the
$\mathcal{W}$ space of a StyleGAN2 [31] model $G$, so that $I_{0}=G(w_{0})$.
(For real images we can use GAN-inversion techniques [57].) The user, with the
help of an image editing framework (e.g., DragGAN [17] or InstructPix2Pix
[14]), edits $I_{0}$ to manipulate one or more of its attributes. The edit
maps to the $\mathcal{W}$ space as $w_{0}\rightarrow w^{\prime}_{0}$. The
resulting edited image can thus be recovered as follows:
$I^{\prime}_{0}=G(w^{\prime}_{0})$. Fig. 3(a) shows an example of the original
and edited image pair, ($I_{0},I^{\prime}_{0}$), where the user intended to
turn the face forward.

Now, given the user edit $I_{0}\rightarrow I^{\prime}_{0}$, we wish to capture
it in a way that can be applied to new images in a generalizable manner; i.e.,
the application of the edit changes the _same_ property in a new image. This
is where a nice property of GANs, and in particular StyleGANs [30, 31] comes
in useful. It has been shown in prior works [26, 27, 28] that it is possible
to discover directions with such properties (using supervised as well as
unsupervised methods) in the learned $\mathcal{W}$ space. In particular, it is
possible to find directions ($\Delta_{w}$) that are _globally consistent_.
Taking motivation from [29], we define a globally consistent direction
$\Delta_{w}$ as the following: for any arbitrary $w$, moving along
$\Delta_{w}$, $w\rightarrow w+\Delta_{w}$ (i) changes the same attribute, and
(ii) by the same amount.

To make the precise user edit $I_{0}\rightarrow I^{\prime}_{0}$ applicable to
other images, it needs to be captured as a globally consistent direction. The
naive way to represent that edit will be through a simple difference in the
$\mathcal{W}$ space: $\Delta_{w}=w^{\prime}_{0}-w_{0}$. However, empirically,
and as we show in Fig. 3(b), applying this $\Delta_{w}$ to the latent code
$w_{1}$ corresponding to a new image $I_{1}$ _does not_ always result in the
same change; while $I_{0}\rightarrow I^{\prime}_{0}$ results in a pose change
of $\sim$30°degrees in yaw, $I_{1}\rightarrow I^{\prime}_{1}$ does not seem to
change the same attribute or at least not by the same amount.

Hence, our goal is to represent the $I_{0}\rightarrow I^{\prime}_{0}$ edit
through a $\mathcal{W}$ space direction that can better satisfy both the
properties of a globally consistent direction. For this task, we first take
motivation from LARGE [29] to introduce a mathematical view of what those
directions mean.

#### Globally consistent direction.

Let’s say that $\Delta_{g}$ is one such direction. Along with a bias term $b$,
we can define a hyperplane as follows:

${w}\cdot{\Delta_{g}}+b=0$ (1)

That is, any point $w$ which lies on the hyperplane will satisfy this
equation. The authors in [29] argued that for such a hyperplane, whose normal
vector is a globally consistent direction ($\Delta_{g}$), the distance of an
arbitrary point $w$ from that hyperplane ($w\cdot{\Delta_{g}}$) will be
linearly correlated to the actual attribute ($y$) that results in the
generated image:

$y=a\times({w}\cdot{\Delta_{g}})+b$ (2)

Here, $a$ and $b$ are unknown linear coefficients. For example, if
$\Delta_{g}$ corresponds to change in pose, then all the front facing people
will lie at the same distance $d$ from the above hyperplane. Because of this,
any hyperplane defined with respect to a globally consistent direction can be
viewed as a semantic hyperplane. For simplicity, we can set $d=0$ for the
edited image $I^{\prime}_{0}$.

Given that the original $\Delta_{w}$ may not always be globally consistent, we
aim to discover, through optimization, a different direction $\Delta^{*}_{w}$
which produces a similar editing effect, but is more likely to be a globally
consistent one. We initialize $\Delta^{*}_{w}$ with 0’s, and design two
objective functions to optimize it. First, to make sure that it produces a
similar effect as the original direction, we use an image reconstruction loss
so that the edited image produced by the new edit matches the original edit
given by the user.

$\mathcal{L}_{img}=\left\|G({w_{0}}+{\Delta_{w}})-G({w_{0}}+{\Delta^{*}_{w}})\right\|_{2}$
(3)

Additionally, user can provide the real value of distance $d$, otherwise we
set the distance of the edited image from the hypothetical hyperplane to be 0
(similar to [29]). We constrain the learned $\Delta^{*}_{w}$ to follow this
property for better interpretability. That is, the new direction should be
such that when we traverse the original latent code in that direction,
$w_{0}\rightarrow w_{0}+\Delta^{*}_{w}$, the resulting image should lie at the
hyperplane defined by that direction. Setting $b=0$ in Eq. 1, we minimize the
edited point’s distance from the hyperplane:

$\mathcal{L}_{att}=\left|(w_{0}+\Delta^{*}_{w})\cdot\Delta^{*}_{w}\right|$ (4)

The overall loss function for optimizing the new direction $\Delta^{*}_{w}$ is
a weighted sum of the two objectives:
$\mathcal{L}_{\Delta}=\mathcal{L}_{img}+\lambda\mathcal{L}_{att}$. We set
$\lambda$ to 0.02 in all experiments to balance the magnitude of
$\mathcal{L}_{img}$ and $\mathcal{L}_{att}$.

Figure 4: Qualitative comparisons between dragging baselines. For ours, green
bounding box indicates automatic transfer from the red bounding box example in
the first row (i.e., no point annotation needed!).

Now, in Fig. 3 (b) and (c), we compare the difference in image editing results
using two different directions when applied to the same new image ($I_{1}$):
$w_{1}\rightarrow w_{1}+\Delta_{w}$ vs. $w_{1}\rightarrow
w_{1}+\Delta^{*}_{w}$. As discussed before, in this case, the editing using
the naively computed direction ($\Delta_{w}$) is not able to accurately
capture the pose rotation the way it did for the user edited example
($I_{0}\rightarrow I^{\prime}_{0}$). On the other hand, with $\Delta^{*}_{w}$,
we see that the pose of the woman changes by a similar amount as the edited
example. However, since the original pose of the woman ($I_{1}$) was not the
same as the pose of the man ($I_{0}$), the new edited image ($I^{\prime}_{1}$)
still does not arrive at the same final state as $I^{\prime}_{0}$. Therefore,
our next goal is to figure out how to scale the learned direction so that
$I^{\prime}_{1}$ does arrive at the same final state as $I^{\prime}_{0}$.

###  3.2 Adjusting Editing Strength for New Images

Given the optimized $\Delta^{*}_{w}$ and some new image $I_{i}=G(w_{i})$, we
wish to edit it in the following way:

$w^{\prime}_{i}=w_{i}+\alpha_{i}n$ (5)

where $\alpha_{i}$ is the editing strength computed separately for each image
and $n=\Delta^{*}_{w}/||\Delta^{*}_{w}||$ is a unit vector in the direction of
$\Delta^{*}_{w}$. Fig. 3 (d) illustrates a geometric perspective that we will
use to compute $\alpha_{i}$. First, we see $\Delta^{*}_{w}$ represented as a
vector and its corresponding hyperplane that is normal to it. Next, we depict
the new latent point $w_{i}$. The goal is to move it along the
$\Delta^{*}_{w}$ direction so that it arrives _onto_ the hyperplane. Through
this depiction, $\alpha_{i}$ can be understood as the distance of $w_{1}$ from
the hyperplane. We can get the distance by projecting ($w^{\prime}_{0}-w_{i}$)
in the direction of $n$. Therefore, $\alpha_{i}=(w^{\prime}_{0}-w_{i})\cdot
n$.

Since for each image, the only unique computation that needs to be done is the
calculation of $\alpha_{i}$, we can see why our method will be much faster
than, for example, annotating every image and running the DragGAN [17]
optimization each of those times. We will show in our experiments the
significant difference in time taken by our method compared to single image
editing baselines.

Importantly, there is an added benefit to computing $\alpha$’s in this way.
Let’s say the goal of the editing process is to rotate $n$ faces and make them
frontal. After completion, all of them do become frontal, each with their own
editing strengths $\\{\alpha_{1},\alpha_{2},...,\alpha_{n}\\}$ computed using
the above formula. Suppose the user now wants the same faces facing a bit left
instead. To do this, the user _does not_ need to re-annotate the original
training example and run the optimization one more time. In an interactive
manner, they can simply scale the $\alpha$ for the training example to match
the desired edit, and all other $\alpha$’s can be automatically recomputed.

Figure 5: Qualitative comparisons to text-guided baselines. Ours transfers the
edit from example (red), to other test images (green).

##  4 Experiments

We study how well our method models and transfers the edits from an example,
and how efficient it is compared to single image editing baselines using their
official code.

#### Categories.

We evaluate on a variety of domains: Human faces (FFHQ) [30], AFHQ Cats, Dogs
[58], MetFaces [59], Human bodies [60]. For each domain, we use the
corresponding pretrained StyleGAN model to perform editing.

Figure 6: Additional qualitative results on various domains.

###  4.1 Qualitative results

We start with qualitative comparisons with (i) interactive point-dragging, and
(ii) text-based image editing baselines.

#### Interactive point based editing.

We compare against DragGAN [17], UserControllableLT [25], and DragDiffusion
[16]. For each baseline, we, as a user, desire to edit a bunch of images to
have the same final state for an attribute. Fig. 4 shows the results of
editing two kinds of edits to four images (leftmost column). In the left case,
the goal is to make everyone smile by the same degree. In the right, it is to
vertically compress everyone’s face by the same degree. For all three
baselines, we manually annotate points for each test image. For our method, we
only annotate points and perform DragGAN-based edit on one image (top row),
then automatically transfer the edit to the three other test images.

For the smiling case (left), we notice that when DragGAN drags the upper/lower
lips up/down respectively, it can either make people smile (1st, 2nd and 4th
rows), or it can make people look shocked (blonde girl; 3rd row). Both edits
are technically correct based on lips movement, but it won’t match the user
expectation of making everyone smile by the same degree. Since all four images
are edited independently, this is not unexpected. The other two baselines
sometimes have issues introducing the same edit across different images; e.g.,
UserControllableLT can make the blonde girl smile a little bit, but not the
other images. In general, we notice in our experiments that to introduce the
same exact edit that we want, we often need to play around with some
hyperparameters (e.g., # of iterations). In comparison, our method produces
edited images that are smiling more equally (left) and have been compressed by
a similar amount (right), and with much less human effort.

#### Text-driven image editing.

We consider two baselines: StyleCLIP [23] and InstructPix2Pix [14]. Each takes
an image and text as input to produce an edited image: $I\rightarrow
I^{\prime}$. Fig. 5 shows the results of Ours vs. StyleCLIP (left) and Ours
vs. InstructPix2Pix (right). (For the latter, we invert the output of
InstructPix2Pix into StyleGAN’s latent space using [43] for our method.) In
each case, for the baselines, we use the same text prompt, e.g., “A photo of a
person with pale skin” for each edited image (in four rows). For ours, we
capture the edit from the first editing result of a baseline (first row), and
then automatically transfer the edit to the remaining three images.

Our goal with this particular setup is to test (i) how consistently the
baselines introduce the edit denoted by text to different images (e.g., do all
people become equally pale?), and (ii) how consistently our method captures
and transfers the _particular_ edit of the first example to the rest of the
images, irrespective of how good that first edit was. For (i), we find that
StyleCLIP sometimes has issues; e.g., the paleness of the edited face in first
example is different from the second (first vs second rows). For
InstructPix2Pix, sometimes it can give consistent results; the childness of
different faces seems similar. But, it can have consistency issues in other
examples; e.g., the _type/strength_ of maleness introduced is different in
each image. And this is where, we believe, the utility of our method lies: if
one desires to edit every face to be _male_ in a particular way (i.e., to the
same degree) as the first example, e.g., thick eyebrows & light beard, our
method has an advantage.

Finally, we show our method’s results for non-facial domains in Fig. 6. For
the lion examples (top left), we see that it can preserve not just the type of
edit, e.g., dragging to make the lion roar, but also the strength of the roar.
The strength being preserved can be observed more easily for the human body
poses (bottom left), where we see that the extent of legs split, hand
movements, is consistent enough to almost align the edited test images with
the user edited one. Overall, results on these diverse set of domains
highlight an observation that many kinds of edits can be thought of as a
combination of {type, strength}, both of which can indeed be captured and
transferred according to our needs.

| Point Dragging | Time | Anno.
---|---|---|---
Method | Dist. | FID | 1 img | 1k imgs | # imgs
UserControl. [25] | 13.64 | 25.32 | 0.03s | 30s | 1000
DragGAN [17] | 4.165 | 9.28 | 2s | 33.33m | 1000
DragDiffusion [16] | 26.56 | 36.55 | 60s | 16.67h | 1000
Ours | 9.467 | 9.35 | 2s | 82s | 1

Table 1:  Time is estimated for 1 point drag, without human annotation time.
(82s includes 2s to perform edit on the example, 30s to optimize the editing
direction, and 50s to transfer the edit to 1000 test images.) Our method
requires only one image annotation in total, while the baselines need one
annotation per test image.

###  4.2 Quantitative results

Next, perform quantitative experiments following the setup proposed in [17,
19, 16]. We first randomly sample 10 facial test images and pair them with a
randomly sampled target face image. We use dlib-ml [61] to detect keypoints in
the test and target images. Our goal is to see if the keypoints in the test
images can be moved to the target locations specified by keypoints in the
target image. For the baselines [25, 17, 16], we perform dragging for each
test image. For our method, we perform dragging on _one_ test image, and then
transfer the edit to the remaining nine images. This is repeated 100 times;
i.e., each time a random target image is paired with 10 random test images.

We report the Euclidean distance between the keypoints of edited and target
images in Table 1. Even without requiring annotations for every test image,
our method can move the points very close to the target; almost as close as
DragGAN, which requires annotation each time. We also compute FID [62] between
the original test images and their edited versions to ensure that our method
does not distort the image; and our quality is comparable to DragGAN.

##  5 Deeper Analysis

We perform deeper analysis on the usefulness of our batch image editing
framework by focusing on a specific edit: face pose rotation. In particular,
if we wish to rotate many faces to front, how do we visualize the improvements
brought by different components of our method?

Fig. 7 depicts the setup. We use DragGAN to perform dragging on one image
(top; $w_{0}\rightarrow w^{\prime}_{0}$). We then transfer the edit to 1000
other test images (two shown in rows 2-3). We see that naively applying
$\Delta_{w}=w^{\prime}_{0}-w_{0}$ from DragGAN to test images cannot make them
frontal (0°). Using our optimized $\Delta^{*}_{w}$ (without dynamic scaling)
does help it bring closer to facing front. Does this mean that
$\Delta^{*}_{w}$ might be a more _global_ direction? We study this in Fig. 8
(a), where we visualize the degree to which the correlation property of global
directions (Eq. 2) holds for $\Delta_{w}$ and $\Delta^{*}_{w}$ individually.
We see that there is indeed a better correlation ($R^{2}$) between the
distances of latents from hyperplanes and the Yaw degree of resulting images.
(We predict Yaw using 6DRepNet [63].) However, as we discussed before,
$\Delta^{*}_{w}$ in itself is insufficient: to completely bring the facial
pose to front, we need to scale it with corresponding $\alpha$’s to bring them
much closer to front (Fig. 7, 4th column). To study this in a more systematic
way, we visualize the effect on all 1000 test images. Similar to Fig. 8 (a),
we visualize distance-to-hyperplane vs. Yaw degree for the original test
images (blue) and edited images (red) in Fig. 8 (b). We can see that the
variation in Yaw for the edited images _collapses_ at around 0°; i.e., they
mostly face front as we would like.

We also compute the time it takes to perform editing in Table 1, last three
columns. Our method only requires annotating one image, while the baselines
require annotating each image (e.g., 1000 total in this case). Note the times
do not include human annotation time.

Figure 7: Effect of adjusted and optimized editing for test images. Yaw degree
of each image is provided on top of each images.

|  | Time complexity
---|---|---
Method | MAE | Prepare | Inference
Random | 11.295 $\pm$ 8.972 | - | -
DragGAN [17] \+ GANgealing [64] | 8.141 $\pm$ 7.221 | 30s | 2000s
Ours | 2.120 $\pm$ 1.818 | 32s | 50s

Table 2:  Ablation Study: DragGAN + GANgealing.  Figure 8: (a) After
optimization, the editing direction is more linearly correlated with the yaw
attribute. (b) With automatically adjusted editing scale for each test image.
Figure 9: Ablation study set up. We use GANgealing [64] to transfer the
annotated points from example to test image.

#### Other ways to automate batch image editing.

One baseline for batch image editing (i.e., so that all edited images achieve
the same final state) is shown in Fig. 9. After the user annotates the
source/target points in the example image, we use GANgealing [64] to transfer
the points to corresponding locations in each new test image. The edited
images are then obtained using DragGAN (DragGAN + GANgealing). We compare our
method to this baseline when the user edits one face image to become front
(Yaw=0°) and transferring it to 1000 other test images.

Results are shown in Table 2. We report mean absolute error (MAE) between the
Yaw degree of the edited image and its ideal frontal image (Yaw=0°). ‘Random’
shows the variations in Yaw for the original images (before editing). While
‘DragGAN + GANgealing’ does help in reducing the variation (8.14 $<$ 11.29),
our edited images are much more frontal, with an MAE of 2.12. This is because
the baseline has no way to bring all edited images to the same final state. On
top of that, since our method does not rely on additional information about
keypoints (which ‘DragGAN + GANgealing’ does), we believe that it is better
suited for batch image editing for many kinds of domains (e.g., Fig. 6), where
we might not have such information.

Figure 10: Interactive Batch Image Editing. As users adjust the editing
strength in the example image, all test images will be automatically updated.
(Red bounding boxes indicate the $\Delta_{w}$ according to the drag points).

##  6 Conclusion

We introduced the problem of interactive batch image editing. Given a user
edit in an example image, our approach automatically transfers that edit to
other test images, maintaining a consistent final state of the edit across
images. Extensive experiments demonstrated that our method produces comparable
quality to state-of-the-art single-image-editing methods while saving
significant time and human effort. We are currently limited to StyleGAN
models. Extending this problem and solution to diffusion-based models for more
edits types would be an exciting future direction.

## References

  * Ho and Zhou [2021] Man M. Ho and Jinjia Zhou.  Deep preset: Blending and retouching photos with color style transfer.  In _WACV_ , 2021.
  * Afifi et al. [2021] Mahmoud Afifi, Konstantinos G. Derpanis, Bjorn Ommer, and Michael S. Brown.  Learning multi-scale photo exposure correction.  In _CVPR_ , 2021.
  * Baek et al. [2023] Jong-Hyeon Baek, DaeHyun Kim, Su-Min Choi, Hyo-jun Lee, Hanul Kim, and Yeong Jun Koh.  Luminance-aware color transform for multiple exposure correction.  In _ICCV_ , 2023.
  * Cong et al. [2020] Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang.  Dovenet: Deep image harmonization via domain verification.  In _CVPR_ , 2020.
  * Tsai et al. [2017] Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-Hsuan Yang.  Deep image harmonization, 2017.
  * Guo et al. [2021] Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, and Bing Zheng.  Intrinsic image harmonization.  In _CVPR_ , 2021.
  * Ling et al. [2021a] Jun Ling, Han Xue, Li Song, Rong Xie, and Xiao Gu.  Region-aware adaptive instance normalization for image harmonization.  In _CVPR_ , 2021a.
  * Jiang et al. [2021] Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, and Zhangyang Wang.  Ssh: A self-supervised framework for image harmonization.  In _ICCV_ , 2021.
  * Cong et al. [2022] Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, and Liqing Zhang.  High-resolution image harmonization via collaborative dual transformations.  2022\.
  * Valanarasu et al. [2022] Jeya Maria Jose Valanarasu, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Jose Echevarria, Yinglan Ma, Zijun Wei, Kalyan Sunkavalli, and Vishal M Patel.  Interactive portrait harmonization.  2022\.
  * Zhang et al. [2020] Lingzhi Zhang, Tarmily Wen, and Jianbo Shi.  Deep image blending.  In _The IEEE Winter Conference on Applications of Computer Vision_ , pages 231–240, 2020.
  * Epstein et al. [2023] Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, and Aleksander Holynski.  Diffusion self-guidance for controllable image generation.  _arXiv preprint arXiv:2306.00986_ , 2023.
  * Rombach et al. [2021a] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  2021a.
  * Brooks et al. [2022] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  2022\.
  * Meng et al. [2022] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  SDEdit: Guided image synthesis and editing with stochastic differential equations.  In _ICLR_ , 2022.
  * Shi et al. [2023] Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and Song Bai.  Dragdiffusion: Harnessing diffusion models for interactive point-based image editing.  2023\.
  * Pan et al. [2023] Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt.  Drag your gan: Interactive point-based manipulation on the generative image manifold.  In _SIGGRAPH_ , 2023.
  * Couairon et al. [2022] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance, 2022.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang.  Dragondiffusion: Enabling drag-style manipulation on diffusion models.  2023\.
  * Yang et al. [2022] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen.  Paint by example: Exemplar-based image editing with diffusion models.  _arXiv preprint arXiv:2211.13227_ , 2022.
  * Con [2023] _Adding Conditional Control to Text-to-Image Diffusion Models_ , 2023.
  * gli [2023] _GLIGEN: Open-Set Grounded Text-to-Image Generation_ , 2023. arXiv:2301.07093.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _ICCV_ , 2021.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _CVPR_ , 2023.
  * Endo [2022] Yuki Endo.  User-controllable latent transformer for stylegan image layout editing.  _Computer Graphics Forum_ , 2022.
  * Härkönen et al. [2020] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  In _NeurIPS_ , 2020.
  * Shen et al. [2020] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In _CVPR_ , 2020.
  * Shen and Zhou [2021] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in gans.  In _CVPR_ , 2021.
  * Nitzan et al. [2022] Yotam Nitzan, Rinon Gal, Ofir Brenner, and Daniel Cohen-Or.  Large: Latent-based regression through gan semantics.  In _CVPR_ , 2022.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  2019\.
  * Karras et al. [2020a] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In _CVPR_ , 2020a.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _NeurIPS_ , 2014.
  * LeCun [1998] Yann LeCun.  The mnist database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ , 1998.
  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv_ , 2018.
  * Karras et al. [2018] Terro Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  In _ICLR_ , 2018.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Yu et al. [2015a] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  2015a.
  * Jahanian et al. [2020] Ali Jahanian, Lucy Chai, and Phillip Isola.  On the ”steerability” of generative adversarial networks.  In _ICLR_ , 2020.
  * Bau et al. [2020a] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba.  Understanding the role of individual units in a deep neural network.  _Proceedings of the National Academy of Sciences_ , 2020a.
  * Wu et al. [2020] Zongze Wu, Dani Lischinski, and Eli Shechtman.  Stylespace analysis: Disentangled controls for stylegan image generation.  In _ICLR_ , 2020.
  * Tov et al. [2021a] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  In _ACM Transactions on Graphics_ , 2021a.
  * Wang et al. [2022] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen.  High-fidelity gan inversion for image attribute editing.  In _CVPR_ , 2022.
  * Tov et al. [2021b] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  _arXiv preprint arXiv:2102.02766_ , 2021b.
  * Roich et al. [2021] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  _ACM Trans. Graph._ , 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  2021\.
  * Rombach et al. [2021b] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  2021b.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2020.
  * Mokady et al. [2022] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  _arXiv preprint arXiv:2211.09794_ , 2022.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ling et al. [2021b] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler.  Editgan: High-precision semantic image editing.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021b.
  * Bau et al. [2020b] David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba.  Rewriting a deep generative model.  In _ECCV_ , 2020b.
  * Bar et al. [2022] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A. Efros.  Visual prompting via image inpainting.  2022\.
  * Nguyen et al. [2023] Thao Nguyen, Yuheng Li, Utkarsh Ojha, and Yong Jae Lee.  Visual instruction inversion: Image editing via visual prompting.  In _NeurIPS_ , 2023.
  * Sun et al. [2023] Yasheng Sun, Yifan Yang, Houwen Peng, Yifei Shen, Yuqing Yang, Han Hu, Lili Qiu, and Hideki Koike.  Imagebrush: Learning visual in-context instructions for exemplar-based image manipulation.  2023\.
  * Hertzmann et al. [2023] Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin.  Image analogies.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , 2023.  URL https://doi.org/10.1145/3596711.3596770.
  * Liao et al. [2017] Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang.  Visual attribute transfer through deep image analogy.  2017\.
  * Xia et al. [2022] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang.  Gan inversion: A survey.  _TPAMI_ , 2022.
  * Choi et al. [2020] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Starganv2: Diverse image synthesis for multiple domains.  In _CVPR_ , 2020.
  * Karras et al. [2020b] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data, 2020b.
  * Fu et al. [2022] Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Chen Qian, Chen-Change Loy, Wayne Wu, and Ziwei Liu.  Stylegan-human: A data-centric odyssey of human generation.  _arXiv preprint_ , arXiv:2204.11823, 2022.
  * King [2009] Davis E. King.  Dlib-ml: A machine learning toolkit.  _J. Mach. Learn. Res._ , 2009.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Gunter Klambauer, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _NeurIPS_ , 2017.
  * Hempel et al. [2022] Thorsten Hempel, Ahmed A. Abdelrahman, and Ayoub Al-Hamadi.  6d rotation representation for unconstrained head pose estimation.  In _ICIP_ , 2022.
  * Peebles et al. [2022] William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei Efros, and Eli Shechtman.  Gan-supervised dense visual alignment.  In _CVPR_ , 2022.
  * Yu et al. [2015b] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_ , 2015b.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.

##  7 Supplementary

Figure 11: Interactive Batch Image Editing. As the user adjusts the editing
strength $\alpha$ in the example image (first row), all test images will be
automatically updated, ensuring consistency in the final state (yaw degree).
(Red bounding box indicates the $\Delta_{w}$ yielded by dragging points).

We present results from Interactive Batch Image Editing in Section 7.1, where
adjustments in the example are automatically updated to test images.
Subsequently, we demonstrate the effectiveness of our method when multiple
edits are applied to an example before reaching the final state in Section
7.2. Limitations are discussed in Section 7.3, followed by additional
qualitative results in Section 7.4 and implementation details in Section 7.5.

###  7.1 Interactive Batch Image Editing

We demonstrate the advantages of computing adaptive $\alpha$ values for
individual example images. Consider a scenario where the editing objective is
to rotate and bring into frontal view a set of $n$ faces. User will first
annotate dragging points for example image to rotate this face to frontal
(Fig. 11, 1st row, $\alpha=1$). Upon completion, each test images will also
become frontal, each with unique editing strengths represented by
${\alpha_{1},\alpha_{2},...,\alpha_{n}}$, computed using
$\alpha_{i}=(w^{\prime}_{0}-w_{i})\cdot n$ with
$n=\Delta^{*}_{w}/||\Delta^{*}_{w}||$ (Fig. 11, row 2-5, 5th column).

Now, if the user wishes to have the same faces facing slightly to the left,
there is no need to re-annotate the original training examples and re-run the
optimization process. In an interactive fashion, user can simply adjust the
scaling of the $\alpha$ for the specific training example to achieve the
desired edit (e.g., $\alpha=-4$), and all other $\alpha$ values will be
automatically recalculated. Qualitative results are shown in Fig. 11. As can
be seen, all poses (yaw degrees) in test images are changing accordingly to
the example image (yaw degree is consistent). It is worth noting that it takes
roughly 0.05s to compute a new image (about 0.03s to recompute $\alpha$ for
each test image and 0.02s to generate a new image). Thus, it is fast enough
for real-time batch image editing.

Additional qualitative results for interactive batch image editing are
provided in Fig. 10 (dog pose rotation), Fig. 14 (mountain
enlargement/removal), Fig. 15 (face slimming/enlargement), Fig. 16 (anime hair
shortening/lengthening), Fig. 17 (tiger roaring) and Fig. 18 (model leg pose).

Figure 12: Multiple edits can be applied to example image before being
transferred to test images.

###  7.2 Multiple Edits

Multiple edits can be applied to the example to reach the final state, which
is then transferred to the test images without intermediate steps. For
example, a user may choose to initially rotate the face to the left (1st edit)
and subsequently enhance the person’s smile (2nd edit). The resulting final
state incorporates both edits. Our method can directly transfer the final
state to test images, including both edits (face rotation and smiling) (Fig.
12, columns 1-3).

Two additional examples of multiple edits are presented in Fig. 12. Columns
4-6 showcase uplifting the dog’s ear (1st edit) and then opening the mouth
(2nd edit). Columns 7-9 illustrate shortening the dress (1st edit) and
adjusting the pose (2nd edit). Despite variations in the initial states of the
test images (e.g., variations in mouth openness), our methods ensure
consistency in the final states across test images (e.g., all dogs have an
open mouth and uplifted ears).

Figure 13: Limitations. (a) Failure Case: Our method may encounter challenges
in capturing fine details (e.g., curling trunk of an elephant). (b) Example-
Test Similarity: For optimal results, the example and test images should
belong to the same semantic domain (e.g., both featuring long hair) to ensure
correctly transferred edits. (c-d) Interesting Cases: Edits can be mistakenly
interpreted, resulting in unexpected outcomes such as winking in the wrong eye
(c) or unintentionally flipping the horse (d).

###  7.3 Limitations

While our method has demonstrated effectiveness in various applications, there
are some limitations/failure cases that we notice. (1) Failure to capture
small details: Our method may encounter challenges in accurately transferring
small details (e.g., curling elephant trunk, Fig. 13a). (2) Semantic
dissimilarity in example/test images: When test images deviate significantly
in semantic content from the example (e.g., vastly differing in hair lengths),
the transferred results may exhibit suboptimal performance (Fig. 13b). (3)
Potential editing errors: In certain scenarios, the editing might be
misinterpreted and lead to unexpected outcomes. For example, attempts to make
a person wink at the left eye may occasionally lead to the wink being
transferred to the right eye (Fig. 13c). Another example is pose changes, when
adjusting the pose of horse legs, there are instances where the outcome
unexpectedly mirrors the horse in the same position as the example image (Fig.
13d).

Figure 14: Interactive Batch Image Editing. As the user adjusts the editing
strength $\alpha$ in the example image (first row), all test images will be
automatically updated, ensuring consistency in the final state (mountain
height). (Red bounding box indicates the $\Delta_{w}$ yielded by dragging
points). Figure 15: Interactive Batch Image Editing. As the user adjusts the
editing strength $\alpha$ in the example image (first row), all test images
will be automatically updated, ensuring consistency in the final state (facial
shape). (Red bounding box indicates the $\Delta_{w}$ yielded by dragging
points).

###  7.4 Additional Qualitative Results

Along with the domains presented in the main paper (Human faces (FFHQ) [30],
Lions, Dogs [58], MetFaces [59], Human bodies [60]), we provide additional
qualitative results for Cats [58], and Horses [65] domains in Fig. 20, 19
respectively.

Figure 16: Interactive Batch Image Editing. As the user adjusts the editing
strength $\alpha$ in the example image (first row), all test images will be
automatically updated, ensuring consistency in the final state (hair length).
(Red bounding box indicates the $\Delta_{w}$ yielded by dragging points).
Figure 17: Interactive Batch Image Editing. As the user adjusts the editing
strength $\alpha$ in the example image (first row), all test images will be
automatically updated, ensuring consistency in the final state (roar degree).
(Red bounding box indicates the $\Delta_{w}$ yielded by dragging points).
Figure 18: Interactive Batch Image Editing. As the user adjusts the editing
strength $\alpha$ in the example image (first row), all test images will be
automatically updated, ensuring consistency in the final state (leg position).
(Red bounding box indicates the $\Delta_{w}$ yielded by dragging points).
Figure 19: Additional qualitative results on Horses. Figure 20: Additional
qualitative results on Cats.

###  7.5 Implementation Details

We use the AdamW optimizer [66] to optimize $\Delta^{*}_{w}$ for 1000
iterations, with a learning rate of $0.001$ and a batch size of 16. All
experiments are performed on a single NVIDIA RTX 3090 machine.


# A Dynamic Population Model of Strategic
Interaction and Migration under Epidemic Risk

Ezzat Elokda, Saverio Bolognani and Ashish R. Hota E. Elokda and S. Bolognani
are with ETH Zurich, 8092 Zurich, Switzerland<EMAIL_ADDRESS>A.
R. Hota is with the Department of Electrical Engineering, Indian Institute of
Technology Kharagpur, India<EMAIL_ADDRESS>research leading to
these results was partly supported by the Swiss National Science Foundation
(SNSF) via the NCCR Automation.

###### Abstract

In this paper, we show how a dynamic population game can model the strategic
interaction and migration decisions made by a large population of agents in
response to epidemic prevalence. Specifically, we consider a modified
susceptible-asymptomatic-infected-recovered (SAIR) epidemic model over
multiple zones. Agents choose whether to activate (i.e., interact with
others), how many other agents to interact with, and which zone to move to in
a time-scale which is comparable with the epidemic evolution. We define and
analyze the notion of equilibrium in this game, and investigate the transient
behavior of the epidemic spread in a range of numerical case studies,
providing insights on the effects of the agents’ degree of future awareness,
strategic migration decisions, as well as different levels of lockdown and
other interventions. One of our key findings is that the strategic behavior of
agents plays an important role in the progression of the epidemic and can be
exploited in order to design suitable epidemic control measures.

##  I Introduction

Infectious diseases or epidemics spread through society by exploiting social
interactions. As the disease becomes more prevalent, individuals reduce their
social interaction and even migrate to safer locations in a strategic and non-
myopic manner [1, 2], which plays a significant role in epidemic evolution.
Accordingly, past work has explored decentralized or game-theoretic protection
strategies against epidemics on (static) networks [3, 4, 5, 6, 7]. More
recently, evolution of network topology and epidemic states in a comparable
time-scale have been studied in the framework of activity-driven networks [8,
9]. Game-theoretic decision-making in this framework was recently studied in
[10] where myopic bounded rational agents decide whether to activate or not as
a function of the epidemic prevalence.

To the best of our knowledge, there have been few rigorous game-theoretic
formulations that model agents that

  * •

decide their degree of activation, and consequently influence the resulting
network topology,

  * •

decide whether to migrate to different locations, and

  * •

maximize both current and long-run future pay-off

in the same time-scale as epidemic evolution. In this work, we present a
framework to address the above research gap.

Motivated by the presence of asymptomatic carriers in COVID-19 [11, 12], we
build upon the SAIR epidemic model studied in [13, 14]. We consider a large
population regime where the state of an individual agent is characterized by
its infection state and its location (or zone). At discrete time instants,
each agent decides its degree of activation and its next location. The agent
is then paired randomly with other agents, and its infection state evolves
following an augmented SAIR epidemic model which also takes into account
_unknowingly recovered_ agents as described in Section II. Agents maximize a
discounted infinite horizon expected reward which is a function of the
aggregate infection state, the zonal distribution of the agents, and the
policy followed by the population.

In a departure from the conventional assumption of a static population
distribution in the classical population game setting [15], epidemic evolution
leads to a dynamically evolving population which makes the analysis
challenging. We utilize the recently developed framework of _dynamic
population games_ [16], in which the authors show a reduction of the dynamic
setting to a static population game setting [15]. This simplifies the analysis
in comparison to the existing approaches such as anonymous sequential games
[17, 18] and mean field games [19, 20, 21], and is particularly useful for
epidemic models. As a consequence of the reduction, standard evolutionary
models [15] can be adapted for the coupled dynamics of the agents’ states and
strategic decision making, which evolve on the same time-scale.

The paper is structured as follows: the dynamic population model is presented
in Section II, and its stationary equilibria are analyzed in Section III. The
evolutionary update of agents’ policies is modeled in Section IV. Numerical
experiments reported in Section V provide compelling insights into agents’
behavior, effects of lockdown measures and strategic mobility patterns. For
instance, we observe that if recovered agents are exempt from lockdown
measures, then an increased level of activity by susceptible and asymptomatic
agents can happen without having much impact on the peak and total infections.
Their strategic behavior does not lead to a higher infection level and the
social welfare improves due to overall higher activity levels.

##  II Model

We consider a homogeneous population of non-atomic agents or individuals. The
state and the dynamics of this population are described by the following
elements.

###  II-A States

We augment the SAIR epidemic model to distinguish between recovered agents who
are aware of being recovered and those who are recovered, but unaware of ever
being infected. Specifically, each agent can be in one of the following
infection states: _Susceptible_ (${\mathtt{S}}$), _infected Asymptomatic_
(${\mathtt{A}}$), _Infected symptomatic_ (${\mathtt{I}}$), _Recovered_
(${\mathtt{R}}$), _Unknowingly recovered_ (${\mathtt{U}}$) (agents that have
recovered without showing symptoms). Agents in states ${\mathtt{R}}$ and
${\mathtt{U}}$ are immune from further infection. Moreover, each agent resides
in one of $Z$ _zones_ or locations. Formally, we define the state of each
agent as $(s,z)\in{\mathcal{S}}\times{\mathcal{Z}}$, where
${\mathcal{S}}=\set{{\mathtt{S}},{\mathtt{A}},{\mathtt{I}},{\mathtt{R}},{\mathtt{U}}}$
and ${\mathcal{Z}}=\set{1,\dots,Z}$. The state distribution is
$d\in{\mathcal{D}}=\Delta({\mathcal{S}}\times{\mathcal{Z}})$, where
$\Delta(X)$ is the space of probability distributions supported in $X$. We
write $d[s,z]$ to denote the proportion of agents with infection state $s$
residing in zone $z$.

###  II-B Actions and policies

We consider a dynamic environment that evolves in discrete-time (e.g., each
time interval representing a day). At each time step, each agent strategically
chooses:

  * •

its activation degree $a\in{\mathcal{A}}=\set{0,1,\dots,{a_{\textup{max}}}}$,
which denotes the number of other agents it chooses to interact with ($a=0$
signifies no activation), and

  * •

the zone ${\tilde{z}}\in{\mathcal{Z}}$ where to move for the next day.

The combined action is denoted
$(a,{\tilde{z}})\in{\mathcal{A}}\times{\mathcal{Z}}$. A (Markovian) policy is
denoted by
$\pi:{\mathcal{S}}\times{\mathcal{Z}}\rightarrow\Delta(A\times{\mathcal{Z}})$,
and it maps an agent’s state $(s,z)\in{\mathcal{S}}\times{\mathcal{Z}}$ to a
randomization over the actions
$(a,{\tilde{z}})\in{\mathcal{A}}\times{\mathcal{Z}}$. The set of all possible
policies is denoted by $\Pi$. Explicitly, $\pi[a,{\tilde{z}}\mid s,z]$ is the
probability that an agent plays $(a,{\tilde{z}})$ when in state $(s,z)$. All
agents are homogeneous and follow the same policy $\pi$. Further, agents that
have never shown symptoms act in the same way, i.e.
$\pi[\cdot\mid{\mathtt{S}},z]=\pi[\cdot\mid{\mathtt{A}},z]=\pi[\cdot\mid{\mathtt{U}},z]$.
Note that $\pi$ is _time-varying_ ; agents change their strategies as the
epidemic unfolds. The dynamics of $\pi$ are detailed in Section IV.

###  II-C State transitions

We now derive a dynamic model of the evolution of state distribution $d$ when
the agents adopt a policy $\pi$. We denote the policy-state pair $(\pi,d)$ as
the social state. The state of each agent changes at every time step according
to transition probabilities encoded by the _stochastic matrix_

$P[s^{+},z^{+}\mid s,z](\pi,d)\\\ =\sum_{a,{\tilde{z}}}\pi[a,{\tilde{z}}\mid
s,z]\>p[s^{+},z^{+}\mid s,z,a,{\tilde{z}}](\pi,d),$ (1)

where $p[s^{+},z^{+}\mid s,z,a,{\tilde{z}}](\pi,d)$ denotes the probability
distribution over the next state when an agent in infection state $s$ and zone
$z$ chooses action $(a,\tilde{z}$) in social state $(\pi,d)$.

Note that the Markov chain $P[s^{+},z^{+}\mid s,z](\pi,d)$ is not time-
homogeneous as the social state $(\pi,d)$ is time-varying. The state
transition function is defined as

$p[s^{+},z^{+}\mid s,z,a,{\tilde{z}}](\pi,d)\\\ :={\mathbb{P}}[s^{+}\mid
s,z,a](\pi,d)\>{\mathbb{P}}[z^{+}\mid z,{\tilde{z}}](\pi,d).$ (2)

The zone transition probabilities are assumed to be independent of $(\pi,d)$
and given by

${\mathbb{P}}[z^{+}\mid z,{\tilde{z}}](\pi,d)=\begin{cases}1&\text{if
}z^{+}={\tilde{z}},\\\ 0&\text{otherwise}.\end{cases}$

In order to derive the infection state transition probabilities
${\mathbb{P}}[s^{+}\mid s,z,a](\pi,d)$ (which are schematically represented in
Figure 1), we combine the transition rules of the (augmented) SAIR model with
the specific activation actions as follows.

  * •

At a given time, an agent in state $s$ in zone $z$ chooses its activation
degree $a$ according to policy $\pi$. Then, it is paired randomly with up to
$a$ other individuals in zone $z$ with the probability of being connected with
another agent being proportional to the activation degree of the target agent
(analogous to the configuration model [22]).

  * •

The agent could also fail to pair with one or more of the $a$ other
individuals. This occurs with increasing probability as the _total amount of
activity_ in the zone (to be defined hereafter) is low111This represents, for
example, when the public space (streets, buildings) are largely empty because
most agents are staying at home..

  * •

Once the network is formed, a susceptible agent becomes asymptomatically
infected with probability $\beta_{\mathtt{A}}\in[0,1]$ per each asymptomatic
neighbor and with probability $\beta_{\mathtt{I}}\in[0,1]$ per each infected
neighbor.

  * •

An asymptomatic agent becomes (symptomatically) infected222This represents
both the aggravation of the illness and the outcome of testing in the
asymptomatic population. with probability
$\delta_{\mathtt{A}}^{\mathtt{I}}\in(0,1]$, and recovers without being aware
of it with probability $\delta_{\mathtt{A}}^{\mathtt{U}}\in[0,1]$.

  * •

An infected agent recovers with probability
$\delta_{\mathtt{I}}^{\mathtt{R}}\in(0,1]$.

  * •

An individual in state ${\mathtt{U}}$ becomes aware of its recovery with
probability $\delta_{\mathtt{U}}^{\mathtt{R}}\in[0,1]$ (for example via
serological tests on the population).

  * •

The network thus formed gets discarded at the next time step and the process
repeats.

Note that with the exception of the transition from state ${\mathtt{S}}$ to
${\mathtt{A}}$, all other state transition probabilities are defined via
exogenous parameters and do not depend on the social state. In order to
compute the transition probability from ${\mathtt{S}}$ to ${\mathtt{A}}$, we
define the _total amount or mass of activity_ in zone $z$ as

$e_{z}(\pi,d)=\sum_{s}d[s,z]\>\sum_{a,{\tilde{z}}}a\>\pi[a,{\tilde{z}}\mid
s,z],$

which is determined by the mass of active agents and their degrees of
activation. Similarly, the mass of activity by _asymptomatic_ and
_symptomatic_ agents in zone $z$ are

$\displaystyle e^{\mathtt{A}}_{z}(\pi,d)$
$\displaystyle=d[{\mathtt{A}},z]\>\sum_{a,{\tilde{z}}}a\>\pi[a,{\tilde{z}}\mid{\mathtt{A}},z],$
$\displaystyle e^{\mathtt{I}}_{z}(\pi,d)$
$\displaystyle=d[{\mathtt{I}},z]\>\sum_{a,{\tilde{z}}}a\>\pi[a,{\tilde{z}}\mid{\mathtt{I}},z].$

In order to consider the event of failing to pair with an agent when the
amount of activity in the zone $e_{z}(\pi,d)$ is low, we introduce a small
constant amount $\epsilon>0$ of fictitious activation that does not belong to
any of the agents. Consequently in zone $z$, the probability of not
interacting with any agent, the probability of a randomly chosen agent being
asymptomatic and the probability of a randomly chosen agent being symptomatic
are, respectively,

$\displaystyle\gamma^{\emptyset}_{z}(\pi,d)$
$\displaystyle=\frac{\epsilon}{e_{z}(\pi,d)+\epsilon},$ (3)
$\displaystyle\gamma^{\mathtt{A}}_{z}(\pi,d)$
$\displaystyle=\frac{e^{\mathtt{A}}_{z}(\pi,d)}{e_{z}(\pi,d)+\epsilon},\quad\gamma^{\mathtt{I}}_{z}(\pi,d)=\frac{e^{\mathtt{I}}_{z}(\pi,d)}{e_{z}(\pi,d)+\epsilon}.$
(4)

Note that for a given $\epsilon$, the probability of encountering an infected
agent (symptomatically or not) goes to zero as the amount of infections goes
to zero, as desired.

Figure 1: Infection state transition diagram. Self loops are not shown.

As a result, the probability of a susceptible agent to _not_ get infected upon
activation with degree $a$ is

${\mathbb{P}}[s^{+}={\mathtt{S}}\mid s={\mathtt{S}},z,a](\pi,d)\\\
=\left(1-\beta_{\mathtt{A}}\>\gamma^{\mathtt{A}}_{z}(\pi,d)-\beta_{\mathtt{I}}\>\gamma^{\mathtt{I}}_{z}(\pi,d)\right)^{a}.$

It is easy to see that when a susceptible agent does not interact with any
other agent (i.e., $a=0$), it remains susceptible. When it participates in
exactly one interaction ($a=1$) in $z$, the probability that its neighbor is
asymptomatic (respectively, infected) is $\gamma^{\mathtt{A}}_{z}(\pi,d)$
(respectively, $\gamma^{\mathtt{I}}_{z}(\pi,d)$). When it draws $a>0$
independent agents to interact with, it must not get infected in any of the
interactions to remain susceptible, and this occurs with the specified
probability. As a consequence, we have

$\displaystyle{\mathbb{P}}[s^{+}={\mathtt{A}}\mid s={\mathtt{S}},z,a](\pi,d)=$
$\displaystyle\quad\qquad 1-{\mathbb{P}}[s^{+}={\mathtt{S}}\mid
s={\mathtt{S}},z,a](\pi,d).$

The remaining transition probabilities follow directly as:

$\displaystyle{\mathbb{P}}[s^{+}={\mathtt{I}}\mid
s={\mathtt{A}}]=\delta_{\mathtt{A}}^{\mathtt{I}},\quad{\mathbb{P}}[s^{+}={\mathtt{U}}\mid
s={\mathtt{A}}]=\delta_{\mathtt{A}}^{\mathtt{U}},$
$\displaystyle{\mathbb{P}}[s^{+}={\mathtt{A}}\mid
s={\mathtt{A}}]=1-\delta_{\mathtt{A}}^{\mathtt{I}}-\delta_{\mathtt{A}}^{\mathtt{U}},\quad{\mathbb{P}}[s^{+}={\mathtt{R}}\mid
s={\mathtt{R}}])=1$ $\displaystyle{\mathbb{P}}[s^{+}={\mathtt{R}}\mid
s={\mathtt{I}}]=\delta_{\mathtt{I}}^{\mathtt{R}},\quad{\mathbb{P}}[s^{+}={\mathtt{I}}\mid
s={\mathtt{I}}]=1-\delta_{\mathtt{I}}^{\mathtt{R}},$
$\displaystyle{\mathbb{P}}[s^{+}={\mathtt{R}}\mid
s={\mathtt{U}}]=\delta_{\mathtt{U}}^{\mathtt{R}},\quad{\mathbb{P}}[s^{+}={\mathtt{U}}\mid
s={\mathtt{U}}]=1-\delta_{\mathtt{U}}^{\mathtt{R}}$

where we have suppressed $(z,a,\pi,d)$ for better readability. These
expressions completely specify ${\mathbb{P}}[s^{+}\mid s,z,a](\pi,d)$ (Figure
1) and therefore the state transition function (2).

###  II-D Rewards

Each agent’s own state $(s,z)$ and action $(a,\tilde{z})$ yields an immediate
reward for the agent, composed of a reward ${r_{\textup{act}}}[s,z,a]$ for
their activation decision, a reward ${r_{\textup{mig}}}[s,z,{\tilde{z}}]$ for
their migration decision, and a reward ${r_{\textup{dis}}}[s]$ for how the
agent’s health is affected by the disease. Formally,

$r[s,z,a,{\tilde{z}}]:={r_{\textup{act}}}[s,z,a]+{r_{\textup{mig}}}[s,z,{\tilde{z}}]+{r_{\textup{dis}}}[s].$
(5)

The activation reward is defined as

${r_{\textup{act}}}[s,z,a]:=o[a]-c[s,z,a],$

where $o[a]\in\mathbb{R}_{+}$ denotes the social benefit of interacting with
$a$ other agents and is assumed to be non-decreasing in $a$ with $o[0]=0$, and
$c[s,z,a]\in\mathbb{R}_{+}$ denotes the cost imposed by the authority in zone
$z$ to discourage activity. We assume that $c[s,z,a]$ are non-decreasing in
$a$ and satisfy

$\\!\\!\\!c[{\mathtt{I}},z,a]\geq
c[{\mathtt{S}},z,a]=c[{\mathtt{A}},z,a]=c[{\mathtt{U}},z,a]\geq
c[{\mathtt{R}},z,a]$

element-wise, since lockdown measures can be more stringent against
individuals showing symptoms and more benign for individuals who are known to
be immune.

The migration reward encodes the non-negative cost of migrating to a new zone.
In this work, we define

${r_{\textup{mig}}}[s,z,{\tilde{z}}]:=\begin{cases}0&\text{if
}{\tilde{z}}=z,\\\ -{c_{\textup{mig}}}&\text{if }{\tilde{z}}\neq
z.\end{cases}$

However, one may consider a richer cost function that incorporates specific
travel restrictions between zones, etc. The third term in (5) encodes the cost
of being ill:

${r_{\textup{dis}}}[s]:=\begin{cases}-{c_{\textup{dis}}}&\text{if
}s={\mathtt{I}},\\\ 0&\text{otherwise}.\end{cases}$

###  II-E Agents’ Strategic Decisions

We now introduce the strategic decision-making process of the agents. The
immediate expected reward of an agent in state $(s,z)$ when it follows policy
$\pi$ is

$R[s,z](\pi)=\sum_{a,{\tilde{z}}}\pi[a,{\tilde{z}}\mid
s,z]\>r[s,z,a,{\tilde{z}}],$

with $r[s,z,a,{\tilde{z}}]$ as defined in (5). The expected discounted
infinite horizon reward of an agent in state $(s,z)$ with discount factor
$\alpha\in[0,1)$ following the homogeneous policy $\pi$ is recursively defined
as

$V[s,z](\pi,d)=R[s,z](\pi)\\\ +\alpha\sum_{s^{+},z^{+}}P[s^{+},z^{+}\mid
s,z](\pi,d)\>V[s^{+},z^{+}](\pi,d),$

or, equivalently in vector form,

$V(\pi,d)=(I-\alpha\>P(\pi,d))^{-1}\>R(\pi).$ (6)

Equation (6) is the well-known Bellman equation. Note that it is continuous in
the social state $(\pi,d)$, as $I-\alpha\>P(\pi,d)$ is guaranteed to be
invertible for $\alpha\in[0,1)$.

While an agent can compute the expected discounted reward $V(\pi,d)$ at a
given social state $(\pi,d)$, the policy $\pi$ may not be optimal for the
agent. Thus, we assume that each agent chooses its current action
$(a,{\tilde{z}})$ in order to maximize

$Q[s,z,a,{\tilde{z}}](\pi,d):=r[s,z,a,{\tilde{z}}]\\\
+\alpha\sum_{s^{+},z^{+}}p[s^{+},z^{+}\mid
s,z,a,{\tilde{z}}](\pi,d)\>V[s^{+},z^{+}](\pi,d),$ (7)

i.e., the agent is aware of the immediate reward and the effect of its action
on their future state; however, it assesses the future reward based on a
stationarity assumption on the social state $(\pi,d)$. In other words, the
agent chooses its action to maximize a single-stage deviation from the
homogeneous policy $\pi$ [23, Section 2.7], and assumes that its own actions
are not going to affect the social state significantly.

##  III Equilibrium Analysis

We start by introducing the notion of best response based on the single-stage
deviation reward defined in (7).

######  Definition 1 (Best Response).

The best response of an agent in state $(s,z)$ at the social state $(\pi,d)$
is the set valued correspondence
$B_{s,z}:\Pi\times{\mathcal{D}}\rightrightarrows\Delta({\mathcal{A}}\times{\mathcal{Z}})$
given by

$B_{s,z}(\pi,d)\in\Big{\\{}\sigma\in\Delta({\mathcal{A}}\times{\mathcal{Z}}):\forall\sigma^{\prime}\in\Delta({\mathcal{A}}\times{\mathcal{Z}})\\\
\sum_{a,{\tilde{z}}}\left(\sigma[a,{\tilde{z}}]-\sigma^{\prime}[a,{\tilde{z}}]\right)\>Q[s,z,a,{\tilde{z}}](\pi,d)\geq
0\Big{\\}}.$ (8)

The above notion of best response is from the perspective of an individual
agent in state $(s,z)$ when all other agents are following the homogeneous
policy $\pi$ and their states are distributed as per $d$. The agent will
choose any distribution $\sigma$ over the actions which maximizes its expected
single-stage deviation reward $Q$ at the current state $(s,z)$.

Consequently, a social state $({\boldsymbol{\pi}},{\boldsymbol{d}})$ is
stationary when agents in all states are playing their best response when they
follow the policy ${\boldsymbol{\pi}}$, and the state distribution
${\boldsymbol{d}}$ is stationary under this policy. Thus, we have the
following definition of a stationary equilibrium.

######  Definition 2 (Stationary equilibrium).

A stationary equilibrium is a social state
$({\boldsymbol{\pi}},{\boldsymbol{d}})\in\Pi\times{\mathcal{D}}$ which
satisfies

$\displaystyle{\boldsymbol{\pi}}[\cdot\mid s,z]$ $\displaystyle\in
B_{s,z}({\boldsymbol{\pi}},{\boldsymbol{d}}),\;\forall(s,z)\in{\mathcal{S}}\times{\mathcal{Z}},$
(SE.1) $\displaystyle{\boldsymbol{d}}$
$\displaystyle={\boldsymbol{d}}\>P({\boldsymbol{\pi}},{\boldsymbol{d}}).$
(SE.2)

Thus, at the equilibrium, the state Markov chain
$P({\boldsymbol{\pi}},{\boldsymbol{d}})$ (1) is time-homogeneous, and the
agents behave optimally in the corresponding Markov decision process [23].
Note that we denote stationary equilibria with boldface notation. When
$({\boldsymbol{\pi}},{\boldsymbol{d}})$ is a stationary equilibrium,
${\boldsymbol{\pi}}$ corresponds to the Nash equilibrium under state
distribution ${\boldsymbol{d}}$.

######  Theorem 1 (Theorem 1 in [16]).

A stationary equilibrium $({\boldsymbol{\pi}},{\boldsymbol{d}})$ for the
proposed dynamic population game is guaranteed to exist.

We refer to [16] for the details of the proof, which relies on a fixed-point
argument. There, a general dynamic population game is considered in which the
state and action spaces are finite, and the state transition and reward
functions are continuous in the social state $(\pi,d)$. By definition, our
model is an instance of such a dynamic population game.

The following proposition shows that the stationary distribution
${\boldsymbol{d}}$ does not have any asymptomatic or infected agents (i.e.,
eventually, the epidemic dies out). The final stationary distribution is
however not unique.

######  Proposition 1.

Let ${r}_{\textup{act}}^{*}[s,z]:=\max_{a}{r_{\textup{act}}}[s,z,a]$ be the
maximum activation reward in state $s$ and zone $z$, and let
${\mathcal{A}}_{s,z}^{*}$ be the set of activation levels that achieve such
reward. Let $\bar{r}_{\textup{act}}[s]:=\max_{z}{r}_{\textup{act}}^{*}[s,z]$
be the maximum activation reward in state $s$ across all zones, and let
$\delta_{\mathtt{U}}^{\mathtt{R}},\alpha,{c_{\textup{mig}}}>0$. Let us define
the subset of zones

$\displaystyle\bar{\mathcal{Z}}_{s}$
$\displaystyle=\operatorname*{argmax}_{z}{r}_{\textup{act}}^{*}[s,z],$
$\displaystyle\mathcal{Z}^{0}_{s}$
$\displaystyle=\left\\{z\mid\bar{r}_{\textup{act}}[s]-{r}_{\textup{act}}^{*}[s,z]>\frac{1-\alpha}{\alpha}{c_{\textup{mig}}}\right\\}.$

Then, any social state $({\boldsymbol{\pi}},{\boldsymbol{d}})$ satisfying

$\begin{cases}{\boldsymbol{d}}[{\mathtt{A}},z]={\boldsymbol{d}}[{\mathtt{I}},z]={\boldsymbol{d}}[{\mathtt{U}},z]=0&\forall
z,\\\ {\boldsymbol{d}}[s,z\in\mathcal{Z}^{0}_{s}]=0&\forall s,\\\
{\boldsymbol{\pi}}[a\notin\mathcal{A}^{*}_{s,z},{\tilde{z}}\mid
s,z]=0&\forall{\tilde{z}},s,z,\\\
{\boldsymbol{\pi}}[a,{\tilde{z}}\notin\bar{\mathcal{Z}}_{s}\mid
s,z\in\mathcal{Z}^{0}_{s}]=0&\forall a,s,\\\
{\boldsymbol{\pi}}[a,{\tilde{z}}\neq z\mid
s,z\notin\mathcal{Z}^{0}_{s}]=0&\forall a,s.\end{cases}$

is a stationary equilibrium.

The proof is presented in the extended version [24]. Proposition 1 shows how
stationary equilibria can be computed without solving a fixed-point problem,
and directly identifies some dominant strategies for the agents. The
identification of dominant strategies is insightful for the design of
interventions (e.g., lockdown measure for the different compartments). For
example, it is possible to verify that

$\mathcal{A}_{{\mathtt{R}},z}^{*}=\operatorname*{argmax}_{a}\
{r_{\textup{act}}}[{\mathtt{R}},z,a]=\operatorname*{argmax}_{a}\
\left(o[a]-c[{\mathtt{R}},z,a]\right)$

corresponds to the dominant activation strategies for agents in ${\mathtt{R}}$
(_knowingly_ immune agents). Notice that the activation caused by immune
agents appears in the denominator of the probability that a generic agent
interacts with an infectious agent – see (4) – and therefore looser lockdown
measures for recovered agents can be used to reduce the spreading as shown in
simulations in Section V.

##  IV Social State Dynamics

Different stationary equilibria correspond to drastically different outcomes
in terms of the impact of the epidemic on the population. For this reason, we
investigate the _transient behavior_ leading to the equilibrium, which is
determined by the state dynamics from Section II-C, i.e., $d^{+}=d\>P(\pi,d)$,
and the way in which agents update their policies. For this second part, we
get inspiration from the _evolutionary dynamic_ models in classical population
games [15], and more precisely from the _perturbed best response dynamics_.

We assume that the agents are not perfectly rational, with the bounded
rationality factor $\lambda\in[0,\infty)$. When they are making a decision on
which action to play, they follow the _logit choice_ function [15, Section
6.2], given by

${\tilde{\pi}}[a,{\tilde{z}}\mid
s,z](\pi,d)=\frac{\exp{(\lambda\>Q[s,z,a,{\tilde{z}}](\pi,d))}}{\sum_{a^{\prime},{\tilde{z}}^{\prime}}\exp{(\lambda\>Q[s,z,a^{\prime},{\tilde{z}}^{\prime}](\pi,d))}}.$

For $\lambda=0$, it results in a uniform distribution over all the actions. At
the limit $\lambda\rightarrow\infty$, we recover the perfect best response. At
finite value of $\lambda$, it assigns higher probabilities to actions with
higher payoffs.

In order to model the fact that agents update their policies gradually, we
consider the discrete-time update

$\pi^{+}[\cdot\mid s,z]=(1-\eta)\>\pi[\cdot\mid
s,z]+\eta\>{\tilde{\pi}}[\cdot\mid s,z],$

where $\eta\in(0,1]$ is a parameter that controls the rate of policy change:
for $\eta<1$, agents have _inertia_ in their decision making, while for
$\eta=1$ agents promptly update their action decision to the perturbed best
response ${\tilde{\pi}}$. Note that this update model leads to a perturbed
version of the equilibrium policy ${\boldsymbol{\pi}}$ at the rest points,
rather than the exact policy [15].

##  V Numerical Case Studies

We present a select number of case studies to showcase

  * •

the effect of agents’ strategic activation decisions on the spread of the
epidemic,

  * •

the impact of lockdown measures on both epidemic containment and social
welfare, and

  * •

the effect of strategic migration decisions on how the epidemic spreads across
multiple locations.

For this purpose, we consider an infectious epidemic characterized by
$\beta_{\mathtt{A}}=\beta_{\mathtt{I}}=0.2$,
$\delta_{\mathtt{A}}^{\mathtt{I}}=\delta_{\mathtt{A}}^{\mathtt{U}}=0.08$, and
$\delta_{\mathtt{I}}^{\mathtt{R}}=0.04$. The agents can activate up to degree
${a_{\textup{max}}}=6$, and the activation reward is linear in the activation
degree, with a unit reward for maximum activation $o[{a_{\textup{max}}}]=1$.
The illness is quite severe, with a discomfort cost ${c_{\textup{dis}}}=10$.
Initially, we let the agents choose an activation degree uniformly at random
and do not plan any move. The agents are highly rational ($\lambda=10$) and
unless otherwise stated, we consider that they update their decisions with an
inertia $\eta=0.2$. We consider both a single zone and a two zone setting, and
denote the zones by ${\mathtt{Z1}}$ and ${\mathtt{Z2}}$. In all cases, the
epidemic starts in ${\mathtt{Z1}}$ with a proportion of $2\%$ of that zone’s
population asymptomatic (${\mathtt{A}}$), and $1\%$ infected (${\mathtt{I}}$).

We further consider that authorities can enforce lockdown regulations through
the parameter ${a_{\textup{lock}}}[s,z]$, which represents the _maximum
allowed activation degree_ and can differ between zones and for agents in
different infection states. Lockdown is implemented by setting $c[s,z,a]=0$ if
$a\leq{a_{\textup{lock}}}[s,z]$, and $c[s,z,a]=3\>o[a]$ otherwise. Regardless
of the lockdown measures, we always assume that the discomfort of the illness
is sufficient to prevent symptomatically infected agents from activating. As a
consequence, the main threat of the epidemic is due to the presence of
asymptomatically infected agents in the population.

###  V-A Strategic activation under lockdown measures

We first investigate the single zone scenario, with a focus on how agents of
different cognitive ability react under various lockdown measures, and the
resulting effects on the epidemic spread. Figure 2 shows an example with
lockdown degree ${a_{\textup{lock}}}=2$ and three cases. In cases (2a) and
(2b), the lockdown is enforced on the whole population, and the cases differ
in the agents’ cognitive level of the future. In (2a), the agents are
completely myopic. Notice how they simply adhere to the lockdown degree333A
slight deviation to the expected degree of activation is due to the agents’
bounded rationality., which aligns with standard epidemic models. Farsighted
agents (2b), on the other hand, actively adjust their activation decisions in
response to the epidemic threat, and volunteer to limit their activity beyond
the lockdown requirement at peak infection times. The reduction in activity
levels leads to a less severe epidemic spread, with a smaller total and peak
infection.

(a)

(b)

(c)

(a)

(b)

(c)

Figure 2: The effect of the agents’ farsightedness and of the exclusion of
immune agents from lockdown measures on the epidemic. The dashed black lines
indicate the maximum allowed activation degree.

(a)

(b)

(c)

(d)

Figure 3: The effect of the severity of the lockdown measures on three key
epidemic indicators, for different scenarios.

Case (2c) also considers farsighted agents, but this time _recovered_ agents
are exempt from lockdown and thus activate at the maximum degree, as per their
dominant strategy. This leads to a significant reduction of the total amount
of infections. In fact, due the prevalence of activity of immune agents, it
becomes less likely for a susceptible agent to encounter an infected agent.
Consequently, susceptible, asymptomatic and unknowingly recovered agents too
increase their level of activation. Nevertheless, the peak infection remains
being largely unchanged.

This insight is further explored in Figure 3, which shows the effect of
different lockdown measures on three main performance indices: the _total
infections_ (${\mathtt{R}}+{\mathtt{U}}$ at the end of the epidemic), the
_peak infections_ (highest value of ${\mathtt{I}}$444We do not consider
infected but asymptomatic agents since they do not contribute to the load on
medical facilities.), and the _average welfare_ (mean reward (5) in the
population over the duration of the epidemic).

We depict the same cases considered in Figure 2 and perform a parameter sweep
over the strictness of the lockdown ${a_{\textup{lock}}}$. Additionally, we
showcase the effect of performing _serological tests_ to increase the amount
of knowingly immune agents, at rate $\delta_{\mathtt{U}}^{\mathtt{R}}=0.05$.
We observe that myopic agents perform poorly along all the performance
metrics. For farsighted agents, the exemption of recovered agents and
serological tests lead to significant improvements in the average welfare
because the knowingly immune agents are able to achieve their maximum
activation degree without increasing the threat of the epidemic.

###  V-B Strategic migration

We showcase the effect of strategic migration in a setting with two zones,
with zone ${\mathtt{Z1}}$ initially holding $90\%$ of the total population,
and zone ${\mathtt{Z2}}$ initially infection free. In both zones, (knowingly)
recovered agents are exempt from lockdown, and the lockdown restrictions for
the other agents are different in the zones. Namely, ${\mathtt{Z1}}$ has a
looser lockdown, with a maximum allowed activation degree of 4, whereas
${\mathtt{Z2}}$ only allows a maximum of 2. The migration cost is
${c_{\textup{mig}}}=2$, and the agents are farsighted with $\alpha=0.9$. Note
that with these parameters, susceptible agents will want to move to
${\mathtt{Z1}}$ when the epidemic is not prevalent, as per Proposition
1555Here, ${r_{\textup{act}}^{*}}[{\mathtt{S}},{\mathtt{Z2}}]=\frac{1}{3}$ and
${\bar{r}_{\textup{act}}}[{\mathtt{S}}]=\frac{2}{3}$, therefore
${\mathtt{Z2}}\in{\mathcal{Z}}^{0}_{\mathtt{S}}$.. The inertia in the policy
update is $\eta=0.1$.

Additionally, both zones perform serological testing at rate
$\delta_{\mathtt{U}}^{\mathtt{R}}=0.01$. The resulting epidemic spread is
displayed in Figure 4. First, notice how the proportion of unknowingly
recovered agents decays, in contrast to Figure 2, in which no serological
testing is performed.

We now focus on the strategic migration behavior of agents who are either
susceptible or think they are (${\mathtt{S}}$, ${\mathtt{A}}$,
${\mathtt{U}}$). Note that symptomatically infected and recovered agents never
move since they are immune to the threat of the epidemic, and the activation
costs are the same for them in both zones. Initially, the epidemic risk is
still small in ${\mathtt{Z1}}$, and the occupants of ${\mathtt{Z2}}$ start
moving there to benefit from the more lenient lockdown measure. This trend
soon reverses, however, with the rise of infections in ${\mathtt{Z1}}$: the
strategic agents elect to move to the zone with stricter lockdown to escape
the epidemic risk. Since a proportion of the movers are asymptomatically
infected, this leads to an outbreak of the epidemic in ${\mathtt{Z2}}$ as
well, with lower, but significant, peak infections than ${\mathtt{Z1}}$.
Eventually, once the infections in ${\mathtt{Z1}}$ has decreased sufficiently
(at approximately day 50), some ${\mathtt{Z2}}$ residents move to
${\mathtt{Z1}}$ again, initiating a _second wave of infections_ in
${\mathtt{Z1}}$. At the end of the epidemic, all the remaining agents move
from ${\mathtt{Z2}}$ as per their dominant strategy.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 4: Effect of strategic migration/activation decisions on the epidemic
spreading between two zones with different lockdown measures. The dashed black
lines indicate the maximum allowed activation degree. The top left and bottom
center plots are clipped for display purposes; at the end of the epidemic all
non-recovered agents in Zone 2 move to Zone 1. The total infections in Zones 1
and 2 are, respectively, 62.1% and 24.5%. The peak infections in Zones 1 and 2
are, respectively, 11.1% and 9.5%.

##  VI Conclusions and Future Works

In this paper, we propose a model of strategic behavior at both individual and
societal level based on first principles, demonstrate its potential to explain
complex activation and migration patterns, and to guide the design of more
effective epidemic control measures. We characterize the stationary equilibria
in the proposed dynamic population game setting and illustrate how a better
understanding of the emergent behavior can be leveraged to design effective
mitigation strategies. For instance, we show that withdrawing restrictions on
recovered agents leads to higher levels of activity by susceptible agents
without increasing the peak and total infection levels. This observation
provides a rigorous justification for conducting large-scale serological
testing and letting individuals with immunity to interact freely for
significantly improving the welfare of the society. A natural extension of
this model includes considering the behavior of vaccinated individuals and
designing optimal intervention strategies that incorporates the strategic
response of agents.

### Proof of Proposition 1

The first property follows the fact that states ${\mathtt{A}}$,
${\mathtt{I}}$, and ${\mathtt{U}}$ are _transient_ in the infection state
Markov chain (see Figure 1), and evolve independently of the agents’
migrations. We omit the formal steps, which follow standard Markov chain
theory arguments.

We now observe that, given a distribution that is only supported on the two
compartments ${\mathtt{S}}$ and ${\mathtt{R}}$, no transition between
compartments of the extended SAIR model is possible. The reward for agents in
both these compartments then becomes independent from the actions of others,
and the set $\mathcal{A}^{*}_{s,z}$ defines their _dominant activation
strategies_.

We now consider migration strategies of the agents at the equilibrium
$({\boldsymbol{\pi}},{\boldsymbol{d}})$. We first consider an agent in
infection state $s\in\\{{\mathtt{S}},{\mathtt{R}}\\}$ residing in zone
$z\in\bar{\mathcal{Z}}_{s}$. In this zone, the activation reward is maximum
among all other zones for the agent in state $s$. Since the infection state
remains unchanged, and migration is costly (${c_{\textup{mig}}}>0$), migrating
to a different zone does not lead to a beneficial single-stage deviation for
the agent.

We now consider an agent in zone
$z\in{\mathcal{Z}}^{n}_{s}:={\mathcal{Z}}\setminus\\{\bar{\mathcal{Z}}_{s}\cup\mathcal{Z}^{0}_{s}\\}$.
According to policy ${\boldsymbol{\pi}}$, the agent does not migrate to a
different zone. Consequently, the value function for the agent is

$V[s,z]({\boldsymbol{\pi}},{\boldsymbol{d}})=\frac{{r}_{\text{act}}^{*}[s,z]}{1-\alpha}.$

Now, consider a single-stage deviation where the agent moves to location
$z^{\prime}\in\bar{\mathcal{Z}}_{s}$ and chooses activation degree
$a^{\prime}\in\mathcal{A}^{*}_{s,z}$. Consequently, using the definition of
${\mathcal{Z}}^{n}_{s}$,

$Q[s,z,a^{\prime},z^{\prime}]=r^{*}_{\text{act}}[s,z]-{c_{\textup{mig}}}+\alpha\frac{\bar{r}_{\text{act}}[s]}{1-\alpha}<V[s,z]({\boldsymbol{\pi}},{\boldsymbol{d}}).$

In other words, an agent in a zone in ${\mathcal{Z}}^{n}_{s}$ does not find it
beneficial to migrate anywhere else in a single-stage deviation.

It remains to show that an agent in zone $z\in\mathcal{Z}^{0}_{s}$ finds it
beneficial to move to a zone in $\bar{\mathcal{Z}_{s}}$ and consequently, we
must have $d[s,z]=0$ for $z\in\mathcal{Z}^{0}_{s}$. Under policy
${\boldsymbol{\pi}}$, we have

${\boldsymbol{\pi}}[a\notin\mathcal{A}^{*}_{s,z},z^{\prime}\notin\bar{\mathcal{Z}}_{s}|s,z]=0.$

Consequently, the value function for the agent is

$\displaystyle V[s,z]({\boldsymbol{\pi}},{\boldsymbol{d}})$
$\displaystyle=r^{*}_{\text{act}}[s,z]-{c_{\textup{mig}}}+\alpha\frac{\bar{r}_{\text{act}}[s]}{1-\alpha}>\frac{r^{*}_{\text{act}}[s,z]}{1-\alpha};$

in other words, the policy ${\boldsymbol{\pi}}$ yields a higher value compared
to any policy that does not include migration. Similarly, one can show that
migrating to a zone in ${\mathcal{Z}}^{n}_{s}$ instead does not yield a
beneficial single-stage deviation since
$r^{*}_{\text{act}}[s,z^{\prime}]<\bar{r}_{\text{act}}[s]$ for any zone
$z^{\prime}\in{\mathcal{Z}}^{n}_{s}$. ∎

## References

  * [1] M. Patino, A. Kessler, and S. Holder. (2021) More Americans are leaving cities, but don’t call it an urban exodus. Accessed: 2021-08-26. [Online]. Available: https://www.bloomberg.com/graphics/2021-citylab-how-americans-moved/
  * [2] W. H. Frey. (2021) Pandemic population change across metro America: Accelerated migration, less immigration, fewer births and more deaths. Accessed: 2021-08-26. [Online]. Available: https://brook.gs/3hFPAFf
  * [3] S. Trajanovski, Y. Hayel, E. Altman, H. Wang, and P. Van Mieghem, “Decentralized protection strategies against SIS epidemics in networks,” _IEEE Transactions on Control of Network Systems_ , vol. 2, no. 4, pp. 406–419, 2015.
  * [4] A. R. Hota and S. Sundaram, “Game-theoretic vaccination against networked SIS epidemics and impacts of human decision-making,” _IEEE Transactions on Control of Network Systems_ , vol. 6, no. 4, pp. 1461–1472, 2019\.
  * [5] K. Paarporn, C. Eksin, J. S. Weitz, and J. S. Shamma, “Networked SIS epidemics with awareness,” _IEEE Transactions on Computational Social Systems_ , vol. 4, no. 3, pp. 93–103, 2017.
  * [6] C. Eksin, J. S. Shamma, and J. S. Weitz, “Disease dynamics on a network game: A little empathy goes a long way,” _Scientific Reports_ , vol. 7, p. 44122, 2017.
  * [7] Y. Huang and Q. Zhu, “A differential game approach to decentralized virus-resistant weight adaptation policy over complex networks,” _IEEE Transactions on Control of Network Systems_ , vol. 7, no. 2, pp. 944–955, 2020\.
  * [8] L. Zino, A. Rizzo, and M. Porfiri, “An analytical framework for the study of epidemic models on activity driven networks,” _Journal of Complex Networks_ , vol. 5, no. 6, pp. 924–952, 2017.
  * [9] ——, “Analysis and control of epidemics in temporal networks with self-excitement and behavioral changes,” _European Journal of Control_ , vol. 54, pp. 1 – 11, 2020.
  * [10] A. R. Hota, T. Sneh, and K. Gupta, “Impacts of game-theoretic activation on epidemic spread over dynamical networks,” _arXiv:2011.00445_ , 2020.
  * [11] Z. Hu, C. Song, C. Xu, G. Jin, Y. Chen, X. Xu, H. Ma, W. Chen, Y. Lin, Y. Zheng _et al._ , “Clinical characteristics of 24 asymptomatic infections with COVID-19 screened among close contacts in Nanjing, China,” _Science China Life Sciences_ , vol. 63, no. 5, pp. 706–711, 2020.
  * [12] W. Pang, “Public health policy: COVID-19 epidemic and SEIR model with asymptomatic viral carriers,” _arXiv:2004.06311_ , 2020.
  * [13] S. Ansumali, S. Kaushal, A. Kumar, M. K. Prakash, and M. Vidyasagar, “Modelling a pandemic with asymptomatic patients, impact of lockdown and herd immunity, with applications to sars-cov-2,” _Annual Reviews in Control_ , 2020.
  * [14] R. H. Chisholm, P. T. Campbell, Y. Wu, S. Y. Tong, J. McVernon, and N. Geard, “Implications of asymptomatic carriers for infectious disease transmission and control,” _Royal Society open science_ , vol. 5, no. 2, p. 172341, 2018\.
  * [15] W. H. Sandholm, _Population games and evolutionary dynamics_. MIT Press, 2010.
  * [16] E. Elokda, A. Censi, and S. Bolognani, “Dynamic population games,” _arXiv preprint arXiv:2104.14662_ , 2021. [Online]. Available: https://arxiv.org/abs/2104.14662
  * [17] B. Jovanovic and R. W. Rosenthal, “Anonymous sequential games,” _Journal of Mathematical Economics_ , vol. 17, no. 1, pp. 77–87, 1988.
  * [18] S. Adlakha, R. Johari, and G. Y. Weintraub, “Equilibria of dynamic games with many players: Existence, approximation, and market structure,” _Journal of Economic Theory_ , vol. 156, pp. 269–316, 2015.
  * [19] M. Huang, R. P. Malhamé, P. E. Caines _et al._ , “Large population stochastic dynamic games: closed-loop mckean-vlasov systems and the nash certainty equivalence principle,” _Communications in Information & Systems_, vol. 6, no. 3, pp. 221–252, 2006.
  * [20] D. A. Gomes, J. Mohr, and R. R. Souza, “Discrete time, finite state space mean field games,” _Journal de mathématiques pures et appliquées_ , vol. 93, no. 3, pp. 308–328, 2010.
  * [21] B. A. Neumann, “Stationary equilibria of mean field games with finite state and action space,” _Dynamic Games and Applications_ , 2020.
  * [22] M. Newman, _Networks: An Introduction_. Oxford University Press, 2010.
  * [23] J. Filar and K. Vrieze, _Competitive Markov decision processes_. Springer Science & Business Media, 2012.
  * [24] E. Elokda, S. Bolognani, and A. R. Hota, “A dynamic population model of strategic interaction and migration under epidemic risk,” _PLACEHOLDER_ , 2020.