# GeoFill: Reference-Based Image Inpainting with Better Geometric
Understanding

Yunhan Zhao111footnotemark: 1, Connelly Barnes2, Yuqian Zhou2,3, Eli
Shechtman2, Sohrab Amirghodsi2, Charless Fowlkes1
1UC Irvine 2Adobe Research 3IFP, UIUC
{yunhaz5<EMAIL_ADDRESS>{cobarnes, elishe<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Reference-guided image inpainting restores image pixels by leveraging the
content from another single reference image. The primary challenge is how to
precisely place the pixels from the reference image into the hole region.
Therefore, understanding the 3D geometry that relates pixels between two views
is a crucial step towards building a better model. Given the complexity of
handling various types of reference images, we focus on the scenario where the
images are captured by freely moving the same camera around. Compared to the
previous work, we propose a principled approach that does not make heuristic
assumptions about the planarity of the scene. We leverage a monocular depth
estimate and predict relative pose between cameras, then align the reference
image to the target by a differentiable 3D reprojection and a joint
optimization of relative pose and depth map scale and offset. Our approach
achieves state-of-the-art performance on both RealEstate10K and
MannequinChallenge dataset with large baselines, complex geometry and extreme
camera motions. We experimentally verify our approach is also better at
handling large holes.

**footnotetext: Work done while an intern at Adobe.

##  1 Introduction

Image inpainting aims to plausibly restore missing pixels within a given hole
region. Existing single image inpainting models [65, 59, 66] solve this
problem without additional information by leveraging the knowledge learned
from large scale training data or existing patches within the image. These
methods become less reliable when input images contains large holes and the
filling regions are complex in structures and textures.

In 2021, Zhou et al. [80] proposed a novel inpainting task called reference-
based image inpainting. It aims at filling the hole regions of a “target”
image using another single source photo taken of the same scene. This
foreground removal application is particularly useful when people take photos
at museums or famous landmarks where the background is unique. It is nearly
impossible for existing single image inpainting to faithfully restore what was
_actually there in the background_ for such cases. Inpainting with a reference
image is appealing and indispensable, and also feasible, since other photos
taken at different viewpoints of the same scene can be available, e.g. in the
users’ albums or even downloaded from the Internet.

Figure 1:  Given a reference image and a target image with hole, GeoFill
utilizes predicted correspondence matching and depth maps to estimate a 3D
mesh and relative camera pose and intrinsics. Compared to TransFill, the
previous state-of-the-art approach, GeoFill handles complex scenes better by
iteratively refining predicted depth maps and relative pose.  Figure 2:
Overview of our system pipeline. We are given a target image with a hole and a
source image. We aim at warping the single-source image to the target to fill
the hole. We first estimate the relative pose as well as predict the monocular
depth of the source, and then adjust the scale and offset of the depth map.
After that, to mitigate the potential errors caused by deep models, we jointly
optimize camera relative pose and depth map scale and offset to make the depth
map and image contents well-align near the hole region. Finally, we render the
reprojected source and refine it using post-processing.

However, reference-based image inpainting is very challenging and not well
explored. That is because reference-based inpainting is a photography-oriented
task that has only one single reference frame as “source”, and has usually
large baseline or challenging camera movements between the target and source.
Therefore, the target and source images cannot be easily aligned to fill in
the hole due to issues such as parallax. To address the alignment problems,
the previous state-of-the-art method TransFill [80] has a strong assumption
that the scenes can be blended by multiple planar structures. It clusters the
predicted depth on matched feature points and utilizes multiple homographies
to fill the hole. However, real scenes rarely consist of only a few planar
surfaces, and even when they do, it can be hard to identify the relevant
planar surfaces. This indicates that to better solve the problem of reference-
based inpainting, _it is crucial to understand the camera positions and
geometry of the 3D scene, especially near the hole region_ in order to find
the appropriate content for hole filling.

In this work, we propose a more principled approach that fills the hole region
by explicitly estimating the 3D scene structure from two limited views.
Specifically, we first estimate sparse feature correspondences, from which we
derive an initial relative pose between the two views. We then predict a
monocular dense depth map of the source image and determine a scale and offset
that aligns the depth map with the target using a sparse 3D triangulated point
cloud. To mitigate the prediction errors and improve alignment accuracy, we
next jointly optimize the depth scale and offset and the relative pose using a
fast differentiable 3D reprojection of the source image to the target. We
synthesize a warped source image by rendering a textured source mesh with the
optimized depth and target pose, and fill dis-occluded regions with single
image inpainting. Lastly, we adjust the exposure, white balance, lighting, and
correct any residual misalignments before pasting the result back into the
hole region.

In summary, our GeoFill is the first to apply a more principled approach for
reference-based inpainting, i.e., the first to leverage an explicit non-planar
3D scene representation given only limited two-view RGB images (no camera pose
information). Compared with the previous state-of-the-art, GeoFill better
handles complex 3D scene structures within the hole, wide-baseline image pairs
and larger holes. Extensive experiments demonstrate that our methods achieve
the best perceptual quality on various scenes in benchmarks and real user
cases.

##  2 Related Work

Image inpainting. Traditional image inpainting models rely on hand-crafted
heuristics. Diffusion-based approaches [6] propagate pixel colors from the
background to the hole region. These approaches generate artifacts when the
hole size is large or texture variations are significant. Alternatively,
patch-based approaches [57, 3] search for similar patches outside the hole
region to complete the missing regions. Although these approaches offer high
quality texture by copying texture patches, the filled regions may be
inconsistent with regions around the hole due to lack of high-level structural
understanding of the entire image.

Recent deep models fill the hole by learning from large amounts of training
data. Context encoders [42] generate semantically plausible content in the
hole by encoding the surroundings. Iizuka et al. [26] adopt two discriminators
to ensure the inpainted content is both locally and globally consistent.
Artifacts can be reduced along the hole boundary by filtering using partial
[32] or gated [66] convolutions. Some recent inpainting models improve the
generated image quality with additional information, such as edges [39],
segmentation masks [51], and low frequency structures [47, 31]. Moreover,
several papers show deep neural networks can fill holes on high resolution
images [62, 64, 68]. Despite the significant advancements in single inpainting
models, filling with one single image remains fundamentally an ill-posed
problem [76]. Image inpainting with additional information is also explored in
the literature, such as inpainting with stereo images [55, 7, 2, 38, 37] and
utilizing more than one image [52]. TransFill [80] is closely related to our
work: it performs reference-guided inpainting by warping a reference image
with multiple homographies. However, due to the planar nature of homographies,
TransFill has a limited ability to handle image pairs with complex 3D
structures, wide baselines, or significant disocclusions.

Video Inpainting. Classical works mainly focus on globally optimizing patch-
based energies [43, 58, 19]. Recent work often adopts deep generative models
for better inpainting performance. Wang et al. introduce a data-driven
framework that jointly learns temporal structure and spatial details [53].
Onion-Peel Network (OPN) proposes to fill in the missing region progressively
with the spatio-temporal attention [40]. Spatial-Temporal Transformer Network
(STTN) adopts a deep generative model with adversarial training along the
spatial-temporal dimension to mitigate blurriness and temporal artifacts [67].
Note that video inpainting approaches heavily exploit the dense temporal
information in the video while we only have one single reference image which
is a much harder scenario.

Two-view Geometry. SfM establishes correspondences between two monocular
frames and subsequently estimates 3D structure [21, 50, 78, 54, 36]. In
classic geometric vision, it is well understood that the camera poses as well
as depth for corresponding points can be computed from feature matching points
alone [33, 23]. Traditional methods utilize hand-crafted descriptors [34, 5,
48] to build sparse correspondence for the subsequent fundamental matrix
estimation with the 8-point algorithm [22]. Learned local features have shown
great success in recent works [63, 13, 14] together with the learning-based
feature matching models, such as SuperGlue [49] or differentiable formations
of RANSAC [8, 45, 9]. Another alternative is to directly estimate relative
pose using an end-to-end pose estimation network [27]. We leverage these
recent advances (specifically OANet [69]). Our method uses components inspired
by SfM, however, our differentiable joint optimization stage is novel and has
been carefully formulated for our task.

Monocular Depth Estimation. Predicting depth from a single image is an ill-
posed problem. However, learning based approaches have shown impressive
performance by either treating monocular depth estimation as a regression or
classification task [16, 28, 60, 20, 61, 18, 24, 1, 29, 75, 25, 74]. Recent
advances include BTS [29], which introduces local planar guidance layers to
guide the features to full resolution instead of standard upsampling layers
during the decoding phase. DAV [25] proposes to exploit the co-planarity of
objects in the scene via depth-attention volume. DPT [44] leverages the high
quality intermediate representation from transformers and become state-of-the-
art.

##  3 Method

Suppose we are given a target image ${\bf I}_{t}$ with a hole ${\bf M}$ to be
filled, and a reference (source) image ${\bf I}_{s}$ of the same scene. Our
goal is to find a 3D-aware warped source image ${\bf I}_{s\rightarrow t}$ that
geometrically aligns the source image to the target image which can be used to
fill the hole. The final composite image can be represented as ${\bf
I}_{t}^{\mathrm{comp}}={\bf I}_{t}\odot{\bf M}+({\bf M}_{\mathrm{single}}{\bf
I}_{s\rightarrow t}+(1-{\bf M}_{\mathrm{single}})\odot{\bf
I}_{\mathrm{single}})\odot(1-{\bf M})$, where ${\bf M}_{\mathrm{single}}$ is
the blending map to merge the warped source with the single image inpainting
result ${\bf I}_{\mathrm{single}}$. Note that ideally there should be enough
contents that GeoFill can copy from the source image to the target hole
region, i.e. the source image $I_{s}$ is useful. Cases with very few target-
source content pixels overlapping inside the hole will cause GeoFill to fall
back to the single image inpainting ${\bf I}_{\mathrm{single}}$.

To compute the final warping matrix and use it to reproject the source image,
as shown in Figure 2, we propose a pipeline consisting of three stages named
initialization, joint optimization, and rendering and post-processing. In the
first stage, we establish sparse correspondences between ${\bf I}_{s}$ and
${\bf I}_{t}$ and estimate the relative pose ${\bf T}_{\mathrm{rel}}$ between
two views. Meanwhile, we obtain a dense depth map of the source image using a
pretrained deep model. Then we align the scale and offset of the predicted
depth map with the sparse 3D triangulated feature points. In the second stage,
we mitigate the potential errors of the initial guess of pose and depth by
optimizing the related parameters so contents well-align near the hole.
Finally, we render the warped image using the optimized parameters, and post-
process it to address any residual spatial and color misalignments like
TransFill. We will introduce each stage in the following sections.

###  3.1 Initialization Stage

Initialize Relative Pose Our approach first estimates the relative pose ${\bf
T}_{\mathrm{rel}}$ based on predicted sparse correspondences. We extract
sparse corresponding feature points ${\bf P}_{t}$ and ${\bf P}_{s}$ between
the target and source images, and compute the fundamental matrix ${\bf F}$
between ${\bf I}_{s}$ and ${\bf I}_{t}$ via the normalized 8-point algorithm
[22] using RANSAC [17]. From ${\bf F}$, we derive the relative pose ${\bf
T}_{\mathrm{rel}}$ using the classic multi-view geometry algorithm mentioned
in [22].

Initialize Dense Depth Map We then predict the inverse depth map on the source
image using a pretrained monocular depth estimator as the cues to estimate the
real source depth. Mathematically, we only need the source depth and ${\bf
T}_{\mathrm{rel}}$ to compute the 3D-aware warp ${\bf I}_{s\rightarrow t}$.
However, the source depth is predicted up to an unknown scale and offset by a
pretrained deep model, therefore, we need to solve for these such that the
initial source depth ${\bf D}_{s}^{i}$ best matches the estimated relative
pose. Note the estimated translation in ${\bf T}_{\mathrm{rel}}$ is normalized
and will not match the arbitrary scale in the predicted depth maps from
pretrained deep models. As suggested in [73], aligning dense depth to sparse
triangulated points is much simpler than rescaling the relative pose.
Therefore, we first triangulate points with the relative pose, then align the
scale of depth predictions with triangulation to subsequently match the scale
of the relative pose.

Specifically, a 3D triangulated point ${\bf x}$ with point $q_{s}\in{\bf
P}_{s}$ and $q_{t}\in{\bf P}_{t}$ is computed as,

${\bf x}^{*}=\mathop{\rm argmin}_{{\bf x}}[E({\bf r}_{s},{\bf x})]^{2}+[E({\bf
r}_{t},{\bf x})]^{2},$ (1)

where ${\bf r}_{s}$ represents the ray shooting from the source camera center
through the point $q_{s}$ on the image plane, ${\bf r}_{t}$ is the ray from
the target camera following a similar analogy, and $E$ measures the Euclidean
distance between two inputs. In this way, we compute a set of 3D triangulated
points ${\bf X}$ using all matching sparse correspondences. In order to form
the linear problem to compute the scale and offset, we first compute a sparse
triangulated depth map ${\bf D}_{\mathrm{tri}}$ by projecting 3D triangulated
points to source camera coordinates. Note ${\bf D}_{\mathrm{tri}}$ is in the
same scale as the relative pose. Therefore, we correct ${\bf D}_{s}$ to match
${\bf D}_{\mathrm{tri}}$, which subsequently matches the scale of the relative
pose. We correct ${\bf D}_{s}$ by estimating two scalars, the scale $s^{i}$
and offset $b^{i}$ associated with the depth map, by solving a linear least
square problem. The initial depth maps are then expressed as: ${\bf
D}_{s}^{i}=s^{i}{\bf D}_{s}+b^{i}$.

###  3.2 Joint Optimization Stage

To mitigate the effects of potential errors in sparse correspondence and depth
estimation since deep models can be not robust or generalized enough, we
further introduce an optimization module to improve the quality of ${\bf
I}_{s\rightarrow t}$. We optimize the depth scale, offset, and the relative
pose that jointly define ${\bf I}_{s\rightarrow t}$ in the 3D scene.
Specifically, we convert the rotation matrix into quaternions, which leads to
a total of 9 parameters to optimize. Both relative pose and the initial depth
computed in the previous section are used as the initial guess for the
optimization. Our optimization contains 3 different loss functions: a
multiscale photometric loss $\mathcal{L}_{\mathrm{photo}}$, a feature
correspondence loss $\mathcal{L}_{\mathrm{feat}}$, and a negative depth
penalty $\mathcal{L}_{\mathrm{negD}}$.

Multiscale Photometric Loss $\mathcal{L}_{\mathrm{photo}}$ measures the pixel-
level color difference between ${\bf I}_{s\rightarrow t}$ and the ${\bf
I}_{t}$ outside the hole region. We downsample both ${\bf I}_{s\rightarrow t}$
and ${\bf I}_{t}$ and sum the normalized color difference across different
resolutions. Specifically, we build Gaussian pyramids on both ${\bf
I}_{s\rightarrow t}$ and ${\bf I}_{t}$ using an RGB representation for the
source image and an alpha-premultiplied RGBA representation on the target
image to incorporate the hole region properly into the target image. Computing
a multi-scale photometric loss within each iteration is obviously
computationally expensive. Moreover, the optimization might also get trapped
into the local minima associated with the finest resolution due to poor
initialization [81]. To accelerate the computation speed and find better
solutions, we adopt a coarse-to-fine optimization strategy, which means we
first compute photometric loss on the most coarse level and move to the finer
level once the convergence criteria at the current pyramid level are met.
Additionally, instead of building a 3D triangle mesh and rendering from the
target view at each iteration, we use a much more efficient differentiable 3D
reprojection to find a warping field that computes ${\bf I}_{s\rightarrow t}$
from ${\bf I}_{s}$ with bilinear interpolation. Mathematically, we have:

${\bf I}_{s\rightarrow t}=\mathrm{bilinear}({\bf I}_{s},\mathrm{reproj}({\bf
K},{\bf T}_{\mathrm{rel}},{\bf D}_{s}^{o})),$ (2)

where $\mathrm{reproj}()$ represents the reprojection operation. The
photometric loss at a given resolution with the pyramid is:

$\mathcal{L}_{\mathrm{photo}}=\frac{1}{|{\bf M}|}\sum{\bf W}\odot||{\bf
I}_{s\rightarrow t}\odot{\bf M}-I_{t}\odot{\bf M}||^{2}.$ (3)

Here ${\bf W}$ is a pixel importance weight map discussed shortly.

Feature Correspondence Loss $\mathcal{L}_{\mathrm{feat}}$ computes the
distance between reprojected matching feature points in the source images and
the target images. We use the $\mathrm{reproj}()$ operator on all 2D image
coordinates in ${\bf P}_{s}$ to get another set ${\bf P}_{s\rightarrow t}$.
Then, we compute the average distance between ${\bf P}_{s\rightarrow t}$ and
${\bf P}_{t}$. However, the average distance of all points is very sensitive
to outliers, i.e., very few outliers dominate the loss function. To reduce the
effects of the outliers on the loss function, we adopt the general robust loss
function from [4]. The general form of the loss function is:

$f(x,\alpha,c)=\frac{|\alpha-2|}{\alpha}\left(\left(\frac{(x/c)^{2}}{|\alpha-2|}+1\right)^{\alpha/2}-1\right),$
(4)

where $\alpha$ and $c$ are the shape and scale parameters, respectively. In
our experiments, we set $\alpha=-2$ and $c=10$. Then, feature correspondence
loss is written as:

$\mathcal{L}_{\mathrm{feat}}=\frac{1}{|P_{s\rightarrow
t}|}\sum\limits_{m=0}^{|P_{s\rightarrow t}|}{\bf W}({\bf q}^{m}_{t})f(||{\bf
q}^{m}_{s\rightarrow t}-{\bf q}^{m}_{t}||,\alpha,c),$ (5)

where $f$ is the general robust loss function, ${\bf q}^{m}_{s\rightarrow t}$
and ${\bf q}^{m}_{t}$ is the $m^{th}$ point in $P_{s\rightarrow t}$ and
$P_{t}$, respectively.

Negative Depth Penalty $\mathcal{L}_{\mathrm{negD}}$ aims to penalize negative
values in the remapped depth. Although depth predictions have arbitrary scale
and offset, they should have all positive values, meaning that a fragment of
geometry associated with a pixel should never move behind the camera.
Mathematically, we adopt a hinge loss function:

$\mathcal{L}_{\mathrm{negD}}=\sum\max\\{0,-{\bf D}_{s}^{o}\\}.$ (6)

Our final objective function is written as:
$l=\lambda_{1}\mathcal{L}_{\mathrm{photo}}+\lambda_{2}\mathcal{L}_{\mathrm{feat}}+\lambda_{3}\mathcal{L}_{\mathrm{negD}}$,
where $\\{\lambda_{j}\\}$ are weights. The convergence criteria is discussed
in the supplemental material.

Pixel Importance Weight Map W. When computing the photometric loss or feature
correspondence loss, we assign weights to each pixel to replace uniform
weighting. It encourages our optimization to better align the warped source
with the target image both locally and globally.

The first type of weighting strategy is hole-distance weighting ${\bf W}_{h}$.
We encourage the optimization to focus on the regions close to the hole
boundary since those pixels are more important for filling the hole regions.
To achieve this, we apply the distance transform to the hole image and obtain
a distance map ${\bf M}_{h}$, where each pixel records the Euclidean distance
to the closest boundary pixel of the hole ${\bf M}_{h}$. We compute the weight
at each pixel using a Gaussian function ${\bf W}_{h}=\exp(-{\bf
M}_{h}^{2}/2\sigma^{2})$, where $\sigma$ is a hyperparameter that adjusts how
weights change w.r.t distance to the hole boundary.

The second type is the edge-based weighting ${\bf W}_{e}$. This is because
high-gradient edge regions are more salient while checking the alignment
quality so we intend to give strong edges larger weights. We compute a multi-
scale Canny edge map by first applying Gaussian blur on ${\bf I}_{t}$ with N
different kernel sizes, running the Canny edge detector [10], and dilating
each edge to get $\\{e_{1},e_{2},\cdots,e_{N}\\}$. Our pixel-level edge-based
weight map becomes ${\bf W}_{e}=\sum_{k=1}^{N}e_{k}/\sum_{p}e_{k}(p)$, where
the inner sum is over spatial coordinates. Our overall weighting map is ${\bf
W}={\bf W}_{h}\odot{\bf W}_{e}$.

###  3.3 Rendering and Postprocessing Stage

Mesh Rendering. After optimization, we find ${\bf T}_{\mathrm{rel}}^{o}$,
$s^{o}$, and $b^{o}$, the camera relative pose and depth map scale and offset,
respectively. However, computing ${\bf I}_{s\rightarrow t}$ using depth
reprojection always has some gaps between valid pixels due to it relying on a
forward warping for efficiency in the inner loop of the optimization. One way
to overcome this problem is interpolation but that has two disadvantages.
First, the reprojected image is sparse and for regions with multiple layers of
depth, pixels for a far-away depth layer might splat between pixels of a
closer depth layer, which could result in interpolations that do not fully
remove hidden surfaces. Additionally, interpolation cannot distinguish between
holes due to disoccluded regions or simple gaps between the pixels due to the
forward warping.

To address the above problems, we choose to render a textured mesh to get the
final ${\bf I}_{s\rightarrow t}$. We first build a triangle mesh with a
regular grid from the source view. The mesh vertices are computed by
projecting the optimized depth ${\bf D}_{s}^{o}$ to the 3D space and the
texture is the RGB colors of the source image. After obtaining the mesh, we
drop the edges around depth discontinuities. We adopt a simplified version of
the footprint algorithm [56] by comparing the depth values between connected
vertices. We drop the edge between two vertices $v_{i}$ and $v_{j}$ if
$\frac{2|d(v_{i})-d(v_{j})|}{d(v_{i})+d(v_{j})}>\epsilon_{\mathrm{edge}}$,
where $d(v_{i})$ is the depth value of the vertex $v_{i}$ and
$\epsilon_{\mathrm{edge}}$ is a predefined threshold. After building the
triangle mesh, we render the target view with ${\bf T}_{\mathrm{rel}}^{o}$.
Note that we also normalize the mesh to the unit size before rendering. The
textured mesh densely fills in pixels and removes hidden surfaces. It also
allows us to use the rendered alpha channel to find pixels where there is no
ray intersection with the mesh, which represent disoccluded regions near depth
discontinuities or regions outside the photo: these are later filled by single
image inpainting. We use PyTorch3D [46] as our renderer.

Refinement and Merging. With the rendered image, we apply the color-spatial
transformation (CST) module from TransFill to further improve any small
residual spatial misalignments and correct color and exposure differences.
Lastly, we merge the output from CST with results from the single image
inpainting model as in TransFill to handle the disocclusions and regions
outside the photo.

Figure 3: Qualitative comparison of GeoFill against other baselines on user-
provided images (top 2 rows), RealEstate10K (mid 2 rows), and
MannequinChallenge dataset (last two rows).

##  4 Experiments

Datasets. No large-scale image-based dataset for reference-based inpainting is
available. So we follow TransFill to randomly sample multiple image pairs from
video-based datasets because it is easier to simulate user behaviours and
analyze the target-source difference (or camera view changes). During
evaluation we only use one single reference frame. This is different from what
video inpainting works did. We follow DeepFillv2 [66] to generate random free-
form brush stroke masks and evaluate GeoFill and other baselines on the
following datasets.
RealEstate10K [79]: It contains a diverse collection of YouTube video
sequences shot from a moving camera for both indoor and outdoor scenes. Each
video clip contains a variety of views of the same scene. We randomly sample
500 videos and select one pair of images in each video sequence with a
specific frame difference (FD). Specifically, we sample FD=25, 50, and 75 to
build three different sets with a resolution of $720\times 1280$ while
automatically filtering out image pairs without enough overlapping content
inside the hole by checking the number of matching sparse features. Note that
our filtering mechanism is only for the purpose of simulating user behaviors
and removing non-useful image pairs. In practice, we believe users could
simply check overlap between photos by visual inspection.
MannequinChallenge [30]: This is a challenging dataset with video sequences
shot from a hand-held camera freely roaming in a scene of people with frozen
poses. This dataset contains more than 170K frames and corresponding camera
poses derived from about 2k YouTube videos. The camera motion in this dataset
is more extreme and scene complexity is much higher due to diverse frozen
human poses and rich background objects. We reduce the sampling FD to ensure
enough overlap between image pairs. Similar to the previous dataset, we
randomly sample 3 subsets with FD=10, 20, 30, where each contains 500 image
pairs of $720\times 1280$.
Real User-provided Images: We also evaluate our approach on the real user-
provided images like TransFill to validate the generalization ability and
practicability.

Baselines. In addition to TransFill, which is directly related to our work, we
compare our approach against several different types of baselines to evaluate
final inpainting performance. The first type is state-of-the-art video
completion models. OPN [40] achieves high quality inpainting results with
spatio-temporal attention. STTN [67] proposes to optimize a spatial-temporal
adversarial loss function. In addition, we compare against state-of-the-art
single-image inpainting methods including ProFill [68] and CoModGAN [72]. We
use their off-the-shelf pretrained weights directly since they are both
trained on more diverse scenes in Places2 [77] than RealEstate10K and have
strong generalization ability. Lastly, we compare a two-view SfM based
approach [73] — which we refer to as JointDP — by warping the source image
with the jointly estimated relative pose and depth using dense correspondence.
To ensure fairness in the comparison, we use the same depth predictor and
rendering process as in GeoFill while keeping all other settings the same as
in the original work.

Implementation Details. We follow TransFill to extract SIFT features [35] and
feed them into OANet [69] to reject outliers and establish correspondences.
This combination was already proven to be quite robust. A better matching
strategy could always be adopted to further improve results while leaving the
rest of our framework intact. Our pretrained monocular depth predictor is DPT
[44]. No ground-truth camera intrinsic information is required to run our
approach. We use fixed camera intrinsic parameters for all images by setting
focal length to 750 and principal point to the image center. An ablation study
with different focal lengths is included in the supplemental material. We use
the pretrained CST module from TransFill [80] without finetuning. This module
generalizes well to MannequinChallenge and user provided images. Our pipeline
is implemented with PyTorch [41] and we choose DiffGrad [15] as our optimizer
due to its fast convergence speed. In the optimization step, we use a constant
learning rate $10^{-2}$ and the maximum number of iteration is set to
$10^{4}$. The loss weights $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are
10, 10, 0.5, respectively. In the coarse-to-fine optimization strategy, the
number of pyramid levels is 4 and the maximum number of cumulative iterations
at each level from coarse to fine are $4\times 10^{3}$, $7\times 10^{3}$,
$9\times 10^{3}$, $10^{4}$. We set the $\sigma$ in hole-based weighting to 192
pixels. In edge-based weight, we compute 4 different Canny edge maps and
dilate each of them with a kernel size equal to 4. In mesh rendering, the edge
threshold $\epsilon_{\mathrm{edge}}$ is $4\times 10^{-2}$.

Table 1: Quantitative comparisons of GeoFill against other baselines on both
RealEstate10K and MannequinChallenge.

|  RealEstate10K: PSNR $\uparrow$/ SSIM $\uparrow$ / LPIPS $\downarrow$ |  MannequinChallenge: PSNR $\uparrow$/ SSIM $\uparrow$ / LPIPS $\downarrow$
---|---|---
Model | FD=25 | FD=50 | FD=75 | FD=10 | FD=20 | FD=30
JointDP [73] | 22.46 / 0.9469 / 0.1011 | 21.76 / 0.9457 / 0.1063 | 20.89 / 0.9423 / 0.1122 | 20.13 / 0.9346 / 0.1087 | 19.52 / 0.9290 / 0.1195 | 19.38 / 0.9315 / 0.1177
OPN [40] | 28.41 / 0.9684 / 0.0525 | 27.80 / 0.9669 / 0.0570 | 26.91 / 0.9634 / 0.0624 | 25.63 / 0.9628 / 0.0605 | 24.92 / 0.9584 / 0.0698 | 24.84 / 0.9591 / 0.0702
STTN [67] | 28.83 / 0.9696 / 0.0710 | 28.26 / 0.9697 / 0.0721 | 27.59 / 0.9680 / 0.0751 | 25.60 / 0.9623 / 0.0803 | 25.09 / 0.9602 / 0.0865 | 24.94 / 0.9613 / 0.0844
ProFill [68] | 27.45 / 0.9642 / 0.0775 | 27.67 / 0.9654 / 0.0755 | 27.37 / 0.9639 / 0.0768 | 25.04 / 0.9589 / 0.0808 | 25.02 / 0.9582 / 0.0836 | 25.22 / 0.9599 / 0.0810
CoModGAN [72] | 26.02 / 0.9594 / 0.0703 | 26.14 / 0.9607 / 0.0686 | 25.88 / 0.9596 / 0.0697 | 23.39 / 0.9504 / 0.0770 | 23.14 / 0.9486 / 0.0808 | 23.36 / 0.9503 / 0.0791
TransFill [80] |  32.03 / 0.9764 / 0.0461 | 30.64 / 0.9732 / 0.0540 | 29.24 / 0.9694 / 0.0608 | 28.01 / 0.9680 / 0.0569 | 26.56 / 0.9628 / 0.0688 | 26.17 / 0.9632 / 0.0701
GeoFill (Ours) |  32.57 / 0.9775 / 0.0467 | 31.47 / 0.9748 / 0.0525 | 30.43 / 0.9717 / 0.0581 |  28.85 / 0.9702 / 0.0553 | 27.72 / 0.9658 / 0.0652 | 27.44 / 0.9664 / 0.0665

###  4.1 Quantitative Results

The quantitative results comparing our approach with other baselines are shown
in Table 1. We report the PSNR, SSIM and LPIPS[71] on the RealEstate10K and
the MannequinChallenge datasets. Single image inpainting models are not
competitive enough for image pairs with larger scale differences and wider
baselines. Video inpainting approaches also show bad performance due to the
lack of dense temporal information and multiple reference frames. JointDP is
based on optical flow and is not able to accurately estimate parameters like
the camera pose needed to correctly align the image pairs. Our method
demonstrates superiority over TransFill because we have a better understanding
of the 3D structures of the scenes, and better leverage the depth estimation.
Note that GeoFill has higher performance gain over TransFill on
MannequinChallenge than on RealEstate10K dataset. TransFill relies on the
fusion module to handle multiple homographies for the final hole filling and
it is less robust on the images different from the training data. In contrast,
GeoFill only has a single proposal to merge during inpainting, which also
greatly reduces blending artifacts that arise from multiple proposals.
Therefore, our GeoFill is robust and stable on image pairs with even larger
frame differences.

###  4.2 Qualitative Results

Figure 3 shows the visual comparisons with other baseline algorithms on the
user-provided images, the RealEstate10K and the MannequinChallenge dataset.
JointDP utilizes estimated optical flow for the initial matching, thus the
results fail to obtain accurate depth and camera pose if the baseline of the
image pair is wide. The contents inside the hole are often misaligned with the
target image. The original OPN uses five reference frames to achieve a more
efficient non-local matching among frames, but a single reference frame makes
the results less visually-pleasing. ProFill is not able to take advantage of
the reference image contents, and TransFill usually has blending artifacts or
content misalignment issues when objects inside the hole regions occupy
multiple depth planes. However, GeoFill avoids the blending artifacts by using
one single proposal, and aligns the objects well by understanding the camera
poses and reconstructing the 3D scene from two images.

###  4.3 Ablation Study

In the following sections, we study how each optimization loss function and
the pixel importance weight maps contribute to the final results.
Additionally, we present experimental results of TransFill and GeoFill on
cases with larger holes and their alignment accuracy without CST. All
experiment results in the following section are reported for the RealEstate10K
FD=50 subset unless specified. We also include more in-depth analyses of the
joint depth pose optimization module on RealEstate10K and ScanNet [12], the
timing of each module in our system pipeline, the convergence criteria of the
optimization step, and more visualizations in the supplemental material.

Table 2: Ablations on the objective functions in the joint optimization stage
of GeoFill.

$\mathcal{L}_{\mathrm{photo}}$ | $\mathcal{L}_{\mathrm{feat}}$ | $\mathcal{L}_{\mathrm{negD}}$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---|---|---
✓ | ✓ | ✓ | 31.47 | 0.9748 | 0.0525
✗ | ✓ | ✓ | 31.19 | 0.9742 | 0.0533
✓ | ✗ | ✓ | 30.88 | 0.9734 | 0.0554
✓ | ✓ | ✗ | 31.23 | 0.9742 | 0.0532

Table 3: Ablations on the pixel importance weight map ${\bf W}$.

Hole ${\bf W}_{h}$ |  Edge ${\bf W}_{e}$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---|---
✓ | ✓ | 31.47 | 0.9748 | 0.0525
✓ | ✗ | 31.20 | 0.9740 | 0.0534
✗ | ✓ | 31.12 | 0.9739 | 0.0539
✗ | ✗ | 30.95 | 0.9734 | 0.0552

Optimization: Objective Functions. We claim that all the objective functions
used in the optimization process contribute to a higher perceptual and
reconstruction quality as shown in Table 2. Comparing GeoFill without
photometric loss against without feature correspondence loss, the feature loss
contributes to the performance the most. We find that photometric loss by
itself may get distracted by local textures and fall into local minima, such
that it achieves lower RGB errors on average but ignores the global structure.
However, using the photometric loss does help improve alignments. Lastly,
GeoFill without negative depth penalty still has a performance drop,
suggesting negative depth penalty is able to prevent corner cases where depth
scale or offset estimates are non-robust.

Optimization: Pixel Importance Weight Map. We present results of GeoFill with
different combinations of pixel importance weight maps in the optimization in
Table 3. This suggests that hole-distance weighting ${\bf W}_{h}$ which puts
higher weights around the hole lead to better local alignment around the hole.
Edge-based weighting ${\bf W}_{e}$ also helped by matching strong edges within
the image. Using a uniform weighting map leads to the worst performance,
indicating the effectiveness of the weighting maps.

Performance w.r.t Hole Size. This ablation study aims to examine the
performance of our approach against TransFill under more difficult settings.
As the hole becomes larger, inevitably we have fewer matching points, which
makes ${\bf I}_{s\rightarrow t}$ hard to align with ${\bf I}_{t}$. We generate
holes with different average stroke widths ranging from 90 to 210 pixels. As
shown in Figure 4, GeoFill has an increasing performance gain over TransFill
until the hole average stroke width reaches 180 pixels. As the hole size
grows, the complexity of the content in the hole also increases. In other
words, we are more likely to encounter a growing number of depth layers, a
more complicated objects layout, and a higher chance of occlusions due to
camera translation in a larger hole. These problems are harder for homography-
based models, therefore, GeoFill has a greater advantage when the hole is
larger. The drop in performance gain at the end of the curve can be because
there are not enough matching points for GeoFill to infer an accurate 3D
structure.

Initial Alignment Comparisons against TransFill. We adopted the CST module
from TransFill to adjust auto exposure, lighting conditions, and potential
remaining misalignments. In this experiment, we analyze the performance of
using the aligned images directly, e.g. without using CST for both models. We
retain the multiple homographies used by TransFill, drop the CST module, and
keep the merging module so TransFill can merge its different regions. Table 4
shows that GeoFill maintains better performance, dropping 2.12 in PSNR, while
TransFill drops 4.62 without CST. This experimentally validates the proposal
from GeoFill is much more accurate than TransFill by better leveraging the
depth.

Figure 4: Performance gain of our method compared to TransFill w.r.t the average hole size. GeoFill has a greater advantage when the hole is larger. Table 4: Ablations on Initial alignment comparisons of our method compared to TransFill without the CST Module. Model |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---
GeoFill | 31.47 | 0.9748 | 0.0525
GeoFill (no CST) | 29.35 | 0.9688 | 0.0579
TransFill | 30.64 | 0.9732 | 0.0540
TransFill (no CST) | 26.03 | 0.9598 | 0.0742

Per-Sample Improvement Study. This ablation is designed to break down the
numbers shown in Table 1 by comparing the performance of GeoFill against
TransFill over each individual sample. We sort and plot the per-image PSNR
difference over TransFill, shown in Figure 5. Specifically, PSNR difference is
computed as GeoFill PSNR $-$ TransFill PSNR, therefore a positive PSNR gain
indicates GeoFill is better. GeoFill improves performance on the majority of
samples, specifically for around 75.4% of the entire subset. Moreover, the
PSNR gains when GeoFill outperforms are much greater: up to 3 dB, as opposed
to the PSNR losses when TransFill outperforms.

##  5 Discussion, Limitations, and Conclusion

Limitations and Future Work. GeoFill inpaints the hole with one image pair
with no auxiliary pose or depth information from sensors, therefore, our
pipeline may not work well when the quality of feature matching points is
poor, e.g., matching points are inaccurate or too few. Under these cases, our
relative pose and triangulated points can be inaccurate, which may be hard for
optimization to correct. Additionally, our pipeline is also sensitive to depth
prediction quality: artifacts such as blurry depth discontinuities or wrong
order of depth planes can lead to potential bad inpainting results. Future
work might mitigate these problems by jointly reasoning about monocular depth
and the stereo cues established by triangulation. In the optimization, we used
a 3D reprojection based on forward warping because it is much faster than
rendering a triangle mesh even though it does not fully remove hidden surfaces
in the rare cases where a mesh occludes itself: this could be addressed in
future work by testing and pruning those splatted points. GeoFill utilizes the
CST module to adjust auto exposure and lighting condition changes, which still
suffers when the scene environment changes drastically, e.g., day to night,
spring to fall. Future work could better address these by incorporating
specialized lighting estimation (e.g., [70]) and relighting modules. One last
limitation of GeoFill is that our pipeline only handles static scenes. Any
dynamic objects, e.g., walking people or moving cars, not in the hole could
lead to bad relative pose estimation and our optimization may cause
misalignments under such cases. A simple solution would be masking out objects
that are likely to move such as people and cars, but how to correctly identify
all moving objects is still an open question.

Figure 5: Per-sample performance gain of ours compared to TransFill. The blue
vertical line separates positive and negative PSNR gain.

Conclusion. In this work, we proposed a novel reference-based inpainting
approach that achieves state-of-the-art performance. Our method relies on off-
the-shelf depth estimation and sparse correspondence methods, and thus we can
expect its performance to improve if these modules improve in performance.
Compared to previous homography-based work, GeoFill is better at image pairs
with large camera motions and scenes with complicated structures in the hole.
GeoFill ensures accurate alignment between the filling content and the hole by
jointly optimizing depth predictions and relative poses without explicitly
knowing camera intrinsics. Moreover, our approach inpaints the hole in a more
principled way by utilizing a single proposal without heuristic planar
assumptions.

## References

  * [1] Ibraheem Alhashim and Peter Wonka.  High quality monocular depth estimation via transfer learning.  arXiv preprint arXiv:1812.11941, 2018.
  * [2] Seung-Hwan Baek, Inchang Choi, and Min H Kim.  Multiview image completion with space structure propagation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 488–496, 2016.
  * [3] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman.  Patchmatch: A randomized correspondence algorithm for structural image editing.  ACM Trans. Graph., 28(3):24, 2009.
  * [4] Jonathan T Barron.  A general and adaptive robust loss function.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4331–4339, 2019.
  * [5] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  In European conference on computer vision, pages 404–417. Springer, 2006.
  * [6] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester.  Image inpainting.  In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 417–424, 2000.
  * [7] Arnav V Bhavsar and Ambasamudram N Rajagopalan.  Inpainting in multi-image stereo.  In Joint Pattern Recognition Symposium, pages 172–181. Springer, 2010.
  * [8] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother.  Dsac-differentiable ransac for camera localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6684–6692, 2017.
  * [9] Eric Brachmann and Carsten Rother.  Neural-guided ransac: Learning where to sample model hypotheses.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4322–4331, 2019.
  * [10] John Canny.  A computational approach to edge detection.  IEEE Transactions on pattern analysis and machine intelligence, (6):679–698, 1986.
  * [11] Kefan Chen, Noah Snavely, and Ameesh Makadia.  Wide-baseline relative camera pose estimation with directional learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3258–3268, 2021.
  * [12] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017.
  * [13] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Toward geometric deep slam.  arXiv preprint arXiv:1707.07410, 2017.
  * [14] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224–236, 2018.
  * [15] Shiv Ram Dubey, Soumendu Chakraborty, Swalpa Kumar Roy, Snehasis Mukherjee, Satish Kumar Singh, and Bidyut Baran Chaudhuri.  diffgrad: An optimization method for convolutional neural networks.  IEEE transactions on neural networks and learning systems, 31(11):4500–4511, 2019.
  * [16] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  arXiv preprint arXiv:1406.2283, 2014.
  * [17] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [18] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2002–2011, 2018.
  * [19] Miguel Granados, Kwang In Kim, James Tompkin, Jan Kautz, and Christian Theobalt.  Background inpainting for videos with dynamic objects and a free-moving camera.  In European Conference on Computer Vision, pages 682–695. Springer, 2012.
  * [20] Zhixiang Hao, Yu Li, Shaodi You, and Feng Lu.  Detail preserving depth estimation from a single image using attention guided networks.  In 2018 International Conference on 3D Vision (3DV), pages 304–313. IEEE, 2018.
  * [21] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [22] Richard I Hartley.  In defense of the eight-point algorithm.  IEEE Transactions on pattern analysis and machine intelligence, 19(6):580–593, 1997.
  * [23] Richard I Hartley and Peter Sturm.  Triangulation.  Computer vision and image understanding, 68(2):146–157, 1997.
  * [24] Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani.  Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1043–1051. IEEE, 2019.
  * [25] Lam Huynh, Phong Nguyen-Ha, Jiri Matas, Esa Rahtu, and Janne Heikkilä.  Guiding monocular depth estimation using depth-attention volume.  In European Conference on Computer Vision, pages 581–597. Springer, 2020.
  * [26] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics (ToG), 36(4):1–14, 2017.
  * [27] Alex Kendall, Matthew Grimes, and Roberto Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In Proceedings of the IEEE international conference on computer vision, pages 2938–2946, 2015.
  * [28] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In 2016 Fourth international conference on 3D vision (3DV), pages 239–248. IEEE, 2016.
  * [29] Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh.  From big to small: Multi-scale local planar guidance for monocular depth estimation.  arXiv preprint arXiv:1907.10326, 2019.
  * [30] Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T Freeman.  Learning the depths of moving people by watching frozen people.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4521–4530, 2019.
  * [31] Liang Liao, Jing Xiao, Zheng Wang, Chia-Wen Lin, and Shin’ichi Satoh.  Guidance and evaluation: Semantic-aware image inpainting for mixed scenes.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16, pages 683–700. Springer, 2020.
  * [32] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro.  Image inpainting for irregular holes using partial convolutions.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 85–100, 2018.
  * [33] H Christopher Longuet-Higgins.  A computer algorithm for reconstructing a scene from two projections.  Nature, 293(5828):133–135, 1981.
  * [34] David G Lowe.  Object recognition from local scale-invariant features.  In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1150–1157. Ieee, 1999.
  * [35] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [36] Haoyu Ma, Liangjian Chen, Deying Kong, Zhe Wang, Xingwei Liu, Hao Tang, Xiangyi Yan, Yusheng Xie, Shih-Yao Lin, and Xiaohui Xie.  Transfusion: Cross-view fusion with transformer for 3d human pose estimation.  arXiv preprint arXiv:2110.09554, 2021.
  * [37] Liqian Ma, Stamatios Georgoulis, Xu Jia, and Luc Van Gool.  Fov-net: Field-of-view extrapolation using self-attention and uncertainty.  IEEE Robotics and Automation Letters, 6(3):4321–4328, 2021.
  * [38] Wei Ma, Mana Zheng, Wenguang Ma, Shibiao Xu, and Xiaopeng Zhang.  Learning across views for stereo image completion.  IET Computer Vision, 14(7):482–492, 2020.
  * [39] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi.  Edgeconnect: Generative image inpainting with adversarial edge learning.  arXiv preprint arXiv:1901.00212, 2019.
  * [40] Seoung Wug Oh, Sungho Lee, Joon-Young Lee, and Seon Joo Kim.  Onion-peel networks for deep video completion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4403–4412, 2019.
  * [41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32:8026–8037, 2019.
  * [42] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [43] Kedar A Patwardhan, Guillermo Sapiro, and Marcelo Bertalmio.  Video inpainting of occluding and occluded objects.  In IEEE International Conference on Image Processing 2005, volume 2, pages II–69. IEEE, 2005.
  * [44] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12179–12188, 2021.
  * [45] René Ranftl and Vladlen Koltun.  Deep fundamental matrix estimation.  In Proceedings of the European conference on computer vision (ECCV), pages 284–299, 2018.
  * [46] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv preprint arXiv:2007.08501, 2020.
  * [47] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li.  Structureflow: Image inpainting via structure-aware appearance flow.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 181–190, 2019.
  * [48] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  Orb: An efficient alternative to sift or surf.  In 2011 International conference on computer vision, pages 2564–2571. Ieee, 2011.
  * [49] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947, 2020.
  * [50] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016.
  * [51] Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C-C Jay Kuo.  Spg-net: Segmentation prediction and guidance network for image inpainting.  arXiv preprint arXiv:1805.03356, 2018.
  * [52] Theo Thonat, Eli Shechtman, Sylvain Paris, and George Drettakis.  Multi-view inpainting for image-based scene editing and rendering.  In 2016 Fourth International Conference on 3D Vision (3DV), pages 351–359. IEEE, 2016.
  * [53] Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang.  Video inpainting by jointly learning temporal structure and spatial details.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5232–5239, 2019.
  * [54] Jianyuan Wang, Yiran Zhong, Yuchao Dai, Stan Birchfield, Kaihao Zhang, Nikolai Smolyanskiy, and Hongdong Li.  Deep two-view structure-from-motion revisited.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8953–8962, 2021.
  * [55] Liang Wang, Hailin Jin, Ruigang Yang, and Minglun Gong.  Stereoscopic inpainting: Joint color and depth completion from stereo images.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [56] Lee Westover.  Footprint evaluation for volume rendering.  In Proceedings of the 17th annual conference on Computer graphics and interactive techniques, pages 367–376, 1990.
  * [57] Yonatan Wexler, Eli Shechtman, and Michal Irani.  Space-time video completion.  In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 1, pages I–I. IEEE, 2004.
  * [58] Yonatan Wexler, Eli Shechtman, and Michal Irani.  Space-time completion of video.  IEEE Transactions on pattern analysis and machine intelligence, 29(3):463–476, 2007.
  * [59] Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo.  Foreground-aware image inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5840–5848, 2019.
  * [60] Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe.  Multi-scale continuous crfs as sequential deep networks for monocular depth estimation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5354–5362, 2017.
  * [61] Dan Xu, Wei Wang, Hao Tang, Hong Liu, Nicu Sebe, and Elisa Ricci.  Structured attention guided convolutional neural fields for monocular depth estimation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3917–3925, 2018.
  * [62] Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li.  High-resolution image inpainting using multi-scale neural patch synthesis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6721–6729, 2017.
  * [63] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua.  Lift: Learned invariant feature transform.  In European conference on computer vision, pages 467–483. Springer, 2016.
  * [64] Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, and Zhan Xu.  Contextual residual aggregation for ultra high-resolution image inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7508–7517, 2020.
  * [65] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505–5514, 2018.
  * [66] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Free-form image inpainting with gated convolution.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4471–4480, 2019.
  * [67] Yanhong Zeng, Jianlong Fu, and Hongyang Chao.  Learning joint spatial-temporal transformations for video inpainting.  In The Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [68] Yu Zeng, Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, and Huchuan Lu.  High-resolution image inpainting with iterative confidence feedback and guided upsampling.  In European Conference on Computer Vision, pages 1–17. Springer, 2020.
  * [69] Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao.  Learning two-view correspondences and geometry using order-aware network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5845–5854, 2019.
  * [70] Jinsong Zhang, Kalyan Sunkavalli, Yannick Hold-Geoffroy, Sunil Hadap, Jonathan Eisenman, and Jean-Franccois Lalonde.  All-weather deep outdoor lighting estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10158–10166, 2019.
  * [71] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [72] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu.  Large scale image completion via co-modulated generative adversarial networks.  arXiv preprint arXiv:2103.10428, 2021.
  * [73] Wang Zhao, Shaohui Liu, Yezhi Shu, and Yong-Jin Liu.  Towards better generalization: Joint depth-pose learning without posenet.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9151–9161, 2020.
  * [74] Yunhan Zhao, Shu Kong, and Charless Fowlkes.  Camera pose matters: Improving depth prediction by mitigating pose distribution bias.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15759–15768, 2021.
  * [75] Yunhan Zhao, Shu Kong, Daeyun Shin, and Charless Fowlkes.  Domain decluttering: Simplifying images to mitigate synthetic-real domain shift and improve depth estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3330–3340, 2020.
  * [76] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai.  Pluralistic image completion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1438–1447, 2019.
  * [77] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
  * [78] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1851–1858, 2017.
  * [79] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  arXiv preprint arXiv:1805.09817, 2018.
  * [80] Yuqian Zhou, Connelly Barnes, Eli Shechtman, and Sohrab Amirghodsi.  Transfill: Reference-guided image inpainting by merging multiple color and spatial transformations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2266–2276, 2021.
  * [81] Shizhan Zhu, Cheng Li, Chen Change Loy, and Xiaoou Tang.  Face alignment by coarse-to-fine shape searching.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4998–5006, 2015.

Appendices

_In the supplementary document, we provide additional ablation studies to
further support our findings, as well as details of our experiments and more
visualizations. Below is the outline._

  * •

Section A: Convergence criteria. We describe the convergence criteria of our
optimization step in detail.

  * •

Section B: Joint optimization evaluation: inpainting performance. We
demonstrate the importance of the joint optimization module by comparing the
performance of GeoFill with parameters estimated before and after the joint
optimization.

  * •

Section C: Joint optimization evaluation: depth and pose accuracy. We further
evaluate the performance of the optimization module by measuring the accuracy
of depth maps and camera poses individually against ground-truth using
ScanNet.

  * •

Section D: Performance w.r.t intrinsic parameters. Quantitative comparisons of
our approach with different focal lengths.

  * •

Section E: Further analyses of initial alignments. We provide additional
qualitative and quantitative evaluations of GeoFill and TransFill without the
CST module.

  * •

Section F: Visualizations of hole ablation study. We include qualitative
comparisons between GeoFill and TransFill under various hole sizes.

  * •

Section G: Average running time of GeoFill. We report the average running time
of each step in GeoFill.

  * •

Section H: User study against other baselines. A user study of GeoFill against
OPN, ProFill, and TransFill.

  * •

Section I: Handling appearance changes from camera movement. We show
qualitative results of GeoFill handling some common appearance changes due to
camera movement.

  * •

Section J: Failure cases. Visual examples of failure cases of GeoFill.

  * •

Additional Visual Results. We include more inpainting results in Fig. 11 and
12.

##  Appendix A Convergence Criteria

The convergence criteria define when the optimization should stop. Our
optimization halts the loop at a given scale and continues to the next scale
if the following condition is met or the predefined maximum number of
iteration is achieved. The formula below measures the objective function value
changes within the last $m$ iterations.

$\epsilon_{i}=\frac{|\sum_{i-(m/2)-1}^{i}l_{i}-\sum_{i-(m/2)}^{i-m-1}l_{i}|}{\sum_{i-(m/2)-1}^{i}l_{i}},$
(7)

where $i$ represents the $i^{th}$ iteration. If $\epsilon_{i}$ is smaller than
a predefined $\epsilon_{opt}$, we assume the objective function has converged.
Since we adopt a coarse-to-fine optimization strategy, we check the same
condition at every level of the pyramid. In other words, we move to the finer
scale level only if Eqn. 7 is met or maximum number of iterations at the
current level is reached. We also keep track of the optimal parameters at each
level and use them as the initialization in the next level. In practice, we
set the convergence threshold $\epsilon_{opt}$ to $10^{-6}$ for all levels.
The number of loss values to track in computing convergence criteria is
$m=10$.

##  Appendix B Joint Optimization Evaluation: Inpainting Performance

We show the importance of our optimization module by comparing the performance
of GeoFill with initial estimated parameters and optimized parameters. As
shown in Table 5, GeoFill with optimized parameters has substantially better
performance. Initial parameters are computed from SIFT and pretrained models
such as OANet, which can make erroneous predictions, especially for image
pairs with holes. Experimental results demonstrate our optimization module
successfully mitigates such errors and improves the inpainting performance.

##  Appendix C Joint Optimization Evaluation: Depth, Pose Accuracy

In this section, we further demonstrate the effectiveness of our joint
optimization step by measuring the depth and pose accuracy before and after
the optimization. Since both RealEstate10K and MannequinChallenge do not have
ground-truth labels, we choose the ScanNet [12] dataset which comes with
ground-truth camera poses and depth maps. We randomly sampled 75 pairs of
images with approximately 30 frame difference. We generate random holes in the
same manner as described in our main paper line 388. Each image pair comes
from a unique scene in the dataset. Note that ScanNet includes images with
heavy motion blur, which we manually filtered out. We evaluate depth and
relative camera pose _separately_ by providing the ground-truth for one of
these (depth or camera pose) when evaluating the accuracy of the other one.
For example, when evaluating the accuracy of depth maps, we first follow the
same pipeline described in the main paper. Then, instead of estimating the
relative pose, we provide the ground-truth camera pose and evaluate the
accuracy of the depth map determined by our pipeline before and after the
optimization. Note that we only optimize scale and offset when evaluating
depth accuracy. A similar analogy applies when evaluating the accuracy of
camera poses: we provide the ground truth depth map to our pipeline and then
evaluate the accuracy of the relative pose before and after optimization. For
pose evaluation, we report the geodesic errors [11] for both rotations and
translation directions. For depth evaluation, we follow the commonly adopted
metrics used in the literature [16, 18]. As shown in Table 6 and Table 7, both
depth and pose errors are significantly reduced after the optimization module,
which demonstrates the ability of the optimization module to find more
accurate depth and poses in our challenging case where the images have holes.

Table 5: Quantitative comparison of our method with initially estimated parameters and optimized parameters. Model |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---
GeoFill (optim) | 31.47 | 0.9748 | 0.0525
GeoFill (init) | 30.66 | 0.9719 | 0.0548
Table 6: Relative camera pose evaluation of initial guess and our optimized results, where $R$ and $t$ represent the rotation and translation, respectively. | $R\downarrow$ |  | ${t\downarrow}$
---|---|---|---
|  mean (∘) |  med (∘) |  |  mean (∘) |  med (∘)
GeoFill (optim) | 1.588 | 1.062 |  | 3.688 | 3.457
GeoFill (init) | 7.378 | 7.807 |  | 11.861 | 11.096
Table 7:  Depth evaluation of our initial guess and optimized results. The evaluation metrics include absolute relative difference (Absr), squared relative difference (Sqr), root mean squared log error (RMS-log), and accuracy with a relative error threshold of $\delta^{k}<1.25^{k}$, k = 1, 2.  Models |  Abs${}^{r}\downarrow$ |  Sq${}^{r}\downarrow$ |  RMS-log$\downarrow$ | $\delta^{1}\uparrow$ | $\delta^{2}\uparrow$
---|---|---|---|---|---
GeoFill (optim) | .258 | .233 | .302 | .683 | .851
GeoFill (init) | .372 | .766 | .403 | .609 | .788
Figure 6: Visual plots showing the performance of GeoFill with different focal
lengths. PSNR diff is computed by using GeoFill with new focal length subtract
GeoFill with focal length equals to 750.

##  Appendix D Performance w.r.t Intrinsic Parameters

GeoFill handles incoming image pairs using fixed camera intrinsic parameters
instead of explicitly knowing the ground-truth camera intrinsic parameters. In
the main paper, we use fixed camera intrinsic parameters by setting the focal
length of all images to 750 pixels and the principal point to the center of
the image. It is intuitive to set the principal point to the center of the
images with unknown intrinsic parameters, therefore, we focus on studying the
effect of focal lengths. We compare the performance of GeoFill with the camera
focal lengths of 600, 750, 900, 1050, and 1200 pixels. As shown in Fig. 6,
GeoFill with different focal lengths has very slight differences in terms of
PSNR. There is a slight trend that the performance drops as the focal length
increases. We believe this indicates that the ground-truth focal length is
close to 600 and higher focal lengths make the optimization have a harder time
finding improved relative poses. Nevertheless, GeoFill can still adapt to
different focal lengths by jointly optimizing depth scale, offset, and
relative pose, therefore, it still can render similar images across a variety
of focal lengths.

Figure 7: Qualitative comparisons of our approach against TransFill with different hole sizes. Please zoom in to see that ours looks good but there are broken structures, ghosting, and distortion artifacts in TransFill. Table 8: Initial alignment comparisons of our method compared to TransFill without the CST Module on MannequinChallenge dataset (FD=10). Model |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---
GeoFill | 28.85 | 0.9702 | 0.0553
GeoFill (no CST) | 26.84 | 0.9626 | 0.0615
TransFill | 28.01 | 0.9680 | 0.0569
TransFill (no CST) | 24.04 | 0.9526 | 0.0760

##  Appendix E Further Analyses of Initial Alignments

GeoFill adopts the CST module from TransFill to further improve any small
residual spatial misalignments and correct color and exposure differences. We
first visually compare the quality of our single proposal to the merged
proposal from TransFill without the CST on RealEstate10K dataset. As shown in
Fig. 8, the single proposal from GeoFill is significantly more accurate than
merged heuristic proposals from TransFill, demonstrating the superiority of
our approach over TransFill. Additionally, we also show the quantitative
comparisons of GeoFill and TransFill without the CST module on the
MannequinChallenge dataset. As shown in Table 8, we find GeoFill without the
CST module has a huge advantage over TransFill merged homographies.

##  Appendix F Visualizations for Hole Ablation Study

In the main paper, we show the quantitative comparisons between our approach
and TransFill with various hole sizes. Here, we provide visual comparisons to
better understand the performance boost for larger holes. We simulate larger
holes by generating the same hole shape with larger stroke width. As shown in
Fig. 7, GeoFill has a robust performance while TransFill has ghosting
artifacts and misalignments as the hole grows larger.

Figure 8: Visual comparisons of GeoFill and TransFill without the CST module. Figure 9: Visual examples of failure cases of GeoFill. Table 9: User study results of GeoFill against ProFill, OPN, and TransFill. | Filtered | Non-Filtered
---|---|---
Model | PR | p-value | PR | p-value
ProFill | 100% | $p<10^{-6}$ | 96.25% | $p<10^{-6}$
OPN | 97.37% | $p<10^{-6}$ | 95.00% | $p<10^{-6}$
TransFill | 70.90% | $p<2\times 10^{-3}$ | 68.13% | $p<2\times 10^{-3}$

##  Appendix G Average Running Time of GeoFill

We randomly sampled 50 images at 1280x720 pixels and compute the average time
of each step. Monocular depth estimation takes 3.83s, sparse correspondence
estimation takes 0.596s, triangulation takes 0.0009s, initial relative pose
takes 0.0052s, joint optimization takes 58.2s, mesh rendering takes 1.03s,
refinement and merging step takes 2.53s. The reported time uses default
parameters described in the experiment section. Although the joint
optimization step takes up the vast majority of the time, its current
implementation is naive and not optimized. If desired, various engineering
optimizations could be made such as using a custom kernel with proper low-
level optimizations such as fusion for the renderer instead of a naive pure
PyTorch implementation, using FP16 mode, using only the sparse edge map pixels
during optimization (these are quite sparse so significant acceleration should
be possible), using second-order optimization techniques that could
potentially converge in fewer steps, carefully tuning input resolution, number
of pyramid levels, iteration limits, break thresholds, etc. We considered
these to be lower-level engineering details that we did not focus on in our
paper’s implementation, since we were focusing more on research aspects.

##  Appendix H User Study

To better evaluate the performance of GeoFill against other baselines, we
conduct a user study via Amazon Mechanical Turk (AMT). We compare our method
against OPN, ProFill, and TransFill by showing users image pairs with binary
choice questions. The users are requested to choose the inpainting results
that look more realistic and faithful. To improve the quality of collected
data, we adopt a qualification test with trivial questions to filter noisy
results. For each method pair, we randomly sampled 80 examples in
RealEstate10K dataset with FD=50, and each example was evaluated by 7
independent users. We present two approaches to computing the preference rate.
The first one is the filtered approach, in which we filter the responses to
retain only those where one method is “preferred” if 6 or more users select
it. The filtering helps suppress noise in the responses of Mechanical Turk
workers, whose work quality can vary. The second one is the non-filtered
approach where we retain all responses and choose the method as “preferred”
where a simple majority of 4 or more users select it. We reported GeoFill’s
Preference Rate (PR) in Table 9. GeoFill has much higher preference rates
against OPN and ProFill. Compared against TransFill, we receive a PR around
70% on filtered and non-filtered approaches. TransFill is still very robust on
small holes and relatively small camera motions in the randomly sampled data.
Therefore, GeoFill is favored by users over TransFill but less strongly than
in the other comparisons. We performed a one sample permutation t test with
$10^{6}$ simulations using the null hypothesis that each pair are preferred
equally by users: the p-values are all sufficiently small that the preference
for our method is statistically significant.

Figure 10: Qualitative results of GeoFill handling some common appearance
changes such as in white balance and exposure due to camera movement.

##  Appendix I Handling Appearance Changes from Camera Movement

As we stated in the main paper, we focus on the common scenario of capturing
photos with the same camera freely moving around. However, there are potential
appearance changes of the same parts of the scene due to the camera movement,
for example, changes due to automatic exposure or automatic white balance
between source and target images. This is a common yet non-trivial challenge
when applying GeoFill in real-world applications. In this section, we show
some visual examples of image pairs with appearance changes in the dataset. As
shown in Figure 10, GeoFill still inpaints plausible results even when the
appearance of the same part of the scene is different between source and
target images.

##  Appendix J Failure Cases

We show some failure cases of GeoFill under extreme conditions. Fig. 9 shows
three common failure cases of GeoFill. The image pair on the left contains
transparent surfaces in the images. These objects often cause monocular depth
estimators to fail and can lead to bad optimization results. In the second
failure case, the drastic changes in the lighting environment affect the
feature correspondence matching and depth prediction, which makes the final
result from GeoFill less accurate. In the last case, dynamic objects, e.g.,
pedestrians, make our optimization module estimate inaccurate parameters. We
discuss in the last section of our main paper ways that future work might
address these issues.

## Additional Visual Results

We include additional qualitative comparisons of GeoFill against other
baselines in Fig. 11. Additionally, we also show the inpainting performance of
GeoFill on user-provided images, RealEstate10K, and MannequinChallenge dataset
in Fig. 12.

Figure 11: Qualitatively comparison of GeoFill against other baselines on
user-provided images (top 3 rows), RealEstate10K (mid 3 rows), and
MannequinChallenge dataset (last 3 rows).

Figure 12: Visual illustration of inpaiting performance of GeoFill on user-
provided images, RealEstate10K, and MannequinChallenge dataset.