# Regularized Shallow Image Prior for Electrical Impedance Tomography

Zhe Liu, Zhou Chen, Qi Wang, , Sheng Zhang, and Yunjie Yang Zhe Liu, Zhou Chen
and Yunjie Yang are with the SMART Group, Institute for Digital
Communications, School of Engineering, The University of Edinburgh, Edinburgh
EH9 3JL, U.K. (e-mail<EMAIL_ADDRESS>y.yang@ed.ac.uk).
Qi Wang is with the School of Electronics $\&$ Information Engineering,
Tiangong University, Tianjin 300387, China (e-mail: wangqitju@163.com).
Sheng Zhang is with the Peking University Shenzhen Hospital, Shenzhen, China.
Manuscript received xxxx, xxxx; revised xxxx, xxxx.

###### Abstract

Untrained Neural Network Prior (UNNP) based algorithms have gained increasing
popularity in tomographic imaging, as they offer superior performance compared
to hand-crafted priors and do not require training. UNNP-based methods usually
rely on deep architectures which are known for their excellent feature
extraction ability compared to shallow ones. Contrary to common UNNP-based
approaches, we propose a regularized shallow image prior method that combines
UNNP with hand-crafted prior for Electrical Impedance Tomography (EIT). Our
approach employs a 3-layer Multi-Layer Perceptron (MLP) as the UNNP in
regularizing 2D and 3D EIT inversion. We demonstrate the influence of two
typical hand-crafted regularizations when representing the conductivity
distribution with shallow MLPs. We show considerably improved EIT image
quality compared to conventional regularization algorithms, especially in
structure preservation. The results suggest that combining the shallow image
prior and the hand-crafted regularization can achieve similar performance to
the Deep Image Prior (DIP) but with less architectural dependency and
complexity of the neural network.

###### Index Terms:

Inverse problem, electrical impedance tomography, shallow multi-layer
perceptron, untrained neural network prior, hand-crafted prior

##  1 Introduction

Iverse Problems (IPs) exist in various imaging techniques, such as Computed
Tomography (CT) [1, 2] and Magnetic Resonance Imaging (MRI) [3, 4, 5]. The
task of the inverse problem in imaging is to reconstruct an unknown image from
noisy measurements. As IPs are ill-posed problems, reliable priors are needed
to improve their invertibility. Conventional methods usually adopt hand-
crafted priors, such as $L_{1}$ regularization, Total Variation ($TV$)
regularization, and so forth. This type of priors has a poor discriminative
ability which may cause undesired solutions. Recently, Deep Learning (DL) has
offered a different paradigm for IPs due to its remarkable nonlinear fitting
and feature extraction abilities. Reported work has demonstrated that DL-based
methods can solve certain problems that hand-crafted prior-based algorithms
cannot effectively address [6, 7].

Like other imaging fields, researchers in Electrical Impedance Tomography
(EIT) also devote themselves to IPs using DL. As a fast functional imaging
modality, EIT is widely investigated in industrial processes [8, 9] and bio-
medicine [10, 11, 12]. However, its further development is limited by the low
spatial resolution caused by the severe ill-posedness of the EIT inversion.
Therefore, DL has naturally been leveraged to solve such a challenging
problem.

Figure 1: The flowchart of the regularized shallow image prior (R-SIP) based
EIT image reconstruction algorithm. $\boldsymbol{\theta}^{0}$ represents the
initial MLP’s parameters. The cross symbol in a blue circle denotes the matrix
product. Dashed arrows either denote or point toward a data operation and
solid arrows represent data flow.

The past several years have witnessed many DL-based EIT image reconstruction
algorithms. For example, Hamilton et al. proposed to use U-Net as a post-
processing method for deconvolving the convolved direct reconstructions of the
D-bar method and demonstrated structure-enhanced EIT images [13]. Wei et al.
also adopted the U-Net architecture to process the neural network’s
multichannel inputs originating from the dominant parts of the Induced
Contrast Current (ICC) and produced quality-improved EIT images. In addition,
a V-shaped dense denoising convolutional neural network was proposed to
enhance the EIT images reconstructed using the model-based algorithm [14].
Wang et al. designed a Convolutional Neural Network (CNN) to post-process the
results of the model-based algorithm [15]. These methods either use the
reconstructed images or certain intermediate quantities of other model-based
algorithms as the neural network’s inputs. Another type of DL-based approach
directly learns the mapping from the voltage measurements to the conductivity
distribution. For instance, a CNN was proposed to directly solve the inverse
problem of EIT [16]. In [17], a densely connected U-Net was adopted to solve
the same problem. To endow DL-based algorithms with interpretability, model-
based learning is explored. Herzberg et al. proposed a Graph Convolutional
Newton-type Method (GCNM) for solving EIT image reconstruction and their
results illustrated a good generalization ability on distinct domain shapes
and meshes [18]. Colibazzi et al. designed an unrolled Gauss-Newton network
for EIT inversion and demonstrated improved EIT image quality compared with
the model-based Gauss-Newton algorithm [19]. Other than these single-modal
algorithms, DL-based multi-modal methods have been reported that utilize
information from auxiliary imaging modalities to further improve EIT image
quality [20, 21].

The aforementioned approaches rely on Trained Neural Networks (TNNs), which
usually need a significant amount of training data. This requirement can limit
the algorithm performance and generalization capability, as it depends heavily
on the data volume and quality. UNNP-based methods can address the data-
dependency problem and generate comparable results to TNN-based approaches.
The initial UNNP-based work was described in [22], which established a new
connection between the inverse problem and DL. Within the UNNP framework, a
neural network acts as the regularizer to the inverse problem. Subsequent
research has proposed various UNNP-based approaches [23, 24, 25, 26]. In the
context of EIT, Liu et al. introduced the UNNP-based method for 2D
reconstruction and showed superior image quality compared to conventional
model-based algorithms [27].

The UNNP-based algorithm has demonstrated great potential in EIT inversion and
is worth further exploration. Existing UNNP-based approaches usually involve
carefully designed deep architectures as a deep image prior. In this paper, we
explore the possibility of employing 3-layer MLPs in EIT image reconstruction
in an untrained manner. Additionally, we introduce hand-crafted smoothing
regularization (e.g. $TV$ or $Laplacian$ regularization) to improve image
quality. We refer to this type of priors as Regularized Shallow Image Priors
(R-SIPs). The contributions of this study are as follows:

  * •

We prove that a 3-layer MLP regularizer with a hand-crafted prior, i.e. R-SIP,
is an effective regularization strategy for 2D and 3D EIT imaging. R-SIP-based
algorithms demonstrate comparable performance to DIP-based algorithms and
outperform to conventional regularization-based algorithms.

  * •

The architecture of MLPs can be chosen from a large range, which minimizes the
dependence on the neural network architecture and provides an easy way to
leverage the advantages of UNNPs.

The rest of this paper is organized as follows: Section II introduces the
principle of EIT. Section III details the proposed image reconstruction
algorithm. Section IV describes the experimental setup. Section V gives the
simulation and experimental results and analyzes the properties of our method.
Section VI draws the conclusion and discusses future work.

##  2 EIT Image Reconstruction

Time-difference imaging is adopted in this study. It is based on the
linearized EIT forward model [28]:

$\displaystyle\mathbf{V}=\mathbf{J}\boldsymbol{\sigma},$ (1)

where
$\mathbf{V}=(\mathbf{V}_{o}-\mathbf{V}_{r})/\mathbf{V}_{r}\in\mathbb{R}^{M}$
and
$\boldsymbol{\sigma}=-(\boldsymbol{\sigma}_{o}-\boldsymbol{\sigma}_{r})/\boldsymbol{\sigma}_{r}\in\mathbb{R}^{N}$
represent normalized voltage measurements and conductivity distribution,
respectively. $\mathbf{V}_{o}\in\mathbb{R}^{M}$ denotes the voltage
measurements at the observation time point and
$\mathbf{V}_{r}\in\mathbb{R}^{M}$ stands for those at the reference time
point. Similarly, $\boldsymbol{\sigma}_{o}\in\mathbb{R}^{N}$ represents the
conductivity distribution at the observation time point and
$\boldsymbol{\sigma}_{r}\in\mathbb{R}^{N}$ denotes those at the reference time
point. $M$ and $N$ account for the number of measurements and image
pixels/voxels, respectively. Vector division ${}^{\prime}/^{\prime}$ means
element-wise division. $\mathbf{J}\in\mathbb{R}^{M\times N}$ represents the
normalized sensitivity matrix.

Due to the ill-posedness of EIT inversion, priors are required to reduce the
feasible set size. Thus, EIT image reconstruction can be generally expressed
as:

$\min_{\boldsymbol{\sigma}}\quad||\mathbf{V}-\mathbf{J}\boldsymbol{\sigma}||^{2}+\phi
R(\boldsymbol{\sigma}),$ (2)

where $||\cdot||$ denotes the $l_{2}$ norm.
$R:~{}\mathbb{R}^{n}\rightarrow\mathbb{R}$ stands for the regularization
function and $\phi>0$ is the parameter.

##  3 Methodology

This section first introduces the concept of image representation with MLPs.
Then we describe the Regularized Shallow Image Prior (R-SIP) based 3D image
reconstruction algorithm. Finally, we introduce its extension to the 2D case.

###  3.1 Image Representation with MLP

In Kernel method [29], the unknown conductivity distribution
$\boldsymbol{\sigma}$ can be represented by a linear equation:

$\boldsymbol{\sigma}=\mathbf{K}\boldsymbol{\tau},$ (3)

where $\mathbf{K}\in\mathbb{R}^{N\times N}$ is the kernel matrix that encodes
a certain prior and $\boldsymbol{\tau}\in\mathbb{R}^{N}$ denotes the kernel
coefficients. From the network viewpoint, the Kernel method can be regarded as
an iterative two-layer neural network-based reconstruction algorithm [30].
Extending this idea, like [30, 31, 32] does, $\boldsymbol{\sigma}$ can be
generally expressed by the nonlinear representation:

$\boldsymbol{\sigma}=\kappa(\boldsymbol{\theta};\boldsymbol{\rho}),$ (4)

where $\kappa$ is a neural network. In this study, we refer to $\kappa$ as an
MLP. $\boldsymbol{\theta}\in\mathbb{R}^{Q}$ denotes the parameters of the MLP
and $Q$ is the number of the parameters. $\boldsymbol{\rho}\in\mathbb{R}^{G}$
stands for the random noise, which is the input of the MLP and $G$ represents
the number of input neurons.

###  3.2 Regularized Shallow Image Prior for 3D EIT Image Reconstruction

We first consider the 3D EIT image reconstruction problem, which can be
formulated as a penalized nonlinear optimization problem:

$\displaystyle\min_{\boldsymbol{\theta}}\quad\mathcal{L}(\boldsymbol{\theta};\boldsymbol{\rho}))=||\mathbf{V}-\mathbf{J}\kappa(\boldsymbol{\theta};\boldsymbol{\rho})||^{2}+\frac{\eta}{N}R_{H}(\kappa(\boldsymbol{\theta};\boldsymbol{\rho})),$
(5)

where $R_{H}$ denotes the hand-crafted regularization function. $\eta>0$
represents the parameter for $R_{H}$. In the later section (Section 5), we
demonstrate that a natural image can not always be guaranteed with only SIP.
Therefore, hand-crafted priors are added to further regularize the image. For
simplicity, we denote $\kappa(\boldsymbol{\theta};\boldsymbol{\rho})$ as
$\boldsymbol{\sigma}$ thereafter.

We investigate the performance of R-SIP with two hand-crafted smoothing
regularizations, i.e. Total Variation ($TV$) and $Laplacian$ regularizations.
For $TV$ regularization, $R_{H}=R_{TV}$, where $R_{TV}$ denotes the isotropic
$TV$ regularization function, which provides global prior information and
facilitates edge-preserving and noise smoothing. The expression of $R_{TV}$ is
[33]:

$\displaystyle R_{TV}(\boldsymbol{\sigma})$
$\displaystyle=\sum_{i,j,k}\sqrt{\begin{aligned}
&((\nabla_{h}\boldsymbol{\sigma})_{i,j,k})^{2}+((\nabla_{w}\boldsymbol{\sigma})_{i,j,k})^{2}\\\
&+((\nabla_{d}\boldsymbol{\sigma})_{i,j,k})^{2}+\epsilon\end{aligned}}~{},$
(6) $\displaystyle(\nabla_{h}\boldsymbol{\sigma})_{i,j,k}$
$\displaystyle=(\boldsymbol{\sigma})_{i+1,j,k}-(\boldsymbol{\sigma})_{i,j,k}~{},$
(7) $\displaystyle(\nabla_{w}\boldsymbol{\sigma})_{i,j,k}$
$\displaystyle=(\boldsymbol{\sigma})_{i,j+1,k}-(\boldsymbol{\sigma})_{i,j,k}~{},$
(8) $\displaystyle(\nabla_{d}\boldsymbol{\sigma})_{i,j,k}$
$\displaystyle=(\boldsymbol{\sigma})_{i,j,k+1}-(\boldsymbol{\sigma})_{i,j,k}~{},$
(9)

where $\nabla_{h}\boldsymbol{\sigma}\in\mathbb{R}^{N}$,
$\nabla_{w}\boldsymbol{\sigma}\in\mathbb{R}^{N}$ and
$\nabla_{d}\boldsymbol{\sigma}\in\mathbb{R}^{N}$ represent the gradient
components of $\boldsymbol{\sigma}$ along the height, width, and depth
directions, respectively. $i$, $j$ and $k$ represent the voxel indices when
$\boldsymbol{\sigma}$ is reshaped to a 3D image. $\epsilon$ is a small
constant whose value is set as $10^{-10}$ throughout this paper.

For $Laplacian$ regularization, $R_{H}=R_{Lap}$, where $R_{Lap}$ represents
the $Laplacian$ regularization function, which is a local prior and smooths
sudden intensity variation by punishing its second-order gradient. $R_{Lap}$
is expressed by [28]:

$R_{Lap}(\boldsymbol{\sigma})=||\mathbf{L}\boldsymbol{\sigma}||^{2},$ (10)

where $\mathbf{L}\in\mathbb{R}^{N\times N}$ is the $Laplacian$ matrix and
$\mathbf{L}\boldsymbol{\sigma}$ is the discrete $Laplacian$.
$||\mathbf{L}\boldsymbol{\sigma}||^{2}$ can be acquired by convolving the 3D
version of $\boldsymbol{\sigma}$ with a small kernel
$\boldsymbol{\varsigma}\in\mathbb{R}^{3\times 3}$ , then adding up its all
squared elements. The kernel adopted in this study is:

$\boldsymbol{\varsigma}=\frac{1}{26}\begin{bmatrix}\begin{bmatrix}2&3&2\\\
3&6&3\\\ 2&3&2\end{bmatrix};\begin{bmatrix}3&6&3\\\ 6&-88&6\\\
3&6&3\end{bmatrix};\begin{bmatrix}2&3&2\\\ 3&6&3\\\
2&3&2\end{bmatrix}\end{bmatrix}.$ (11)

The problem in (5) can be solved with various standard optimization
algorithms. In this study, we solve (5) with Adam [34], a widely used
optimizer in network training. In order to perform Adam, at each iteration,
the first step is to calculate the gradient of $\mathcal{L}$ with respect to
neural network parameters $\boldsymbol{\theta}$, i.e.:

$\displaystyle\frac{\partial\mathcal{L}}{\partial\boldsymbol{\theta}}=$
$\displaystyle\left(\frac{\partial
F}{\partial\boldsymbol{\sigma}}\right)^{T}\frac{\partial\boldsymbol{\sigma}}{\partial\boldsymbol{\theta}}+\frac{\eta}{N}\left(\frac{\partial
R_{H}}{\partial\boldsymbol{\sigma}}\right)^{T}\frac{\partial\boldsymbol{\sigma}}{\partial\boldsymbol{\theta}},$
(12)

where we denote $||\mathbf{V}-\mathbf{J}\boldsymbol{\sigma}||^{2}$ as $F$. As
each term in (5) is expressed by $Tensors$, the PyTorch built-in class, we
implement (12) by using PyTorch’s automatic differentiation engine called
$torch.autograd$. Afterward, the neural network parameters can be updated
according to the parameter update rules of Adam. The number of iterations $t$
is considered as the algorithm’ parameter. Thus, there are three parameters
for R-SIP-based reconstruction, i.e $\eta$, $t$, and the learning rate of Adam
(denoted by $lr$). The flowchart of the R-SIP algorithm is visually
illustrated in Fig. 1.

###  3.3 Extension to 2D EIT Image Reconstruction

The 3D image reconstruction algorithm can be directly applied to the 2D case
with two modifications. First, the $TV$ regularization only involves height
and width directions. Second, the $Laplacian$ matrix used in the 2D situation
is the same as that in [28]. The other aspects of the algorithm remain the
same.

##  4 Experimental Setup

###  4.1 Data for Evaluation

####  4.1.1 2D Simulation

We modeled a circular region and attached 16 electrodes to its boundary (see
Fig. 2a) in COMSOL Multiphysics. In this circular region, two types of
conductivity distributions (labeled as case 1 and case 2) are constructed.
Case 1 simulates a triangular object whose conductivity is 0.8 $S/m$. Case 2
includes a triangular object with a conductivity of 0.4 $S/m$ and a
rectangular bar with a conductivity of 1.2 $S/m$. The background conductivity
is 2 $S/m$ for cases 1 and 2. The adjacent measurement protocol [35] is
adopted for 2D imaging, producing 104 voltage measurements.

Figure 2: a, Simulated 2D and 3D phantoms. The inclusions are highlighted with
bluish violet. b, Thoracic phantom with the illustration of the cross-layer
stimulation-measurement pattern. In a and b, the golden color indicates
electrodes. c, The measured reference voltage data.

Figure 3: Image reconstruction comparison on simulated 2D EIT data.

Figure 4: Image reconstruction comparison on simulated 3D EIT data. The
smaller x/y/z slice number corresponds to the smaller x/y/z coordinate and
vice versa. The word following the slice number is the inclusion’s name. As
the z slice contains all inclusions, we do not add any suffixes.

####  4.1.2 3D Simulation

For 3D imaging, we modeled cylindrical and thoracic regions for EIT imaging
(see Fig. 2a and 2b). Thirty-two electrodes arranged as 2 layers are placed on
the boundaries, and each electrode layer includes 16 electrodes. For the
cylindrical region, we utilize various objects to form three types of
conductivity distributions (labeled as cases 3, 4, and 5). For case 3, we put
a cone in the imaging region and set its conductivity as 1 $S/m$; For case 4,
we place a cone and a cylinder in the imaging region. The conductivities of
the cone and cylinder are 0.8 $S/m$ and 1.2 $S/m$, respectively. For case 5,
we put a cone, a sphere, and a cuboidal slab in the imaging region, and their
conductivities are 0.4 $S/m$, 1 $S/m$, and 1.2 $S/m$ accordingly. The
background medium’s conductivity is 2 $S/m$ for all cases. For the thoracic
region, we modeled lungs with pulmonary edema. The background conductivity is
0.24 $S/m$. The conductivity of healthy lung tissue (indicated by green) and
unhealthy lung tissue (indicated by fuchsia) are 0.06 $S/m$ and 0.24 $S/m$,
respectively.

Figure 5: a, Convergence of the proposed algorithm. b, The noise resistance
performance of our method. c, The image reconstruction result of case 1
without hand-crafted prior. d, Image reconstruction results of case 3 without
hand-crafted prior. e, Image reconstruction results of case 3 without voltage
data from the cross-layer stimulation-measurement pattern. f, $RE$ and $MSSIM$
variation with the different numbers of MLP’s input and hidden neurons.

We number the electrodes ($e_{1}$, $e_{2}$, …, $e_{32}$) in the thoracic model
to describe our customized sensing strategy for 3D imaging (see Fig. 2b). Our
sensing strategy is divided into 3 stages, including 2 intra-layer
stimulation-measurement patterns and 1 cross-layer stimulation-measurement
pattern. The adjacent strategy [35] is adopted for intra-layer stimulation and
measurement. Following the adjacent strategy, we first collect voltages at the
bottom layer of electrodes and then collect voltages at the top layer of
electrodes. There are 104 measurements for each intra-layer stimulation-
measurement pattern.

Figure 6: Thoracic image reconstruction comparison. The smaller z slice number
corresponds to the smaller z coordinate and vice versa. Letters ’L’ and ’R’
mark the left lung and the right lung, respectively.

For the cross-layer stimulation-measurement pattern, the current is injected
into a certain electrode at the bottom layer and flows out from the electrode
right over that electrode. Then, voltages are collected from the rest
electrode pairs. Each measurement pair of electrodes aligns vertically. In
addition, we only collect independent voltages according to the reciprocal
theorem [36]. According to this rule, the first stimulation pair is ($e_{1}$,
$e_{17}$), and measurement pairs are ($e_{2}$, $e_{18}$), ($e_{3}$, $e_{19}$),
…, ($e_{16}$, $e_{32}$); the second stimulation pair is ($e_{2}$, $e_{18}$),
and measurement pairs are ($e_{3}$, $e_{19}$), ($e_{4}$, $e_{20}$), …,
($e_{16}$, $e_{32}$); …; the last stimulation pair is ($e_{15}$, $e_{31}$),
and the measurement pair is ($e_{16}$, $e_{32}$). Thus, the number of
measurements is 120 for the cross-layer stimulation-measurement pattern.
Eventually, we acquire 328 measurements in total for 3D imaging. The simulated
reference voltage data is illustrated in Fig. 2c.

####  4.1.3 Real-World Data

In real experiments, we imaged the cylindrical region using the EIT system
[37] developed at The University of Edinburgh. The cylindrical EIT sensor has
the same configuration as that in the simulation. For all experiments, the
stimulation current frequency was 10 kHz, and the customized sensing strategy
described was adopted. We place 3 sets of objects in the imaging region (see
the first column of Fig. 7). The first set includes a red cuboid made of
plastics, the second set contains two lung models made of silicone, and the
third set consists of three objects (i.e. cone, cuboid, and a slab) made of
plastics.

###  4.2 Imaging Domain Discretization

In 2D imaging, we use square inverse mesh. The x/y axis between the minimum
and maximum x/y coordinates of the imaging region is split into 64 segments.
Thus, there are 3228 pixels within the circular region. In 3D imaging, we
adopt cuboidal inverse mesh and take a similar splitting method to the 2D
case. The x, y, and z axes are split into 32, 32, and 40 segments. Eventually,
there are 32480 voxels in the cylindrical imaging region and 27418 voxels in
the thoracic imaging region.

###  4.3 Structure of MLP

We design three decoder-type 3-layer MLPs labeled as MLP-Cir, MLP-Cyl, and
MLP-Tho for imaging circular, cylindrical, and thoracic regions. Due to
distinct numbers of voxels/pixels corresponding to different imaging domains,
the numbers of MLP’s output neurons are 3228, 32480, and 27418 for MLP-Cir,
MLP-Cyl, and MLP-Tho, respectively. These 3 MLPs adopt the same input (328
neurons) and hidden (2000 neurons) layers. All MLPs take the leaky rectified
linear unit $LeakyReLu$ as the activation function for the hidden layer. In
biomedical applications, such as thoracic imaging and cell monitoring, the
conductivity of the region of interest (RoI) usually decreases during certain
physiological or pathological processes, causing $\boldsymbol{\sigma}$ within
the range of $(0,~{}1)$. Therefore, the $Sigmoid$ is selected as the
activation function of the output layer for all MLPs. If the $TV$
regularization is selected as the hand-crafted prior with an MLP, e.g. MLP-
Cyl, in the R-SIP algorithm, the algorithm is then labeled as MLP-Cyl-TV. If
the $Laplacian$ regularization is chosen, the algorithm is labeled as MLP-Cyl-
Lap. Other MLPs also follow this labeling rule.

###  4.4 Comparison Algorithms

We choose model-based $TV$ and $Laplacian$ regularization for comparison in 2D
and 3D imaging. These two algorithms are labeled as TV and Laplace,
respectively. They adopt the same expressions of $TV$ and $Laplacian$
regularization functions as those in our proposed method, and we also use Adam
to solve the problem. The parameters of both TV and Laplace include the
regularization parameter, the learning rate of Adam, and the number of
iterations. In the 2D comparison, we additionally choose the $TV$
regularization-based algorithm in [27], which is a UNNP-based algorithm with a
deep net. The reason that we only compare with it in 2D cases is that the
neural network is designed only for 2D imaging in the original paper. This
algorithm is labeled as DeepEIT-TV. There are three differences in
implementing the DeepEIT-TV algorithm. First, we adopt the algorithm framework
for time-difference imaging while the original paper conducts absolute
imaging. Second, we divide the $TV$ regularization term by the number of
pixels. Third, we use the $64\times 64$ inverse mesh for 2D imaging while the
original paper employs the $128\times 128$ inverse mesh. The modifications aim
to make the comparison algorithms consistent with our method in as many
aspects as possible for a fair comparison.

###  4.5 Parameter Settings

Parameter selection of all algorithms is based on trial and error and follows
the settings in this subsection if not specified. For the R-SIP algorithm, we
fix $t$ to 2000 for both simulation and real experiment studies. $lr$ is set
as $10^{-4}$ for 2D simulation and real experiments, and set as $5\times
10^{-4}$ for 3D simulation. Regarding TV and Laplace, we fix the learning rate
as $10^{-2}$ for simulation and real experiments; the number of iterations is
set as 2000 for the simulation study and 1000 for real experiments. For cases
1 and 2, the number of iterations and learning rate of the DeepEIT-TV are set
as 8000 and 0.005, respectively. The regularization parameter of all
algorithms is shown with the reconstructed images. The regularization
parameter is denoted by $\phi_{T}$, $\phi_{L}$, and $\phi_{D}$ for TV,
Laplace, and DeepEIT-TV, respectively.

###  4.6 Quantitative Metrics

We employ Image Relative Error ($RE$) to evaluate the performance in
differentiating distinct conductivity levels, and Mean Structural Similarity
Index ($MSSIM$) to evaluate the algorithm’s structure preservation ability.
$RE$ and $MSSIM$ are defined as:

$RE=\frac{||\boldsymbol{\sigma}_{p}-\boldsymbol{\sigma}_{g}||}{||\boldsymbol{\sigma}_{g}||}~{},$
(13)
$MSSIM=\frac{1}{N}\sum\frac{\left(2\boldsymbol{\iota}_{p}\boldsymbol{\iota}_{g}+\chi_{1}\right)\left(2\boldsymbol{\delta}_{p,g}+\chi_{2}\right)}{\left(\boldsymbol{\iota}_{p}^{2}+\boldsymbol{\iota}_{g}^{2}+\chi_{1}\right)\left(\boldsymbol{\delta}_{p}^{2}+\boldsymbol{\delta}_{g}^{2}+\chi_{2}\right)}~{},$
(14)

where $\boldsymbol{\sigma}_{p}$ and $\boldsymbol{\sigma}_{g}$ denote the
reconstructed and true normalized conductivity distributions.
$\boldsymbol{\iota}_{p}\in\mathbb{R}^{N}$,
$\boldsymbol{\iota}_{g}\in\mathbb{R}^{N}$,
$\boldsymbol{\delta}_{p}\in\mathbb{R}^{N}$,
$\boldsymbol{\delta}_{g}\in\mathbb{R}^{N}$ and
$\boldsymbol{\delta}_{p,g}\in\mathbb{R}^{N}$ denote the local means, standard
deviations and cross-covariance for $\boldsymbol{\sigma}_{p}$ and
$\boldsymbol{\sigma}_{g}$, respectively. $\chi_{1}=(\psi_{1}\gamma)^{2}$ and
$\chi_{2}=(\psi_{2}\gamma)^{2}$ are constants in which $\psi_{1}$, $\psi_{2}$
and $\gamma$ are set as 0.01, 0.03 and 1, respectively. The standard deviation
of the isotropic Gaussian function in $MSSIM$ calculation is set as 0.35.
Summation in (14) is over all elements of operands. Due to the intrinsic
modeling error of the linear EIT forward model, absolute reconstruction cannot
be expected. Therefore, all reconstruction results are normalized with
$\boldsymbol{\sigma}/max(abs(\boldsymbol{\sigma}))$ before evaluation and
display. $abs()$ converts each element of a vector to its absolute value.
$max()$ returns the maximum element of a vector.

##  5 Results and Discussion

###  5.1 Simulation Results

Fig. 3 shows the image reconstruction results using noise-free 2D data.
Quantitative metrics are placed under each image. The results indicate that
neural network based algorithms have better performance on background artifact
suppression than conventional algorithms. Especially, our method achieves the
best structure preservation performance. For example, our method reconstructs
the triangular object with the sharpest vertices and the straightest edges. In
addition, our method successfully preserves the rectangular bar in case 2 and
separates it from the triangular object. The highest $MSSIM$ of the MLP-Cir-TV
and MLP-Cir-Lap also supports this conclusion. For $RE$, our method has the
lowest value in case 1, while the $RE$ in case 2 is not the lowest. This
result is reasonable because, in case 2, our method estimates bigger
triangular inclusion, the rectangular bar is slightly deformed, and the
conductivity levels are not perfectly estimated. These factors make an impact
on the result.

Fig. 4 compares algorithms on noise-free data collected from the cylindrical
imaging region. Quantitative metrics are displayed under each EIT image block.
For each image block, we display not only the 3D volume images but also four
selected slices. For 3D images, we transparentize low-absolute-value voxels
based on trials for highlighting the inclusions’ structure. The results
illustrate that our method achieves considerable improvements in structure
prevention and background artifact suppression. For example, for cases 3, 4,
and 5, only our method can reconstruct the cone while other algorithms
reconstruct a round-shaped rod. For other geometrical objects in these three
cases, our method can still preserve their geometrical characteristics, but
other algorithms cannot make it. Regarding the quantitative metrics, our
method has the lowest $RE$ and highest $MSSIM$, which further proves that our
method reconstructs EIT images with the best quality compared with TV
regularization-based and Laplace regularization-based algorithms.

Figure 7: Image reconstruction comparison based on experimental data. The
smaller z slice number corresponds to the smaller z coordinate and vice versa.
For lung imaging, the left and right lungs are indicated in selected images by
’L’ and ’R’, respectively. Regards 3-object phantom imaging, digits ’1’ $\sim$
’3’ are shown with selected images. ’1’ represents the cone, ’2’ denotes the
cuboid, and ’3’ accounts for the slab.

Fig. 5a illustrates the convergence property of our method based on case 5. It
displays the variation of the objective of (5) with the iteration. The
vertical axis takes the logarithmic scale. The objective decreases smoothly,
indicating a stable convergence property.

We use case 4 to discuss the noise resistance ability of the proposed method.
We add various levels of Gaussian noise to the voltage data, forming noisy
data with SNR 10 dB $\sim$ 90 dB. The MLP-Cyl with $TV$ is chosen as the
R-SIP. The variation of the $RE$ and $MSSIM$ with the SNR is illustrated in
Fig. 5b. In addition, we also display 3D images corresponding to selected
noise levels. The results indicate that the performance of the proposed method
becomes stable when SNR is over 30 dB. Even when the noise level is as low as
20 dB, our method can still correctly recover the inclusions, suggesting that
the proposed algorithm has good noise resistance ability.

We use cases 1 and 3 as examples to discuss the function of the hand-crafted
regularization, i.e. $TV$ or $Laplacian$ regularization, in our framework. 2D
and 3D image reconstruction results without hand-crafted priors are shown in
Fig. 5c and in Fig. 5d, respectively. The left image block in Fig. 5d adopts
2000 iterations, the middle image block takes 8000 iterations, and the right
image block uses 20000 iterations. Other parameters are the same as those
described in subsection 4.5. For the 2D situation, a satisfactory result is
acquired. For the 3D situation, we observe that with only the neural network
regularizer, our method can still suppress most background artifacts, capture
the ROI and recover some structural information. However, the reconstructed
images suffer from discontinuities and irregular boundaries. With the
iteration increasing, this phenomenon still remains. It indicates that SIP
imposes imperfect regularization on the conductivity estimation in complex
setups, such as 3D imaging. Therefore, the hand-crafted prior is added to
further regularize the SIP, though it is not always necessary in certain
situations.

We adopt case 3 to demonstrate the necessity of the cross-layer stimulation-
measurement pattern in capturing vertical structure information for 3D EIT.
The MLP-Cyl with $Laplacian$ regularization is selected as the R-SIP. $lr$,
$\eta$, and $t$ are set as $10^{-4}$, $5\times 10^{-4}$ and 2000. The image
reconstruction result is shown in Fig. 5e. From the y slices 21 and 25, we see
that the bottom of the inclusion reaches the bottom of the imaging region,
inconsistent with the ground truth (see Fig. 4). Thus, the cross-layer
stimulation-measurement pattern can provide more vertical information in 3D
imaging.

Fig. 5f demonstrates the influence of the number of input and hidden neurons
on the algorithm performance using case 3. We do not change the number of
output neurons as it is determined by the imaging region. MLP-Cyl with
$Laplacian$ regularization is selected as the R-SIP. Algorithm parameters are
stated in subsection 4.5. According to the results, $RE$ becomes the lowest,
and $MSSIM$ reaches the highest when the number of input neurons is over 140
and the number of hidden neurons is over 1400. If the number of input neurons
is small, e.g. 80, we can increase the number of hidden neurons to increase
the algorithm performance. The number of optional configurations of the input
layer and hidden layer is extremely large. There are two aspects that are
noteworthy. First, if we fine-tune the parameters for each MLP, the
performance of certain MLPs may become better. Second, if we increase the
number of MLP layers, regularization may not be necessary. Although it is not
possible to investigate all situations, the results offer sufficient evidence
that the 3-layer MLP is an effective choice of UNNP in 2D and 3D EIT imaging.
Moreover, the results suggest a wide range of options for the number of input
and hidden neurons, indicating that the proposed method requires minimal
effort in designing the neural network architecture.

Finally, we evaluate our method in thoracic imaging and the results are shown
in Fig. 6. For each algorithm, we display 3D images from two different angles
and some selected slices. We can see that the proposed method demonstrates
better performance on background artifact suppression and structure
preservation. Especially, for the left lung, our method generates a more
accurate structure compared to other methods. For example, the left lung in
the z slices 21 $\sim$ 25 of the $TV$ results is an ellipse, which is far from
the ground truth. In addition, our method achieves the lowest $RE$ and highest
$MSSIM$, further validating its superior performance.

###  5.2 Real Experiment Results

Fig. 7 gives the image reconstruction results based on experimental data. For
each algorithm, we display the 3D image and selected 2D slices. From the 2D
images, we can observe that our method has better performance on background
artifact suppression than other algorithms. In addition, the proposed method
can reconstruct objects with more precise structures than conventional
algorithms. For example, z slices 19 $\sim$ 21 of the cuboid image indicate
that our method can reconstruct the bar-type shape while TV and Laplace
reconstruct a circular shape. For lung model imaging, our method also
reconstructs the lung model with a more accurate shape. For the three-object
phantom, we can reconstruct the slab while other algorithms can not
differentiate the slab from other objects (see z slices 21 and 22). The x
slice 8 shows that only our approach can reconstruct the downward cone which
other algorithms fail to do. It is noted that the objects in the images of the
three objects imaging distort, which is possibly caused by the imperfection of
the EIT measurements and inevitable errors in the experiments.

##  6 Conclusion

We propose an image reconstruction framework based on the shallow image prior
with hand-crafted regularization for 2D and 3D EIT image reconstruction. The
shallow image prior is implemented by representing the conductivity
distribution with a 3-layer MLP, and the hand-crafted regularization is
combined to further improve image quality. Our method provides a way to
develop UNNP-based algorithms, which considerably reduce the time cost of
designing the neural network architecture and release the algorithm’s
dependency on it. We demonstrate that our method has a smooth convergence
property. Simulation and real experiments show that our method achieves the
best performance compared with given algorithms, especially in structure
preservation. Future work will extend this method to multi-frequency EIT
imaging and apply it to quantitative analysis in tissue engineering.

## References

  * [1] J. Wang, T. Li, H. Lu, and Z. Liang, “Penalized weighted least-squares approach to sinogram noise reduction and image reconstruction for low-dose X-ray computed tomography,” IEEE transactions on medical imaging, vol. 25, no. 10, pp. 1272-1283, 2006.
  * [2] H. Zhang, B. Liu, H. Yu, and B. Dong, “MetaInv-Net: Meta inversion network for sparse view CT image reconstruction,” IEEE Transactions on Medical Imaging, vol. 40, no. 2, pp. 621-634, 2020.
  * [3] J. Trzasko, and A. Manduca, “Highly Undersampled Magnetic Resonance Image Reconstruction via Homotopic $\ell_{0}$-Minimization,” IEEE Transactions on Medical imaging, vol. 28, no. 1, pp. 106-121, 2008.
  * [4] C. Qin, J. Schlemper, J. Caballero, A. N. Price, J. V. Hajnal, and D. Rueckert, “Convolutional recurrent neural networks for dynamic MR image reconstruction,” IEEE transactions on medical imaging, vol. 38, no. 1, pp. 280-290, 2018.
  * [5] S. Ravishankar, and Y. Bresler, “MR image reconstruction from highly undersampled k-space data by dictionary learning,” IEEE transactions on medical imaging, vol. 30, no. 5, pp. 1028-1041, 2010.
  * [6] G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, “Deep learning techniques for inverse problems in imaging,” IEEE Journal on Selected Areas in Information Theory, vol. 1, no. 1, pp. 39-56, 2020.
  * [7] G. Wang, J. C. Ye, and B. De Man, “Deep learning for tomographic image reconstruction,” Nature Machine Intelligence, vol. 2, no. 12, pp. 737-748, 2020.
  * [8] C. Dang, M. Darnajou, C. Bellis, G. Ricciardi, S. Mylvaganam, and S. Bourennane, “Improving EIT-based visualizations of two-phase flows using an eigenvalue correlation method,” IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-9, 2021.
  * [9] S. Wang, F. Giorgio-Serchi, and Y. Yang, “A virtual platform of electrical tomography for multiphase flow imaging,” Physics of Fluids, vol. 34, no. 10, pp. 107104, 2022.
  * [10] I. Frerichs, J. Hinz, P. Herrmann, G. Weisser, G. Hahn, M. Quintel, and G. Hellige, “Regional lung perfusion as determined by electrical impedance tomography in comparison with electron beam CT imaging,” IEEE transactions on medical imaging, vol. 21, no. 6, pp. 646-652, 2002.
  * [11] K. Zhang, R. Guo, M. Li, F. Yang, S. Xu, and A. Abubakar, “Supervised descent learning for thoracic electrical impedance tomography,” IEEE Transactions on Biomedical Engineering, vol. 68, no. 4, pp. 1360-1369, 2020.
  * [12] E. Ravagli, S. Mastitskaya, N. Thompson, F. Iacoviello, P. R. Shearing, J. Perkins, A. V. Gourine, K. Aristovich, and D. Holder, “Imaging fascicular organization of rat sciatic nerves with fast neural electrical impedance tomography,” Nature Communications, vol. 11, no. 1, pp. 6241, 2020.
  * [13] S. J. Hamilton, and A. Hauptmann, “Deep D-bar: Real-time electrical impedance tomography imaging with deep neural networks,” IEEE transactions on medical imaging, vol. 37, no. 10, pp. 2367-2377, 2018.
  * [14] X. Zhang, Z. Wang, R. Fu, D. Wang, X. Chen, X. Guo, and H. Wang, “V-shaped dense denoising convolutional neural network for electrical impedance tomography,” IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1-14, 2022.
  * [15] Q. Wang, H. Zhang, X. Li, X. Duan, J. Wang, R. Zhang, H. Zhang, Y. Ma, H. Wang, and J. Jia, “Error-constraint deep learning scheme for electrical impedance tomography (EIT),” IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1-11, 2021.
  * [16] C. Tan, S. Lv, F. Dong, and M. Takei, “Image reconstruction based on convolutional neural network for electrical resistance tomography,” IEEE Sensors Journal, vol. 19, no. 1, pp. 196-204, 2018.
  * [17] F. Li, C. Tan, and F. Dong, “Electrical resistance tomography image reconstruction with densely connected convolutional neural network,” IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-11, 2020.
  * [18] W. Herzberg, D. B. Rowe, A. Hauptmann, and S. J. Hamilton, “Graph convolutional networks for model-based learning in nonlinear inverse problems,” IEEE transactions on computational imaging, vol. 7, pp. 1341-1353, 2021.
  * [19] F. Colibazzi, D. Lazzaro, S. Morigi, and A. Samoré, “Learning nonlinear electrical impedance tomography,” Journal of Scientific Computing, vol. 90, no. 1, pp. 58, 2022.
  * [20] Z. Liu, P. Bagnaninchi, and Y. Yang, “Impedance-optical dual-modal cell culture imaging with learning-based information fusion,” IEEE Transactions on Medical Imaging, vol. 41, no. 4, pp. 983-996, 2021.
  * [21] Z. Liu, R. Zhao, G. Anderson, P. -O. Bagnaninchi and Y. Yang, ”Enhanced Multi-Scale Feature Cross-Fusion Network for Impedance–Optical Dual-Modal Imaging,” IEEE Sensors Journal, vol. 23, no. 5, pp. 4455-4465, 2023, doi: 10.1109/JSEN.2022.3200758.
  * [22] D. Ulyanov, A. Vedaldi, and V. Lempitsky, ”Deep image prior,” IEEE CVPR, pp. 9446–9454, 2018.
  * [23] A. Qayyum, W. Sultani, F. Shamshad, R. Tufail, and J. Qadir, “Single-shot retinal image enhancement using untrained and pretrained neural networks priors integrated with analytical image priors,” Computers in Biology and Medicine, vol. 148, pp. 105879, 2022.
  * [24] Y. Gandelsman, A. Shocher, and M. Irani, ”double-dip: Unsupervised image decomposition via coupled deep-image-priors,” arXiv:1812.00467, 2018.
  * [25] R. Heckel and P. Hand, ”Deep decoder: Concise image representations from untrained non-convolutional networks,” arXiv:1810.03982, 2018.
  * [26] S. Rey, S. Segarra, R. Heckel, and A. G Marques, ”Untrained graph neural networks for denoising,” arXiv:2109.11700, 2021.
  * [27] D. Liu, J. Wang, Q. Shan, D. Smyl, J. Deng and J. Du, ”DeepEIT: Deep Image Prior Enabled Electrical Impedance Tomography,” IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2023.3240565.
  * [28] Z. Liu, H. Gu, Z. Chen, P. Bagnaninchi and Y. Yang, ”Dual-modal Image Reconstruction for Electrical Impedance Tomography with Overlapping Group Lasso and Laplacian Regularization,” IEEE Transactions on Biomedical Engineering, doi: 10.1109/TBME.2023.3243781.
  * [29] Z. Liu and Y. Yang, ”Multimodal Image Reconstruction of Electrical Impedance Tomography Using Kernel Method,” in IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1-12, 2022, Art no. 5002012, doi: 10.1109/TIM.2021.3132830.
  * [30] K. Gong, C. Catana, J. Qi and Q. Li, ”PET Image Reconstruction Using Deep Image Prior,” in IEEE Transactions on Medical Imaging, vol. 38, no. 7, pp. 1655-1665, July 2019, doi: 10.1109/TMI.2018.2888491.
  * [31] K. Gong et al., ”Iterative PET Image Reconstruction Using Convolutional Neural Network Representation,” in IEEE Transactions on Medical Imaging, vol. 38, no. 3, pp. 675-685, March 2019, doi: 10.1109/TMI.2018.2869871.
  * [32] K. Gong, C. Catana, J. Qi and Q. Li, ”Direct Reconstruction of Linear Parametric Images From Dynamic PET Using Nonlocal Deep Image Prior,” in IEEE Transactions on Medical Imaging, vol. 41, no. 3, pp. 680-689, March 2022, doi: 10.1109/TMI.2021.3120913.
  * [33] G. González, J. M. J. Huttunen, V. Kolehmainen, A. Seppänen, and M. Vauhkonen, “Experimental evaluation of 3D electrical impedance tomography with total variation prior,” Inverse Problems Sci. Eng., vol. 24, no. 8, pp. 1411–1431, Oct. 2016.
  * [34] D. P. Kingma and J. Ba, “ADAM: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [35] B. H. Brown and A. D. Seagar, “The Sheffield data collection system,” Clin. Phys. Physiol. Meas., vol. 8, no. 4A, p. 91, Nov. 1987.
  * [36] D. B. Geselowitz, “An application of electrocardiographic lead theory to impedance plethysmography,” IEEE Transactions on biomedical Engineering, no. 1, pp. 38-41, 1971.
  * [37] Y. Yang and J. Jia, “A multi-frequency electrical impedance tomography system for real-time 2D and 3D imaging,” Rev. Sci. Instrum., vol. 88, no. 8, Aug. 2017, Art. no. 085110.


11institutetext: Center for Machine Vision and Signal Analysis, University of
Oulu, Finland 22institutetext: Tampere University, Finland
22email<EMAIL_ADDRESS>
# Predicting Novel Views Using
Generative Adversarial Query Network

Phong Nguyen-Ha 11 Lam Huynh 11 Esa Rahtu 22 Janne Heikkila 11

###### Abstract

The problem of predicting a novel view of the scene using an arbitrary number
of observations is a challenging problem for computers as well as for humans.
This paper introduces the Generative Adversarial Query Network (GAQN), a
general learning framework for novel view synthesis that combines Generative
Query Network (GQN) and Generative Adversarial Networks (GANs). The
conventional GQN encodes input views into a latent representation that is used
to generate a new view through a recurrent variational decoder. The proposed
GAQN builds on this work by adding two novel aspects: First, we extend the
current GQN architecture with an adversarial loss function for improving the
visual quality and convergence speed. Second, we introduce a feature-matching
loss function for stabilizing the training procedure. The experiments
demonstrate that GAQN is able to produce high-quality results and faster
convergence compared to the conventional approach.

###### Keywords:

novel view synthesis generative adversarial query network mean feature
matching loss

##  1 Introduction

Humans are easily able to build a mental understanding of the 3D scene based
on just 2D images. With such ability, we can effortlessly imagine unseen views
of 3D scenes and objects which is currently extremely challenging for computer
based systems. Instead of reconstructing the scene as an explicit 3D model,
humans can approximate new views by combining images obtained from nearby
poses. Such task of predicting an image from a novel view point, given a
limited set of other images from the same scene, is referred as a novel view
synthesis in computer vision literature.

Novel view synthesis can be considered as a fundamental problem in computer
vision and it is still being studied actively by the community. Despite of the
tremendous progress obtained during the years, the problem is still far from
being solved. There are several reasons making novel view synthesis extremely
challenging. First, a perfect solution would require knowledge of the full
light field of all visible objects which is usually not possible to obtain due
to occlusions and limited number of samples. Second, due to

Figure 1: Illustration of the structure of GAN [9], GQN [7] and our proposed
GAQN where $x$, $p$ and $p_{q}$ are input images, camera poses and queried
poses, respectively. $z$ is the latent representation, $y$ is labelled output
as real/fake and finally, $x$’ is the generated image.

short distance to the scene geometry, even small changes in the viewpoint may
lead to substantial changes in the image appearance. Finally, large parts of
the scenes are texture-less and textured areas are typically small. Accurate
rendering of novel views can be a useful component for many machine vision
applications such as robotics and augmented reality.

The outline of this paper is as follows. In Section 2, we present works
related to novel view synthesis methods. A brief background of two previous
works that inspired our method is presented in Section 3. In Section 4.1 and
Section 4.2, we introduce our Generative Adversarial Query Network with
feature matching loss. In Section 5, we show our experimental results and
discuss the effectiveness of adversarial training and feature matching loss to
the previously proposed methods. Finally, we conclude our research in Section
6.

##  2 Related work

In this paper, we propose a solution to the task of novel view synthesis when
multiple source images are given. Early works on this domain manage to cope
with small viewing angle changes by interpolation [5], warping [27] or stereo
reconstruction [26]. When the input cameras are further separated from each
others, these methods do not perform well due to the sparsity of sampled
plenoptic function. Traditional structure-from-motion, structure-from-depth
and multi-view geometry methods [4, 16, 31, 34] predict novel views through
the estimated 3D structure (point clouds, mesh clouds or a collection of
predefined primitives) of the environment. Although, these methods show good
results on abundant source data, they are unable to recover the desired target
views with a limited number of input images due to the ambiguity of 3D
environments. Moreover, estimating the full 3D scene structure may be more
challenging problem than that of synthesizing novel images from new
viewpoints.

Recent works on deep generative model such as Variational Auto Encoder (VAE)
[19], Generative Adversarial Network (GAN) [9] and their variants [1, 17, 20,
33] have demonstrated remarkable results in generating highly photo-realistic
novel images. Based on these results, one could expect that similar
architectures would be applicable to predict 3D structure of the environment.
However, the results demonstrated so far, are far from the desired quality.
For instance, the viewpoint transformation networks explicitly learn the
relationship between input views of the same 3D scene, but the result is
limited in scale such as predicting novel views of an individual rotated
objects [6, 12, 28, 29, 32, 36] or predicting a small displacement between
stereo camera views [8, 35].

A recent generative model that has shown promises in learning representation
for 3D scenes structure is Generative Query Network (GQN) [7]. GQN makes use
of an iterative latent variable density model [10] to generate images of the
3D scene. Using multiple source images as input, GQN presents an end-to-end
learning framework that generates the novel view from the queried pose by
leveraging learned knowledge of the 3D scene representation. However, GQN is
known for large memory consumption and the predicted novel views are sometimes
blurry.

##  3 Background

In this section, we provide the reader with a brief background of Generative
Adversarial Networks (GANs) [9] and Generative Query Network (GQN) [7]. Figure
1 shows the overall structure of our method and compares it to the previously
proposed GANs and GQN.

###  3.1 Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs) [9] consist of two competing
architectures referred as a generator (G) and a discriminator (D) (see Figure
1(a)). The generator G maps a given latent representation $z$ (possibly a
vector with random values) into a novel image $x^{\prime}=G(z)$ that is then
passed to the discriminator network D. The discriminator aims to determine if
the given sample is produced by the generator, or if it is a real image taken
from the training set. Denoting the real training samples as $x$, the
conventional generator loss $\mathcal{L}_{G}^{GAN}$ and discriminator loss
$\mathcal{L}_{D}^{GAN}$ are defined as:

$\displaystyle\mathcal{L}_{G}^{GAN}$
$\displaystyle=-\mathbb{E}_{\bm{z}\thicksim P_{\bm{z}}}[\log{D(G(\bm{z}))}]$
(1) $\displaystyle\mathcal{L}_{D}^{GAN}$
$\displaystyle=-\mathbb{E}_{\bm{x}\thicksim
P_{x}}[\mathrm{log}D(\bm{x})]-\mathbb{E}_{\bm{z}\thicksim
P_{\bm{z}}}[\mathrm{log}(1-D(G(\bm{z}))]$ (2)

Both networks are trained simultaneously in an alternating fashion. In the
ideal case, the procedure guides the generator to produce images that are
indistinguishable from the training image distribution. However, in practice
the training procedure is challenging due to various problems such as mode
collapses [21].

###  3.2 Generative Query Network (GQN)

Generative Query Network (GQN) [7] is a deep generative model that learns the
scene representation to perform novel view synthesis. Using an arbitrary
number of observations, GQN can be trained to generate new views from the same
environment. A GQN network includes a scene encoder(Enc) and a decoder(Dec) as
can be seen in Figure 1(b).

First, the Enc network tries to compress all of the 3D scene information into
a latent representation $z$ from multiple pairs of input views $x$ and their
camera poses $p$. The Enc network processes each pair of input views and
camera poses by using a feed-forward deep convolutional neural network. Each
camera pose $p$ is represented by a 7 dimensional vector of $x_{t}$, $y_{t}$,
$z_{t}$, $sin(yaw)$, $cos(yaw)$, $sin(pitch)$ and $cos(pitch)$ where $x_{t}$,
$y_{t}$, $z_{t}$ are 3D translation, and $yaw$ and $pitch$ are parameters of
the 3D rotation matrix. The latent scene information $z$ is the summation of
all output pairs from the Enc network.

The goal of the GQN is to predict novel views by using the queried pose
$p_{q}$. Therefore, $p_{q}$ and $z$ are input to the Dec network to generate
the new view $x^{\prime}$. The Dec network is a conditional latent variable
model DRAW [11] which includes $M$ pairs of Inference and Generation recurrent
sub network [10]. In the language of GANs, the term of decoder Dec and
generator G have many similarities since they both synthesize data from the
latent vector $z$ (could be also random vector). Both Enc and Dec are trained
jointly to minimize the evidence lower bound (ELBO) loss $\mathcal{L}_{GQN}$ :

$\mathcal{L}_{GQN}=-\mathbb{E}_{(x,p),z}\bigg{[}-\ln\mathcal{N}(x_{gt}|x^{\prime})+\sum_{n=1}^{M}\Big{[}\mathcal{N}(q_{m},{\pi}_{m})\Big{]}\bigg{]}$
(3)

The GQN training loss $\mathcal{L}_{GQN}$ is the expected value over the
negative log-likelihood of the target image $x_{gt}$ given the target
distribution regularized by the cumulative Kullback Leibler divergence between
obtained posterior $q_{m}$ and prior $\pi_{m}$ distributions from $m^{th}$
generation step.

##  4 Generative Adversarial Query Network

The proposed Generative Adversarial Query Network (GAQN) builds on the GQN
architecture by introducing two novel contributions. We will explain both of
these in the following subsections. The corresponding experimental results are
presented in Section 5. Figure 1(c) shows the overall architecture of our
method.

###  4.1 Adversarial loss

As illustrated in Figure 1(c), the proposed GAQN consists of three components:
an encoder network Enc, a decoder network Dec, and a discriminator network D.
The GAQN utilities the same Enc and Dec architecture as standard GQN to
generate a novel view $x^{\prime}=Dec(Enc(x,p),p_{q})$. However, inspired by
the recent advancement of GANs, we include an additional discriminator network
D to distinguish between the generated fake images from the GQN and the ground
truth view in the training data. In this way, the discriminator acts as a
learn-able loss function that boosts the learning process of GQN.

Numerous works [1, 21, 24, 25] have shown that the training of the GAN may be
unstable due to the vanishing gradients problem caused by the binary cross
entropy loss as defined in Equation (1) and (2). In this paper, we avoid the
above problem by adopting the least-square loss function from the previously
proposed Least Squares Generative Adversarial Networks (LSGANs) [20]. The idea
of LSGANs has been proved to be effective since it tries to pull the fake
samples closer to the decision boundary of the least-square loss function.
Based on the distance between the sampled data and the decision boundary,
LSGANs manages to generate better gradients to update its generator.
Furthermore, LSGANs also proves to exhibit less mode-seeking behaviour [3]
which also stabilize the training process. Equation (4) and (5) provide the
generator and discriminator loss of LSGANs that we use to train the GQN
decoder and our proposed discriminator, respectively.

$\displaystyle\mathcal{L}_{G}^{LSGAN}$
$\displaystyle=\mathbb{E}_{\bm{z}\thicksim
P_{\bm{z}}}\Big{[}(D(G(\bm{z}))-1)^{2}\Big{]}$ (4)
$\displaystyle\mathcal{L}_{D}^{LSGAN}$
$\displaystyle=\mathbb{E}_{\bm{z}\thicksim
P_{\bm{z}}}\Big{[}(D(G(\bm{z})))^{2}\Big{]}+\mathbb{E}_{\bm{x}\thicksim
P_{\bm{x}}}\Big{[}(D(x^{\prime})-1)^{2}\Big{]}$ (5)

Inspired by recent works [33, 22] on GANs, our proposed Discriminator $D$
network adopts the residual blocks architecture [13]. Instead of directly
classifying the generated image as real or fake, we follow the patch-based
discriminator [15] to restrict the attention to the structure in the local
patches. Table 1 shows the design of our discriminator network.

Figure 2: Comparison between generated novel view using the GQN and the
proposed GAQN architectures. The first four columns depict the training views
of the scene and the fifth column shows the ground truth image for the novel
view. The last two columns illustrate the results obtained using GQN
architecture and the proposed approach, respectively. Our method is able to
obtain clearly sharper images compared to the plain GQN architecture.

The original GQN includes per-pixel variance annealing technique [7] that is
aimed to focus the learning for the scene environment (wall, floor and sky),
location and color of objects at the early stage of the training and enhance
the low level details later. Even though the annealing strategy was shown to
work, we argue that the adversarial loss is able to further speed up the
learning and to generate sharper object details at the early stages of the
training. Therefore, our GAQN can achieve better visualized novel view earlier
than GQN.

Figure 2 illustrates the predicted novel view using the plain GQN architecture
and the proposed GAQN architecture. Although the obtained GQN model
successfully predicts the correct location and color of three objects in the
3D scene, their edges are blurry. Meanwhile our proposed method manages to
produce sharper object edges, especially, to the middle green icosahedron.
Further results and implementation details are provided in Section 5.

Table 1: Architecture of our discriminator $D$ network using 3 residual blocks [13]. Layers | Input size | Output size
---|---|---
conv$2\times 2$, stride = 2, channels = 32, ReLU | $64\times 64\times 3$ | $32\times 32\times 32$
ResBlock down 64 | $32\times 32\times 32$ | $16\times 16\times 64$
ResBlock down 128 | $16\times 16\times 64$ | $8\times 8\times 128$
ResBlock down 256 | $8\times 8\times 128$ | $4\times 4\times 256$
ResBlock down 512 | $4\times 4\times 256$ | $2\times 2\times 512$
conv$1\times 1$, stride = 1, channels = 1024, ReLU | $2\times 2\times 512$ | $2\times 2\times 1024$

###  4.2 Feature-matching loss

Inspired by the recent works [2, 25, 30] on improving the stability of the GAN
training, we add an extra feature matching loss to train the generator
network. The main idea of this feature matching loss is to use the
discriminator network as a feature extractor and guide the generator to
generate data that matches the feature statistics of the real data. There are
several approaches on exploiting the feature matching loss in training the
generator.

Specifically, the common GAN generator loss as shown in Equation (1) is being
replaced by a mean feature matching loss. It has been argued that this mean
feature matching loss helps preventing the gradient vanishing problem during
the training. In our research, we have already adopted the least square loss
to address the above problem (Section 4.1) but there is no guarantee that the
problem is completely solved. Therefore, we train our GAQN generator network
using a unified loss function $\mathcal{L}_{G}^{GAQN}$ as the combination of
LSGANs generator loss $\mathcal{L}_{G}^{LSGAN}$ and mean feature matching loss
$\mathcal{L}_{FM}$. Let $f_{D}()$ be the mean of the output feature maps from
the 3rd layer (ResBlock down 128 in Table 1) of the discriminator network, the
mean feature matching loss is define as follow:

$\mathcal{L}_{FM}=||\mathbb{E}_{\bm{x}\thicksim
P_{\bm{x}}}f_{D}(x_{gt})-\mathbb{E}_{\bm{z}\thicksim
P_{\bm{z}}}f_{D}(x^{\prime})||_{2}^{2}$ (6)

In this paper, we jointly train GAQN Enc and G by using
$\mathcal{L}_{EG}^{GAQN}$ which is the conventional GQN ELBO loss (Equation
(3)). The D parameters are being updated by the least squares loss (Equation
(5)) from LSGANs, and we adopt the mean feature matching loss (Equation (6))
to update G. Finally, Equations (7), (8) and (9) show all the loss functions
we have used to train our GAQN model.

$\displaystyle\mathcal{L}_{G}^{GAQN}$
$\displaystyle=\mathcal{L}_{G}^{LSGAN}+\mathcal{L}_{FM}$ (7)
$\displaystyle\mathcal{L}_{D}^{GAQN}$ $\displaystyle=\mathcal{L}_{D}^{LSGAN}$
(8) $\displaystyle\mathcal{L}_{EG}^{GAQN}$ $\displaystyle=\mathcal{L}_{GQN}$
(9)

##  5 Experiments

###  5.1 Experimental settings

We evaluate our method using the rooms ring camera dataset that was provided
and used by Eslami et al. in [7]. The dataset contains various rendered 3D
square rooms that contain random objects of various shapes, colors and
locations. Moreover, the scene textures, walls and lights are also randomly
generated. Therefore, the task of predicting the novel views on this data-set
is a relatively challenging problem.

The original GQN construction proposed in [7] consumes a large amount of
computing and memory resources. Due to computational restrictions, we
demonstrate the advantages of the proposed architectural changes using a
smaller version of the basic GQN network and a subset of the training data.
Originally, training a 12 generative-layers GQN model with batch size of 36
requires 4 NVIDIA K80 GPUs as shown in [7]. In this paper, we use a single
NVIDIA Tesla P100 GPU to train our GAQN model which has 8 generative layers
and batch size is 20. As far as we experimented, this change would not affect
the quality of results. Despite of the smaller network size, our method is
able to produce results (see figures 2 and 4) that are very close to those
presented in the original work [7] that uses clearly larger network and
training set. This further, emphasized the benefits of the proposed
architectural modifications. Moreover, we only use the first halve of the
original GQN’s training and testing data for faster training procedure.

We implement our model on PyTorch [23] and our GAQN model is trained end-to-
end using ADAM optimization [18] with hyper-parameter $\beta_{1}=0.9$ and
$\beta_{2}=0.999$. The generator network is trained using the learning rate of
$10^{-4}$ and the discriminator is trained using a learning rate of $4\times
10^{-4}$. Recent work by Heusel et al. [14] shows that if we update the
generator slower than the discriminator then it helps to reach the convergence
easier.

###  5.2 Results

Table 2: Comparison of training, testing loss, KL testing loss and SSIM testing score between our GAQN model (GQN + LSGANs + FM), original GQN and a variant (training GQN with LSGANs loss). | GQN | GQN + LSGANs |  | GAQN
---
(ours)
Training loss | 7012 | 6988 | 6951
Testing loss | 7003 | 6974 | 6957
KL testing loss | 17.59 | 19.36 | 22.91
SSIM | 0.742 | 0.815 | 0.865
Figure 3: Comparison of generator and discriminator training loss between our
proposed GAQN (a) and GQN + GAN (b). Both generator and discriminator of
GQN+GAN are suffering from mode collapsing and vanishing gradients. Using the
least-square loss and the mean feature loss, our GAQN achieves a stable
learning process. Figure 4: Generated novel view comparison between our
proposed GAQN and variants

Our approach relies on two new components, namely, an adversarial training
pipeline using the least-square GANs loss and the discriminator feature
matching loss functions to enhance the previously proposed GQN model. In this
section, we will experimentally investigate the impact of the both components.
The results corresponding to only GQN and adversarial loss are denoted as (GQN
+ LSGANs), whereas the results with the proposed full model are denoted as
GAQN. We train the above methods using the same hyper-parameters (reported in
Section 5.1) and record their training and testing loss for evaluation
purposes. We also use the structure similarity index (SSIM) to assess on the
quality of the target image and the predicted novel on a held-out test set.

Table 2 contains the obtained results. The proposed full GAQN method has
clearly the smallest training and testing loss and the largest
Kullback–Leibler (KL) testing loss. The KL loss represents the distance
between the estimated and true distribution produced by GQN’s generation
network and this KL loss is the second term in the ELBO loss (Equation (3)).
If the model manages to produce high KL loss then the predicted novel view
tends to be close to the ground-truth data. It is also evident that the
adversarial loss is able to improve the results compared to the plain GQN
architecture. However, the largest gain is obtained by combining both of the
proposed contributions.

In Figure 4, we show qualitative examples of the generated novel views
produced by different versions of the proposed GAQN method and the baseline.
Although the baseline manages to correctly generate the colors and positions
of objects in most cases, theirs edges are blurry. Based on the SSIM, our GAQN
model achieves highest score and produce sharper edges on predicted images.

Finally, we illustrate how the least squares loss of LSGANs and discriminator
feature matching loss affect the training procedure of the method. In this
experiment, we compare the generator and discriminator training loss of our
proposed GAQN and GQN + GAN. As can be seen in Figure 3 (b), the training
procedure of GQN + GAN is highly unstable due to mode collapsing and
diminishing gradients. Our GAQN model eliminates both problems by using the
least-squares loss and the discriminator mean feature loss as shown in Figure
3 (a).

##  6 Conclusion

We have introduced an novel adversarial training pipeline to improve the
previously proposed GQN network architecture. Our experimental results
demonstrate that training an additional discriminator network encourages the
GQN model to predict more accurate novel views. Moreover, using the
combination of least square loss and the feature matching loss helps
stabilizing both generator and discriminator training process. In our future
work, we will explore how to generate novel views in the bigger scale of
indoor scenes where there are more objects and different lighting conditions.

## References

  * [1] Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017. pp. 214–223 (2017), http://proceedings.mlr.press/v70/arjovsky17a.html
  * [2] Bao, J., Chen, D., Wen, F., Li, H., Hua, G.: CVAE-GAN: fine-grained image generation through asymmetric training. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. pp. 2764–2773. IEEE Computer Society (2017). https://doi.org/10.1109/ICCV.2017.299, https://doi.org/10.1109/ICCV.2017.299
  * [3] Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg (2006)
  * [4] Chaurasia, G., Duchene, S., Sorkine-Hornung, O., Drettakis, G.: Depth synthesis and local warps for plausible image-based navigation. ACM Trans. Graph. 32(3), 30:1–30:12 (Jul 2013). https://doi.org/10.1145/2487228.2487238, http://doi.acm.org/10.1145/2487228.2487238
  * [5] Chen, S.E., Williams, L.: View interpolation for image synthesis. In: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques. pp. 279–288. SIGGRAPH ’93, ACM, New York, NY, USA (1993). https://doi.org/10.1145/166117.166153, http://doi.acm.org/10.1145/166117.166153
  * [6] Dosovitskiy, A., Springenberg, J.T., Brox, T.: Learning to generate chairs with convolutional neural networks. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1538–1546 (June 2015). https://doi.org/10.1109/CVPR.2015.7298761
  * [7] Eslami, S.M.A., Jimenez Rezende, D., Besse, F., Viola, F., Morcos, A.S., Garnelo, M., Ruderman, A., Rusu, A.A., Danihelka, I., Gregor, K., Reichert, D.P., Buesing, L., Weber, T., Vinyals, O., Rosenbaum, D., Rabinowitz, N., King, H., Hillier, C., Botvinick, M., Wierstra, D., Kavukcuoglu, K., Hassabis, D.: Neural scene representation and rendering. Science 360(6394), 1204–1210 (2018). https://doi.org/10.1126/science.aar6170, http://science.sciencemag.org/content/360/6394/1204
  * [8] Flynn, J., Neulander, I., Philbin, J., Snavely, N.: Deepstereo: Learning to predict new views from the world’s imagery. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [9] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. pp. 2672–2680. NIPS’14, MIT Press, Cambridge, MA, USA (2014), http://dl.acm.org/citation.cfm?id=2969033.2969125
  * [10] Gregor, K., Besse, F., Jimenez Rezende, D., Danihelka, I., Wierstra, D.: Towards conceptual compression. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 3549–3557. Curran Associates, Inc. (2016), http://papers.nips.cc/paper/6542-towards-conceptual-compression.pdf
  * [11] Gregor, K., Danihelka, I., Graves, A., Rezende, D., Wierstra, D.: Draw: A recurrent neural network for image generation. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 1462–1471. PMLR, Lille, France (07–09 Jul 2015), http://proceedings.mlr.press/v37/gregor15.html
  * [12] Habtegebrial, T., Varanasi, K., Bailer, C., Stricker, D.: Fast view synthesis with deep stereo vision. arXiv preprint arXiv:1804.09690 (2018)
  * [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pp. 770–778. IEEE Computer Society (2016). https://doi.org/10.1109/CVPR.2016.90, https://doi.org/10.1109/CVPR.2016.90
  * [14] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 6626–6637. Curran Associates, Inc. (2017), http://papers.nips.cc/paper/7240-gans-trained-by-a-two-time-scale-update-rule-converge-to-a-local-nash-equilibrium.pdf
  * [15] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)
  * [16] Jimenez Rezende, D., Eslami, S.M.A., Mohamed, S., Battaglia, P., Jaderberg, M., Heess, N.: Unsupervised learning of 3d structure from images. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 4996–5004. Curran Associates, Inc. (2016)
  * [17] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018), https://openreview.net/forum?id=Hk99zCeAb
  * [18] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [19] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  * [20] Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., Smolley, S.P.: On the effectiveness of least squares generative adversarial networks. IEEE transactions on pattern analysis and machine intelligence (2018)
  * [21] Metz, L., Poole, B., Pfau, D., Sohl-Dickstein, J.: Unrolled generative adversarial networks. arXiv preprint arXiv:1611.02163 (2016)
  * [22] Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: International Conference on Learning Representations (2018), https://openreview.net/forum?id=B1QRgziT-
  * [23] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch. In: NIPS-W (2017)
  * [24] Qi, G.J.: Loss-sensitive generative adversarial networks on lipschitz densities. arXiv preprint arXiv:1701.06264 (2017)
  * [25] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. In: Advances in neural information processing systems. pp. 2234–2242 (2016)
  * [26] Scharstein, D.: View Synthesis Using Stereo Vision. Springer-Verlag, Berlin, Heidelberg (1999)
  * [27] Seitz, S.M., Dyer, C.R.: Physically-valid view synthesis by image interpolation. In: Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV’95). pp. 18–25 (June 1995). https://doi.org/10.1109/WVRS.1995.476848
  * [28] Sun, S.H., Huh, M., Liao, Y.H., Zhang, N., Lim, J.J.: Multi-view to novel view: Synthesizing novel views with self-learned confidence. In: The European Conference on Computer Vision (ECCV) (September 2018)
  * [29] Tatarchenko, M., Dosovitskiy, A., Brox, T.: Multi-view 3d models from single images with a convolutional network. In: European Conference on Computer Vision (ECCV) (2016)
  * [30] Warde-Farley, D., Bengio, Y.: Improving generative adversarial networks with denoising feature matching. In: International Conference on Learning Representations 2017 (Conference Track) (2017), https://openreview.net/forum?id=S1X7nhsxl
  * [31] Wu, J., Zhang, C., Xue, T., Freeman, B., Tenenbaum, J.: Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 82–90. Curran Associates, Inc. (2016)
  * [32] Yan, X., Yang, J., Yumer, E., Guo, Y., Lee, H.: Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 1696–1704. Curran Associates, Inc. (2016), http://papers.nips.cc/paper/6206-perspective-transformer-nets-learning-single-view-3d-object-reconstruction-without-3d-supervision.pdf
  * [33] Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. arXiv preprint arXiv:1805.08318 (2018)
  * [34] Zhang, Y., Xu, W., Tong, Y., Zhou, K.: Online structure analysis for real-time indoor scene reconstruction. ACM Trans. Graph. 34(5), 159:1–159:13 (Nov 2015). https://doi.org/10.1145/2768821, http://doi.acm.org/10.1145/2768821
  * [35] Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: Learning view synthesis using multiplane images. In: SIGGRAPH (2018)
  * [36] Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A.: View synthesis by appearance flow. In: European conference on computer vision. pp. 286–301. Springer (2016)