# Curvature-induced repulsive effect on the lateral Casimir-Polder–van der
Waals force

Danilo T. Alves<EMAIL_ADDRESS>Faculdade de Física, Universidade Federal do
Pará, 66075-110, Belém, Pará, Brazil  Centro de Física, Universidade do Minho,
P-4710-057, Braga, Portugal  Lucas Queiroz<EMAIL_ADDRESS>Faculdade
de Física, Universidade Federal do Pará, 66075-110, Belém, Pará, Brazil  Edson
C. M. Nogueira<EMAIL_ADDRESS>Faculdade de Física,
Universidade Federal do Pará, 66075-110, Belém, Pará, Brazil  N. M. R. Peres
<EMAIL_ADDRESS>Centro de Física, Universidade do Minho, P-4710-057,
Braga, Portugal  Departamento de Física, Universidade do Minho, P-4710-057,
Braga, Portugal

(August 27, 2024)

###### Abstract

We consider a perfectly conducting infinite cylinder with radius $R$, and
investigate the Casimir-Polder (CP) and van der Waals (vdW) interactions with
a neutral polarizable particle constrained to move in a plane distant
$x_{0}>R$ from the axis of the cylinder. We show that when the relative
curvature $x_{0}/R\lesssim 6.44$, this particle, under the action of the
lateral CP force (which is the projection of the CP force onto the mentioned
plane), is attracted to the point on the plane which is closest to the
cylinder surface. On the other hand, when $x_{0}/R\gtrsim 6.44$, we also show
that, for certain particle orientations and anisotropy, the lateral CP force
can move the particle away from the cylinder. This repulsive behavior of such
a component of the CP force reveals a nontrivial dependence of the CP
interaction with the surface geometry, specifically of the relative curvature.
In the vdW regime, we show that a similar nontrivial repulsive behavior
occurs, but for the relative curvature $x_{0}/R\gtrsim 2.18$, which means that
this effect requires a smaller cylinder curvature in the vdW regime than in
the CP one. In addition, we also show that there are classical counterparts of
these effects, involving a neutral particle with a permanent electric dipole
moment. The prediction of such geometric effects on this force may be relevant
for a better controlling of the interaction between a particle and a curved
surface in classical and quantum physics.

##  I Introduction

The quantum electromagnetic dispersive force between a polarizable particle
and a macroscopic body is, generally, denominated as the Casimir-Polder (CP)
force, or as van der Waals (vdW) force in the nonretarded regime [1, 2, 3, 4,
5]. Their attractive or repulsive character can be influenced, for instance,
by the particle anisotropy and geometry of the bodies [6, 7, 8, 9, 10]. For
example, an anisotropic polarizable particle can feel a normal repulsive force
when it is put on the symmetry axis of a thin metal plate with a hole [6] or
of a perfectly conducting toroid [7]. When corrugations are considered to the
body surface, lateral CP-vdW forces appear, and nontrivial geometric effects
can be predicted, especially for these particular components of these forces
[11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 10, 21, 22, 23]. For instance, in
Ref. [21], it was shown that, when a sinusoidal corrugation with period $L$ is
introduced on an infinite flat conducting surface at $x=0$, an anisotropic
particle, kept constrained to move on a plane $x=x_{0}$ above the surface,
feels a lateral vdW force that leads it to the nearest corrugation peak when
sufficiently small values of the ratio $x_{0}/L$ are considered. Otherwise,
increasing the ratio $x_{0}/L$, the lateral vdW force can change its sign,
moving the particle away from the corrugation peak [21]. Moreover, in Ref.
[23], the collective effect of the peaks was removed by considering a flat
surface, still infinite, with only a single slight protuberance. In this case,
it was shown that a sign inversion in the lateral vdW force can still occur
when the ratio $x_{0}/l$ increases, with $l$ being the characteristic width of
the protuberance. In addition, in Ref. [23], it was observed a clear
distinction between sign inversions in the lateral vdW force originated by a
purely individual effect from those originated only in the presence of a
collective of protuberances.

In the present paper, motivated by the predictions from Refs. [21, 23], we
investigate the occurrence of a similar sign inversion in the lateral force
for a perfectly reflecting infinite cylinder, in both retarded (CP) and
nonretarded (vdW) regimes. Our initial conjecture (which will be confirmed
along this paper), is that a neutral anisotropic polarizable particle, kept
constrained to move on a given plane distant $x_{0}>R$ from the cylinder axis
(with $R$ being the cylinder radius), could feel a lateral CP-vdW force that
takes it back to the point closest to the cylinder when it is slightly
dislocated from this point [see Fig. 1(i)]. On the other hand, for
sufficiently large values of the relative curvature $x_{0}/R$, the lateral CP-
vdW force would change its sign, moving the particle away from the cylinder
[see Fig. 1(ii)].

The cylindrical geometry has been considered in problems involving dispersive
interactions [24, 25, 26, 27, 28, 29, 30, 31, 32, 33], with curvature effects
discussed, for example, in Refs. [29, 33]. In addition, the problem of how to
probe geometric effects on the quantum vacuum fluctuations, by considering the
lateral CP force, has been addressed [11, 12]. Thus, the investigation carried
out here, by predicting a curvature-induced repulsive effect on the lateral
CP-vdW force, may be relevant for a better controlling of the interaction
between a particle and a curved surface, as well as an additional way to
understand geometric effects on the quantum vacuum fluctuations.

The paper is organized as follows. In Sec. II, we investigate the CP-vdW
interaction between a perfectly reflecting infinite cylinder and a neutral
polarizable particle, kept constrained to move on a given plane near the
cylinder. We also discuss the classical interaction involving a neutral
particle with a permanent electric dipole moment. In Sec. III we discuss some
implications of our results, and present our final comments.

Figure 1:  Illustration of our initial conjecture, which is investigated (and
confirmed) in the present paper: the sign inversion in the lateral force for
an infinite cylinder of radius $R$, in both retarded and nonretarded regimes,
as the relative curvature increases. The arrows represent the lateral force
acting on a neutral anisotropic polarizable particle (elliptic figures), kept
constrained to move on a given plane $x=x_{0}$ (horizontal dotted lines). In
(i), the particle feels a lateral force $F_{\text{CP-vdW}}^{(y)}$ that takes
it back to $A$, which is the point on the plane closest to the cylinder
surface. In (ii), for an increased relative curvature $x_{0}/R$, $F_{\text{CP-
vdW}}^{(y)}$ moves the particle away from $A$, or from the cylinder. This
repulsive behavior reveals a nontrivial dependence of the CP-vdW interaction
on the cylinder relative curvature.

##  II Casimir-Polder–van der Waals interaction

Let us start by considering the retarded (CP) regime, and investigate the
quantum energy interaction $U_{\text{CP}}$ between a perfectly reflecting
infinite cylinder and a neutral polarizable particle, oriented in space in
such a way that its principal axes are parallel to the $xyz$ axes of a
Cartesian system, as illustrated in Fig. 2. In this way, an electric field
applied on the particle, along any one of these Cartesian axes, induces a
dipole moment in the same direction [34] (this is just a convenient choice
that does not imply any loss of generality for the conclusions obtained in the
present work). We also consider this particle characterized by a frequency
dependent polarizability tensor
$\overleftrightarrow{\alpha}\left(\omega\right)$, whose representation
$\boldsymbol{\alpha}^{(\text{cart})}\left(\omega\right)$ in the Cartesian
system is given by

$\boldsymbol{\alpha}^{(\text{cart})}\left(\omega\right)=\left[\begin{array}[]{ccc}\alpha_{xx}\left(\omega\right)&0&0\\\
0&\alpha_{yy}\left(\omega\right)&0\\\
0&0&\alpha_{zz}\left(\omega\right)\end{array}\right].$ (1)

The tensor $\overleftrightarrow{\alpha}\left(\omega\right)$, for a translated
particle as illustrated in Fig. 2, is represented in cylindrical coordinate
system $(\rho,\phi,z)$ by means of the matrix
$\boldsymbol{\alpha}^{(\text{cil})}\left(\omega,\phi\right)$ given by

$\boldsymbol{\alpha}^{(\text{cil})}\left(\omega,\phi\right)=\left[\begin{array}[]{ccc}\alpha_{\rho\rho}\left(\omega,\phi\right)&\alpha_{\rho\phi}\left(\omega,\phi\right)&0\\\
\alpha_{\phi\rho}\left(\omega,\phi\right)&\alpha_{\phi\phi}\left(\omega,\phi\right)&0\\\
0&0&\alpha_{zz}\left(\omega\right)\end{array}\right],$ (2)

where $\alpha_{\rho\rho}\left(\omega,\phi\right)$
$=\alpha_{yy}\left(\omega\right)\sin^{2}\phi+\alpha_{xx}\left(\omega\right)\cos^{2}\phi$,
$\alpha_{\rho\phi}\left(\omega,\phi\right)$$=\alpha_{\phi\rho}\left(\omega,\phi\right)=\left[\alpha_{yy}\left(\omega\right)-\alpha_{xx}\left(\omega\right)\right]\sin\phi\cos\phi$,
$\alpha_{\phi\phi}\left(\omega,\phi\right)$
$=\alpha_{yy}\left(\omega\right)\cos^{2}\phi+\alpha_{xx}\left(\omega\right)\sin^{2}\phi$.
Note that
$\boldsymbol{\alpha}^{(\text{cart})}\left(\omega\right)=\boldsymbol{\alpha}^{(\text{cil})}\left(\omega,0\right)$,
$\cos\phi=x/\rho$, and $\sin\phi=y/\rho$.

Figure 2: Illustration of a conducting cylinder, a neutral polarizable
particle (shown in two positions), and the Cartesian system $xyz$ ($z$ axis is
perpendicular to the paper plane). The particle is oriented in space in such a
way that its principal axes are parallel to the $xyz$ axes. In one position of
the particle, for which $\phi=0$, these principal axes coincide with the
cylindrical ones. When the particle is translated to another position for
which $\phi\neq 0$, these axes do not coincide anymore.

To investigate the behavior of $U_{\text{CP}}$, we take into account Eqs. (17)
and (27)-(29) found in Ref. [31] (the same formulas are also obtained in Ref.
[32]). Specifically, we consider in these formulas the limit $\lambda_{ji}\to
0$ ($E_{ji}\to\infty$), with $\lambda_{ji}$ and $E_{ji}$ being, respectively,
the wavelength and energy of a typical transition between the states $i$ and
$j$ of the particle [31]. The obtained formula can be written in terms of the
static polarizability tensor $\overleftrightarrow{\alpha}\left(0\right)$,
specifically of the diagonal components $\rho\rho$, $\phi\phi$, and $zz$. For
a translated particle (see Fig. 2), these components depend on $\phi$
according to Eq. (2), so that we use $\alpha_{\rho\rho}(0,\phi)$
$=\alpha_{xx}\left(0\right)x^{2}/\rho^{2}+\alpha_{yy}\left(0\right)y^{2}/\rho^{2}$,
and $\alpha_{\phi\phi}(0,\phi)$
$=\alpha_{xx}\left(0\right)y^{2}/\rho^{2}+\alpha_{yy}\left(0\right)x^{2}/\rho^{2}$
in the obtained formula. Thus, one has

$\displaystyle U_{\text{CP}}(\overline{x},\overline{y})$
$\displaystyle=-\frac{1}{\left(4\pi\right)^{2}\epsilon_{0}R^{4}}$
$\displaystyle\times\bigg{[}\Xi_{\rho}^{(\text{CP})}\left(\overline{\rho}\right)\left(\alpha_{xx}\left(0\right)\frac{\overline{x}^{2}}{\overline{\rho}^{2}}+\alpha_{yy}\left(0\right)\frac{\overline{y}^{2}}{\overline{\rho}^{2}}\right)$
$\displaystyle+\Xi_{\phi}^{(\text{CP})}\left(\overline{\rho}\right)\left(\alpha_{xx}\left(0\right)\frac{\overline{y}^{2}}{\overline{\rho}^{2}}+\alpha_{yy}\left(0\right)\frac{\overline{x}^{2}}{\overline{\rho}^{2}}\right)$
$\displaystyle+\Xi_{z}^{(\text{CP})}\left(\overline{\rho}\right)\alpha_{zz}\left(0\right)\bigg{]},$
(3)

where:

$\displaystyle\Xi_{\rho}^{(\text{CP})}\left(\overline{\rho}\right)$
$\displaystyle=2\sum_{m=0}^{\infty}{\vphantom{\sum}}^{\prime}\int_{0}^{\infty}duu\left\\{u^{2}\frac{I_{m}\left(u\right)}{K_{m}\left(u\right)}\left[K_{m}^{\prime}\left(u\overline{\rho}\right)\right]^{2}\right.$
$\displaystyle\left.-\frac{m^{2}}{\overline{\rho}^{2}}\frac{I_{m}^{\prime}\left(u\right)}{K_{m}^{\prime}\left(u\right)}\left[K_{m}\left(u\overline{\rho}\right)\right]^{2}\right\\},$
(4) $\displaystyle\Xi_{\phi}^{(\text{CP})}\left(\overline{\rho}\right)$
$\displaystyle=2\sum_{m=0}^{\infty}{\vphantom{\sum}}^{\prime}\int_{0}^{\infty}duu\left\\{-u^{2}\frac{I_{m}^{\prime}\left(u\right)}{K_{m}^{\prime}\left(u\right)}\left[K_{m}^{\prime}\left(u\overline{\rho}\right)\right]^{2}\right.$
$\displaystyle\left.+\frac{m^{2}}{\overline{\rho}^{2}}\frac{I_{m}\left(u\right)}{K_{m}\left(u\right)}\left[K_{m}\left(u\overline{\rho}\right)\right]^{2}\right\\},$
(5)
$\Xi_{z}^{(\text{CP})}\left(\overline{\rho}\right)=4\sum_{m=0}^{\infty}{\vphantom{\sum}}^{\prime}\int_{0}^{\infty}duu^{3}\frac{I_{m}\left(u\right)}{K_{m}\left(u\right)}\left[K_{m}\left(u\overline{\rho}\right)\right]^{2},$
(6)

with
$\sum_{m=0}^{\infty}{\vphantom{\sum}}^{\prime}f_{m}=\frac{1}{2}f_{0}+\sum_{m=1}^{\infty}f_{m}$,
$R$ being the cylinder radius, $\overline{x}={x}/{R}$, $\overline{y}={y}/{R}$,
$\overline{\rho}={\rho}/{R}$.

We start our analysis with the idealized case in which
$\alpha_{xx}(0)=\alpha_{zz}(0)=0$. The behavior of $U_{\text{CP}}(x,y)$ is
such that the CP force ${\bf
F}_{\text{CP}}=F_{\text{CP}}^{(x)}{\bf\hat{x}}+F_{\text{CP}}^{(y)}{\bf\hat{y}=}-\boldsymbol{\nabla}U_{\text{CP}}$
always attracts the particle to the cylinder when no constraint is imposed to
the particle. Now, let us keep the particle constrained to move on a given
plane $\overline{x}=\overline{x}_{0}>1$. From Eq. (3), we have that $\partial
U_{\text{CP}}/\partial y=0$ along the dashed and dot-dashed lines shown in
Fig. 3, with $[\partial^{2}U_{\text{CP}}(x_{0},y)/\partial y^{2}]>0$ [minimum
points of $U_{\text{CP}}(x_{0},y)$] along the dashed line, and
$[\partial^{2}U_{\text{CP}}(x_{0},y)/\partial y^{2}]<0$ [maximum points of
$U_{\text{CP}}(x_{0},y)$] along the dot-dashed one. Thus, in Fig. 3, a
particle in the dark region ($1<\overline{x}<6.44$), slightly dislocated from
a point in the plane $y=0$, feels a force $F_{\text{vdW}}^{(y)}$ that takes it
back to $y=0$. On the other hand, in the light region ($6.44<\overline{x}$),
the particle is moved away from $y=0$, or, in other words, away from the
cylinder (see Fig. 3). This sign inversion in $F_{\text{CP}}^{(y)}$, changing
from an attractive character to a repulsive one, is a nontrivial geometric
effect regulated by the relative curvature $\overline{x}$. The behavior of
${U}_{\text{CP}}(\overline{x}_{0},\overline{y})$, for some values of
$\overline{x}_{0}$, is shown in Fig. 4.

Figure 3:  Some features of the CP interaction between a perfectly reflecting
cylinder and a polarizable particle (illustrated in several positions by the
ellipsoidal figures), with $\alpha_{xx}(0)=\alpha_{zz}(0)=0$, and kept
constrained to move on a given plane $\overline{x}=\overline{x}_{0}>1$ (three
of them represented by the vertical dotted lines). Note that the cylinder
circular section and the axes $\overline{x}$ and $\overline{y}$ are
represented at a same scale. We also remark that the dashed and dot-dashed
lines are plotted taking into account Eq. (3). We have $\partial
U_{\text{CP}}/\partial y=0$ on both dashed and dot-dashed lines, with the
dashed line corresponding to minimum points of $U_{\text{CP}}(x_{0},y)$,
whereas the dot-dashed one corresponding to maximum points. In the dark region
($1<\overline{x}<6.44$), when the particle is dislocated, along the $y$-axis,
for instance, from the point $A$ to $A^{\prime}$ (or $A^{\prime\prime}$), it
feels a force $F_{\text{CP}}^{(y)}$ (represented by the arrows) which takes it
back to $A$. In the light region ($6.44<\overline{x}$), when the particle is
dislocated, for instance, from the point $B$ to $B^{\prime}$ (or
$B^{\prime\prime}$), it feels a force $F_{\text{CP}}^{(y)}$ which moves it
away from $B$, and, consequently, away from the cylinder. This sign inversion
in $F_{\text{CP}}^{(y)}$ is a nontrivial geometric effect regulated by the
relative curvature $\overline{x}=x/R$. When the particle is dislocated, along
$\overline{x}=\overline{x}_{0}$, for instance, from the point $C$ to
$C^{\prime}$ (or from $D$ to $D^{\prime}$), it feels a force
$F_{\text{CP}}^{(y)}$ that moves it back to $C$ ($D$).

Figure 4:  The behavior of $U_{\text{CP}}(\overline{x}_{0},\overline{y})$ for
some values of $\overline{x}_{0}$. In (a), $\overline{x}_{0}=6.00$. In (b),
$\overline{x}_{0}=6.44$. In (c), $\overline{x}_{0}=7.00$. In (d),
$\overline{x}_{0}=7.50$. Panels (a) and (b) correspond to the dark region in
Fig. 3, whereas (c) and (d) to the light one.

For a particle with $\alpha_{xx}(0)=\alpha_{zz}(0)=\beta\alpha_{yy}(0)$, with
$0\leq\beta<0.2$, we have a behavior similar to that shown in Fig. 3, but with
the border between the dark and light regions occurring at a value
$\overline{x}=\gamma(\beta)$. In Fig. 5, we show the configurations of $\beta$
and $\overline{x}$ for which $y=0$ is a minimum (dark region) or a maximum
(light region) point of $U_{\text{CP}}(x_{0},y)$. Thus, the dark region of
this figure represents configurations for which
$[\partial^{2}U_{\text{CP}}(x_{0},y)/\partial y^{2}]_{y=0}>0$, and the
behavior of the lateral force is similar to that described for the dark region
in Fig. 3. Besides this, the light region represents
$[\partial^{2}U_{\text{CP}}(x_{0},y)/\partial y^{2}]_{y=0}<0$, and the
behavior is similar to that described for the light region in Fig. 3. The
border between the two regions in Fig. 5 is given by the curve
$\overline{x}=\gamma(\beta)$. Note that, $\gamma(0)\approx 6.44$, which
corresponds to the situation discussed in Fig. 3. We remark that for
$\beta\gtrsim 0.2$ we just have a dark region, so that any repulsive effect of
the lateral force is suppressed.

Figure 5:  For a particle characterized by
$\alpha_{xx}(0)=\alpha_{zz}(0)=\beta\alpha_{yy}(0)$, it is shown the
configurations of $\beta$ and $\overline{x}$ for which $y=0$ is a minimum
(dark region) or a maximum (light region) point of $U_{\text{CP}}(x_{0},y)$.
The border between these two regions is given by the curve
$\overline{x}=\gamma(\beta)$. We remark that $\gamma(0)\approx 6.44$ (the
situation discussed in Fig. 3), and that above $\beta\approx 0.2$ we just have
a dark region.

It is worth to mention that these curvature-induced effects are affected by
the particle orientation in relation to the plane where it is kept constrained
to move. As an example, let us consider the particle with
$\alpha_{xx}(0)=\alpha_{zz}(0)=\beta\alpha_{yy}(0)$ ($0\leq\beta\leq 1$)
oriented as illustrated in Fig. 2, and kept constrained to move on a given
plane $y=y_{0}>1$ (see Fig. 6). In this case, we have that the point $x=0$ is
always a minimum point of $U_{\text{CP}}(x,y_{0})$, independent on the value
of $\beta$ or $\overline{y}$. This means that, when this particle is slightly
dislocated from a point in the plane $x=0$, it feels a force
$F_{\text{vdW}}^{(x)}$ that takes it back to $x=0$ in all values of
$\overline{y}$ (see Fig. 6).

Figure 6:  Illustration of some features of the vdW-CP interaction between a
perfectly reflecting cylinder and a polarizable particle (represented in two
positions by the ellipsoidal figures), kept constrained to move on a given
plane $\overline{y}=\overline{y}_{0}>1$ (represented by the dotted line). The
dashed line corresponds to minimum points of $U_{\text{CP-vdW}}(x,y_{0})$, so
that when the particle is dislocated, for instance, from the point $A$ to
$A^{\prime}$ (or $A^{\prime\prime}$), it feels a force $F_{\text{CP-
vdW}}^{(x)}$ (represented by the arrows) which takes it back to $A$.

Now, let us consider the nonretarded (vdW) regime, and investigate the quantum
energy interaction $U_{\text{vdW}}$. To this end, we take the limit
$\lambda_{ji}\to\infty$ ($E_{ji}\to 0$) in Eqs. (17) and (27)-(29) found in
Ref. [31]. After this, one obtains the formula for $U_{\text{vdW}}$ given by
Eq. (19) in Ref. [30], which is written in terms of the expectation values
$\langle\hat{d}_{\rho}^{2}\rangle$, $\langle\hat{d}_{\phi}^{2}\rangle$, and
$\langle\hat{d}_{z}^{2}\rangle$, where $\hat{d}_{s}$ are components of the
dipole moment operator. For a translated particle (see Fig. 2), the components
$\alpha_{\rho\rho}$ and $\alpha_{\phi\phi}$ depend on $\phi$ according to Eq.
(2), so that we write
$\langle\hat{d}_{\rho}^{2}\rangle(\phi)=\frac{\hbar}{\pi}\int_{0}^{\infty}d\xi\,\alpha_{\rho\rho}\left(i\xi,\phi\right)$
$=\langle\hat{d}_{x}^{2}\rangle x^{2}/\rho^{2}+\langle\hat{d}_{y}^{2}\rangle
y^{2}/\rho^{2}$, and
$\langle\hat{d}_{\phi}^{2}\rangle(\phi)=\frac{\hbar}{\pi}\int_{0}^{\infty}d\xi\,\alpha_{\phi\phi}\left(i\xi,\phi\right)$
$=\langle\hat{d}_{x}^{2}\rangle y^{2}/\rho^{2}+\langle\hat{d}_{y}^{2}\rangle
x^{2}/\rho^{2}$. Thus, one obtains

$\displaystyle U_{\text{vdW}}(\overline{x},\overline{y})$
$\displaystyle=-\frac{1}{4\pi\epsilon_{0}R^{3}}$
$\displaystyle\times\bigg{[}\Xi_{\rho}^{(\text{vdW})}\left(\overline{\rho}\right)\left(\langle\hat{d}_{x}^{2}\rangle\frac{\overline{x}^{2}}{\overline{\rho}^{2}}+\langle\hat{d}_{y}^{2}\rangle\frac{\overline{y}^{2}}{\overline{\rho}^{2}}\right)$
$\displaystyle+\Xi_{\phi}^{(\text{vdW})}\left(\overline{\rho}\right)\left(\langle\hat{d}_{x}^{2}\rangle\frac{\overline{y}^{2}}{\overline{\rho}^{2}}+\langle\hat{d}_{y}^{2}\rangle\frac{\overline{x}^{2}}{\overline{\rho}^{2}}\right)$
$\displaystyle+\Xi_{z}^{(\text{vdW})}\left(\overline{\rho}\right)\langle\hat{d}_{z}^{2}\rangle\bigg{]},$
(7)

where:

$\Xi_{\rho}^{(\text{vdW})}\left(\overline{\rho}\right)=\frac{2}{\pi}\sum_{m=0}^{\infty}{\vphantom{\sum}}^{\prime}\int_{0}^{\infty}duu^{2}\frac{I_{m}\left(u\right)}{K_{m}\left(u\right)}\left[K_{m}^{\prime}\left(u\overline{\rho}\right)\right]^{2},$
(8)
$\Xi_{\phi}^{(\text{vdW})}\left(\overline{\rho}\right)=\frac{2}{\pi}\sum_{m=1}^{\infty}\frac{m^{2}}{\overline{\rho}^{2}}\int_{0}^{\infty}du\frac{I_{m}\left(u\right)}{K_{m}\left(u\right)}\left[K_{m}\left(u\overline{\rho}\right)\right]^{2},$
(9)
$\Xi_{z}^{(\text{vdW})}\left(\overline{\rho}\right)=\frac{2}{\pi}\sum_{m=0}^{\infty}{\vphantom{\sum}}^{\prime}\int_{0}^{\infty}duu^{2}\frac{I_{m}\left(u\right)}{K_{m}\left(u\right)}\left[K_{m}\left(u\overline{\rho}\right)\right]^{2},$
(10)

with
$\langle\hat{d}_{j}^{2}\rangle=\frac{\hbar}{\pi}\int_{0}^{\infty}d\xi\,\alpha_{jj}\left(i\xi\right)$,
for $j=x,y,z$.

For the vdW case, we carry out an analysis similarly to that done for the CP
case. We start, again, focusing on the idealized case of a particle
characterized by
$\langle\hat{d}_{x}^{2}\rangle=\langle\hat{d}_{z}^{2}\rangle=0$. For a
particle kept constrained to move on a given plane
$\overline{x}=\overline{x}_{0}>1$, from Eq. (7), we have that the dashed and
dot-dashed lines shown in Fig. 7 represent the minimum and maximum points of
$U_{\text{vdW}}(x_{0},y)$, respectively. In this figure, the division between
the dark and light regions occurs at $\overline{x}\approx 2.18$. The behavior
of ${U}_{\text{vdW}}(\overline{x}_{0},\overline{y})$, for some values of
$\overline{x}_{0}$, is shown in Fig. 8. For a particle with
$\langle\hat{d}_{x}^{2}\rangle=\langle\hat{d}_{z}^{2}\rangle=\beta\langle\hat{d}_{y}^{2}\rangle$,
we show in Fig. 9 configurations of $\beta$ and $\overline{x}$ for which $y=0$
is a minimum (dark region) or a maximum (light region) point of
$U_{\text{vdW}}(x_{0},y)$. For $\beta\gtrsim 0.32$ we just have a dark region,
so that any repulsive effect of the lateral force is suppressed. For
$0\leq\beta\leq 0.32$, we can have dark and light regions, depending on
$\overline{x}$ and $\beta$. The border between these two regions is given by
the curve $\overline{x}=\gamma(\beta)$. In this figure, note that
$\gamma(0)\approx 2.18$, which corresponds to the situation discussed in Fig.
7. In summary, for a particle with $\beta\gtrsim 0.32$ the behavior
$F_{\text{vdW}}^{(y)}$ is given by the dark region in Fig. 7 for all
$\overline{x}$, and only particles with $\beta\lesssim 0.32$ can exhibit the
repulsive behavior of $F_{\text{vdW}}^{(y)}$ shown by the light region in Fig.
7. Lastly, these curvature-induced effects are affected by the particle
orientation in relation to the plane where it is kept constrained to move,
similar to the CP case, so that if we consider a particle oriented as
illustrated in Fig. 6, we also have a force $F_{\text{vdW}}^{(x)}$ that always
take the particle to $x=0$, regardless of the value of $\overline{y}$ (see
Fig. 6).

Figure 7:  Some features of the vdW interaction between a perfectly reflecting
cylinder and a polarizable particle (represented in several positions by the
ellipsoidal figures), with
$\langle\hat{d}_{x}^{2}\rangle=\langle\hat{d}_{z}^{2}\rangle=0$, and kept
constrained to move on a given plane $\overline{x}=\overline{x}_{0}>1$ (three
of them represented by the vertical dotted lines). Note that the cylinder
circular section and the axes $\overline{x}$ and $\overline{y}$ are
represented at a same scale. We also remark that the dashed and dot-dashed
lines are plotted taking into account Eq. (7). We have $\partial
U_{\text{vdW}}/\partial y=0$ on both dashed and dot-dashed lines, with the
dashed line corresponding to minimum points of $U_{\text{vdW}}(x_{0},y)$,
whereas the dot-dashed one corresponding to maximum points. In dark region
($1<\overline{x}<2.18$), when the particle is dislocated, along the $y$-axis,
for instance, from the point $A$ to $A^{\prime}$ (or $A^{\prime\prime}$), it
feels a force $F_{\text{vdW}}^{(y)}$ (represented by the arrows) which takes
it back to $A$. In light region ($2.18<\overline{x}$), when the particle is
dislocated, for instance, from the point $B$ to $B^{\prime}$ (or
$B^{\prime\prime}$), it feels a force $F_{\text{vdW}}^{(y)}$ which moves it
away from $B$, and, consequently, away from the cylinder. This sign inversion
in $F_{\text{vdW}}^{(y)}$ is a nontrivial geometric effect regulated by the
relative curvature $\overline{x}$. When the particle is dislocated, along
$\overline{x}=\overline{x}_{0}$, for instance, from the point $C$ to
$C^{\prime}$ (or from $D$ to $D^{\prime}$), it feels a force
$F_{\text{vdW}}^{(y)}$ that moves it back to $C$ ($D$).

Figure 8:  The behavior of $U_{\text{vdW}}(\overline{x}_{0},\overline{y})$ for
some values of $\overline{x}_{0}$. In (a), $\overline{x}_{0}=1.50$. In (b),
$\overline{x}_{0}=2.18$. In (c), $\overline{x}_{0}=2.50$. In (d),
$\overline{x}_{0}=3.00$. Panels (a) and (b) correspond to the dark region in
Fig. 7, whereas (c) and (d) to the light one.  Figure 9:  For a particle
characterized by
$\langle\hat{d}_{x}^{2}\rangle=\langle\hat{d}_{z}^{2}\rangle=\beta\langle\hat{d}_{y}^{2}\rangle$,
it is shown the configurations of $\beta$ and $\overline{x}$ for which $y=0$
is a minimum (dark region) or a maximum (light region) point of
$U_{\text{vdW}}(x_{0},y)$. The border between these two regions is given by
the curve $\overline{x}=\gamma(\beta)$. We remark that $\gamma(0)\approx 2.18$
(the situation discussed in Fig. 7), and that above $\beta\approx 0.32$ we
just have a dark region.

It is worth to mention that these curvature-induced effects have a classical
counterpart. The interaction energy $U_{\text{cla}}$, between a perfectly
reflecting conducting infinite cylinder and a neutral point particle with a
dipole moment vector $\textbf{d}=d_{x}\hat{{\bf x}}+d_{y}\hat{{\bf
y}}+d_{z}\hat{{\bf z}}$, is given by replacing $\langle d_{i}^{2}\rangle\to
d_{i}^{2}$ in Eq. (7) [30]. In this way, for the case of a particle with
$d_{x}=d_{z}=0$, we have that the dashed and dot-dashed lines shown in Fig. 7
also represent the minimum and maximum points of $U_{\text{cla}}(x_{0},y)$,
respectively.

Figure 10:  Illustration of a particle kept constrained to move on a given
plane $\overline{x}=\overline{x}_{0}>1$, and subjected to a trap harmonic
potential $U_{\text{trap}}(y)$ (illustrated by the dot-dashed lines) with
equilibrium point at $y=0$, near a perfectly reflecting cylinder. The dashed
lines represent $U_{\text{CP-vdW}}(x_{0},y)$. The presence of the cylinder
modifies this oscillation frequency to a new value
$\omega_{\text{trap}}^{\prime}$, resulting in a frequency deviation
$\delta\omega_{\text{trap}}\equiv\omega_{\text{trap}}^{\prime}-\omega_{\text{trap}}$.
In dark region [$1<\overline{x}<\gamma(\beta)$], an experimental apparatus
would detect $\delta\omega_{\text{trap}}>0$. Otherwise, in light region
[$\gamma(\beta)<\overline{x}$], it would detect
$\delta\omega_{\text{trap}}<0$.

##  III Discussions and Final Remarks

The CP-vdW total force ${\bf F}_{\text{CP-
vdW}}=-\boldsymbol{\nabla}U_{\text{CP-vdW}}$ always attracts the particle to a
point closer to the cylinder, so that, focusing on ${\bf F}_{\text{CP-vdW}}$,
the curvature effect may not be directly evident. This effect is a very subtle
one, in the sense that it is easier to be perceived when considering the
particle kept constrained to move on a given plane near the cylinder. When the
relative curvature becomes sufficiently high, for certain particle
orientations and anisotropy, the projected CP-vdW force onto the mentioned
plane changes its sign and, instead of moving the particle to the point on the
plane closest to the cylinder surface, moves it away. We considered the
existence of such curvature-induced sign inversion effect as our initial
research hypothesis, guided by the sign inversions in the lateral force
predicted in previous works [21, 23], but, differently of the models discussed
in Refs. [21, 23], where the infinite extent of the surface did not allow a
direct characterization of the mentioned sign inversion as a change from an
attractive to a repulsive effect of the lateral force, for the cylinder this
characterization can be done directly.

We showed that the repulsive effect on the lateral force requires a larger
cylinder curvature in the retarded (CP) regime than in the nonretarded (vdW)
one: the minimum relative curvature required in the CP regime is $6.44$ [see
Fig. 5], whereas in vdW is $2.18$ [see Fig. 9].

The detection of such geometric effects on the lateral CP-vdW force could be
done, for example, by trapping an anisotropic particle, and measuring the
deviation in the original trap frequency (see, for instance, Refs. [35, 11,
21, 23]). Let us consider a particle with mass $m$ and
$\alpha_{xx}(0)=\alpha_{zz}(0)=\beta\alpha_{yy}(0)$, with $\beta<0.2$, and the
CP regime [or
$\langle\hat{d}_{x}^{2}\rangle=\langle\hat{d}_{z}^{2}\rangle=\beta\langle\hat{d}_{y}^{2}\rangle$,
with $\beta<0.32$, and the vdW regime], kept constrained to move on a given
plane $\overline{x}=\overline{x}_{0}>1$, and subjected to a trap harmonic
potential $U_{\text{trap}}(y)$ with equilibrium point at $y=0$ (see Fig. 10),
and oscillating with a certain frequency $\omega_{\text{trap}}$ in the absence
of the cylinder. The presence of the cylinder modifies this oscillation
frequency to a new value
$\omega_{\text{trap}}^{\prime}=\sqrt{\omega_{\text{trap}}^{2}+m^{-1}\left[\partial^{2}U_{\text{CP-
vdW}}(x_{0},y)/\partial y^{2}\right]}$, resulting in a frequency deviation
$\delta\omega_{\text{trap}}\equiv\omega_{\text{trap}}^{\prime}-\omega_{\text{trap}}$.
When $1<\overline{x}_{0}<\gamma(\beta)$ [with $\gamma(\beta)$ given in Figs. 5
and 9 for the CP and vdW cases, respectively], an experimental apparatus would
detect $\delta\omega_{\text{trap}}>0$ (see dark region in Fig. 10). Otherwise,
when $\gamma(\beta)<\overline{x}_{0}$, one would have
$\delta\omega_{\text{trap}}<0$ (see light region in Fig. 10). Therefore, the
sign inversion in the lateral force, changing from an attractive behavior to a
repulsive one (as illustrated in Fig. 10), manifests, in this scenario, as a
sign inversion in the frequency deviation $\delta\omega_{\text{trap}}$, with
this behavior revealing a nontrivial dependence of the CP-vdW interaction with
the surface geometry, specifically of the relative curvature.

###### Acknowledgements.

L.Q. and E.C.M.N. were supported by the Coordenação de Aperfeiçoamento de
Pessoal de Nível Superior - Brasil (CAPES), Finance Code 001.

## References

  * R. Eisenschitz and F. London [1930] R. Eisenschitz and F. London, Über das Verhältns der van der Waalsschen Kräfte zu den homöopolaren Bindungskräften, Zeitschrift für Physik 60, 491 (1930).
  * London [1930] F. London, Zur Theorie und Systematik der Molekularkräfte, Zeitschrift für Physik 63, 245 (1930).
  * Casimir and Polder [1946] H. B. G. Casimir and D. Polder, Influence of Retardation on the London–van der Waals Forces, Nature 158, 787 (1946).
  * Casimir and Polder [1948] H. B. G. Casimir and D. Polder, The Influence of Retardation on the London-van der Waals Forces, Phys. Rev. 73, 360 (1948).
  * Feinberg and Sucher [1970] G. Feinberg and J. Sucher, General theory of the van der Waals interaction: A model-independent approach, Phys. Rev. A 2, 2395 (1970).
  * Levin  _et al._ [2010] M. Levin, A. P. McCauley, A. W. Rodriguez, M. T. H. Reid, and S. G. Johnson, Casimir repulsion between metallic objects in vacuum, Phys. Rev. Lett. 105, 090403 (2010).
  * Abrantes  _et al._ [2018] P. P. Abrantes, Y. França, F. S. S. da Rosa, C. Farina, and R. de Melo e Souza, Repulsive van der Waals interaction between a quantum particle and a conducting toroid, Physical Review A 98, 012511 (2018).
  * Eberlein and Zietal [2011] C. Eberlein and R. Zietal, Casimir-Polder interaction between a polarizable particle and a plate with a hole, Physical Review A 83, 052514 (2011).
  * Marchetta  _et al._ [2021] J. J. Marchetta, P. Parashar, and K. V. Shajesh, Geometrical dependence in Casimir-Polder repulsion, Physical Review A 104, 032209 (2021).
  * Buhmann  _et al._ [2016] S. Y. Buhmann, V. N. Marachevsky, and S. Scheel, Impact of anisotropy on the interaction of an atom with a one-dimensional nano-grating, Int. J. Mod. Phys. A 31, 1641029 (2016).
  * Dalvit  _et al._ [2008a] D. A. R. Dalvit, P. A. M. Neto, A. Lambrecht, and S. Reynaud, Probing Quantum-Vacuum Geometrical Effects with Cold Atoms, Phys. Rev. Lett. 100, 040405 (2008a).
  * Dalvit  _et al._ [2008b] D. A. R. Dalvit, P. A. M. Neto, A. Lambrecht, and S. Reynaud, Lateral Casimir-Polder force with corrugated surfaces, Journal of Physics A: Mathematical and Theoretical 41, 164028 (2008b).
  * Döbrich  _et al._ [2008] B. Döbrich, M. DeKieviet, and H. Gies, Scalar Casimir-Polder forces for uniaxial corrugations, Phys. Rev. D 78, 125022 (2008).
  * Messina  _et al._ [2009] R. Messina, D. A. R. Dalvit, P. A. M. Neto, A. Lambrecht, and S. Reynaud, Dispersive interactions between atoms and nonplanar surfaces, Physical Review A 80, 022119 (2009).
  * Moreno  _et al._ [2010a] G. A. Moreno, D. A. R. Dalvit, and E. Calzetta, Bragg spectroscopy for measuring Casimir–Polder interactions with Bose–Einstein condensates above corrugated surfaces, New Journal of Physics 12, 033009 (2010a).
  * Contreras-Reyes  _et al._ [2010] A. M. Contreras-Reyes, R. Guérout, P. A. M. Neto, D. A. R. Dalvit, A. Lambrecht, and S. Reynaud, Casimir-Polder interaction between an atom and a dielectric grating, Physical Review A 82, 052517 (2010).
  * Moreno  _et al._ [2010b] G. A. Moreno, R. Messina, D. A. R. Dalvit, A. Lambrecht, P. A. Maia Neto, and S. Reynaud, Disorder in Quantum Vacuum: Casimir-Induced Localization of Matter Waves, Physical Review Letters 105, 210401 (2010b).
  * Bimonte  _et al._ [2014] G. Bimonte, T. Emig, and M. Kardar, Casimir-Polder interaction for gently curved surfaces, Physical Review D 90, 081702 (2014).
  * Marachevsky [2015] V. Marachevsky, Fluctuation potential of the interaction between a neutral atom and a diffraction grating., Theoretical & Mathematical Physics 185 (2015).
  * Bennett [2015] R. Bennett, Spontaneous decay rate and Casimir-Polder potential of an atom near a lithographed surface, Phys. Rev. A 92, 022503 (2015).
  * Nogueira  _et al._ [2021] E. C. M. Nogueira, L. Queiroz, and D. T. Alves, Peak, valley, and intermediate regimes in the lateral van der Waals force, Physical Review A 104, 012816 (2021).
  * Queiroz  _et al._ [2021] L. Queiroz, E. C. M. Nogueira, and D. T. Alves, Regimes of the lateral van der Waals force in the presence of dielectrics, Physical Review A 104, 062802 (2021).
  * Nogueira  _et al._ [2022] E. C. M. Nogueira, L. Queiroz, and D. T. Alves, Sign inversion in the lateral van der Waals force, Phys. Rev. A 105, 062816 (2022).
  * Rosenfeld and Wasan [1974] J. Rosenfeld and D. Wasan, The London force contribution to the van der Waals force between a sphere and a cylinder, Journal of Colloid and Interface Science 47, 27 (1974).
  * Gu and Li [1999] Y. Gu and D. Li, The van der Waals interaction between a spherical particle and a cylinder, Journal of Colloid and Interface Science 217, 60 (1999).
  * Kirsch [2003] V. Kirsch, Calculation of the van der Waals force between a spherical particle and an infinite cylinder, Advances in Colloid and Interface Science 104, 311 (2003).
  * Blagov  _et al._ [2005] E. V. Blagov, G. L. Klimchitskaya, and V. M. Mostepanenko, Van der waals interaction between microparticle and uniaxial crystal with application to hydrogen atoms and multiwall carbon nanotubes, Phys. Rev. B 71, 235401 (2005).
  * Bordag [2006] M. Bordag, Casimir effect for a sphere and a cylinder in front of a plane and corrections to the proximity force theorem, Phys. Rev. D 73, 125018 (2006).
  * Gies and Klingmüller [2006] H. Gies and K. Klingmüller, Casimir effect for curved geometries: Proximity-force-approximation validity limits, Phys. Rev. Lett. 96, 220401 (2006).
  * Eberlein and Zietal [2007] C. Eberlein and R. Zietal, Force on a neutral atom near conducting microstructures, Phys. Rev. A 75, 032516 (2007).
  * Eberlein and Zietal [2009] C. Eberlein and R. Zietal, Retarded Casimir-Polder force on an atom near reflecting microstructures, Phys. Rev. A 80, 012504 (2009).
  * Bezerra  _et al._ [2011] V. B. Bezerra, E. R. Bezerra de Mello, G. L. Klimchitskaya, V. M. Mostepanenko, and A. A. Saharian, Exact Casimir–Polder potential between a particle and an ideal metal cylindrical shell and the proximity force approximation, The European Physical Journal C 71, 1614 (2011).
  * Valchev  _et al._ [2017] G. Valchev, P. Djondjorov, V. Vassilev, and D. Dantchev, Van der Waals interactions between planar substrate and tubular lipid membranes undergoing pearling instability, in  _AIP Conference Proceedings_ , Vol. 1895 (AIP Publishing LLC, 2017) p. 090002.
  * Feynman  _et al._ [2006] R. Feynman, R. Leighton, and M. Sands,  _The Feynman Lectures on Physics_, Vol. II (Pearson/Addison-Wesley, 2006).
  * Buhmann [2012] S. Y. Buhmann,  _Dispersion Forces I_, Springer Tracts in Modern Physics, Vol. 247 (Springer Berlin Heidelberg, Berlin, Heidelberg, 2012).


# Robust Alignment of Multi-Exposed Images with Saturated Regions

Jun Jiang  Key Laboratory of Metallurgical Equipment and Control Technology,
Wuhan University of Science and Technology, Wuhan, China  Zhengguo Li
Institute for Infocomm Research, Singapore  Shoulie Xie  Institute for
Infocomm Research, Singapore  Shiqian Wu  Institute of Robotics and
Intelligent Systems, School of Information Science and Engineering, Wuhan
University of Science and Technology, Wuhan, China  Liangcai Zeng
Corresponding<EMAIL_ADDRESS>Key Laboratory of Metallurgical
Equipment and Control Technology, Wuhan University of Science and Technology,
Wuhan, China

###### Abstract

It is challenging to align multi-exposed images due to large illumination
variations, especially in presence of saturated regions. In this paper, a
novel image alignment algorithm is proposed to cope with the multi-exposed
images with saturated regions. Specifically, the multi-exposed images are
first normalized by using intensity mapping functions (IMFs) in consideration
of saturated pixels. Then, the normalized images are coded by using the local
binary pattern (LBP). Finally, the coded images are aligned by formulating an
optimization problem by using a differentiable Hamming distance. Experimental
results show that the proposed algorithm outperforms state-of-the-art
alignment methods for multi-exposed images in terms of alignment accuracy and
robustness to exposure values.

##  1 Introduction

Image alignment is an important process in many image processing applications
[1], visual odometry and SLAM [2], and panorama imaging [3]. A robust image
alignment algorithm should cope with illumination variations and large motion
variations while giving a sub-pixel accurate alignment. Existing image
alignment approaches can be classified into two categories [1]: intensity-
based methods and feature-based methods. The intensity-based methods find
motion parameters directly from image intensities on brightness constancy
assumption. In contrast, the feature-based methods, working with image
features (such as SIFT-based features), are usually used to deal with
illumination-varying images.

Since the brightness constancy assumption is not true for differently exposed
images [4], the feature-based methods are widely applied to align differently
exposed images. Ordering features are usually used to represent invariant
property of illumination-variation images. Median threshold bitmap (MTB) [5]
is a global descriptor for the representation of illumination-variation
images. But the alignment is not accurate due to errors from unequal division
on severely saturated regions and ordering problems for many pixels with the
same intensity [6]. Local descriptors, such as census transform (CT) [7],
local binary pattern (LBP) [8], BRIEF [9], ORB [10] and so on, have been
widely used to describe illumination-robust features. These descriptors depend
solely on comparison between a specific pixel and its neighborhood (or a pair
of pixels in its neighborhood), and are therefore invariant under intensity
changes.

Several learning-based binary descriptors have been proposed to project a
local patch to a binary descriptor. Semantic hashing [11] trains a multi-layer
neural network to learn compact binary codes. Spectral hashing (SpeH) [12]
generates efficient binary codes by spectral graph partitioning. BinBoost
descriptor [13] applies boosting to learn a set of binary hash functions that
achieve an accurate binary descriptor. The work in [14] presented a binary
online learned descriptor by applying LDA criterion. DeepBit [15] generates a
compact binary descriptor by designing a CNN in an unsupervised manner. The
work in [16] proposed a compact binary face descriptor (CBFD) to learn evenly-
distributive and energy-saving local binary codes. Duan et al. [17] presented
a context-aware local binary feature learning (CA-LBFL) approach using the
contextual information of adjacent bits. A supervised structured binary code
(SUBIC) with a one-hot block structure was proposed in [18]. Those methods
utilize the rigid sign function or a hand-crafted threshold for binarization,
which is easily affected by exposure change and leads to unequal binarization
[19]. Normally, the learning-based matching pipeline consists of three sub-
components [20]: keypoints detection, orientation estimation and features
extraction. Each subcomponent is trained individually. However, the overall
performance may not be optimal when integrating these separately optimized
subcomponents into a matching pipeline [21]. Therefore, end-to-end trainable
matching networks [22, 23, 24] have been proposed to optimize the matching
performance. But, they require hand-crafted detectors or other functions to
initialize the training process [23, 24]. More importantly, the learning-based
methods are very data-dependent and require a large amount of data to train
the network repeatedly, which is complex, time-consuming and not suitable for
using on mobile devices. In contrast, the traditional binary descriptors is a
better and simpler choice to represent the differently exposed images.

For binary descriptors [9, 10, 14], Hamming distance was adopted to measure
the similarity of two pixels, and can be computed with a bitwise XOR operation
followed by a bit count. However, the Hamming distance is non-differentiable
and unsuitable for optimization. Wu et al. [6] used the decimal representation
to represent the LBP coded images and matched the difference between two coded
images under the squared Euclidean norm. However, the decimal representation
does not capture the closeness of two bit-strings and it is sensitive to the
rotation. It is thus desired to have a differentiable distance which is
equivalent to the Hamming distance for aligning of differently exposed images.
Such a distance function was recently introduced in [26]. Besides the issues
on the brightness constancy assumption and the distance, one more challenging
issue for aligning multi-exposed images is possibly saturated regions in the
images [26]. Due to the saturated regions, one feature detected in one image
may not occur on another; and a specific intensity in one image may map to
multiple intensities in the other image, and vice versa. It is thus desired to
develop a new image alignment algorithm for multi-exposed images with
saturated regions.

In this paper, a novel alignment algorithm is proposed for multi-exposed
images with saturated regions by using the differentiable “Hamming” distance
function to binary descriptors in [26]. Specifically, a novel intensity
mapping function (IMF) based method is first proposed to normalize the multi-
exposed images so that the saturated regions are same when they are aligned.
Then the normalized images are coded by the LBP. The difference between two
coded images is measured by the differentiable “Hamming” distance in [26]
which is equivalent to the Hamming distance on top of the bitwise XOR. Same as
[6], the alignment of two differently exposed images is formulated as a
quadratic optimization which can be easily solved. Since the differentiable
“Hamming” distance captures the closeness of two bit-strings better than the
decimal representation in [6], the proposed algorithm is more robust than the
algorithm in [6]. Experimental results demonstrate that the proposed algorithm
outperforms state-of-the-art alignment algorithms for multi-exposed images.
Two major contributions of this paper are summarized as follows: 1) A new
normalization method for differently exposed images is proposed which is
effective to reduce the effect of saturation to image alignment; and 2) A
simple image alignment algorithm for differently exposed images is presented
which is robust to the exposure values (EV).

The rest of this paper is organized as follows. Multi-exposed images are
normalized in section 2 and aligned in section 3. Experimental results are
provided in section 4 to validate the proposed algorithm. Finally, conclusions
are provided in section 5.

##  2 Normalization of Multi-Exposed Images

In this section, a novel normalization method is first introduced for multi-
exposed images. Unlike the method in [28], the exposure times of the images
are not required by the proposed method.

Consider two images $Z_{1}$ and $Z_{2}$ of an identical scene with two
different exposures $\Delta t_{1}$ and $\Delta t_{2}$. $Z_{1}(p)(Z_{2}(p))$,
$Z_{1}(p^{\prime})(Z_{2}(p^{\prime}))$ and $E(p)(E(p^{\prime}))$ are two
intensities and radiances respectively at pixel positions $p$ and
$p^{\prime}$. There are two possible issues for two differently exposed
images. One is that their intensity values are different for two co-located
pixels. Fortunately, the following relationship is generally held due to the
monotonic property:

$E(p)>E(p^{\prime})\Rightarrow
Z_{1}(p)>Z_{1}(p^{\prime}),\mbox{~{}and~{}}Z_{2}(p)>Z_{2}(p^{\prime}).$ (1)

Clearly, exposures change the intensities, but keep the relative order of
intensities if they are not saturated. Thus, the LBP code could be adopted to
represent two differently exposed images [6].

One possible issue for multi-exposed images is the saturation, i.e., part of
an image could be over/under-exposed. The saturation is a very challenging
issue for alignment [26]. Assume that $\Delta t_{1}$ is larger than $\Delta
t_{2}$. Let $\alpha$ and $\beta$ be two predefined constants, and their values
are selected as 5 and 254, respectively. For simplicity, a pixel $Z(p)$ is
regarded as over-exposed if its value is larger than $\beta$, and under-
exposed if its value is smaller than $\alpha$. Here, a novel method is
introduced to synchronize the under-exposed regions of the images
$\hat{Z}_{1}$ and $\hat{Z}_{2}$ as well as the over-exposed regions of the
images $\hat{Z}_{1}$ and $\hat{Z}_{2}$ by using the IMFs between the images
$Z_{1}$ and $Z_{2}$.

Let $f_{12}(z)$ and $f_{21}(z)$ be the IMFs from image $Z_{1}$ to image
$Z_{2}$ and vice versa, respectively. By finding the matched points between
the cumulative histograms of $Z_{1}$ and $Z_{2}$, the IMF $f_{12}(z)$ which
maps intensity from $Z_{1}$ to $Z_{2}$ and the inverse IMF $f_{21}(z)$ can be
obtained [25].

Figure 1:  Normalization of 1st and 6th images in “BigTree”. (a) Original
images. The right image is rotated by 5∘, and shifted by 30 pixels and 10
pixels in y-axis and x-axis respectively. (b) IMFs of the two images. (c)
Normalized images using a unidirectional mapping function[6]. It maps the
image with more information to the one with less information. (d) Normalized
images using the proposed bi-directional mapping function. The intensities
larger than $\zeta_{1}$ in the left image are unchanged, the remaining
intensities in right image are mapped to left image to ensure the over-
exposure regions in both images are consistent. The intensities less than
$\zeta_{2}$ in the right image remain constant, the remaining intensities in
the left image are mapped to right image to ensure the under-exposure regions
in both images are consistent.

It was pointed out in [29] that the IMFs $f_{12}(z)$ and $f_{21}(z)$ have the
following properties: 1) The IMF $f_{12}(Z_{1}(p))$ is not accurate if
$Z_{1}(p)$ is over-exposed; and 2) The IMF $f_{21}(Z_{2}(p))$ is not accurate
if $Z_{2}(p)$ is under-exposed. On top of the two observations, two constants
are defined as

$\left\\{\begin{array}[]{l}\zeta_{1}=\max_{z_{1}}\\{z_{1}|f_{12}(z_{1})=\alpha\\}\\\
\zeta_{2}=\min_{z_{2}}\\{z_{2}|f_{21}(z_{2})=\beta\\}\end{array}\right.,$ (2)

$\zeta_{1}$ and $\zeta_{2}$ are further adjusted as

$\displaystyle\left\\{\begin{array}[]{l}\zeta_{1}=\min\\{\zeta_{1},\beta\\}\\\
\zeta_{2}=\max\\{\zeta_{2},\alpha\\}\end{array}\right..$ (5)

The image $Z_{1}$ is mapped as

$\hat{Z}_{1}(p)=\left\\{\begin{array}[]{ll}Z_{1}(p);&\mbox{if~{}}Z_{1}(p)\geq\zeta_{1}\\\
f_{12}(Z_{1}(p));&\mbox{otherwise}\end{array}\right.,$ (6)

and the image $Z_{2}$ is mapped as

$\hat{Z}_{2}(p)=\left\\{\begin{array}[]{ll}Z_{2}(p);&\mbox{if~{}}Z_{2}(p)\leq\zeta_{2}\\\
f_{21}(Z_{2}(p));&\mbox{otherwise}\end{array}\right..$ (7)

It can be easily shown that

$\displaystyle\left\\{\begin{array}[]{l}\hat{Z}_{1}(p)\in[0,\alpha]\cup[\zeta_{1},255]\\\
\hat{Z}_{2}(p)\in[0,\zeta_{2}]\cup[\beta,255]\end{array}\right..$ (10)

Clearly, the under-exposed regions of $\hat{Z}_{1}$ and $\hat{Z}_{2}$ as well
as the over-exposed regions of $\hat{Z}_{1}$ and $\hat{Z}_{2}$ are the same if
they are aligned. The translational vector and the rotational angle are
estimated by using the mapped images $\hat{Z}_{1}$ and $\hat{Z}_{2}$.

As the functions $f_{12}(\cdot)$ and $f_{21}(\cdot)$ are monotonically non-
decreasing functions, and

$\left\\{\begin{array}[]{l}f_{12}(Z_{1}(p))\leq Z_{1}(p)\\\
f_{21}(Z_{2}(p))\geq Z_{2}(p)\end{array}\right..$ (11)

it can be easily verified that the LBP codes of the mapped images
$\hat{Z}_{1}$ and $\hat{Z}_{2}$ are almost the same as those of the original
images $Z_{1}$ and $Z_{2}$. Clearly, both the IMF $f_{12}(\cdot)$ and the IMF
$f_{21}(\cdot)$ are applied to improve the robustness of LBP codes with
respect to possible saturation in the proposed algorithm. The IMFs of images 1
and 6 in “BigTree” sequence is plotted in Fig.1(a). As shown in Fig.1(c), the
IMFs result in inconsistence in the two normalized images, such as trees in
bright area, and tree trunk in dark area. This is because the region circled
by the two ellipses in Fig.1(b) is not reliable due to under/over exposure and
multi-value mapping. Thus, the parameters $\alpha$ and $\beta$ are introduced
to reduce their effects, and improve the similarity of the two normalized
images, as shown in Fig.1(d).

##  3 Alignment of Multi-Exposed Images

In this section, a new alignment algorithm is proposed for multi-exposed
images by using a differentiable “Hamming” distance [26].

Let the luminance components of the two normalized images $\hat{Z}_{1}$ and
$\hat{Z}_{2}$ be denoted as $Y_{1}$ and $Y_{2}$, respectively. In order to
reduce the sensitivity of the descriptor to noise, the images $Y_{1}$ and
$Y_{2}$ can be smoothed with a Gaussian filter in a $3\times 3$ neighborhood
($\sigma=0.5$) [26] or an edge-preserving smoothing filter such as the
weighted guided image filter (WGIF) in a $3\times 3$ neighborhood
($\lambda=1/256$) [27]. For the pixel $p$, $\chi(j)(j=1,2,\cdots,8)$ are its
eight neighboring pixels. The Census Transform or the LBP code is [7, 8]

$S^{(j)}_{Y_{i}}(p)=\left\\{\begin{array}[]{ll}1;&\mbox{if~{}}Y_{i}(\chi(j))\bowtie
Y_{i}(p)\\\ 0;&\mbox{otherwise}\\\ \end{array}\right.,$ (12)

where the operation $\bowtie\in\\{>,<,\geq,\leq\\}$. Since the binary
descriptor requires only eight comparisons, it is commonly stored as a byte
according to [6]

$S_{Y_{i}}(p)=\sum_{j=1}^{8}2^{j-1}S^{(j)}_{Y_{i}}(p).$ (13)

Unfortunately, the above binary descriptor does not maintain the morphological
invariance to intensity changes due to different weights for different
neighborhood pixels. For example, consider two pairs of bit-strings
$a=\\{1,0,1,0,1,1,1,0\\}$ and $b=\\{1,1,1,0,1,1,1,0\\}$,
$c=\\{1,1,1,0,1,1,1,1\\}$ and $d=\\{1,1,1,0,1,1,1,0\\}$. Both the pairs differ
at a single bit, and they are clearly similar. However, if the equation (13)
is used and matched under Euclidean norm, their distance become $64$ and $1$,
respectively. Clearly, the equation (13) does not capture the closeness in the
descriptor space and it is sensitive to the rotational. To address this
problem, the binary descriptor must be matched using a binary norm, such as
the Hamming distance, which counts the number of mismatched bits as follows:

$\rho_{h}(S_{Y_{i}}(p),S_{Y_{i}}(p^{\prime}))=\sum_{j=1}^{8}(S^{(j)}_{Y_{i}}(p)\odot
S^{(j)}_{Y_{i}}(p^{\prime})).$ (14)

where $\rho_{h}(a,b)$ is Hamming distance denoted as $a\odot b$ and $\odot$ is
the XOR operation. It is noted that equation (14) cannot be formulated as an
optimization problem due to the non-differentiability of the Hamming distance.
As such, an equivalently differentiable measurement is derived in the
following.

Proposition 1: The Hamming distance in the equation (14) is equivalent to the
following differentiable cost distance:

$\rho(S_{Y_{i}}(p),S_{Y_{i}}(p^{\prime}))=\sum_{j=1}^{8}(S^{(j)}_{Y_{i}}(p)-S^{(j)}_{Y_{i}}(p^{\prime}))^{2}.$
(15)

Proof Notice that

$\displaystyle a\odot b$
$\displaystyle=\left\\{\begin{array}[]{ll}0;&\mbox{if~{}}\\{a=0,b=0\\}\mbox{~{}or~{}}\\{a=1,b=1\\}\\\
1;&\mbox{if~{}}\\{a=1,b=0\\}\mbox{~{}or~{}}\\{a=0,b=1\\}\\\
\end{array}\right.,$ $\displaystyle(a-b)^{2}$
$\displaystyle=\left\\{\begin{array}[]{ll}0;&\mbox{if~{}}\\{a=0,b=0\\}\mbox{~{}or~{}}\\{a=1,b=1\\}\\\
1;&\mbox{if~{}}\\{a=1,b=0\\}\mbox{~{}or~{}}\\{a=0,b=1\\}\\\
\end{array}\right..$

It can be easily shown that

$\displaystyle(a-b)^{2}=a\odot b$ (16)

holds if both $a$ and $b$ are in the set $\\{0,1\\}$. Therefore, the Hamming
distance in the equation (14) is equivalent to the differentiable cost
distance in the equation (15). $\square$

The difference of two LBP coded images is computed as

$J=\sum_{p}\rho(S_{Y_{1}}(p),S_{Y_{2}}(\psi(p))).$ (17)

where $\psi(p)$ is the corresponding pixel in the image $\hat{Z}_{2}$ .

Let $p$ be denoted as $(x,y)$. Same as [6], it is assumed that the motion
between the images $\hat{Z}_{1}$ and $\hat{Z}_{2}$ is the Euclidean
transformation, i.e.,

$\psi(p)=[x,y]R^{T}(\theta)+[t_{x},t_{y}],$ (18)

where $R(\theta)$ is a 2D rotational matrix. The values of $\theta$, $t_{x}$,
and $t_{y}$ are obtained from the following optimization problem:

$\arg\min_{\theta,t_{x},t_{y}}J(\theta,t_{x},t_{y}).$ (19)

If the value of $\theta$, $t_{x}$ and $t_{y}$ are small, then image
$S_{Y_{2}}$ can be derived as below [6],

$\displaystyle S_{Y_{2}}^{(j)}(\psi(p))$
$\displaystyle=S_{Y_{1}}^{(j)}([x,y]R^{T}(\theta)+[t_{x},t_{y}])$
$\displaystyle\approx S_{Y_{1}}^{(j)}(x-y\theta+t_{x},y+x\theta+t_{y})$
$\displaystyle\approx S_{Y_{1}}^{(j)}(x,y)+\frac{\partial
S_{Y_{1}}^{(j)}}{\partial x}(t_{x}-y\theta)+\frac{\partial
S_{Y_{1}}^{(j)}}{\partial y}(x\theta+t_{y})$
$\displaystyle=S_{Y_{1}}^{(j)}(x,y)+\frac{\partial S_{Y_{1}}^{(j)}}{\partial
x}t_{x}+\frac{\partial S_{Y_{1}}^{(j)}}{\partial y}t_{y}+(\frac{\partial
S_{Y_{1}}^{(j)}}{\partial y}x-\frac{\partial S_{Y_{1}}^{(j)}}{\partial
x}y)\theta.$

For simplicity, define a $3\times 3$ symmetric matrix $A$ as

$\displaystyle\left\\{\begin{array}[]{l}A_{11}=\sum_{x,y}\sum_{j=1}^{8}(\frac{\partial
S_{Y_{1}}^{(j)}}{\partial x})^{2}\\\
A_{12}=A_{21}=\sum_{x,y}\sum_{j=1}^{8}\frac{\partial S_{Y_{1}}^{(j)}}{\partial
y}\frac{\partial S_{Y_{1}}^{(j)}}{\partial x}\\\
A_{13}=A_{31}=\sum_{x,y}\sum_{j=1}^{8}(\frac{\partial
S_{Y_{1}}^{(j)}}{\partial y}x-\frac{\partial S_{Y_{1}}^{(j)}}{\partial
x}y)\frac{\partial S_{Y_{1}}^{(j)}}{\partial x}\\\
A_{22}=\sum_{x,y}\sum_{j=1}^{8}(\frac{\partial
S_{Y_{1}}^{(j)}}{\partial}{y})^{2}\\\
A_{23}=A_{32}=\sum_{x,y}\sum_{j=1}^{8}(\frac{\partial
S_{Y_{1}}^{(j)}}{\partial y}x-\frac{\partial S_{Y_{1}}^{(j)}}{\partial
x}y)\frac{\partial S_{Y_{1}}^{(j)}}{\partial y}\\\
A_{33}=\sum_{x,y}\sum_{j=1}^{8}(\frac{\partial S_{Y_{1}}^{(j)}}{\partial
y}x-\frac{\partial S_{Y_{1}}^{(j)}}{\partial x}y)^{2}\\\ \end{array}\right.,$
(26)

and a vector $b=[b_{1},b_{2},b_{3}]^{T}$ as

$\displaystyle\left\\{\begin{array}[]{l}b_{1}=\sum_{x,y}\sum_{j=1}^{8}\frac{\partial
S_{Y_{1}}^{(j)}}{\partial x}(S^{(j)}_{1}-S_{Y_{2}}^{(j)})\\\
b_{2}=\sum_{x,y}\sum_{j=1}^{8}\frac{\partial S_{Y_{1}}^{(j)}}{\partial
y}(S^{(j)}_{1}-S_{Y_{2}}^{(j)})\\\
b_{2}=\sum_{x,y}\sum_{j=1}^{8}(\frac{\partial S_{Y_{1}}^{(j)}}{\partial
y}x-\frac{\partial S_{Y_{1}}^{(j)}}{\partial
x}y)(S^{(j)}_{1}-S_{Y_{2}}^{(j)})\\\ \end{array}\right..$ (30)

It is easily shown that the equation
$A\left[\begin{array}[]{lll}\theta&t_{x}&t_{y}\end{array}\right]^{T}=b$ can be
derived from the optimization problem (19). Once the optimal values of
$\theta^{*}$, $t_{x}^{*}$, and $t_{y}^{*}$ are obtained, the image $Z_{2}$ is
aligned as $Z_{2}([x,y]R^{T}(-\theta^{*})+[-t_{x}^{*},-t_{y}^{*}])$.

The optimization is very fast due to linear regression, and the solution can
be achieved to sub-pixel accuracy. It is noted that the Taylor series involved
in the estimation of $(\theta,t_{x},t_{y})$ is a good approximation only when
the motion is very small. To further increase speed and robust to large
motion, the optimization is implemented by coarse-to-fine technique which is
represented using the Gaussian pyramid [6], shown in Algorithm 1, where the
wrap() function is an affine transformation and is used to update the position
of pixels.

Algorithm 1 Pseudo code of the proposed method

Input: Two differently exposed images $Z_{1}$(Refernce images) and
$Z_{2}$(Slave image).
Output: Motion parameters between image pairs $Z_{1}$ and $Z_{2}$

1:$\hat{Z}_{1}$,$\hat{Z}_{2}$ = $Normalization(Z_{1},Z_{2})$

2:Estimate $t_{x}^{0}$, $t_{y}^{0}$ and $\theta^{0}$ by historgram-based
matching

3:$\hat{Z}_{1}$ = Wrap $(\hat{Z}_{1},t_{x}^{0},t_{y}^{0},\theta^{0})$

4:Build N-tier pyramid Gaussian pyramid of $\hat{Z}_{1}$,$\hat{Z}_{2}$

5:for pyrlevel = N:1 do

6: Code $\hat{Z}_{1}^{pyrlevel}$, $\hat{Z}_{2}^{pyrlevel}$ by equation (12)

7: Estimate $t_{x}^{pyrlevel}$, $t_{y}^{pyrlevel}$ and $\theta^{pyrlevel}$ by
“Hamming Distance”

8: Update $t_{x}^{pyrlevel-1}=2*t_{x}^{pyrlevel}$,
$t_{y}^{pyrlevel-1}=2*t_{y}^{pyrlevel}$,
$\theta^{pyrlevel-1}=\theta^{pyrlevel}$

9: Wrap $\hat{Z}_{1}^{pyrlevel-1}$ with $t_{x}^{pyrlevel}$, $t_{y}^{pyrlevel}$
, $\theta^{pyrlevel}$

10:return $t_{x}^{*}$, $t_{y}^{*}$ and $\theta^{*}$

It should be pointed out that the single scale with a few iterations is enough
if the initial values of $\theta$, $t_{x}$ and $t_{y}$ can be computed by
using the inertial measurement unit (IMU) integration [31] or other methods,
such as histogram-based matching [6]. Besides, the LBP code is also a simple
way to reduce the effects of haze [32].

##  4 Experimental Results

In this section, the proposed method is evaluated with a variety of
synthesized images from benchmark datasets, public datasets (Cai’s dataset)
[33] and real images. All datasets are released in open source[34]. Readers
can browse the full-size images in the digital version and zoom in to better
observe the differences. The results of the proposed method are compared with:
1) existing non-parametric ordering features using MTB [5], CT [7], LBP [8]
and BRIEF [9] methods, while employing the alignment algorithm shown in
Section 3; 2) the intensity-based method IMF; 3) the feature-based method SIFT
[35] and the hybrid method IMF+SIFT; 4) IMF+BRIEF and IMF+BRIEF+“Hamming”
distance (“HD”); 5) IMF+LBP[6]; and 6) Learning-based matching methods
SuperPoint [22] and LF-Net [23].

###  4.1 Test on synthetic images

9 well geometrically aligned sequences in the benchmark datasets, and 37
randomly selected and well geometrically aligned sequences in [33] are used to
evaluate different alignment algorithms. These multi-exposure image sequences
are captured by different cameras, which cover a broad range of scenes,
subjects and lighting conditions. Each sequence has 3 to 18 differently
exposed images and is arranged from the longest exposure to the shortest
exposure. Some sample sequences are presented in Fig. 2(d).

(a) “BigTree” sequence. Fields, trees and mountains in first image are over-
exposed.

(b) “Snowman” sequence. The first image is almost over-exposed due to sun
glare.

(c) “Inscription” sequence. The text on the inscription in the first image is
over-exposed and cannot be seen clearly.

(d) “Pillar” sequence.

Figure 2: Multi-exposed image sequences. Each sequence is arranged from long
exposure to short exposure and the first image in each sequence is reference
image. The interval between images in (a) and (b) is 1EV. The exposure levels
in (c) and (d) are manually set based on the lighting ratio of the scene. The
full-size images have been uploaded to the internet, readers can zoom them for
better observation[34].

.

To evaluate the alignment performance, the first image in each sequence is
selected as the reference, and other images are rotated by 5∘, and shifted 30
pixels and 10 pixels in $y$-axis and $x$-axis, respectively, to produce
unaligned samples for latter experiments. The alignment performances are
verified in terms of average errors between estimated motion parameters and
the true values (5∘, 30,10), and the results are shown in Table 1 and 2. The
average error of the proposed method is the smallest, which implies the
proposed method is suitable for various scene sets alignment and robust to
intensity variations.

A typical sequence named “Big Tree” (1EV interval) in benchmark datasets,
shown in Fig. 2(d)(a), is selected to test each algorithm’s robustness to
exposure changes. Table 3 summarizes the motion errors in aligning for the
“BigTree” sequence. The SIFT, BRIEF, SuperPoint and LF-Net do not provide
motion parameters but they provide the matching point pairs of two images.
Thus, the performances of the four methods are assessed by the number of
consistent points and mismatching ones. Table 3 reveals that: 1) Among the
order feature-based methods, the error of the MTB method is the smallest,
which illustrates that the MTB is robust to the varying exposures. But, as
seen in Table 4-5, the performance of MTB is not good in the other two
sequences, such as “Snowman” shown in Fig.2(d)(b) because the reference image
is seriously over-exposed. 2) The performances of the IMF, CT, LBP, SIFT and
BRIEF methods are not good, but the IMF+SIFT and IMF+LBP are robust to
intensity variations. This shows that IMF improves the similarity of the
differently exposed images. 3) BRIEF and IMF+BRIEF do not offer good results
due to large intensity variations, while IMF+BRIEF+“HD” and the proposed
method demonstrate high robustness to intensity variations. This indicates
that the proposed “Hamming” distance improves the alignment accuracy of binary
operators, especially for the rotational angle $\theta$. The proposed
normalization method reduces the effects of saturation and noise in the dark
regions, and improves the similarity of two differently exposed images. 4) It
can be observed from Table 3 and Table 4 that the performance of the learning
based methods on “Snowman” sequence is worse than their performance on
“BigTree” sequence because the reference image on “Snowman” sequence is
completely washed out. The SuperPoint is more robust to intensity variations
than the LF-Net, i.e., it can find 6 pairs of consistent points in the third
image, but 0 pair from the fourth image due to large exposure difference
between them, as shown in Fig. 3(a). By contrast, the method of LF-net can
find more feature points, but can not match these feature points correctly, as
shown in Fig. 3(b). 5) In comparison with the above methods, the proposed
method is robust to large EV intervals (7EV in “Bigtree”), especially for
over-exposed image sequences, in which the reference image is completely
washed out. The LBP based alignment algorithm does not work as indicated in
Table 4. This was also pointed out in [26]. The performance of SuperPoint on
sequences “Pillar” and “Inscriptionn” is better than its performance on
“Snowman” and “BigTree” sequences because the reference images in sequences
“Pillar” and “Inscriptionn” contain rich features as shown in Fig. 4.
Nevertheless, the proposed method is also superior to other methods as shown
in Table 5 and Table 6. The proposed image alignment algorithm outperforms the
DVO algorithm in [26] and multi-exposed image alignment in [6].

Table 1: Overall results performed on benchmark database. $\Delta\theta$, $\Delta t_{y}$ and $\Delta t_{x}$, are the average errors between estimated motion parameters and (5∘, 30, 10).  Method |  $\Delta\theta$(∘) | $\Delta t_{y}$ | $\Delta t_{x}$
---|---|---|---
Mean | Max | Min | Mean | Max | Min | Mean | Max | Min
MTB | 2.8 | 23.5 | 0 | 17.8 | 149 | 0.8 | 9.6 | 92 | 0.4
CT | 6 | 39 | 0 | 64 | 539 | 0 | 49 | 195 | 0.5
LBP | 3 | 23 | 0 | 26 | 213 | 0.8 | 34 | 282 | 0
IMF | 1.3 | 6.2 | 0 | 4.1 | 32 | 0.1 | 6.1 | 130 | 0
IMF+LBP | 1.1 | 13 | 0 | 8.5 | 192 | 0.9 | 10 | 254 | 0
IMF+BRIEF +“HD” | 1.9 | 14.1 | 0 | 14.6 | 181 | 0.3 | 15.8 | 253 | 0.3
Ours | 0.6 | 10 | 0 | 3.8 | 34 | 0.7 | 1.8 | 11.1 | 0.1
Table 2: Overall results performed on 37 Cai’s database[33]. Method |  $\Delta\theta$(∘) | $\Delta t_{y}$ | $\Delta t_{x}$
---|---|---|---
Mean | Max | Min | Mean | Max | Min | Mean | Max | Min
MTB | 0.9 | 23 | 0 | 7.2 | 177 | 0.1 | 4.2 | 151 | 0
CT | 5.2 | 62 | 0 | 86 | 542 | 0.5 | 155 | 984 | 0
LBP | 0.7 | 12 | 0 | 11 | 137 | 0.1 | 8.3 | 369 | 0
IMF | 1.6 | 6 | 0 | 7.1 | 201 | 0.1 | 7.3 | 386 | 0
IMF+LBP | 0.9 | 24 | 0 | 8.8 | 195 | 0 | 8.3 | 346 | 0
IMF+BRIEF +“HD” | 0.7 | 14.8 | 0 | 11.2 | 199 | 0 | 9.1 | 352 | 0
Ours | 0.2 | 3.6 | 0 | 6.9 | 241 | 0 | 1.5 | 9.9 | 0
Table 3: Performances of different methods on “BigTree” sequence in terms of motion parameters $(\Delta\theta,\Delta t_{y},\Delta t_{x})$. Method | 2 | 3 | 4 | 5 | 6 | 7
---|---|---|---|---|---|---
MTB | $\Delta\theta$ | 0 | 0 | 0 | 0 | 0 | 0
$\Delta t_{y},\Delta t_{x}$ | 1.6,1.1 | 1.6,0.9 | 1.7,0.8 | 1.7,0.7 | 1.7,0.7 | 1.6,0.7
CT | $\Delta\theta$ | 0 | 0 | 1.8 | 1.4 | 3 | 5
$\Delta t_{y},\Delta t_{x}$ | 1,1.1 | 0.6,1.2 | 34,19 | 84,175 | 21,195 | 59,192
LBP | $\Delta\theta$ | 0 | 0 | 0.5 | 1 | 3.2 | 11.7
$\Delta t_{y},\Delta t_{x}$ | 2,1 | 2,1 | 5.5,0.6 | 19,138 | 17,33 | 36,65
IMF | $\Delta\theta$ | 2.4 | 1.6 | 1.9 | 3 | 2.6 | 0
$\Delta t_{y},\Delta t_{x}$ | 0.4,3 | 0.1,2.2 | 0.2,2.3 | 2,5.5 | 1.1,5.1 | 1.8,1.2
IMF+LBP | $\Delta\theta$ | 0 | 0 | 0 | 0 | 0 | 4.4
$\Delta t_{y},\Delta t_{x}$ | 1.7,1.3 | 1.7,1.3 | 1.7,1.3 | 1.6,1.3 | 1.7,1.3 | 15,11
IMF+BRIEF+“HD” | $\Delta\theta$ | 0 | 0 | 0 | 0 | 0 | 6
$\Delta t_{y},\Delta t_{x}$ | 3,1.2 | 2,5.5 | 2.3,0 | 1,5 | 2.7,4.3 | 31,12
Ours | $\Delta\theta$ | 0 | 0 | 0 | 0 | 0 | 0
$\Delta t_{y},\Delta t_{x}$ | 1.6,1.4 | 1.6,1.4 | 1.6,1.4 | 1.6,1.5 | 3.2,0 | 1.5,1.5
SIFT | $\surd$ | $\surd$ | $\times$ | $\times$ | $\times$ | $\times$
IMF+SIFT | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\times$ | $\times$
BRIEF | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
IMF+BRIEF | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
SuperPoint | $\surd$ | $\surd$ | $\surd$ | $\times$ | $\times$ | $\times$
LF-Net | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
$\surd$ indicates good alignment, $\times$indicates wrong alignment.
Table 4: Performances of different methods on “Snowman” sequence in terms of motion parameters $(\Delta\theta,\Delta t_{y},\Delta t_{x})$. Method | 2 | 3 | 4 | 5 | 6
---|---|---|---|---|---
MTB | $\Delta\theta$ | 0.7 | 8.5 | 10 | 10 | 8.6
$\Delta t_{y},\Delta t_{x}$ | 3,1 | 32,4 | 38,15 | 40,17 | 33,4
CT | $\Delta\theta$ | 0.2 | 1.6 | 7 | 20 | 39
$\Delta t_{y},\Delta t_{x}$ | 0,1 | 32,150 | 2,77 | 39,38 | 20,75
LBP | $\Delta\theta$ | 3 | 5 | 7.4 | 5.4 | 0.4
$\Delta t_{y},\Delta t_{x}$ | 17,34 | 51,130 | 0,189 | 31,145 | 157,127
IMF | $\Delta\theta$ | 6 | 6 | 5.4 | 2.4 | 5.2
$\Delta t_{y},\Delta t_{x}$ | 16,3 | 16,3 | 30,9 | 32,10 | 17,10
IMF+LBP | $\Delta\theta$ | 0 | 0 | 0 | 0.1 | 0.1
$\Delta t_{y},\Delta t_{x}$ | 1.4,0.8 | 1.6,0.7 | 1.4,1 | 1.4,1 | 1.5,1
IMF+BRIEF+“HD” | $\Delta\theta$ | 1.3 | 0.5 | 0.1 | 0.7 | 0.7
$\Delta t_{y},\Delta t_{x}$ | 11,5 | 1,0 | 15,16 | 5,6 | 11,10
Ours | $\Delta\theta$ | 0 | 0 | 0 | 0 | 0
$\Delta t_{y},\Delta t_{x}$ | 1.6,1.1 | 1.7,1.0 | 1.8,1.0 | 1.8,1.0 | 1.8,0.8
SIFT | $\surd$ | $\surd$ | $\times$ | $\times$ | $\times$
IMF+SIFT | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$
BRIEF | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
IMF+BRIEF | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
SuperPoint | $\surd$ | $\surd$ | $\times$ | $\times$ | $\times$
LF-Net | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
$\surd$ indicates good alignment, $\times$indicates wrong alignment.
Table 5: Performances of different methods on “Pillar” sequence in terms of motion parameters $(\Delta\theta,\Delta t_{y},\Delta t_{x})$. Method | 2 | 3 | 4 | 5 | 6 | 7 | 8
---|---|---|---|---|---|---|---
MTB | $\Delta\theta$ | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.3 | 20.5
$\Delta t_{y},\Delta t_{x}$ | 2.1,1.2 | 1.6,1.5 | 1.3,1.9 | 1.4,1.5 | 0.8,1.4 | 0.1,0.7 | 156.5,1.7
CT | $\Delta\theta$ | 0 | 0 | 0 | 4.7 | 3.6 | 3.3 | 6
$\Delta t_{y},\Delta t_{x}$ | 2,0.8 | 1.6,1.1 | 1.2,1.5 | 89,20 | 470,86 | 471,115 | 205,595
LBP | $\Delta\theta$ | 0 | 0 | 0 | 0.1 | 0.1 | 6.3 | 5.7
$\Delta t_{y},\Delta t_{x}$ | 2,0.8 | 1.6,1.1 | 1.2,1.5 | 1.4,1.1 | 1.6,0.8 | 47,16.2 | 46.3,29
IMF | $\Delta\theta$ | 4.3 | 4.6 | 2.2 | 0.8 | 0.6 | 0.4 | 0.5
$\Delta t_{y},\Delta t_{x}$ | 9.7,7.7 | 12,9 | 11,6 | 3.4,2.7 | 2.5,2.2 | 1.2,0.8 | 1.3,0.2
IMF+LBP | $\Delta\theta$ | 0 | 0 | 0 | 0 | 0 | 0.2 | 3
$\Delta t_{y},\Delta t_{x}$ | 2,0.9 | 1.6,1.2 | 1.3,1.6 | 1.4,1.5 | 1.5,1.4 | 1.7,0.7 | 15,6.1
IMF+BRIEF+“HD” | $\Delta\theta$ | 0.2 | 0.1 | 0.6 | 0 | 1.2 | 0.4 | 0.5
$\Delta t_{y},\Delta t_{x}$ | 3.9,12.6 | 0,7.1 | 9.8,16.3 | 5.1,5.5 | 1.6,30 | 33.6,6.7 | 12.7,8.9
Ours | $\Delta\theta$ | 0 | 0 | 0 | 0 | 0 | 0.1 | 0.1
$\Delta t_{y},\Delta t_{x}$ | 2,0.9 | 1.6,1.2 | 1.2,1.6 | 1.4,1.4 | 1.5,1.2 | 1.2,0.4 | 1.2,0.2
SIFT | $\surd$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
IMF+SIFT | $\surd$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
BRIEF | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
IMF+BRIEF | $\surd$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
SuperPoint | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\times$
LF-Net | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
$\surd$ indicates good alignment, $\times$indicates wrong alignment. |  |
Table 6: Performances of different methods on “Inscription” sequence in terms of motion parameters $(\Delta\theta,\Delta t_{y},\Delta t_{x})$. Method | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
---|---|---|---|---|---|---|---|---
MTB | $\Delta\theta$ | 0.3 | 0.1 | 0 | 0 | 0 | 0.1 | 0.1 | 0.1
$\Delta t_{y},\Delta t_{x}$ | 3.8,1.2 | 1.0,1.4 | 0.8,1.5 | 0.4,1.8 | 0.5,1.7 | 1.1,1.4 | 1.2,1.5 | 1.8,1.3
CT | $\Delta\theta$ | 0 | 0.1 | 6 | 6.8 | 7.4 | 3.6 | 2.6 | 7
$\Delta t_{y},\Delta t_{x}$ | 0.9,1 | 0.8,1.1 | 15,233 | 48,4 | 82.5,523 | 51,570 | 36,675 | 63,707
LBP | $\Delta\theta$ | 0 | 0 | 0.1 | 0.1 | 0.1 | 0 | 0 | 0
$\Delta t_{y},\Delta t_{x}$ | 1.3,1.2 | 1.1,1.3 | 1.0,1.1 | 0.6,1.4 | 0.8,1.2 | 1.4,0.8 | 1.4,0.7 | 1.8,0.5
IMF | $\Delta\theta$ | 2.2 | 4 | 2.1 | 3.2 | 3.3 | 3.3 | 3.5 | 0.1
$\Delta t_{y},\Delta t_{x}$ | 4.8,6.6 | 23,7.5 | 7.4,6.6 | 11.4,7.9 | 11.5,8 | 11,7.9 | 8.4,9.5 | 2.2,0.9
IMF+LBP | $\Delta\theta$ | 0 | 0 | 0.1 | 0.1 | 0.1 | 0 | 0 | 24
$\Delta t_{y},\Delta t_{x}$ | 1.3,1.4 | 1.1,1.4 | 1,1.5 | 0.7,1.8 | 0.8,1.7 | 1.4,1.3 | 1.4,1.3 | 190,333
IMF+BRIEF+“HD” | $\Delta\theta$ | 0 | 0 | 0 | 0.2 | 0.1 | 0.1 | 0 | 14.8
$\Delta t_{y},\Delta t_{x}$ | 4.3,17 | 1.2,2 | 4.6,20.5 | 1.5,0.3 | 0.9,2.3 | 4.5,14.4 | 0.6,13.5 | 27,11
Ours | $\Delta\theta$ | 0 | 0 | 0.1 | 0.1 | 0.1 | 0 | 0 | 0
$\Delta t_{y},\Delta t_{x}$ | 1.2,1.3 | 1.1,1.4 | 1,1.4 | 0.5,1.6 | 0.7,1.6 | 1.3,1.1 | 1.3,1.2 | 1.7,1.1
SIFT | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\times$ | $\times$
IMF+SIFT | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\times$
BRIEF | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
IMF+BRIEF | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
SuperPoint | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$
LF-Net | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
$\surd$ indicates good alignment, $\times$indicates wrong alignment. |  |  |
Figure 3: Comparison of learning-based methods and the proposed method on
“Snowmann” sequence. In this figure, red lines indicate consistent point
pairs, and green lines indicate mismatching point pairs. Figure 4: Comparison
of learning-based methods and the proposed method on “Pillar” sequence. In
this figure, red lines indicate consistent point pairs, and green lines
indicate mismatching point pairs.

###  4.2 Test on real images

For further objective evaluation, 35 sequences of real scenes are used here,
which are captured using a NIKON D300, and a CANON EOS-1Ds Mark II, they are
comprised of various scenarios including small/big camera motion,
static/dynamic scenes, little/severe saturation, 1EV/2EV intervals and so on.
Each sequence contains 4 to 11 images, two sample sequences of multi-exposure
image datasets are presented in Fig. 5. As motion parameters are unknown,
alignment results are evaluated in terms of mutual information (MI) [36].
Selecting the brightest image as the reference, the MIs in different exposures
are shown in Fig. 6. The performance of alignment methods usually decreases as
the EV interval increases. This is because the normalization error increases
with the increment of EV interval. On the other hand, the performances of the
CT, IMF+LBP and the proposed method are robust to 7EV and 8 EV intervals,
among which, the proposed method achieves the largest mutual information,
especially when the EV interval between the two images is large. The proposed
normalization method reduces the errors of IMF in under/over exposed regions,
and improves the similarity of two large-exposure-ratio images. It is also
observed from Fig. 6 that the minimal, median and maximal MI of the proposed
method is the largest, which implies the proposed method is suitable for
various scene sets alignment and robust to intensity variations.

(a) “Snow” sequence.

(b) “Table” sequence.

Figure 5: Real multi-exposed images. Each sequence is arranged from long
exposure to short exposure and the reference and first image in each sequence
have over-exposed regions.

Figure 6: Mutual information for 35 real sequences.

###  4.3 tests on efficiency

In order to demonstrate the efficiency of the proposed method, the operation
times of different methods are calculated respectively. The learning-based
methods are running in Python code111SuperPoint,
https://github.com/magicleap/SuperPointPretrainedNetwork ;LF-Net,
https://github.com/vcg-uvic/lf-net-release, the rest are running in MATLAB
code. The experiments are performed on DELL computer with i7-6700 CPU, 8GB
RAM, and 64bit OS. However, the performances of MTB, LBP, CT, IMF, SIFT,
BRIEF, IMF+BRIEF and the learning-based methods failed to align multi-exposed
images in the previous experiment. The descriptor of IMF+BRIEF+“HD” is a
256-vector which needs expensive computation to match two descriptors under
“Hamming distance”. Therefore, the operation times of the proposed method are
compared with IMF+LBP and IMF+SIFT. The operation times (second) of different
algorithms are tabulated in Table 7.

Two sequences are randomly selected to test the efficiency. The results are
given in Table 7. The operation time of the algorithm is proportional to image
size and the number of features. IMF+SIFT is expensive in computation due to
complex feature extraction and description as shown in Table 8. The
normalization process of the proposed method is computationally more expensive
than that of [6], but its alignment is 2.5 times faster than that of IMF+LBP
and IMF+SIFT methods.

Table 7: Operation times (second) of different algorithms. All the codes are written in Matlab. Sequence | No. of images | Image size | IMF+LBP | IMF+SIFT | Ours
---|---|---|---|---|---
Snowman | 6 | 693*505 | 68.6 | 183.2 | 48.4
Snow | 4 | 960*680 | 57.3 | 1209 | 36.6
Table 8: Operation times (second) of normalization, feature extraction and description, alignment when aligning the first two “Snowman” images. All the codes are written in Matlab. Method | Normalization | Feature extraction and descripition | Alignment
---|---|---|---
IMF+SIFT | 0.7 | 59.8 | 6.0
IMF+LBP | 0.7 | 16.3
Ours | 2.7 | 6.1

##  5 Conclusion

In this paper, a novel image alignment algorithm has been proposed for multi-
exposed images with saturated regions by using a differentiable “Hamming”
distance. The images are first normalized by using intensity mapping functions
to reduce the effects of saturation. The normalized images are then coded by
LBP and aligned by the proposed algorithm. Experimental results show that the
proposed algorithm outperforms state-of-the-art alignment algorithms for
differently exposed images.

There are many applications of the proposed algorithm. For example, it can be
used in the autonomous SLAM [2] with prior knowledge on the mapping
environment, the direct visual odometry [26] with the assistance of IMU
integration [31], and panorama imaging for tele-operation of unmanned ground
vehicles [3]. All these applications will be studied in our future research.

## References

  * [1] R. Szeliski,  “Image alignment and stitching: a tutorial,”  Foundations and Trends in Computer Graphics and Vision, vol. 2, no.1, pp.1-104, 2007.
  * [2] T. Whelan, S. Leutenegger, R. F. Salas-Moreno, B. Glocker, and A. J. Davison,  “ElasticFusion: dense SLAM without a pose graph,”  in Robotics: Science & Systems, 2015.
  * [3] W. Yao and Z. G. Li,  “Instant color matching for mobile panorama imaging,”  IEEE Signal Processing Letters, vol. 22, no. 1, pp. 6-10, Jan. 2015.
  * [4] P. E. Debevec and J. Malik,  “Recovering high dynamic range radiance maps from photographs,”  in proceedings of ACM SIGGRAPH, ACM, pp.369-378, 1997.
  * [5] G. Ward,  “Fast, robust image registration for compositing high dynamic range photographs from hand-held exposures,”  Journal of graphics tools, vol.8, no.2, pp.17-30, 2003.
  * [6] S. Q. Wu, Z. G. Li, J. H. Zheng, and Z. J. Zhu,  “Exposure robust method for aligning differently exposed images,”  IEEE Signal Processing Letters, vol. 21, no. 7, pp. 885–889, Jul. 2014.
  * [7] R. Zabih and J. Woodfill,  “Non-parametric local transforms for computing visual correspondence,”  in Computer Vision-ECCV. Springer, pp. 151-158, 1994.
  * [8] T. Ojala, M. Pietikainen, and D. Harwood  “A comparative study of texture measures with classification based on featured distributions,”  Pattern recognition, vol.29, no.1, pp.51-59, 1996.
  * [9] M. Calonder, V. Lepetit, M. Ozuvsal, T. Trzcinski, C. Strecha, and P. Fua,  “BRIEF: Computing a local binary descriptor very fast,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.34, no.7, pp.1281-1298, 2001.
  * [10] E. Rublee, V. Rabaud, K. Konolige, and G.R. Bradski,  “ORB: an efficient alternative to SIFT or SURF,”  in International Conference on Computer Vision,2012.
  * [11] R. Salakhutdinov and G. Hinton,  “Semantic hashing,”  in International Journal of Approximate Reasoning, vol.50, no.7, pp. 969-978, 2009.
  * [12] Y. Weiss, A. Torralba, and R. Fergus,  “Spectral hashing,”  in International Conference on Neural Information Processing Systems, 2008.
  * [13] T. Trzcinski, M. Christoudias, P. Fua, and V. Lepetit,  “Boosting binary beypoint descriptors,”  in IEEE Conference on Computer Vision and Pattern Recognition, pp.2874-2881, 2013.
  * [14] V. Balntas, L. Tang, and K. Mikolajczyk,  “BOLD - Binary online learned descriptor for efficient image matching,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [15] K. Lin, J. Lu, C.S. Chen, and J. Zhou,  “Learning compact binary descriptors with unsupervised deep neural networks,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [16] J. Lu, V. E. Liong, X. Zhou and J. Zhou,  “Learning compact cinary face descriptor for face recognition,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 10, pp. 2041-2056, Oct. 2015.
  * [17] Y. Duan, J. Lu, J. Feng and J. Zhou,  “Context-aware local binary feature learning for face recognition,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 5, pp. 1139-1153, 2018.
  * [18] H. Jain, J. Zepeda, P. Pérez, and R. Gribonval,  “SUBIC: A supervised, structured binary code for image search,”  in IEEE International Conference on Computer Vision (ICCV), Venice, pp. 833-842, 2017.
  * [19] Y. Duan, J. Lu, Z. Wang, J. Feng and J. Zhou,  “Learning deep binary descriptor with multi-quantization,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 8, pp. 1924-1938, 2019.
  * [20] K. He, Y. Lu, and S. Sclaroff,  “Local descriptors optimized for average precision,”  in IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 596-605, 2018.
  * [21] K. Yi, E. Trulls, V. Lepetit, and P. Fua,  “Lift: Learned invariant feature transform,”  in ECCV, pp 467-483, 2016.
  * [22] D. DeTone, T. Malisiewicz, and A. Rabinovich,  “Superpoint: Self-supervised interest point detection and description,” 2017.
  * [23] Y. Ono, E. Trulls, P. Fua, and K. M. Yi,  “LF-net: Learning local features from images,” 2018.
  * [24] X. Shen, C. Wang, X. Li, Z. Yu, and Z. He,  “RF-Net: An end-to-end image matching network based on receptive field,”  in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [25] M. Grossberg and S. Nayar  “Determining the camera response from images: What is knowable?”  IEEE Transactions on pattern analysis and machine intelligence, vol.25, no.11, pp.1455-1467, Nov. 2003.
  * [26] H. Alismail1, M. Kaess, B. Browning, and S. Lucey,  “Direct visual odometry in low light using binary descriptors,”  in IEEE International Conference on Robotics and Automation (ICRA), Singapore, 2017.
  * [27] Z. G. Li, J. H. Zheng, Z. J. Zhu, W. Yao, and S. Q. Wu,  “Weighted guided image filtering,”  IEEE Trans. on Image Processing, vol. 24, no. 1, pp. 120-129, Jan. 2014.
  * [28] J. Engel, V. Koltun, and D. Cremers,  “Direct sparse odometry,”  IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 3, pp. 611–625, Mar. 2018.
  * [29] Z. G. Li, J. H. Zheng, Z. J. Zhu, and S. Q. Wu,  “Selectively detail-enhanced fusion of differently exposed images with moving objects,”  IEEE Trans. on Image Processing, vol. 23, no. 10, pp. 4372–4382, Oct. 2014.
  * [30] J. Bouguet  “Pyramidal implementation of the Lucas Kanade feature tracker description of the algorithm,”  Intel, Microprocessor Research Labs, pp. 1-9, Jan. 2000.
  * [31] J. Henawy, Z. G. Li, W. Y. Yau, and G. Seet,  “Accurate IMU factor using switched linear systems for VIO,”  IEEE Transactions On Industrial Electronics, Accepted, Jun. 2020.
  * [32] Z. G. Li and J. H. Zheng,  “ Single image haze removal Using globally guided image filtering,”  IEEE Trans. on Image Processing, vol. 27, no. 1, pp. 442–450, Jan. 2018.
  * [33] J. Cai, S. Gu, and L. Zhang,  “Learning a deep single image contrast enhancer from multi-exposure images,”  IEEE Transactions on Image Processing, vol.27, no.4, pp.2049-2062. Apr. 2018.
  * [34] https://github.com/gelinlan/Robust-Alignment-of-Multi-Exposed-Images-with-Saturated-Regions.
  * [35] D. G. Lowe,  “Distinctive image features from scale-invariant keypoints,”  International Journal of Computer Vision, vol.60, no.2, pp.91-110, Feb. 2004.
  * [36] P. Viola and W. M. Wells  “Alignment by maximization of mutual information,”  International Journal of Computer Vision, vol.24, no.2, pp.137-154, Feb. 1997.