# DUDF: Differentiable Unsigned Distance Fields with Hyperbolic Scaling

Miguel Fainstein${}^{\text{1}}$
<EMAIL_ADDRESS>Viviana Siless${}^{\text{1}}$
<EMAIL_ADDRESS>Emmanuel Iarussi${}^{\text{1},\text{2}}$
<EMAIL_ADDRESS>${}^{\text{1}}$Universidad Torcuato DiTella,
${}^{\text{2}}$CONICET

###### Abstract

In recent years, there has been a growing interest in training Neural Networks
to approximate Unsigned Distance Fields (UDFs) for representing open surfaces
in the context of 3D reconstruction. However, UDFs are non-differentiable at
the zero level set which leads to significant errors in distances and
gradients, generally resulting in fragmented and discontinuous surfaces. In
this paper, we propose to learn a hyperbolic scaling of the unsigned distance
field, which defines a new Eikonal problem with distinct boundary conditions.
This allows our formulation to integrate seamlessly with state-of-the-art
continuously differentiable implicit neural representation networks, largely
applied in the literature to represent signed distance fields. Our approach
not only addresses the challenge of open surface representation but also
demonstrates significant improvement in reconstruction quality and training
performance. Moreover, the unlocked field’s differentiability allows the
accurate computation of essential topological properties such as normal
directions and curvatures, pervasive in downstream tasks such as rendering.
Through extensive experiments, we validate our approach across various data
sets and against competitive baselines. The results demonstrate enhanced
accuracy and up to an order of magnitude increase in speed compared to
previous methods.

##  1 Introduction

Surface representation is a fundamental aspect in the field of 3D geometry
processing, with explicit methods such as triangle meshes, point clouds, and
voxelized representations being traditional choices. Implicit surface
representations, on the other hand, have been an integral part of the graphics
pipeline for many decades. They encapsulate surfaces as the zero-level set of
a function, providing a compact and continuous geometry representation. The
novelty in recent years has emerged from parameterizing these implicit
functions with Neural Networks (NNs), combining their learning capabilities
with the advantages of implicit representations.

Figure 1: Distance field cross-sections comparative. First row displays ground
truth unsigned distance fields. Methods based on signed distances, like SIREN
[35], mistakenly fill gaps in open surfaces. While methods based on unsigned
distances such as CAP-UDF [45] can represent non-watertight surfaces, the
learned distance field do not approximate the true function. DUDF’s
differentiable formulation successfully overcomes these challenges.

Signed Distance Functions (SDFs) have traditionally been the chosen
formulation for implicit surface representation [8, 25] due to their well-
defined gradients and the ease they offer for computing constructive solid
geometry operations and mesh reconstruction. However, they are inherently
limited to closed surfaces (Fig. 1), which poses a significant challenge for
representing open surfaces with implicit methods. This limitation arises from
the inside/outside sign flip on the zero level set, which is impossible to
define for surfaces that do not enclose a volume.

The advent of Unsigned Distance Functions (UDFs) extended representation
capabilities to open surfaces. However, this advancement introduced a
challenge: the non-differentiable nature of the distance functions at the zero
level set. Such non-differentiability leads to inaccuracies in distances and
gradients learned by NNs, particularly near the surface where precision is
paramount. This paper tackles the challenge of representing open surfaces with
neural networks, focusing on a formulation that ensures the surface continuity
and smoothness. Our framework (DUDF) combines a hyperbolic scaling of the
distance field, together with a new Eikonal problem featuring tailored
boundary conditions. This formulation facilitates the training of continuously
differentiable implicit neural representation networks, while preserving the
essential attributes of the unsigned distance fields. Additionally, the
differentiable nature of our representations enables precise calculations of
crucial topological characteristics, including curvatures and normals, which
are pervasive in downstream tasks such as rendering. In contrast, earlier
techniques depended on mesh reconstruction methods like Marching Cubes [19,
14, 46] for rendering purposes.

We evaluate our framework with experiments that rigorously assess performance
on a variety of challenging data sets and compare to competitive benchmarks in
the field. Our method overcomes several limitations of previous approaches,
which struggled with the non-differentiability of UDFs. Results demonstrate
that our approach not only enhances the reconstruction quality of open
surfaces but also accelerates the training process. We believe the dual
improvement in precision and efficiency enhance the applicability of neural
UDFs representations in geometric processing for real-world scenarios.

##  2 Related Works

The representation and reconstruction of three-dimensional geometry is a
fundamental challenge in the field of computer graphics, vision, and
computational geometry. We now briefly discuss the key contributions and
methodologies adopted in recent literature to address the main challenges in
the field, distinguishing between closed and open surface paradigms.

Closed surfaces. Recent years have seen a surge in the use of Neural Implicit
Functions (NIFs) for modeling and reconstructing 3D shapes [39]. These
representations are usually implemented with Multi-Layer Perceptrons (MLP) or
Convolutional Neural Networks (CNNs), but differ in the learning task [21, 4,
29, 5]. On the one hand, some methods [21, 4, 29, 22, 5] learn an indicator
function or binary occupancies which are later used to reconstruct the 3D
surface. On the other hand, alternative methods focus on estimating the SDF at
any point in 3D space [27, 15, 23, 9, 11, 24]. The landscape of NIFs has
further expanded with the introduction of novel approaches like implicit
moving least-squares surfaces [16], a differentiable Poisson solver [28], a
complex Gabor wavelet [32], and a level set alignment loss [20]. Notably,
central works in the field [13, 35] approached the problem as finding
solutions to the Eikonal equation alongside expressive boundary conditions,
building upon the literature on solving PDEs with neural networks [34, 30]. In
particular, SIREN [35] achieved remarkable results utilizing periodic
activation functions, allowing to successfully control the function’s
differential fields. Despite these advancements, a fundamental limitation of
current NIF approaches remains in their inability to represent open surfaces,
a characteristic often exhibited by real-world objects like scene walls,
clothing, or vehicles with inner structures.

Open surfaces. In order to model general non-watertight surfaces, Chibane et
al. introduced the idea of learning UDFs, pioneering the handling of open
surfaces with neural networks [6]. Following this, several methods have aimed
to improve the performance of open-surface representation using neural
networks [36, 37, 42, 41, 43, 45]. For instance, GIFS [42] and Neural Vector
Fields [41] model the relationships between every two points instead of the
relationships between points and surfaces. HSDF separately learns sign and
distance fields to handle surfaces with arbitrary topologies[38]. NeuralUDF
[18] and NeUDF [17] focused on learning an unsigned distance field by volume
rendering for multi-view reconstruction of surfaces with arbitrary topologies.
Closer to our work is CAP-UDF [45], which optimized models on raw point clouds
by learning to move 3D point queries until reaching the surface with a field
consistency constraint. Despite these developments, since UDFs are not
differentiable at the zero level set, the gradient field is ill-defined near
the isosurface. This leads to difficulties during training, but specially
during surface reconstruction, where gradients are often used to determine
surface presence between points [14, 46], compute surface normals, and project
points onto the isosurface [6] through gradient descent, among others. Recent
work by Zhou et al. aimed to solve this issue by introducing constraints at
the zero level set in the form of losses leading to a smoother surface [44].
In contrast, our approach formulates an Eikonal problem which is solved by
learning a scaled distance function that remains differentiable close to the
surface. We demonstrate that our method not only enhances reconstruction
quality, addressing the prevailing challenges associated with the non-
differentiability of the unsigned distance, but also improves _state-of-the-
art_ in terms of training time performance.

##  3 Proposed Approach

###  3.1 Mathematical background

Methods addressing closed surfaces approach the learning of SDFs as finding
the solution to a system of Partial Differential Equations (PDEs) governed by
the homogeneous Eikonal equation with Dirichlet and Neumann boundary
conditions. Formally, given a closed surface $\mathcal{S}$ in
$\mathcal{C}\subseteq\mathbb{R}^{3}$ they seek to find a continuous function
$f$ that satisfies $\left\|\nabla f\right\|=1$, with boundary conditions
$f_{\rvert_{\mathcal{S}}}=0$, $\nabla
f_{\rvert_{\mathcal{S}}}=\mathbf{n}_{\mathcal{S}}$. Where
$\mathbf{n}_{\mathcal{S}}$ denotes the unitary normal field at the surface.
Although SDFs are not differentiable at every point and represent only a weak
solution to the Eikonal equation, recent research has demonstrated
considerable success in solving this problem with continuously differentiable
implicit neural representation networks [35, 13, 26, 7]. These architectures
use periodic activations, facilitating smoother optimization processes and
improved control over the solution’s gradient field. This is achievable
because locations where the signed distance lacks differentiability are
distant from the isosurface, hence the approximation tends to be good in a
close neighborhood of the zero level set. In particular, using these networks
enable accurate computation of critical topological properties, including mean
and Gaussian curvature [26]. This contrasts with networks featuring piecewise
linear activations, such as ReLUs, which have null second-order derivatives
[35].

In the context of representing UDFs, these functions cannot be a solution to
the Eikonal equation both in the interior and exterior regions of the surface,
without losing differentiability at the zero level set. This presents
challenges for continuously differentiable implicit neural representation
networks in achieving satisfactory outcomes, being that the approximation
errors happen at the isosurface where accuracy is paramount. To our knowledge,
there has been no successful report of a solution to the Eikonal equation
accurately approximating unsigned distance functions.

Figure 2: In (a), view of different distance fields for a 2D slice of a torus.
Note the effect of hyperbolic scaling near the isosurface (bottom). In (b),
sign and gradient for each distance field $f$. In signed distance, the
gradient $\nabla f$ at the isosurface is equal to the surface’s normal field
$\mathbf{n}_{\mathcal{S}}$ (top). In unsigned distance, the gradient is
undefined at the isosurface due to the change in orientation (middle). Our
maximum curvature field vectors $\mathbf{v}_{1}$ align with the surface’s
unoriented normals (bottom).

###  3.2 Problem statement

Our key insight is to overcome these challenges by redefining the unsigned
distance field, particularly through the application of an hyperbolic scaling.
In light of this understanding, we propose to learn the parameters $\theta$ of
a neural network $f_{\theta}$ with periodic activation functions [35] to
approximate the function:

$\mathit{t_{\mathcal{S}}}(\mathbf{x})=\mathit{d}_{\mathcal{S}}(\mathbf{x})\tanh(\alpha\mathit{d}_{\mathcal{S}}(\mathbf{x})),$
(1)

where $\mathit{d}_{\mathcal{S}}$ is the unsigned distance to surface
$\mathcal{S}$; and $\alpha$ is a constant value. The function
$\mathit{t}_{\mathcal{S}}$ is a differentiable approximation of
$\mathit{d}_{\mathcal{S}}$, whose zero level set is the surface $\mathcal{S}$.
In Fig. 2 (a), we show the impact of our hyperbolic scaling on the unsigned
distance field, mainly characterized by quadratic smoothing in proximity to
the isosurface and a linear behavior in the distance. The parameter $\alpha$
controls the distance at which the smoothing occurs.

The scaling enables the application of continuously differentiable implicit
neural representation networks to solve an Eikonal equation, while retaining
the UDF’s open surface representation capabilities. For this task, we aim to
address an heterogeneous Eikonal equation, for which we know $t_{\mathcal{S}}$
(Eq. 1) is a weak solution:

$\left\\{\begin{matrix}\left\|\nabla f\right\|=\phi\\\
f_{\rvert_{\mathcal{S}}}=0\\\ \nabla
f_{\rvert_{\mathcal{S}}}=\mathbf{0}\end{matrix}\right.$ (2)

where $\phi$ is defined as the L2 norm of the gradient of Eq. 1, formally:

$\phi(\mathbf{x})=\tanh(\alpha\mathit{d}_{\mathcal{S}}(\mathbf{x}))+\alpha\mathit{d}_{\mathcal{S}}(\mathbf{x})(1-\tanh^{2}(\alpha\mathit{d}_{\mathcal{S}}(\mathbf{x}))).$
(3)

This formulation assumes that the domain boundary is defined by a closed
surface $\mathcal{S}$, where Dirichlet and Neumann boundary conditions can be
imposed. However, that framework does not directly apply to open surfaces,
which do not enclose a well-defined domain and thus cannot support the
imposition of boundary constraints. To address this limitation, we reformulate
the problem as an initial value problem where the Dirichlet condition on
$\mathcal{S}$ prescribes the initial values for $f$, and where the Neumann
condition can be omitted given that is already satisfied by the Eikonal
equation when $\phi(\mathbf{s})=0$ for $\mathbf{s}\in\mathcal{S}$.

The problem outlined in Eq. 2 does not incorporate the information of the
surface normals. This contrasts with prior research focused on closed
surfaces, where leveraging such information has led to improved accuracy and
enhanced reconstruction quality by further constraining the set of feasible
solutions. Our second key insight is that although the gradient of
$t_{\mathcal{S}}$ (Eq. 1) becomes null at the isosurface, the direction in
which the gradient norm increases most rapidly remains non-null. This
direction is indicated by the maximum curvature of $f$, defined as the
eigenvector associated with the largest eigenvalue of the Hessian matrix. Note
that in this context, the maximum curvature does not refer to the surface’s
curvature, but rather to the curvature of the hyperbolic scaled unsigned
distance field $f$, where the function’s rate of change varies most rapidly.
Formally, $\mathbf{H}_{t_{\mathcal{S}}}$ is the Hessian matrix of
$t_{\mathcal{S}}$ which has 3 real eigenvalues
$\left|\lambda_{1}\right|\geq\left|\lambda_{2}\right|\geq\left|\lambda_{3}\right|$;
be $\mathbf{v_{1}}$ a unitary eigenvector associated with $\lambda_{1}$ at
$\mathbf{s}$, it can be shown that
$\mathbf{v_{1}}=\pm\mathbf{n}_{\mathcal{S}}$ for every
$\mathbf{s}\in\mathcal{S}$. Therefore, we further condition the solution to
Eq. 2 by adding an extra boundary condition enforcing alignment between the
unitary directions associated to the maximum curvature field and the unitary
normal field of the surface.

###  3.3 Implicit function learning

Following the aforementioned definitions, our neural networks are trained to
minimize the following loss function:

$\mathcal{L}=\lambda_{e}\mathcal{L}_{\mathit{Eikonal}}+\lambda_{d}\mathcal{L}_{\mathit{Dirichlet}}+\lambda_{n}\mathcal{L}_{\mathit{Neumann}}+\lambda_{g}\mathcal{L}_{\mathit{MCurv}},$
(4)

with $\lambda_{i}$ constant weights controlling the relevance of each term.
The term which favors a solution to the Eikonal PDE is defined as:

$\mathcal{L}_{\mathit{Eikonal}}=\int_{\mathcal{C}}\left|\left\|\nabla
f_{\theta}(\mathbf{x})\right\|-\phi(\mathbf{x})\right|\quad d\mathbf{x}.$ (5)

The Dirichlet boundary condition loss term controlling function values at the
surface is formally defined as:

$\mathcal{L}_{\mathit{Dirichlet}}=\int_{\mathcal{S}}\left|f_{\theta}(\mathbf{x})\right|\quad
d\mathbf{x}.$ (6)

Similarly to previous works [26, 7], we extend this conditioning to points far
from the surface ${\mathcal{S}}$. This is achieved by computing an
approximation of the function $\mathit{t_{S}}$ (Eq. 1) based on nearest
neighbors. Neumann’s boundary condition is expressed in the loss term:

$\mathcal{L}_{\mathit{Neumann}}=\int_{\mathcal{S}}\left\|\nabla
f_{\theta}(\mathbf{x})\right\|\quad d\mathbf{x}.$ (7)

Finally, we ensure the second-order boundary condition by aligning the
directions of maximum curvature with the surface normal through the
computation of the following integral:

$\mathcal{L}_{\mathit{MCurv}}=\int_{\mathcal{S}}1-\left|\mathbf{v_{1}}(\mathbf{x})\cdot\mathbf{n}_{\mathcal{S}}(\mathbf{x})\right|\quad
d\mathbf{x},$ (8)

where $\mathbf{v_{1}}(\cdot)$ is the unitary eigenvector associated with the
largest eigenvalue of $\mathbf{H}_{f_{\theta}}(\mathbf{x})$, the Hessian
matrix of $f_{\theta}$ at $\mathbf{x}$. In practice, we approximate all
integrals discretely using a data set comprised of tuples
$\\{(\mathbf{x}_{i},\mathit{d}_{\mathcal{S}}(\mathbf{x}_{i}),\mathbf{n}_{\mathcal{S}}(\mathbf{x}_{i}))\\}_{i}$.
At every training iteration, the gradient and Hessian of the neural network
are computed through automatic differentiation.

###  3.4 Isosurface refinement

In pursuit of solutions that exhibit a consistent, near-zero value at the
isosurface —essential for smooth reconstructions— we further fine-tune our
networks in a second optimization step. Formally, we minimize the following
loss function:

$\mathcal{L}_{Refinement}=\lambda_{\mu}\rvert\,\mu({f_{\theta}}_{\rvert_{\mathcal{S}}})\,\rvert+\lambda_{\sigma}\,\sigma({f_{\theta}}_{\rvert_{\mathcal{S}}}),$
(9)

where $\mu(\cdot)$ is the mean value and $\sigma(\cdot)$ the standard
deviation.

By minimizing the mean and variance of the learned function values at the
isosurface, we ensure they are as close to zero as possible. This refinement
process effectively reduces the oscillations and deviations that can occur at
the critical boundary, thereby enhancing the overall quality of the
reconstructed surfaces.

###  3.5 Normals and curvature computation

A significant advantage of our formulation over prior efforts [6, 46] is the
fact that gradient fields and higher-order derivatives are well-defined in the
vicinity of the isosurface, allowing the direct computation of topological
properties during inference and reconstruction. This means that we are able to
render unsigned distance fields using standard algorithms such as sphere
tracing. However, directly using the gradients of $f_{\theta}$ at the
isosurface as normals for rendering purposes can be unreliable since their
norm is close to 0. Hence, for a given surface value threshold $\epsilon$,
when a point $\mathbf{s}$ is identified such that
$f_{\theta}(\mathbf{s})<\epsilon$, we determine the normal direction to the
surface by calculating the unit eigenvectors associated with the maximum
eigenvalue of $\mathbf{H}_{f_{\theta}}(\mathbf{s})$. Note that there are two
unitary eigenvectors per surface point, $\mathbf{v_{1}}$ and
$-\mathbf{v_{1}}$; however this is usually the desired behavior since open
surfaces may not be orientable, such as the möbius strip. To address this
ambiguity in the sign we consider the position of the camera when rendering.
Additionally, other relevant topological properties such as the mean and
Gaussian curvature are also available by means of the maximum unit eigenvector
field. See Sec. 4.6 for more details.

##  4 Results and evaluation

###  4.1 Experimental setup

We conducted a series of experiments to assess the performance of our method.
To target a broad amount of surfaces, we experimented on three well-known data
sets: ShapeNet cars [3], Multi-Garment [2], and DeepFashion [48]. For each
data set we trained individual networks on 30 randomly selected oriented point
clouds. For the Deep Fashion and Multi-Garment dataset, we sampled 100,000
surface points, while for ShapeNet cars we doubled the sample size to account
for the higher degree of complexity. On each training iteration, the loss was
computed on a subset of 30,000 points, equally distributed into three distinct
groups: surface points (1), far domain points (2), and near domain points (3).
Surface points (1) were randomly selected from the ground truth point cloud.
Far points (2) were uniformly generated within the function’s domain: a cube
with side length 2 and centered at the origin. All shapes were uniformly
scaled to fit within this volume before sampling. The distance to the point
clouds was approximated using a tree-based search algorithm [47]. Finally,
since we found that a biased near-the-surface sampling improved accuracy, we
used near domain points (3) during training. We constructed this set by
randomly displacing the sampled surface points in their normal direction.
Displacements were sampled from a normal distribution centered at zero with a
standard deviation of $0.01$. The corresponding unsigned distance was
approximated with the distance to the undisplaced surface point. This was
experimentally found to be sufficiently accurate and fast.

###  4.2 Network architecture

Our network takes spatial coordinates $x,y,z$ as input, and outputs the
predicted $t_{\mathcal{S}}$ value from Eq. 1. The architecture consists of an
8-layer 256-units MLP with _sine_ activations. The same number of layers and
parameter count were used for baseline methods. Loss weights were
experimentally set to $\lambda_{e},\lambda_{d},\lambda_{n}=1e^{4}$,
$\lambda_{g}=1e^{3}$ and $\lambda_{\mu},\lambda_{\sigma}=1e^{5}$. Models were
implemented in Python using Pytorch library, and distance metrics were
computed using Pytorch3D [31]. We trained each network for 3,000 iterations,
the first third of them utilized a learning rate of $1\mathrm{e}^{-4}$, in the
following third we lowered it to $1\mathrm{e}^{-5}$, and in the final
iterations (only the refinement loss active) we set the learning rate to
$1\mathrm{e}^{-7}$ with a cosine decay. These final small learning rates
allowed the refinement of the isosurface without losing the gained accuracy in
the rest of the field. Regarding the value of $\alpha$ (Eq. 1), we
experimentally found that choosing a value of 100 kept a good balance between
training accuracy and reconstruction quality. Experiments were run on Ubuntu,
an Intel(R) Xeon(R) Silver 4310 CPU, 256Gb of RAM, and 2 Nvidia A100 graphics
cards of 80Gb VRAM.

###  4.3 Mesh reconstruction

For a thorough comparison of our method against existing techniques, we
undertake mesh reconstruction using two gradient-based Marching Cubes
algorithms, referred here as MC1 [46] and MC2 [14]. These methods perform
linear interpolation between grid values for accurate placement of triangle
vertices. Given this requirement and considering that the function
$t_{\mathcal{S}}$, as described in Equation 1, exhibits quadratic behavior
near the isosurface, we retrieve the true unsigned distance by first dividing
by $\alpha$ and then applying the square root. For a comprehensive analysis we
include SIREN [35], which does not approximate an unsigned distance, as
illustrated in Figure 1. In this case, we used traditional Marching Cubes[19]
for the reconstruction.

###  4.4 Surface reconstruction

####  4.4.1 Closed shapes

Method | time(s) $\downarrow$ | L1CD $\downarrow$ | L2CD $\downarrow$
---|---|---|---
DeepSDF [27] | $\mathbf{94}$ | $19.40$ | $0.206$
SIREN [35] | $379$ | $15.40$ | $0.171$
CAP-UDF [46] | $1080$ | $\mathbf{9.48}$ | $0.030$
Ours (MC1 [46]) | $319$ | $\mathbf{9.48}$ | $\mathbf{0.028}$
Ours (MC2 [14]) | $9.49$ | $\mathbf{0.028}$

Table 1: Training time, L1, and L2 mean Chamfer distances ($\times 10^{3}$)
for the closed ShapeNet cars data set. DeepSDF and SIREN were trained on
signed distances, while CAP-UDF and our method were trained on unsigned
distances.

We benchmarked DUDF against state-of-the-art methods for representing closed
surfaces, where signed distances are well-defined. Given that our primary
focus is not on closed surface representation, we limited this comparison to
the ShapeNet car dataset [3], modifying the meshes for closure and omitting
internal structures [40]. The comparative results are detailed in Table 1.
While DeepSDF [27] excels in speed, it falls short in reconstruction quality.
Being primarily designed for shape generation, DeepSDF is outperformed by
methods tailored for accurate single shape representation, as evidenced by the
Chamfer distance metrics between ground truth and reconstructed point clouds.

Figure 3: Comparisons on DeepFashion [48] (top row), Multi-Garment [2] (middle
row), and ShapeNet cars [3] (bottom row) data sets. DUDF preserves fine
details and accurately represents complex geometries without closing holes,
outperforming SIREN (SDF), which tends to smooth and round models, and CAP-
UDF, which captures sharp features but often closes open surfaces.
Reconstructions for CAP-UDF and DUDF performed with MC1 [46].

SIREN [35] offers improved shape representation with enough detail and
smoothness. However, since its zero level set satisfies the implicit function
theorem, it cannot properly represent shapes with right angles and thin
structures. In contrast, CAP-UDF [46] can better approximate these sharp
features because it does not rely on continuously differentiable implicit
neural representation networks, thereby achieving higher accuracy. However,
the training time is high and the reliance on gradient based Marching Cubes
methods for surface reconstruction tends to create surfaces with small holes
and discontinuities which hinders the visual quality in some cases. Our
approach captures a differentiable function’s zero level set like SIREN, but
uniquely bypasses the implicit function theorem by having null gradients at
the isosurface. This allows our method to effectively represent sharp features
and complex geometries, leading to improved precision in surface
reconstruction, particularly in scenarios where surfaces are not inherently
smooth.

####  4.4.2 Open surfaces

When evaluating open-surface representation, we primarily compare our method
against CAP-UDF [46]. This approach was shown to be more effective than
earlier techniques (i.e. NDF [6]), that required a complex process for
creating a dense point cloud and reconstructing a triangulated surface using
the Ball Pivoting algorithm [1], which lead to poor quality meshes with great
amounts of holes and irregularities Additionally, a recent method by Zhou et
al. [44] has introduced constraints on the zero level set, which the authors
claim it leads to better reconstructions. However, the absence of available
implementation code in their official repository precluded a fair comparison
with our method. Therefore, we concentrate our analysis on the most recent and
available methodologies.

| Method
---
SIREN [35]
CAP-UDF [46]
Ours (MC1 [46])
Ours (MC2 [14])
| DeepFashion [48]
---
time(s) $\downarrow$ | L1CD $\downarrow$ | L2CD $\downarrow$
$376$ | $27.3$ | $1.980$
$1390$ | $18.1$ | $1.110$
$\mathbf{326}$ | $\mathbf{9.01}$ | $\mathbf{0.025}$
$9.14$ | $0.027$
| Multi-Garment [2]
---
time(s) $\downarrow$ | L1CD $\downarrow$ | L2CD $\downarrow$
$374$ | $40.5$ | $8.810$
$1440$ | $18.5$ | $1.190$
$\mathbf{318}$ | $\mathbf{8.70}$ | $\mathbf{0.024}$
$8.82$ | $0.026$
| ShapeNet cars [3]
---
time(s) $\downarrow$ | L1CD $\downarrow$ | L2CD $\downarrow$
$751$ | $16.9$ | $0.170$
$1040$ | $\mathbf{10.9}$ | $0.071$
$\mathbf{317}$ | $12.3$ | $\mathbf{0.057}$
$13.7$ | $0.081$

Table 2: Training time, L1, and L2 mean Chamfer distances ($\times 10^{3}$)
for the evaluated open surface data sets. SIREN was trained as described in
the original paper, while CAP-UDF and our method were trained on unsigned
distances.

Qualitative comparisons can be observed in Fig. 3 and a full quantitative
analysis is presented in Table 2. On the one hand, our method demonstrates a
significant improvement in efficiency, consuming up to an order of magnitude
less computational time than CAP-UDF, while also showing enhanced performance
across all three data sets. This advantage is particularly evident in the
DeepFashion and Multi-Garment data sets, where CAP-UDF tends to inaccurately
close openings (like those at the ends of sleeves in clothing). On the other
hand, since SIREN was devised for closed surfaces, it creates a negative
distance valued shell around the isosurface. This causes to either close every
hole or a thickened surface enclosing the ground-truth level set.

In addition to these experiments, we compare our method with CAP-UDF in the
context of sphere tracing rendering. This comparison is presented in Fig. 4,
where we showcase two rendering examples. Since learned unsigned distance
fields in CAP-UDF do not grow linearly away from the surface, the marching
steps in sphere tracing often fail to accurately intersect the surface,
leading to undesirable visual artifacts. Even when hitting the surface,
gradients might be undefined, leading to noisy shaded images. In contrast, our
proposed distance function does not exhibit these problems, offering a more
robust framework for direct rendering. Furthermore, our ability to compute
normals using the maximum curvature field is useful for shading purposes,
enhancing the rendering quality without undergoing an intermediate 3D mesh
reconstruction step.

###  4.5 Ablation study

Figure 4: Rendering examples using sphere tracing. CAP-UDF struggles with non-
linear growth of their unsigned distance fields, causing the sphere tracing
marching step to miss the surface. Conversely, our method demonstrates
precision in direct rendering scenarios.

We now dissect the impact of individual loss components on the network’s
accuracy through an ablation study. The methodology involves selectively
deactivating terms in our loss function (Eq. 4) to isolate their
contributions. We adopted the experimental framework from the prior section,
training on 30 meshes from the DeepFashion dataset [48]. To elucidate the role
of boundary conditions in enhancing reconstruction quality, we incorporated a
normal consistency metric into our analysis. This metric is computed as the
mean value of the absolute cosine similarities between each point to its
closest neighbor in the other point cloud. We report metrics for
reconstructions performed using MC2 [14], yielding results analogous to those
obtained with MC1 [46]. Additionally, we contrasted the effects of
approximating the distance function $d_{\mathcal{S}}$ with $t_{\mathcal{S}}$
(Eq. 1), utilizing sine and $ReLU$ activation functions (each with distinct
Eikonal problems).

The findings presented in Table 3 underscore the significance of both the
Eikonal equation (Eq. 5) and the $MCurv$ boundary condition (Eq. 8) in
achieving superior normal consistency, which notably influences the rendering
quality. Moreover, the refinement step during training is an important element
for normal consistency improvement, offering substantial increased performance
for a small additional computational cost.

Method | time(s) $\downarrow$ | L1CD $\downarrow$ | L2CD $\downarrow$ | NC $\downarrow$
---|---|---|---|---
Baseline | $326$ | $\mathbf{9.14}$ | $\mathbf{0.027}$ | $\mathbf{0.020}$
$\lambda_{e}=0$ | $312$ | $9.43$ | $0.028$ | $0.033$
$\lambda_{g}=0$ | $150$ | $9.16$ | $\mathbf{0.027}$ | $0.021$
$\lambda_{e},\lambda_{g}=0$ | $\mathbf{109}$ | $9.44$ | $\mathbf{0.027}$ | $0.035$
$\lambda_{\mu},\lambda_{\sigma}=0$ | $302$ | $9.24$ | $\mathbf{0.027}$ | $0.031$
$d_{\mathcal{S}}$ (sine) | $150$ | $31.2$ | $0.830$ | $0.057$
$d_{\mathcal{S}}$ ($ReLU$) | $145$ | $47.5$ | $2.500$ | $0.149$
Table 3: Quantitative impact of each loss component on the reconstruction
accuracy of our network. We report training time, L1, and L2 mean Chamfer
distances $\times 10^{3}$, and Normal Consistency (NC).

###  4.6 Computing mean and Gaussian curvature

A significant advantage of our method over previous approaches is the
possibility to compute curvatures. Previous work learned non-differentiable
functions, which preclude the direct computation of geometrical properties
such as mean and Gaussian curvature. Our differentiable UDFs facilitates the
extraction of these curvature values. Leveraging the divergence
($\nabla\cdot$) of the normal field ($\mathbf{n}_{\mathcal{S}}$), the mean
curvature $H$ at any point on the surface $\mathbf{s}$ is then computed [12,
10, 26] as:

$H(\mathbf{s})=\frac{1}{2}\nabla\cdot\mathbf{n}_{\mathcal{S}}(\mathbf{s}).$
(10)

Additionally, the Gaussian curvature $K$ can be computed using the determinant
of a matrix composed of the Jacobian matrix
$J_{\mathbf{n}_{\mathcal{S}}}(\mathbf{s})$ of the unit normal field
($\mathbf{n}_{\mathcal{S}}$) along with the normal field itself. Formally, we
compute:

$K(\mathbf{s})=-\det\left[\begin{array}[]{c|c}J_{\mathbf{n}_{\mathcal{S}}}(\mathbf{s})&\mathbf{n}_{\mathcal{S}}(\mathbf{s})\\\
\hline\cr\mathbf{n}_{\mathcal{S}}(\mathbf{s})^{t}&0\end{array}\right].$ (11)

In Fig. 5 we show mean and Gaussian curvatures for open and closed surfaces
computed with our method.

Figure 5: Mean and Gaussian curvatures computed with our method for closed
surfaces (left) and open surfaces (right).

##  5 Limitations and future work

Our method encounters certain limitations. Primarily, by approximating the
function $t_{\mathcal{S}}$ (Eq. 1), the Eikonal equation requires supervision
with ground truth unsigned distances, in contrast to signed distance
approaches where it is constant everywhere. In addition, the quadratic
behavior of the function near the isosurface produces a wider near-zero strip
which can lead to slightly inflated representations. This effect can be
lessened by computing square roots, at the expense of an increased
computational cost. Secondly, we observed that surfaces extracted using
gradient-based Marching Cubes methods can sometimes be less smooth than
directly rendering the function using sphere tracing. This is likely due to
the current reconstruction algorithm’s sensitivity to noise, an issue that
might be addressed by employing a tangency-aware surface reconstruction
strategy [33]. Additionally, computing the maximum curvature direction field
is computationally expensive, requiring three network passes and the
diagonalization of the Hessian matrix. However, future improvements in the
network’s design could reduce the number of parameters, thus lessening the
computational burden associated with computing topological features.

##  6 Conclusion

In this work we introduced Differentiable Unsigned Distance Fields (DUDF) with
Hyperbolic Scaling, a novel approach that addresses inherent limitations of
traditional UDFs in representing open surfaces. By applying a hyperbolic
transformation to the distance field, we define a new variant of the Eikonal
problem, tailored with unique boundary conditions. This adaptation enables our
model to capitalize on the robust framework of continuously differentiable
neural networks, thus enhancing reconstruction quality and training
efficiency. The conducted experiments provide evidence that our approach can
lead to improvements in reconstruction quality and computational performance
when compared to several state-of-the-art methods. Moreover, the ability to
accurately calculate topological features such as normals and curvatures is a
notable benefit of our model, offering additional utility in geometric
processing tasks and rendering. While our results are promising, showing
better accuracy and efficiency in most cases, we acknowledge the need for
further research and development. The potential applications of DUDF in 3D
geometry processing are broad, and we are optimistic about its utility in
practical scenarios. Future work will focus on exploring these applications
and enhancing the model’s performance.

##  7 Aknowledgments

This project was supported by grants from Salesforce, USA (Einstein AI 2020),
National Scientific and Technical Research Council (CONICET), Argentina (PIP
2021-2023 GI -11220200102981CO), and Universidad Torcuato Di Tella, Argentina.
We also thank Facundo Fainstein and Tomas Delgado for valuable discussions on
the Eikonal problem formulation.

## References

  * [1] Fausto Bernardini, J. Mittleman, Holly Rushmeier, Cláudio Silva, and Gabriel Taubin.  The ball-pivoting algorithm for surface reconstruction.  Visualization and Computer Graphics, IEEE Transactions on, 5:349 – 359, 11 1999.
  * [2] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll.  Multi-garment net: Learning to dress 3d people from images.  In IEEE International Conference on Computer Vision (ICCV). IEEE, oct 2019.
  * [3] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [4] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  * [5] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll.  Implicit functions in feature space for 3d shape reconstruction and completion.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6970–6981, 2020.
  * [6] Julian Chibane, Gerard Pons-Moll, et al.  Neural unsigned distance fields for implicit function learning.  Advances in Neural Information Processing Systems, 33:21638–21652, 2020.
  * [7] Mattéo Clémot and Julie Digne.  Neural skeleton: Implicit neural representation away from the surface.  Computers & Graphics, pages 368–378, 2023.
  * [8] Brian Curless and Marc Levoy.  A volumetric method for building complex models from range images.  In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303–312, 1996.
  * [9] Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson.  On the effectiveness of weight-encoded neural implicit 3d shapes.  arXiv preprint arXiv:2009.09808, 2020.
  * [10] Manfredo P Do Carmo.  Differential geometry of curves and surfaces: revised and updated second edition.  Courier Dover Publications, 2016.
  * [11] Yueqi Duan, Haidong Zhu, He Wang, Li Yi, Ram Nevatia, and Leonidas J Guibas.  Curriculum deepsdf.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16, pages 51–67. Springer, 2020.
  * [12] Ron Goldman.  Curvature formulas for implicit curves and surfaces.  Computer Aided Geometric Design, 22(7):632–658, 2005.
  * [13] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  arXiv preprint arXiv:2002.10099, 2020.
  * [14] Benoit Guillard, Federico Stella, and Pascal Fua.  Meshudf: Fast and differentiable meshing of unsigned distance field networks, 2022.
  * [15] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al.  Local implicit grid representations for 3d scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6001–6010, 2020.
  * [16] Shi-Lin Liu, Hao-Xiang Guo, Hao Pan, Peng-Shuai Wang, Xin Tong, and Yang Liu.  Deep implicit moving least-squares functions for 3d reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1788–1797, 2021.
  * [17] Yu-Tao Liu, Li Wang, Jie Yang, Weikai Chen, Xiaoxu Meng, Bo Yang, and Lin Gao.  Neudf: Leaning neural unsigned distance fields with volume rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 237–247, 2023.
  * [18] Xiaoxiao Long, Cheng Lin, Lingjie Liu, Yuan Liu, Peng Wang, Christian Theobalt, Taku Komura, and Wenping Wang.  Neuraludf: Learning unsigned distance fields for multi-view reconstruction of surfaces with arbitrary topologies.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20834–20843, 2023.
  * [19] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In Seminal graphics: pioneering efforts that shaped the field, volume 21, pages 347–353. 1998.
  * [20] Baorui Ma, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han.  Towards better gradient consistency for neural signed distance functions via level set alignment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17724–17734, 2023.
  * [21] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460–4470, 2019.
  * [22] Zhenxing Mi, Yiming Luo, and Wenbing Tao.  Ssrnet: Scalable 3d surface reconstruction network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 970–979, 2020.
  * [23] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.  Deep level sets: Implicit surface representations for 3d shape inference.  arXiv preprint arXiv:1901.06802, 2019.
  * [24] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.  Implicit surface representations as layers in neural networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4743–4752, 2019.
  * [25] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon.  Kinectfusion: Real-time dense surface mapping and tracking.  In 2011 10th IEEE international symposium on mixed and augmented reality, pages 127–136. Ieee, 2011.
  * [26] Tiago Novello, Guilherme Schardong, Luiz Schirmer, Vinicius da Silva, Helio Lopes, and Luiz Velho.  Exploring differential geometry in neural implicits, 2022.
  * [27] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  * [28] Songyou Peng, Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger.  Shape as points: A differentiable poisson solver.  Advances in Neural Information Processing Systems, 34:13032–13044, 2021.
  * [29] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 523–540. Springer, 2020.
  * [30] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis.  Physics informed deep learning (part ii): Data-driven discovery of nonlinear partial differential equations, 2017.
  * [31] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv:2007.08501, 2020.
  * [32] Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk.  Wire: Wavelet implicit neural representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18507–18516, 2023.
  * [33] Silvia Sellán, Christopher Batty, and Oded Stein.  Reach for the spheres: Tangency-aware surface reconstruction of sdfs.  arXiv preprint arXiv:2308.09813, 2023.
  * [34] Justin Sirignano and Konstantinos Spiliopoulos.  DGM: A deep learning algorithm for solving partial differential equations.  Journal of Computational Physics, pages 1339–1364, 2018.
  * [35] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Advances in neural information processing systems, 33:7462–7473, 2020.
  * [36] Rahul Venkatesh, Tejan Karmali, Sarthak Sharma, Aurobrata Ghosh, R Venkatesh Babu, László A Jeni, and Maneesh Singh.  Deep implicit surface point prediction networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12653–12662, 2021.
  * [37] Bing Wang, Zhengdi Yu, Bo Yang, Jie Qin, Toby Breckon, Ling Shao, Niki Trigoni, and Andrew Markham.  Rangeudf: Semantic surface reconstruction from 3d point clouds.  arXiv preprint arXiv:2204.09138, 2022.
  * [38] Li Wang, Weikai Chen, Xiaoxu Meng, Bo Yang, Jintao Li, Lin Gao, et al.  Hsdf: Hybrid sign and distance field for modeling surfaces with arbitrary topologies.  Advances in Neural Information Processing Systems, 35:32172–32185, 2022.
  * [39] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar.  Neural fields in visual computing and beyond.  In Computer Graphics Forum, volume 41, pages 641–676. Wiley Online Library, 2022.
  * [40] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann.  Disn: Deep implicit surface network for high-quality single-view 3d reconstruction.  CoRR, abs/1905.10711, 2019.
  * [41] Xianghui Yang, Guosheng Lin, Zhenghao Chen, and Luping Zhou.  Neural vector fields: Implicit representation by explicit learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16727–16738, 2023.
  * [42] Jianglong Ye, Yuntao Chen, Naiyan Wang, and Xiaolong Wang.  Gifs: Neural implicit function for general shape representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12829–12839, 2022.
  * [43] Fang Zhao, Wenhao Wang, Shengcai Liao, and Ling Shao.  Learning anchored unsigned distance functions with gradient direction alignment for single-view garment reconstruction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12674–12683, 2021.
  * [44] Junsheng Zhou, Baorui Ma, Shujuan Li, Yu-Shen Liu, and Zhizhong Han.  Learning a more continuous zero level set in unsigned distance fields through level set projection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 3181–3192, 2023.
  * [45] Junsheng Zhou, Baorui Ma, Yu-Shen Liu, Yi Fang, and Zhizhong Han.  Learning consistency-aware unsigned distance functions progressively from raw point clouds.  Advances in Neural Information Processing Systems, 35:16481–16494, 2022.
  * [46] Junsheng Zhou, Baorui Ma, Yu-Shen Liu, Yi Fang, and Zhizhong Han.  Learning consistency-aware unsigned distance functions progressively from raw point clouds.  In Advances in Neural Information Processing Systems (NeurIPS), 2022\.
  * [47] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.  Open3D: A modern library for 3D data processing.  arXiv:1801.09847, 2018.
  * [48] Heming Zhu, Yu Cao, Hang Jin, Weikai Chen, Dong Du, Zhangye Wang, Shuguang Cui, and Xiaoguang Han.  Deep fashion3d: A dataset and benchmark for 3d garment reconstruction from single images.  CoRR, 2020.


$\partial_{\\_}{\overline{z}}\varphi_{\\_}k=\mu_{\\_}k\partial_{\\_}{z}\varphi_{\\_}k$
for all $k$ and let $\psi_{\\_}k:\mathbb{C}\to\mathbb{C}$ be the unique
$K$-quasiconformal homeomorphism satisfying the same differential equation, as
well as $\psi_{\\_}k(0)=0$, $\psi_{\\_}k(1)=1$ (see [5, Theorem 4.30]).

By Lemma A.3, up to subsequences there exists a $K$-quasiconformal
homeomorphism $\psi_{\\_}\infty$ such that $\psi_{\\_}k\to\psi_{\\_}\infty$
and $\psi_{\\_}k^{-1}\to\psi_{\\_}\infty^{-1}$ in
$C^{0}_{\\_}{loc}(\mathbb{C})$.

By the chain rule (see [8, Lemma III.6.4]), the map
$\psi_{\\_}k\circ\varphi_{\\_}k^{-1}:\mathbb{C}\to\mathbb{C}$ is a
biholomorphism and fixes the origin, so it equals the multiplication by a
nonzero complex number $\lambda_{\\_}k$, i.e.
$\psi_{\\_}k=\lambda_{\\_}k\varphi_{\\_}k$. On the other hand,

$\displaystyle\left|\lambda_{\\_}k\right|=\min_{\\_}{x\in\partial
B_{\\_}1^{2}}\left|\psi_{\\_}k(x)\right|\to\min_{\\_}{x\in\partial
B_{\\_}1^{2}}\left|\psi_{\\_}\infty(x)\right|\in(0,\infty).$

Hence, up to further subsequences we can suppose that
$\lambda_{\\_}k\to\lambda_{\\_}\infty\in\mathbb{C}\setminus\left\\{0\right\\}$.
The statement follows with
$\varphi_{\\_}\infty:=\lambda_{\\_}\infty^{-1}\psi_{\\_}\infty$. ∎

###### Remark A.5.

In general, given $\varphi_{\\_}k\in\mathcal{D}_{\\_}K$ (for
$k\in\mathbb{N}\cup\left\\{\infty\right\\}$) with
$\varphi_{\\_}k\to\varphi_{\\_}\infty$ and
$\varphi_{\\_}k^{-1}\to\varphi_{\\_}\infty^{-1}$ locally uniformly, it is not
true that the corresponding Beltrami coefficients satisfy
$\mu_{\\_}k\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\mu_{\\_}\infty$ in
$L^{\infty}(\mathbb{C})$. For instance, let $\mu_{\\_}0(z):=\frac{1}{2}$ if
$\Re(z)\in\bigcup_{\\_}{n\in\mathbb{Z}}\left[n,n+\frac{1}{2}\right)$ and
$\mu_{\\_}0(z):=-\frac{1}{2}$ otherwise. Then the bi-Lipschitz homeomorphism
$\psi_{\\_}0:\mathbb{C}\to\mathbb{C}$ given by

$\displaystyle\psi_{\\_}0(x+iy):=\begin{cases}n+\frac{9}{5}(x-n)+\frac{3}{5}iy=n+\frac{6}{5}(z-n)+\frac{3}{5}(\overline{z}-n)&n\leq
x\leq n+\frac{1}{2}\\\
n+\frac{4}{5}+\frac{x-n}{5}+\frac{3}{5}iy=n+\frac{4}{5}+\frac{2}{5}(z-n)-\frac{1}{5}(\overline{z}-n)&n+\frac{1}{2}\leq
x\leq n+1\end{cases}$

satisfies
$\partial_{\\_}{\overline{z}}\psi_{\\_}0=\mu_{\\_}0\partial_{\\_}z\psi_{\\_}0$,
with the normalization $\psi_{\\_}0(0)=0$ and $\psi_{\\_}0(1)=1$. So
$\mu_{\\_}k:=\mu_{\\_}0(2^{k}\cdot)$ and
$\psi_{\\_}k:=2^{-k}\psi_{\\_}0(2^{k}\cdot)$ satisfy
$\partial_{\\_}{\overline{z}}\psi_{\\_}k=\mu_{\\_}k\partial_{\\_}z\psi_{\\_}k$
with the same normalization. Moreover, they converge uniformly to
$\psi_{\\_}\infty(x+iy)=x+\frac{3}{5}iy=\frac{4}{5}z+\frac{1}{5}\overline{z}$,
together with their inverses. The homeomorphism $\psi_{\\_}\infty$ satisfies
$\partial_{\\_}{\overline{z}}\psi_{\\_}\infty=\mu_{\\_}\infty\partial_{\\_}z\psi_{\\_}\infty$
with $\mu_{\\_}\infty:=\frac{1}{4}$, but
$\mu_{\\_}k\stackrel{{\scriptstyle*}}{{\rightharpoonup}}0$. Dividing each
$\psi_{\\_}k$ by $\min_{\\_}{\left|z\right|=1}\left|\psi_{\\_}k(z)\right|$, we
obtain a counterexample in the class $\mathcal{D}_{\\_}{3}$.

## References

    * Chodosh and Mantoulidis [2018] O. Chodosh and C. Mantoulidis.  Minimal surfaces and the Allen–Cahn equation on 3-manifolds: index, multiplicity, and curvature estimates.  _ArXiv preprint1803.02716_, 2018.
    * Ejiri and Micallef [2008] N. Ejiri and M. Micallef.  Comparison between second variation of area and second variation of energy of a minimal surface.  _Adv. Calc. Var._ 1(3):223–239, 2008\.  doi:10.1515/ACV.2008.009.
    * Hélein [2002] F. Hélein.  _Harmonic maps, conservation laws and moving frames (second edition)_ , vol. 150 of _Cambridge Tracts in Mathematics_.  Cambridge University Press, Cambridge, 2002.
    * Hummel [1997] C. Hummel.  _Gromov’s compactness theorem for pseudo-holomorphic curves_ , vol. 151 of _Progress in Mathematics_.  Birkhäuser Verlag, Basel, 1997
    * Imayoshi and Taniguchi [1992] Y. Imayoshi and M. Taniguchi.  _An introduction to Teichmüller spaces_.  Springer–Verlag, Tokyo, 1992.
    * Ketover and Liokumovich [2017] D. Ketover and Y. Liokumovich.  On the existence of unstable minimal Heegaard surfaces.  _ArXiv preprint1709.09744_, 2016.
    * Ketover and Marques and Neves [2016] D. Ketover, F. C. Marques and A. Neves.  The catenoid estimate and its geometric applications.  _ArXiv preprint1601.04514_, 2016.
    * Lehto and Virtanen [1973] O. Lehto and K. I. Virtanen.  _Quasiconformal mappings in the plane (second edition)_ , vol. 126 of _Grundlehren der mathematischen Wissenschaften_.  Springer–Verlag, New York–Heidelberg–Berlin, 1973.
    * Marques and Neves [2012] F. C. Marques and A. Neves.  Rigidity of min-max minimal spheres in three-manifolds.  _Duke Math. J._ , 161(14):2725–2752, 2012.  doi:10.1215/00127094-1813410.
    * Marques and Neves [2016] F. C. Marques and A. Neves.  Morse index and multiplicity of min-max minimal hypersurfaces.  _Camb. J. Math._ , 4(4):463–511, 2016.  doi:10.4310/CJM.2016.v4.n4.a2.
    * Michelat [2018] A. Michelat.  On the Morse index of critical points in the viscosity method.  _ArXiv preprint1806.09578_, 2018.
    * Pigati and Rivière [2017] A. Pigati and T. Rivière.  The regularity of parametrized integer stationary varifolds in two dimensions.  _ArXiv preprint1708.02211_, 2017.
    * Rivière [2017] T. Rivière.  A viscosity method in the min-max theory of minimal surfaces.  _Publ. Math. Inst. Hautes Études Sci._ , 126:177–-246, 2017.  doi:10.1007/s10240-017-0094-z.
    * Rivière [2017] T. Rivière.  The regularity of conformal target harmonic maps.  _Calc. Var. Partial Differential Equations_ , 56(4):117, 2017\.  doi:10.1007/s00526-017-1215-8.
    * Rivière [2018] T. Rivière.  Lower semi-continuity of the index in the viscosity method for minimal surfaces.  _ArXiv preprint1808.00426_, 2018.
    * Rudin [1987] W. Rudin.  _Real and complex analysis (third edition)_.  McGraw–Hill Book Co., New York, 1987.
    * Sacks and Uhlenbeck [1981] J. Sacks and K. Uhlenbeck.  The existence of minimal immersions of $2$-spheres.  _Ann. of Math. (2)_ , 113(1):1–24, 1981.  doi:10.2307/1971131.
    * Simon [1983] L. Simon.  _Lectures on geometric measure theory_ , vol. 3 of _Proceedings of the Centre for Mathematical Analysis_.  Australian National University, Canberra, 1983.
    * Song [2017] A. Song.  Local min-max surfaces and strongly irreducible minimal Heegaard splittings.  _ArXiv preprint1706.01037_, 2017.
    * Zhou [2015] X. Zhou.  Min-max minimal hypersurface in $(M^{n+1},g)$ with $Ric>0$ and $2\leq n\leq 6$.  _J. Differential Geom._ , 100(1):129–160, 2015.  doi:10.4310/jdg/1427202766.
    * Zhou [2017] X. Zhou.  On the existence of min-max minimal surface of genus $g\geq 2$.  _Commun. Contemp. Math._ , 19(4):1750041, 2017.  doi:10.1142/S0219199717500419.
    * Zhou [2019] X. Zhou.  On the multiplicity one conjecture in min-max theory.  _ArXiv preprint1901.01173_, 2019.


Citation

C. Zhou, G. AlRegib, A. Parchami, and K. Singh, ”Learning Trajectory-
Conditioned Relations to Predict Pedestrian Crossing Behavior,” in IEEE
International Conference on Image Processing (ICIP), Bordeaux, France, Oct.
16-19 2022.

Review

Date of acceptance: June 20, 2022

Codes

https://github.com/olivesgatech/cross_predict

Copyright

©2022 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

Keywords

Intent prediction, Relation learning, Pedestrian crossing

Contact

<EMAIL_ADDRESS>OR<EMAIL_ADDRESS>
https://ghassanalregib.info/
http://alregib.ece.gatech.edu/

# Learning Trajectory-Conditioned Relations to Predict Pedestrian Crossing
Behavior

###### Abstract

In smart transportation, intelligent systems avoid potential collisions by
predicting the intent of traffic agents, especially pedestrians. Pedestrian
intent, defined as future action, e.g., start crossing, can be dependent on
traffic surroundings. In this paper, we develop a framework to incorporate
such dependency given observed pedestrian trajectory and scene frames. Our
framework first encodes regional joint information between a pedestrian and
surroundings over time into feature-map vectors. The global relation
representations are then extracted from pairwise feature-map vectors to
estimate intent with past trajectory condition. We evaluate our approach on
two public datasets and compare against two state-of-the-art approaches. The
experimental results demonstrate that our method helps to inform potential
risks during crossing events with 0.04 improvement in F1-score on JAAD dataset
and 0.01 improvement in recall on PIE dataset. Furthermore, we conduct
ablation experiments to confirm the contribution of the relation extraction in
our framework.

Index Terms—  Intent Prediction, Relation Learning, Pedestrian Crossing

##  1 Introduction

Intelligent systems are at a crossroad between perception and prediction.
There has been significant advance on tackling the challenges of perception
using learning-based algorithms. For instance, robust models [1, 2, 3, 4] have
been developed for image perception-based applications under challenging
conditions. In addition, the success of learning-based models on natural
images has fostered its usage on computational image analysis including
seismic [5, 6] and medical [7] fields. While perception algorithms endows an
intelligent system with the ability to see the current surroundings,
prediction on future states informs intent and suggests safe maneuver.

Fig. 1: Example frames from the PIE dataset where a pedestrian intends to
cross while interacting with traffic objects. The left figure shows a
pedestrian 1) passes the vehicle b at further distance 2) slows down as the
closer vehicle a brakes. The observed trajectory is shown in green dashed
curve.

Predicting future states of traffic agents, especially pedestrians, in complex
urban scenes is safety-critical for intelligent transportation systems. The
future states of pedestrians can be in the form of upcoming actions [8], e.g.,
road crossing. In this paper, we consider predicting future crossing as
pedestrian intent prediction.

Prediction can be challenging when pedestrian’s future crossing depends on
other traffic agents and environment. Traffic agents constantly avoid
collisions by interacting within their local neighborhood as illustrated in
Fig. 1. Looking into local interaction at different regions, e.g., pedestrian
$\leftrightarrow$ car a and pedestrian $\leftrightarrow$ car b together
informs how their behavior change from one neighborhood to another across time
from a global respective. Hence, we hypothesize that learning relation from
spatiotemporal region-wise interaction with respect to a target agent informs
intent. We develop a framework that learns relation from pedestrian visual
interaction and motion dynamics to estimate the probability of binary crossing
intent. The contributions of this paper include:

  1. 1.

We develop a relation extraction module that infers relation between
pedestrian and surroundings from region-wise local interaction.

  2. 2.

We show that integrating region-wise relation learning into prediction
framework achieves competitive performance against existing methods. The
improvement in F1-score and recall suggests lower ratio of missed crossing
predictions.

##  2 Related Work

Pedestrian Intent prediction The objective of pedestrian intent prediction,
specifically crossing prediction, is to anticipate whether a target pedestrian
will cross the road at some time in the future. [9] extends DenseNet to
extract spatio-temporal features from image sequences to estimate pedestrians’
crossing actions. Many algorithms rely on various input modalities and
architectures [10, 8, 11]. For instance, the framework in [10] utilizes
traffic scene features and pedestrians looking and walking attributes by
convolutional networks to forecast crossing action. Some recent methods
utilize recurrent modeling [8, 12]. The authors in [8] develop a framework
with different input modalities including pedestrian trajectories, poses,
traffic scenes, and fuse them hierarchically within a stacked recurrent model.

Relation Modeling Some studies model the dependency between pedestrians and
traffic surroundings with object-level representations. For instance, the
authors in [13] design a module to extract object-wise interaction between
various traffic objects based on their pre-detected location and visual
states. Different from their method, we develop a relation extraction module
that aggregates region-wise joint interaction to inform prediction.

##  3 Technical Approach

In this section, we introduce our framework to predict the probability of
pedestrian crossing intent with region-wise relation representations and
motion dynamics. We formalize the intent prediction problem in subsection 3.1.
We describe how relation representations with respect to motion dynamics are
learned through our framework in subsection 3.2.

###  3.1 Problem Definition

We define the pedestrian intent prediction as binary classification that
predicts whether a target pedestrian will cross the road in the future based
on the observation data. Without loss of generality, we assume that the
observed trajectory of a target pedestrian is obtained from an object
detection and tracking framework in advance. Let
$X_{i}=[X_{i}^{t-\tau+1},...,X_{i}^{t}]$ denote the past trajectory of
$i^{th}$ pedestrian from time $t-\tau+1$ to time $t$, where
$X_{i}^{t}=(u_{i}^{t},v_{i}^{t},w_{i}^{t},h_{i}^{t})\in{\rm I\\!R}^{4}$
denotes the bounding box center, width, height of $i^{th}$ pedestrian at time
$t$. Given the past trajectory $X_{i}$ along with the observed frames
$I=[I^{t-\tau+1},...,I^{t}]$, we estimate the $i^{th}$ pedestrian’s crossing
probability $\hat{y}_{i}$.

Fig. 2: Joint information between object and surroundings is encoded within
feature maps by a visual encoder.

###  3.2 Framework Overview

Our prediction framework consists of three major components: (1) a visual
encoder to extract region-wise interaction evolving along time; (2) a relation
extraction module that jointly relate local interaction from different regions
to obtain relation from a global perspective; (3) a trajectory recurrent
encoder that encodes pedestrian’s observed motion dynamics. We elaborate each
component in more details in the following.

Visual Encoder We employ a visual convolutional encoder that extracts local
joint interaction between object and surroundings from observation frame
sequence $I$ as shown in Fig.2. The visual encoder can be divided into two
parts: (1) a spatial module that encodes frame-level joint local interaction
within feature-map vectors; (2) a temporal module that aggregates frame-level
feature-map vectors to extract spatiotemporal local interaction. The spatial
module takes $\tau$ observed frames $I=[I^{t-\tau+1},...,I^{t}]$ as inputs and
passes them through a 2D ResNet34 [14] backbone $G_{sf}$ to generate frame-
level features $f_{s}\in{\rm I\\!R}^{\tau\times h\times w\times c}$. The
feature vectors (cells) in $f_{s}$ encode joint interaction within local
regions, i.e, the feature vector in orange encodes pedestrian-car information.
To capture temporal changes of joint information locally, we pass $f_{s}$
through a temporal module $G_{tf}$ to generate spatiotemporal local
interaction representations $f_{st}$. Specifically, we employ $\tau\times
1\times 1$ convolutional 3D filters with $c=$512 channels to aggregate cells
$f_{s}^{(k)}\in{\rm I\\!R}^{\tau\times 1\times 1\times c}$ in $f_{s}$ along
temporal axis to generate $f_{st}\in{\rm I\\!R}^{h\times w\times c}$, within
which a cell is a $c$-dimensional spatiotemporal feature vector
$f_{st}^{(k)}\in{\rm I\\!R}^{c}$.

Fig. 3: The detailed architecture of our intent prediction framework. Each
pairwise relation between $f_{st}^{m}$ and $f_{st}^{n}$ is conditioned on past
agent-specific motion states $q_{i}$.

Relation Extraction For a target pedestrian, relating local interaction from
different cells $f_{st}^{(k)}$ (e.g., car-car and pedestrian-car) informs how
behavioral interaction changes from one local region to another when objects
moving around in the scene. While the visual encoder encodes spatiotemporal
local interaction representations $f_{st}^{(k)}$, it less explicitly relate
region-wise representations to jointly learn relation. Hence, we employ a
relation extraction module to extract global-wise relation from feature
vectors $f_{st}^{(1)},...f_{st}^{(k)},...,f_{st}^{(h\times w)}$ in $f_{st}$.
Inspired by [15], we extend the notion of entity to a spatiotemporal feature
representation encoded from a local region across time, which is
$f_{st}^{(k)}$ in our case. Hence, we define an entity set as
$f_{st}=\\{f_{st}^{(1)},...f_{st}^{(k)},...,f_{st}^{(h\times w)}\\}$.
Following such definition, we learn the relation between entities in $f_{st}$
via a composite function $R(*)$ as following:

$R(f_{st})=f_{\phi}\left(\sum_{m,n}g_{\theta}\left(f_{st}^{m},f_{st}^{n}\right)\right)$
(1)

where $g_{\theta}(*)$ denotes a pair-wise relation function that takes a pair
of feature vectors $f_{st}^{m},f_{st}^{n}$ in $f_{st}$ as input to learn their
relation. The outcomes of $g_{\theta}(*)$ from all feature vector pairs are
then aggregated to be passed through $f_{\phi}(*)$ to infer relations
globally. In general, $f_{\phi}(*)$ and $g_{\theta}(*)$ can be simple neural
network functions, e.g., multi-layer perceptrons (MLPs). We use one linear
layer as $g_{\theta}$ and an identity function as $f_{\phi}$. In the high
level, $R(*)$ receives pair-wise features that represents region-wise local
interaction, if any, and integrate them to form global relation.

We introduced how to model global relation from region-wise interactions via a
relation module. We then elaborate on how we incorporate extracted relation
representations $R(f_{st})$ into our prediction framework.

Trajectory Encoding The past observation of pedestrian’s bounding box
implicitly captures the motion dynamics in relative distance to the
surroundings, which is informative to relation learning. Thus, We employ a
recurrent encoder to extract motion dynamics of a target pedestrian. The
encoder consists of gated recurrent union (GRU) with hidden size 256 that
generates motion hidden states $q_{i}$ from i-th pedestrian’s trajectory
$X_{i}$, as shown in the lower part of Fig. 3.

The relation features $R(f_{st})$ are agent-agnostic representations. Given
the same observation frames $I$ with multiple pedestrians and traffic objects,
the intentions of different pedestrians can be influenced by different
relation context. To learn agent-specific relation context for each target
pedestrian, $R(f_{st})$ should be conditioned on pedestrian’s historical
motion dynamics. Hence, we condition relation learning on $q_{i}$ for $i$-th
pedestrian and use the trajectory-conditioned relation features
$R(f_{st},q_{i})$ to predict intent, as shown in the relation extraction block
in Fig. 3. Specifically, we concatenate $q_{i}$ to every concatenated pair of
$f_{st}^{m},f_{st}^{n}$ and pass triplets $(f_{st}^{m},f_{st}^{n},q_{i})$ to
$g_{\theta}$ and $f_{\phi}$ to obtain $R(f_{st},q_{i})$. We append a MLP as
classifier to estimate the intent probability $\hat{y}_{i}$ given relation
features $R(f_{st},q_{i})$.

Learning Objectives We employ the binary cross-entropy objective for intent
estimation $\hat{y}_{i}$ and ground-truth intent $y_{i}$ for each pedestrian.
The loss function can be expressed as following:

$\mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}(y_{i}\log{\hat{y}_{i}}+(1-y_{i})\log({1-\hat{y}_{i}}))$
(2)

where $N$ denotes the total number of training pedestrians.

##  4 Experiments

In this section, we empirically evaluate our intent prediction framework on
two public datasets: Joint Attention for Autonomous Driving (JAAD) [16] and
Pedestrian Intention Estimation (PIE) [17] that contain scenarios where
pedestrians approaching the road and might cross in the future. We provide a
comparative study to demonstrate the efficacy of our approach and an ablation
discussion on relation extraction.

Table 1: Results comparison of our approach with other methods on JAAD and PIE datasets. Methods | PIE | JAAD
---|---|---
Acc | AUC | F1 | P | R | Acc | AUC | F1 | P | R
SF-GRU[8] | 0.82 | 0.79 | 0.69 | 0.67 | 0.70 | 0.84 | 0.84 | 0.65 | 0.54 | 0.84
ARN[13] | 0.84 | 0.88 | 0.90 | 0.96 | 0.85 | 0.87 | 0.92 | 0.70 | 0.66 | 0.75
Ours (w/o relation) | 0.82 | 0.81 | 0.84 | 0.89 | 0.80 | 0.81 | 0.84 | 0.70 | 0.63 | 0.82
Ours | 0.84 | 0.85 | 0.90 | 0.94 | 0.86 | 0.84 | 0.89 | 0.74 | 0.67 | 0.84

###  4.1 Datasets and Experimental Setting

Datasets The two datasets contain video sequences collected in urban driving
scenes. JAAD contains 323 video clips with total 2786 pedestrians. PIE
contains long videos from 6 sets covering 6 hours of driving footage with
total 1842 pedestrians. There are 2100 not-crossing pedestrians and 686
crossing pedestrians in the JAAD dataset. PIE contains 1322 non-crossing
pedestrians and 512 crossing ones. The pedestrian tracks in both datasets are
annotated at 30Hz.

Experimental Setting Throughout our experiments, we follow the same data
preprocessing procedure and evaluation protocol as previous work [17, 8]. We
use the standard data splits of JAAD as in [18] where the dataset is divided
into 177 training clips, 29 validation clips and 117 test clips, respectively.
The pedestrian tracks are divided into 1355, 202 and 1023 for train/val/test
splits in the JAAD data. For the PIE dataset, we follow the same data splits
defined in [17]: videos from set01, set02 and set06 are used for training,
set04 and set05 for validation and set03 for test. The number of pedestrian
tracks in PIE is 880, 243 and 719 in training, validation and test sets. To
predict the crossing probability of a pedestrian, we subsample each pedestrian
track into multiple observation sequences with fixed length of $\tau=$16
frames (0.5s observation). The sample overlap ratio is set to 0.6 for PIE and
0.8 for JAAD dataset. The last frame of each subsampled observation sequence
falls between 30 and 60 frames (or 1s – 2s) prior to the annotated starting
point of a crossing event, as illustrated in Fig. 4. Such time-to-event (TTE)
setting allows prediction beforehand as some time should be allotted for
emergency maneuver.

Fig. 4: Subsampling for pedestrian crossing prediction: sample observation
frames prior to the starting point of crossing. We subsample the observation
data by varying time-to-event.

Evaluation Metrics. To evaluate the performance of our prediction framework,
we use several binary classification metrics: accuracy, area under the curve
(AUC), precision, recall and F1-score.

###  4.2 Results

We compare our method with two other state-of-the-art prediction methods as
following:

  1. 1.

SF-GRU[8]: a framework that hierarchically fuses features from trajectories,
poses and traffic scenes within a stacked recurrent model.

  2. 2.

ARN[13]: a framework that extracts object-wise interaction from object
location and visual states.

The evaluation results of state-of-the-art methods and ours are shown in Table
1. Our method outperforms SF-GRU model in several metrics, and achieves
similar or superior performance in recall and F1-score on both datasets.
Specifically, our model results in 0.04 increase in F1-score on JAAD dataset
and 0.01 increase in recall on PIE dataset. The improvement in recall and
F1-score implies lower ratio of missed crossing predictions. Similar trend can
be observed when comparing the fusion model SF-GRU and the interaction model
ARN. This indicates that incorporating relation learning into prediction
framework helps to alarm the intelligent system about potential risks. Our
method yields lower acc and AUC on JAAD. The pedestrians in JAAD are close to
the curbside with similar trajectories while less interaction with traffic
infrastructures. The relation representations $R(f_{st},q_{i})$ on JAAD is
majorly influenced by the less discriminative $q_{i}$. Hence, there are fewer
True Negatives, leading to the drop of acc and AUC.

To investigate the effect of relation extraction within our framework, we
conduct an ablation experiment by removing the relation extraction. We employ
a model that consists of the same visual encoder, the same trajectory encoder
and the same final classifier on the same inputs. However, we replace the
relation extraction ($g_{\theta}$ and $f_{\phi}$) with an MLP with same number
of layers. This MLP connects to the concatenated features of full feature maps
$f_{st}$ and $q_{i}$, instead of triplets of vectors
$(f_{st}^{m},f_{st}^{n},q_{i})$. Thus, there are more parameters in this
model. We include the results of this non-relation model in the third row of
the Table1 as Ours (w/o relation). The improved performance by using relation
extraction confirms its contribution in the estimation of pedestrian crossing
intent.

##  5 Conclusion

In this work, we develop a framework with trajectory-conditioned relation
learning to predict pedestrian crossing behavior. A relation extraction module
is developed to learn relations between a target pedestrian and surroundings
from region-wise interaction through our framework. We evaluate our method on
two public intent prediction datasets and compare against state-of-the-art
approaches. The experimental results demonstrate that our framework helps to
inform potential risks during crossing events with 0.04 increase in F1-score
on JAAD dataset and 0.01 increase in recall on PIE dataset. We further conduct
ablation experiments to verify the contribution of the relation extraction in
our framework.

## References

  * [1] Mohit Prabhushankar and Ghassan AlRegib,  “Extracting causal visual features for limited label classification,”  in 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021, pp. 3697–3701.
  * [2] Jinsol Lee and Ghassan AlRegib,  “Open-set recognition with gradient-based representations,”  in 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021, pp. 469–473.
  * [3] Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib,  “Novelty detection through model-based characterization of neural networks,”  in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 3179–3183.
  * [4] Charles Lehman, Dogancan Temel, and Ghassan AlRegib,  “On the structures of representation for the robustness of semantic segmentation to input corruption,”  in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 3239–3243.
  * [5] Ryan Benkert, Oluwaseun Joseph Aribido, and Ghassan AlRegib,  “Explaining deep models through forgettable learning dynamics,”  in 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021, pp. 3692–3696.
  * [6] Ahmad Mustafa and Ghassan AlRegib,  “Man-recon: Manifold learning for reconstruction with deep autoencoder for smart seismic interpretation,”  in 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021, pp. 2953–2957.
  * [7] Dogancan Temel, Melvin J Mathew, Ghassan AlRegib, and Yousuf M Khalifa,  “Relative afferent pupillary defect screening through transfer learning,”  IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 3, pp. 788–795, 2019.
  * [8] Amir Rasouli, Iuliia Kotseruba, and John K Tsotsos,  “Pedestrian action anticipation using contextual feature fusion in stacked rnns,”  arXiv preprint arXiv:2005.06582, 2020.
  * [9] Khaled Saleh, Mohammed Hossny, and Saeid Nahavandi,  “Real-time intent prediction of pedestrians for autonomous ground vehicles via spatio-temporal densenet,”  in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 9704–9710.
  * [10] Amir Rasouli, Iuliia Kotseruba, and John K Tsotsos,  “Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior,”  in Proceedings of the IEEE International Conference on Computer Vision Workshops, 2017, pp. 206–213.
  * [11] Amir Rasouli, Tiffany Yau, Mohsen Rohani, and Jun Luo,  “Multi-modal hybrid architecture for pedestrian action prediction,”  arXiv preprint arXiv:2012.00514, 2020.
  * [12] Smail Ait Bouhsain, Saeed Saadatnejad, and Alexandre Alahi,  “Pedestrian intention prediction: A multi-task perspective,”  arXiv preprint arXiv:2010.10270, 2020.
  * [13] Yu Yao, Ella Atkins, Matthew Johnson Roberson, Ram Vasudevan, and Xiaoxiao Du,  “Coupling intent and action for pedestrian crossing behavior prediction,”  arXiv preprint arXiv:2105.04133, 2021.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [15] Adam Santoro, David Raposo, David GT Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap,  “A simple neural network module for relational reasoning,”  arXiv preprint arXiv:1706.01427, 2017.
  * [16] Iuliia Kotseruba, Amir Rasouli, and John K Tsotsos,  “Joint attention in autonomous driving (jaad),”  arXiv preprint arXiv:1609.04741, 2016.
  * [17] Amir Rasouli, Iuliia Kotseruba, Toni Kunic, and John K Tsotsos,  “Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6262–6271.
  * [18] Amir Rasouli, Iuliia Kotseruba, and John K Tsotsos,  “It’s not all about size: On the role of data properties in pedestrian detection,”  in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0–0.