# Hybrid Approach for 3D Head Reconstruction: Using Neural Networks and Visual
Geometry

Oussema Bouafif12, Bogdan Khomutenko1 and Mohamed Daoudi2  1MCQ-Scan, Lille,
France  2IMT Lille Douai, Univ. Lille, CNRS UMR 9189 CRIStAL, Lille, France
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>mohamed.daoudi@imt-
lille-douai.fr

###### Abstract

Recovering the 3D geometric structure of a face from a single input image is a
challenging active research area in computer vision. In this paper, we present
a novel method for reconstructing 3D heads from a single or multiple image(s)
using a hybrid approach based on deep learning and geometric techniques. We
propose an encoder-decoder network based on the U-net architecture and trained
on synthetic data only. It predicts both pixel-wise normal vectors and
landmarks maps from a single input photo. Landmarks are used for the pose
computation and the initialization of the optimization problem, which, in
turn, reconstructs the 3D head geometry by using a parametric morphable model
and normal vector fields. State-of-the-art results are achieved through
qualitative and quantitative evaluation tests on both single and multi-view
settings. Despite the fact that the model was trained only on synthetic data,
it successfully recovers 3D geometry and precise poses for real-world images.

##  I Introduction

In the last few years, several computer vision applications have used 3D face
models. Three-dimensional face model reconstruction encounters different
challenges, such as difficult orientations and illumination variations [1].
Several applications such as facial recognition [2], facial animation [3],
gender classification [4], and facial expression recognition [5] have achieved
better results using 3D face models. There is a number of different techniques
to tackle the 3D face reconstruction problem. Some approaches are based on
structure from motion, optical flow, or shape from shading [6]. However, these
approaches all have limitations since most of them are sensitive to lighting
conditions, reflections, shadows, and image quality.

Recently, machine learning has been used to solve these problems [7, 8].
Neural networks are good at implicit modeling of complex variations of
lighting, shading, and so on, which leads to a robust geometry estimation. On
the other hand, one of the most well-known difficulties in applying neural
networks is the lack of 3D face data sets. In some cases, this problem makes
end-to-end learning systems less efficient than geometric methods. The use of
synthetic data or fitted 3D models from [6] has been proposed in many
approaches to meet this need, even if the latter produces only an
approximation of the ground truth, which may lead to a low reconstruction
accuracy. Another drawback of end-to-end machine-learning-based approaches is
the lack of control over the reconstruction process.

Another challenge in 3D face reconstruction is geometry representation. One
possible way is a 3D Morphable Model [9] (3DMM). Such models guarantee that
the result will be a plausible head reconstruction, and at the same time they
are flexible and expressive enough to represent a wide variety of face
morphologies. The challenge is to estimate the model parameters. Several
approaches to fit morphable models have been proposed [10, 3, 11]. They are
usually based either on a set of discrete landmarks [12] or on contours [13].
We argue that pixel-wise fitting (used in some other 3D reconstruction
applications [14]) is the best way to exploit as much information present in
the image as possible. But the application of this technique to 3D head
reconstruction is problematic. In the classical photometric approach, we would
require texture and lighting model, which are generally not available.

In this paper, we propose a hybrid approach composed of deep learning and
geometric optimization methods (See Fig. 1), capable of reconstructing a 3D
head model from one or multiple facial images. The key concept is that surface
normals have a high degree of invariance (scale, translation), they don’t
depend on texture, shadows, and lighting; and they convey rich information
about geometry. First, we use an encoder-decoder network that translates a
facial input image into a landmarks map ($\cal Z$) and a facial normal map
($\cal N$). Then, using these maps in a parametric regression algorithm, we
reconstruct the 3D facial model.

The main contributions of this paper are:

  * •

The proposed approach predicts both pixel-wise normal vectors and landmarks
maps from a single input photo.

  * •

We present a mesh fitting strategy based on surface normal vectors which
address the reconstruction of the head directly from a single or multiple
images and aims to recover a complete craniofacial human head as well as its
pose.

  * •

We show that our approach achieves state-of-the-art performance on the BU-3DFE
[15] data set. We show that the proposed model generalizes well to real-world
images, even though it has been trained only on synthetic data.

Figure 1: Overview of our proposed 3D face reconstruction method. Given an
input facial image (a), we estimate two different maps (normal surface map
$\cal N$ (b), Landmarks map $\cal Z$ (c)) used to reconstruct the 3D facial
shape via a fitting process with the LYHM [16] morphable model.

##  II Related Work

###  II-A Morphable Models

The use of 3D morphable models (3DMM) is one of the fundamental approaches
proposed to address 3D face reconstruction from a single input image. The most
well-known model is perhaps the BFM model [9]. This model is constructed using
Principal Component Analysis (PCA) given a large set of 3D faces. By changing
the model parameters, one can obtain a variety of 3D facial geometries.

###  II-B Deep Learning Face Reconstruction Methods

In the last few years, Convolutional Neural Networks (CNNs) have demonstrated
better performances with fitting the 3DMM to an input image. For example,
fitting via inverse rendering as it was shown in [3, 11], or regressing the
3DMM shape parameters directly from an input image [17].

Other techniques directly map the input photo onto diverse kinds of images
which contain information on the geometry, such as [10], which produces a
depth map and a 3D correspondence map. After that, a template mesh is deformed
in order to fit these produced maps. In [7] Richardson et al. proposed
CoarseNet and FineNet to provide both coarse and fine details of facial shape.
Another method was proposed by [18], which learns 3D face curves from
horizontal and vertical epipolar plane images (EPI) of light field images
utilizing a densely connected network (FaceLFnet). This model-free approach
produced curves that are combined together to obtain a more accurate combined
3D point cloud. An end-to-end 3D face reconstruction method has been proposed
in [8], where the problem was divided into two sub-tasks: predicting the
neutral 3D face and the expression parameters. In [19] Wu et al. proposed a
multi-view facial approach based on an end-to-end trainable CNN to regress
3DMM parameters via a differentiable dense optical flow estimator for the
alignment and the photometric losses. In [20], a transfer function between an
input image and the UV map was learned using a CNN. This UV map is a 2D
representation of the 3D shape of the 3DMM. A direct approach was proposed in
[21] to reconstruct the 3D face with the use of a volumetric CNN regression.

To be able to use an unlimited amount of data for training, unsupervised
learning methods were proposed. A differentiable renderer was used in [22] for
the optimization task after learning 3DMM parameters from facial identity
encoding. In [23], RingNet was trained using a shape consistency loss similar
to the triplet loss to regress all images of the same person to a unique
latent form vector. A novel deep 3D face reconstruction approach in [24] uses
a hybrid loss function for weakly-supervised learning.

More recently, Wang et al. [25] uses an off-the-shelf face recognition neural
network trained on high-quality synthetic data to fully reconstruct facial
geometries based on a single selfie. Inspired from [26] and based on a GAN
model, Face-Normal-Net [27] learned from a synthetic data set to regress a
normal map, used in a weighted least square solver to reconstruct a surface
approximating the 3D shape. The DF2Net network [28] uses three combined
modules, which are each trained on a separate type of data set with different
training strategies. The model progressively refines the subtle facial details
like small crow’s feet and wrinkles. MMFace [29] is a multi-metric regression
network composed of two sub-networks: a volumetric sub-network to estimate an
intermediate face geometry and a parametric sub-network to infer the
corresponding 3DMM parameters.

###  II-C Normal Map Prediction

Multiple works in computer vision have been proposed to solve tasks such as
depth map or surface curvature prediction, semantic segmentation, and edge
detection. Among these works, prediction of the normal map has been studied by
different learning-based approaches since producing high-quality normal maps
for complex objects represents a challenging task. Several methods have been
dedicated to the generation of normal maps from (outdoor/indoor) scenes or
objects. In [30], a skip-network model has been trained to predict surface
normal which is the input to another two networks that estimate the pose and
the style of the object image. In [31], $DeepLiDAR$ has been proposed to
predict depth map with a normal surface of outdoor scenes from sparse LiDAR
Data and a gray-scale image.

Another category of normal-map-prediction methods has been proposed, where the
network takes sketches as input images. In [32], Hudon et al.propose a multi-
scale representation of the input images to ensure accuracy of produced maps.
An interactive method was proposed by Su et al. [33] where they proposed a
conditional GAN framework based on U-Net model [34]. As we present here,
another type of approach has proposed to predict normal maps from facial
images to use it in a 3D reconstruction task. In [26], a fully convolutional
network trained on various synthetic and real-world data sets. The obtained
normal map is then used in the Frankot-Chellappa method [35] to reconstruct 3D
facial shape. In [36], a novel architecture SfSNet learns from a mixture of
labeled synthetic and unlabeled real images to solve the problem of inverse
face rendering. It utilizes residual blocks to disentangle normal and albedo
maps into separate subspaces. More recently, Bouafif et al. [27] trained a GAN
model to generate normal maps from synthetic facial images which are used in a
variational 3D reconstruction method. In [37], a new cross-modal learning
architecture was proposed to resolve the limited available ground truth data.
Core to this approach is a novel module called deactivable skip connections,
which allows integrating both the auto-encoded and image-to-normal branches
within the same architecture that can be trained end-to-end. This allows
learning of a rich latent space that can accurately capture the normal
information.

###  II-D Training With Synthetic Data

Several facial analysis approaches like facial recognition [38], face
detection [39] and facial expression analysis [40] have utilized synthetic
data to overcome the lack of large-scale training data during the use of
neural networks. 3D face reconstruction is among these applications. In [17,
41], a semi-synthetic data set was produced by implementing an optimization-
based algorithm with proven precision to a database containing real-world
faces. Other approaches [11] have also reinforced this track by integrating a
technique of inverse rendering. Another option was brought up by [7, 8, 10,
28, 42], which consists in producing a full synthetic data set. For example,
they use the BFM morphable model [9] for identities with Face Warehouse [43]
or 3DDFA [44] model for expressions and producing with it, several types of
images according to the need of the method. However, the performance of these
methods might be limited by lighting models used for rendering, as well as the
representativity of the subjects participated in the acquisition to create the
model.

To deal with some of these problems, a short time ago, novel approaches have
been proposed to boost realism in 3D synthetic face data sets. To achieve
that, more productive models and more accurate rendering techniques are
suggested. [18] proposed a photo-realistic light field image synthesis method
to generate a large-scale Epipolar Plane Images (EPI) data set selecting some
examples from the BU-3DFE [15]. In [25], Wang et al.proposed an augmented 3D
head data set with UV texture and a high-quality engine for rendering. More
recently, the LYHM [16] morphable model was used in [27] with hair model [45]
and texture to construct a synthetic data head generator that can be used in
various approaches related to facial analysis.

##  III Proposed Method

In this section, we describe the details of our proposed framework as shown in
Fig.1. Our 3D reconstruction method takes a facial image as the only input and
produces two output maps from an encoder-decoder model. These two maps are a
normal surface map $\cal N$ and a map which contains facial landmarks $\cal
Z$. We use these outputs in a 3D fitting algorithm to recover the 3D Model of
the head. In Section III-A, we describe our synthetic 3D head generator which
allowed us to produce our training data set. In Section III-B, we show our
network architecture which regresses two different maps that are aligned with
the input image. The first is a normal surface map ($\cal N$). The second
output is a map that contains facial landmarks ($\cal Z$). Finally, we explain
the details of 3DMM fitting in Section III-C.

Figure 2: Training data samples. From top to bottom: Synthetic facial images,
Normal surface maps $\cal N$ and Landmarks maps $\cal Z$.

###  III-A Synthetic Data Generation

We propose a fully synthetic data set for deep neural network training. To
achieve this, we have set up a synthetic data generator based on the one
described in [27]. The main source of information for geometric fitting is the
normal map $\cal N$ produced by the face generator (second row in Fig. 2).

To ensure alignment during the adjustment process, we have selected 24
vertices on the head model which are distributed over the eyes, nose, mouth,
chin, and ears. After that, the points that are not hidden by the hair or
difficult poses are projected, and the landmark map ($\cal Z$) is generated
(third row in Fig. 2). This map is composed of 24 channels, each one
containing a positive value at the corresponding landmark projection and zeros
everywhere else.

###  III-B Network Architecture

Figure 3: Details of Normal-Landmark Network. Our encoder-decoder architecture
produces two different maps ($\cal N$ and $\cal Z$) (shown on the right) from
a facial input image (shown on the left). The spatial size and number of
layers are shown below and above each block, respectively.

We propose for the network architecture an encoder-decoder based on the Su et
al. [33] model which trained a GAN model to map a normal surface from a sketch
and binary point mask inputs. This model uses the symmetric connection between
the encoder and the decoder to reduce the information loss among successive
layers, thus implementing the U-Net model [34].

In order to adapt the network to our problem, a certain number of changes have
been made. The discriminator has been discarded. For the training data, we
give for the input the facial image (RGB), while we have only two generated
maps $\cal N$ (3 channels) and $\cal Z$ (24 channels). For the loss function,
we use a composed pixel-wise loss to measure the difference between the
generated maps and the real input ones as described here:

$\begin{gathered}\mathrm{Loss}=L_{\cal N}+L_{\cal Z}\\\ L_{\cal N}=||{\cal
N}_{\textrm{GT}}-{\cal N}||_{2}^{2}\quad,\quad L_{\cal Z}=||{\cal
Z}_{\textrm{GT}}-{\cal Z}||_{2}^{2}\end{gathered}$ (1)

Where it enforces the rendered images $\cal N$ and $\cal Z$ to be similar to
the inputs $\cal N_{\textrm{GT}}$ and $\cal Z_{\textrm{GT}}$ respectively. The
architecture of Normal-Landmark Network is illustrated in Fig. 3.

####  III-B1 Training Details

To train our encoder-decoder model, we generate 60,000 facial images scaled to
$128$ x $128$ (equally distributed between males and females) and their
corresponding $\cal N$ and $\cal Z$ maps. We train the model for about $3000$
epochs with a learning rate of $1e-5$, $32$ as batch size, and we adopt
RMSprop as the optimizer. To make the training images more realistic we add
random blur effect and Gaussian noise as data augmentation. When minimizing
energy function in (7), we use $\lambda_{\cal N}=1$ , $\lambda_{\cal Z}=0.8$
and, $\lambda_{P}=0.4$.

###  III-C 3D Morphable Model fitting

The core idea of the present work is the geometry-based fitting of a morphable
model to a predicted field of normals, using a camera projection model. This
subsection describes in details the problem formulation.

####  III-C1 3D Morphable Model

In our reconstruction method, we use the LYHM [16] morphable model. By adding
parameterized deformations to the mean face model $X_{0}$, we can generate
variability in head identity, as follows:

$\displaystyle X$ $\displaystyle=X_{0}+W\boldsymbol{y}$ (2)

Where $X\in\mathbb{R}^{3\times N_{\\!X}}$ is the generated 3D head with
$N_{\\!X}=11,510$ vertices; $X_{0}\in\mathbb{R}^{3\times N_{\\!X}}$ is the
mean face, computed over the aligned facial 3D scans in the Headspace [46]
collection; $\boldsymbol{y}\in\mathbb{R}^{N_{\\!y}}$ is the identity parameter
vector with $N_{\\!y}=100$; $W\in\mathbb{R}^{3\times N_{\\!X}\times N_{\\!y}}$
is a tensor representing principal components of the shape model.

####  III-C2 Morphable Model’s Normals

In order to fit the normal map $\cal N$, the normal vector
$\boldsymbol{n}_{i}$ is calculated for each vertex location
$\boldsymbol{p}_{i}\in\mathbb{R}^{3}$, given the set of adjacent vertices
$\left\\{\boldsymbol{q}_{i,1},\boldsymbol{q}_{i,2},...,\boldsymbol{q}_{i,M}\right\\}\subset\mathbb{R}^{3}$.
First an unnormalized version $\tilde{\boldsymbol{n}}$ is computed:

$\begin{gathered}\tilde{\boldsymbol{n}}_{i}=\sum_{j=1}^{N_{\\!X}}\frac{(\boldsymbol{q}_{i,j}-\boldsymbol{p}_{i})\times(\boldsymbol{q}_{i,j+1}-\boldsymbol{p}_{i})}{\|(\boldsymbol{q}_{i,j}-\boldsymbol{p}_{i})\times(\boldsymbol{q}_{i,j+1}-\boldsymbol{p}_{i})\|}\end{gathered}$
(3)

Then the actual normal $\boldsymbol{n}_{i}$ is obtained by a simple vector
normalization:

$\boldsymbol{n}_{i}=\frac{\tilde{\boldsymbol{n}}_{i}}{||\tilde{\boldsymbol{n}}_{i}||}$
(4)

This is not the most exact method of computing normals, but it is fast, which
is important since it is computed at every optimization iteration.

####  III-C3 Projection Model

In order to get the reference normal for each vertex $\boldsymbol{p}$, the
latter is projected onto the image plane of $\cal N$ using the Pinhole camera
model. The bi-cubic interpolation is used to get the corresponding reference
normal from $\cal N$ (which is a discrete grid). The projection process can be
expressed as follows:

$\begin{gathered}\boldsymbol{a}=\frac{1}{z}KR(\boldsymbol{p}+\boldsymbol{t})\end{gathered}$
(5)

Where $\boldsymbol{a}\in\mathbb{R}^{2}$ is the projected point;
$R\in\mathrm{SO(3)}$ is the rotation matrix, parametrized by the three
rotation angles of roll, pitch, and yaw, and denoted by $\boldsymbol{r}$; and
$\boldsymbol{t}\in\mathbb{R}^{3}$ is a translation vector. In this context $z$
in the denominator represents the normalization of a 3D point to bring it to
the normal plane ($z$ is taken after multiplying the vector by $R$ but before
multiplying it by $K$). The projection matrix K contains three parameters
$f,u_{0},v_{0}$ and defines the projection model:

$K=\begin{pmatrix}f\;&0\;&u_{0}\\\ 0\;&f\;&v_{0}\end{pmatrix}$ (6)

Since the camera calibration is approximative, we think that it is enough to
have only three intrinsic camera parameters.

####  III-C4 Parametric Regression

Our implementation follows standard practices (for example, [11, 7, 10, 17])
which use the morphable model in a fitting process. It aims to form an image
as close as possible to a target image by finding the most suitable
combination of parameters. The target images ($\cal N$ and $\cal Z$) are
produced from our encoder-decoder network described in Section III-B. The
problem can be expressed as the minimization of an energy function that
represents the error between the produced maps and those generated by the
morphable model. Our energy function contains three main components $E_{\cal
N}$, $E_{\cal Z}$ and $E_{P}$:

$\begin{gathered}E=\lambda_{\cal N}E_{\cal N}+\lambda_{\cal Z}E_{\cal
Z}+\lambda_{P}E_{P}\\\ \end{gathered}$ (7)

Let us describe each term. $E_{\cal N}$ represents the difference between
interpolated normals $\cal N(\boldsymbol{a})$ from the 2D projection of the
morphable model onto $\cal N$ and vertex normals $\boldsymbol{n}$ which are
computed from the Morphable model using (3). $E_{\cal N}$ is defined by:

$E_{\cal N}=\dfrac{1}{2}\sum_{i=1}^{N_{\\!X}}||{\cal
N}(\boldsymbol{a}_{i})-\boldsymbol{n}_{i}||^{2}$ (8)

The landmarks loss $E_{\cal Z}$ is defined as the distance between the
detected landmarks $\boldsymbol{z}_{j}$ and the projections
$\boldsymbol{b}_{j}$ of the corresponding subset of 3D vertices using the same
projection model described in Eq.5.

$\begin{gathered}E_{\cal
Z}=\dfrac{1}{2}\sum_{j=1}^{N_{z}}||\boldsymbol{z}_{j}-\boldsymbol{b}_{j}||^{2}\end{gathered}$
(9)

Finally, the shape prior loss $E_{P}$ ensures plausibility of reconstructed
heads by assuming priors given by the statistical head model represented by
singular values $\sigma_{k}$. In most 3DMM reconstruction methods [9, 22, 11,
10, 25, 19], this term is used to prevent the degeneration of the
reconstructed geometry:

$\begin{gathered}E_{P}=\dfrac{1}{2}\boldsymbol{y}^{T}{\cal
C}^{-1}_{y}\boldsymbol{y}\end{gathered}$ (10)

${\cal C}_{y}$ is the covariance matrix, which in this case is a diagonal
matrix containing
$\sigma_{1}^{2}\\!,\sigma_{2}^{2}\\!,..\sigma_{N_{\\!y}}^{2}$.

The goal of the parameter regressor is to predict morphable model parameters
$y$ and those related to the projection of the model from 3D space to 2D one.
This can be formulated as a minimization problem as the following:

${\boldsymbol{y}}^{*},{\boldsymbol{t}}^{*},{\boldsymbol{r}}^{*},{K}^{*}=\underset{{\boldsymbol{y}},\boldsymbol{r},\boldsymbol{t},K}{\mathrm{argmin}}\;E$
(11)

where $\boldsymbol{y}$ is the head shape parameter vector, $\boldsymbol{t}$
and $\boldsymbol{r}$ are the translation and rotation parameters defining the
head pose with respect to the camera, $K$ is the intrinsic camera matrix. This
is a non-linear least-square problem which can be solved efficiently using the
Levenberg-Marquardt algorithm. The Ceres solver [47] is used as the
optimization back-end.

####  III-C5 Head Pre-Alignment

If we use only normals to fit the head model, we may encounter a certain
number of convergence issues. First, it might take some time for the optimizer
to converge to the desired position and orientation. Second, The optimization
might get stuck in a local minimum. It is a good thing for all local-search
optimizers to have a good initial guess. We can achieve it by computing a
plausible initial pose. It is done by optimizing $E_{\cal Z}$ over
$\boldsymbol{t}_{0},\boldsymbol{r}_{0}$, while fixing $\boldsymbol{y}$ to 0
and $K$ to a certain prior value, in other words, by using the mean head model
and landmarks given by the network. Since it is a purely geometric
optimization problem, it almost always converges to the global solution and
does it quickly, considering the low number of errors (up to 48 in our case)
and unknowns (6 DoF). This approach allows us to make the system robust with
respect to the exact head pose and orientation in the image.

###  III-D Multi-view 3D Face Reconstruction

Figure 4: An illustration of our method for multi-view fitting. We estimate
the two maps $\cal N$ (b) and $\cal Z$ (c) from each input image (a), then we
use them in the same fitting process to get a unique 3D head reconstruction.

Our approach can be employed to regress 3DMM parameters from multiple facial
images of the same person in different views. Note that our approach can be
generalized to any number of input images. We produce ${\cal N}_{i}$ and
${\cal Z}_{i}$ maps for all input images, then we use output maps in the same
fitting process.

Poses and camera parameters are computed independently for each image while
$\boldsymbol{y}$ is shared for all images. In Fig. 4, we show an example of
3DMM fitting from multiple images of the same person.

##  IV Experiments and Results

In this section, we evaluate the quality of our 3D head reconstruction results
obtained using the proposed framework. Firstly, we show the qualitative
results in Section IV-A. Section IV-B is dedicated to the ability of the
system to reconstruct 3D heads from real-world images by quantitative
experiments on a benchmark database with available ground truth, where we
compare our results to state-of-the-art methods.

###  IV-A Qualitative Evaluation

Figure 5: Examples of visual comparisons from some celebrities facial images
with other methods. Rows contain in order; input image, predicted $\cal N$
map, input image with predicted landmarks (red squares) and dense alignment
results (projected vertices of the 3DMM produced by our fitting process), Ours
(frontal view), Ours (aligned) RingNet [23], PRN [20] and R-C-Nets [24]

In Fig. 5, we show our monocular 3D reconstruction process on the photos of
certain celebrities captured from different poses. This evaluation
demonstrates how the network learns to represent the head structures of real
people. From row 2, we notice our network produces high-quality normal maps as
well as accurate landmarks, although it was trained with fully synthetic data.
It successfully separates the head from the hair and the background. One can
see our method produces high-quality results, which fit the overall structure
well. As we use the LYHM [16] morphable model that includes the cranial part,
our method allows us to recover the 3D model of the head. The cranial region
approximates the areas hidden by the hair whereas we do not estimate the
invisible parts of the face skin.

A comparison between the monocular and a multi-view processes using an example
of the BU-3DFE [15] data set is illustrated in Fig. 6. We can notice a
significant visual resemblance between the 3D reconstruction and the ground
truth using the multi-image process—the 3D reconstruction better captures the
shape of the face. We notice that for some of the other examples in BU-3DFE,
the results are less accurate.

###  IV-B Quantitative Evaluation

For quantitative evaluation, we demonstrate the effectiveness of our approach
using the BU-3DFE [15] data set, which contains 100 subjects (56 female, 44
male), with different ages and a variety of ethnic/racial ancestries, and
where each 3D model has corresponding 2D images captured under controlled
settings. In our evaluation, we use both frontal and profile images of all
subjects to evaluate our algorithm’s performance with both mono and multi-view
reconstruction. Since our 3DMM does not contain expressions, we use only
neutral expressions faces. We performed a rough pre-alignment process between
the reconstructed model and the ground truth using 6 pre-selected vertices.
Next, the alignment and registration processes with the ground truth model are
performed using the iterative closest point (ICP) solver, after that, we
compute the point-to-plane distances and the absolute depth errors between the
reconstructed 3D models and the ground-truth 3D meshes. We eliminate examples
when the alignment process fails (which is about 3-4%).

We use the point-to-plane Root Mean Square Error (RMSE) as the performance
measure. We argue that it is a good way of evaluating performance, which is
not sensitive to either the number of vertices in the model or the way of how
they are distributed. It is computed as follows. Once the models are aligned,
for each vertex $\boldsymbol{p}_{i}$ of the reference model, we compute the
nearest neighbor $\boldsymbol{q}_{i}$ from the reconstructed model to be
evaluated, as well as normal vectors $\boldsymbol{n}_{i}$ of the reference
model. Second, we compute the error projected to the normal, and compute the
RMSE:

$\varepsilon=\sqrt{\frac{\sum_{i=1}^{N}\left((\boldsymbol{p}_{i}-\boldsymbol{q}_{i})\cdot\boldsymbol{n}_{i}\right)^{2}}{N}}$
(12)

where $N$ is the number of vertices of the reference model. Once $\varepsilon$
is computed for each subject, we can find the mean and standard deviation
across the benchmark data set.

####  IV-B1 Results and Discussion

In Table I, we report the numeric results in the form of
$\mathrm{mean}\pm\mathrm{std}$ for the point-to-plane (RMSE) and for the
absolute depth errors, we report mean ($\mu$), standard deviation ($\sigma$),
median ($\widetilde{m}$), and the average ninety percent most significant
error ($\delta_{90\%}$). We have compared our system in both monocular (Mono)
and multi-view (Multi) settings to the state-of-the-art systems with code
available online [23, 20, 24]. We used the exact same procedure for all the
tested systems to make sure that all of them are in the same conditions.

Figure 6: An illustration of stereo and mono fitting from BU-3DFE [15] data
set example. Input images (a), normals surface map $\cal N$ (b), landmarks map
$\cal Z$ (c). (Multi-view fitting): 3D head reconstruction using all images in
the same fitting process. (Mono fitting) : 3D head fitted exploiting only the
frontal image in the fitting process (third row). (GT) : the ground-truth of
the 3D head mesh.

We can see that the mean error of the multi-view approach is lower than that
which we have with the monocular configuration. In most cases, the morphology
of the entire head is better captured using several images. In some other
cases, this multi-view approach gives less good result. And since the
evaluation is only done on the facial part, it does not contribute to the
precision.

One can see that the proposed method is performing at the level of the state
of the art, being clearly outperformed only by the R-C-Nets. The precision
obtained by us for the latter is somehow worse than the one reported in the
corresponding publication [24], which is $1.40\pm 0.31$ for the point-to-plane
(RMSE) and which can be due to some differences in the evaluation technique.

TABLE I: Quantitative comparison on the BU-3DFE [15] data set. Lowers are better Method | Ours (Mono) | Ours (Multi) | RingNet [23] | PRN [20] | R-C-Nets [24]
---|---|---|---|---|---
RMSE | 1.74 $\pm$ 0.44 | 1.67 $\pm$ 0.43 | 1.90 $\pm$ 0.49 | 1.86 $\pm$ 0.47 | 1.60 $\pm$ 0.41
$\mu$ | 2.21 | 2.17 | 3.42 | 1.83 | 1.64
$\sigma$ | 1.08 | 1.04 | 1.58 | 1.70 | 1.69
$\widetilde{m}$ | 2.09 | 2.05 | 3.23 | 1.43 | 1.27
$\delta_{90\%}$ | 3.61 | 3.53 | 5.60 | 3.46 | 3.00

##  V Conclusions

In this paper, we presented a novel approach composed of both deep learning
and visual-geometry-based methods for estimating the complete 3D human head
shape from a single or multiple images. Our method employs an encode-decoder
network that maps the input image to a rasterized normal map $\cal N$ and a
landmark map $\cal Z$. These maps are then used in a fitting process to
regress the 3DMM parameters of the face identity from the LYHM model. To our
knowledge, pixel-wise fitting is generally based on photometric information
and not on output from a neural network, we strongly believe this process is
intrinsically better than integration-based surface-reconstruction methods.

Our network has been trained only on a synthetic facial data set. The network
has shown good results in terms of both accuracy and generalization on real-
world images. In addition to the 3D reconstruction of the head, the landmark
map $\cal Z$ can be directly used for face tracking and pose estimation, an
essential part of our reconstruction pipeline. The landmarks are used to find
a good initial guess for the pose before the fine-grained fitting. It improves
the convergence rate of the regression process and minimizes chances of
reaching a local minimum.

We have performed quantitative and qualitative experiments to evaluate our
pipeline performance. We demonstrate that our proposed framework achieves
state-of-the-art performance in 3D face reconstruction for both single and
multi-view settings. Overall, the multi-view setting gives promising results
but further work is required to take full advantage of it.

Despite the robust performance in many cases, our method has some limitations.
The used 3DMM does not include facial expressions and has a limited age range.
This is why it is difficult to reconstruct the finest details in 3DMM because
the accuracy of the recovered geometry is limited to the flexibility of this
model. This limitation is not fundamental to our proposed method. It can be
overcome by adopting a more expressive morphable model for the synthetic data
generator and for the fitting process. Another limitation is that synthetic
data can have unrealistic features, which in turn can introduce certain biases
in the learning process. The use of GAN architectures in combination with
classic 3D rendering may improve the realism of generated photos.

## References

  * [1] A. F. Abate, M. Nappi, D. Riccio, and G. Sabatino, “2D and 3D face recognition: A survey,” _Pattern Recognition Letters_ , vol. 28, no. 14, pp. 1885–1906, 2007. [Online]. Available: https://doi.org/10.1016/j.patrec.2006.12.018
  * [2] H. Drira, B. Ben Amor, A. Srivastava, M. Daoudi, and R. Slama, “3D face recognition under expressions, occlusions, and pose variations,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 35, no. 9, pp. 2270–2283, 2013. [Online]. Available: https://doi.org/10.1109/TPAMI.2013.48
  * [3] J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Nießner, “Face2face: Real-time face capture and reenactment of rgb videos,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2387–2395.
  * [4] B. Xia, B. Ben Amor, H. Drira, M. Daoudi, and L. Ballihi, “Combining face averageness and symmetry for 3D-based gender classification,” _Pattern Recognit._ , vol. 48, no. 3, pp. 746–758, 2015. [Online]. Available: https://doi.org/10.1016/j.patcog.2014.09.021
  * [5] B. Ben Amor, H. Drira, S. Berretti, M. Daoudi, and A. Srivastava, “4-D facial expression recognition by learning geometric deformations,” _IEEE Trans. Cybernetics_ , vol. 44, no. 12, pp. 2443–2457, 2014. [Online]. Available: https://doi.org/10.1109/TCYB.2014.2308091
  * [6] I. Kemelmacher-Shlizerman and R. Basri, “3D face reconstruction from a single image using a single reference face shape,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 33, no. 2, pp. 394–405, 2010\.
  * [7] E. Richardson, M. Sela, R. Or-El, and R. Kimmel, “Learning detailed face reconstruction from a single image,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1259–1268.
  * [8] P. Dou, S. K. Shah, and I. A. Kakadiaris, “End-to-end 3D face reconstruction with deep neural networks,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 21–26.
  * [9] V. Blanz and T. Vetter, “A morphable model for the synthesis of 3D faces,” in _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_. ACM Press/Addison-Wesley Publishing Co., 1999, pp. 187–194.
  * [10] M. Sela, E. Richardson, and R. Kimmel, “Unrestricted facial geometry reconstruction using image-to-image translation,” _arxiv_ , 2017.
  * [11] Y. Guo, J. Cai, B. Jiang, J. Zheng _et al._ , “Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 6, pp. 1294–1307, 2018.
  * [12] Y. Yanga, X.-J. Wu, and J. Kittler, “Landmark weighting for 3dmm shape fitting,” _arXiv preprint arXiv:1808.05399_ , 2018.
  * [13] A. Bas, W. A. Smith, T. Bolkart, and S. Wuhrer, “Fitting a 3d morphable model to edges: A comparison between hard and soft correspondences,” in _Asian Conference on Computer Vision_. Springer, 2016, pp. 377–391.
  * [14] J. Engel, T. Schöps, and D. Cremers, “Lsd-slam: Large-scale direct monocular slam,” in _European conference on computer vision_. Springer, 2014, pp. 834–849.
  * [15] L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rosato, “A 3D facial expression database for facial behavior research,” in _7th international conference on automatic face and gesture recognition (FGR06)_. IEEE, 2006, pp. 211–216.
  * [16] H. Dai, N. Pears, W. A. P. Smith, and C. Duncan, “A 3D morphable model of craniofacial shape and texture variation,” in _The IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * [17] A. Tuan Tran, T. Hassner, I. Masi, and G. Medioni, “Regressing robust and discriminative 3D morphable models with a very deep neural network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5163–5172.
  * [18] M. Feng, S. Zulqarnain Gilani, Y. Wang, and A. Mian, “3D face reconstruction from light field images: A model-free approach,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 501–518.
  * [19] F. Wu, L. Bao, Y. Chen, Y. Ling, Y. Song, S. Li, K. N. Ngan, and W. Liu, “Mvf-net: Multi-view 3D face morphable model regression,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 959–968.
  * [20] Y. Feng, F. Wu, X. Shao, Y. Wang, and X. Zhou, “Joint 3d face reconstruction and dense alignment with position map regression network,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 534–551.
  * [21] A. S. Jackson, A. Bulat, V. Argyriou, and G. Tzimiropoulos, “Large pose 3d face reconstruction from a single image via direct volumetric cnn regression,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1031–1039.
  * [22] K. Genova, F. Cole, A. Maschinot, A. Sarna, D. Vlasic, and W. T. Freeman, “Unsupervised training for 3D morphable model regression,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8377–8386.
  * [23] S. Sanyal, T. Bolkart, H. Feng, and M. J. Black, “Learning to regress 3D face shape and expression from an image without 3D supervision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7763–7772.
  * [24] Y. Deng, J. Yang, S. Xu, D. Chen, Y. Jia, and X. Tong, “Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [25] R. Wang, C.-F. Chen, H. Peng, X. Liu, O. Liu, and X. Li, “Digital twin: Acquiring high-fidelity 3D avatar from a single image,” _arXiv preprint arXiv:1912.03455_ , 2019.
  * [26] G. Trigeorgis, P. Snape, S. Zafeiriou, and I. Kokkinos, “Normal Estimation For ”in-the-wild” Faces Using Fully Convolutional Networks,” in _Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [27] O. Bouafif, B. Khomutenko, and M. Daoudi, “Monocular 3D head reconstruction via prediction and integration of normal vector field,” in _Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications-Volume 5: VISAPP, Valletta, Malta_ , 2020, pp. 359–369.
  * [28] X. Zeng, X. Peng, and Y. Qiao, “Df2net: A dense-fine-finer network for detailed 3D face reconstruction,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 2315–2324.
  * [29] H. Yi, C. Li, Q. Cao, X. Shen, S. Li, G. Wang, and Y.-W. Tai, “Mmface: A multi-metric regression network for unconstrained face reconstruction,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7663–7672.
  * [30] A. Bansal, B. Russell, and A. Gupta, “Marr revisited: 2D-3D alignment via surface normal prediction,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 5965–5974.
  * [31] J. Qiu, Z. Cui, Y. Zhang, X. Zhang, S. Liu, B. Zeng, and M. Pollefeys, “Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3313–3322.
  * [32] M. Hudon, M. Grogan, R. Pagés, and A. Smolić, “Deep normal estimation for automatic shading of hand-drawn characters,” in _European Conference on Computer Vision_. Springer, 2018, pp. 246–262.
  * [33] W. Su, D. Du, X. Yang, S. Zhou, and H. Fu, “Interactive sketch-based normal map generation with deep neural networks,” _Proceedings of the ACM on Computer Graphics and Interactive Techniques_ , vol. 1, no. 1, p. 22, 2018.
  * [34] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [35] R. T. Frankot and R. Chellappa, “A method for enforcing integrability in shape from shading algorithms,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 10, no. 4, pp. 439–451, 1988. [Online]. Available: https://doi.org/10.1109/34.3909
  * [36] S. Sengupta, A. Kanazawa, C. D. Castillo, and D. W. Jacobs, “Sfsnet: Learning shape, reflectance and illuminance of facesin the wild’,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6296–6305.
  * [37] V. F. Abrevaya, A. Boukhayma, P. H. Torr, and E. Boyer, “Cross-modal deep face normals with deactivable skip connections,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4979–4989.
  * [38] A. Kortylewski, A. Schneider, T. Gerig, B. Egger, A. Morel-Forster, and T. Vetter, “Training deep face recognition systems with synthetic data,” _arXiv preprint arXiv:1802.05891_ , 2018.
  * [39] J. Han, S. Karaoglu, H.-A. Le, and T. Gevers, “Improving face detection performance with 3D-rendered synthetic data,” _arXiv preprint arXiv:1812.07363_ , 2018.
  * [40] I. Abbasnejad, S. Sridharan, D. Nguyen, S. Denman, C. Fookes, and S. Lucey, “Using synthetic data to improve facial expression analysis with 3D convolutional networks,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 1609–1618.
  * [41] X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li, “Face alignment across large poses: A 3D solution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 146–155.
  * [42] E. Richardson, M. Sela, and R. Kimmel, “3D face reconstruction by learning from synthetic data,” in _2016 Fourth International Conference on 3D Vision (3DV)_. IEEE, 2016, pp. 460–469.
  * [43] C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou, “Facewarehouse: A 3D facial expression database for visual computing,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 20, no. 3, pp. 413–425, 2013.
  * [44] X. Zhu, Z. Lei, J. Yan, D. Yi, and S. Z. Li, “High-fidelity pose and expression normalization for face recognition in the wild,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 787–796.
  * [45] L. Hu, C. Ma, L. Luo, and H. Li, “Single-view hair modeling using a hairstyle database,” _ACM Transactions on Graphics (Proceedings SIGGRAPH 2015)_ , vol. 34, no. 4, July 2015.
  * [46] C. Duncan, H. Dai, N. Pears, and W. Smith, “A novel shape based outcomes analysis tool for craniofacial surgeons,” in _17th Biennial Congress of the International Society of Craniofacial Surgery (ISCFS)_ , vol. 1, no. 2, 2017\.
  * [47] S. Agarwal, K. Mierle, and Others, “Ceres solver,” http://ceres-solver.org.


# Consistent equivalence principle tests with fast radio bursts

Robert Reischke1, Steffen Hagstotz2 and Robert Lilow3
1 Ruhr University Bochum, Faculty of Physics and Astronomy, Astronomical
Institute (AIRUB),
German Centre for Cosmological Lensing, 44780 Bochum, Germany
2 The Oskar Klein Centre for Cosmoparticle Physics, Department of Physics,
Stockholm University, Roslagstullsbacken 21A, SE-106 91 Stockholm, Sweden
3 Department of Physics, Technion, Haifa 3200003, Israel
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Fast radio bursts (FRBs) are astrophysical transients of still debated origin.
So far several hundred events have been detected, mostly at extragalactic
distances, and this number is expected to grow significantly over the next
years. The radio signals from the burst experience dispersion as they travel
through the free electrons along the line-of-sight characterised by the
dispersion measure (DM) of the radio pulse. In addition, each photon also
experiences a gravitational Shapiro time delay while travelling through the
potentials generated by the large-scale structure. If the equivalence
principle (EP) holds, the Shapiro delay is the same for photons of all
frequencies. In case the EP is broken, one would expect an additional
dispersion to occur which could be either positive or negative for individual
sources. Here we suggest to use angular statistics of the DM fluctuations to
put constraints on the EP parametrized by the post-Newtonian parameter
$\gamma$. Previous studies suffer from the problem that the gravitational
potential responsible for the delay diverges in a cosmological setting, which
our approach avoids. We carry out a forecast for a population of FRBs
observable within the next years and show that any significant detection of
the DM angular power spectrum will place constraints on the EP that are by a
few orders of magnitude more stringent than current limits.

###### keywords:

cosmology: theory, large-scale structure of Universe, radio continuum:
transients

††pagerange: Consistent equivalence principle tests with fast radio
bursts–Acknowledgments††pubyear: 2020

##  1 Introduction

Fast radio bursts (FRBs) are short transients lasting usually only a few
milliseconds with frequencies ranging from $\sim 100$ MHz to several GHz. Due
to the free electrons in the ionized intergalactic medium (IGM), the pulse
experiences a dispersion $\Delta t\propto\nu^{-2}$, where the amplitude is
called the dispersion measure (DM) (e.g. Thornton et al., 2013; Petroff et
al., 2015; Connor et al., 2016; Champion et al., 2016; Chatterjee et al.,
2017) and is proportional to the integrated electron density along the line-
of-sight. While the mechanism for the radio emission is still under
debate,111For a compilation of currently proposed mechanisms for FRBs, see
https://frbtheorycat.org (Platts et al., 2019). their isotropic occurrence and
large observed DM suggest an extragalactic origin, so that the DM can be used
to test the distribution of diffuse electrons in the large-scale structure
(LSS). Several authors therefore proposed to use the DM inferred from FRBs as
a cosmological probe, using either the averaged signal (Zhou et al., 2014;
Walters et al., 2018) or the statistics of DM fluctuations (Masui & Sigurdson,
2015; Shirasaki et al., 2017; Rafiei-Ravandi et al., 2020; Reischke et al.,
2021; Bhattacharya et al., 2020; Takahashi et al., 2021).

Recently, FRBs have also been employed to test the equivalence principle (EP),
which is one of the key axioms of General Relativity. If the EP holds, photons
of different frequencies, as well as any other freely falling massless
particles, should follow the same null-geodesic. Measuring any deviation from
this would immediately indicate EP-breaking physics beyond the current
standard model. There is a number of papers (e.g. Krauss & Tremaine, 1988; Wei
et al., 2015, 2016a, 2016b; Wu et al., 2016; Wu et al., 2017; Yang et al.,
2017; Yu et al., 2018; Xing et al., 2019; Yao et al., 2020) dealing with this
type of measurements, which all rely on measuring the Shapiro time delay
(Shapiro, 1964) for different messengers. The most recent ones focus on
gravitational waves, gamma-ray bursts or FRBs. It is key to all Shapiro delay
tests that the transient is short compared to the associated Shapiro delay,
since this in principle determines the sensitivity of the measurement. All of
the measurements listed above assume a metric with weak perturbations that
vanish at infinity. In the parametrized post-Newtonian (PPN) formalism, this
leads to the well known Shapiro delay equation:

$t_{\mathrm{grav}}=-\frac{1+\gamma}{c^{3}}\int_{\bm{r}_{\mathrm{e}}}^{\bm{r}_{\mathrm{o}}}\mathrm{d}\lambda\;U\bigl{(}\bm{r}(\lambda)\bigr{)}\;,$
(1)

where the integral is performed from the source at $r_{e}$ to the observer at
$r_{o}$. We introduced the PPN parameter $\gamma$ measuring deviations from
the Newtonian time delay (Will, 2014), $U$ denotes the Newtonian gravitational
potential, and $c$ is the speed of light. For General Relativity one finds
$\gamma=1$, independent of frequency or particle type. Deviations from the EP
can be parametrized in terms of the difference $\Delta\gamma$ between the
$\gamma$ values of different messengers. Wei et al. (2015), for example, used
FRBs to constrain $\Delta\gamma$ between photons of different frequencies.

Minazzoli et al. (2019) pointed out that the standard formulation is not well
defined in a cosmological setting. Since the starting point is a perturbed
Minkowski metric, eq. 1 can only be applied to nearby sources. For corrections
in a cosmological setting, also see Nusser (2016). In addition, since the
cosmic density field does not vanish at infinity, the resulting naive Shapiro
delay in eq. 1 diverges. It is usually argued that only taking into account
contributions from some structures along the line-of-sight (and neglecting the
remaining cosmic structure) leads to conservative limits, but Minazzoli et al.
(2019) demonstrate that the neglected contributions can be positive or
negative. Thus, any partial reconstruction of the density field is
insufficient.

In this paper we demonstrate how FRBs can be used to set limits on EP breaking
similar to the procedure suggested by Nusser (2016), but by using statistical
observables of the DM. This avoids dealing with unphysical quantities and the
problematic boundary conditions for a cosmological setting. If the EP is
violated, the observed DM consists of a term arising due to the dispersion
induced by the free electrons in the LSS structure and and by a second term
arising from the fact that the photons at two different frequencies experience
a different Shapiro delay. Since FRBs are very short, any breaking of the EP
would lead to a very strong deviation in the time delay due to the long paths
involved in a cosmological setting. Therefore one can expect that any
measurement of statistics of the DM at the expected level immediately rules
out any substantial deviation from $\Delta\gamma=0$.

The remainder of the paper is organized as follows: we review the current
status of EP measurements using FRBs in section 2 and highlight the associated
problems. Section 3 introduces the DM statistic and describes how a
statistical analysis of FRBs removes the problem of the conventional analysis.
We present the results in section 4. In section 5 we summarize our findings.
Throughout the paper we will assume a $\Lambda$-cold-dark-matter
($\Lambda$CDM) scenario with fiducial parameters given by the best fit values
of Aghanim et al. (2020).

##  2 Shapiro delay tests of the equivalence principle

###  2.1 Using FRBs to test the equivalence principle

The observed time delay, $\Delta t_{\mathrm{obs}}$, between different
frequency bands of an astrophysical transient can be split into several
contributions:

$\Delta t_{\mathrm{obs}}=\Delta t_{\mathrm{int}}+\Delta
t_{\mathrm{LIV}}+\Delta t_{\mathrm{m}}+\Delta t_{\mathrm{grav}}\;.$ (2)

$\Delta t_{\mathrm{int}}$ is the intrinsic time delay due to the source and
the type of transient. In the case of FRBs this can be split into the DM
contribution $\Delta t_{\mathrm{DM}}$ and a potential source contribution
$\Delta t_{\mathrm{s}}$. The second term, $\Delta t_{\mathrm{LIV}}$, describes
Lorentz invariance violation and is ignored here. The term $\Delta
t_{\mathrm{m}}$ is a potential additional dispersion in case the photon is
massive. From the relativistic dispersion relation

$E^{2}=c^{2}p^{2}+m_{\gamma}^{2}c^{4}$ (3)

one finds that massive photons of different frequencies propagate at different
speeds. This also introduces a delay in the arrival time between different
frequencies of the FRB signal:

$\Delta
t_{\mathrm{m}}(z)=\left(\frac{m_{\gamma}c^{2}}{4\pi\hbar}\right)^{2}A(z)\left(\nu_{1}^{-2}-\nu_{2}^{-2}\right)\>,$
(4)

with the redshift evolution given by

$A(z)=\int\mathrm{d}z^{\prime}\frac{1}{(1+z^{\prime})^{2}H(z^{\prime})}\;.$
(5)

Note that the frequency dependence has the same shape as the dispersion caused
by the intergalactic plasma, so for any single source this would contribute to
the total DM.

Both $\Delta t_{\mathrm{LIV}}$ and $\Delta t_{\mathrm{m}}$ produce non-zero
effects on the cosmological background. As the current study will be only
concerned with perturbations, these two contributions are ignored in what
follows. With this we are left with

$\Delta t_{\mathrm{obs}}=\Delta t_{s}+\Delta t_{\mathrm{DM}}+\Delta
t_{\mathrm{grav}}\;,$ (6)

where the last term is the difference in the gravitational time delay, eq. 1,
between photons of different frequency bands.

###  2.2 Problems with cosmological tests

The general problems with EP tests based on the Shapiro delay have already
been pointed out by Minazzoli et al. (2019), but we will briefly recap them
here for the case of FRBs. So far, most of the cosmological tests consider a
localised FRB (or any other object suitable for a differential Shapiro delay
measurement as outlined in section 1) and use the reconstructed potential, for
example the one of the Milky Way, to calculate the predicted Shapiro delay.
Since the Shapiro delay is usually much larger than the delay induced by the
dispersion of the free electrons along the line-of-sight, any EP breaking
($\Delta\gamma\neq 0$) would completely dominate the time delay. It is then
assumed that this gives a conservative bound on $\Delta\gamma$ since the
Shapiro delay will just increase when adding more reconstructed data, i.e.
another cluster along the line-of-sight.

The problem with this procedure is that the gravitational potential along the
line-of-sight is influenced by all particles in the Universe. Thus, the
Shapiro delay diverges if the density does not fall off. The reason for this
problem is the choice of coordinates associated with eq. 1. At spatial
infinity the potential and its derivative are expected to vanish. This
boundary condition might be suitable for isolated objects, but it is not
fulfilled in the cosmological setting with constant background density. In
particular, Minazzoli et al. (2019) show that the cosmological part completely
dominates the naive expectation of the Shapiro delay from any individual
Keplerian potential, e.g. of the Milky Way.

Lastly, Minazzoli et al. (2019) show how to renormalise the divergence by
choosing an appropriate time coordinate and derive an expression for the
Shapiro delay which, however, is no longer monotonic with respect to the
number of sources used to reconstruct the potential along the line-of-sight.
Therefore one cannot place a conservative bound on $\Delta\gamma$, at least on
cosmological scales, by just considering a subset of sources for which a
reconstructed potential along the line-of-sight is readily available.

###  2.3 A way out

In a cosmological setting, we can avoid the problem of a diverging time delay
by considering a weakly perturbed Friedman-Robertson-Walker (FRW) line element
in conformal Newtonian gauge within the PPN formulation:

$\mathrm{d}s^{2}=-\biggl{(}1+\frac{2\phi}{c^{2}}\biggr{)}\,\mathrm{c}^{2}dt^{2}+a^{2}(t)\;\biggl{(}1-\frac{2\gamma\phi}{c^{2}}\biggr{)}\,\mathrm{d}\mathbf{x}^{2}$
(7)

with the gauge potential $\phi$, the scale factor $a$ and the comoving
coordinates $\mathbf{x}$. The corresponding time delay between photons of
different frequencies can then be written as (Nusser, 2016)

$\Delta
t_{\mathrm{grav}}(\hat{\bm{x}})=\frac{\Delta\gamma}{c^{3}}\int^{\chi_{\mathrm{s}}}_{0}\mathrm{d}\chi\;a(\chi)\,\phi\big{(}\hat{\bm{x}}\chi,a(\chi)\big{)}\;,$
(8)

where $\chi$ is the comoving distance at the background level. This expression
does not diverge in a cosmological setting since it respects the cosmological
symmetry assumptions by construction. While this looks very similar to eq. 1,
the perturbation $\phi$ is a Gaussian random field with zero mean, and
therefore the time delay can acquire positive and negative contributions along
the line-of-sight.

Note that this expression is not suited for individual nearby lines-of-sight
dominated by objects that cannot be described as small, linear perturbations
to a FRW background. Furthermore, the absolute value of $\phi$ in any finite
volume is not an observable quantity. However, these problems can be avoided
by observing angular correlations of time delays. They are insensitive to
individual nearby objects, and are independent of monopole contributions and
thus the absolute value of $\phi$. The fact that local fluctuations of $\Delta
t_{\mathrm{grav}}$ can be negative poses no problem since any EP breaking
changes the expected correlations. Angular correlations can therefore be used
to place an upper bound on the strength of EP violation.

##  3 Dispersion measure and time delay statistics

The observed time delay in direction $\hat{\bm{x}}$ for a source at redshift
$z$ is interpreted as a total DM via

$\Delta
t_{\mathrm{obs}}(\hat{\bm{x}},z)\propto\mathrm{DM}_{\mathrm{tot}}(\hat{\bm{x}},z)\nu^{-2}\;.$
(9)

More explicitly, the time delay between two measured frequencies $\nu_{1,2}$
is

$\begin{split}\Delta t_{\mathrm{obs}}(\hat{\bm{x}},z)=&\
t_{\nu_{1},\mathrm{obs}}(\hat{\bm{x}},z)-t_{\nu_{2},\mathrm{obs}}(\hat{\bm{x}},z)\\\
=&\
\mathcal{K}\,\mathrm{DM}_{\mathrm{tot}}(\hat{\bm{x}},z)\,\left(\nu^{-2}_{1}-\nu^{-2}_{2}\right)\;,\end{split}$
(10)

where we absorb all the constants in $\mathcal{K}=e^{2}/(2\uppi m_{e}c)$.
Here, $e$ and $m_{e}$ denote the charge and mass of an electron, respectively.
The DM and time delays can therefore be used interchangeably. We will mostly
work with the DM from now on since it is the quantity of interest for FRB
correlation studies.

Any additional delay in the arrival time of different pulse frequencies due to
EP breaking leads to a shift

$\mathrm{DM}_{\mathrm{tot}}(\hat{\bm{x}},z)\to\mathrm{DM}_{\mathrm{tot}}(\hat{\bm{x}},z)+\mathcal{D}_{\mathrm{grav}}(\hat{\bm{x}},z)\;,$
(11)

where $\mathcal{D}_{\mathrm{grav}}(\hat{\bm{x}},z)$ is the EP-breaking time
delay from eq. 8 interpreted as a DM in direction $\hat{\bm{x}}$ and up to
redshift $z$:

$\mathcal{D}_{\mathrm{grav}}(\hat{\bm{x}},z)=\frac{\Delta\gamma}{\mathcal{K}c^{3}\left(\nu^{-2}_{1}-\nu^{-2}_{2}\right)}\int_{0}^{\chi(z)}\mathrm{d}\chi^{\prime}\;a(\chi^{\prime})\,\phi\big{(}\hat{\bm{x}}\chi^{\prime},z(\chi^{\prime})\big{)}\;.$
(12)

As mentioned before, this contribution can be positive and negative, a problem
which has not been addressed in previous studies. Note that this
identification is subject to the $\nu^{-2}$ law, thus providing a preferred
frequency shape of the EP-breaking term. However, the null hypothesis is
$\Delta\gamma=0$, as predicted by GR, and any additional contribution will
immediately show up in the inferred DM budget.

The non-gravitational contribution in eq. 11 is usually split into three
parts:

$\mathrm{DM}_{\mathrm{tot}}(\hat{\bm{x}},z)=\mathrm{DM}_{\mathrm{LSS}}(\hat{\bm{x}},z)+\mathrm{DM}_{\mathrm{MW}}(\hat{\bm{x}})+\mathrm{DM}_{\mathrm{host}}(z)\;.$
(13)

$(i)$ For the contribution from the Milky Way,
$\mathrm{DM}_{\mathrm{MW}}(\hat{\bm{x}})$, models of the galactic electron
distribution predict $\mathrm{DM}_{\mathrm{MW}}\lower
2.15277pt\hbox{$\;\buildrel<\over{\sim}\;$}50\;\mathrm{pc}\;\mathrm{cm}^{-3}$
(Yao et al., 2017). We will assume that this contribution can be modelled and
subtracted from the signal. $(ii)$ For the host galaxy contribution,
$\mathrm{DM}_{\mathrm{host}}(z)$, similar values are expected inducing a
random scatter on the total DM. $(iii)$ The large-scale structure contribution
can be written as

$\mathrm{DM}_{\mathrm{LSS}}(\hat{\bm{x}},z)=\int_{0}^{z}n_{\mathrm{e}}({\bm{x}},z^{\prime})\,\frac{1+z^{\prime}}{H(z^{\prime})}\mathrm{d}z^{\prime}\;.$
(14)

Here $H(z)$ is the Hubble expansion rate, and $n_{e}$ is the electron number
density. The latter is a functional of the matter density contrast
$\delta_{\mathrm{m}}(\bm{x},z)$:

$n_{\mathrm{e}}({\bm{x}},z)=\frac{\rho_{\mathrm{b}}({\bm{x}},z)}{m_{\mathrm{p}}}=\frac{\bar{\rho}_{\mathrm{b}}(z)}{m_{\mathrm{p}}}\left[1+b_{\mathrm{e}}({\bm{x}},z)\,\delta_{\mathrm{m}}({\bm{x}},z)\right]\;,$
(15)

with the baryon mass density $\rho_{\mathrm{b}}({\bm{x}},z)$, its mean value
$\bar{\rho}_{\mathrm{b}}(z)$, the proton mass $m_{\mathrm{p}}$ and the
electron clustering bias $b_{\mathrm{e}}({\bm{x}},z)$. We assumed that the
bias is linear, i.e. we consider only the lowest-order response of the
electron density to the density of matter. This assumption should be valid on
the scales we are considering. Rewriting eq. 15 yields:

$\mathrm{DM}_{\mathrm{LSS}}(\hat{\bm{x}},z)=\mathcal{A}\int_{0}^{z}\mathrm{d}z^{\prime}\frac{1+z^{\prime}}{E(z^{\prime})}F(z^{\prime})[1+b_{\mathrm{e}}({\bm{x}},z^{\prime})\delta_{\mathrm{m}}({\bm{x}},z^{\prime})]\;,$
(16)

with

$\mathcal{A}=\frac{3H_{0}^{2}\Omega_{\mathrm{b}0}\chi_{H}}{8\uppi
Gm_{\mathrm{p}}}\;,$ (17)

where $\chi_{H}$ is the Hubble radius today, $E(z)=H(z)/H_{0}$ the
dimensionless expansion rate, and $F(z)$ the mass fraction of electrons in the
IGM, which itself can be expressed as follows:

$F(z)=f_{\mathrm{IGM}}(z)[Y_{\mathrm{H}}X_{\mathrm{e},\mathrm{H}}(z)+Y_{\mathrm{He}}X_{\mathrm{e},\mathrm{He}}(z)]\;.$
(18)

Here $Y_{\mathrm{H}}=0.75$ and $Y_{\mathrm{He}}=0.25$ are the mass fractions
of hydrogen and helium, respectively, $X_{\mathrm{e},\mathrm{H}}(z)$ and
$X_{\mathrm{e},\mathrm{He}}(z)$ are their ionization fractions, and
$f_{\mathrm{IGM}}(z)$ is the fraction of electrons in the IGM. We assume
$X_{\mathrm{e},\mathrm{H}}=X_{\mathrm{e},\mathrm{He}}=1$ and
$f_{\mathrm{IGM}}(z)=90\%\;(80\%)$ at $z\lower
2.15277pt\hbox{$\;\buildrel>\over{\sim}\;$}1.5\;(\lower
2.15277pt\hbox{$\;\buildrel<\over{\sim}\;$}0.4)$ (Meiksin, 2009; Becker et
al., 2011; Shull et al., 2012).

To avoid dealing with the monopole, we study statistical properties of the DM.
The angular power spectrum of the DM has been calculated in Masui & Sigurdson
(2015); Rafiei-Ravandi et al. (2020); Reischke et al. (2021). In the presence
of a potentially EP-breaking time delay, the correlator of the observed DM
fluctuations includes additional terms. We begin by splitting the LSS
contribution to the DM into background and perturbation contributions:

$\mathrm{DM}_{\mathrm{LSS}}(\hat{\bm{x}},z)=\overline{\mathrm{DM}}_{\mathrm{LSS}}(z)+\mathcal{D}_{\mathrm{LSS}}(\hat{\bm{x}},z)\,.$
(19)

$\mathcal{D}_{\mathrm{LSS}}(\hat{\bm{x}},z)$ is the effective DM induced by
the fluctuations in the LSS and is given by the weighted line-of-sight
integral over the electron density perturbation in the second term of eq. 16.
Note that there is no background contribution for the gravitational part, eq.
12, which is exactly the problem in the ordinary measurements of EP breaking
with FRBs. The perturbations to the DM from the electrons in the LSS and the
Shapiro delay are therefore given by

$\mathcal{D}(\hat{\bm{x}},z)=\mathcal{D}_{\mathrm{LSS}}(\hat{\bm{x}},z)+\mathcal{D}_{\mathrm{grav}}(\hat{\bm{x}},z)\;.$
(20)

Given a normalised source redshift distribution $n(z)$, satisfying
$\int\mathrm{d}z\;n(z)=1$, and the associated distance distribution
$n(\chi)=n(z)\,\mathrm{d}z/\mathrm{d}\chi$, eq. 20 can be averaged over
redshift:

$\mathcal{D}(\hat{\bm{x}})=\int_{0}^{\chi_{H}}\\!\\!\mathrm{d}\chi\;n(\chi)\,\mathcal{D}\big{(}\hat{\bm{x}},z(\chi)\big{)}\;.$
(21)

By rearranging the integration limits we find

$\mathcal{D}(\hat{\bm{x}})=\int_{0}^{\chi_{H}}\\!\\!\mathrm{d}\chi\;W_{\mathcal{D}}(\hat{\bm{x}},\chi)\,\delta_{\mathrm{m}}\big{(}\hat{\bm{x}},z(\chi)\big{)}\;,$
(22)

Figure 1: Possible constraints on the EP-breaking parameter $\Delta\gamma$ for
a survey with $10^{3}$ FRB and $\alpha=3.5$ and varying number of bins
$n_{\mathrm{tomo}}$ in the DM distribution.

with the averaged weighting function

$W_{\mathcal{D}}(\chi)=\bigg{(}W_{\mathrm{LSS}}(\hat{\bm{x}},\chi)+W_{\mathrm{grav}}(\hat{\bm{x}},\chi)\bigg{)}\int_{\chi}^{\chi_{H}}\mathrm{d}\chi^{\prime}n(\chi^{\prime})\;.$
(23)

Here, $W_{\mathrm{LSS}}$ and $W_{\mathrm{grav}}$ are defined via eq. 16 and
eq. 12, respectively:

$\displaystyle W_{\mathrm{LSS}}(\hat{\bm{x}},\chi)$
$\displaystyle=\left|\frac{\mathrm{d}z}{\mathrm{d}\chi}\right|\mathcal{A}\,\frac{F\big{(}z(\chi)\big{)}\,\big{(}1+z(\chi)\big{)}}{E\big{(}z(\chi)\big{)}}\,b_{\mathrm{e}}(\hat{\bm{x}},\chi)\,,$
(24) $\displaystyle W_{\mathrm{grav}}(\hat{\bm{x}},\chi)$
$\displaystyle=\frac{3\Omega_{\mathrm{m0}}\Delta\gamma\Delta^{-1}}{2\mathcal{K}c\chi_{H}^{2}\left(\nu^{-2}_{1}-\nu^{-2}_{2}\right)}\;,$
(25)

where $\Delta^{-1}$ is the inverse Laplacian relating the potential
fluctuations $\phi$ to the matter density contrast $\delta_{\mathrm{m}}$ via
the Poisson equation. The angular power spectrum of DM correlations for the
source distribution is then given by

$\begin{split}C^{\mathcal{D}\mathcal{D}}(\ell)&=\frac{2}{\uppi}\int\mathrm{d}\chi_{1}\int\mathrm{d}\chi_{2}\int
k^{2}\mathrm{d}k\;W_{\mathcal{D}}(k,\chi_{1})\,W_{\mathcal{D}}(k,\chi_{2})\\\
&\quad\times\sqrt{P_{\mathrm{mm}}(k,\chi_{1})\,P_{\mathrm{mm}}(k,\chi_{2})}\,j_{\ell}(k\chi_{1})\,j_{\ell}(k\chi_{2})\;,\end{split}$
(26)

where we express the weight function in Fourier space, so that the inverse
Laplacian $\Delta^{-1}=k^{-2}$ for a comoving wavenumber $k$. The electron
bias $b_{\mathrm{e}}$ is allowed to have a scale dependence. Note that eq. 26
depends on the two observed frequencies. Compared to the sensitivity to
$\Delta\gamma$ this dependence is, however, very weak.

The host galaxy acts as a stochastic source with an intrinsic width of
$\sigma^{2}_{\mathrm{host}}$. Due to the finite number of sources per solid
angle, $\bar{n}$, this adds a white noise contribution to the observed
spectrum:

$C^{\mathcal{D}\mathcal{D}}(\ell)\to
C^{\mathcal{D}\mathcal{D}}(\ell)+\frac{\sigma^{2}_{\mathrm{host}}}{\bar{n}}\;.$
(27)

As shown in Reischke et al. (2021), the shot noise contribution is small
compared to cosmic variance on large scales even if only a few thousand FRBs
are available. It should also be noted that any redshift dependence of the
host galaxy contribution to the DM has been averaged out, weighted by the
source redshift distribution. We approximate this distribution by the
following form:

$n(z)\propto z^{2}\exp\left(-\alpha z\right)\;,$ (28)

where $\alpha$ determines the depth of the survey.

In Reischke et al. (2021) the influence of the uncertainty on the DM in the
absence of redshift information as well as its implication for the source
redshift distribution and the corresponding angular power spectra was
discussed. The DM is translated into a redshift assuming a fiducial cosmology
without EP breaking. This can lead to dispersion space distortions (Masui &
Sigurdson, 2015) which we will, however, ignore here. However, the associated
scatter of the redshift from the uncertainty of the DM is incorporated into
the redshift distribution:

$n(z)=\int\mathrm{d}z_{\mathrm{DM}}\;n(z_{\mathrm{DM}})p(z_{\mathrm{DM}}|z)\;,$
(29)

where $p(z_{\mathrm{DM}}|z)$ is well approximated by a Gaussian distribution,
as shown in (Jaroszynski, 2019):

$p(z_{\mathrm{DM}}|z)\sim\mathcal{N}(\langle
z_{\mathrm{DM}}\rangle(z),\sigma^{2}_{z_{\mathrm{DM}}(z)})$ (30)

with corresponding mean

$\langle
z_{\mathrm{DM}}\rangle=\langle\mathrm{DM}_{\mathrm{host}}\rangle(z)+\langle\mathrm{DM}_{\mathrm{LSS}}\rangle(z)$
(31)

and variance

$\sigma^{2}_{z_{\mathrm{DM}}}=\sigma^{2}_{\mathrm{host}}(z)+\sigma^{2}_{\mathrm{LSS}}(z)\;.$
(32)

The latter consists of the host contribution $\sigma^{2}_{\mathrm{host}}$ and
the cosmological contribution $\sigma^{2}_{\mathrm{LSS}}$.

We want to stress that any $\Delta\gamma\neq 0$ will introduce large
fluctuations to the DM. Here we are analysing the data with respect to a
$\Lambda$CDM scenario without EP breaking. In this sense we are assuming that
we can convert a given DM into a corresponding redshift on the background
level, while accounting for the scatter due to the perturbations in the
electron density and the host contribution (see Reischke et al. (2021) for
more details). Rafiei-Ravandi et al. (2020) discussed how cross-correlations
could be used to further improve FRB distance measurements in the absence of
redshift information.

Lastly it should be noted that everything could also be formulated in terms of
the time delay directly, so that a time delay correlation function would be
computed. This would be more closely related to the effects considered here.
However, since FRB measurements are usually done in DM space, we stick to this
notation here.

Figure 2: Constraints on the EP-breaking parameter $\Delta\gamma$ as a
function of the signal-to-noise ratio of a measurement of the DM angular power
spectrum.

##  4 Results

In this section we will discuss our results for a forecast of an EP null test.
For this we assume a survey following the source distribution eq. 28 with
$\alpha=3.5$, which roughly corresponds to the distribution of currently
observed FRBs (Petroff et al., 2016). Note that we do not assume that the
redshifts for the events are known, but only use the probabilistic conversion
from DM to $z$ in eq. 29. Furthermore, we assume a total of $10^{3}$ observed
FRBs. Lastly, the electron bias is assumed to be $b_{\mathrm{e}}=0.8$ and it
reaches unity at $z=3$, see (Reischke et al., 2021) for more details.

We consider an almost full sky survey, $f_{\mathrm{sky}}=0.8$, and decompose
the DM map into spherical harmonics from which the angular power spectrum can
be estimated. The sensitivity with respect to $\Delta\gamma$ is calculated
using a Fisher forecast for a Gaussian likelihood of the spherical harmonic
modes with covariance due to cosmic variance and Poisson noise. The latter
describes the intrinsic DM scatter caused by the host and the finite number of
FRB sources. Additionally, the DM maps can be separated into sub-samples by
considering DM bins, as demonstrated in (Reischke et al., 2021). This methods
recovers some of the scale-dependent effects that are smoothed out by the
line-of-sight projection. A pulse width of 0.3 GHz is assumed (corresponding
to 1.5 and 1.2 Ghz), which enters in eq. 25, and the host galaxy contribution
is $\sigma_{\mathrm{host}}(z)=50\mathrm{pc}\,\mathrm{cm}^{-3}\,(1+z)^{-1}$.

In fig. 1 we show the resulting $1$-$\sigma$ limit on $\Delta\gamma$ for the
described survey as a function of the maximum multipole considered and for
different numbers of tomographic bins $n_{\mathrm{tomo}}$. The constraints
presented by Wei et al. (2015) are of the order of $10^{-10}$ while Nusser
(2016) found $\sigma_{\Delta\gamma}\sim 10^{-13}$ for different FRBs. The
constraints here are better by roughly two orders of magnitude. Since Nusser
(2016) was interested in the rms-value and not the full correlation, the
sensitivity to $\Delta\gamma$ is smaller than in the case studied here, as the
angular power spectrum contains terms scaling quadratically with
$\Delta\gamma$. The highest impact of the Shapiro delay measurements can be
seen at low multipoles, while the constraining power settles very quickly at
higher $\ell$. This happens for two reasons: $(i)$ the low multipoles are most
affected due to the $k^{-2}$ factor in the Shapiro delay contribution coming
from the Poisson equation. Therefore, the contribution of the gravitational
time delay will dominate the DM most strongly on the largest scales. $(ii)$
Due to the small number of FRBs, the shot noise starts dominating over the
signal at higher $\ell$. Increasing the number of tomographic bins increases
the information content of the measurement to some degree, as it gets easier
to pick up the scale-dependent imprint on the angular spectrum. However, the
increased shot noise diminishes the gain when more than three bins are
considered.

To present the constraints independent of the specific survey settings, we
show in fig. 2 again the $1$-$\sigma$ limit on $\Delta\gamma$, but now as a
function of the signal-to-noise ratio of the measurement of the angular power
spectrum. It is clear from the figure that any significant detection of the
angular correlation in the DM of FRBs will immediately put very stringent
constraints on deviations from the EP, $\Delta\gamma<10^{-15}$, with
increasing signal-to-noise only yielding mild improvements. The reason is that
the integrated Shapiro delay over cosmological distances in combination with
the short pulse durations yield $C^{\mathcal{DD}}_{\mathrm{grav}}$
contributions to the angular spectrum that are much larger than any dispersion
caused by electrons.

For all forecasts made here we assumed all other parameters in the measurement
to be fixed. Marginalizing over those, however, would not change the results
dramatically since the effect of the Shapiro delay dominates the other
contributions by many orders of magnitude if the EP is broken.

##  5 Conclusions

In this work we re-investigated constraints on the breaking of the equivalence
principle (EP), characterized by the post-Newtonian parameter difference
$\Delta\gamma$, with fast radio bursts by exploiting the Shapiro time delay.
Previous studies had focused on individual FRBs with redshift information by
accounting for the Keplerian potentials of known objects along the line-of-
sight. This was shown to be inconsistent by (Minazzoli et al., 2019). Here we
proposed instead to use statistical properties of the dispersion measure (DM)
of FRBs to constrain deviations from the EP. This measurement does not suffer
from divergences of the time delay and correctly accounts for the Shapiro
delay induced by the LSS. In particular, we looked at the angular power
spectrum of the DM of FRBs. While the Shapiro delay along any given line-of-
sight can be positive or negative (causing problems with previous studies), in
our approach the imprint on the angular DM correlations can always be detected
as a change in the correlation structure. We performed a forecast for a null
test of EP violation by considering a fiducial $\Lambda$CDM cosmology and
placing upper limits on $\Delta\gamma$ between observed frequencies. We would
like to stress that the present parametrization is just a phenomenological one
without a direct link to any specific EP-breaking theory. It is in this sense
just an agnostic test of the degree up to which the EP holds between two
frequencies. We summarize our principal results as follows:

$(i)$ We calculated the contribution to the DM angular correlation function
from the free electron distribution in the LSS and the induced Shapiro delay
by the LSS.

$(ii)$ Using correlations of the DM measured from an FRB population avoids the
usual problems with Shapiro delay measurements by satisfying appropriate
boundary conditions in a cosmological setting.

$(iii)$ Detecting the angular power spectrum of the FRB DM with any
significance at the predicted level will immediately place tight constraints
on $\Delta\gamma$, which are a few orders of magnitudes better than the
current limits.

While we presented the theoretical predictions in DM space, one could
alternatively formulate everything in time delay space, providing a closer
connection to the Shapiro delay. Since the measurements of the angular
correlation function will happen on very larges scales, due to the relatively
high shot noise contribution from the limited number of expected FRB
detections, it will be necessary to calculate general relativistic projection
effects to the angular power spectrum to make accurate theoretical
predictions. We intend to do this in a future paper. Due to the high
sensitivity on $\Delta\gamma$ this will, however, not change our main
conclusion that EP can be tested to unprecedented precision once the angular
correlation of the FRB’s DM has been detected.

## Acknowledgments

RR is supported by the European Research Council (Grant No. 770935). SH
acknowledges support from the Vetenskapsrådet (Swedish Research Council)
through contract No. 638-2013-8993 and the Oskar Klein Centre for
Cosmoparticle Physics. RL acknowledges support by a Technion fellowship.

## References

  * Aghanim et al. (2020) Aghanim N., et al., 2020, Astronomy & Astrophysics, 641, A6
  * Becker et al. (2011) Becker G. D., Bolton J. S., Haehnelt M. G., Sargent W. L. W., 2011, Monthly Notices of the Royal Astronomical Society, 410, 1096
  * Bhattacharya et al. (2020) Bhattacharya M., Kumar P., Linder E. V., 2020, arXiv:2010.14530 [astro-ph]
  * Champion et al. (2016) Champion D. J., et al., 2016, Monthly Notices of the Royal Astronomical Society, 460, L30
  * Chatterjee et al. (2017) Chatterjee S., et al., 2017, Nature, 541, 58
  * Connor et al. (2016) Connor L., Sievers J., Pen U.-L., 2016, Monthly Notices of the Royal Astronomical Society, 458, L19
  * Jaroszynski (2019) Jaroszynski M., 2019, Monthly Notices of the Royal Astronomical Society, 484, 1637
  * Krauss & Tremaine (1988) Krauss L. M., Tremaine S., 1988, Physical Review Letters, 60, 176
  * Masui & Sigurdson (2015) Masui K. W., Sigurdson K., 2015, Physical Review Letters, 115, 121301
  * Meiksin (2009) Meiksin A. A., 2009, Reviews of Modern Physics, 81, 1405
  * Minazzoli et al. (2019) Minazzoli O., Johnson-McDaniel N. K., Sakellariadou M., 2019, Physical Review D, 100, 104047
  * Nusser (2016) Nusser A., 2016, ApJ, 821, L2
  * Petroff et al. (2015) Petroff E., et al., 2015, MNRAS, 447, 246
  * Petroff et al. (2016) Petroff E., et al., 2016, Publ. Astron. Soc. Aust., 33, e045
  * Platts et al. (2019) Platts E., Weltman A., Walters A., Tendulkar S. P., Gordin J. E. B., Kandhai S., 2019, Physics Reports, 821, 1
  * Rafiei-Ravandi et al. (2020) Rafiei-Ravandi M., Smith K. M., Masui K. W., 2020, Physical Review D, 102, 023528
  * Reischke et al. (2021) Reischke R., Hagstotz S., Lilow R., 2021, Physical Review D, 103, 023517
  * Shapiro (1964) Shapiro I. I., 1964, Phys. Rev. Lett., 13, 789
  * Shirasaki et al. (2017) Shirasaki M., Kashiyama K., Yoshida N., 2017, Physical Review D, 95, 083012
  * Shull et al. (2012) Shull J. M., Smith B. D., Danforth C. W., 2012, The Astrophysical Journal, 759, 23
  * Takahashi et al. (2021) Takahashi R., Ioka K., Mori A., Funahashi K., 2021, Monthly Notices of the Royal Astronomical Society, 502, 2615
  * Thornton et al. (2013) Thornton D., et al., 2013, Science, 341, 53
  * Walters et al. (2018) Walters A., Weltman A., Gaensler B. M., Ma Y.-Z., Witzemann A., 2018, The Astrophysical Journal, 856, 65
  * Wei et al. (2015) Wei J.-J., Gao H., Wu X.-F., Mészáros P., 2015, Phys. Rev. Lett., 115, 261101
  * Wei et al. (2016a) Wei J.-J., Wu X.-F., Gao H., Mészáros P., 2016a, Journal of Cosmology and Astroparticle Physics, 08, 031
  * Wei et al. (2016b) Wei J.-J., Wang J.-S., Gao H., Wu X.-F., 2016b, The Astrophysical Journal Letters, 818, L2
  * Will (2014) Will C. M., 2014, Living Reviews in Relativity, 17, 4
  * Wu et al. (2016) Wu X.-F., Gao H., Wei J.-J., Mészáros P., Zhang B., Dai Z.-G., Zhang S.-N., Zhu Z.-H., 2016, Physical Review D, 94, 024061
  * Wu et al. (2017) Wu X.-F., Wei J.-J., Lan M.-X., Gao H., Dai Z.-G., Mészáros P., 2017, Physical Review D, 95, 103004
  * Xing et al. (2019) Xing N., Gao H., Wei J.-J., Li Z., Wang W., Zhang B., Wu X.-F., Mészáros P., 2019, The Astrophysical Journal Letters, 882, L13
  * Yang et al. (2017) Yang C., Zou Y.-C., Zhang Y.-Y., Liao B., Lei W.-H., 2017, Monthly Notices of the Royal Astronomical Society, 469, L36
  * Yao et al. (2017) Yao J. M., Manchester R. N., Wang N., 2017, The Astrophysical Journal, 835, 29
  * Yao et al. (2020) Yao L., Zhao Z., Han Y., Wang J., Liu T., Liu M., 2020, The Astrophysical Journal, 900, 31
  * Yu et al. (2018) Yu H., Xi S.-Q., Wang F.-Y., 2018, The Astrophysical Journal, 860, 173
  * Zhou et al. (2014) Zhou B., Li X., Wang T., Fan Y.-Z., Wei D.-M., 2014, Phys. Rev. D, 89, 107303