# Tell-the-difference: Fine-grained Visual Descriptor via a Discriminating
Referee

Shuangjie Xu111Shuangjie Xu and Feng Xu have equal contribution to the paper.
<EMAIL_ADDRESS>Feng Xu222Shuangjie Xu and Feng Xu have equal
contribution to the paper<EMAIL_ADDRESS>Yu Cheng
<EMAIL_ADDRESS>Pan Zhou<EMAIL_ADDRESS>Huazhong University of
Science and Technology, Luoyu Road 1037, Wuhan, China  Microsoft Research AI,
Microsoft Building 99, 14820 NE 36th St., Redmond, Wash., USA

###### Abstract

In this paper, we investigate a novel problem of telling the difference
between image pairs in natural language. Compared to previous approaches for
single image captioning, it is challenging to fetch linguistic representation
from two independent visual information. To this end, we have proposed an
effective encoder-decoder caption framework based on Hyper Convolution Net. In
addition, a series of novel feature fusing techniques for pairwise visual
information fusing are introduced and a discriminating referee is proposed to
evaluate the pipeline. Because of the lack of appropriate datasets to support
this task, we have collected and annotated a large new dataset with Amazon
Mechanical Turk (AMT) for generating captions in a pairwise manner (with 14764
images and 26710 image pairs in total). The dataset is the first one on the
relative difference caption task that provides descriptions in free language.
We evaluate the effectiveness of our model on two datasets in the field and it
outperforms the state-of-the-art approach by a large margin.

###### keywords:

Relative difference caption, Fine-grained difference description,
Discriminating Referee

††journal: Journal of Pattern Recognition

##  1 Introduction

Visual difference description has long been relatively less studied but
crucial task in the visual-linguistic field, which is demanding because of the
necessity for understanding the pairwise visual information. Most previous
works could only detect visual differences within certain attribute domain
designed by experts [1], which precludes its extension to new domains.

Su et al. [2] focused on the phrase generation of object attributes for visual
differences description between instances, utilizing data annotated by non-
expert workers. However, the discrepancies sentences generation is still
challenging. What we aim for in this paper is a difference captioner that not
only adapted to varied expressions from non-expert workers but also is capable
of generating syntactical-flexible sentences.

Traditional caption tasks [3, 4, 5] concentrated on single image caption tasks
($\bm{I}\to\bm{S}$, which is widely used in automatic photo captioning, human-
computer interaction and so on). Recently, the discriminating caption task [6,
7], which utilizes image pair in training procedure to improve the quality of
single image caption task, extends the caption task to the pairwise caption
task $(\bm{I_{1}},\bm{I_{2}})\to\bm{S}$. However, the discriminating caption
task only has a fuzzy perception of the image difference, leading to a limited
ability to tell the difference in detail. In this work, we aim to distinguish
image pair with detailed attributes. For example, the discriminating captioner
tells ”An ultralight plane flying through a blue sky”, but our model tells
”Has no landing gear can be seen and is an ultralight plane”, where the
detailed difference receives more attention by the model than the entire
scene.

Difference caption is such a difference telling task between image pair with
vast application scenarios: a) guide online shopping with human-computer
dialog; b) characterize differences of patient behavior in medical monitoring;
c) describe dangerous behavior in traffic. However, relative caption seems to
be a rather daunting task at first sight, since we obviously need to conquer a
series of challenges: 1) Absence of dataset for describing pairwise image
differences in natural language; 2) A novel framework for processing pairwise
visual information; 3) Adaptive evaluation metric (e.g., “has propeller
engine” is also the right caption for Fig 1 but will get a low score in
automatic NLP metric).

Figure 1: Semantic difference description for image pairs. The left pair is
from AMT-20K dataset which is about shoes; the right pair is from OID dataset
for planes. Both descriptions only take the difference characteristics shown
in the first image into consideration, which is actually how human express.

We researched plenty of popular caption datasets, however they are not capable
of covering the need of our task: comprehensive difference in detail; not for
entire scene, but for specific object. Datasets in [8, 9] failed to fit our
need obviously: they are proposed to retrieve, not to distinguish. Using
caption datasets with individual image is only capable to get ”rough”
difference caption, which is not what we need. We begin with collecting a
dataset of difference caption named AMT 20K dataset by requesting workers to
annotate visual difference description $\bm{S}$ with given image pair
$(\bm{I_{1}},\bm{I_{2}})$ based on the first image $\bm{I_{1}}$, which is the
manner that more like human. Examples are shown in Fig 1. Since that
annotations are from non-expert workers, the fine-grained discrepancies is
abundant with more subjectivity such as “a cowboy look” and “gold rivet
accents”.

Utilizing the collected AMT 20K dataset, we introduced a series of novel
tactics to fuse visual information obtained from two siamese encoders into
unified representations, which are prepared for the decoder. Experimental
caption results demonstrate the effectiveness of our feature fusing tactics
with exquisite discrepancies descriptions generated. The model of our work is
designed under the scenario of visual difference caption, contributing to that
the input format and evaluation are different with most previous works [6, 8,
9]. Moreover, a referee, which has been proved to be more suitable for our
task than automatic NLP metric, is introduced to evaluate our descriptor. We
first validate the reliability of our referee, who ranks the correct image
with 90.48% accuracy given ground truth from AMT 20K dataset, while it reached
at 82.34% accuracy given captions generated by our model. We also compare our
work with the state-of-the art of Su et al.[2] on the OID:Aircraft dataset,
achieving a score of 2.88 than their 2.72 evaluated with user study method.

Contributions of this paper. In summary, our work makes the following
contributions:

  * 1.

A novel caption problem is to tell the difference in the given image pair and
provide human-style description, which can be widely used in various
applications. In this paper, we develop a complete system for this problem,
containing datasets, caption model and evaluation metric.

  * 2.

We introduce a new large-scale dataset for fine-grained discrimination caption
named AMT 20K of nearly 32000 image pairs and relative annotation. Besides, an
additional dataset named OID:Aircraft is re-annotated with 9400 image pairs
based on the OID dataset [2]. Moreover, a low-cost method for data enrichment
is proposed. Fig 1 shows some representative samples of both datasets.

  * 3.

We propose a series of feature fusing modules for the visual differences
caption task, fusing multiple visual representations of either high or low
level into one, that a unified representation is consist of either high or low
level features, which can be adopted by most encoder-decoder caption models.

  * 4.

In this work, a referee model is introduced to evaluate our relative
captioner. Extensive experiments demonstrate that the referee is the most
suitable evaluation metric compared to automatic metric and user study.

##  2 Related Work

Caption Generation Task. Generating descriptions on target image has been
extensively studied in both Computer Vision and Natural Language Processing
[10, 11, 12, 13]. Early researches take advantage of sentence template and
heavily hand-designed systems [14, 15, 16], which limits the application so
far as to cause sensitivity to disturbance. Traditional approaches of
captioning image comprise Encoder-Decoder structure and end-to-end training
method. The complete framework extracts feature representation from pre-
trained deep convolutional networks ,then the language model decodes image
features to generate descriptive sentences. LSTM [17] and GRU [18] are favored
in demonstrating sequential relations among words. Proposed by Vinyals et al.
[3], this well-designed structure has been hitherto adopted by many state-of-
the-art caption generation frameworks [19, 20, 21].

Recently, attention-based models [4, 5, 22, 23, 24, 25] have achieved higher
performance by a large margin. Attentive mechanisms that embedded in neural
networks are capable of capturing local regions out of the full image. Looking
into regional details facilitates recognition to the image, which is in
accordance with human intuition [26]. Xu et al. [4] proposed two variants of
attention models. Moreover, they validate the use of attention with state-of-
the-art performance. You et al. [5] applied a policy that extracts richer
information from the image and couples them with an RNN selectively attend on
semantic attributes. Karpathy et al. [24] provided an image-sentence ranking
mechanism for visual-semantic alignment. Different from most of the one-image-
based caption models, our work contributes by learning the representation of
differences between image pairs.

Inspired by human attention as well, V. Mnih et al. [27] introduced a
Recurrent Attention Model that a deep recurrent neural architecture iterates
image features from previous steps, while DT-RAM [28] utilized learnable
parameters to govern whether stop the computation and output the results at
each time step. As to our formulation, attention mechanism effects both in
descriptions generating and differences emphasizing.

Image difference Detection. Though less popularity like relation detection
enjoys, present works along with remarkable outcome preview great potential in
visual comparison application. Yu et al. [29] proposed a local learning-to-
rank approach by which the model predicts which one exhibits the attribute
more from a fine-grained image pair. The team further their work [30, 31]
towards deciding whether these differences in attributes is perceptible, or
the relative strength of an attribute. The Attribute-based visual difference
has been studied in [2] as well. The determinant of relationships among
objects in image understanding is emphasized by Dai et al. [32] and Cheng et
al. [33] respectively.

Further image difference detection works include generating descriptions
afterward, for example, Liu et al. [34] and Su et al.[2]. To be specific, the
latter team utilized encoder-decoder framework to generate attribute phrases
in accordance with image pairs from OID Dataset[35], which we have modified so
that apply to our model (Section5.1). Such work

Similarly, Andreas et al.[1] and Vedantam et al.[6] treated the second image
as a distractor, while the former focused on scene description and the latter
features using training data that only describe a concept or an image in
isolation. Different from present works, we emphasise on the flexible
syntactical structure of our difference captioner that facilitate generating
“realistic” human-like difference description out of raw image pairs.

##  3 Model

Overview. Deep Neural Networks are powerful machine learning models that
achieve outstanding performance in sequence to sequence translation [36],
image caption [3, 4] and many other natural language processing fields. In
this work, we introduce an amusing issue about telling the difference of image
pair in a human-like style. Given a correct difference description $\bm{S}$
and the source image pair ($\bm{I}_{1}$, $\bm{I}_{2}$), this issue can be
modeled as finding the description that maximizes the following log
probability:

$\log p(\bm{S}|\bm{I}_{1},\bm{I}_{2})=\sum_{t=0}^{T}\log
p(\bm{S}_{t}|\bm{I}_{1},\bm{I}_{2};\bm{S}_{0},\bm{S}_{1},...,\bm{S}_{t-1})$
(1)

, where $\bm{S}_{t}$ is the word in the description at location $t$, and
$\bm{S}$ has $T$ words in total. In the training process, in order to maximize
the log probability of a correct description, we set the training objective
as:

$Objective=-\sum\limits_{i=1}^{N}{\log
p\left({{\bm{S}^{i}}|\bm{I}_{1}^{i},\bm{I}_{2}^{i}}\right)}$ (2)

, where $N$ denotes the number of the training set. Once training process is
completed, we inference difference description by finding the sentence
$\bm{\hat{S}}$ that maximizing the probability above:
${\bm{\hat{S}}}=\arg\mathop{\max}\limits_{\bm{S}}p(\bm{S}|\bm{I}_{1},\bm{I}_{2})$.

In this section, we proposed a pairwise generator model, shown in Fig 2, which
followed the state-of-the-art framework of generating image caption in a
features encoder-decoder structure. A feature extractor (Section 3.1) is used
as encoder to encode images to visual features. These pairwise features are
then fused utilizing a series of fusion tactics (described in Section 3.2). In
the last step, long short-term memory (LSTM) with attention mechanism (Section
3.3) as a decoder is adopted to realize the function shown in Equation 1.

###  3.1 Feature Extractor

As mentioned in Section 2, we use a Convolutional Neural Network (CNN) that
extracts representations of images. Our particular choice of CNN leads to deep
residual learning framework [37]. Known as Deep residual network (ResNet), it
features ”shortcut connections” that connect lower layers by skipping one or
plural layers and added to the output of stacked layers:

$\mathbf{y=\mathcal{F}(x,\left\\{W_{i}\right\\})+W_{s}x}$ (3)

, where $\mathcal{F}(x,\left\\{W_{i}\right\\})$ is the residual mapping to be
learned, and $\mathbf{W_{s}x}$ is a linear projection that shortcut
connections. ResNet, as one of the most competitive image classification
model, can provide the decoder with strong and stable representations.
Concretely, we juxtaposed two ResNets-101 initialized with a pre-trained model
on ImageNet [38], which share identical parameters with each other.

Compressed high-level representation benefits category classification task,
while properly preserved low-level representation is in favor of fine-grained
features like texture and color attributes hence facilitate our model for
difference detection. Therefore, both high-level and low-level representations
are adopted in our work. 1) We obtain the outputs of the last convolution
block as the low-level representation, denoting as
$\bm{C}\in\mathbb{R}^{k\times l\times l}$, where $k=2048$ represents the
channel number, $l$ represents the width and height of the feature maps; 2)
The high-level representation $\bm{fc}\in\mathbb{R}^{k}$ is obtained by
operating average pooling on $\bm{C}$. According to these two different
represents, we introduce a series of fusion tactics. Now all the visual
features are eligible for our fusion model.

Figure 2: Our visual descriptor framework. Pairwise images are fed into Visual
Feature Extractors (ResNet[37] in our work) respectively which share
parameters with each other to obtain visual features. After that, a series of
Feature Fusing Tactics are introduced to fuse features from two independent
images into one, which is obtained for Description Generator to tell the
difference. We illustrated four fusion methods to fusion pairwise features
from different point of view, introduced in Section 3.2.

###  3.2 Fusion tactics

In this part, the frameworks of four fusion methods will be illustrated,
followed by Section 5.4, where we reveal evaluation results and performance
analysis among them. The goal of the fusion layer is to combine each pair of
feature vectors or matrixs from image pairs into a holistic representation,
which can be adopted by mainstream caption models. These feature vectors and
matrixs are identical with the visual features mentioned in the last paragraph
of Section 3.1. As an effort to make a further step, we have experimented on
multiple methods that enable us to combine each pair of
$\bm{fc}_{1}/\bm{C}_{1}$ and $\bm{fc}_{2}/\bm{C}_{2}$ into a vector $\bm{r}$.
Some of them are unconvincing and show up with poor results, while the four
fusion methods listed below appear to be promising.

  * 1.

Tensor Concatenation Our first move from intuition is to directly concat two
feature vectors $\bm{fc}$ into vector $\bm{r}\in\mathbb{R}^{2k}$. Denote $[*]$
as the vector concat or matrix channel-stack manipulation, $\bm{r}$ is
obtained by:

$\bm{r}=[\bm{fc}_{1},\bm{fc}_{2}].$ (4)

  * 2.

Linear Neural Net The second choice of fusion layer is a simple but effective
network with affine transformation prior and $\rm ReLU$ activation afterwards.
Denoted as:

$\bm{r}={\rm ReLU}(\bm{W}[\bm{fc}_{1},\bm{fc}_{2}]+\bm{b})$ (5)

, where $\bm{W}\in\mathbb{R}^{2k\times k}$, and the result
$\bm{r}\in\mathbb{R}^{k}$.

  * 3.

Feature Sharing Net Thanks to the flexibility of visual feature tensors, it is
our third option that $\bm{fc}_{1}$ and $\bm{fc}_{2}$ have been affinely
transformed before multiplication with feature sharing matrix $\bm{M}$ with a
$\rm ReLU$ activation following:

$\bm{r}={\rm
RELU}\left\\{{\left({{\bm{W}_{1}}\cdot\bm{fc}_{1}+{\bm{b}_{1}}}\right)\cdot\bm{M}\cdot\left({{\bm{W}_{2}}\cdot\bm{fc}_{2}+{\bm{b}_{2}}}\right)}\right\\}$
(6)

, where ${\bm{W}_{1}},\bm{W}_{2}\in\mathbb{R}^{k\times m}$; parameters of
$\bm{M}\in\mathbb{R}^{m\times m\times m}$ will be updated when training. The
result $\bm{r}\in\mathbb{R}^{m}$.

  * 4.

Hyper Convolution Last fusion method achieves the prime outcome. Visual
features with spatial information $\bm{C}$, reserved from last convolution
layer of ResNet, is fed into our Hyper Convolution (HC) layer. In HC layer, we
concat $\bm{C}_{1}$ and $\bm{C}_{2}$ in the nominal channel dimension before a
set of $1\times 1$ convolution layers. Moreover, operations that batch
normalization and ReLU activation enable boosting of the training procedure.
The result $\bm{r}\in\mathbb{R}^{k}$ is denoted as:

$\bm{r}={\rm AvgPooling}\left\\{{{\rm RELU}\left({{\rm Bn}\left({{\rm
Conv}\left({\left[{{\bm{C}_{1}},{\bm{C}_{2}}}\right]}\right)}\right)}\right)}\right\\}.$
(7)

Once the fused vector $\bm{r}$ is obtained, the description generator is able
to decode $\bm{r}$ to tell the difference.

###  3.3 Description Generator

We leverage a single-layer n-timestep LSTM as our language model as well as
RNN decoder, by generating one word at each time step given attention
weight($\alpha_{t},t=1,...,n,\alpha_{t}\in\mathbb{R}^{d}$ ) and context
vector(${{\tilde{z}}_{t},t=1,...,n,{\tilde{z}}_{t}\in\mathbb{R}^{v\times
d}}$).

In order to derive ${\tilde{z}}_{t}$, two prerequisite variables should be
provided, which are $\rho_{i}$ and $\alpha_{i}$. Here,
$\rho_{i},i=1,...,d,\rho_{i}\in\mathbb{R}^{w}$ are local feature vectors,
sliced from $\Upsilon\in\mathbb{R}^{w\times d}$, where $\Upsilon$ is computed
by average 2d pooling from fused vector r in Section 3.2. Specifically, $d$ is
alterable from user setting, deciding how many local regions to scope within
the visual feature vector $\bm{r}$. Each $\rho_{i}$ is tied up with a positive
weight $\alpha_{i}$. The attention mechanism($f_{att}$), a multilayer
perceptron, derives each weight $\alpha_{i}$ which indicates each annotation
vector $\rho_{i}$’s relative importance, conditioned on previous hidden state
(${e_{ti}}={f_{att}}(\rho_{i},{h_{t-1}})$). Concretely, each $\alpha_{r}$ is
activated by softmax
function(${a_{tr}}={\exp({e_{tr}})}/({\sum\nolimits_{i=1}^{L}{\exp({e_{ti}})}})$).

Then, context vector ${\tilde{z}}_{t}$ is presented in a conditional
expectation while output word probability is acquired given the word
embeddings($E{y_{t-1}}\in\mathbb{R}^{1\times 1}$, 1 is the size of
vocabulary), hidden state($h_{t}$), and context vector(${{\tilde{z}}_{t}}$),
as introduced by Bahdanau et al.[39] and Xu et al.[4] respectively.

In nature, the attentive model is trained on fusion features and ground truth
sentences, so as to emphasize the most obvious difference from human
intuition.

##  4 Discriminating Referee

What accessible to other image captioning tasks could be obscure for ours. In
addition to applying automatic metrics (Section 5.2), we need an alternative
method to evaluate our results. The reason for this is that multiple notable
differences exist given two images and these objective discrepancies may vary
from image to image, but also some key points are indistinguishable for the
automatic metric. For example, the plane image pair in Fig 1 has multiple
notable such as that the former plane has less windows and is smaller, which
will cause sentences that focus on attributes out of ground truth get a worse
point with automatic matrices. Besides, for this image pair, the score of
automatic matrices is high if generator inferences ”facing right” as ”facing
left”, which is totally wrong in this task.

Also important is that although we introduced methods of caption generator and
a referee, it does not necessarily indicate that we are referring to a
generative adversarial structure [40]. Using the referee auxiliarily to train
our generator models by utilizing it as an adversarial loss shows a visible
increase in successful judgments of the referee, whereas, it has raised little
progress in automatic metric evaluations and also led to poor quality texts
according to human study. Moreover, allowing the referee to play both roles in
training and evaluation process provides inappropriate advantages to the
referee, which seems like cheating. The ultimate goal of our task is to
improve the readability and precision of generated captions, rather than
improve the accuracy of our referee’s judgments, for example.

Figure 3: Our referee framework. Visual feature pair and linguistic feature
are obtained from VGG-16 [41] and Text CNN [42] respectively. Then we define
the visual-linguistic score by vector multiplication between visual and
linguistic feature. Concatenating of two scores, the final score vector is
adopted with a distinguish classification to judge which image is described by
our descriptor.

Motivated by Su et al. [2], we introduced a discriminating referee for
evaluation process. In the referee task, the training data is the same with
the generator task. The referee takes $\bm{I_{1}}$, $\bm{I_{2}}$ and $\bm{S}$
as inputs, and inference index $y$ as ground truth. $y=0$ means the difference
description $\bm{S}$ is based on $\bm{I_{1}}$. We randomly exchange the
position of $\bm{I_{1}}$ and $\bm{I_{2}}$ to make ${y}\sim B(1,0.5)$. The
overall referee framework shows in figure 3. Annotate the visual net
(ResNet-18 [37]) as $\nu$ and the sentence-to-vector embedding net (Text CNN
[42]) as $\omega$, the visual-linguistic score $s_{vl}$ formulates as:

$s_{vl}=v^{T}l=\nu(\bm{I^{\prime}};\theta_{\nu})^{T}\omega(\bm{S};\theta_{\omega})$
(8)

, where $v$ represents visual feature, $l$ represents linguistic feature.
$\bm{I^{\prime}}$ shows the order of images that appears to referee. Network
$\nu$ and $\omega$ have linear layer in the last layer respectively to map
vectors of different size into the same size $k=1024$ before vector product.
Therefore, we get the probability of $\bm{S}$ based on $\bm{I_{1}^{\prime}}$:

$\begin{split}p\left({y=0|{\bm{I}_{1}^{\prime}},{\bm{I}_{2}^{\prime}},\bm{S}}\right)={{\exp\left({s_{vl}^{1}}\right)}\mathord{\left/{\vphantom{{\exp\left({s_{vl}^{1}}\right)}{\left({\exp\left({s_{vl}^{1}}\right)+\exp\left({s_{vl}^{2}}\right)}\right)}}}\right.\kern-1.2pt}{\left({\exp\left({s_{vl}^{1}}\right)+\exp\left({s_{vl}^{2}}\right)}\right)}}\end{split}$
(9)

. The target of referee is to find
$\Theta=\left\\{{{\theta_{\nu}},{\theta_{\omega}}}\right\\}$ that maximum the
probability as below:

$\begin{split}\Theta=\mathop{\arg\max}\limits_{\Theta}\\{{\left({1-y}\right)p\left({y=0|{\bm{I}_{1}^{\prime}},{\bm{I}_{2}^{\prime}},\bm{S}}\right)+}{y\left({1-p\left({y=0|{\bm{I}_{1}^{\prime}},{\bm{I}_{2}^{\prime}},\bm{S}}\right)}\right)}\\}.\end{split}$
(10)

It translates reference game into a two-category classification task, making
it trainable with the same data as our descriptor. The referee is now able to
judge whether the description our model generates tells the difference or not,
no matter whether the description has attributes outside the ground truth.
Besides, it’s sensitive to those key points mistakes.

##  5 Experiments

We evaluate our generator and referee on two different datasets - one is
collected by us and the other is collected based on existing dataset. In the
Section 5.1, the AMT 20K dataset, one major contribution of our paper, will be
introduced. Section 5.2 introduce the exploration of the best evaluation
metric in the difference caption task. Model study are shown in Section 5.3.
Section 5.4 demonstrate the experiment results and compare with the state of
the art respectively.

###  5.1 Dataset details

There are comparatively few datasets for fine-grained difference attribute
phrases in computer vision,

the situation is even worse for who focuses on difference caption of the image
pair. As far as we know, there is no such fine-grained difference description
dataset for our task at present. As a pathfinder in image difference caption,
we introduce two datasets, one is collected by ourselves, and the other is
transformed from the OID aircraft attribute phrases dataset [2].

AMT 20K Dataset. UT Zappos50K dataset [43] is a large shoe dataset for
pairwise attributes comparison task, consisting of 50,025 catalog images
collected from Zappos.com. Based on those 14764 chosen images from UT
Zappos50K dataset, we annotate image pairs with different descriptions to
collect our own AMT 20K dataset. The annotation method is shown in Fig 4,
which is released to a group of workers to annotate on the Internet. The
length of collected annotation is in the range between 1 to 13, with an
average number of 5.28, shown in Fig 4.

Figure 4: (a) The left figure is the annotation interface used to collect
difference captions; (b) The middle figure is the sentence length histograms
of images in AMT 20K dataset; (b) The right figure is the sentence length
histograms of images in OID:Aircraft dataset.

The annotations are organized into 26710 image pairs (12416 images) in
training set, and 5041 pairs (2348 images) in test set. Since that image pairs
are annotated by different workers without any constraint in advance such as
attribute range, sentence pattern and so on, the difference descriptions are
diverse of open-ended ”realistic” human-style. Besides, the annotations
contain various attributes/key points, such as color, texture, accessories,
size and so on, which enriches the visual comparison details compared to those
attribute-based datasets. More statistics of AMT 20K dataset and its
comparison with other datasets are given in Table 1.

Table 1: Comparison of our AMT 20K dataset with existing datasets

Dataset |  | Resolution
---
(w$\times$h)
ImgNum | PairNum | Max1 | Min1 | Avg1 | SPP1
AMT 20K | 280$\times$280 | 14764 | 31751 | 13 | 1 | 5.28 | 1
OID:Aircraft | 1200$\times$7562 | 7541 | 9400 | 21 | 5 | 8.22 | 13

  * 1

Max is the maximum sentence length; Min is the minimal sentence length; Avg is
the average sentence length; SPP: sequence number per pair.

  * 2

The Resolution has a random error of 0-20 pixels.

  * 3

The OID dataset refer to the vision of difference captions.

OID:Aircraft Dataset. Aiming to investigate models that understand fine-
grained object categories with rich descriptions, Vedaldi et al. [35]
introduced OID Aircraft Dataset of 7,413 airplanes annotated with attributes
according to components. Additionally, Su et al. [2] derived annotations on
images from OID Aircraft dataset, picked images pairs uniformly at random
within the OID aircraft dataset, and relied on human annotators to discover
the space of descriptive attribute phrases. The annotations are organized into
4700 image pairs (1851 images) in training set, 2350 pairs (1730 images) in
validation set, and 2350 pairs (2705 images) in test set.

In our task, since the attribute phrases are similar to sentences, we utilize
a series of templates to map two or three attribute phrases to difference
captions, according to the word formation and category of phrases.
Furthermore, obeying certain probability distribution, generated sentences
alter both in length and structure for the sake of descriptive variety. Once
the transformation is completed, three workers refine the sentences generated
by templates. By this tactic, difference captions are obtained in a convenient
way. The split of this dataset is followed the previous work.

Data enrichment. As mentioned above, the main challenges of difference caption
datasets are the costliness for data collection, which limited the scale. One
contribution of our paper is the proposal of a low-cost method for data
enrichment. The simplest way is template-based caption by generating template
sentence with attribute words or phrases. However, it contradicts our goal
that promote natural language generation of artificial intelligence. Instead,
we employ a neural language style transfer methods.

Style transfer has been popular in image processing [44, 45, 46]. Shen et al.
[47], taking a further step on textual style transfer , formulated a shared
latent content space($\mathcal{Z}$) between two sentence
domains($\mathcal{X}_{1},\mathcal{X}_{2}$), and sentences of contrary
sentiment that respectively compose $\mathcal{X}_{1}$ and $\mathcal{X}_{2}$
could be transferred bi-directionally. Thus, we utilized a lingual model for
sentence completion.

Table 2: Comparison between template-based captions and style transferred captions Template-based | Neural style transfer
---|---
has white color and is a big plane | is a bigger plane with white color
has a propeller engine and has multiple windows | has more windows with a propeller engine

First, we use attributes from original UT Zappos50K and OID dataset to
construct sentences of template-based structure (See left column of Table 2)
as $\mathcal{X}_{1}$, while difference descriptions have been provided by AMT
20K and updated OID dataset as $\mathcal{X}_{2}$.

Then, it is trained on two sentence domains following classical training
procedure. Finally, obtaining translations from raw phrases into coherent and
fluent descriptions (See right column of Table 2) by sending a larger set of
template-based sentences to the best-trained model. Despite that rare
grammatical errors do occur, it does not necessarily preclude the fruitful
achievement that marks effectiveness of our model.

###  5.2 Evaluation Metrics

To quantitatively evaluate the generated sentences, we introduced three
evaluation metrics for different objectives in our work, namely automatic
metric for machine translation, a discriminating referee and user study
method.

Automatic metric. Like other common caption generation tasks, we deploy
automatic metric for results reporting. Specifically, we tested on BLEU[48]
and ROUGE[49]. They are essentially based on computing unigram or n-gram from
the candidate sentence that are found in the reference sentence. Additionally,
BLEU multiplied by a brevity penalty to prevent very short candidates from
receiving too high a score while METEOR using the harmonic mean to combine
precision and recall, the latter which is proportional to the ratio that the
number of unigrams in the candidate to the number of ‘ in the reference.

Discriminating referee. As introduced in Section 4, an alternative evaluation
method is to use a carefully designed model, where generated sentences
$\bm{S}$ are given information and how well the referee could distinguish
within the image pair $({\bm{I}_{1}^{\prime}},{\bm{I}_{2}^{\prime}})$ whose
position is exchanged randomly. Evolved from automatic metric, our referee
focuses on whether the vital information is attended by the generator. For
example, candidate sentence ‘the plane on the left is a commercial plane’
resemble reference sentence ‘the plane on the left is a military plane’ and
scores high in the automatic metric such as BLEU. Unfortunately, it utterly
misapprehends the use of the plane in the current image.

GT results in Table. 3 and Table. 4 show the accuracy of our referee in the
task of
$({\bm{I}_{1}^{\prime}},{\bm{I}_{2}^{\prime}},{\bm{S}})\to\left\\{{0,1}\right\\}$,
compared with human referee. Due to the noise of datasets and the error of
human judgment, the accuracy of human referees is slightly lower than $100\%$.
The accuracy of our discriminating referee reaches nearly $90\%$, which is a
qualified result to the duties of the referee. The accuracy of the referee in
AMT 20K dataset are higher than that in the OID:Aircraft, thanks to the more
abundant visual difference in AMT 20K dataset.

User study. We resort to human evaluation as well, which is extensive and the
most reliable though time-consuming. We leverage a well-designed web page in
Fig 5 and request the workers to evaluate each generated sentence of the image
pair. Each sentence rank is independent, indicating that there could be
multiple or no perfect answer. We took the arithmetic mean of scores from a
group of graders. In this work, ten workers are involved for the evaluation of
both datasets with 100 data pairs per worker.

Figure 5: User study interface. According to the image pair, workers are asked
to rank difference captions in the left of figure with a range of 1 to 4,
where 4 is the highest rank for worker’s intuition.

###  5.3 Experiments Settings

In this work, we run our experiments on five GEFORCE GTX 1080 Ti GPUs, and the
implementation framework is Pytorch [50]. More details are described in the
following.

Generator Model. In the feature extractor, the output of ResNet-101
$C\in\mathbb{R}^{2048\times l\times l}$ and ${\bm{f}c}\in\mathbb{R}^{2048}$,
where $l=14$. In the Feature Sharing Net, $m=128$ for sharing matrix $M$. As
to the one-layer LSTM, the hidden size is set to 512, and the encoding size of
vocabulary is 512. When the attention module is used, the hidden size of the
attention MLP is also 512.

During the training process, the ResNet-101 is initialized with weights pre-
trained on ImageNet dataset [38]. Using Adam optimizer [51] with initial
learning rate $lr=0.0004$, $\beta_{1}=0.9$, $\beta_{2}=0.999$. The training
process is persisted 100 epoches with 64 batch size. During the testing
process, we use beam search with beam size 10 to generate difference captions.

Discriminating Referee. Both visual feature and linguistic are mapped to
$1024$ vector. In text CNN, there are three kernels with size 3, 4, 5
respectively, with 100 kernel number in common. During the training process,
the ResNet-18 is also initialized with pre-trained weights. We use Adam
optimizer with initial learning rate $lr=0.001$. Batch size is set to $16$ and
max epoch number is $50$. There is a batch normalization layer [52] in the
last of Text Net.

Data augmentation. Difference caption data are in the form of coupled images
along with the corresponding text description, making the data acquisition
difficult compared to other tasks. In order to conquer the over-fitting issue,
we apply data augmentation on both datasets. For both datasets, each image is
randomly cropped 10 pixels in the width and height respectively. In the AMT
20K dataset, random horizontal flips are employed additionally thanks to the
lack of ‘left’ and ‘right’ attributes. Besides, we add the identical image
pair data (3000 pairs for AMT dataset and 1000 for OID dataset) to improve the
robustness of our model. The identical image pair data don’t be augmented.
Besides, both datasets are enriched by 10% utilizing the method mentioned
before.

Table 3: Evaluation on different caption models with AMT 20K dataset

Models | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE_L | Referee |  | User
---
study
GT | - | - | - | - | - | 89.15% | 3.64
SATTC | 0.17 | 0.06 | 0.02 | 0.01 | 0.20 | 71.49% | 2.43
SATLNN | 0.18 | 0.06 | 0.02 | 0.01 | 0.21 | 76.45% | 2.68
SATFSN | 0.18 | 0.06 | 0.02 | 0.01 | 0.22 | 76.16% | 2.64
SATHC | 0.21 | 0.08 | 0.03 | 0.01 | 0.24 | 81.77% | 2.83

  * 1

GT: ground truth; SAT [4]: show, attend and tell model; SATTC: Tensor
Concatenation Net with SAT; SATLNN: Linear Neural Net with SAT; SATFSN:
Feature Sharing Net with SAT; SATHC: Hyper Convolution Net with SAT.

Figure 6: Difference caption results by our SATHC model. In each pair, the
upper sentence is the discrepancy description collected from human annotators,
which we regarded as ground truth, while the lower sentence is the generated
caption.

###  5.4 Model Analyze

Results on the AMT 20K dataset

Table 3 shows the results of different methods on the AMT 20K dataset. We
utilize various evaluation metric to evaluate our captioner, but take the user
study rank as the benchmark. The value of BLEU and ROUGE_L metric are low in
all models, and sightly reflect the performance compared to the referee and
user study. The reason is that the discrimination descriptions are various for
one pair image. Besides, the captions are usually small due to the minor
differences. Therefore, we use referee and user study rank instead automatic
metric to evaluate captioner. The Hyper Convolution Net achieves the best
performance in both Referee metric and human rank, reaching 82.34% and 2.88
respectively compared to the SATTC model of only 72.54% and 2.48 respectively.
It’s turned that representation fusion in the low-level is better than the
high-level.

Comparison with the state of the art on the OID dataset

We compare our results with the most recent state-of-the-art image difference
describing task of Su et al. [2], by user study method. They trained a _simple
speaker_(SS) that takes a single image as input and produces a description,
and a _discerning speaker_ (DS) that takes two images as input and produces a
single (or a pair of) description(s). Given that they only generated short
phrases, automatic metric in Section 5.2 is naturally not applicable. Although
we both developed a player-referee demo, listeners of Su et al. and
Discriminating Referee of our group are trained on different datasets with
discrepancy, and hence evaluations made by computer referee is not comparable.
Finally, comparison of accuracy marked by human graders, is shown is Table 4,
illustrating that our SATHC model achieved 2.88, compared to 2.72 of DS [2].
Table 5 demonstrates the traditional evaluation matrix result of the SATHC
method, which is higher than it in AMT dataset due to the sequence per image
is five in OID dataset, but only one in AMT dataset.

Table 4: Comparison of user study results with other methods Dataset | GT | SS | DS | SATTC | SATLNN | SATFSN | SATHC
---|---|---|---|---|---|---|---
Referee | 90.48% | - | - | 72.54% | 77.82% | 77.66% | 82.34%
User study | 3.71 | 2.65 | 2.72 | 2.48 | 2.65 | 2.68 | 2.88
Table 5: Traditional evaluation matrix of the SATHC method Matrix | ROUGE_L | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4
---|---|---|---|---|---
SATHC | 0.54 | 0.70 | 0.55 | 0.41 | 0.29

Generation Results

In each pair in the figure above, the upper sentence is the discrepancy
description collected from human annotators, which we regarded as ground
truth?, while the lower sentence is the generated caption The saliency of our
model is demonstrated as: a) It is as careful as human annotators that tiny
rivet could be named along with multiple other attributes (3rd of Shoe pairs
in Fig. 6). b) In addition to evident differences such as color, it is capable
of discovering subtle but instructive differences annotations even more
distinct than the ones from datasets (1st of Plane pairs in Fig. 6). This
figure also illustrates that n-gram recall-oriented automatic metric is
inappropriate for our task, because the model is permitted to disagree on
differences labeled by annotators and thus receiving poor accuracy, even if it
is actually correct. Furthermore, in Fig. 7, diverse relative captions from
5-best beam searching seek out the discrepancy independently, with various
visual points. It’s proved that our SATHC model has learned something deeper
with visual-linguistic representation, and is capable of providing a healthy
diversity.

Figure 7: Five-best sentences beam searched by our SATHC model with beam size
setting to 10. Bold phases indicate a novel discrepancy different from other
generations.

##  6 Conclusion

We have introduced AMT 20K, a large dataset of differential descriptions on
fine-grained image pairs. Based on its data, we proposed an encoder-decoder
framework with various feature fusing tactics for image difference detection
and description generation. Moreover, we enriched our dataset with textual
style transfer method unprecedentedly, proved to be reliable as the
substitution of human annotation. Overall, our model reached outstanding
results on difference caption task and achieved state-of-the-art performance.
In the future work, we would like to introduce more competent relative caption
models such as Generative Discriminative structure. Besides, more data will be
collected for our AMT Dataset.

## References

## References

  * [1] J. Andreas, D. Klein, Reasoning about pragmatics with neural listeners and speakers, arXiv preprint arXiv:1604.00562.
  * [2] J.-C. Su, C. Wu, H. Jiang, S. Maji, Reasoning about fine-grained attribute phrases using reference games, arXiv preprint arXiv:1708.08874.
  * [3] O. Vinyals, A. Toshev, S. Bengio, D. Erhan, Show and tell: A neural image caption generator, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3156–3164.
  * [4] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, Y. Bengio, Show, attend and tell: Neural image caption generation with visual attention, in: International Conference on Machine Learning, 2015, pp. 2048–2057.
  * [5] Q. You, H. Jin, Z. Wang, C. Fang, J. Luo, Image captioning with semantic attention, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4651–4659.
  * [6] R. Vedantam, S. Bengio, K. Murphy, D. Parikh, G. Chechik, Context-aware captions from context-agnostic supervision, in: Computer Vision and Pattern Recognition (CVPR), Vol. 3, 2017.
  * [7] R. Luo, B. Price, S. Cohen, G. Shakhnarovich, Discriminability objective for training descriptive captions, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [8] R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, T. Darrell, Natural language object retrieval, in: CVPR, IEEE, 2016, pp. 4555–4564.
  * [9] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, K. Murphy, Generation and comprehension of unambiguous object descriptions, in: CVPR, 2016, pp. 11–20.
  * [10] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, V. Goel, Self-critical sequence training for image captioning, arXiv preprint arXiv:1612.00563.
  * [11] Z. Ren, X. Wang, N. Zhang, X. Lv, L.-J. Li, Deep reinforcement learning-based image captioning with embedding reward, arXiv preprint arXiv:1704.03899.
  * [12] C. Gan, Z. Gan, X. He, J. Gao, L. Deng, Stylenet: Generating attractive visual captions with styles, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3137–3146.
  * [13] R. Kiros, Y. Zhu, R. Salakhutdinov, R. S. Zemel, A. Torralba, R. Urtasun, S. Fidler, Skip-thought vectors, arXiv preprint arXiv:1506.06726.
  * [14] G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, T. L. Berg, Babytalk: Understanding and generating simple image descriptions, IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (12) (2013) 2891–2903.
  * [15] M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, H. Daumé III, Midge: Generating image descriptions from computer vision detections, in: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, 2012, pp. 747–756.
  * [16] Y. Yang, C. L. Teo, H. Daumé III, Y. Aloimonos, Corpus-guided sentence generation of natural images, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2011, pp. 444–454.
  * [17] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (8) (1997) 1735–1780.
  * [18] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using rnn encoder-decoder for statistical machine translation, arXiv preprint arXiv:1406.1078.
  * [19] T. Yao, Y. Pan, Y. Li, Z. Qiu, T. Mei, Boosting image captioning with attributes, OpenReview 2 (5) (2016) 8.
  * [20] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, J. C. Niebles, Dense-captioning events in videos, in: Proceedings of the IEEE International Conference on Computer Vision, Vol. 1, 2017, p. 6.
  * [21] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, K. Murphy, Improved image captioning via policy gradient optimization of spider, arXiv preprint arXiv:1612.00370.
  * [22] J. Fu, H. Zheng, T. Mei, Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [23] Z. Yang, X. He, J. Gao, L. Deng, A. Smola, Stacked attention networks for image question answering, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [24] A. Karpathy, L. Fei-Fei, Deep visual-semantic alignments for generating image descriptions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3128–3137.
  * [25] S. Xu, Y. Cheng, K. Gu, Y. Yang, S. Chang, P. Zhou, Jointly attentive spatial-temporal pooling networks for video-based person re-identification, arXiv preprint arXiv:1708.02286.
  * [26] R. Desimone, J. Duncan, Neural mechanisms of selective visual attention, Annual review of neuroscience 18 (1) (1995) 193–222.
  * [27] V. Mnih, N. Heess, A. Graves, et al., Recurrent models of visual attention, in: Advances in neural information processing systems, 2014, pp. 2204–2212.
  * [28] Z. Li, Y. Yang, X. Liu, S. Wen, W. Xu, Dynamic computational time for visual attention, arXiv preprint arXiv:1703.10332.
  * [29] A. Yu, K. Grauman, Fine-grained visual comparisons with local learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 192–199.
  * [30] A. Yu, K. Grauman, Just noticeable differences in visual attributes, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2416–2424.
  * [31] A. Yu, K. Grauman, Semantic jitter: Dense supervision for visual comparisons via synthetic images, Tech. rep., Technical Report (2016).
  * [32] B. Dai, Y. Zhang, D. Lin, Detecting visual relationships with deep relational networks, arXiv preprint arXiv:1704.03114.
  * [33] G. Cheng, D. Gao, Y. Liu, J. Han, Multi-scale and discriminative part detectors based features for multi-label image classification., in: IJCAI, 2018, pp. 649–655.
  * [34] X. Liu, H. Li, J. Shao, D. Chen, X. Wang, Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data, arXiv preprint arXiv:1803.08314.
  * [35] A. Vedaldi, S. Mahendran, S. Tsogkas, S. Maji, R. Girshick, J. Kannala, E. Rahtu, I. Kokkinos, M. B. Blaschko, D. Weiss, Understanding objects in detail with fine-grained attributes, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3622–3629.
  * [36] I. Sutskever, O. Vinyals, Q. V. Le, Sequence to sequence learning with neural networks, in: Advances in neural information processing systems, 2014, pp. 3104–3112.
  * [37] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016\.
  * [38] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, Imagenet large scale visual recognition challenge, International Journal of Computer Vision 115 (3) (2015) 211–252.
  * [39] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473.
  * [40] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Advances in neural information processing systems, 2014, pp. 2672–2680.
  * [41] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, Computer Science.
  * [42] Y. Kim, Convolutional neural networks for sentence classification, arXiv preprint arXiv:1408.5882.
  * [43] A. Yu, K. Grauman, Fine-grained visual comparisons with local learning, in: Computer Vision and Pattern Recognition (CVPR), 2014.
  * [44] L. A. Gatys, A. S. Ecker, M. Bethge, A neural algorithm of artistic style, arXiv preprint arXiv:1508.06576.
  * [45] D. Ulyanov, V. Lebedev, A. Vedaldi, V. S. Lempitsky, Texture networks: Feed-forward synthesis of textures and stylized images., in: ICML, 2016, pp. 1349–1357.
  * [46] J. Johnson, A. Alahi, L. Fei-Fei, Perceptual losses for real-time style transfer and super-resolution, in: European Conference on Computer Vision, Springer, 2016, pp. 694–711.
  * [47] T. Shen, T. Lei, R. Barzilay, T. Jaakkola, Style transfer from non-parallel text by cross-alignment, in: Advances in Neural Information Processing Systems, 2017, pp. 6833–6844.
  * [48] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting on association for computational linguistics, Association for Computational Linguistics, 2002, pp. 311–318.
  * [49] C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, Text Summarization Branches Out.
  * [50] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer, Automatic differentiation in pytorch.
  * [51] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980.
  * [52] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: International conference on machine learning, 2015, pp. 448–456.


# Parity violation in the observed galaxy trispectrum

Pritha Paul1, Chris Clarkson1,2, and Roy Maartens2,3,4
1 Astronomy Unit, School of Physical & Chemical Sciences, Queen Mary
University of London, London E1 4NS, UK
2 Department of Physics & Astronomy, University of the Western Cape, Cape Town
7535, South Africa
3 National Institute for Theoretical and Computational Sciences, Cape Town
7535, South Africa
4 Institute of Cosmology & Gravitation, University of Portsmouth, Portsmouth
PO1 3FX, UK

###### Abstract

Recent measurements of the 4-point correlation function in large-scale galaxy
surveys have found apparent evidence of parity violation in the distribution
of galaxies. This cannot happen via dynamical gravitational effects in general
relativity. If such a violation arose from physics in the early Universe it
could indicate important new physics beyond the standard model, and would be
at odds with most models of inflation. It is therefore now timely to consider
the galaxy trispectrum in more detail. While the _intrinsic_ 4-point
correlation function, or equivalently the trispectrum, its Fourier
counterpart, is parity invariant, the _observed_ trispectrum must take
redshift-space distortions into account. Although the standard Newtonian
correction also respects parity invariance, we show that sub-leading
relativistic corrections do not. We demonstrate that these can be significant
at intermediate linear scales and are dominant over the Newtonian parity-
invariant part around the equality scale and above. Therefore when _observing_
the galaxy 4-point correlation function, we should expect to detect parity
violation on large scales.

###### keywords:

cosmology: theory. cosmology: large-scale structure of Universe.

## Introduction

A detection of parity violation in large-scale galaxy statistics could be a
signature of physics beyond the standard model and could provide information
about the early universe, dark matter, and dark energy. Parity inversion is a
point reflection, defined as reversing the sign at any event of each spatial
Cartesian coordinate axis, is a symmetry that is obeyed by most physical
processes. On cosmological scales, structure formation is dominated by
gravitation which is parity invariant in the dominant scalar sector. Hence,
probing parity violation in large-scale structure could also provide an
insight into early universe physics as we should expect it to be invariant
under a parity transformation. In order to provide constraints on parity
violation on large scales, we need to investigate observables that are
sensitive to parity. Observables that are constructed from scalar fields, such
as the galaxy density fluctuation, pose difficulties to being parity
sensitive. For example, the power spectrum, the Fourier counterpart of the $2$
point correlation function (2PCF) is not sensitive to parity. However, with
future galaxy surveys, such as DESI (www.desi.lbl.gov) and Euclid (www.euclid-
ec.org), we will be able to constrain higher-order statistical measures such
as the bispectrum and trispectrum. The intrinsic bispectrum or 3PCF is not
sensitive to a parity transformation because a triangle under reflection and
rotation by $\pi$ (i.e., a point reflection) is mapped back to itself. The
trispectrum, which is the Fourier counterpart of the 4-point correlation
function (4PCF), is the lowest-order statistic that can probe parity violation
in the scalar gravity sector.

A number of cosmological studies have considered the $4$PCF of the cosmic
microwave background (CMB). Most of the parity violation studies of the CMB
have been focused on the polarisation of the CMB (Kamionkowski & Souradeep,
2011; Shiraishi et al., 2011) or on gravitational waves (Nishizawa &
Kobayashi, 2018; Jeong & Kamionkowski, 2020; Orlando et al., 2021). The CMB
study of parity violation (Minami & Komatsu, 2020) has shown $2.5\sigma$
evidence of cosmic birefringence. Eskilt & Komatsu (2022) have presented a new
analysis of cosmic birefringence and found a statistical significance of
$3.6\sigma$. The equivalent 4PCF of large-scale structure has not been
explored as much. Recently, Cahn et al. (2021) have proposed using the $4$PCF
as a test for detecting cosmological parity violation in $3$D large-scale
structure. They have taken quadruples of galaxies forming a tetrahedron, which
is the lowest order $3$D shape which cannot be superimposed on its mirror
image by a rotation. As a result, an imbalance between the tetrahedrons and
their mirror images (assuming statistical isotropy) can provide information
about parity violating parity-odd modes in the $4$PCF from 3D large-scale
structure surveys. Philcox (2022) has done a blind test on the odd-parity part
of the 4PCF using the Baryonic Oscillation Spectroscopic (BOSS) CMASS
(Rodr\́mathrm{i\,}guez-Torres et al., 2016) sample of the Sloan Digital Sky
Survey (SDSS) III (Beutler et al., 2014). This test considered separations in
the range $20-160h^{-1}\mathrm{Mpc}$ and found $2.9\sigma$ evidence for the
detection of an odd-parity $4$PCF. Hou et al. (2023) have also measured these
parity-odd modes from the currently available spectroscopic sample, BOSS. They
have performed an analysis of two sets of data, DR$12$ LOWZ $(\bar{z}=0.32)$
and DR$12$ CMASS $(\bar{z}=0.57)$ (Slepian & Eisenstein, 2015; Cannon et al.,
2006). They have found a $3.1\sigma$ evidence and a $7.1\sigma$ evidence for
an odd-parity $4$PCF in the above data sets respectively.

These important findings imply that theoretical modelling of the galaxy
trispectrum should now be investigated in more detail. Within the standard
cosmology of simple inflation plus general relativity, there can be no parity
violation within the intrinsic trispectrum of the real space galaxy
distribution. However, galaxies are observed in redshift space on our past
lightcone, and not in real space. The Kaiser redshift-space distortions (RSD)
account for the first approximation in the mapping from real space to redshift
space (Scoccimarro et al., 1999). The usual Newtonian RSD term accounts for
much of these corrections on small scales, and these do not break the parity
invariance of the trispectrum. But on larger scales, a host of relativistic
corrections come into play which are important on large scales (McDonald,
2009; Yoo, 2010; Bonvin & Durrer, 2011; Bertacca et al., 2012; Bonvin, 2014;
Bertacca et al., 2014; Umeh et al., 2017; Giusarma et al., 2017; Breton et
al., 2018). All these effects beyond the Kaiser RSD are suppressed at least by
a factor of $\mathcal{H}/k$, where $\mathcal{H}$ is the comoving Hubble rate,
and $k$ the wavenumber.

In this work, we show that these relativistic contributions also break the
parity invariance of the trispectrum, and therefore need to be included in a
full analysis. Peculiar velocities and gravitational redshift are responsible
for breaking the cylindrical symmetry along the line of sight and generically
give rise to odd-multipoles in $n$-point statistics which break parity
invariance (Bonvin et al., 2014; Clarkson et al., 2019). These odd-multipoles
reflect these relativistic corrections which become more important as we go to
larger scales. In the trispectrum we show that the odd parity violating part
picks up these important effects at order $\mathcal{H}/k$, and are
surprisingly significant even on scales $k\sim 0.01-0.1h$ Mpc-1. While these
corrections are not present at leading order in $\mathcal{H}/k$ in the
monopole of the trispectrum (so are presumably unlikely to be the cause of the
parity violation in Hou et al. (2023)) they will play a role in future
trispectrum analyses, in particular in detecting large-scale relativistic
effects.

In this work, we provide a theoretical description of the observed galaxy
trispectrum in a relativistic framework. We take into account RSD, Doppler and
gravitational redshift effects in our analysis, up to third order in
perturbation theory which is necessary for a consistent description of the
dominant part of the trispectrum.

## Parity violation in the power- and bi-spectra

We first review how relativistic effects enter the power and bispectra, and
gather the relevant quantities we need for the trispectrum calculation. The
principal redshift space effect on galaxy number counts at first order is
given by the Kaiser RSD term,
$\delta_{g}(\bm{k})=(b_{1}+f\mu^{2})\delta(\bm{k})$, where
$\mu=\bm{n}\cdot\hat{\bm{k}}$, with $\bm{n}$ the line of sight direction, $f$
the growth rate, and $b_{1}$ the linear bias (we omit the functional
dependence on redshift here and below for convenience). The principal local
correction to this effect is a Doppler term (Kaiser, 1987; McDonald, 2009;
Challinor & Lewis, 2011; Bertacca et al., 2012) (see also Raccanelli et al.,
2018; Hall & Bonvin, 2017; Abramo & Bertacca, 2017) proportional to
$\bm{v}\cdot\bm{n}$, where $\bm{v}$ is the peculiar velocity:

$\delta_{g}(\bm{k})=\Big{(}b_{1}+f\mu^{2}+\mathrm{i\,}\mu{A}\,f\frac{\mathcal{H}}{k}\Big{)}\delta(\bm{k})=\mathcal{K}(\bm{k})\delta(\bm{k})\,,$
(1)

where ${A}={b_{\rm e}+{3}\Omega_{m}/2-3+(2-5s)(1-{1/r\mathcal{H}})}$. We
assume a $\Lambda$CDM background, so that
$\mathcal{H}^{\prime}/\mathcal{H}^{2}=1-3\Omega_{m}/2$, where $\mathcal{H}$ is
the conformal Hubble rate and $\Omega_{m}$ is the evolving matter density. The
evolution of comoving galaxy number density is $b_{\rm
e}=\partial{\ln}(a^{3}\bar{n}_{g})/\partial\ln a$ and
$s=-(2/5)\partial\ln\bar{n}_{g}/\partial\ln L$ is the magnification bias ($L$
is the threshold luminosity), while $r$ is the comoving radial distance
($\partial_{r}=\bm{n}\cdot\bm{\nabla}$).

Equation (1) tells us how each of the terms scales with $\mathcal{H}/k$. On
small scales, terms which are proportional to $\mathcal{H}/k$, are suppressed
but they become more important near and above the equality scale. Note that
the galaxy density contrast is no longer invariant under $\bm{k}\to-\bm{k}$
and it is complex. However, the power spectrum is real and remains parity
invariant:

$\big{\langle}\delta_{g}(\bm{k})\delta_{g}(-\bm{k})\big{\rangle}=\Big{[}\big{(}b_{1}+f\mu^{2}\big{)}^{2}+\Big{(}A\,f\mu{\mathcal{H}\over
k}\Big{)}^{2}\Big{]}\big{\langle}\delta(\bm{k})\delta(-\bm{k})\big{\rangle}\,,$

since $\mu_{-\bm{k}}=-\mu_{\bm{k}}$ means that the imaginary parts cancel.

The bispectrum is more complicated because it requires the density contrast at
second order. The galaxy bispectrum is defined in Fourier space by

$\displaystyle B_{g}(\mathbf{k}_{1},\mathbf{k}_{2},\bm{k}_{3})$
$\displaystyle=\Big{[}3{\mathcal{K}(\bm{k}_{1})\mathcal{K}({\bm{k}}_{2})\mathcal{K}^{(2)}(\mathbf{k}_{1},\mathbf{k}_{2},\bm{k}_{3})}$
$\displaystyle~{}~{}~{}\times
P(k_{1})P(k_{2})\Big{]}_{{\circlearrowright}_{\tiny{1\cdots 3}}}.$ (2)

Here $\circlearrowright$ denotes symmeterisation111We use this in the tensor
notation manner, i.e.,
$X(\bm{k}_{1}\cdots\bm{k}_{n})_{{\circlearrowright}_{\tiny{1\cdots
n}}}=\frac{1}{n!}\left[X(\bm{k}_{1}\cdots\bm{k}_{n})+\text{all perms
of~{}}\bm{k}_{1}\cdots\bm{k}_{n}\right]$ on $\bm{k}_{1\cdots 3}$ and
$\mathcal{K}^{(2)}$ is symmetric on $\bm{k}_{1,2}$, leaving 2 permutations.
$P$ is the matter power spectrum. The first-order kernel
$\mathcal{K}=\mathcal{K}_{\rm N}+\mathcal{K}_{\rm R}$ is given by the term in
brackets in (1). At second order, $\mathcal{K}^{(2)}=\mathcal{K}_{\rm
N}^{(2)}+\mathcal{K}_{\rm R}^{(2)}$, where the Newtonian kernel is (Verde et
al., 1998; Tellarini et al., 2016)

$\displaystyle\mathcal{K}_{\rm{N}}^{(2)}(\mathbf{k}_{1},\mathbf{k}_{2},\bm{k}_{3})$
$\displaystyle=b_{2}+b_{1}F_{2}(\mathbf{k}_{1},\mathbf{k}_{2})$
$\displaystyle~{}~{}~{}+f\mu_{3}^{2}\,G_{2}(\mathbf{k}_{1},\mathbf{k}_{2})+{\cal
Z}_{2}({\bm{k}_{1}},{\bm{k}_{2}})\,.$ (3)

Here $F_{2}({\bm{k}_{1}},{\bm{k}_{2}}),G_{2}({\bm{k}_{1}},{\bm{k}_{2}})$ are
the second-order density and velocity kernels, and ${\cal
Z}_{2}({\bm{k}_{1}},{\bm{k}_{2}})$ is the second-order RSD kernel. We use a
local bias model (Desjacques et al., 2018), but ignore tidal bias.

The leading relativistic correction at second order is (Bertacca et al., 2014;
Bertacca, 2015; Yoo & Zaldarriaga, 2014; Di Dio et al., 2014; Jolicoeur et
al., 2017; Di Dio & Seljak, 2019):

$\displaystyle\delta^{(2)}_{g{\rm
R}}=A\,\bm{v}^{(2)}\\!\\!\cdot\bm{n}+2{C}(\bm{v}\cdot\bm{n})\delta+2{{E}\over\mathcal{H}}(\bm{v}\cdot\bm{n})\partial_{r}(\bm{v}\cdot\bm{n})$
(4)
$\displaystyle~{}~{}~{}~{}~{}+2{b_{1}\over\mathcal{H}}\phi\,\partial_{r}\delta+{2\over\mathcal{H}^{2}}\big{[}\bm{v}\cdot\bm{n}\,\partial_{r}^{2}\phi-\phi\,\partial_{r}^{2}(\bm{v}\cdot\bm{n})\big{]}-{2\over\mathcal{H}}\partial_{r}(\bm{v}\cdot\bm{v}),$

where $\phi$ is the gravitational potential,
$C=b_{1}(A+f)+{b_{1}^{\prime}/\mathcal{H}}+2(1-{1/r\mathcal{H}}){\partial
b_{1}/\partial\ln L}$ and $E={4-2A-3\Omega_{m}/2}$. While most of the
corrections are peculiar velocity contributions, there is also a contribution
from the density gradient. The corrections in (4) all scale as
$(\mathcal{H}/k)\delta^{2}$ and so can be significant on scales approaching
equality and above. These corrections appear in the bispectrum in the kernel
(Jolicoeur et al., 2018),

$\displaystyle\mathcal{K}^{(2)}_{\mathrm{R}}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3})={\mathrm{i\,}}{\mathcal{H}}\Big{[}-{3}\mu_{1}{k_{1}\over
k_{2}^{2}}\Omega_{m}b_{1}+4\nu_{12}{\mu_{1}\over k_{2}}f^{2}$
$\displaystyle\qquad\quad+2{\mu_{1}\over k_{1}}{C}f-{3}\mu_{1}^{3}{k_{1}\over
k_{2}^{2}}\Omega_{m}f+{\mu_{1}^{2}\mu_{2}\over
k_{2}}{\Big{(}3\Omega_{m}-2Ef\Big{)}f}$
$\displaystyle\qquad\quad+{\mu_{3}\over
k_{3}}G_{2}(\bm{k}_{1},\bm{k}_{2}){A}f\Big{]}_{\circlearrowright_{1,2}},~{}~{}~{}~{}$
(5)

where $\nu_{ij}=\hat{\bm{k}}_{i}\cdot\hat{\bm{k}}_{j}$ and
$\mu_{i}=\hat{\bm{k}}_{i}\cdot\bm{n}$. The Newtonian kernel (3) scales as
$(\mathcal{H}/k)^{0}$, while the Doppler kernel (5) scales as
$(\mathcal{H}/k)$. More generally there are many other corrections to the
relativistic kernel but these scale as $(\mathcal{H}/k)^{2}$ or higher, and
are sub-dominant – see De Weerd et al. (2020) for details.

The bispectrum will be parity invariant provided it is invariant under a sign
change of each $\bm{k}_{i}$ – i.e., if the momentum triangle flips
orientation. Therefore, the parity-violating part of the bispectrum is found
by calculating

$\displaystyle B_{\text{odd}}(\mathbf{k}_{1},\mathbf{k}_{2},\bm{k}_{3})$
$\displaystyle=\frac{1}{2}\Big{[}B_{g}(\mathbf{k}_{1},\mathbf{k}_{2},\bm{k}_{3})$
$\displaystyle~{}~{}~{}~{}~{}-B_{g}(-\mathbf{k}_{1},-\mathbf{k}_{2},-\bm{k}_{3})\Big{]}\,.$
(6)

To see which contributions cause this to be non-zero, let us examine the
kernels in turn. First, under $\bm{k}_{i}\to-\bm{k}_{i}$ we have $k_{i}\to
k_{i}$ and $\mu_{i}\to-\mu_{i}$. Therefore contributions with odd powers of
$\mu_{i}$ will not necessarily be parity invariant. The Newtonian kernels
consist of terms with even powers of $\mu_{i}$ only (including zero), while
leading-order relativistic terms consist only of odd powers of $\mu_{i}$. To
order $(\mathcal{H}/k)$ which we consider here, in forming the bispectrum by
multiplying out the kernels $\mathcal{K}_{N}+\mathcal{K}_{R}$ we see that the
bispectrum structure is

$\displaystyle\text{Newtonian
terms}~{}\mathcal{O}(\mathcal{H}/k)^{0}\times\text{even powers
of}~{}\mu_{i}\text{'s}$ $\displaystyle+\,\text{Relativistic
terms}~{}\mathrm{i\,}\times\mathcal{O}(\mathcal{H}/k)^{1}\times\text{odd
powers of}~{}\mu_{i}\text{'s}\,.$

Consequently, it follows that

$B_{\text{odd}}\sim\text{Relativistic
terms}~{}\mathrm{i\,}\times\mathcal{O}(\mathcal{H}/k)^{1}\times\text{odd
powers of}~{}\mu_{i}\text{'s}\,,$

and that relativistic terms introduce parity violation in the observed
bispectrum. This is equivalent to the bispectrum having odd multipoles (De
Weerd et al., 2020) .

## Relativistic contributions to the trispectrum

While the large-scale distribution of galaxies is statistically isotropic and
homogeneous, for a redshift survey, this symmetry is broken due to our line of
sight. Standard (Newtonian) RSD that arise from peculiar velocities have
already been taken into account in studies of the trispectrum. The standard
RSD, which are due to peculiar velocities, give rise to the even multipoles in
the $4$PCF, reflecting a cylindrical symmetry about the line of sight – real
space spheres are distorted into redshift space ellipsoids. However, on large
scales, it is not enough to only take into account the standard RSD effects
because the relativistic terms also come into play, as is the case for the
bispectrum. Additional Doppler, gravitational redshift and other effects are
responsible for breaking this symmetry – redshift-space ellipsoids are further
distorted into egg-like shapes. This contribution to the observable changes
sign when $\bm{n}\to-\bm{n}$. When we consider correlations between quadruples
of galaxies, these effects give rise to an odd component in the 4PCF. In the
trispectrum, these give rise to an imaginary part which changes sign under a
parity transformation.

The imaginary part of the trispectrum is a direct consequence of relativistic
effects, which are not present in a Newtonian analysis. In order to compute
this trispectrum, it requires us to go up to third order in perturbation
theory, including terms with $\bm{v}^{(3)}\cdot\bm{n}$. In addition to this,
there is a large variety of coupling terms between different orders. These
terms include radial gradients of the potential and the density contrast, as
well as a host of Doppler contributions at each order, including transverse
velocities. The leading correction with all the local relativistic effects up
to order $\mathcal{H}/k$ was calculated in Di Dio & Seljak (2019)222Note: our
$\bm{n}$ is minus theirs; their convention
$\delta_{g}^{(1)}+\delta_{g}^{(2)}+\delta_{g}^{(3)}$, differs from ours:
$\delta_{g}^{(1)}+\delta_{g}^{(2)}/2+\delta_{g}^{(3)}/3!$.. All these
relativistic corrections are a consequence of the projection along the line of
sight $\bm{n}$ in redshift space.

We have neglected any terms that are integrated along the line of sight. These
include the lensing magnification and the integrated Sachs-Wolf effect. We
also adopt the standard plane-parallel approximation that is used on large
scales. However, wide-angle effects should be taken into account in a more
complete treatment, since they could be of a similar size to contributions we
consider here at redshifts $z\sim 1$ (Paul et al., 2023). We drop any terms
that scale as $(\mathcal{H}/{k})^{2}$ and higher, since these do not
contribute to the parity violation that we consider. (These will contribute
corrections to the parity violation only at order $(\mathcal{H}/{k})^{3}$.) We
also do not include any primordial non-Gaussianity.

In general, the trispectrum of the observed galaxy number density fluctuation
is defined by

$\displaystyle\big{\langle}\delta_{g}(\bm{k}_{1})\delta_{g}(\bm{k}_{2})\delta_{g}(\bm{k}_{3})\delta_{g}(\bm{k}_{4})\big{\rangle}$
$\displaystyle=(2\pi)^{3}T_{g}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3},\bm{k}_{4})$
$\displaystyle~{}~{}~{}\times\delta^{D}(\bm{k}_{1}+\bm{k}_{2}+\bm{k}_{3}+\bm{k}_{4}).$
(7)

At third order, the only combination of terms that contribute at tree level
are

$\displaystyle\big{\langle}\delta_{g}(\bm{k}_{1})\delta_{g}(\bm{k}_{2})\delta_{g}(\bm{k}_{3})\delta_{g}(\bm{k}_{4})\big{\rangle}=$
$\displaystyle\qquad\qquad~{}~{}\Big{[}4\langle\delta_{g}^{(1)}(\bm{k}_{1})\delta_{g}^{(1)}(\bm{k}_{2})\delta_{g}^{(1)}(\bm{k}_{3})\delta_{g}^{(3)}(\bm{k}_{4})\rangle$
$\displaystyle\qquad\qquad~{}~{}~{}~{}+6\langle\delta_{g}^{(1)}(\bm{k}_{1})\delta_{g}^{(1)}(\bm{k}_{2})\delta_{g}^{(2)}(\bm{k}_{3})\delta_{g}^{(2)}(\bm{k}_{4})\rangle\Big{]}_{\circlearrowright_{1\cdots
4}}.$ (8)

Symmetrisation gives 5 distinct permutations of type 1113 and 7 of type
1122.333Note that contributions of the form 1111 are just power spectra
squared and removed from the analysis, and contributions 1112 are zero under
the assumption of Gaussian initial conditions. Using Wick’s theorem, this
leads to

$\displaystyle T_{g}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3},\bm{k}_{4})=$ (9)
$\displaystyle\Big{[}\Big{\\{}4\mathcal{K}^{(3)}(-\bm{k}_{1},-\bm{k}_{2},-\bm{k}_{3},\bm{k}_{4})\mathcal{K}^{(1)}(\bm{k}_{1})\mathcal{K}^{(1)}(\bm{k}_{2})\mathcal{K}^{(1)}(\bm{k}_{3})P(k_{3})$
$\displaystyle~{}+3\mathcal{K}^{(1)}(\bm{k}_{1})\mathcal{K}^{(1)}(\bm{k}_{2})\mathcal{K}^{(2)}(-\bm{k}_{1},\bm{k}_{1}+\bm{k}_{3},\bm{k}_{3})$
$\displaystyle~{}~{}\times\mathcal{K}^{(2)}(-\bm{k}_{2},-\bm{k}_{1}\\!-\bm{k}_{3},\bm{k}_{4})P(|\bm{k}_{1}+\bm{k}_{2}|)\\!\Big{\\}}P(k_{1})P(k_{2})\Big{]}_{\circlearrowright_{1\cdots
4}}.$

We break down the third-order kernel into Newtonian and relativistic parts,
$\mathcal{K}^{(3)}=\mathcal{K}^{(3)}_{\mathrm{N}}+\mathcal{K}^{(3)}_{\mathrm{R}}.$
The Newtonian kernel is

$\displaystyle\mathcal{K}^{(3)}_{\mathrm{N}}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3},\bm{k}_{4})=b_{1}F_{3}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3})-b_{2}F_{2}(\bm{k}_{1},\bm{k}_{2})+\frac{1}{6}b_{3}$
$\displaystyle+\frac{\mu_{3}(\mu_{3}k_{+}\mu_{4}k_{4})f}{2k_{3}}\Big{[}2b_{1}F_{2}(\bm{k}_{1},\bm{k}_{2})+b_{2}+2\mu_{4}^{2}fG_{2}(\bm{k}_{1},\bm{k}_{2})\Big{]}$
$\displaystyle+\frac{(\mu_{1}k_{1}+\mu_{2}k_{2})(\mu_{1}k_{1}+\mu_{2}k_{2}+\mu_{3}k_{3})f}{|\bm{k}_{1}+\bm{k}_{2}|^{2}}G_{2}(\bm{k}_{1},\bm{k}_{2})(b_{1}+\mu_{3}^{2}f)$
$\displaystyle+\frac{\mu_{1}\mu_{2}\mu_{3}}{6k_{1}k_{2}k_{3}}(\mu_{1}k_{1}+\mu_{2}k_{2}+\mu_{3}k_{3})^{3}f^{3}$
$\displaystyle+\frac{\mu_{2}\mu_{3}}{2k_{2}k_{3}}(\mu_{1}k_{1}+\mu_{2}k_{2}+\mu_{3}k_{3})^{2}f^{2}+\mu^{2}_{4}fG_{3}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3}).$
(10)

Here $F_{3}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3})$ and
$G_{3}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3})$ are the third order density and
velocity kernels (Jain & Bertschinger, 1994). The relativistic correction to
(Relativistic contributions to the trispectrum) in Fourier space follows from
Di Dio & Seljak (2019) and the full expression is given in the Appendix (a
follow-up paper will present the detailed derivation). The Newtonian kernel is
proportional to $(\mathcal{H}/{k})^{0}$, while the relativistic kernel is
proportional to $(\mathcal{H}/{k})^{1}$. We have verified numerically that the
third-order relativistic kernel is required for an accurate calculation of the
trispectrum, as the 1113 contribution is similar to the 1122, depending on the
configuration.

### Parity violation in the trispectrum

The parity-violating part of the trispectrum can be found by calculating

$\displaystyle T_{\text{odd}}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3},\bm{k}_{4})$
$\displaystyle=\frac{1}{2}\Big{[}T_{g}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3},\bm{k}_{4})$
$\displaystyle~{}~{}~{}~{}~{}~{}-T_{g}(-\bm{k}_{1},-\bm{k}_{2},-\bm{k}_{3},-\bm{k}_{4})\Big{]},$
(11)

and we compare this to the parity-preserving part,

$\displaystyle T_{\text{even}}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3},\bm{k}_{4})$
$\displaystyle=\frac{1}{2}\Big{[}T_{g}(\bm{k}_{1},\bm{k}_{2},\bm{k}_{3},\bm{k}_{4})$
$\displaystyle~{}~{}~{}~{}~{}~{}+T_{g}(-\bm{k}_{1},-\bm{k}_{2},-\bm{k}_{3},-\bm{k}_{4})\Big{]}.$
(12)

We expect $T_{\text{odd}}\neq 0$ for the same reason that the observed
bispectrum is parity violating: because the contributions with odd powers of
$\bm{n}$ will not be parity invariant. The Newtonian kernels have only even
powers of $\bm{n}$ and consequently of $\mu_{i}$. 444By this we mean that any
term with $\mu_{1}^{i_{1}}\mu_{2}^{i_{2}}\mu_{3}^{i_{3}}\mu_{4}^{i_{4}}$ has
$i_{1}+i_{2}+i_{3}+i_{4}$ an even number or zero. On the other hand, the terms
in the relativistic kernels have all odd powers of $\bm{n}$ and odd powers of
$\mu_{i}$. The terms responsible for the odd powers in the relativistic
kernels include $\bm{v}\cdot\bm{n}$ and $\partial_{r}\delta$. As a result, the
terms that scale as $(\mathcal{H}/k)^{1}$ will introduce parity violation in
the trispectrum, while the terms that scale $(\mathcal{H}/k)^{0}$ will not. We
calculate the even and odd parts neglecting terms $(\mathcal{H}/k)^{2}$ and
higher, which is consistent with our truncation of the relativistic kernels.
This ensures that the even part is purely Newtonian, $O(\mathcal{H}/k)^{0}$,
while the odd part will arise from a coupling of the leading relativistic part
with Newtonian terms. More generally, relativistic terms would enter the even
part at order $(\mathcal{H}/k)^{2}$ (the same order as primordial non-
Gaussianity at the perturbative order that we are considering).

We now turn to calculating the amplitude of the odd parity part relative to
the even Newtonian part. In order to do this we write the geometry of the
quadrilateral as follows. We let
$\bm{k}=\bm{k}_{1}+\bm{k}_{2}=-\bm{k}_{3}-\bm{k}_{4}$ be the vector which
breaks the quadrilateral into two triangles. We choose the $z$ axis in the
direction of $\bm{k}$ and place the triangle $\bm{k}-\bm{k}_{1}-\bm{k}_{2}=0$
in the $x=0$ plane. Let $\Theta$ be the angle between the $z$-axis and
$\bm{k}_{2}$ and $\Phi$ be the angle between the $z$-axis and $\bm{k}_{3}$.
The angle between the 2 quadrilateral triangle flaps is $\Psi$. Then the shape
is fixed by the set $k,s=k_{2}/k,t=k_{3}/k,\Theta,\Phi,\Psi$ implying

$\displaystyle
k_{1}=k\sqrt{1+s^{2}+2s\cos\Theta}\,,~{}~{}k_{4}=k\sqrt{1+t^{2}+2t\cos\Phi}\,.$
(13)

Finally $\bm{n}$ is an arbitrary vector with spherical coordinates
$(\theta,\phi)$.

In order to demonstrate the amplitude of the odd-parity modes, we consider two
types of survey which affect the relativistic terms via the magnification and
evolution biases. The first survey is an SKA-like intensity mapping (IM) of 21
cm neutral hydrogen radio emission at $z=1$. This has bias parameters
$b_{1}=0.856,b_{2}=-0.321,b_{1}^{\prime}=-0.5\times
10^{-4},b_{e}=-0.5,b_{e}^{\prime}=0,s=2/5$ (Umeh et al., 2016; Fonseca et al.,
2018) (we ignore tidal bias and set third-order bias to zero for simplicity).
Second is a Euclid-like optical/infrared spectroscopic survey, with
$b_{1}=1.3,b_{2}=-0.74,b_{1}^{\prime}=-1.6\times
10^{-4},b_{e}=-4,b_{e}^{\prime}=0,s=-0.95$ (Camera et al., 2018; Yankelevich &
Porciani, 2019). (We set $\partial b_{1}/\partial\ln L=0$ for simplicity.) The
$\Lambda$CDM model has parameters
$\Omega_{m0}=0.31,h=0.68,f_{\text{baryon}}=0.157,n_{s}=0.968$. Plots are
presented using linear power spectra with the fitting formula of Eisenstein &
Hu (1998).

The trispectrum is a function of many variables, so we shall consider a few
cases to illustrate the behaviour we find. We consider the behaviour as a
function of scale, viewing orientation, and tetrahedron shape.

Figure 1: The odd (dashed) and even (solid) trispectra as a function of scale,
for both Euclid-like and SKA-like surveys. The geometry – flat quadrilateral
with the line of sight along $\bm{k}$ – is shown on the inset. In both cases,
the odd part is dominant on large scales, and important even on intermediate
scales. (Note the scaling on the vertical axis is linear.)

Figure 1 displays the odd and even parts of the trispectrum for both Euclid-
and SKA-like surveys at $z=1$. These are for a flattened configuration with
the tetrahedron lying in a plane ($\Psi=\pi$), viewed along the folding vector
$\bm{k}$ (the $z$-axis in our coordinates). We find in this example that
corrections for the Euclid-like case are larger than 10% for $k\lesssim 0.06h$
Mpc-1, and for the SKA-like survey, $k\lesssim 0.04h$ Mpc-1. The parity-
violating part is similar in size to the parity-preserving part for $k\lesssim
0.02h$ Mpc-1 in the Euclid-like case and $k\lesssim 0.008h$ Mpc-1 for SKA-
like. Even for scales $k\sim 0.1h$ Mpc-1, the parity-violating contribution is
$\sim$5%.

Figure 2: The ratio of the odd to even trispectrum for a SKA-like survey as a
function of scale and viewing orientation $\theta$ ($\phi=0$). Note the
colouring is a log scale, and the dark red curve is where the even part
changes sign.

In Fig. 2 we show the ratio of the odd- to even-parity trispectrum for an SKA-
like survey at $z=1$. We fix the shape of the quadrilateral and change the
scale and viewing orientation as indicated. We see that on all scales
$k\lesssim 0.1h$ Mpc-1, the parity-violating part is substantial, well over
10% – and is clearly dominant above the equality scale.

Figure 3: The odd and even parts of the trispectrum as a function of
tetrahedron shape for a Euclid-like survey at $k=0.01h$ Mpc-1. For fixed
double-triangle geometry, we vary the opening angle $\Psi$, and for fixed
opening angle, we vary the angle of the $\bm{k}_{2}$ vector with respect to
$\bm{k}$ (the $z$-axis).

Figure 3 investigates the effect of changing the tetrahedron shape for fixed
scale and viewing angle. Again, we see that the odd-parity part is dominant in
many cases, and never negligible.

## Conclusions

We have investigated for the first time the effect that large-scale
relativistic RSD have on the observed galaxy trispectrum. These relativistic
projection effects induce parity violation even in the case of a primordial
trispectrum that is parity preserving. The effects that need to be included
are considerably more complicated than in the Newtonian RSD case, and a third-
order analysis is required. The leading relativistic contribution is of order
$\mathcal{H}/k$ and changes sign under $\bm{k}\to-\bm{k}$, thus breaking the
parity invariance of the Newtonian contribution. We have shown that the
changes are significant, and cannot necessarily be neglected, even for scales
$k\sim 0.1h$ Mpc-1, with corrections around 5%, becoming even more significant
on larger scales – and dominant on equality scales and above. Although the
monopole does not have this parity violation as a result of the relativistic
contributions, the fact that the dipole has such a significant contribution
implies that measurements of the monopole have to be very careful in order to
remove possible contaminants which are naturally parity-violating.

## Acknowledgements

We thank Dionysis Karagiannis and Scott Melville for discussions. CC is
supported by STFC Consolidated Grant ST/T000341/1. RM is supported the South
African Radio Astronomy Observatory (SARAO) and the National Research
Foundation (Grant No. 75415).

## References

  * Abramo & Bertacca (2017) Abramo L. R., Bertacca D., 2017, Phys. Rev., D96, 123535
  * Bertacca (2015) Bertacca D., 2015, Class. Quant. Grav., 32, 195011
  * Bertacca et al. (2012) Bertacca D., Maartens R., Raccanelli A., Clarkson C., 2012, JCAP, 10, 025
  * Bertacca et al. (2014) Bertacca D., Maartens R., Clarkson C., 2014, JCAP, 1409, 037
  * Beutler et al. (2014) Beutler F., et al., 2014, MNRAS, 443, 1065
  * Bonvin (2014) Bonvin C., 2014, Class. Quant. Grav., 31, 234002
  * Bonvin & Durrer (2011) Bonvin C., Durrer R., 2011, Phys. Rev., D84, 063505
  * Bonvin et al. (2014) Bonvin C., Hui L., Gaztañaga E., 2014, Phys. Rev., D89, 083535
  * Breton et al. (2018) Breton M.-A., Rasera Y., Taruya A., Lacombe O., Saga S., 2018, MNRAS, 483, 2671
  * Cahn et al. (2021) Cahn R. N., Slepian Z., Hou J., 2021 (arXiv:2110.12004)
  * Camera et al. (2018) Camera S., Fonseca J., Maartens R., Santos M. G., 2018, MNRAS, 481, 1251
  * Cannon et al. (2006) Cannon R., et al., 2006, MNRAS, 372, 425
  * Challinor & Lewis (2011) Challinor A., Lewis A., 2011, Phys. Rev., D84, 043516
  * Clarkson et al. (2019) Clarkson C., De Weerd E. M., Jolicoeur S., Maartens R., Umeh O., 2019, MNRAS, 486, L101
  * De Weerd et al. (2020) De Weerd E. M., Clarkson C., Jolicoeur S., Maartens R., Umeh O., 2020, JCAP, 05, 018
  * Desjacques et al. (2018) Desjacques V., Jeong D., Schmidt F., 2018, Phys. Rept., 733, 1
  * Di Dio & Seljak (2019) Di Dio E., Seljak U., 2019, JCAP, 04, 050
  * Di Dio et al. (2014) Di Dio E., Durrer R., Marozzi G., Montanari F., 2014, JCAP, 12, 017
  * Eisenstein & Hu (1998) Eisenstein D. J., Hu W., 1998, Astrophys. J., 496, 605
  * Eskilt & Komatsu (2022) Eskilt J. R., Komatsu E., 2022, Phys. Rev., D106, 063503
  * Fonseca et al. (2018) Fonseca J., Maartens R., Santos M. G., 2018, MNRAS, 479, 3490
  * Giusarma et al. (2017) Giusarma E., Alam S., Zhu H., Croft R. A. C., Ho S., 2017 (arXiv:1709.07854)
  * Hall & Bonvin (2017) Hall A., Bonvin C., 2017, Phys. Rev., D95, 043530
  * Hou et al. (2023) Hou J., Slepian Z., Cahn R. N., 2023, MNRAS, 522, 5701
  * Jain & Bertschinger (1994) Jain B., Bertschinger E., 1994, Astrophys. J., 431, 495
  * Jeong & Kamionkowski (2020) Jeong D., Kamionkowski M., 2020, Phys. Rev. Lett., 124, 041301
  * Jolicoeur et al. (2017) Jolicoeur S., Umeh O., Maartens R., Clarkson C., 2017, JCAP, 09, 040
  * Jolicoeur et al. (2018) Jolicoeur S., Umeh O., Maartens R., Clarkson C., 2018, JCAP, 03, 036
  * Kaiser (1987) Kaiser N., 1987, MNRAS, 227, 1
  * Kamionkowski & Souradeep (2011) Kamionkowski M., Souradeep T., 2011, Phys. Rev., D83, 027301
  * McDonald (2009) McDonald P., 2009, JCAP, 11, 026
  * Minami & Komatsu (2020) Minami Y., Komatsu E., 2020, Phys. Rev. Lett., 125, 221301
  * Nishizawa & Kobayashi (2018) Nishizawa A., Kobayashi T., 2018, Phys. Rev., D98, 124018
  * Orlando et al. (2021) Orlando G., Pieroni M., Ricciardone A., 2021, JCAP, 03, 069
  * Paul et al. (2023) Paul P., Clarkson C., Maartens R., 2023, JCAP, 04, 067
  * Philcox (2022) Philcox O. H., 2022, Phys. Rev., D106, 063501
  * Raccanelli et al. (2018) Raccanelli A., Bertacca D., Jeong D., Neyrinck M. C., Szalay A. S., 2018, Phys. Dark Univ., 19, 109
  * Rodr\́mathrm{i\,}guez-Torres et al. (2016) Rodr\́mathrm{i\,}guez-Torres S. A., et al., 2016, MNRAS, 460, 1173
  * Scoccimarro et al. (1999) Scoccimarro R., Zaldarriaga M., Hui L., 1999, Astrophys. J., 527, 1
  * Shiraishi et al. (2011) Shiraishi M., Nitta D., Yokoyama S., 2011, Prog. Theor. Phys., 126, 937
  * Slepian & Eisenstein (2015) Slepian Z., Eisenstein D. J., 2015 (arXiv:1510.04809)
  * Tellarini et al. (2016) Tellarini M., Ross A. J., Tasinato G., Wands D., 2016, JCAP, 06, 014
  * Umeh et al. (2016) Umeh O., Maartens R., Santos M., 2016, JCAP, 03, 061
  * Umeh et al. (2017) Umeh O., Jolicoeur S., Maartens R., Clarkson C., 2017, JCAP, 03, 034
  * Verde et al. (1998) Verde L., Heavens A. F., Matarrese S., Moscardini L., 1998, MNRAS, 300, 747
  * Yankelevich & Porciani (2019) Yankelevich V., Porciani C., 2019, MNRAS, 483, 2078
  * Yoo (2010) Yoo J., 2010, Phys. Rev., D82, 083508
  * Yoo & Zaldarriaga (2014) Yoo J., Zaldarriaga M., 2014, Phys. Rev., D90, 023513